Re: [Tails-dev] Automated tests specification

Hi,

On Wed, Aug 26, 2015 at 03:38:19PM +0200, anonym wrote:
> On 06/25/2015 11:41 AM, bertagaz wrote:
>
> Looks great! For the record, I looked at the spec as of commit e70f8e7.

Thanks! I'm cheating, most of the work has already been done when we
designed the autobuilds. :D

> > # Facts
> >
> > Running the full test suite on 1 isotester hosted on Lizard takes
> > around 8 hours.
>
> FYI, with the test/6094-improved-snapshots branch (previously
> test/wip-improved-snapshots) a full run on lizard takes ~350 minutes, so
> a bit less than six hours. On my laptop it takes around half that, at
> 180 minutes per full run. It seems

I've seen it going up to a bit more thant 400 minutes, depending on the
network.

> > We intend to run 4 isotesters, so at the moment we would be able
> > to run 12 full test suites per day.
>
> So with test/6094-improved-snapshots we can run 24/6 * 4 = 16 full runs
> per day! :)
>
> OTOH, we'll keep on adding more tests, so these figures will eventually
> becomes obsolete. Until the end of 2015 (when we "probably" get more
> hardware) I think we can assume that the run time of a full test suite
> will not increase by more than 33% due to new tests, but if we assume
> this worst case scenario, then with the test/6094-improved-snapshots
> branch we end up at your 12 runs per day number again.

I came up to the same conclusions. Let's hope we're right. And we WILL
have new hardware for that anyway. :)

> > Scenario 1 : reviewer
>
> vs.
>
> > Scenario 2 : developer
>
> The current proposal seems to be to only start the automated test run of
> a feature branch when it is marked "Ready for QA". This has overloaded
> the meaning of this status so it no longer alone means that the branch
> is ready for review'n'merge; the reviewer also has to wait until the
> automated tester posts the result to the ticket.
>
> We could get rid of this ambiguity by splitting "Ready for QA" into
> "Ready for (automated) testing" (RFT) and "Ready for review" (RFR). Example:
>
> Let's say I have created a new feature branch and think I'm done. I
> assign it to intrigeri (who I want to review it in the end) and mark it
> RFT. And automated build is triggered, and then a full run of the test
> suite. Let's say the test suite fails. Then it gets reassigned to me and
> marked "Dev needed". I fix the issue (which probably was easier to spot
> for me than it would be for intrigeri, since I did the implementation)
> assign it to intrigeri and mark it RFT again. This time the test suite
> passes, and then the ticket is marked RFR automatically. Also, if I run
> the test suite myself and see it pass, then I can just mark it RFR directly.

I've think about this issue too. My own conclusion was to add a new
Custom fiels in redmine, that Jenkins would use, so something similar to
yours. The dev mark the ticket as ReadyForQA, then Jenkins run the test
suite on it and send its report modifying that field accordingly.

But well, the redmine part, as for the automated builds, is a SHOULD, so
it will probably be worked later once the MUST are implemented, a bit
like #9614 ("phase two") and children for automated builds (see #9719).

> I do not know if this complicates the current plans too much, so perhaps
> this could be a future improvement. Or perhaps the current plan will be
> good enough in practice. I suppose it's bad in itself to introduce yet
> another QA status, complicating things.
>
> Also a question:
>
> > And the developer who proposed the merge must be notified
> > And the ticket should be reassigned to the branch submitter
>
> Is deciding "the developer who proposed the merge" and "the branch
> submitter" actually easy? We do not automatically get a mapping between
> Git authors and Redmine users. And parsing the redmine ticket history
> (e.g. who marked the ticket a certain QA status last?) also seems
> hairy, but perhaps it's "easy" via some API (SQL?) that I'm unaware of?

Yes, the redmine <-> Jenkins interaction is not so clear yet. I'm
thinking about it and working on it as a procrastination right now, I
have other task to finish before.
For now the only idea we came up with was to host a file that would have
this commit author <-> redmine user binding.
But to be honest, on the first quick check I made, IIRC all of the devs
had a easy binding: the left part of their email address was
corresponding to their Redmine account.

bert.