On 06/25/2015 11:41 AM, bertagaz wrote:
> I've prepared a blueprint to start this discussion and take notes of the
> decisions:
> https://tails.boum.org/blueprint/automated_builds_and_tests/automated_tests_specs/
Looks great! For the record, I looked at the spec as of commit e70f8e7.
> # Facts
>
> Running the full test suite on 1 isotester hosted on Lizard takes
> around 8 hours.
FYI, with the test/6094-improved-snapshots branch (previously
test/wip-improved-snapshots) a full run on lizard takes ~350 minutes, so
a bit less than six hours. On my laptop it takes around half that, at
180 minutes per full run. It seems
> We intend to run 4 isotesters, so at the moment we would be able
> to run 12 full test suites per day.
So with test/6094-improved-snapshots we can run 24/6 * 4 = 16 full runs
per day! :)
OTOH, we'll keep on adding more tests, so these figures will eventually
becomes obsolete. Until the end of 2015 (when we "probably" get more
hardware) I think we can assume that the run time of a full test suite
will not increase by more than 33% due to new tests, but if we assume
this worst case scenario, then with the test/6094-improved-snapshots
branch we end up at your 12 runs per day number again.
> Scenario 1 : reviewer
vs.
> Scenario 2 : developer
The current proposal seems to be to only start the automated test run of
a feature branch when it is marked "Ready for QA". This has overloaded
the meaning of this status so it no longer alone means that the branch
is ready for review'n'merge; the reviewer also has to wait until the
automated tester posts the result to the ticket.
We could get rid of this ambiguity by splitting "Ready for QA" into
"Ready for (automated) testing" (RFT) and "Ready for review" (RFR). Example:
Let's say I have created a new feature branch and think I'm done. I
assign it to intrigeri (who I want to review it in the end) and mark it
RFT. And automated build is triggered, and then a full run of the test
suite. Let's say the test suite fails. Then it gets reassigned to me and
marked "Dev needed". I fix the issue (which probably was easier to spot
for me than it would be for intrigeri, since I did the implementation)
assign it to intrigeri and mark it RFT again. This time the test suite
passes, and then the ticket is marked RFR automatically. Also, if I run
the test suite myself and see it pass, then I can just mark it RFR directly.
I do not know if this complicates the current plans too much, so perhaps
this could be a future improvement. Or perhaps the current plan will be
good enough in practice. I suppose it's bad in itself to introduce yet
another QA status, complicating things.
Also a question:
> And the developer who proposed the merge must be notified
> And the ticket should be reassigned to the branch submitter
Is deciding "the developer who proposed the merge" and "the branch
submitter" actually easy? We do not automatically get a mapping between
Git authors and Redmine users. And parsing the redmine ticket history
(e.g. who marked the ticket a certain QA status last?) also seems
hairy, but perhaps it's "easy" via some API (SQL?) that I'm unaware of?
Cheers!
