Hi,
bertagaz wrote (25 Jun 2015 09:41:23 GMT) :
> I've prepared a blueprint to start this discussion and take notes of the
> decisions:
> https://tails.boum.org/blueprint/automated_builds_and_tests/automated_tests_specs/
Great work! I've pushed a few minor changes, and a more important one
(21870e1), on top.
> ## When to test the builds
> for base branches, we could envisage to run the full test suite on
> every automatically built ISO (every git push and daily builds) if
> we think that is relevant.
This would be great. A possible optimization would be to do it
(instead of all base branches, all the time) for:
* the stable branch, so that we're always ready to put out an
emergency release;
* the branch that next scheduled release will be based on (can be
either stable, or testing, or devel, depending on when in the
release cycle we are).
> for feature branches, we could run the full test suite only on the
> daily builds, and either only the automated tests related to the
> branch on every git push, and/or a subset of the whole test suite.
I'm not sure what's the benefit of testing a topic branch every day if
no new work has been pushed to it. In the general case, as a developer
I'd rather see them tested on Git push only, with some rate limiting
per-day if needed. See below wrt. one specific case.
> We can also consider testing only the feature branch that are marked
> as ReadyforQA as a beginning, even if that doesn't cover Scenario
> 2 (developers).
Absolutely, I think that would be the top-priority thing to do for
topic branches: let's ensure we don't merge crap.
> We can also maybe find more ways to split the automated test suite
> in faster subsets of feature depending on the context, define
> priorities for built ISO and/or tests.
This feels ambituous and potentially quite complex. I say let's design
something simple and then re-adjust.
> ## How to run the tests
> The automated test suite MUST have access to the artifacts of
> a given automated build corresponding to a given commit, as well as
> to the ISOs of the previous Tails releases.
The ISO of the one last release should be enough, no?
It also needs to know what commit that ISO was built from, in order to
run the test suite from the same commit. Surely we can dynamically get
this information by inspecting the ISO (maybe even in the iso9660
metadata), if passing through the info via Jenkins is too painful.
Maybe that's worth a research ticket?
> The automated test suite MUST be run in a clean environment.
I'm not sure what exactly you had in mind here, but in my experience,
the test suite is now quite resistant to being run multiple times in
a row, so don't bother too much about this -- just using
a fresh --tmpdir should be enough in general. If we really need to
e.g. reboot the isotesterN VMs between test suite runs, I've looked
into it a few weeks ago and dumped results of my research somewhere
(likely in some blueprint). It seemed to be doable, but adds quite
some complexity that I'd happily skip.
> The automated test suite MUST be able to run features in parallel
> for a single automated build ISO. This way, if more than one
> isotester are idle, it can use several of them to test an
> ISO faster.
Wow! Not sure if/how this can work out, or actually optimize things
much, with the upcoming new VM snapshots handling.
Anyway: I doubt we'll have the situation when we have idle
isotesterN's -- we're rather trying to limit the workload to something
they can handle -- so perhaps it's not worth putting too much time
into this? My current feeling is that this is a MAY at best, but I can
totally be missing something.
> The automated suite SHOULD be able when there are more than one ISO
> queued for testing to fairly distribute the parallelizing of
> their features.
> The automated test suite MUST not allocate all the isotesters for
> one ISO when others are waiting to be tested.
These seems to be related to the parallelizing of one test suite run
on multiple VMs, so I'll skip them until we've discussed that
topic more.
> The automated test suite MUST be able to accept a treshold of
> failures for some features before sending notifications. This can
> help if a scenario fails because of a network congestion, but other
> use cases will probably raise.
The current running theory is that the test suite *itself* (as opposed
to the way it's being run e.g. by Jenkins) should handle this itself,
see e.g.
https://labs.riseup.net/code/issues/9515. I prefer it a lot
more to having Jenkins ignore failures, as it also benefits people who
run the test suite outside of Jenkins. But realistically, surely we'll
anyway have transient failures, and I'm not sure what's the best way
to deal with it. I doubt it parameterizes a lot how we design the
whole thing, though: it seems to be only about Jenkins publishers
configuration, and should not impact the rest, so perhaps we can just
postpone this topic (and not call it a MUST) until #9515 and friends
are resolved, and an initial deployment makes our actual needs
clearer? (See, I'm not *always* in favour of over-engineering
things ;)
Cheers,
--
intrigeri
