Author: intrigeri Date: To: The Tails public development discussion list Subject: Re: [Tails-dev] Automated tests specification
Hi,
bertagaz wrote (28 Aug 2015 14:24:51 GMT) : > On Wed, Aug 26, 2015 at 07:46:17PM +0200, anonym wrote:
> I've also added a new section about the result to keep: > ## What kind of result shall be kept > The test suite produces different kind of artifacts: logfiles, screen
> captures for failing steps, snapshots of the test VM, and also videos of
> the running test session. > Videos may be a bit too much to keep, given they slow down the test
> suite
Do we have data about how much they slow down the test suite on our
current Jenkins slaves? See #10001 for partial data about how we could
make it less resources-hungry...
> and might take quite a bit of disk space to store.
... and smaller (#10001). I'm curious how much space a full set of
test suite videos take.
> If we want to keep them, we may want to do so only for failing test
> suite runs.
Fully agreed.
> But then, often the screen capture are enought to identify why
> a step failed to run.
In my experience, sometimes, what would help understanding the problem
has disappeared a while ago when we hit the timeout, fail, and take
a screenshot. That's why I almost always turn on --capture when I'm
doing TDD.
So, I invite you to reconsider the option of storing videos of failed
test suite runs. A first iteration could keep *all* videos for the
entire test suite — presumably this should be just a glob to add to
the list of artifacts we archive, right? If it's more work for you,
let me know.
A later iteration (non-blocker) could keep only those from features
that have failing scenarios. If we agree it would be an improvement,
I can file a non-blocking ticket about it.
And by the way, perhaps having videos optionally split per-scenario
would help a lot making good use of such videos: faster download,
easier to jump to the failing part, less storage needs on our infra.
anonym and Kill Your TV, if you agree it would be useful and not
stupid an idea, I can file a non-blocking ticket about it.
> If we decide to still use them, then we
> probably have to wait for [[!tails_ticket 10001]] too be resolved.
Absolutely. This sounds pretty easy: anyone with a Jessie test suite
environment can trivially confirm my results (should take just a few
minutes) and then I'll submit a branch for review.
> The retention strategy should be the same than for the automatically
> built ISOs.
ACK, let's start like this.
> I don't expect this last one to raise a lot of discussions, so let say that the
> deadline for this thread is next Sunday 12pm CEST unless some points are still
> not clear. I think the rest has already been discussed and drafted enough.
