Re: [Tails-dev] Automated tests specification

On 09/01/2015 12:04 PM, intrigeri wrote:
> Hi,
>
> bertagaz wrote (28 Aug 2015 14:24:51 GMT) :
>> On Wed, Aug 26, 2015 at 07:46:17PM +0200, anonym wrote:
>> I've also added a new section about the result to keep:
>
>> ## What kind of result shall be kept
>
>> The test suite produces different kind of artifacts: logfiles, screen
>> captures for failing steps, snapshots of the test VM, and also videos of
>> the running test session.
>
>> Videos may be a bit too much to keep, given they slow down the test
>> suite
>
> Do we have data about how much they slow down the test suite on our
> current Jenkins slaves? See #10001 for partial data about how we could
> make it less resources-hungry...

Speculation: Didn't we plan for an extra core for this? I suppose we'd
need fours cores, for: video capture, sikuli, libvirt (USB emulation in
particular), cucumber-and-other-stuff. Hmm, perhaps
cucumber-and-other-stuff are happy with the left over from the other
cores? So three cores?

>> and might take quite a bit of disk space to store.
>
> ... and smaller (#10001). I'm curious how much space a full set of
> test suite videos take.

I'll try to remember to collect this info in my next full run.

>> If we want to keep them, we may want to do so only for failing test
>> suite runs.
[...]
> A later iteration (non-blocker) could keep only those from features
> that have failing scenarios. If we agree it would be an improvement,
> I can file a non-blocking ticket about it.
>
> And by the way, perhaps having videos optionally split per-scenario
> would help a lot making good use of such videos: faster download,
> easier to jump to the failing part, less storage needs on our infra.
> anonym and Kill Your TV, if you agree it would be useful and not
> stupid an idea, I can file a non-blocking ticket about it.

Yes, per-scenario videos would be great (my plan was to do this when we
have #8947, but whatever, nothing prevents us from having it now). They
would be more useful than per-feature videos, and actually easier to
implement AFAICT. Please file a ticket!

Cheers!