Re: [Tails-dev] Questions about the automated test suite

Hi,

first of all: I'm really really happy to see yet another person run
our automated test suite! :)))

Kill Your TV wrote (18 Aug 2014 19:15:20 GMT) :
> * Even with 9GB assigned to a "level0-guest", the memory erasure
> test fails to complete due to failures allocating memory. How much
> (available) memory is (generally) required to be able to run the full
> test suite successfully?

I give 10240000 KiB to my tester guest VM.

> I probably need more RAM in order to run everything.

We can help, and you might want to talk to me privately about it.

In the meantime, adding swap to that guest VM should work, although
it'll of course slow things down.

> * Which tests, if any, are expected to fail or even pass inconsistently?
> (I already know about #7060)

See also #6092, which sometimes breaks the USB installer tests badly
(especially since many tests in there depend on previous ones to have
passed successfully). Otherwise, I've nothing else in mind, but if you
report specific failures, then maybe I'll remember :]

> * Are there any tests which are known to only run (or only run well) on
> 'bare metal'?

Many tests haven't been run in a nested virt. setup, so if you don't
have fast nested virt., then possibly some timeouts will be reached.
We can work this around by bumping the timeouts.

> * Generally speaking, how long does running the full suite take? (on
> good or average hardware, or 'nested vm' or 'bare metal'...whatever
> can be answered is fine)

I've never clocked it precisely, but I think it takes about 2-3 hours
in a fast nested virt. environment (Haswell i7-4600U, SSD).
IIRC a little bit more on a slower box (ThinkPad X201, i5-560M), but
running on bare metal. At least 2-3 times slower in nested virt.
on that X201.

I'd be delighted to see other results, especially:

* on AMD where supposedly nested virt. might be faster
* on desktop CPUs (instead of low-voltage mobile ones)
* on normal-voltage mobile CPUs
* with more RAM allocated, so that the temporary directory can have
a tmpfs mounted onto it, which should make the I/O-heavy tests (USB
installer, memory erasure) — which take most of the full run time
here — substantially faster
* with more vCPUs allocated to the Tails system under test (wanna
give #6729 a try? :)

Rationale: these results would 1. help us decide what hardware we want
to run our tests automatically post-commit (#7631); 2. help us decide
what hardware we should put into the hands of developers working on
the test suite, or who need/want to run it often (e.g. to do TDD).
It would be good if, in both usecases, we could run the current test
suite in less than one hour... especially since we have plans to add
many more automated tests.

Cheers,
--
intrigeri