Re: [Tails-dev] automated tests

On Wed, Jun 20, 2012 at 03:03:22AM +0200, intrigeri wrote:
> bertagaz@??? wrote (19 Jun 2012 11:39:17 GMT) :
> > As you wish. As long as jruby is < 1.6 in Debian at least the sikuli
> > gem won't be packageable anyway. Might help to get a jruby upgrade.
>
> Well, if you don't mind, please reference or file the RFP bugs
> yourself, then :)

Ack. Will do.

> > Yeah, that's one of the questions brought by this tool. Might be usefull
> > if it contains a basic domain definition for sure. For consistency of
> > devices name for example.
>
> > To test different feature, there will be need for diffferent VMs
> > configurations, for example some with more memory for the memory wiping
> > on shutdown feature. There are mainly two different ways to achieve this:
>
> > * programatically by using libvirt ruby bindings and use the functions to
> > attach/remove devices, modify the basic VM configuration, etc, which is
> > the current roughly implemented way.
>
> > * enable the tests to be shipped with a full domain definition, that would
> > be automatically loaded rather than a default one.
>
> > Between the two, tests could be shipped only with xml domain snippet
> > corresponding to the configuration changes to be done on the domain.
>
> > The second one would be the easiest, and might have the advantage to be a
> > totally "stop and throw away this domain" solution, so would help not to
> > modify too much the domain definition and keep consistency upon tests. But
> > there might be drawbacks too in this implementation.
>
> I tend to prefer the 2nd way too, for the sake of reproducibility.
> I suggest we try it, and see how it goes.

Ok, will push a domain xml soon then.

> Regardless how the initial domain definition is done, we'll probably
> need to programatically change it while testing,
> e.g. to plug or unplug a USB device.
> So the programmatic interface is good news anyway.

Yeah, and if we want to sniff the network, we'll have to know the
"tested system" IP, and it is easier to do knowing the mac address. We
might want to provide a net-dumpxml too.

> Side question: how often to reboot the tested system?

Good question! That is one of the stuffs to think/implement. It depends
on how the helpers are hooked in the support/env.rb file, or in the step
definitions. Actually, the reboot is happening between each scenario (as
there is only one yet ;) ), but I guess the best would be to reboot
between each feature rather. Or let the dev choose in the setp definitions
when she/he is willing to start/stop a Tails vm.

bert.