Re: [Tails-dev] automated tests

On Tue, Jun 19, 2012 at 01:53:38AM +0200, intrigeri wrote:
>
> This does look awesome!

Thanks I agree ;)

> > It requires to use jruby, as sikuli is java. But there might be
> > other ways to implement it.
>
> Just curious: is this requirement brought in by the sikuli gem?

Totally, because sikuli is java. I didn't choose this dep by taste ;) If
one day sikuli can be used with another language, I'm totally ready to
switch.

>
> > You also need a Jruby >= 1.6 environnement, which sadly isn't
> > possible using debian packages yet
>
> Any hint / idea / pointer why?

See bug #636554, the maintainer seems to say back in september 2011 that
he is willing to upload jruby 1.6 "*after* the testing migration". Not
sure what he means by that, probably after wheeze is released. Adding a
word on this bug is prolly a good idea, so if you want to you're welcome.

>
> Also, it's not clear to me what "guest or host" means in there:
> - "on the VM guest or host who does the tests"
> - "everything happens on the host or guest where the build happen"
>
> Could you please clarify this kind of sentences?
>
> I guess this means "the system", that may be either a bare-metal one,
> or itself some kind of VM (with nested virtualization enabled), right?

Exactly, and that's why that was not easy to word correctly. I'll fix
this.

> Also, I'm unsure about "where the build happen". Do you mean "where
> the tests are run from", or anything else?

Yep, builds and tests are supposed to be started in the same system.

> > It is also possible to add and remove different types of storages,
> > and thus test persistence or filesystem modifications.
>
> Have you by chance found a way to *emulate* a USB 2.0 device in
> software? (qemu-kvm from current Debian testing/sid supports USB 2.0
> passthrough, but this is quite different as far as automated tests
> are concerned.)

Not really investigated in this area yet sorry. Apart from the
pass-through method I haven't seen anything else.

> > but anyway you need this gems to be installed
>
> Do you want to {add references to,file} the corresponsding RFP bugs,
> or shall I?

As you wish. As long as jruby is < 1.6 in Debian at least the sikuli gem
won't be packageable anyway. Might help to get a jruby upgrade.

> > setup a basic VM
>
> Providing a guest XML skeleton would be perfect :)

Yeah, that's one of the questions brought by this tool. Might be usefull
if it contains a basic domain definition for sure. For consistency of
devices name for example.

To test different feature, there will be need for diffferent VMs
configurations, for example some with more memory for the memory wiping
on shutdown feature. There are mainly two different ways to achieve this:

* programatically by using libvirt ruby bindings and use the functions to
attach/remove devices, modify the basic VM configuration, etc, which is
the current roughly implemented way.

* enable the tests to be shipped with a full domain definition, that would
be automatically loaded rather than a default one.

Between the two, tests could be shipped only with xml domain snippet
corresponding to the configuration changes to be done on the domain.

The second one would be the easiest, and might have the advantage to be a
totally "stop and throw away this domain" solution, so would help not to
modify too much the domain definition and keep consistency upon tests. But
there might be drawbacks too in this implementation.

>
> > Set the DISPLAY env to something relevant
>
> What do you mean? Some unused X display?

Yes, on this unused display (so typically > :7), a Xvfb virtual X server
will be attached, and virt-viewer will be told to display the domain
display in full screen there. This is the trick to have sikuli detecting
the display (thgrough the $DISPLAY env variable) and using it for the
tests without having to play to much with vnc. :)

Thanks for the feedbacks.

bert.