Hi,
I'd like to start by saying that I think the work bertagaz did on the
jruby + sikuli + cucumber combo is really great. He called it PoC, but
after having worked on it for some time now I say it's definitely fit
for the task at hand.
Next I'd like to announce that the automated test suite, in its current
unfinished state, actually has found its very first Tails bug. Here's
the cucumber output of when it was found:
[...]
And all Internet traffic has only flowed through Tor
# cucumber/iceweasel/step_definitions/torified_browsing.rb:66
The following IPv6 hosts were contacted:
ff02::1
Full network capture available at: [...censored...]
There were network leaks! (RuntimeError) [...]
In other words, our firewall leaks link-local IPv6 broadcasts even
though it should block everything IPv6 (right?). This is promising (not
that Tails has this particular bug, but that the test suite found it)!
The framework currently has the following test primitives and other
interesting features so far:
* Sikuli stuff, e.g. waiting for an image (or text via OCR) to appear
on the screen, and perform arbitrary mouse and keyboard actions.
* Capturing all of the guest's network traffic and determining whether
the Tails firewall leaked anything. On failure the capture is
preserved so manual inspection is possible.
* Take screenshots. It's currently unused, but I think it could be
useful to save a screenshot upon test failure. If we want to go
really crazy it would be trivial to add support for recording videos
using e.g. `ffmpeg -f x11grab` to record the whole DISPLAY that
virt-viewer + sikuli runs in, which could be helpful for post-failure
analysis.
* Running arbitrary commands inside the guest VM.
* Saving/restoring VM snapshots (not yet pushed to git)
I'd like to present the last two with a bit more depth and hear your
opinions, especially w.r.t. the fact that they alter Tails or "cheat" in
the testing process, so I wonder how "ethical" they are in the context
of test-driven development.
Running arbitrary commands inside the guest VM
==============================================
This is very valuable as it makes many tests that would be truly awkward
to do with sikuli into something trivial. libvirt doesn't seem to have
something like VirtualBox' `vboxmanage guestcontrol execute` (provided
by the VirtualBox guest additions), so I implemented a simple remote
shell (read: a backdoor (listening on port 1337 + firewall exception) so
expect havoc on the Tails forum!) that starts on the guest when the boot
parameter "autotest_never_use_me" is present on the kernel cmdline. If
it's not there, the script that implements the remote shell server is
removed from the filesystem. Of course, it is still accessible from the
read-only fs, but it will refuse to start if the boot parameter isn't
present. My hope is that all these failsafes will make sense to all but
our most "conspiranoid" users. :)
Saving/restoring VM snapshots
=============================
This is how I intend to use it for a given feature:
Background:
Given I restore the background snapshot if it exists
[ ... "real" background steps ... ]
And I save the background snapshot if it does not exist
[ ... Scenarios ... ]
So, for the first scenario all background steps are run normally, and in
the last one the VM state is saved. For all remaining scenarios the VM
state is restored in the first step, and all remaining steps, including
the "real" ones, just pass immediately (i.e. each "real" step starts
with "pass if we restored from a snapshot"). I suppose common background
snapshots can be shared between different features too to improve the
testing speed.
While saving/restoring snapshots greatly improves the speed of the test
suite, this wasn't the main reason for me implementing it. Without it we
would hammer the Tor authorities (once per scenario) since each Tails
boot is a full Tor bootstrap, which clearly isn't acceptable. In the end
we will have *many* scenarios in total, and if we're gonna run the test
suite for each automatic build at some point... well, you get the picture.
An issue with restoring past state like this is that our Tor's circuit
state may get out-of-sync with the circuit state of the relays they use.
For instance, I ran 10 tests that restored to the same post-background
state and all but the first two failed to fetch a web page. Then I ran
10 tests where I do the following after each snapshot restore:
1. Stop Tor.
2. Sync time from host to guest.
3. Start Tor.
And then all 10 tests succeeded, so it seems resetting Tor like this is
highly necessary.
That's all.
Cheers!
