Re: [Tails-dev] Progress report on the automated test suite

Delete this message

Reply to this message
Author: Ague Mill
Date:  
To: The Tails public development discussion list
Subject: Re: [Tails-dev] Progress report on the automated test suite
anonym:
> I'd like to start by saying that I think the work bertagaz did on the
> jruby + sikuli + cucumber combo is really great. He called it PoC, but
> after having worked on it for some time now I say it's definitely fit
> for the task at hand.


Great. :)

> Next I'd like to announce that the automated test suite, in its current
> unfinished state, actually has found its very first Tails bug. Here's
> the cucumber output of when it was found:
>
>     [...]
>     And all Internet traffic has only flowed through Tor
>       # cucumber/iceweasel/step_definitions/torified_browsing.rb:66

>
>       The following IPv6 hosts were contacted:
>       ff02::1
>       Full network capture available at: [...censored...]
>     There were network leaks! (RuntimeError) [...]

>
> In other words, our firewall leaks link-local IPv6 broadcasts even
> though it should block everything IPv6 (right?). This is promising (not
> that Tails has this particular bug, but that the test suite found it)!


I did not run the code itself, but are you sure that these packets came
from Tails and not from the host system?

> Saving/restoring VM snapshots
> =============================
>
> This is how I intend to use it for a given feature:
>
>   Background:
>     Given I restore the background snapshot if it exists
>     [ ... "real" background steps ... ]
>     And I save the background snapshot if it does not exist

>
> [ ... Scenarios ... ]


Those lines feel like noise: they are an implementation detail and
should not appear in the scenarios.

Cucumber offer tags and hooks that should be usable to achieve something
similar while keeping the scenarios as lean as possible. See:
<https://github.com/cucumber/cucumber/wiki/Hooks> and
<http://stackoverflow.com/questions/9994797/cucumber-when-to-use-tags-hooks-vs-backgrounds>

> An issue with restoring past state like this is that our Tor's circuit
> state may get out-of-sync with the circuit state of the relays they use.
> For instance, I ran 10 tests that restored to the same post-background
> state and all but the first two failed to fetch a web page. Then I ran
> 10 tests where I do the following after each snapshot restore:
>
> 1. Stop Tor.
> 2. Sync time from host to guest.
> 3. Start Tor.
>
> And then all 10 tests succeeded, so it seems resetting Tor like this is
> highly necessary.


Indeed, as restoring from a snapshot is likely to break all existing TCP
connections. Have you tried to see if a SIGHUP sent to Tor is sufficient?



Side note: your `try_for` function is very unidiomatic Ruby.
I suggest you have a look at the part about blocks on
<http://www.ruby-doc.org/docs/ProgrammingRuby/html/tut_containers.html>,
and the `yield` and `block_given?` methods.

--
Ague