Re: [Tails-dev] [review'n'merge:1.1] #6559, #7062, #6275: te…

Hi,

intrigeri wrote (10 May 2014 15:22:54 GMT) :
> Test suite run in progress, I'll inspect results and
> report back later (likely tomorrow).

Here we go!

I think my first failure may be a blocker, as it's a regression.
For the others, I don't know.

So, the failures I have seen are:

1. I got this failure once out of two tries in the "Writing files to
a read/write-enabled persistent partition" scenario, once out of
two tries in the "Writing files to a read/write-enabled persistent
partition with the old Tails USB installation" scenario:

    And I completely shutdown Tails                                    # features/step_definitions/common_steps.rb:398
    Then only the expected files should persist on USB drive "current" # features/step_definitions/usb.rb:433
      FindFailed: can not find TailsEmergencyShutdownHalt.png on the screen.

Looking at the video, it seems as if TailsEmergencyShutdownHalt.png
was never clicked. There's a notification left on screen at this
time, so I wonder if it may be interfering somehow (but perhaps I'm
only trying to find justifications for my doubt wrt. dropping the
notification handling code ;) -- in any case, that's a regression,
since I have never seen this before.

2. Scenario: Booting Tails from a USB drive upgraded from DVD with persistence enabled # features/usb_install.feature:182
     [...]
     And the boot device has safe access rights                                        # features/step_definitions/usb.rb:326
     And the expected persistent files are present in the filesystem                   # features/step_definitions/usb.rb:423
       Could not find expected file in persistent directory /etc/NetworkManager/system-connections (RuntimeError)

Same in "Booting Tails from a USB drive upgraded from USB with
persistence enabled" and "Booting a USB drive upgraded from ISO
with persistence enabled".

And on next run, I cannot reproduce this. Weird.

3. Scenario: Iceweasel should not have any plugins enabled                # features/torified_browsing.feature:26
     When I run "iceweasel"                                               # features/step_definitions/common_steps.rb:340
     And Iceweasel has started and is not loading a web page              # features/step_definitions/common_steps.rb:247
     And I open the address "about:plugins" in Iceweasel                  # features/step_definitions/torified_browsing.rb:7
     Then I see "IceweaselNoPlugins.png" after at most 60 seconds         # features/step_definitions/common_steps.rb:302
       FindFailed: can not find IceweaselNoPlugins.png on the screen.

I've seen this once out of three tries, without --keep-snapshots.
Unfortunately, I have no screenshot nor video anymore.

4. Scenario: Memory erasure on an old computer                     # features/erase_memory.fe
   [...]
     And I shutdown and wait for Tails to finish wiping the memory # features/step_definitions/erase_memory.rb:164
     Then I find very few patterns in the guest's memory           # features/step_definitions/erase_memory.rb:140
       Pattern coverage: 0.314% (11 MiB)
       0.314% of the memory is filled with the pattern, but less than 0.250% was expected (RuntimeError)

I got this once out of two tries. Is it an acceptable drawback of
how the test suite works, or a real problem?

Cheers,
--
intrigeri
| GnuPG key @ https://gaffer.ptitcanardnoir.org/intrigeri/intrigeri.asc
| OTR fingerprint @ https://gaffer.ptitcanardnoir.org/intrigeri/otr.asc