11/05/14 13:50, intrigeri wrote:
> Hi,
>
> intrigeri wrote (10 May 2014 15:22:54 GMT) :
>> Test suite run in progress, I'll inspect results and
>> report back later (likely tomorrow).
>
> Here we go!
>
> I think my first failure may be a blocker, as it's a regression.
> For the others, I don't know.
>
> So, the failures I have seen are:
>
> 1. I got this failure once out of two tries in the "Writing files to
> a read/write-enabled persistent partition" scenario, once out of
> two tries in the "Writing files to a read/write-enabled persistent
> partition with the old Tails USB installation" scenario:
>
> And I completely shutdown Tails # features/step_definitions/common_steps.rb:398
> Then only the expected files should persist on USB drive "current" # features/step_definitions/usb.rb:433
> FindFailed: can not find TailsEmergencyShutdownHalt.png on the screen.
>
> Looking at the video, it seems as if TailsEmergencyShutdownHalt.png
> was never clicked. There's a notification left on screen at this
> time, so I wonder if it may be interfering somehow (but perhaps I'm
> only trying to find justifications for my doubt wrt. dropping the
> notification handling code ;) -- in any case, that's a regression,
> since I have never seen this before.
Note that that scenario does the
And all notifications have disappeared
step. The notification killing I removed in commit 13d0ae9 doesn't
affect usb_install.feature in any way. I think this just shows that the
"all notifications have disappeared" step is problematic (and ). That
step will immediately complete once it sees no notifications, so it
really depends on being run after the last notification is has been
shown... I really don't know how to improve that step.
My suggestion would be that we don't use the emergency shitdown, and
simply sends a `halt`, which we did before, and which was much more
reliable. I know you want us to the same ways we except our users to
use, and I agree, but well... can't we just make a dedicated test for
the emergency shutdown instead?
> 2. Scenario: Booting Tails from a USB drive upgraded from DVD with persistence enabled # features/usb_install.feature:182
> [...]
> And the boot device has safe access rights # features/step_definitions/usb.rb:326
> And the expected persistent files are present in the filesystem # features/step_definitions/usb.rb:423
> Could not find expected file in persistent directory /etc/NetworkManager/system-connections (RuntimeError)
>
> Same in "Booting Tails from a USB drive upgraded from USB with
> persistence enabled" and "Booting a USB drive upgraded from ISO
> with persistence enabled".
>
> And on next run, I cannot reproduce this. Weird.
It's notable that that particular persistence preset's directory was
changed in feature/wheezy. What was your --old-iso when you ran the
first test vs the second? Could it have been a Wheezy-based image from
before the persistence preset was changed?
> 3. Scenario: Iceweasel should not have any plugins enabled # features/torified_browsing.feature:26
> When I run "iceweasel" # features/step_definitions/common_steps.rb:340
> And Iceweasel has started and is not loading a web page # features/step_definitions/common_steps.rb:247
> And I open the address "about:plugins" in Iceweasel # features/step_definitions/torified_browsing.rb:7
> Then I see "IceweaselNoPlugins.png" after at most 60 seconds # features/step_definitions/common_steps.rb:302
> FindFailed: can not find IceweaselNoPlugins.png on the screen.
>
> I've seen this once out of three tries, without --keep-snapshots.
> Unfortunately, I have no screenshot nor video anymore.
I just had a look (with --view) at what's actually happening in these
scenarios. I noticed that the Iceweasel window, after appearing, is
minimized and than maximized, which is strange. It would also explain
the randomness of this error as the following key presses surely can be
lost while that's in motion. The reason is that we end up doing two
@screen.wait_and_click("IceweaselRunning.png", ...)
and IceweaselRunning.png was changed from being the title to the task
icon in commit 13dce7f (introduced in the wheezy branch). The
`wait_and_click` was introduced to ensure focus of the window, so it has
to be e.g. the title. This is fixed now.
> 4. Scenario: Memory erasure on an old computer # features/erase_memory.fe
> [...]
> And I shutdown and wait for Tails to finish wiping the memory # features/step_definitions/erase_memory.rb:164
> Then I find very few patterns in the guest's memory # features/step_definitions/erase_memory.rb:140
> Pattern coverage: 0.314% (11 MiB)
> 0.314% of the memory is filled with the pattern, but less than 0.250% was expected (RuntimeError)
>
> I got this once out of two tries. Is it an acceptable drawback of
> how the test suite works, or a real problem?
To me it just shows that with the particular kernel (or whatever) that
we happen to use now require us to bump the highly arbitrary 0.25% to
0.5%, perhaps. Without some more rigorous guideline to what we think is
acceptable, arbitrary is what we've got. What do you think?
Cheers!
%20:%0A>%20>>%20Test%20suite%20run%20in%20progress,%20I'll%20inspect%20results%20and%0A>%20>>%20report%20back%20later%20(likely%20tomorrow).%0A>%20>%20%0A>%20>%20Here%20we%20go!%0A>%20>%20%0A>%20>%20I%20think%20my%20first%20failure%20may%20be%20a%20blocker,%20as%20it's%20a%20regression.%0A>%20>%20For%20the%20others,%20I%20don't%20know.%0A>%20>%20%0A>%20>%20So,%20the%20failures%20I%20have%20seen%20are:%0A>%20>%20%0A>%20>%20%0A>%20>%20%0A>%20%0A>%20Note%20that%20that%20scenario%20does%20the%0A>%20%0A>%20%0A>%20step.%20The%20notification%20killing%20I%20removed%20in%20commit%2013d0ae9%20doesn't%0A>%20affect%20usb_install.feature%20in%20any%20way.%20I%20think%20this%20just%20shows%20that%20the%0A>%20%22all%20notifications%20have%20disappeared%22%20step%20is%20problematic%20(and%20).%20That%0A>%20step%20will%20immediately%20complete%20once%20it%20sees%20no%20notifications,%20so%20it%0A>%20really%20depends%20on%20being%20run%20after%20the%20last%20notification%20is%20has%20been%0A>%20shown...%20I%20really%20don't%20know%20how%20to%20improve%20that%20step.%0A>%20%0A>%20My%20suggestion%20would%20be%20that%20we%20don't%20use%20the%20emergency%20shitdown,%20and%0A>%20simply%20sends%20a%20%60halt%60,%20which%20we%20did%20before,%20and%20which%20was%20much%20more%0A>%20reliable.%20I%20know%20you%20want%20us%20to%20the%20same%20ways%20we%20except%20our%20users%20to%0A>%20use,%20and%20I%20agree,%20but%20well...%20can't%20we%20just%20make%20a%20dedicated%20test%20for%0A>%20the%20emergency%20shutdown%20instead?%0A>%20%0A>%20>%20%0A>%20>%20%0A>%20%0A>%20It's%20notable%20that%20that%20particular%20persistence%20preset's%20directory%20was%0A>%20changed%20in%20feature/wheezy.%20What%20was%20your%20--old-iso%20when%20you%20ran%20the%0A>%20first%20test%20vs%20the%20second?%20Could%20it%20have%20been%20a%20Wheezy-based%20image%20from%0A>%20before%20the%20persistence%20preset%20was%20changed?%0A>%20%0A>%20>%20%0A>%20>%20%20%20I've%20seen%20this%20once%20out%20of%20three%20tries,%20without%20--keep-snapshots.%0A>%20>%20%20%20Unfortunately,%20I%20have%20no%20screenshot%20nor%20video%20anymore.%0A>%20%0A>%20I%20just%20had%20a%20look%20(with%20--view)%20at%20what's%20actually%20happening%20in%20these%0A>%20scenarios.%20I%20noticed%20that%20the%20Iceweasel%20window,%20after%20appearing,%20is%0A>%20minimized%20and%20than%20maximized,%20which%20is%20strange.%20It%20would%20also%20explain%0A>%20the%20randomness%20of%20this%20error%20as%20the%20following%20key%20presses%20surely%20can%20be%0A>%20lost%20while%20that's%20in%20motion.%20The%20reason%20is%20that%20we%20end%20up%20doing%20two%0A>%20%0A>%20%0A>%20and%20IceweaselRunning.png%20was%20changed%20from%20being%20the%20title%20to%20the%20task%0A>%20icon%20in%20commit%2013dce7f%20(introduced%20in%20the%20wheezy%20branch).%20The%0A>%20%60wait_and_click%60%20was%20introduced%20to%20ensure%20focus%20of%20the%20window,%20so%20it%20has%0A>%20to%20be%20e.g.%20the%20title.%20This%20is%20fixed%20now.%0A>%20%0A>%20>%20%0A>%20%0A>%20To%20me%20it%20just%20shows%20that%20with%20the%20particular%20kernel%20(or%20whatever)%20that%0A>%20we%20happen%20to%20use%20now%20require%20us%20to%20bump%20the%20highly%20arbitrary%200.25%%20to%0A>%200.5%,%20perhaps.%20Without%20some%20more%20rigorous%20guideline%20to%20what%20we%20think%20is%0A>%20acceptable,%20arbitrary%20is%20what%20we've%20got.%20What%20do%20you%20think?%0A>%20%0A>%20Cheers!%0A>%20%0A>%20%0A>%20 "Reply to this message")