At the beginning of the month, we organized together with Lunar from Tor
a UX testing session of the last prototype of Tails Greeter, designed by
Alan and Spencer.
The tests took place at B17 in Nantes and everything went really
smoothly. So thank you to everybody who organized or attended the tests.
I'm putting the people who came to do the test in blind copy, in case
they're curious about the results :)
We gave people three different missions:
A. Beginners: "Someone gave you a Tails USB stick and you want to try it
out to visit some website anonymously."
C. Experienced users: "You need to configure a bridge to connect to Tor
while traveling and download a secret document to your persistent storage."
D. Geeks: "You're starting Tails to access a document on the internal
hard disk of the computer and save it to your persistent storage."
Lunar helped me take notes and debrief the tests.
As a first note, I wanted to say that the new Greeter is awesome! There
are surely many little things to improve but it did very well. I was
also surprised to see how good newcomers do with the GNOME widgets. Many
of the testers were interacting with Tails and probably GNOME for the
first time.
The actual results come in two forms:
1. Less interesting: a SUS score (
http://www.measuringu.com/sus.php)
We got a score of 62 which is slightly below average (the average across
many products is 68). So we're doing good but not that much.
Still, we shouldn't pay much attention to it or be disappointed not to
reach the average. It was the first time I was using the SUS
questionnaire and I just wanted to give it a try. I learned from my
mistakes and next time it will be more relevant.
2. More interesting: a rainbow table (in attachment)
- We identified 22 issues.
- Each colored cell in the table represents an occurrence of the
issue with one tester.
- Testers are numbered from 1 to 16 and grouped by their mission.
- The black cells are for issues that do not apply to a given
testers. For example, people who were not expected to configure a
bridge couldn't be confused about how to do so.
I added a possible solution and whether I think it should be a blocker
or not in columns T and U. So here is the list of possible solutions
(the number in bracket refer to the issue number in the rainbow table).
[1]: Virtual keyboard (bonus).
Nobody was blocked by the default keyboard not being their usual one but
many did typos. I wonder how would this be for people using non-Latin
scripts. In case people really can't make it to select their language
with the mouse of the default keyboard, maybe a virtual keyboard would
help. That's anyway something we want to integrate in the Greeter to
protect from keyloggers but that's quite some work.
[2] [6] [11] [12]: Auto-selection according to language (blocker).
Many people either lost time or started with the wrong keyboard because
of a lack of correlation between the language and the rest. Changing the
default keyboard and format based on the language would help. We're
doing this in the current Greeter and that was part of our plans anyway.
[4]: "Keyboard layout" (easy)
Someone thought that "keyboard" was referring to the type of keyboard
(USB, BlueTooth) or drivers, etc. Using "Keyboard layout" which is a
more common phrasing might help.
[5]: Split keyboard variant (bonus)
The list of different keyboard layouts is very long and people were
forced to go through it because of [2]. This was sometimes quite long.
Splitting the selection in two steps: first language, then the variant
(optional) might help. That's a bonus as the auto-selection should
already provide something good and that was not preventing people from
doing their job.
[7]: Formats (won't fix)
I was surprised by how well understood was the "Formats" option. The
calendar icon worked really well. Many people referred to disliking the
American date formats :)
[8]: Redesign help button (blocker)
Nobody clicked on the help button [?] and the only person who felt like
he needed help, looked for it and didn't find it. It needs to be redesigned.
[9]: Allow changing timezone in session (later)
Many people were not configuring the time zone in the Greeter, either
because they were going too fast, didn't care, or were not used to do so
in the current Greeter. We already have plans for solving this from the
desktop.
[10]: Map for time zones (bonus)
Amongst the few people who configured the time zone, most of them
searched by country name (France) instead of by city name (Paris) or
time zone name (Eastern Europe). Adding a map would make this easier.
That's not a blocker since we don't have the backend to handle custom
time zones anyway.
[13]: Doesn't consider saving language settings (won't fix)
Surprisingly, only one person clicked to save the language settings.
Still, I propose not to change anything as that's probably something
that people will learn how to do over time.
[14]: Warning on language settings in cleartext (blocker)
The only person we questioned about saving the language settings thought
that they were saved encrypted as with everything in Tails. We should
really add a warning as planned originally.
[17]: Improve camouflage wording + screenshot (easy)
First of all, it seemed to be confirmed that the Windows camouflage will
be removed from Tails Jessie (January 26). Still, if we get it back
somehow, the way it is presented in the Greeter should make it more
clear that this is about the visual appearance only (and not the
networking behavior for example).
[18]: Click outside of the dialog closes the dialog (bonus)
One person was very confused about the overlay of the additional
settings dialog on top of the main window and had a very hard time
closing it. Clicking outside of the additional settings dialog should
close it.
[19]: "Start Tails" = "Unlock"
Many people were not clicking "Unlock" before clicking "Start Tails"
when activating the persistent storage. We should make "Start Tails"
perform the same action as "Unlock" before starting the session.
[20]: Info bar about bridge configuration
Everybody we had to configure a bridge was confused about the fact that
the bridge information was not entered in the Greeter and was worried
about having made a mistake. We can't really change this now as that's a
much more complex issue but for the time being we could add an info bar
in the Greeter saying that the actual configuration will be done later on.
[21]: "bridge" → "Tor bridge" (easy)
One technical person thought that "bridge" was referring to the
networking device (ISO layer 2 bridge) and not "Tor bridge".
[22]: "Privacy Settings" → "Additional Software" (easy)
Several people who had to configure an administration password expressed
that going into "Privacy Settings" was not clear enough, though they
found their way. I'm proposing to rename this section "Additional Settings".
So what's next?
I propose that Alan and Spencer have a look at the rainbow table and
comment on the solutions I'm proposing. Once we agree on what needs to
be done we'll create Redmine tickets to solve each one of them.
