Last week I tested a beta version of our work on integrating VeraCrypt
in GNOME with 5 people. Here are the results!
Protocol
--------
I did in-person moderated tests with 5 participants. By lack of time I
recruited amongst friends who I already knew would match the profile I
wanted (even if it's not a best practice to do so).
I gave them 3 missions (in attachment).
- M1: Opening a file container (without .hc extension) with a password.
- M2: Opening an entire USB stick with a password.
- M3: Opening an entire USB stick with a password and a keyfile.
Mission 1 is harder than mission 2 but I chose to do them in this order
to match the popularity of the user scenarios as per our quantitative
survey:
https://tails.boum.org/blueprint/veracrypt/#survey
Contrary to segfault's branch, I add disabled automounting of file
systems to match the current behavior of Tails and have more data to
discuss #15628:
https://labs.riseup.net/code/issues/15628.
Participants
------------
I aimed people who:
- Already knew what Tails was and tried it (but were not regular users).
- Could understand our interface in English (to varying levels).
As a matter of fact, all were already familiar with Linux but with very
different technical levels, distributions, and desktop environment.
None of them were familiar with VeraCrypt and only one had tried it.
This added a layer of complexity to the tests as they had to understand
some concepts of VeraCrypt (like file containers and keyfiles) on top of
figuring out how to use our interface.
Still, this part went quite fine. They had little problems understanding
file containers. They had more problems understanding keyfiles and I
helped them a tiny bit on that when needed.
And the missions I gave them could be a realistic scenario: a journalist
who didn't know VeraCrypt receives encrypted volumes from a source.
You'll find more demographics in the spreadsheet in attachment.
Summary
-------
They made it through all the missions except 3. I had to rescue:
- P3 on M1: She got stuck after looking for VeraCrypt in the menus.
- P3 on M3: I had to redirect her to the right section of the doc.
- P4 on M3: I had to redirect her to the right section of the doc.
But they also all went through the doc to complete M1 and then had it
open for M2 and M3.
The average [SUS] score is of 50 which is considered below average
compared to other industry products (68), qualified as "OK".
[SUS]:
https://www.usability.gov/how-to-and-tools/methods/system-usability-scale.html
Spreadsheet
-----------
You'll find a rainbow table with the list of issues in attachment.
- The "M" column correspond to the mission in which the issues occurred.
- The "V", "C", and "V/C" correspond to a rough cost/value ratio based
on a punctuation of severity of the issues (1, 2, or 3) and cost.
- I marked '"' when several issues could share a single solution.
Top issues
----------
Like last time, I'm not making a clear boundary between what is part of
the VeraCrypt project and what is not, since an issue anywhere in Tails
or in our doc can hurt someone trying to use VeraCrypt as much as an
issue in the VeraCrypt integration itself.
In the spreadsheet I'm also listing a few bugs that should be fixed even
though they didn't hurt usability during the tests.
In rough order of importance:
1. People looked unsuccessfully for VeraCrypt in "Open With Other
Application" and the Applications menu. That forced everybody to go
through the doc while at least some of them could probably have made it
without reading the doc.
2. The structure of the documentation was very confusing and all 5
participants ended up following a wrong section of the doc (and not
mostly because they didn't knew VeraCrypt).
3. Bad error messages when unlocking fails. Nobody read the error
message, some interpreted correctly what happened, some didn't.
4. No feedback while the unlocking happens. This was confusing for a
successful unlock (a few seconds) but sometimes badly misleading for an
unsuccessful unlock (people started trying other things and sometimes
even didn't relate the error message with the failed unlocking).
5. Saving files from Tor Browser is a nightmare for newcomers. I ranted
about this in #15678 and prepared video clips (first time!):
https://labs.riseup.net/code/issues/15678
6. Having automounting would have helped P3 and P5 in M1. It would have
been even more helpful when people had to go through Disks in M3.
7. Bad layout of the user interface to point to Disks from GVfs monitor.
With the current layout several people thought they had to use Disks to
locate the keyfile and didn't interpret Disks as a complete alternative
to the GVfs dialog.
--
sajolida
