Hi,
I did five users tests in the last days, which I want to give you a
report on:
# Testing process
I presented them a VM with a running Tails. After a short briefing about
what Tails and Tails Server is, I showed them the documentation (see
attachment) and the already started Pidgin chat client. Then I gave them
the following tasks to solve:
1. Start the Tails Server
2. Add the Mumble service
3. Start the Mumble service
4. Send the connection information needed to connect to the service to
the „Amnesia“ contact in the Pidgin chat client.
5. Use the Mumble client to connect to your service.
6. Configure the Mumble service so that it will automatically start
after a reboot of the system.
7. Modify the password of the Mumble service.
I mainly observed the users and took notes. I also recorded the screen.
I didn't help when they had problems with Tails Server or Mumble, but
when they were stuck for a long time, I gave a hint to read the
documentation (this happened two times). Afterwards I asked them what
they thought about the application, what they liked and what they didn't
like, and if they had any suggestions to improve it. And I gave them a
System Usability Scale questionnaire to fill in.
# Users
I think all five users were more technically qualified than average.
4/5 were IT students. 2/5 used Tails before.
# Results
The feedback was positive, they all said that they liked the
application. The average score of the System Usability Scale was 83 out
of 100, which is a very good result, at least according to some
usability webpages I read. But then I am afraid that the feedback might
be biased, because they all knew or guessed that I was developing this
application, which might have caused more positive feedback, because
they wanted to be friendly.
The users needed 5, 7, 12, 20 and 25 minutes to solve the tasks.
I observed a lot of issues during the tests:
The most frequent issues were with the Mumble certificate warning. This
warning message appears because Mumble enforces the use of TLS
encryption and automatically creates a self-signed certificate for this.
Because this certificate is not signed by a trusted CA, the client
displays this certificate warning when connecting to the server. But to
verify the certificate, the user has to compare the SHA-1 fingerprints,
and the fingerprint of the certificate in the Mumble client is very well
hidden in an aweful scroll box (see attachment).
In conjunction with the Tails Server, we wouldn't actually need TLS
encryption for Mumble, because the connection will already be encrypted
by Tor.
But we don't want to teach our users to ignore SSL warning messages, so
instead I explained in the documentation how to verify the certificate.
But the results regarding this are bad:
- 3/5 didn't verify the certificate
- 1 of these 3 didn't actually read the warning message, but suspected
that it was an error message because he used the wrong connection details.
- 1 of these 3 tried to verify the SHA-1 fingerprint, but couldn't
find the certificates fingerprint in the scroll box
- 1 of these 3 didn't see the fingerprint in the Tails Server's
connection info. But he knew that he doesn't actually need to verify the
fingerprint because the service is already authenticated by the onion
address.
Other problems:
- 2/5 tried to modify the server config via the client
- 1/5 needed some time to understand that the application is hosting a
network service and they had to use another client application to
connect to it.
- 1/5 looked for a way to start the Mumble client from within the Tails
Server
- 1/5 didn't understand the connection info at first and tried to copy
the required information one by one, which didn't work, because the
service was running and the password field was grayed out, so she
couldn't select it to copy it. The latter problem would be solved if we
used the clickable labels or the edit mode. And I will try to make the
connection info less confusing by making it a button which opens a
window to display the connection info - right now it displays only a
part of the Mumble URL in the config panel.
- 1/5 was confused that the loading spinner didn't spin and wondered
whether the application had crashed (the reason of this seems to be that
GNOME disables animations if it runs in a VM, I don't know how to fix this)
- 1/5 needed some seconds to figure out that she had to enable the
persistence option before she could enable the autostart option
- 1/5 was confused that there was no password field in the connection
form in the Mumble client
- 1/5 was wondering whether the "Allow LAN" option was required to
connect to the service locally (on the same machine)
- Maybe we should add "?" icons next to some options and link to the
respective section in the documentation
- 1/5 clicked the connection info's "Copy" button before the service was
connected, so the connection info was still empty, so they had an empty
clipboard.
- I'm already reworking the connection info and will make sure that
this can't happen anymore
- 1/5 suggested that we should allow editing the options while the
service is running and restarting it when the options are applied (like
the edit mode discussed in the other thread).
Not Tails Server-specific, but interesting nonetheless:
- 3/5 complained that there was no search function in the application
menu and they had to look for the Tails Server and/or the Mumble client
for quite some time. The other two already knew Tails and were probably
used to the lack of a search function in the application menu (none of
them used the search function in GNOME's Activities Overview).
