Hi,
I did a second batch of user tests, again with five participants.
I changed a few things before these tests:
- Use the "clickable labels" I implemented and wrote about before. I
will append screenshots of this again. In the previous tests I simply
used textboxes for the editable options, which were grayed out while the
service was running. Because they were grayed out, they could not be
selected, which was a problem for some users, who wanted to copy the
strings. With the clickable labels, the users are able to select and
copy them, because they turn into normal labels while the service is
running.
- Only display the onion address and connection information once the
service has started
- Display the connection information on click in a separate window
This time I did the tests with less skilled users (the first tests were
mainly with IT students). The feedback was less positive, the average
SUS score was only 60.5/100 (it was 83 in the first tests). They also
had a lot more problems and needed more time to solve the tasks (average
of 16 minutes).
The main problems discovered:
- 4/5 had at least some problems with the client-server concept, i.e.
they didn't understand that they are hosting a server on their machine
and have to connect to this server with a client application. And they
tried to modify the server config in the client, especially during step
6. "Configure the Mumble service so that it will automatically start
after a reboot of the system".
- This is what sajolida (and others?) already expected. I'm not sure
what we can do about this. sajolida suggested showing a schematic of the
concept in the placeholder in Tails Server. Maybe we can also display a
line "You can connect to this server with 'Mumble'", and maybe open the
application on click on the application name.
- When asked to modify the options, 4/5 needed at least 1 minute to
figure out that they had to stop the service before they can edit the
options.
- I think I will use the edit mode we discussed instead of the
clickable labels, which will hopefully make it more obvious how to edit
the options.
- 2/5 tried editing the password in the "Connection Information" window,
after they were not able to edit it in the config panel. I think this
would be solved when we solve the problem that the user's don't know how
to edit the options.
- 3/5 needed some time to see the on-switch to actually start the
service after installing it
- I don't know how we can make this more obvious. I think putting
on-switches in the top-right corner is standard in GNOME. And I don't
see any other position where it would be more noticeable without looking
awkward.
- 2/5 were looking for the address and other connection information
before the service had finished starting. They didn't find it, because
the address and the connection info only get populated after the service
was started.
- Maybe we could show them with a hint "Not available until service
started"
- 3/5 were confused how to close the "Connection Information" window
- Apparently it was a bad idea of mine to name the buttons "Copy" and
"Don't Copy". I think I will rename "Don't Copy" to "Close" and make the
"Copy" button not automatically close the window.
- 2/5 pressed the '+' key on their keyboard instead of the '+' button
after reading "Click '+' to create a new service" in the placeholder
Problems in Mumble:
- None of the users verified the certificate
- 1/5 saw the SHA-1 sum in the connection info, but didn't find it in
Mumble
- anonym's Mumble patch will make this a lot easier, because it will
display the fingerprint directly in the warning message
- 1/5 saw the SHA-1 sum in Mumble, but not in the connection info
- I'm not sure how to prevent this. We could display the fingerprint
directly in the config panel.
- 1/5 thought the warning was caused wrong connection data
- The others didn't care at all
- 4/5 were not sure if they could choose the "Label" field in the "Add
Server" form themselves or had to fill in the correct value. One was
stuck for a long time on this, because he thought the certificate
warning was caused by the wrong value in the "Label" or "Username" field.
- 3/5 were confused that there is no "password" field in the "Add
Server" form
Cheers
