UX + Usability Study Take Two!

Translate this post

Usability Study No. 2

The Wikipedia Usability Initiative partnered with Bolt Peters and Davis Research to evaluate the changes we’ve implemented so far and inform our work moving forward.  If you don’t know what changes we are talking about, check out our Beta (including a new skin, new toolbar, improved search, and more) by following these instructions.
Overall, the study confirmed that we are on the right track with our beta features – showing us room for improvements, maybe a bug or two along the way, and work yet to be done!  You can view the full report (and soon the full videos) on our project wiki, but we thought we’d share with you some highlights:
“It was easy, and I wouldn’t have thought it would be that easy.”
“Before there were a lot of tools, and I liked that they were all spread out in front of you, but this actually makes a lot of sense. I had to muddle my way through the older system, but this one seemed fine.”
“Websites don’t have common sense, but programmers do.”
The majority of our 8 interview subjects found and used our features and tools without instruction and with success.  Special victories go to our more spacious and grouped tabbed navigation, improved search and new searchbox location, and built-in toolbar.  In using these features, users were not only less intimidated, but also showed a greater ease of use and increased performance.  All of the 8 users successfully found the “edit” tab with a minimum of hunting; no one resorted to Google to get to the Wikipedia article they were seeking; two of our users even expressed pleasure and delight in the process!  Perhaps small victories, but a major change from our first study if you remember!
Needs Improvement
“Uh-oh, I think I may have made the wrong kind of link before. I’ll go to the preview window to see if this is a link. It would have been nice to just edit it in the preview.”
“This is different, it’s got these hot-links [the table of contents]. That’s nice.”
“Links are so easy to screw up. I’m not sure if we’ve correctly typed the link markup. Ah, there are these buttons…”
Some of our tools are definitely still rough around the edges – their flaws and failures were seen in technicolor when observing people using them.  Our link dialog caused the most confusion.  6 of our 8 users initially made some errors in using it, and some received a false positive assurance when they had not actually accomplished the link behavior they were attempting.  Oops!  Our features need to err on the side of a user’s expectation rather than giving users access to the technical structure or wiki syntax, which they did not in this case.  For example, to create a new link in our prototype, users were asked to specify whether they wanted to create an “external link” (to a website) or “internal” link (to a different article) – a differentiation that exists in wiki code, but not in the eyes of a novice user.  Additionally, our toolbar buttons need to behave consistently and be grouped accordingly.  Having dialogs for links and tables, and not having one for a reference was not acceptable and led to some quite confused and persistent button pushing by our subjects.
Speaking of buttons, our “Bold” and “Italics” toolbar buttons use the roman character “a” – the result of our struggled effort to be accessible to an international community while attempting to take advantage of software standards.  In our effort to generalize, we became too general – even those users who correctly guessed the purpose of these buttons had to hover over or use them to confirm their assumptions.  We’re going all in – look out for our efforts to make our toolbar icons language specific soon!
As if we didn’t already know it – adding media or “embedding a file” was the least understood toolbar action of our study.  Most users avoided it, but when they did the sample text that it inserted provided no additional insight.
Moving Forward
“I’m completely intimidated by that [template].”
“I’m not sure what that is. I’m going to save it and then see, because this preview is too confusing.”
Our study illustrated how large an effect a small change can have and brought to our attention tweaks and enhancements that need to be made to our current features.  It also showed us that we are just a slice of what is a very, very large pie.  We had many deja vu moments seeing users flounder around previewing and saving, many times adopting strange techniques and multiple windows to add a simple sentence.  The terms “code,” “computer lingo,” “html” often came up and highlighted the separation users feel from their content while editing.  The expectation for editing a wiki to be similar to editing a blog or word processing document was still prevalent.  And though our Table of Contents and built-in cheat sheet put out some small fires, when navigating an lengthy article or searching for help, we again heard “there sure is a lot of stuff to read” and “this is where I’d give up.”
As we’ve mentioned before, we cannot tackle the full scope of issues that our study participants surface.  But I think I can speak for our team when I say we all felt a certain amount of satisfaction in the results of those problems we did address and it has only made us more eager to attack new problems and iterate on solutions we’ve proposed.  As always, we look forward to your comments, insights, and feedback!  We also appreciate your contributions during our fundraiser – it’s in part community support like this that makes the Foundation’s work possible.
Parul Vora, Wikimedia Usability Initiative

Archive notice: This is an archived post from blog.wikimedia.org, which operated under different editorial and content guidelines than Diff.

Can you help us translate this article?

In order for this article to reach as many people as possible we would like your help. Can you translate this article to get the message out?

Inline Feedbacks
View all comments

[…] done a post on the process over at the Wikimedia blog, where you can read the report (with videos forthcoming!). And also […]

I may be missing something, but is there any sort of timetable yet as to when we can expect to start seeing these improvements *by default* on “live” Wikimedia projects? I’m a big fan of Vector; as an advanced user, I have been using it since it came out. What I’m really looking forward to, however, is having the improvements delivered to newbies, the sort who see such a positive change from your study! I realize that the improvements are ongoing: it’s understandable to delay as we don’t want the interface to change significantly every other week. I’m simply eager.

Glad to hear it Nihiltres. Making our changes default is definitely one of our goals, and it’s great to have so much community support behind such a move. But as of yet, there is not a definite timetable for that. As you mentioned, we don’t want to be making significant changes once deployed and we are still prototyping and trying a variety of exciting features and enhancements. Also, as you can see with this study, we’ve also surfaced many ways to improve tools we’ve already released in Beta. The best answer we can give is: not yet. Yet being the… Read more »

@Nihiltres: Many of the usability improvements, including Vector, are already the default on the English Wikinews. I’m not sure if any other projects have done so, but I guess if a community reaches a clear consensus, they could also ask to be switched.

Я вам пожертвую денег, когда будете принимать яндекс-деньги! Думаю, многие пользователи из России того же мнения.
[Google translation: “I will donate money, when you take Yandex-money! I think many users from Russia would be of the same opinion.”]

> We’re going all in – look out for our efforts to make our toolbar icons language specific soon!
With the old toolbar this was possible and in use – that makes the new behaviour a regression.