Usability Study Results (Sneak Preview)

Usability Study Sessions

The Wikipedia Usability Team has finished up their analysis of the User Experience and Usability Testing conducted this past March with Bolt | Peters!  As noted, please expect a full report on our project page soon.  While we are getting all of our i’s dotted and t’s crossed, we wanted to share with you some of the major themes and findings & successes and failures.
Outstanding
“Usually it’s the most information in the easiest spot to access.  It always looks very well put together….it boggles my mind how many people can contribute and it still looks like an encyclopedia.” – ‘Galen’
“I like Wikipedia because it’s plain text and nothing flashes” – ‘Claudia’
“It’s always one of the first hits on Google” – ‘Grace’

If there was one thing that was consistent and unanimous across our study participants, it was the assessment that Wikipedia is an incredibly valuable information resource whose accessibility is unparalleled.  Aside from its value as a reference, a time and lifesaver, and an up to the minute news resource, participants also praised it’s simplicity, coherence, and breadth.  Also, an extra thanks to Google, for showcasing just how referenced Wikipedia articles are – consistently making their links one of the top hits – our users count on that!
Room for Improvement
“Rather than making a mess, I’d rather take some time to figure out how to do it right.” – ‘Dan’
“Where are the rules?……I don’t really want to read all of this other stuff about what I”m supposed to do.” – ‘Grace’

All of our participants are Wikipedia readers, but had little or no experience with editing.  Generally the editing process was not a warm and welcoming one.  Before subjects even hit the ‘edit’ or ‘edit this page’ buttons, they voiced concerns about the rules, proper etiquette, formatting, and were naturally conscientious of and inhibited by maintaining the community expectations.  When a few of them attempted to find answers to their questions about rules and etiquette, they were overwhelmed with the amount of information and documentation they encountered.
“ [I felt] kind of stupid.” – ‘Galen’
“It looks all jumbled and crazy…I’m going down to the stuff that looks like it makes sense.” – ‘Tito’
“I’m not a programmer.  I know the letters PHP.” – ‘Seamus’

Once within the editing environment, most subjects commented on the illegibility of the hybrid Wiki syntax and article content – the more complex the article, the more exaggerated the response.  When users made it past their initial reactions, navigating around the syntax to perform basic word processing tasks (correcting a typo, inserting a block of text, bold and italics formatting) proved less problematic than finding a particular section, adding references, using tables, creating and naming links.  But not even our youngest and most computer savvy participants accomplished these tasks with ease.
“It’d be nice to have a GUI, so you could see what you’re editing.  You’ve made these changes and you’re looking at it, and you don’t know how it’s going to look on the page.  It’s a little clumsy to see how it’s going to look.” – ‘Bryan’
“On a blog it looks like the real page when you look at it.  I have a hard time looking at this and going back to the way it actually is.  I’m trying to correlate what’s on the real page.” – ‘Saurab’

Aside from feeling confused by the “code”, “computer lingo”, and “html”, subjects could not correlate what they were seeing within the edit box to what they saw on the article page.  Most subjects opened a separate browser window to view the static article as they were making their changes and used preview and save before they had finished their work to monitor their editing progress and results.
Fail
“[This is] where I’d give up.”  – ‘Shaun’
“ There sure is a lot of stuff to read.” – ‘Dan’

Yes, we can admit it.  In some ways it seems, we are failing our users.  The tasks that users most often failed to accomplish were adding references, creating a new article, and successfully finding help.  When adding references, users questioned where and how to enter their sources.  Once they established a location, they struggled with both the interface and the complex wiki syntax.  Several users, while scanning pages to try to figure out how to create a new article, saw ‘create a book’ and thought “add a wiki page” was what they were looking for.  In both adding references and new articles, some users consulted help.  Help proved to be quite a labyrinth, where the cheat sheet was one source of shining light.
Check out this highlight video (in English only for now) on Commons:
Editing Wikipedia Makes Me Feel Stupid
Explain the Editing Process to Me
I Can’t Tell What This Really Looks Like
Though we cannot tackle the full scope of issues that our study participants brought up, brought to our attention, and validated, it was an eye-opening and learning experience for the whole team.  The study informs us as we take a further look at the most effective changes we can make to lower these barriers for potential Wikipedia editors.  We look forward to sharing our complete report and initial ideas with you – stay tuned!
Parul, Wikimedia Usability Initiative

Archive notice: This is an archived post from blog.wikimedia.org, which operated under different editorial and content guidelines than Diff.

12 Comments
Inline Feedbacks
View all comments

It seems to me there’s an element of capturing the reader’s / editor’s intent, which is important to usability, and not covered in this overview. This is not something that it would be easy for users to vocalize if they had not thought at some length about what was missing in their experience, rather than what frustrated them. For instance — the most enjoyable user interfaces have a natural tutorial, often that happens automatically as you make your way through the first few stages of engaging a game, book, tool, or site. This is different from the Help and other… Read more »

You needed a STUDY to figure this out? Is this what donations are going towards?
That there is no WISIWYG editor is so obvious, you should not have to spend any time or money studying it. Just fix it!

One thought I had is, most new editors are likely to edit mainspace initially – they want to add an article, modify or improve one, or correct one. Mainspace is a lot more open to being simplified, than (say) template space, which has a lot more functional options – and crucially, mainspace may also be open to a “simple” page editor option. In other words, my response to this video would be to develop a simple rich text editor that handles /basic/ wiki markup in mainspace at a minimum: 1/ SPLIT OFF HEADER TEMPLATES, CATEGORIES AND INTERWIKI LINKS: – use… Read more »

Will the final report include a note about how “unwelcome” User:NawlinWiki made the study participants feel when he indefinitely blocked their accounts for “abusing” Wikipedia?
http://en.wikipedia.org/w/index.php?title=Special:Log&page=User%3AUsability_Tester_3

I’m no usability expert – but I think this suggests that it would be wise to create some sort of live preview (preview changes as you type), so that users can correlate wiki-markup with what appears on the screen. (and doesn’t have the problems associated with WYSIWYG)

Thanks Parul, that was really fascinating. I look forward to more reports from the usability team!

Thanks for all of the responses and feedback! SJ – We can’t tell you how much we appreciate and agree with your comments. You’ll be happy to know that these issues were brought up and uncovered in our interviews as well as in our brainstorming! There are so many possibilities for change that think would benefit current and potential readers and editors. Being limited in resources and time we will have to be strategic in moving forward with those that can have the greatest overall impact. Albert Francis – While many of us at the foundation and many community members… Read more »

The simple, the best{{fact}}. We should consider making a ==comming soon== section at the bottom of every article. Just asking people to copy their [old good plain text] there with [WP:ref|reference] if any. We’ll do the fancy stuff.
Reading those survey, I’m sure we would get a lot of unsuspected help that way.

We don’t need it easier to edit. This would allow unintelligent, uneducated anybodies to create more havoc, vandalism, nonsense, etc. The interface is not complicated, you just have to spend a bit of time to understand our style manual, policy and guidelines.

I hope that you guys try and correlate the difficulties that the testers experienced with the decline in Wikipedia editing and the increase in references. It sounds like your study is ignoring the interpersonal aspects of Wikipedia, which I think is a huge mistake. Wikipedia is basically dying right now (see stats below), and a WYSWYG may not be enough to stop it. Four years ago when I started, I could edit articles willy nilly, and no one cared. Now, the well written articles on my watchlist, the GA and B class ones (which tend to be high traffic), are… Read more »

You’re right, one side of the coin could be “if it’s easier to edit, you’ll get more vandals”. However, there’s another side: if it’s easier to edit, you’ll get more contributions from people who otherwise would be discouraged. Take senior citizens, for example, many of whom have a lot of free time on their hands and a large amount of knowledge, resources, and experiences. If we increase the usability of the sites, it’s easier for them to contribute and our sites become a lot better.

I think what is needed is to make a real editor GUI. Much more wikisyntax should be accessible by simply pressing a button.