Wikimedia blog

News from the Wikimedia Foundation and about the Wikimedia movement

Usability Study Results (Sneak Preview)

Usability Study Sessions

The Wikipedia Usability Team has finished up their analysis of the User Experience and Usability Testing conducted this past March with Bolt | Peters!  As noted, please expect a full report on our project page soon.  While we are getting all of our i’s dotted and t’s crossed, we wanted to share with you some of the major themes and findings & successes and failures.

Outstanding

“Usually it’s the most information in the easiest spot to access.  It always looks very well put together….it boggles my mind how many people can contribute and it still looks like an encyclopedia.” – ‘Galen’
“I like Wikipedia because it’s plain text and nothing flashes” – ‘Claudia’
“It’s always one of the first hits on Google” – ‘Grace’

If there was one thing that was consistent and unanimous across our study participants, it was the assessment that Wikipedia is an incredibly valuable information resource whose accessibility is unparalleled.  Aside from its value as a reference, a time and lifesaver, and an up to the minute news resource, participants also praised it’s simplicity, coherence, and breadth.  Also, an extra thanks to Google, for showcasing just how referenced Wikipedia articles are – consistently making their links one of the top hits – our users count on that!

Room for Improvement

“Rather than making a mess, I’d rather take some time to figure out how to do it right.” – ‘Dan’
“Where are the rules?……I don’t really want to read all of this other stuff about what I”m supposed to do.” – ‘Grace’

All of our participants are Wikipedia readers, but had little or no experience with editing.  Generally the editing process was not a warm and welcoming one.  Before subjects even hit the ‘edit’ or ‘edit this page’ buttons, they voiced concerns about the rules, proper etiquette, formatting, and were naturally conscientious of and inhibited by maintaining the community expectations.  When a few of them attempted to find answers to their questions about rules and etiquette, they were overwhelmed with the amount of information and documentation they encountered.

“ [I felt] kind of stupid.” – ‘Galen’
“It looks all jumbled and crazy…I’m going down to the stuff that looks like it makes sense.” – ‘Tito’
“I’m not a programmer.  I know the letters PHP.” – ‘Seamus’

Once within the editing environment, most subjects commented on the illegibility of the hybrid Wiki syntax and article content – the more complex the article, the more exaggerated the response.  When users made it past their initial reactions, navigating around the syntax to perform basic word processing tasks (correcting a typo, inserting a block of text, bold and italics formatting) proved less problematic than finding a particular section, adding references, using tables, creating and naming links.  But not even our youngest and most computer savvy participants accomplished these tasks with ease.

“It’d be nice to have a GUI, so you could see what you’re editing.  You’ve made these changes and you’re looking at it, and you don’t know how it’s going to look on the page.  It’s a little clumsy to see how it’s going to look.” – ‘Bryan’
“On a blog it looks like the real page when you look at it.  I have a hard time looking at this and going back to the way it actually is.  I’m trying to correlate what’s on the real page.” – ‘Saurab’

Aside from feeling confused by the “code”, “computer lingo”, and “html”, subjects could not correlate what they were seeing within the edit box to what they saw on the article page.  Most subjects opened a separate browser window to view the static article as they were making their changes and used preview and save before they had finished their work to monitor their editing progress and results.

Fail

“[This is] where I’d give up.”  – ‘Shaun’
“ There sure is a lot of stuff to read.” – ‘Dan’

Yes, we can admit it.  In some ways it seems, we are failing our users.  The tasks that users most often failed to accomplish were adding references, creating a new article, and successfully finding help.  When adding references, users questioned where and how to enter their sources.  Once they established a location, they struggled with both the interface and the complex wiki syntax.  Several users, while scanning pages to try to figure out how to create a new article, saw ‘create a book’ and thought “add a wiki page” was what they were looking for.  In both adding references and new articles, some users consulted help.  Help proved to be quite a labyrinth, where the cheat sheet was one source of shining light.

Check out this highlight video (in English only for now) on Commons:
Editing Wikipedia Makes Me Feel Stupid
Explain the Editing Process to Me
I Can’t Tell What This Really Looks Like

Though we cannot tackle the full scope of issues that our study participants brought up, brought to our attention, and validated, it was an eye-opening and learning experience for the whole team.  The study informs us as we take a further look at the most effective changes we can make to lower these barriers for potential Wikipedia editors.  We look forward to sharing our complete report and initial ideas with you – stay tuned!

Parul, Wikimedia Usability Initiative

12 Responses to “Usability Study Results (Sneak Preview)”

  1. SJ says:

    It seems to me there’s an element of capturing the reader’s / editor’s intent, which is important to usability, and not covered in this overview.

    This is not something that it would be easy for users to vocalize if they had not thought at some length about what was missing in their experience, rather than what frustrated them. For instance — the most enjoyable user interfaces have a natural tutorial, often that happens automatically as you make your way through the first few stages of engaging a game, book, tool, or site. This is different from the Help and other places you would go for specific future knowledge, but related to how you conceive of the site, and how you learn the basic interface : what sorts of desires and impulses you can realize through the site, what actions and responses are possible, what the final outcomes should be / what they might look like.

    Another helpful facet that many more narrowly focused games or processes have is a metric of success. Either a series of challenges/watermarks one can visibly achieve or try one’s skill against, a set of running tallies that measure progress, or a visualization of yourself within the environment/group/site.

    Since there is no equivalent to either of these in the current site, new visitors could hardly comment on them in an orderly way, but to me they seem central to taking MediaWiki and Wikimedia experiences to a new level of usability.

  2. Albert Francis says:

    You needed a STUDY to figure this out? Is this what donations are going towards?

    That there is no WISIWYG editor is so obvious, you should not have to spend any time or money studying it. Just fix it!

  3. FT2 says:

    One thought I had is, most new editors are likely to edit mainspace initially – they want to add an article, modify or improve one, or correct one. Mainspace is a lot more open to being simplified, than (say) template space, which has a lot more functional options – and crucially, mainspace may also be open to a “simple” page editor option.

    In other words, my response to this video would be to develop a simple rich text editor that handles /basic/ wiki markup in mainspace at a minimum:

    1/ SPLIT OFF HEADER TEMPLATES, CATEGORIES AND INTERWIKI LINKS: – use expanding boxes for “tags at the top of the page >” and “categories >” or “cross-language links >” so the main rich text edit area is kept for the actual core article text for simplicity.

    2/ KEEP IT SIMPLE, MOST BEGINNERS DON’T NEED THE MORE COMPLEX MARKUP: – make the main text area handle sections, lists, bold/italic, and wikilinks, transparently… and a click button for “add reference” or “add a link to another website” which would guide the user to enter the needed data to go with those.

    3/ USE A “BASIC/ADVANCED” EDITOR OPTION: – I’d then have in config whether the “advanced” (current) editor, or the “simple” editor, is the default for a new/IP user on each namespace (you’d want “advanced” for templates for example), and add a “switch to advanced/simple editing” button to the edit screen. That way the current editor won’t scare people off.

    This would take care of transition for new editors, making it much easier to make the first few edits. We can worry about rich text for other users, or more complex text entry, later, but this basic change would at least remove the barrier for users new to wiki.

  4. Gregory Kohs says:

    Will the final report include a note about how “unwelcome” User:NawlinWiki made the study participants feel when he indefinitely blocked their accounts for “abusing” Wikipedia?

    http://en.wikipedia.org/w/index.php?title=Special:Log&page=User%3AUsability_Tester_3

  5. Bawolff says:

    I’m no usability expert – but I think this suggests that it would be wise to create some sort of live preview (preview changes as you type), so that users can correlate wiki-markup with what appears on the screen. (and doesn’t have the problems associated with WYSIWYG)

  6. pfctdayelise says:

    Thanks Parul, that was really fascinating. I look forward to more reports from the usability team!

  7. Parul Vora says:

    Thanks for all of the responses and feedback!

    SJ – We can’t tell you how much we appreciate and agree with your comments. You’ll be happy to know that these issues were brought up and uncovered in our interviews as well as in our brainstorming! There are so many possibilities for change that think would benefit current and potential readers and editors. Being limited in resources and time we will have to be strategic in moving forward with those that can have the greatest overall impact.

    Albert Francis – While many of us at the foundation and many community members have strong opinions and instincts on what the barriers are for new editors, it would be unwise to base our design and development decisions on our feelings alone. The study was conducted in part to validate some of these common assumptions while also allowing us to uncover any other major potential barriers that we had not thought of or experiences….which it did.

    FT2 – Thank you for your thoughts and potential solutions. We have been busy brainstorming implementation solutions, some of which closely resemble yours. As we prototype and iterate on solutions, we will be sharing as much as we can with the community. It would be great to get more feedback and suggestions from you as we go.

    Gregory Kohs – No it will not. But our finding that users were inhibited when they received conflicting messages and by their lack of knowledge of the proper rules and etiquitte will be.

    Bawolff – Discussions of live previews, wysisyg and such are all in session.

    pfctdayelis – Thanks for reading! We are very glad there is such an interest in our work from the community.

  8. Iluvalar says:

    The simple, the best{{fact}}. We should consider making a ==comming soon== section at the bottom of every article. Just asking people to copy their [old good plain text] there with [WP:ref|reference] if any. We’ll do the fancy stuff.

    Reading those survey, I’m sure we would get a lot of unsuspected help that way.

  9. joushi says:

    We don’t need it easier to edit. This would allow unintelligent, uneducated anybodies to create more havoc, vandalism, nonsense, etc. The interface is not complicated, you just have to spend a bit of time to understand our style manual, policy and guidelines.

  10. Peregrine Fisher says:

    I hope that you guys try and correlate the difficulties that the testers experienced with the decline in Wikipedia editing and the increase in references. It sounds like your study is ignoring the interpersonal aspects of Wikipedia, which I think is a huge mistake. Wikipedia is basically dying right now (see stats below), and a WYSWYG may not be enough to stop it. Four years ago when I started, I could edit articles willy nilly, and no one cared. Now, the well written articles on my watchlist, the GA and B class ones (which tend to be high traffic), are mostly a series of well intentioned but undesirable newbie edits followed by a revert from an experienced user. I can’t imagine getting started in that environment. I’d quit as soon as I saw that my edits didn’t stick.

    Couple of studies: http://en.wikipedia.org/wiki/User:Dr_pda/Article_referencing_statistics http://en.wikipedia.org/wiki/User:Dragons_flight/Log_analysis

    You guys should give Dr pda and Dragons flight $1000 each to create a bunch of nice stats, since they already do great jobs for free and I’m sure it would cost much more to get an outside stats person.

  11. Casey Brown says:

    You’re right, one side of the coin could be “if it’s easier to edit, you’ll get more vandals”. However, there’s another side: if it’s easier to edit, you’ll get more contributions from people who otherwise would be discouraged. Take senior citizens, for example, many of whom have a lot of free time on their hands and a large amount of knowledge, resources, and experiences. If we increase the usability of the sites, it’s easier for them to contribute and our sites become a lot better.

  12. FreddyE says:

    I think what is needed is to make a real editor GUI. Much more wikisyntax should be accessible by simply pressing a button.
http://blog.wikimedia.org/2009/04/24/usability-study-results-sneak-preview/
blog.wikimedia.org
crawl=CC-MAIN-2013-20
37899
application/http; msgtype=response
sha1:6JDNMWE4R25H7IO7S6NVEP4FFPMTIOQF
<urn:uuid:1cf39f99-3473-45f1-bc7f-ba28147aca64>
2013-05-24T04:40:15Z
208.80.154.49
sha1:ZQQXLXW7IHL4LLEDJJKG2RZRIZQ7ARWE
<urn:uuid:09f1871b-0d2e-4061-9db4-50e0f525a230>
http://blog.wikimedia.org/2009/04/24/usability-study-results-sneak-preview/
response
<urn:uuid:563179bd-fcb2-4466-a080-43c326b791ba>
robots: classic
hostname: ip-10-60-113-184.ec2.internal
software: Nutch 1.6 (CC)/CC WarcExport 1.0
isPartOf: CC-MAIN-2013-20
operator: CommonCrawl Admin
description: Wide crawl of the web with URLs provided by Blekko for Spring 2013
publisher: CommonCrawl
format: WARC File Format 1.0
conformsTo: http://bibnum.bnf.fr/WARC/WARC_ISO_28500_version1_latestdraft.pdf