{"id":705496,"date":"2020-11-17T11:23:56","date_gmt":"2020-11-17T19:23:56","guid":{"rendered":"https:\/\/newed.any0.dpdns.org\/en-us\/research\/?post_type=msr-blog-post&p=705496"},"modified":"2020-11-17T11:25:24","modified_gmt":"2020-11-17T19:25:24","slug":"a-guide-to-conducting-experience-reviews","status":"publish","type":"msr-blog-post","link":"https:\/\/newed.any0.dpdns.org\/en-us\/research\/articles\/a-guide-to-conducting-experience-reviews\/","title":{"rendered":"A guide to conducting experience reviews"},"content":{"rendered":"

By Denise Carlevato (opens in new tab)<\/span><\/a> and Marcella Silva (opens in new tab)<\/span><\/a><\/p>\n

\"\"<\/p>\n

User researchers have a variety of tools at their disposal, and one of the most flexible of those is the experience review<\/span>\u00a0(ER)<\/span>.\u00a0<\/span>We\u2019d<\/span>\u00a0like to share the process, learnings and best practices gleaned as<\/span>\u00a0we\u2019ve\u00a0worked on\u00a0<\/span>ER<\/span>s for teams across Microsoft<\/span>.<\/span>\u00a0<\/span>\u00a0<\/span><\/p>\n

What is an experience review?\u00a0<\/span><\/i>\u00a0<\/span><\/p>\n

An\u00a0<\/span>ER<\/span>\u00a0is a walk-through of\u00a0<\/span>an experience<\/span>,<\/span>\u00a0<\/span>focusing on a set of\u00a0<\/span>core\u00a0<\/span>tasks<\/span>\u00a0and sub-tasks<\/span>.<\/span>\u00a0<\/span>They are also known as Build Reviews, Expert Reviews, and Walkthroughs.<\/span>\u00a0<\/span><\/p>\n

ER<\/span>s should be approached purposefully\u00a0<\/span>with a\u00a0<\/span>mandate<\/span>\u00a0<\/span>to improve the experience.\u00a0<\/span>The beauty of the\u00a0methodology\u00a0is that it is flexible enough to be done at various stages of a project<\/span>; for example,<\/span>\u00a0checking\u00a0<\/span>craft<\/span>-and-<\/span>polish<\/span>\u00a0prior to shipping<\/span>\u00a0or\u00a0<\/span>reviewing an already-shipped product<\/span>.<\/span>\u00a0<\/span><\/p>\n

Why do an experience review?<\/span><\/i>\u00a0<\/span><\/p>\n

Bringing<\/span>\u00a0the different disciplines and their\u00a0expertise\u00a0together results in a very thorough examination of the\u00a0<\/span>experience<\/span>.\u00a0<\/span>I<\/span>t\u2019s<\/span>\u00a0important to\u00a0<\/span>set rules of engagement in advance to\u00a0<\/span>make<\/span>\u00a0sure<\/span>\u00a0<\/span>the meeting<\/span>\u00a0<\/span>is\u00a0<\/span>inclusive\u00a0<\/span>and\u00a0<\/span>provide<\/span>s<\/span>\u00a0<\/span>opportunities<\/span>\u00a0for all voices to be heard<\/span>, even if\u00a0they\u2019re\u00a0not in agreement<\/span>.\u00a0<\/span>\u00a0<\/span>We are also focused on\u00a0<\/span>building<\/span>\u00a0consensus around the key issues and their severity<\/span>, ensuring that products will\u00a0provide\u00a0the best experience for our customers<\/span>.<\/span>\u00a0<\/span><\/p>\n

What\u00a0<\/span><\/i>are some <\/span><\/i>process<\/span><\/i>\u00a0b<\/span><\/i>est practi<\/span><\/i>ces<\/span><\/i>?<\/span><\/i>\u00a0<\/span><\/p>\n

While each team approaches<\/span>\u00a0<\/span>ER<\/span>s differently,\u00a0<\/span>below\u00a0<\/span>are some basic steps that m<\/span>any<\/span>\u00a0of our<\/span>\u00a0teams follow<\/span>:<\/span>\u00a0<\/span><\/p>\n

    \n
  1. Pre-Review<\/span><\/b>:<\/span><\/b>\u00a0Prior to the\u00a0<\/span>ER<\/span>, we hold a s<\/span>coping meeting\u00a0<\/span>that<\/span>\u00a0determines<\/span>\u00a0the<\/span>\u00a0focus of\u00a0<\/span>the\u00a0<\/span>review<\/span>.\u00a0<\/span>This meeting includes program management<\/span>,\u00a0<\/span>r<\/span>esearch,\u00a0<\/span>and design<\/span>\u00a0to discuss the\u00a0<\/span>review\u2019s scenarios and timing<\/span>. A list of scenarios<\/span>\u00a0and<\/span>\u00a0tasks<\/span>\u00a0<\/span>must<\/span>\u00a0be brought to that meeting<\/span>,<\/span>\u00a0as well as a list of\u00a0<\/span>assumptions, including<\/span>\u00a0who\u00a0<\/span>the customer<\/span>\u00a0is<\/span>, what the customer already know<\/span>s<\/span>,\u00a0<\/span>and\u00a0<\/span>what they already use.<\/span><\/span><\/li>\n
  2. Readiness Review:<\/span><\/b>\u00a0<\/span>Here<\/span>,\u00a0<\/span>the team<\/span>\u00a0determine<\/span>s<\/span>\u00a0whether the\u00a0<\/span>ER<\/span>\u00a0can take place.\u00a0<\/span>Some walking through tasks will occur so the team can make some fixes before the\u00a0<\/span>ER<\/span>.\u00a0<\/span>All disciplines should agree that they are ready to conduct the\u00a0<\/span>ER<\/span>\u00a0before it happens.\u00a0<\/span>When the experience has already\u00a0shipped,\u00a0<\/span>it\u2019s<\/span>\u00a0a little bit different. The team will discuss<\/span>\u00a0what\u00a0shipped\u00a0<\/span>and what was the<\/span>\u00a0plan of record.\u00a0<\/span>They will reach<\/span>\u00a0agreement on readiness, scenarios,\u00a0<\/span>and\u00a0<\/span>assumptions<\/span>.<\/span><\/span><\/li>\n
  3. ER<\/span><\/b>:<\/span><\/b>\u00a0Attendees are the\u00a0<\/span>same people\u00a0<\/span>who attended the pre-review<\/span>,\u00a0<\/span>plus supportability,\u00a0<\/span>content writing, engineering, marketing, and\u00a0<\/span>people who\u00a0<\/span>aren\u2019t<\/span>\u00a0familiar with the\u00a0<\/span>experiences<\/span>, including customers<\/span>\u00a0if desired<\/span>.<\/span>\u00a0Having diverse attendees\u00a0<\/span>with different types\u00a0<\/span>o<\/span>f expertise\u00a0<\/span>allows us to become aware of a range of\u00a0<\/span>concerns<\/span>. The team<\/span>\u00a0will identify design problems that degrade the experience. Each task or subtask will be individually rated by the attendees<\/span>, noting whether the issue is of a blocking or non-blocking nature<\/span>.\u00a0<\/span>
    \n<\/span>NOTE: Two roles should be identified ahead of time:\u00a0<\/span><\/b> 1) <\/span>The notetaker should either be\u00a0<\/span>the<\/span>\u00a0researcher on the experience or a<\/span>\u00a0researcher familiar with\u00a0<\/span>ER<\/span>s. Sometimes,\u00a0<\/span>it\u2019s<\/span>\u00a0necessary to have two notetakers.\u00a0<\/span>\u00a0Recording is also\u00a0an option.<\/span> 2) <\/span>The moderator keeps the process moving and\u00a0<\/span>encourages<\/span>\u00a0attendees to take solution conversations\u00a0<\/span>offline.<\/span>\u00a0They are also responsible for ensuring that the conversation stays unbiased and the quality bar\u00a0remains\u00a0high.<\/span>\u00a0<\/span><\/li>\n<\/ol>\n

    4. Reporting<\/span><\/b>:\u00a0<\/span><\/b>After<\/span>\u00a0completion, the researcher produces a report\u00a0<\/span>of<\/span>\u00a0the\u00a0<\/span>task ratings as well\u00a0<\/span>as issues discovered per task. \u00a0It is also good practice to include recommendations for fixes. \u00a0The product team will then get together and make\u00a0<\/span>decisions an<\/span>d fixes to prepare for\u00a0<\/span>Ship<\/span>\u00a0Room.<\/span><\/p>\n

    \"\"

    Sample task list ratings for an ER report<\/p><\/div>\n

    5. Ship <\/span><\/b>Room:<\/span><\/b>\u00a0<\/span>Completion<\/span>\u00a0of an ER\u00a0<\/span>is considered\u00a0<\/span>to be<\/span>\u00a0<\/span>part of shipping requirements<\/span>.\u00a0<\/span>If mu<\/span>ltiple\u00a0<\/span>tasks\u00a0<\/span>are\u00a0<\/span>marked as\u00a0yellow\u00a0<\/span>\u00a0or<\/span>\u00a0any<\/span>\u00a0tasks\u00a0<\/span>are\u00a0<\/span>marked<\/span>\u00a0red in the\u00a0<\/span>ER<\/span>,<\/span>\u00a0the product or experience\u00a0<\/span>will<\/span>\u00a0not be allowed to ship until those issues are resolved.\u00a0<\/span>\u00a0<\/span><\/p>\n

    Some things to consider:\u00a0<\/span><\/i>\u00a0<\/span><\/p>\n