Research for Product Management
This blog is about a series of moderated usability tests and user interviews which I conducted to learn more about user expectations for the Outlook and Teams calendaring experiences. I led this research initiative myself, with the support of a staff UX Researcher who helped me create a strong research plan.
Note: due to non-disclosure agreements with my former employer, some feature specifics of this project have been omitted or changed.
Context
In early 2022, I was working on the Microsoft 365 Platform SDK team. Our goal was to launch a single Developer SDK which allowed developers to create apps which ran coherently in Teams, Office, and Outlook. This involved consolidating individual SDKs from each of the three platforms, and identifying how to meet developer needs in a single SDK which was backwards-compatible.
I was identifying SDK solutions for the Microsoft Calendar team to enable them to consolidate the Microsoft Teams and Outlook calendars into a single application package. While the two calendars had many UX similarities, there were differences in capabilities and workflows which led to difference in user experience between the two products. In this project, I will highlight investigations for one of these workflows.
The user task is creating and sending a meeting invitation in Microsoft Teams or Outlook — specifically, when a user navigates away from the in-progress meeting invitation before they are ready to send the invite. Users may navigate away from the invite to reference details in an email or chat message to include in the invite. In the diagram below, the user scenario and product behavior for the two platforms is shown: user action in orange, with Microsoft Teams behavior in purple and Outlook behavior in blue.
The difference in platform behaviors when the user navigates away from an in-progress invite was causing developer friction. The developers were unable to design a single app flow which handled both platforms’ behaviors. The testing goal was to learn more about users’ expected behavior and inform the Microsoft Teams and Outlook platforms what changes were recommended.
Method Selection
While beginning research, I had a few constraints and requirements factoring into my research plan:
- There was an upcoming online event with hundreds of Microsoft Teams and Outlook users, and I wanted to take advantage of a large cohort of research subjects. This biased me towards a moderated study so I could talk with users about their perspectives directly.
- I wanted the subjects to specifically encounter the behavior when navigating away from the meeting invite in an organic way, so I scripted a usability test to ensure they would follow a standard scenario.
- Because I did not want customers to record their screens with company or personal information, I used Figma to build a digital wireframe prototype to facilitate the simulate experience.
- I was wary of introducing bias via commentary, so I created a structured interview script and reviewed with a Researcher to ensure as much standardization as possible.
Participant Recruitment
During the time of the study, there was an online conference with hundreds of Microsoft “superusers”. Since the conference participants are verified Teams and Outlook users, I decided to target them for my research. My target persona was a Microsoft Teams or Outlook user who scheduled 5 or more meetings per week, with those meetings having three or more configured values (such as repeating events, conference room, attached files, etc.)
I issued a call for participation to the target user group, hoping for at least five subjects. I was overwhelmed to check the following day and see forty-five sign-ups! I considered down-selecting to a smaller cohort, but I remembered that customers feel positive sentiment from participating in research, so I met with all recruited subjects.
With my script and Figma prototype in hand, I logged into the customer conference and conducted forty-five identical 30-minute sessions.
Insights Gathered
Following the interviews, I annotated interview scripts using digital post-its and sorted them using a digital whiteboard. Here were the findings:
- About 60% of participants found the Microsoft Teams behavior unexpected and unwanted. Another 20% expected the behavior, as they had discovered it while using the product, but still found it unwanted. The final 20% desired the behavior. Less than 5% of the users found the Outlook behavior unexpected.
- Users came up with a variety of product suggestions for how to solve the problem. The most common was to retain the in-progress invite even when the user navigated away. When users learned that technical limitations prevented that, the next-most-common suggestion was to create a “drafts” folder for in-progress invites to be fetched later.
- About 40% of users indicated that unexpected behaviors in the Teams platform led them to choose the Outlook calendar whenever possible. No users said the same about choosing Teams over Outlook.
Outcomes
Following the usability testing, I had a stronger use case for the expected behavior of the system, and used that background to inform recommendations for changes to Microsoft Teams and Outlook to provide a smoother Calendar experience.
With 80% of users not expecting the loss of in-progress meeting invite in Microsoft Teams, it was important that the system provide more transparency to users about the behavior, or that the system be revised to match user expectations. Each of the recommendations was suggested by multiple users in the study, and were presented to the product teams for further exploration.
The product team conducted technical explorations for the suggested revisions, balancing user expectations with time-to-delivery expectations. Eventually the team decided on a short-term solution of a warning message to provide improved transparency on the product behavior, with a longer-term plan of implementing caching such that backgrounded apps would not lose data.
