Just like my mentor Aaron Gustafson, I like to do redesigns of my site out in the open. This page is currently in-progress. The yellow highlights are being written. Once they are completed, the design of the page will be finalized, the inline CSS moved to the main styles.css, and those styles repurposed for additional case studies.
Case study in progress.
IRS Strategic User Research Exploring Chatbots
IRS.gov piloted a chatbot in Refunds and Payments
Situation
The IRS piloted a chatbot on the Refunds and Payments sections of IRS.gov. The MITRE team was tasked with identifying barriers and opportunities for improvement. The sponsor wanted to understand if there were barriers for people with disabilities. They also wanted to understand the impact of overlapping barriers. An initial accessibility audit identified a few issues and interviews would explore more.
Audience
Wide range of American taxpayers
My Role
Lead Researcher
What I Did
Conducted an initial accessibility audit and provided a list of issues and recommendations.
Designed and led generative and usability research sessions, mentored researchers unfamiliar with research with people with disabilities, Veterans, other historically excluded experiences, and intersectionality. I brought a lot of experience with qualitative research with people with disabilities and other historically overlooked characteristics from my work on the VA.gov modernization and at Fidelity and Sabre. While I recognize that 'intersectionality' is currently 'sus' with the 2025 administration, from a UX perspective it is important to consider how barriers can impact the people we serve. If people have multiple barriers, it creates more friction, and increases the potential for some to be unable to access services.
Team
- Project Lead
- Task Lead
- Lead Researcher (moi)
- Two Researchers
Key Tools & Deliverables
- Research Plan
- Conversation Guide
- Observer Instructions
- Participant Schedule
- Analysis Spreadsheet
- Mental Model Diagrams
- Affinity Mapping
- Research Synthesis Report
Skills Used
- Accessibility Assessment
- Chatbot Design
- Design Leadership
- Mentoring Junior Staff
- Research Design
- Qualitative Methods
- UX Research
Challenge
As noted, the IRS sponsor wanted to understand how the chatbot pilot could be improved. They recognized that people with disabilities, that speak other languages, use older devices or have bandwidth challenges, and other considerations might need changes to the chatbot. Our team sought to assess the chatbot for accessibility and usability heuristics, as well as conduct generative and usability research to hear from taxpayers.
Actions
- Accessibility Issues. The accessibility assessment identified that the chatbot was the last item in the focus order, meaning that assistive technology users would be unlikely to ever find it, since they would have to tab through everything else on the page first. Additionally, the chatbot lacked any focus items. If you set your focus within the chatbot you couldn’t tab through the buttons and inputs.
- Design Qualitative Research Protocol. Text.
- Conduct Interviews. Text.
- Analyze and Synthesize Interview Data. Text.
Result
Text to come
-
Accessibility Issues.
We recommended adding a link to access the chatbot be added to the navigation or banner area, so the team added a 'Skip to Chat' link next to the 'Skip to Content' link. The lack of focus was also addressed, as shown in the video.
Skip to Chat link
Chatbot focus - Design Qualitative Research Protocol. Text.
- Conduct Interviews. Text.
- Analyze and Synthesize Interview Data. Text.