Powering Auditors' Work with AI

If I was designing an e-commerce website, even if it was selling products I had never heard of I would be able to somewhat put myself in the shoes of a potential customer and make certain assumptions. Designing for an auditor is different, I cannot simply put myself in the shoes of professionals who have studied and been in the practice for so many years. So here are some things I did to build empathy:

First, I got a better understanding of the audit persona! Auditors assess financial operations and determine whether financial documents, accounting entries and data follow generally accepted accounting principles and regulations.

Next, I tried to understand how auditors currently operate in the form of a user journey map. This step might be obvious but I have come to learn that this is an especially key step in designing AI powered experiences.

Seeing how auditors think through their workflow without AI will help me understand some concerns they might have while using the AI assisted experience.

Here is an overview of the current workflow, the current manual procedure of identifying regulation and the AI automated procedure:

Some questions the auditor might have when using the AI powered experience might be:

I set my design goals to focus on combating these questions and to ensure the auditor doesn't reject the AI recommendations.

At this stage in the project the data science team was still working on improving AI capabilities and could only output limited recommendations. In some cases, detailed descriptions of regulations (which would be helpful for auditors) cannot be surfaced up as part of the recommendation.

The following use case shows an auditor creating a document for a completely new client and how AI will suggest relevant regulations .

The key to building transparency is by showing why AI chose certain recommendations. When an auditor first turns on AI assisted recommendations, they are shown a brief description about how the algorithm works in a tool tip. User testing was done to determine the most optimal way to represent confidence intervals for each recommendation. Originally, designs showed % values, e.g. 97% match. Due to the nature of auditors' work, they are quite sensitive to numbers. Users voiced that they might not understand why a regulation that is only 87% is still surfaced as a recommendation. It was also voiced by the data science team that exact percentage values may be difficult to display. Switching to just language such as "Best Match" eliminated that concern.

This use case shows the flow of an auditor editing a document for an existing client and how AI will highlight sections with changing regulations. The first screen shows an aggregated view of all past documents created and the second shows what happens after the auditor selects a document.

This use case was designed based on the research finding that if an auditor has created a document for a client previously, they will always make edits to the existing document based on new regulation changes. Sections in the document involving a regulation change will be highlighted.
‍
There were lots of ways to design the changed regulation card (on the right of the screen). I wanted to better understand which piece of information will be useful for the auditor to see right away as they skim through the list of changes and which would be additional details. Working within the constraints of what type of information the data science team could output, I came up with the following shows 3 design iterations.

From most prioritized to least prioritized, the card sorting results were: 

Iteration 3 was chosen based on these research findings.

I learnt lots of new research and design techniques from being part of this project, here are some of the challenges I ran into and how I tackled them: 
‍
How to design for users with years of experience in auditing that I know barely anything about?
Throughout the design process I constantly felt like I did not have enough research. I simply didn't know enough about the audit space! I was constantly stuck and it felt like the options were endless. The only way that could narrow my options down was with more research. That leads to my next challenge...

It’s tax season! Auditors are busy and their time is expensive, how do I maximize precious testing sessions?
What really helped was to be prepared. I mapped out all possible answers to the questions I needed answers to. I also prepared wireframes corresponding to those answers. This way, I can show users different variations of how their needs can be met right away. It fits in iterations of design review in one session, which reduced the number of interview sessions needed.

For Auditors who have been creating these documents for years, with their own tips and tricks, how do we design an experience so they don’t intuitively resist that an AI algorithm already did the repetitive work for them?
Understanding the auditor's current workflow helps predict what concerns they might have about the AI algorithm. Setting design goals based on those will help minimize the concerns.

The project's direction pivoted and optimizing a different part of the auditor's process will be prioritized. When this project is picked up again in the future, the next step should be to build an interactive prototype that uses the AI algorithm and conduct testing to see if design changes need to be made to accommodate the performance of the algorithm.

This project solidified my interest in designing AI powered experiences and I will continue to look for opportunities to be on similar projects in the future!