Brice Bai

Generative AI Workflow Generator

Internal AWS Product

Duration
Feb 2024 - Jan 2025
My Role
Lead UX Designer
Stakeholders
UX Researcher
Product Manager,
Engineering Manager,
10 Engineers
Impacted Users
25k monthly active internal users (Cloud Support Engineers)


In order to reduce the time and effort required for Amazon’s internal Cloud Support Engineers to build, manage, and maintain new automated workflows for case support resolution, our team created the Workflow Generator for on-demand, AI-generated workflow code.

I led the North Star and MVP design of this new console, which enabled internal builders to quickstart the workflow coding process through an AI-generated workflow draft. While I transitioned away from AWS right after MVP launch, the design resulted in highly positive user acceptance testing feedback, with users rating the ease of use of the Workflow Generator as 4.42 out of 5. It is expected to directly impact the org's goal to reduce builder effort by at least 50%.

My contributions

  • UX North Star vision, leading to a clear roadmap for product to work backwards from
  • Launched MVP design
  • User journey map
  • Collaboration with UX researcher in 2 usability studies
The Problem

Builders spend extraneous time and manual effort converting Standard Operating Procedures (SOPs) into workflows, which assist in customer support case resolution. The rotational nature of builders and lack of standardization of SOPs make the jump from SOP to workflow code difficult.

The Solution

Automate drafting initial workflows with AI. Create a simplified, single pane of glass console for building and testing workflows so that builders can produce high-quality workflows more rapidly to help automatically resolve customer support cases.

Jump to launched design ↓
Background

What is a workflow, and who builds them?

A workflow is an automation of steps in a support case lifecycle. Workflows drove 2023 case deflections for 9.98% (297.6k) of Support cases, saving ~$42.1M.

Today, builders spend extraneous time and manual effort converting Standard Operating Procedures (SOPs) into automated workflows. Once a rotational builder is assigned to a workflow, they must manually work in Quip to visualize workflow steps, determine API calls, find dependencies, implement unit tests, etc, and then manually create the workflow code in their IDE of choice.

There is an opportunity to simplify this process and configuration to allow builders to create and release workflows more efficiently.


The primary user

The builder is an internal Cloud Support Engineer. They are rotational and get assigned workflows to code.

While publishing a workflow involves many users and levels of approval, we are focusing on the stage where a builder is coding the workflow itself.

Success metrics

Together with my PM, we established success metrics that the design would impact on the workflow generator.

  • Increased number of workflows running in Production
  • Increased number of workflows per builder per rotation
  • Decreased builder time spent coding a workflow (by 50%)
Process

⚠️ Due to the NDA I am under with Amazon, I am unable to reveal full details of my explorations, iterations, and vision work.

However, I can share parts of the internally released MVP experience and general information about my design process.


Research

Creating a builder user journey through understanding pain points and opportunities

To establish an understanding of how a builder codes workflows, I heavily researched the resources available to them in training. I read wikis, followed workshop tutorials, and let a builder walk me through how they coded a workflow. This helped me generate an initial user journey.

Wiki
Builder Workshop Tutorials
Walkthrough with builder

Current CX builder user flow

Competitive analysis of AI-assisted platforms

Additionally, I took inspiration from other AI-assisted coding platforms to see how AI improved developer productivity. I was inspired by:

AWS Code Whisperer’s coding snippet suggestions
Github Copilot’s conversational AI panel
AWS Systems Manager's visual coding functionality

My initial research led to the following tenets I established for this project.

Tenets

Reduce ambiguity.
We make the development process clear, guiding the builder through each step, surfacing instructions or inputs, and providing transparency of workflow status

Simplify manual processes.
We simplify and automate the beginning of this process, giving builders a launch pad to immediately refine details. We are designing for quick starts, small improvements.

Improve education and efficiency for builders of all levels of expertise.
We enable all builders to quickly and accurately deliver workflows, no matter if they're a beginner or a master.

Initial
Explorations

New user journey

I first began designing a North Star journey that covered all stages of the building, refinement, and testing process. In each stage, I made several lo-fi explorations that played around with different layouts for workflow building step information, the code itself, and where AI would be applied.

At a high level, the MVP journey focused on a simplified flow of generating a workflow from an SOP.

Prepare SOP

→

Upload SOP

→

Export workflow

→

Refine in IDE

The quickstart in generating an initial workflow draft tackled the initial stages of manual coding frustration in the builder's current journey. This led to my next step in refining the new flow with generative AI design principles.

Generative AI UX Framework

I used a Gen AI UX framework created by a team member, Principal UX Designer Rad Wendzich, to ensure I would follow AI UX design principles during the stages of user behavior where they interact with AI. In particular I paid attention to Gen AI branded ingress points, refinement options, step and analysis details, transparency of the model, history, and model tuning.

Validation

Partnering with UXR in 2 usability studies

To uncover ambiguities and refine my designs, I partnered with UX Researcher Carlos Cardoso, who led 2 usability studies with 13 builders for 45-60 minute interviews each for the North Star UX design of the Workflow Generator.

Study 1

7 participants

9 research insights

19 design recs

→

Iterate

Fixed inaccurate steps

Improved input guidance

Explored visual coding

→

Study 2

6 participants

10 research insights

64 design recs

→

Final North Star and MVP design

Overall, all believed the new UX would significantly improve the coding process

The UI was laid out clearly and provided a one-stop-shop for their building needs. Given that the AI conversion would provide useful output, it would certainly save time in building. Novice builders would benefit most from the Workflow Generator.

Certain steps needed to be added or removed for an accurate building process

Builders clarified ambiguities over workflow steps and the inputs needed in them. They also helped iron out details such as the number of JSON files in a workflow package. Their input was essential to ensuring accuracy.

Final MVP

Working backwards from North Star to MVP

After incorporating feedback from usability studies, I finalized North Star designs first, then worked backwards with my PM to prioritize what can be accomplished at the MVP stage.

While I cannot show the iterations leading up to the North Star CX in this case study, here are MVP mocks that have internally launched, ultimately prioritizing:

Informing the user on process and prerequisites

Prioritizing education on good inputs for the AI conversion

Emphasis on the converted workflow being a first draft to be refined

Handoff

Fit and finish ensured MVP tackled low-hanging fruit UX to improve the experience

After handing off the mockups to the engineering team, I reviewed the implemented gamma environment to find discrepancies or opportunities to improve UX and tackle technical constraints for the MVP. Through this fit and finish process, I was able to ensure the prioritized inclusion of low-hanging fruit, such as:

  • Adding links on SOP formatting guidance
  • Providing a max ETA notice on conversion time during the loading screen to ensure users of accurate progress
  • Warn users of lost generated workflow if returning to previous step to modify workflow

Results

Users viewed the console as a promising AI-powered tool that saves time

Because I had switched teams immediately after the MVP launched, I was unable to collect post-launch data. However, I obtained user feedback on my UX and the product in a gamma environment with user acceptance testing.

After trying out the Workflow Generator in gamma and seeing it convert SOPs in real-time, 12 users viewed the console as a promising AI-powered tool that saves builders time. They found it very helpful for generating initial workflow boilerplates but noted room for improvement in AI accuracy and visual representation.

On average, users rated the ease of use of the Workflow Generator as 4.42 out of 5 (on a scale of 1-5, 1= not at all helpful, 5 = extremely helpful).


Feedback

This has a lot of value. Let’s say when you’re new to this, this helps immensely generating a workflow document given the proper input. If it’s syntactically correct, that’s important. The workflow will save us this trouble.

Cloud Support Engineer participant in UAT

This is an amazing start. I have tried to prompt our own Bedrock assistant to generate this assistant. And they’re not well-trained to create a workflow. The fact that this has come to play and be honed in to generate workflows, I appreciate that. You don’t have to have a lot of tribal knowledge from people who have designed workflows before. This is democratizing the workflow creation. I anticipate and look forward to improvements to this version.

Cloud Support Engineer participant in UAT

It’s handy enough if I’m just uploading what I need, it’s giving me back more than what I expected it to do. I know how much time this would save getting this initial step out.

Cloud Support Engineer participant in UAT
Future
Directions

This was a highly technical project that required deep knowledge of the workflow building process in order to accurately capture its steps. I realized I could only understand the user and how to improve their pain points through “dogfooding”, learning how to code workflows as if I was a builder myself, and heavily relying on usability studies to validate specific ambiguities. On top of that, I also learned how to incorporate principles of AI UX design and think beyond what I typically consider in non-AI projects. If there’s potential for AI to improve a process, thinking about it early on while mapping out user flows and making low fidelity designs is important to ensure it seamlessly blends into the whole journey and provides genuine value.

In the future, the next designer may consider:

  • Continue working backwards from North Star and prioritize the next round of implementation of the Workflow Generator
  • Consider what other parts of the coding process could be automated with AI

CODED + DESIGNED ON bricebai. 欢迎光临 ✌🏻