AI-Driven Operational Intelligence
Observability redesign, marketing automation, and a new platform for CSMs
Netflix Customer Service Observability
A customer support observability tool at Netflix, used by a number of groups supporting customer streaming experiences and security anomalies. The project involved extensive research and a major redesign to prepare for AI integrations.
6sense Marketing Automation
The 6sense canvas-based visual experience that helps DemandGen and Marketing Ops experts design, execute, and automate omni-channel marketing strategies with the right content for the right users at the right time - all in one place.
Oracle New Platform for Ops & CSMs
An integrated ecosystem at Oracle for CSMs, Ops leaders, and partners that includes web, tablet, and mobile experiences. The system connects all users, provides customer 360 views, simplifies onboarding, and increases renewals.
Case Study 1: Netflix Customer Service Observability
Visibility for Streaming and Security Investigations
The Problem
Users of a Netflix streaming observability tool were reporting difficultly in navigation and fragmentation when running investigations. The team supporting the system needed to prepare for both platform scale and the addition of AI for productivity enhancements.
Project Outcome
The overall outcome was a cleaner, more understandable experience with reduced cognitive load that still supported advanced workflows across multiple teams.
The previous system approach was treating all users as identical. Instead of choosing one extreme, I proposed a layered model (simplified defaults for most users / advanced capabilities progressively revealed for power users).
Grounded Decision-Making
The 5W1H framework provides a solid foundation for the team to work from:
Why are we building this? What gap is being filled or pain point being addressed?
What assumption are we making about users?
What do we need to know about context?
How will we measure this?
The highlights for this project were:
A UX review showed signs of cognitive overload and inconsistencies, along with reported gaps in understanding workflows.
The system needed a clear path toward more usability, scalability, and AI-driven patterns that could be used across the platform.
The initial assumptions to be verified were that the focus was more on deeper work, clarity, and streamlining workflows.
Measurement would come from both telemetry data, in terms of what’s actually being used, and qualitative feedback from users on the overall experience (ie. time saved, productivity enhancements, ease of use, etc.)
Research & User Themes
Using telemetry analytics for the app, I created a list of power users (combined with short bio’s from Slack). Together, these gave me 18 users to meet with, across groups.
I found the following themes (repeating patterns across groups):
Streaming Context: Users could locate a stream in question, but usually had to dig deeper for better context around the ‘before and after‘ (was the issue related to a device, the network, or user behavior)?
Docs and tooltips were missing or needed updates
Cognitive overload was bad for all - but a major roadblock for new user onboarding
Data retention issues were a bottleneck when needing to return to data later.
Navigation: The navigation needed to be redesigned for scale (with AI assistants in mind).
Patterns: There were few standards or reusable patterns being used.
IA Redesign
The redesign of the information architecture took into account contexts that were most used, used occasionally, or never used (sometimes buried), as well as new objects that were coming into play in the roadmap.
This sitemap also provided a rudimentary “‘before and after‘ picture for Engineers to help in creating a more detailed change log.
I used the following approach:
Preserve the location of the most frequently used navigation. This approach saved real estate and consolidated views into like groups - without disruption to existing flows.
Reorganize lower visibility logs and utilities into more logical groupings.
Add a new high profile object to the top level of the navigation, leveraging the format from an existing page-level pattern (the “User“ object that would track an individual users behavior and viewing history).
Mapping Flows
Mapping detailed workflows for common, repetitive use cases happened in 3 parts.
Mapping flows in progressive depth:
The first was strictly based on user feedback. Even this high-level view gave me insight into use cases and where to focus next.
Claude research: I knew this effort would eventually be used as deterministic data to feed to an AI assistant to perform an investigation for a user (as well as the complexity of many of these investigations), so I leveraged Claude for research (which referenced GitHub, Slack, Google docs, and other repositories) to help me build detailed flows.
Validation: The third phase was to get users from each group to validate or correct these flows (a much easier starting point for users with limited time). This happened both in meetings and async on Slack.
Patterns and Prototyping
The approach for usable, scalable patterns was:
Create a menu (with submenu) that could be pinned. Pinning the menu - and persisting that state with cookies across sessions - gave users a choice of less clicks or more real estate.
Reducing Cognitive Load was addressed in 3 primary ways:
Keeping a unique ID by default by placing all other object metadata under a tooltip
Breaking up content for complex pages that included tabs inside of tabs inside of tabs. Resulting pages were easier to read and required less vertical scrolling.
Consistent component placement (page-level actions, CRUD functions, search, filters, and more).
Preventing disruption to existing users. Customization of the new menu (from the original design system variants) was needed to avoid disorientation for users (used to scanning top-level tabs to find the log needed). The DS version of the vertical toolbar had no label variant. We added labels to bridge the gap between old and new.
Context Aware AI Assistants
Aside from initial help chat (RAG system), it became clear that users needed an assistant that immediately understood context based on the trigger (button or link) it was launched from (a common pain point in playback stream investigations).
In this design the user can locate the stream in question, select “Show Stream Context“ from the actions menu, and the system could immediately provide:
Broader context around a wider timeframe (a few hours before and after the stream).
What else was happening across devices, the network, user actions, etc?
A simple output to sum up a conclusion that would have taken a user much more time to investigate.
Combining Assistants, Agents, & Workflows
The previous workflow mapping would become essential for this stage of the design - reviewing strategies with the team for leveraging deterministic workflow data to give an assistant a foundation for running investigations.
The workflow assistant would work as follows:
The system recognizes the logged in user and provides prewritten prompts (a prompt library).
The user selects a prompt (use case).
The user adds other required criteria (typically an ID and time window).
The system then knows which logs, fields, and values to review - as well as how to interpret the values it finds.
The assistant displays a recommended resolution (waiting for human approval).
The user can choose to trust the response or leverage the source links provided for verification.
FUTURE STATE: The assistant can eventually be instructed to close a case and complete the workflow by submitting a case description and resolution to Jira or Zendesk
Decisions and Trade-offs
Find the Right Balance. We preserved the most frequently used navigation items in the their existing locations and avoided disorienting experienced users, while reorganizing lower visibility logs and utilities into more logical groupings.
Designing for Scale. The approach to action buttons, metadata, page-level patterns, and breaking up complex content into individual pages all played into a cleaner, more intuitive, and more scalable experience.
Approach to AI. Assistant features were designed progressively as Engineers and Data Scientists had more tested functionality to offer: (1) RAG-style help (2) context-aware searches (3) automated investigations.
Priorities: The project was broken into 3 primary phases:
Foundation: Set up the navigation, layout, and interactions to be more usable and scalable with patterns that could apply platform-wide.
Content audits: Deprecate logs that are no longer used and look for redundancies.
AI & Automation: Layer on AI assistants for improved productivity and to stitch together any disparate logs, metrics, and traces into connected and coherent workflows.
Case Study 2: 6sense Intelligent Workflows
Omni-channel Marketing Automation
The Problem
The existing product provided simple wizards to allow any platform user the ability to automate marketing-related tasks on a daily or weekly cadence but the product’s adoption and engagement was much lower than expected.
Project Outcome
The research ultimately showed that users, leveraging our omni-channel marketing tool, tended to build fewer but significantly more complex workflows with larger node structures. Because we had intentionally designed for progressive scalability instead of prematurely optimizing around assumptions, the system adapted much more cleanly as complexity increased.
It provided a path toward better adoption, increased revenue, allowed users to start small and scale as needed, and came with the added benefit of simplifying the overall platform experience.
Research and Take-aways
Talking with MarketingOps and DemandGen users, we wanted to know if low adoption was based on limited functionality, price, or being leery of trusting automation without constant visibility into the impact of specific actions.
Users told us the following:
They didn’t know when to engage small automation routines, disconnected from their campaign strategies.
Users need more visibility into what the system is doing.
They were confused by how to measure success.
They wanted better connectivity across channels, not just in Orchestrations but across the platform.
Ideation
From an initial workshop to ongoing design iterations, we moved through a number of design formats. The three main contenders are shown to the right, along with pros and cons.
The decision tree canvas format would provide many benefits over other formats (users can start simple, scale as needed, and can branch logic in multiple ways). It provides extreme flexibility in how users design their marketing strategies.
Eventually, we could connect contexts across the platform that previously existed in silos.
Pricing Models
After many rounds of pricing discussions, the strategy was to provide the app for “free” along with the ABM marketing package. Customers choose a specific tier that determines the number of credits they receive.
Even though each action you take in a workflow consumes credits, customers can start building their workflow with only one or two actions. They can run on-demand tests or automate the workflows daily or weekly.
The model removes barriers toward adoption and allows customers to start small and scale as they become more comfortable with the app.
Project Foundations
I set up libraries of mockups and prototypes for those elements that contained a large number of moving parts - and would create mass confusion without a high amount of organization.
From these libraries, engineers and Product Managers could quickly view any play (workflow template based on a marketing strategy) and the node configuration panels for each action a user could take.
This made communication, iterations, and updates possible in a complex, fast-moving project with multiple stakeholders.
Node Panel Designs
Node Config Panels were needed for 80+ nodes. Interactive Figma components and Auto Layout were used for multiple iterations and fast updates.
Setting layout standards early helped maintain consistency and simplified development.
Consistent labeling and terminology were established
Node menus were organized in a logical sequence from most to least common - and was verified with several beta users.
I started minimalist with visual design until we had customer feedback telling us otherwise. We weren’t yet sure how complex workflows would become and I didn’t want visual clutter to weigh the experience down.
Go-to-Market Templates
Leveraging wireframing tools, such as Miro and Figjam, we iterated on templates that were primarily focused on the most common GTM strategies.
These iterations also help us work through.
Canvas and Node validation
Template selection patterns
Onboarding guides in Pendo
A roadmap for adding in-app help content and videos
Alpha, Beta, and Usability Testing
Meeting weekly with over 30 beta users to go through their flows and any any issues they encountered, we were able to make most updates within the next sprint. We also conducted usability testing with 11 users to address any final questions or concerns around interactions and content.
The app was ready for GA, with the following feedback:
All customers loved the visual canvas and the ability to stitch together everything in the platform that was previously fragmented (design, update, and automate their strategies in one place).
New users highlighted the need for more guidance (an area for AI to add a number of enhancements).
Continued feedback from users (qualitative) and Pendo analytics (quantitative) data was set up to monitor where guidance would be needed most in future releases and to hone-in on future participants for usability tests.
Value Added. Lessons Learned.
The Beta period became more than just a method of getting feedback prior to GA.
Because beta users are given free credits in order to try out the tool, they can run their “tests“ as live production campaigns - and benefit immediately from the testing. So, unlike many passive beta testers, these users were actively engaged.
While we already had basic AI chat capabilities for searching help docs, we needed to expand that with deep linking and recommendations as a next step.
Case Study 3: Oracle Customer Lifecycle Management
From onboarding to renewal
The Problem
The primary focus of CSMs was related to monitoring customer usage and helping them throughout the contract period. However, their primary tools were spreadsheets and half a dozen other internal tools. They needed one system to bring it all together, prevent churn, and help Oracle compete against Amazon, Microsoft, and Google.
Outcomes
The ecosystem that our team created laid a foundation that could be scaled to provide much more value, but the first year of the project gained the following benefits for users (and the company):
Adoption of 4000 users in one year
One source of truth for all data (no more spreadsheets)
At-a-glance views of customer usage (current and trends)
Improved visibility and connectivity across internal users and external partners
Reduced time to onboard
Research and Take-aways
Since CSMs were a new role to Oracle (many coming into the company through acquisitions), no current systems were dedicated to the role and most customer data was stored on spreadsheets or required CSMs to navigate a labyrinth of Oracle homegrown apps.
After a few weeks of almost constant meetings, across CSMs and Ops - and reviewing differences in workflows across services and regions, themes started to emerge:
CSMs typically manage between 10 and 20 customers at any given time.
Everyone needs one data source (one source of truth).
Onboarding needs to be simplified with better visibility for all stakeholders.
Easy monitoring of customer usage was the highest priority.
CSMs and Ops needed better visibility into consulting partners.
Starting Points
After initial research, two primary facts became clear:
Users were not mandated to use our app if it didn’t meet their needs, which made adoption critical.
We would need to move quickly with very few established guidelines and a development team that was learning the code as they were developing it for an enterprise app.
I created strategies for a starting point to the experience within the requirements, ambiguity, and short timeline:
Leverage existing tools and patterns where possible, before creating your own (Oracle JET javascript framework, which included responsive components, and ALTA UI visual libraries).
Create simple layout and navigation patterns that improve on the SaaS group’s information architecture.
Focus on simple default views that can be accessed with very few clicks.
Provide rolled-up “quick views“ of any critical or common information.
Monetization Model
Oracle’s new “Universal Subscriptions“ model allowed customers to choose a monthly spend limit and contract length. We knew CSMs needed to monitor (and be alerted on) usage from the perspective of these new pricing buckets.
While we were creating the app for internal use, we kept our eye on the possibility of CLM becoming a ‘marketable SaaS product’ since an app dedicated to CSMs was not currently offered at Oracle.
MVP Use Case #1
Onboarding Checklists
With very few exceptions, CSMs rarely needed to spend an excessive amount of time with deeper dives. A majority of their primary tasks (the biggest value ad for customers) was in quickly monitoring usage, health, key events, and onboarding statuses.
The first feature (and design goal) was:
Get CSMs off of spreadsheets and on to a dedicated UI with a shared data source.
As with all features for the first release, start minimalist (using simple patterns and content that is fast and easy to access).
MVP Use Case #2
Usage Monitoring
Through continual collaboration after project kickoff, I suggested a simple format that would provide the highest priority CSM requests for the first release:
This expandable row content provides:
At a glance view of where the customer is now
A trend of the last few months. Issues? Do they need more credits?
Financial rollups in local currency and constant dollar
Notifications tied to the more important attributes (>80% usage, etc.)
A method of drilling deeper (when those views become available in future releases)
MVP Use Case #3
Event Tracking
The eventual goal with events and notifications was to have an in-app “Notification Center“ that provided access to all notifications, with search, filter, and foldering options.
This simple timeline view, accessed with one click from the Customers list, provided a few higher priority events to lay the foundation for future work.
Order Booked or Changed
Subscription is past due
Fail to Pay or Suspension
Expiring soon
Overage (current month)
Over 80 usage (current month)
Connecting Partners to the Single Source of Truth
Because many large enterprise customers were onboarded by large consulting partners like Deloitte and Accenture, CSMs needed a way to bring them into the fold so that any project updates reflected immediately in the CLM database.
The team spoke with a few project managers from these partners to understand what core data they needed, what they should have access to, and a little bit about their environment. When learning that many leverage tablets or touch-screen laptops, I quickly redesigned the existing app into a tablet-friendly format by:
Documenting what data could be exposed and what needed to be editable (statuses, notes, etc.)
Moved to a horizontal nav to gain width for tablets (landscape orientation)
Simplified both list and detail screens
Began conversations for better personalization in future releases
Mobile Design for Always On Functionality
Mobile became a core requirement in order for CSMs to stay connected to alerts that may highlight the need for immediate action.
A new mobile project was started on top of the large backlog of items we were currently working through. We needed a simple starting point to leverage existing patterns and platform conventions where ever possible and to organize mocks in a way that streamlined discussions.
To that end, I created a combination prototype and sitemap. Team members could either click through the mobile prototype and see where in the app structure that page was (by following the highlights) OR jump immediately to a page being discussed by clicking on a page in the sitemap.
After getting buy-in on the structure, navigation, and content, I created spec docs for engineers to call out specific layout nuances, icons, colors, etc.
Value Added. Lessons Learned
App adoption went from 0 to 4000 users in a little over a year after different groups at Oracle (Sales, Operations, etc.) found that they could access customer 360 data very quickly with CLM.
The consensus was that CLM provided a solid foundation for growth and created improvements in:
Onboarding
Fast access to customer data
Almost real-time connectivity to customer events and usage