V
Vivek Kumar
Author
June 1, 202513 min read
Business
Featured Image
Managing AI products requires a fundamentally different approach than traditional software. As Vivek Kumar, our CEO, explains: "AI products don't follow the same rules—uncertainty is inherent, user expectations are misaligned, and 'done' is a moving target. The best AI PMs embrace this reality and build products around it."
87%
AI Projects Miss Expectations
3x
Longer Feedback Loops
60%
Time Spent on Data
∞
Continuous Improvement
What Makes AI Products Different
Inherent Uncertainty
Model performance varies, results are probabilistic, and edge cases are abundant and unpredictable
Magical Expectations
Users expect AI to be infallible; failures are confusing and erode trust quickly
Experimentation-Heavy
Development is research—hypothesis-driven with no guaranteed outcomes
Data Dependency
Product quality is bounded by data quality; GIGO applies exponentially
Product Discovery for AI
The AI Validation Framework
Before building, answer these AI-specific questions:
1
Is AI the Right Solution?
Could rules-based logic solve this? AI should be used when patterns are complex and rules can't capture them—not as a default choice.
2
Is Sufficient Data Available?
What data exists? How much is needed? What's the quality? Can you collect more? Data is the constraint most AI projects underestimate.
3
What Accuracy Is Required?
95% may be amazing for some use cases and catastrophic for others. Define the accuracy threshold that delivers user value.
4
How Do Users Handle Errors?
When AI is wrong (it will be), what happens? Design the error experience before the happy path.
Technical Feasibility Assessment
Assess early with your data science team:
- Data availability, quality, and access
- Model complexity vs. accuracy trade-offs
- Latency and performance requirements
- Infrastructure and compute costs
- Ethical considerations and bias risk
User Research for AI Products
Research AI-Specific Questions:
• How do users currently solve this problem? (Baseline for comparison)
• What's their tolerance for errors? (Defines accuracy requirements)
• How much do they trust automated decisions? (Informs UX design)
• What's the value of getting it right vs. cost of getting it wrong?
• How do users currently solve this problem? (Baseline for comparison)
• What's their tolerance for errors? (Defines accuracy requirements)
• How much do they trust automated decisions? (Informs UX design)
• What's the value of getting it right vs. cost of getting it wrong?
Defining Success Metrics
The AI Metrics Hierarchy
| Level | Metrics | Who Owns |
|---|---|---|
| Model | Accuracy, precision, recall, F1, latency | Data Science |
| Product | Adoption, task completion, time saved, satisfaction | Product |
| Business | Revenue impact, cost per prediction, ROI | Leadership |
Don't Optimize Model Metrics Alone: A model with 99% accuracy that no one uses is worthless. A model with 85% accuracy that users love and trust creates massive value. Product metrics matter more than model metrics.
Setting Realistic Expectations
Balance competing trade-offs:
| Trade-off | Consider | |-----------|----------| | Accuracy vs. Speed | Higher accuracy often means slower responses—what do users need? | | Automation vs. Control | Full automation vs. human-in-the-loop—what builds trust? | | Cost vs. Performance | Better models cost more to train and run—what's the ROI? | | Recall vs. Precision | Catch everything vs. never be wrong—what's worse for users? |
User Experience Design for AI
Designing for Uncertainty
"The best AI products make uncertainty visible, not hidden. Users can handle 'I'm 80% confident this is correct'—they can't handle unexplained failures."
- Show confidence appropriately:
- Confidence scores when users can act on them
- Alternative suggestions for uncertain predictions
- Clear indication when AI is guessing vs. confident
- Enable recovery:
- Easy correction mechanisms
- Feedback loops that improve the model
- Manual override when users disagree
Building User Trust
Explain
Show Reasoning
Start Small
Expand Scope
- Transparency builds trust:
- Explain how the AI works (without jargon)
- Show the reasoning behind predictions when possible
- Acknowledge limitations openly
- Gradual rollout strategy:
- 1. Start with low-stakes, low-risk use cases
- 2. Build user confidence through consistent performance
- 3. Expand to higher-stakes applications as trust develops
Error Experience Design
Design the failure mode before the success path:
Graceful Degradation
When AI fails, fall back to non-AI experience—never leave users stuck
Clear Communication
Explain what went wrong and what the user can do about it
Recovery Path
Make it easy to correct errors and continue the task
Learn from Failures
Use error feedback to improve the model over time
Working with AI Teams
Collaboration Model
- Product Manager responsibilities:
- Problem definition and user requirements
- Success criteria and prioritization
- User research and feedback synthesis
- Go-to-market and communication
- Data Science responsibilities:
- Technical feasibility assessment
- Model approach and experimentation
- Performance evaluation and improvement
- Production model quality
- Engineering responsibilities:
- Integration and infrastructure
- Scalability and reliability
- MLOps and monitoring
- Feature engineering pipelines
Effective Communication
- Explain business context—why this matters to users
- Understand technical constraints—what's actually possible
- Regular check-ins—AI work is unpredictable
- Joint success metrics—align incentives
- Share user feedback—close the loop
Development Process
AI-Specific Iteration
- Experimentation-driven approach:
- Hypothesis-driven experiments, not features
- Clear success criteria before starting
- Rapid testing and learning
- Pivot or persevere based on data
- Minimum Viable Model (MVM):
- 1. Build the simplest model that could work
- 2. Test with real users on real problems
- 3. Validate that AI adds value vs. alternatives
- 4. Improve based on feedback and performance data
Data Requirements Planning
Data Questions to Answer Early:
• What data is needed to train the model?
• Does the data exist, and can we access it?
• What's the data quality, and how do we clean it?
• How do we label data if supervised learning?
• How do we handle privacy and compliance?
• What data is needed to train the model?
• Does the data exist, and can we access it?
• What's the data quality, and how do we clean it?
• How do we label data if supervised learning?
• How do we handle privacy and compliance?
Launch Considerations
Beta Programs are Critical
AI products need beta programs more than any other product type:
Launch Communication
- Set accurate expectations:
- What the AI does (and explicitly doesn't do)
- Expected accuracy and limitations
- How to provide feedback
- Roadmap for improvement
Production Monitoring
- Track these metrics from day one:
- Model performance in production (not just test data)
- User behavior patterns and edge cases
- Error rates and types
- User feedback and satisfaction
Post-Launch: Continuous Improvement
AI Products Are Never Done
The Continuous Loop: AI products require ongoing investment. Models degrade over time (drift), user needs evolve, and competitors improve. Build continuous improvement into the team structure and budget.
- Ongoing activities:
- Performance monitoring and alerting
- Feedback collection and triage
- Regular model retraining
- Capability expansion based on usage
Managing Model Drift
Models degrade as the world changes:
- Mitigation:
- Automated performance monitoring
- Retraining triggers and schedules
- Data freshness requirements
- A/B testing model versions
Related Resources
Building AI-Powered Products?
We help product teams navigate the unique challenges of AI product development—from discovery through launch and beyond. Let's discuss your AI product strategy.
Discuss AI Product Strategy →Tags:
Product ManagementAIStrategyProduct DevelopmentUX Design