Humanizing AI Voice for Powered_By with Prompt Engineering and Real-Time Booking
Engagement Highlights
Powered_By, a company focused on AI solutions provider, partnered with us to enhance the quality and realism of their Vapi-powered voice agents and Humanizing AI Voice.
The engagement focused on:
- Enhancing the natural language capabilities of the Powered_By website voice agent.
- Testing and refining prompts across multiple leading AI models including OpenAI, Grok, Anthropic, and Gemini.
- Automating real-time actions through the voice agent, such as calendar scheduling, to improve usability and user experience.
- Building metrics driven validation to assess agent performance, focusing on prompt success rates and user satisfaction.
Company Introduction
Powered_By is a cutting-edge technology company specializing in AI-powered customer experiences. They are on a mission to revolutionize how users interact with businesses through intelligent voice agents. With a focus on innovation and usability, Powered_By continues to push boundaries in conversational AI, aiming to make digital interactions feel natural and human.
Challenges
- The initial voice agent interactions were too mechanical and inconsistent across use cases.
- Prompt responses varied significantly depending on the AI model used, requiring a comparative analysis of different large language models (LLMs).
- The voice agent lacked the ability to perform real-time calendar tasks such as scheduling meetings, leading to limited practical usability.
- Need for robust performance validation across different scenarios and AI responses.
Goals
- Deliver human-like conversation experiences through advanced prompt engineering.
- Compare and optimize performance across top LLMs.
- Seamlessly integrate voice workflows with calendar booking systems, using third-party automation tools.
- Define measurable success metrics to track prompt reliability and user engagement.
Solutions
To address these challenges and meet the goals, we implemented the following solutions:
- Prompt Testing and Fine-Tuning:
Developed structured prompt variations and evaluated them across the different AI models (OpenAI, Grok, Anthropic, Gemini). Based on qualitative and quantitative assessments, we fine-tuned prompts for improved accuracy and contextual understanding.
- AI Model Evaluation Framework:
Set up a testing matrix to log and compare prompt effectiveness across different models, focusing on response clarity, latency, and actionability.
- Workflow Automation for Calendar Booking:
Integrated Make.com and Cal.com into the Vapi voice system. This allowed the voice agent to:
1. Check availability in Powered_By’s calendar.
2. Schedule meetings in real time.
3. Confirm and send invites all through a seamless voice interaction
- Custom Metrics Dashboard:
- Total prompts tested.
- Successful vs. failed prompts.
- Success-to-failure ratio for each model and interaction flow.
Business Impact
- Reduced calendar coordination time by approximately 80% through hands-free booking, eliminating the need for any manual intervention.
- The voice agent became significantly better at understanding and responding to user prompts, especially in conversations that go back and forth (multi-turn).
- Saved engineering hours through reusable automation logic and prompt templates.
- Improved user satisfaction early feedback highlighted that conversations felt more natural and genuinely helpful.
Related Projects
Ensuring Secure Multi-Platform MFT Automation & Compliance
Strengthening Real-World Reliability in a Distributed AI Execution Platform
