Interview Coach: An Open-Source Interview Coaching Platform Based on Multimodal AI

Interview Coach is an open-source AI interview coaching platform developed by mlarsen-source, integrating speech recognition (OpenAI Whisper), sentiment analysis (Audeering wav2vec2), and large language models (Claude/GPT-4o) to provide job seekers with structured feedback on delivery, tone, and answer quality. The project uses a Next.js frontend + FastAPI backend architecture, supports local deployment, and balances data privacy with functional completeness, suitable for multiple scenarios including job seekers, educational institutions, and HR.

• Original author/maintainer: mlarsen-source
• Source platform: GitHub
• Release time: June 2026
• Project overview: Aims to help job seekers get multi-dimensional structured feedback by recording interview answers. Its core value lies in combining speech signal processing (sentiment analysis) with natural language processing (content analysis) to provide data-driven insights for interview preparation.

Workflow

1. Audio Recording & Upload: Frontend Next.js implements browser recording and sends to backend
2. Speech-to-Text: Generate timestamped transcribed text via OpenAI Whisper API
3. Sentiment Analysis: Run Audeering wav2vec2 model locally to output VAD (Valence/Arousal/Dominance) sentiment scores
4. LLM Feedback Generation: Combine transcribed text, sentiment scores, and interview questions to call Claude/GPT-4o for structured feedback
5. Frontend Display: Present results with visual scorecards

Tech Stack

Layer	Technology Selection	Description
Frontend	Next.js (React)	Modern React framework supporting server-side rendering
Backend	FastAPI (Python)	High-performance asynchronous web framework
Speech Transcription	OpenAI Whisper API	Industry-leading speech recognition service
Sentiment Analysis	Audeering wav2vec2	Locally run sentiment model (≈1GB weights)
Feedback Generation	Claude/GPT-4o	Large language model API
Deployment	Vercel+Render/Fly.io	Serverless deployment solutions

1. Multimodal Fusion: Integrate audio sentiment analysis and text content analysis to capture non-textual information like tone and pauses
2. Local Sentiment Model: Sentiment analysis module runs locally, reducing API costs and ensuring data privacy
3. Structured Feedback: LLM generates actionable improvement suggestions based on specific data points instead of vague evaluations

• Job Seekers: Get objective feedback during pre-interview practice to improve expression habits and emotional delivery in a targeted way
• Educational Institutions: Use as a vocational training tool, integrated into simulated interview systems
• HR/Recruitment Teams: Train interviewers on assessment skills and establish standardized interview evaluation systems

1. Unfinished Service Integration: Whisper and LLM services are marked as "todo" and need to be fully integrated
2. Model Generalization: Audeering model is trained on MSP-Podcast, which may have insufficient adaptation to different accents/scenarios
3. Lack of Long-Term Tracking: Currently only supports single analysis; need to add user progress curve tracking functionality

Interview Coach is an excellent open-source case of multimodal AI in the field of interview coaching, providing developers with an end-to-end AI application learning model and job seekers with a low-cost, efficient preparation tool. In the future, its functionality can be enhanced by completing service integration, optimizing model generalization, and adding long-term tracking, with broad prospects.