Chrome Extension Uses Single Stream for Real-Time Caption Translation with Improved Efficiency

The article details the development of a Chrome extension that provides real-time translated captions for video calls on platforms like Meet, Zoom, and Teams. The extension captures audio directly from the browser tab without requiring a bot to join the call, addressing challenges specific to Chrome’s Manifest V3 (MV3) architecture. Key technical hurdles included handling tab audio capture in MV3, which restricts background scripts from accessing Web Audio APIs, and optimizing the translation process to reduce costs and improve efficiency. The author explains how they initially used two services (Deepgram for speech recognition and DeepL for translation), which proved expensive due to repeated translation of incomplete phrases. By switching to Soniox’s stt-rt-v5 model, which combines speech recognition and translation in a single stream, the cost dropped significantly—from $2.55 per hour to $0.154 per hour. The article also covers technical details like routing audio back to speakers after capture, polling for tab capture status to avoid conflicts, and handling latency issues. The author notes that while the new system is more cost-effective, latency remains higher than desired, and challenges like distinguishing between silent tabs and provider inactivity persist. The extension was tested using a simulated environment with multiple Chrome profiles to measure accuracy and performance. The author invites feedback on the MV3 audio capture implementation, particularly regarding more reliable methods than polling getCapturedTabs.