Voice AI Agent Workshop

How a Voice Agent Works

A voice agent is three pieces wired together in a loop. Understanding this loop is the whole conceptual foundation. Everything else is implementation detail.

The ear captures your audio and transcribes it in real time using speech-to-text. Latency here is everything: if your STT is slow, the whole agent feels sluggish. Deepgram's STT runs at sub-300ms end-to-end, which is fast enough to feel like a real conversation.

The brain receives the transcript and decides what to say. This is your LLM: it reasons over the conversation history and your system prompt, generates a response, and can call any functions you've given it access to. It can look things up, run code, fetch data — anything you wire in.

The mouth takes the LLM's text response and streams it back as audio. Fast streaming matters here too: you want audio to start playing before the full response is generated, or the agent feels like it's thinking too hard.

With Deepgram, all three pieces run over a single WebSocket connection. You're not juggling three separate APIs — it's one socket, one loop, about 80 lines of code to start.

Design Decisions That Actually Matter

Most tutorials skip these. They're the difference between an agent that feels like a demo and one that feels like a tool.

Interruption Handling

In a real conversation, you don't wait for the other person to finish before you start talking. Your agent shouldn't either. When the user starts speaking mid-response, the agent needs to stop its output and listen. Getting this wrong is the fastest way to make an agent feel robotic.

Deepgram's Voice Agent API handles this for you. The WebSocket connection detects voice activity on both ends and manages the interrupt logic. You don't have to implement it yourself — just don't override it.

Conversation State

Every turn in the conversation needs to be threaded correctly. The LLM needs the full context of what's been said so far — by both sides — to give coherent responses. This means maintaining a message history and passing it in with every LLM call.

Keep your context window in mind. Very long conversations will eventually push older turns out of the context. A simple solution: keep the system prompt, the last N turns, and let the rest roll off. The agent won't remember everything, but the conversation will stay coherent.

Latency

The number that matters is end-to-end: from when the user stops speaking to when audio starts playing back. Under 500ms feels conversational. Over 1 second feels like a phone call with bad signal.

Three places latency hides: STT processing time, LLM first-token time, and TTS start time. Streaming helps with all three. Start playing audio as soon as the first TTS chunk arrives. Choose an LLM model that prioritises speed over capability for conversational use — a smaller, faster model is often better here than a smarter, slower one.

Getting Set Up

Five steps to a running agent:

1. Sign up at console.deepgram.com. Your account comes with $200 in free credit — more than enough for this workshop and a weekend of hacking.
2. Grab your API key from the console. It lives under API Keys in the left sidebar. Copy it somewhere safe.
3. Clone the starter repo. We'll walk through this together in the workshop, but the pattern is:

   git clone <starter-repo-url>
   cd <repo-name>

4. Add your API key to the environment. Create a .env file in the project root:

   DEEPGRAM_API_KEY=your_key_here

5. Install dependencies and run it:

   # Node
   npm install && npm start
   
   # Python
   uv venv && uv pip install -r requirements.txt
   python main.py

If it's working, you'll see a connection message in the terminal and the agent will greet you when you speak. If it's not working, come find us at the booth.

Make It Yours — Three Modifications

The starter app works, but it's generic. These three modifications are where the session becomes yours. Each one takes about five to seven minutes. Do them in order, or skip to whichever interests you most.

system_prompt = "You are a helpful assistant."

system_prompt = """You are Alf, a friendly AI assistant at a hackathon.
You give encouragement, suggest project ideas, and answer questions
about voice AI. You're enthusiastic but concise - people are busy building."""

import random

def get_project_idea():
    ideas = [
        "A voice-controlled to-do list that reads back your tasks",
        "An AI study buddy that quizzes you out loud",
        "A real-time translator that speaks back in the target language",
        "A voice journalling app that summarises your entries",
    ]
    return random.choice(ideas)

What You Can Build From Here

Now that you have a working, personalised, function-capable voice agent, here's what that unlocks for your hackathon project:

• Accessibility layer — add voice input and output to any existing interface. Users who can't type or read small text get a completely different experience.
• In-game NPC — drop the agent into a game as a character that actually talks back. Hook the function calling to your game state so it knows what's happening.
• Voice-controlled developer tool — talk to your build process, your deploy pipeline, your monitoring dashboard. Voice is an unusually good interface for things you want to do hands-free.
• Multilingual support — Deepgram's STT handles dozens of languages. The LLM can respond in whatever language the user speaks. Global voice interface, almost for free.

Get Help

Building something? Stuck on something? Here's where to find us:

• At the event: the Deepgram booth — we're here all weekend
• Docs: developers.deepgram.com
• Community: Deepgram's Discord

The Deepgram challenge prize this weekend goes to the team that builds the most creative voice-powered experience. Come say hi, show us what you're building, and let us know if you want feedback on your voice integration before judging.