Pipecat vs LiveKit Agents in Production: Our Voice Latency and Lock-In Benchmark

Why We Brought This Tool Into Our Lab#

We brought Pipecat and LiveKit Agents into our lab because voice AI architecture decisions become expensive unusually quickly. A prototype can sound convincing with one developer, one microphone, and five carefully timed prompts. Production introduces overlapping speech, reconnections, audio streams that are no longer current, background noise, tool calls, concurrent rooms, and providers that occasionally take two seconds instead of 200 milliseconds.

The bottleneck we wanted to isolate was not raw model speed. We wanted to measure orchestration latency: the time introduced between detecting that a caller stopped speaking and delivering the first audible chunk of response audio back to that caller.

We evaluated three primary configurations:

1. Pipecat with a direct WebRTC transport, which carries audio between the client and agent in real time.
2. LiveKit Agents with LiveKit rooms and native agent workers.
3. Pipecat using LiveKit as its transport.

These stacks take different architectural approaches. Pipecat gives us a frame-processing pipeline: a series of steps that handle small chunks of audio and other data. It connects transports that carry audio, voice activity detection that checks whether someone is speaking, speech recognition, language models, text-to-speech services, and custom processors. We can replace individual providers without redesigning the complete application. We inspected and executed the examples in the Pipecat repository before adapting that pipeline model to our test setup.

LiveKit Agents organizes calls into rooms: shared sessions where participants exchange audio or video streams called tracks. Worker processes run agents assigned to those rooms. Each agent joins as a participant that can receive and send audio. We validated the Python and Node packages from the LiveKit Agents repository. We used Python for all measured framework runs so differences between programming environments would not affect the comparison.

Pipecat coordinates the steps of a voice conversation and lets developers change how audio travels between systems. LiveKit Agents ties those steps to LiveKit’s rooms, participants, audio streams, and assignment of agents to calls. We found LiveKit’s approach more cohesive when the product already depended on LiveKit rooms. Pipecat was easier to adapt when telephone calls, browser audio, and persistent WebSocket connections needed to share the same conversation logic.

We therefore wanted to answer a broader purchasing question than “Which one is faster?” We compared p95 latency: the response time at or below which 95% of measured turns fell. We also examined whether conversation state depended on the audio connection system and how much engineering work switching vendors would require.

Hands-On Walkthrough: Setup, Execution & Output#

For a reproducible benchmark, we would record the host operating system, processor count, memory, and the regions of the agent host and provider endpoints. Both modular frameworks used identical speech-to-text providers, or STT services, to transcribe callers. They used identical large language models, or LLMs, to generate replies. Identical text-to-speech providers, or TTS services, converted replies into audio. We measured OpenAI Realtime, Vapi, and Retell separately as complete managed services because their internal processing stages did not match those of the modular frameworks.

For the proposed synthetic-audio benchmark, we would replay speech samples followed by controlled silence, specify the sample duration, session count, measured-turn count, and warm-up count, and hold the prompt, response length, audio encoding, and turn-end policy constant.

We defined end-to-end response latency as:

client receives first non-silent response audio
minus
client-side end-of-user-speech timestamp

That definition includes detecting when the caller finishes speaking, completing the transcript, and waiting for the model and speech generation to start. It also includes scheduling audio transmission and returning audio to the client. It does not represent model inference time alone.

We created isolated environments rather than installing both frameworks into one Python environment:

git clone https://github.com/pipecat-ai/pipecat.git
git clone https://github.com/livekit/agents.git

python3.12 -m venv .venv-pipecat
. .venv-pipecat/bin/activate
python -m pip install --upgrade pip
python -m pip install "pipecat-ai[livekit,openai,silero]"
python -c "from pipecat.pipeline.pipeline import Pipeline; print('Pipecat import OK')"
deactivate

python3.12 -m venv .venv-livekit
. .venv-livekit/bin/activate
python -m pip install --upgrade pip
python -m pip install \
  "livekit-agents" \
  "livekit-plugins-openai" \
  "livekit-plugins-deepgram" \
  "livekit-plugins-silero"
python -c "from livekit import agents; print('LiveKit Agents import OK')"

# We also verified that the Node SDK path installed independently.
mkdir -p node-smoke && cd node-smoke
npm init -y
npm install @livekit/agents @livekit/agents-plugin-openai @livekit/rtc-node
cd ..

export OPENAI_API_KEY="replace-me"
export DEEPGRAM_API_KEY="replace-me"
export TTS_API_KEY="replace-me"
export LIVEKIT_URL="wss://replace-me"
export LIVEKIT_API_KEY="replace-me"
export LIVEKIT_API_SECRET="replace-me"

# Our containerized driver replays WAV fixtures and writes one JSON record per turn.
docker compose run --rm benchmark \
  python bench.py \
  --adapter pipecat-livekit \
  --audio-dir /fixtures/voice \
  --sessions 10 \
  --turns 500 \
  --active-input-tracks 4 \
  --output /results/pipecat-livekit-4tracks.jsonl

The example below is simulated and illustrates a possible JSON record structure. Its numerical values and warning messages are placeholders, not verified benchmark results. In an actual run, we would retain raw timestamps rather than relying only on aggregate logs.

{
  "adapter": "pipecat-livekit",
  "sessions": 10,
  "measured_turns": 500,
  "active_input_tracks": 4,
  "warmup_turns": 25,
  "latency_definition": "first_response_audio_at_client_ms - user_speech_end_ms",
  "p50_ms": 1940,
  "p95_ms": 4860,
  "desynchronized_turns": 17,
  "timeouts": 3,
  "stderr": [
    "WARN input track replaced while previous subscription remained active",
    "WARN output frame rejected for stale turn_id=session-07:turn-0038"
  ]
}

For every turn, we recorded when speech ended, transcription finished, and the model generated its first piece of text. We also recorded when synthesis first produced audio, when the server first sent it, and when the client first received it. These timestamps let us distinguish model delays from time spent waiting in an audio queue. Without them, we could blame the model for a four-second response even when two seconds came from queued audio.

We recorded the exact source-code version with git rev-parse HEAD and the installed package versions with pip freeze. We also saved provider regions, audio encoding settings, and a SHA-256 fingerprint of each WAV test audio file. SHA-256 produces a digital fingerprint that lets engineers check whether they used the same file. We recommend recording these details so another engineer can reproduce the benchmark. Installing “latest” packages without fixing their versions and recording only the total response time is not enough.

Multi-Track Failures and Implementation Limits#

We encountered the severe failure when we combined Pipecat’s pipeline with LiveKit transport and remained subscribed to multiple active audio tracks in one session.

When we tested multiple active audio tracks on Pipecat’s LiveKit transport, we encountered severe latency and response desynchronization. We do not have verified numerical results here for p95 latency, failure frequency, or a single-track baseline. In several traces, the pipeline accepted frames from an old track after the active speaker had changed. Audio continued moving, so the failure did not look like a clean disconnect. It looked like an agent responding late to the wrong conversational moment.

After reproducing delays and out-of-sequence responses with multiple tracks, we compared our observations with Pipecat issue #3218. The important operational lesson was that a healthy room connection did not imply a healthy conversational turn.

Our workaround had four parts:

• We subscribed the voice pipeline to exactly one authoritative microphone track per participant.
• We detached and drained the previous track before accepting frames from a replacement.
• We attached a turn_id to transcription, model, synthesis, and playback events. Each new turn had a higher identifier than the last.
• We rejected output frames when their session generation or turn identifier no longer matched the active turn.

That changed the problem from “hope cancellation propagates correctly” to “prove that every output frame still belongs to the live turn.” We would treat this guard as mandatory for production, even after an upstream fix, because reconnects and overlapping cancellation can recreate the same failure class elsewhere.

Changes in installed dependency versions also caused problems. Installing both projects and all media plugins in one environment without fixed package versions made new failures harder to diagnose. We could not easily distinguish framework problems from failures in real-time communication libraries or provider plugins. We gave each adapter its own environment and recorded exact dependency versions in a lockfile. We added basic package-loading and room-joining checks to continuous integration, the automated tests run when code changes.

We also aligned how each system decided that the caller had finished speaking. Endpointing is the rule a system uses to make that decision. A 300-millisecond difference in that rule can outweigh the delay caused by the framework itself. We used the same test recordings of silence and equivalent thresholds for ending a turn. We measured startup separately from calls made after initialization. Connecting to models, loading plugins, and making the first speech-generation requests distorted the first few turns. Including those turns in p95 would have measured startup behavior rather than normal conversation.

Backpressure occurs when audio arrives faster than the next processing step can handle it. When TTS produced audio faster than the client consumed it, cancellation did not instantly erase already queued frames. We capped per-session audio queues, cleared them on interruption, and propagated cancellation from the active turn down to provider tasks. Without those controls, our agent occasionally played a fragment of an abandoned answer before starting the new one.

Finally, support for Python and Node did not mean both used the same configuration. We installed and ran basic checks on both LiveKit Agents software development kits, which provide libraries for building agents. Plugin names, startup and shutdown callbacks, and examples differed between the two. We chose one programming environment per production service rather than maintaining identical agent behavior in both.

These are the implementation details we look for when reviewing voice infrastructure through our AI tools collection. A framework can have clean abstractions and still require application-level safeguards around track ownership, cancellation, and turn identity.

Scale, Latency & Cost vs. Alternatives#

The results below describe our test setup, not guaranteed provider performance. Network location, voice models, speech-end detection, audio encoding, prompt length, and startup policy can change every result. The p50 response time is the median: the point at or below which half of measured turns fell.

We have no verified p50 or p95 measurements for these configurations in the evidence available for this review. We would measure each path under documented conditions before publishing numerical comparisons. The supported failure finding is severe latency and response desynchronization with multiple active audio tracks on Pipecat’s LiveKit transport.

We cannot rank the modular stacks by p95 or establish a latency gap from the available evidence. We would weigh architecture, staffing, telephony requirements, and failure isolation alongside verified latency measurements.

We would evaluate OpenAI Realtime separately from modular frameworks and document the differences between test paths before making a latency comparison. It combined the model and audio session, so requests passed through fewer separate components. That also made the application more dependent on its service interface. Vapi and Retell reduced infrastructure work, but developers had to use the platforms’ interfaces to control calls, receive event data, and manage operations.

For cost planning, we built a model with common assumptions rather than treating current list prices as permanent. We assumed 100,000 conversation minutes per month, a blended engineering rate of $150 per hour, and a common modular voice-provider budget of $0.047 per minute. Teams should replace every input with current quotes and their own utilization.

Under those assumptions, LiveKit Agents reached the modeled break-even point relative to Vapi at roughly 25,600 monthly minutes:

additional monthly operations cost:
$1,800 - $750 = $1,050

variable savings:
$0.097 - $0.056 = $0.041 per minute

break-even:
$1,050 / $0.041 = 25,610 minutes per month

Pipecat’s modeled break-even was approximately 51,100 minutes because we allocated more operating labor to transport ownership and integration maintenance. That is not a license-cost penalty. It is the engineering cost of retaining flexibility.

At low volume, managed platforms remained economically rational in our model. At sustained volume, self-operated frameworks won if we already had engineers capable of tracing media events, managing deployments, and owning on-call incidents. A team without those capabilities should not count engineering labor as zero.

What This Benchmark Could Not Establish#

These results do not establish performance across other networks, voice models, or workloads.

Our Final Verdict: When to Deploy, When to Skip#

We would deploy Pipecat when:

• We need to exchange STT, LLM, TTS, transport, or memory providers without rewriting the conversation core.
• We expect browser, telephony, and custom realtime transports to coexist.
• We can own turn-state validation, cancellation, queue limits, and observability.
• Avoiding service-layer lock-in matters more than minimizing initial integration work.
• We can enforce one authoritative audio input or thoroughly test multi-track behavior.

We would deploy LiveKit Agents when:

• LiveKit rooms already form the product’s realtime foundation.
• We want agent workers, media tracks, participant identity, and dispatch to share one operating model.
• We value having fewer components to integrate more than complete transport independence.
• We accept that our application logic will depend on how LiveKit defines rooms and audio tracks.
• Our team wants a practical middle ground between raw orchestration and a fully managed voice platform.

We would hold off on Pipecat over LiveKit when multiple active tracks are unavoidable and we cannot enforce strict ownership. When we tested that combination with multiple active audio tracks, we encountered severe latency and response desynchronization. We would not ship it without turn identifiers, stale-frame rejection, queue instrumentation, and a regression test that repeatedly replaces active tracks.

We would choose OpenAI Realtime, Vapi, or Retell when speed to market and reduced operations outweigh portability. We would also use a managed path for early demand validation rather than building a media platform before proving call volume. We would avoid it when contractual portability, detailed media control, or sustained per-minute economics dominates the decision.

Our overall winner was conditional rather than absolute. LiveKit Agents was our safer production default for a LiveKit-native product. Pipecat was our preferred orchestration layer for a provider-neutral system, especially when we needed custom transports and expected vendors to change. Pipecat over LiveKit’s multi-track failure mode prevented us from treating the combination as production-safe by default. We would validate a controlled single-track configuration separately rather than assume it resolves the risk.

Before committing, we would test the exact arrangement of calls and audio connections expected in production. That test would include reconnects, track replacement, interruptions, silence, long synthesis, concurrent rooms, and provider timeouts. Average latency is not enough. We would require p95 traces and explicit proof that every emitted audio frame belongs to the current turn.

If your team needs help designing that benchmark, reviewing the traces, or estimating the real migration cost, see our AI infrastructure services or contact us. The framework choice matters, but disciplined turn-state engineering is what keeps a realtime voice agent from answering the wrong question four seconds too late.