Mervin Praison

Pipecat is an open-source Python toolkit for stitching together streaming speech, language models, and playback behind a single real-time loop—so product teams can focus on dialog design instead of rewiring transports for every vendor swap.

%%{init: {"theme": "base", "themeVariables": {"background": "transparent", "lineColor": "#000000"}}}%%
flowchart LR
  A["Caller audio"] --> B["Transport WebRTC or WebSocket"]
  B --> C["Optional VAD"]
  C --> D["Speech to text"]
  D --> E["LLM or speech to speech"]
  E --> F["Text to speech or native audio"]
  F --> G["Return stream"]

  classDef agent fill:#8B0000,color:#fff
  classDef hook fill:#189AB4,color:#fff
  classDef decision fill:#444,color:#fff
  class A,G agent
  class B,C,D,E,F hook

Why teams reach for Pipecat

Coverage snapshot

Quick start (from upstream docs)

Python 3.11 is the minimum supported release; maintainers recommend 3.12+. Core install stays lean—pull optional extras only for the providers you wire in.

# Scaffold a project (requires uv per README)
pipecat init quickstart

# Or add to an existing uv project
uv add pipecat-ai

# Provider-specific wheels (example pattern)
uv add "pipecat-ai[deepgram,cartesia,openai]"

Adjacent ecosystem

• Pipecat Flows for structured state machines when a single linear pipeline is not enough.
• Subagents repository for multi-agent buses when conversations need specialist hand-offs.
• Voice UI Kit for client-side React and native SDKs that pair with the same transports.
• CLI and Pipecat Cloud helpers for packaging agents once they leave a laptop.

At a glance

When two different coding agents share the same goal primitive, a messaging-first orchestrator can route long-running implementation to one stack and structured critique to another—without you babysitting shells from a desk.

%%{init: {"theme": "base", "themeVariables": {"background": "transparent", "lineColor": "#000000"}}}%%
flowchart TD
  U["User message from chat"] --> G["Gateway such as Telegram"]
  G --> H["Hermes agent session"]
  H --> P["Pick next worker for goal primitive"]
  P --> B["Codex build pass"]
  B --> S["Work completes"]
  S --> R["Claude Code review pass"]
  R --> K["Kanban or board state refresh"]

  classDef agent fill:#8B0000,color:#fff
  classDef hook fill:#189AB4,color:#fff
  classDef decision fill:#444,color:#fff
  class U,G,K agent
  class H,P,B,S,R hook

What each layer contributes

Why split builder and reviewer stacks

Controls operators still need

• Budget caps: background goals on two providers can burn tokens quickly—set spend alerts and pause hooks before enabling always-on queues.
• Deterministic gates: community feedback on the original thread highlights lint, type-check, and CI scripts as non-negotiable back-stops once models disagree.
• Secrets hygiene: remote chat triggers should stay behind paired accounts, command allowlists, and isolated working directories as described in Hermes security guidance.

At a glance

Floci is an MIT-licensed, AWS-shaped local emulator you run beside your laptop stack: point the normal SDK or CLI at localhost:4566 and exercise a broad slice of control-plane behaviour without touching a live cloud account.

%%{init: {"theme": "base", "themeVariables": {"background": "transparent", "lineColor": "#000000"}}}%%
flowchart LR
  A["App or test suite"] --> B["AWS SDK or CLI"]
  B --> C["Endpoint override localhost:4566"]
  C --> D["Floci router"]
  D --> E["In-process services"]
  D --> F["Docker-backed runtimes"]
  F --> G["Docker Engine on host"]
  E --> H["In-memory or disk-backed state"]

  classDef agent fill:#8B0000,color:#fff
  classDef hook fill:#189AB4,color:#fff
  classDef decision fill:#444,color:#fff
  class A,B agent
  class C,D,E,F,G hook
  class H decision

What changed in the landscape

Architecture at a glance

README benchmarks (idle laptop profile)

Minimal bring-up

# docker-compose.yml (from upstream quick start)
services:
  floci:
    image: floci/floci:latest
    ports:
      - "4566:4566"
    volumes:
      - ./data:/app/data
      - /var/run/docker.sock:/var/run/docker.sock

# Then:
docker compose up

export AWS_ENDPOINT_URL=http://localhost:4566
export AWS_DEFAULT_REGION=us-east-1
export AWS_ACCESS_KEY_ID=test
export AWS_SECRET_ACCESS_KEY=test

When to trust the emulator

• Treat Floci as a fast feedback loop for integration tests and demos—not a contractual guarantee of full AWS parity.
• Community threads already flag API conformance gaps on complex services such as DynamoDB; pair emulator runs with selective tests against real sandboxes when behaviour must be exact.
• Anything that shells out to Docker inherits host resource limits, socket permissions, and image pull policies—budget CI time accordingly.

Takeaways

A practical split for interactive science demos—especially 3D biology explorers—is to let a GPT Image-class model shape the interface while a strong coding model such as Gemini 3.1 Pro owns orchestration, scene logic, and tool calls.

%%{init: {"theme": "base", "themeVariables": {"background": "transparent", "lineColor": "#000000"}}}%%
flowchart LR
  B[Science brief] --> U["GPT Image style pass"]
  U --> M[UI frames and assets]
  B --> C["Gemini 3.1 Pro coding"]
  C --> A[App logic and 3D wiring]
  M --> P[Interactive web app]
  A --> P

  classDef agent fill:#8B0000,color:#fff
  classDef hook fill:#189AB4,color:#fff
  class B,P agent
  class U,M,C,A hook

Why split UI and code models

Implementation notes

• Keep scientific claims reviewable: wire 3D assets to curated datasets instead of letting the model invent structures.
• Stream partial UI updates from the image side while the coding model focuses on deterministic simulation parameters.
• Reuse the same session transcripts as regression prompts so UI refreshes do not break physics constraints.

At a glance

Remotion ships an official Agent Skills bundle so coding agents such as Claude Code learn Remotion timing, media, and composition rules—installable with one CLI line alongside new `bun create video` scaffolding.

%%{init: {"theme": "base", "themeVariables": {"background": "transparent", "lineColor": "#000000"}}}%%
flowchart LR
  P[Natural language brief] --> I["npx skills add remotion-dev/skills"]
  I --> A[Agent with Remotion skills]
  A --> R[React Remotion project]
  R --> V[Programmatic video output]

  classDef agent fill:#8B0000,color:#fff
  classDef hook fill:#189AB4,color:#fff
  class P,V agent
  class I,A,R hook

Install path

What the skills are for

They encode best practices for Remotion’s React-based, programmatic video model so agents avoid common timing, asset, and composition mistakes. Remotion links the catalogue to the open Agent Skills standard and hosts the source beside the main repository.

At a glance

Gemini 3.1 Flash-Lite is now positioned as Google’s fastest, lowest-cost Gemini 3 tier for high-volume work—agentic pipelines, translation, and light extraction—while keeping a stable API model id for production calls.

%%{init: {"theme": "base", "themeVariables": {"background": "transparent", "lineColor": "#000000"}}}%%
flowchart LR
  V[High volume traffic] --> M["Gemini 3.1 Flash-Lite"]
  M --> T[Text and structured outputs]
  M --> A[Tool calling and routing]

  classDef agent fill:#8B0000,color:#fff
  classDef hook fill:#189AB4,color:#fff
  class V agent
  class M,T,A hook

What shipped

Why teams pick Flash-Lite

API snapshot

At a glance

Cursor announced a /orchestrate skill aimed at the Cursor SDK that recursively spawns agents for large jobs—backed with internal figures on token savings and cold-start latency.

%%{init: {"theme": "base", "themeVariables": {"background": "transparent", "lineColor": "#000000"}}}%%
flowchart TD
  U[User goal] --> O["/orchestrate skill"]
  O --> A1[Sub-agent A]
  O --> A2[Sub-agent B]
  O --> A3[Sub-agent N]
  A1 --> M[Merge results]
  A2 --> M
  A3 --> M
  M --> R[SDK outcome]

  classDef agent fill:#8B0000,color:#fff
  classDef hook fill:#189AB4,color:#fff
  class U,R agent
  class O,A1,A2,A3,M hook

What Cursor said

How it fits the wider Cursor 3.3 drop

The same-day Cursor 3.3 notes emphasise parallel work with async subagents—Build in Parallel on plans, editor /multitask, and pinned skills as quick actions—so teams can mix UI-driven parallelism with skill-driven orchestration patterns.

Skills mechanics (context)

Cursor loads project and user skill folders (for example .cursor/skills/), exposes them to Agent, and supports explicit /skill-name invocation; skills can bundle scripts and references for progressive loading—useful when an orchestration skill coordinates long-running SDK sessions.

At a glance

OpenAI shipped GPT-Realtime-2 in the Realtime API so voice agents can keep a live conversation moving while applying GPT-5-class reasoning, parallel tools, and larger session context—alongside new streaming translation and transcription models.

%%{init: {"theme": "base", "themeVariables": {"background": "transparent", "lineColor": "#000000"}}}%%
flowchart LR
  U[User audio] --> R[GPT-Realtime-2 session]
  R --> T[Tools and retrieval]
  T --> R
  R --> O[Spoken reply]

  classDef agent fill:#8B0000,color:#fff
  classDef hook fill:#189AB4,color:#fff
  class U,O agent
  class R,T hook

Three models in one drop

What GPT-Realtime-2 adds for builders

Pricing snapshot (API)

All three run through the Realtime API; the announcement pairs them with customer examples (travel, telecom, property search) where low-latency speech, translation, or live captions must stay aligned with changing user intent.

At a glance

Anthropic’s SpaceX compute agreement adds Colossus 1 capacity so Claude Code and Claude API limits can move up the same day—here is what changed for subscribers and API callers.

%%{init: {"theme": "base", "themeVariables": {"background": "transparent", "lineColor": "#000000"}}}%%
flowchart LR
  S[SpaceX Colossus 1 capacity] --> A[Anthropic training and inference]
  A --> C[Claude Code higher ceilings]
  A --> P[Claude API Opus limits]

  classDef agent fill:#8B0000,color:#fff
  classDef hook fill:#189AB4,color:#fff
  class S,A agent
  class C,P hook

What went live on 6 May 2026

What the SpaceX slice adds

The same announcement frames wider infrastructure work (other hyperscaler and infrastructure partners) and notes interest in future orbital compute at gigawatt scale; day-one user impact above is the limit moves for Claude Code and Opus API traffic.

At a glance

Claude Managed Agents now pair always-on work sessions with an explicit success rubric, asynchronous memory curation, and HTTPS webhooks so long-running agent jobs can finish, self-correct, and notify your stack without constant polling.

%%{init: {"theme": "base", "themeVariables": {"background": "transparent", "lineColor": "#000000"}}}%%
graph TD
  A[Managed agent session] --> B[Outcome rubric and iterations]
  B --> C[Separate grader context]
  C -->|needs revision| B
  C -->|satisfied| D[Idle with deliverables]
  A --> E[(Memory store)]
  E --> F[Dream job reviews transcripts]
  F --> E
  A --> G[HTTPS webhooks on milestones]

  classDef agent fill:#8B0000,color:#fff
  classDef hook fill:#189AB4,color:#fff
  classDef decision fill:#444,color:#fff

  class A agent
  class E,F,G hook
  class C decision

What changed on 6 May 2026

The live Code with Claude stream introduced dreaming as a research preview inside Managed Agents, while outcomes, multi-agent orchestration, and webhooks moved to public beta alongside the existing memory features.

Dreaming: curate memory without mutating the source store

Dreaming runs as an asynchronous job that reads a memory store and up to one hundred session transcripts, then writes a brand-new store containing merged facts, removed contradictions, and freshly surfaced patterns. The input store stays read-only until you adopt the output, which keeps experiments reversible.

curl -s https://api.anthropic.com/v1/dreams \
  -H "x-api-key: $ANTHROPIC_API_KEY" \
  -H "anthropic-version: 2023-06-01" \
  -H "anthropic-beta: managed-agents-2026-04-01,dreaming-2026-04-21" \
  -H "content-type: application/json" \
  -d '{"inputs":[{"type":"memory_store","memory_store_id":"'"$STORE"'"},
              {"type":"sessions","session_ids":["'"$SESSION_A"'","'"$SESSION_B"'"]}],
           "model":"claude-opus-4-7",
           "instructions":"Focus on durable coding preferences; drop one-off debugging notes."}'

Outcomes: rubrics, graders, and iteration budgets

When you emit user.define_outcome, the harness spins up a grader that scores each criterion independently in its own context window, then returns either a pass or a precise gap list that feeds the next agent revision. You can supply the rubric inline or via the Files API, cap the loop with max_iterations (default three, hard maximum twenty), and subscribe to session.outcome_evaluation_ended webhooks when grading rounds finish.

Multi-agent orchestration: coordinators, threads, and limits

Coordinators declare a roster of delegate agents (maximum twenty unique IDs, single hop only) and spawn isolated session threads that keep their own transcripts while sharing the container filesystem. Up to twenty-five threads may run concurrently; the primary session stream stays a condensed feed while per-thread streams expose full tool traces when you need forensic detail.

Webhooks: verify signatures, then hydrate objects yourself

Endpoints must be public HTTPS on port 443. Each delivery includes a signing secret (whsec_) and an X-Webhook-Signature header; Anthropic’s SDK unwrap() helper validates the payload and rejects anything older than five minutes, which also gives you a safe retry discriminator because duplicate retries reuse the same event.id.

Clip from the live announcement

Where to go next

Read the Managed Agents product announcement for customer vignettes and headline benchmark figures.

Deep-dive the platform guides for dreams, outcomes, multi-agent sessions, and webhooks.

Request dreaming access via the Managed Agents intake form if you want the research preview enabled for your organisation.

At a glance