Connecting a voice agent to a reasoning agent with MCP

Picture this. You're in the car, driving to meet one of your most important prospects. Something is teasing you, at the back of your mind, but just out of reach. A commitment you made after the last meeting. A key insight from the discovery call. You can't dig through the material now. Your hands are on the wheel.

We solved this by combining two agents, one for talking and the other for reasoning.

What does it do?

You talk, it answers. Ask who you're meeting tomorrow and it tells you. Ask what you agreed with an account last month and it reads the meeting summary back to you. Ask in Spanish, it responds in Spanish. Switch language and it follows. When you want to go deeper, it offloads the question to Ask Vibe, our agent with custom tools. That gives it the capability to dig through transcripts, CRM records or account information for much more granular detail, or answers that aggregate across a variety of calls.

Why two agents?

A voice agent can answer fast, with low latency, support interruptions and keep the conversation fluid, multi-turn taking without long pauses. Offloading data intensive operations lets us remove disruptive pauses from the conversation. And by keeping the tool calling inside VibeSell, we control the quality of the response, trace executions and optimise queries.

How does it do it?

We use ElevenLabs for the conversational agent. We use Ask Vibe, our own agent, for tool calling, web search, citations and reasoning. The two are connected through MCP, where the voice agent delegates questions to Ask Vibe. Ask Vibe is the destination either way. Typing in the chat reaches it directly, speaking reaches it through the voice agent, and both land in the same conversation. The chat is the shared surface: what you say is transcribed into it, what Ask Vibe answers is written into it, and anything you type mid-call joins the spoken thread.

Both agents track memory of the conversation independently, and Ask Vibe sends tool calling updates back to the conversational agent in plain language, so it can give feedback on questions that take longer to process.

The ask-vibesell call is asynchronous. A request is fired with an immediate acknowledgement, leading to a natural response. While Ask Vibe is reasoning, the conversation can continue, and new questions that can be answered from carried context get answered immediately. Runs are concurrent: if the agent refines its own question mid-research, both keep going, and only a fresh question from you supersedes the one in flight, so a stale answer never talks over a new one.

Intermittent and final responses from ask-vibesell arrive over our own WebSocket to the browser, which injects them into the ElevenLabs session as a turn-taking trigger with new context. The injection is deliberately a turn rather than a passive context update: passive context lands silently and the agent never speaks it, which left users sitting in silence for the whole lookup.

Security

Voice runs on dedicated speech infrastructure, but your identity and your data never move there. The voice session holds an opaque id. Every request is re-authenticated at VibeSell's MCP server, resolved back to the signed-in user, and executed under row-level security with owner-scoped tools.

Four credentials gate that path, and each one does a different job:

1. Your login. The session request carries your normal signed-in credentials, so nothing starts without an authenticated user. This is the only place identity is captured.
2. A connection token. Our backend mints it against our provider API key and hands it to the browser. It is short-lived and it is what opens the voice call, nothing more.
3. The session id. A random value per session, bound server-side to your identity and the page you are on, expiring with the login it came from. It says which user, and it is the only one of the four the voice agent ever sees.
4. The bearer secret. Configured on the agent itself and checked by our MCP endpoint on every call. It never reaches the browser, so nobody inspecting the page can carry it away.

The session id is not an identity claim, it is a pointer only our servers can resolve. If it leaked, it would name nobody and unlock nothing outside a live session. From that point on, the request executes as you and only you. Row-level security applies at the database, and every tool the agent wields is owner-scoped: a meeting search, a contact lookup, or an account query can only return records the signed-in user owns.

So can a stranger talk to the voice agent and ask about your pipeline? They would need all four. Without the login there is no session, without the connection token the call never opens, without the bearer secret the MCP endpoint returns 401, and without an unexpired session id the request resolves to nobody. Miss one and there is nothing to answer with.

The speech layer is configured for zero retention: audio and transcripts are not kept after a session, none of it is used to train third-party models, and processing runs on EU-hosted infrastructure under GDPR. The voice agent gets the session id and nothing else: no name, no email, no access tokens. At no point does it learn who you are, and at no point does your data leave VibeSell's infrastructure.

Back to that car. "What did I commit to last time?" Ten seconds, then: "You said you would come five minutes earlier." You smash that gas.

Add VibeSell to Chrome