A phone call is audio and nothing else. A web call has a page around it, and that page can hand the agent things a phone never could — a photo, a screen, the state of the form the caller is filling in — and take instructions back from it. Everything here assumes the setup on web calls: a session started with useTalqingSession and wrapped in TalqingSessionProvider. The dashboard’s own test call is built out of exactly these pieces, so what you see there is what these hooks give you.

Images

The caller attaches a photo mid-call and the agent looks at it. The bytes go straight from the file input to the agent over the call’s own data channel — there is no upload endpoint and nothing to correlate by hand.
The agent deliberately does not speak when a photo lands. The caller is mid-conversation and about to say what the photo is for; answering unprompted would talk over them. The image goes into the model’s view and the agent replies on the caller’s next turn. That means silence is the expected response, so it cannot double as “it arrived” — render the per-image status, or the caller has no way to tell a delivered photo from a lost one. send(file) resolves once the bytes are on their way and pushes an entry onto sent; the outcome lands on that entry afterwards. unknown is the state people forget: the agent can go away mid-upload, and then no answer is ever coming. Four states, not two.

Limits

The SDK exports the three steps separately if you want to run them yourself: talqingImageRejection(file) returns the sentence to show or null, talqingDownscaleImage(file) returns the resized Blob, and talqingImageDataUrl(file) does both and returns the data: URL a text message carries.
Whether an agent can read an image at all is decided by its model, not by a setting — the catalog’s vision flag. Send one to an agent whose model cannot see and the image comes back failed with this agent's model cannot read images. See vision and images.

Screen share

The agent watches the caller’s screen while they share it, and is handed the newest frame at the end of each of their turns.
start() must be called from a user click. The browser’s picker requires a transient user activation, so nothing on our side can open it for the person. An agent can ask for a screen; it can never take one. Write the prompt so it asks, and keep the button visible for the whole call rather than revealing it mid-sentence.
The capture and publish options are fixed — TALQING_SCREEN_SHARE_CAPTURE and TALQING_SCREEN_SHARE_PUBLISH, both exported so you can read them. One frame a second, h264, 1920×1080, contentHint: "text". All four are deliberate:
  • contentHint: "text" tells the encoder to keep spatial detail. Lose it and the agent starts misreading identifiers on screen.
  • videoCodec: "h264" is what keeps that hint. livekit-client forces contentHint = "motion" on a screen share published as vp9 or av1, together with scalabilityMode: "L1T3" — so publishing on an SVC codec costs you legibility, and it is the first thing to check if the agent starts getting characters wrong.
  • screenShareEncoding.maxFramerate: 1 is the cap that actually binds. resolution.frameRate reaches getDisplayMedia as a bare number, which WebRTC reads as ideal — a wish, not a limit. The default publish ceiling is 15 fps, and every delivered frame is copied into the agent’s process before it can be dropped. One frame a second is ample: the agent is handed the newest frame at the end of each turn, and turns are seconds apart.
  • resolution is set explicitly rather than inherited, because the default 1080p is also only an ideal.
The agent side is vision_input.screenshare on the config, and it has its own rules — cascade pipeline only, a model that reads images, and a prompt paragraph that is added for you. See vision and images.

Userdata from the page

userdata is the session’s own state: what the agent knows about the person, readable as {{userdata.field}} from its prompt and its tools. The page can read and write it while the call is running.
set(patch, responseTimeout?) resolves to { ok, error? }ok: false with agent participant is not connected when the agent has not joined yet. get(keys?, responseTimeout?) returns the session’s userdata as an object, or only the keys you name, and throws when the agent is not connected. Both are RPCs to the worker running the call, with a 3-second response timeout by default. That is a real round trip over the call’s data channel while the agent is talking, so write to it when something changes rather than on every keystroke, and never block a render on get. Two constraints come from the agent side: keys beginning with _talqing are reserved — a set containing one is refused and a get never returns one — and one RPC payload is capped at 16 KB. Outside a provider, the same two calls take the room directly: setTalqingUserdata(room, patch, responseTimeout = 3) and getTalqingUserdata(room, keys?, responseTimeout = 3). Whatever the call ends with is merged onto the contact’s record, so it is there again on their next call. See userdata.

Frontend actions

A tool’s frontend_rpc operation calls into your page — to open a checkout, highlight a row, or read something only the page knows. The agent sends one envelope method, talqing.frontend_rpc, and the SDK dispatches on envelope.method.
The envelope on the wire is { "method": "open_page", "payload": { … } }. Your handler receives { payload, responseTimeout?, callerIdentity? } where payload is the JSON string of envelope.payload"{}" when the operation sent none — and returns a string. That string is what the tool gets back: valid JSON is parsed, and anything else is wrapped as { "result": "<your string>" }.
  • Register a "*" handler to catch every method. It receives the whole envelope as its payload string, method included.
  • An unrecognised method with no "*" handler throws back as RPC error code 1404, No frontend RPC handler registered for <method>. Any other throw becomes code 1500 with the error’s message; throw a TalqingRpcError to choose the code yourself.
  • The agent waits 5 seconds for your answer unless the operation sets its own timeout. A handler that awaits a person is a handler that times out.
  • Pass false as the second argument to unregister — useTalqingRpcHandlers(handlers, isReady).
If all you want is to render what the agent asked for, useTalqingFrontendRpcs() registers the wildcard for you and keeps the latest payload per method:
The operation side — declaring the method, its payload and what the model does with the answer — is on frontend RPC.

The avatar’s video

A video agent joins wearing an Anam avatar, published by a second participant on the agent’s behalf. useTalqingAvatarTrack finds it without you touching participants, falling back to the agent’s own camera or screen-share track if there is no avatar worker.
TalqingVideoTrack reads the publication’s own dimensions, so the element sizes itself; style it with className. Talqing does not pin the avatar’s frame size — each avatar model publishes at its own — so size the container by aspect ratio and crop rather than assuming a resolution. The avatar’s voice is the room’s audio like any other, already played by TalqingSessionProvider. Adding an audio element for the avatar plays the call twice.

The agent’s state and the transcript

{ isConnected, connectionState, room, start, end }
The call itself. room is what useTalqingImages takes; end() hangs up.
{ state, isFinished }
Where the agent is. state is one of disconnected, connecting, pre-connect-buffering, initializing, idle, listening, thinking, speaking or failed. isFinished is true once the client has disconnected from the agent, expectedly or not.
{ messages, send, isSending }
The live transcript, and a way to type into the call. send(text) puts a text message on a voice call — useful for a spelling, an order number, or an accessible path into an audio agent.
Each message carries id, timestamp, message and, where there is one, from. from?.isLocal distinguishes the caller from the agent, and type says which stream it came from — userTranscript, agentTranscript or chatMessage. The room’s state is not the agent’s state. isConnected goes true partway through the join, while the agent is still connecting, pre-connect-buffering or initializing — a UI keyed off isConnected alone tells the caller to speak to somebody who is not listening yet. Wait for listening. messages belongs to the live session. Once the call has ended, the stored transcript — with every tool call, its output, the cost and the per-stage latency — is on GET /v1/calls/{session_id}. See calls. For a noisy problem, setTalqingBrowserLogLevel("debug") turns up the underlying LiveKit client’s logging.

Next

Browser SDK reference

Every export with its full signature.

Frontend RPC

Building the tool operation that calls these handlers.

Vision and images

Which models can see, and the vision_input.screenshare rules.