useTalqingSession and wrapped in TalqingSessionProvider. The
dashboard’s own test call is built out of exactly these pieces, so what you see
there is what these hooks give you.
Images
The caller attaches a photo mid-call and the agent looks at it. The bytes go straight from the file input to the agent over the call’s own data channel — there is no upload endpoint and nothing to correlate by hand.send(file) resolves once the bytes are on their way and pushes an entry onto
sent; the outcome lands on that entry afterwards.
unknown is the state people forget: the agent can go away mid-upload, and then
no answer is ever coming. Four states, not two.
Limits
The SDK exports the three steps separately if you want to run them yourself:
talqingImageRejection(file) returns the sentence to show or null,
talqingDownscaleImage(file) returns the resized Blob, and
talqingImageDataUrl(file) does both and returns the data: URL a text message
carries.
Whether an agent can read an image at all is decided by its model, not by a
setting — the catalog’s
vision flag. Send one to an agent whose model cannot
see and the image comes back failed with this agent's model cannot read images. See vision and images.Screen share
The agent watches the caller’s screen while they share it, and is handed the newest frame at the end of each of their turns.TALQING_SCREEN_SHARE_CAPTURE and
TALQING_SCREEN_SHARE_PUBLISH, both exported so you can read them. One frame a
second, h264, 1920×1080, contentHint: "text". All four are deliberate:
contentHint: "text"tells the encoder to keep spatial detail. Lose it and the agent starts misreading identifiers on screen.videoCodec: "h264"is what keeps that hint.livekit-clientforcescontentHint = "motion"on a screen share published as vp9 or av1, together withscalabilityMode: "L1T3"— so publishing on an SVC codec costs you legibility, and it is the first thing to check if the agent starts getting characters wrong.screenShareEncoding.maxFramerate: 1is the cap that actually binds.resolution.frameRatereachesgetDisplayMediaas a bare number, which WebRTC reads asideal— a wish, not a limit. The default publish ceiling is 15 fps, and every delivered frame is copied into the agent’s process before it can be dropped. One frame a second is ample: the agent is handed the newest frame at the end of each turn, and turns are seconds apart.resolutionis set explicitly rather than inherited, because the default 1080p is also only anideal.
vision_input.screenshare on the config, and it has its own
rules — cascade pipeline only, a model that reads images, and a prompt paragraph
that is added for you. See vision and images.
Userdata from the page
userdata is the session’s own state: what the agent knows about the person,
readable as {{userdata.field}} from its prompt and its tools. The page can read
and write it while the call is running.
set(patch, responseTimeout?) resolves to { ok, error? } — ok: false with
agent participant is not connected when the agent has not joined yet.
get(keys?, responseTimeout?) returns the session’s userdata as an object, or
only the keys you name, and throws when the agent is not connected.
Both are RPCs to the worker running the call, with a 3-second response timeout by
default. That is a real round trip over the call’s data channel while the agent
is talking, so write to it when something changes rather than on every keystroke,
and never block a render on get.
Two constraints come from the agent side: keys beginning with _talqing are
reserved — a set containing one is refused and a get never returns one — and
one RPC payload is capped at 16 KB.
Outside a provider, the same two calls take the room directly:
setTalqingUserdata(room, patch, responseTimeout = 3) and
getTalqingUserdata(room, keys?, responseTimeout = 3).
Whatever the call ends with is merged onto the contact’s record, so it is there
again on their next call. See userdata.
Frontend actions
A tool’sfrontend_rpc operation calls into your page — to open a checkout,
highlight a row, or read something only the page knows. The agent sends one
envelope method, talqing.frontend_rpc, and the SDK dispatches on
envelope.method.
{ "method": "open_page", "payload": { … } }. Your
handler receives { payload, responseTimeout?, callerIdentity? } where payload
is the JSON string of envelope.payload — "{}" when the operation sent none
— and returns a string. That string is what the tool gets back: valid JSON is
parsed, and anything else is wrapped as { "result": "<your string>" }.
- Register a
"*"handler to catch every method. It receives the whole envelope as its payload string,methodincluded. - An unrecognised method with no
"*"handler throws back as RPC error code 1404,No frontend RPC handler registered for <method>. Any other throw becomes code 1500 with the error’s message; throw aTalqingRpcErrorto choose the code yourself. - The agent waits 5 seconds for your answer unless the operation sets its own timeout. A handler that awaits a person is a handler that times out.
- Pass
falseas the second argument to unregister —useTalqingRpcHandlers(handlers, isReady).
useTalqingFrontendRpcs()
registers the wildcard for you and keeps the latest payload per method:
The avatar’s video
Avideo agent joins wearing an Anam avatar, published by
a second participant on the agent’s behalf. useTalqingAvatarTrack finds it
without you touching participants, falling back to the agent’s own camera or
screen-share track if there is no avatar worker.
TalqingVideoTrack reads the publication’s own dimensions, so the element sizes
itself; style it with className. Talqing does not pin the avatar’s frame size —
each avatar model publishes at its own — so size the container by aspect ratio and
crop rather than assuming a resolution.
The avatar’s voice is the room’s audio like any other, already played by
TalqingSessionProvider. Adding an audio element for the avatar plays the call
twice.
The agent’s state and the transcript
{ isConnected, connectionState, room, start, end }
The call itself.
room is what useTalqingImages takes; end() hangs up.{ state, isFinished }
Where the agent is.
state is one of disconnected, connecting,
pre-connect-buffering, initializing, idle, listening, thinking,
speaking or failed. isFinished is true once the client has disconnected
from the agent, expectedly or not.{ messages, send, isSending }
The live transcript, and a way to type into the call.
send(text) puts a text
message on a voice call — useful for a spelling, an order number, or an accessible
path into an audio agent.id, timestamp, message and, where there is one, from.
from?.isLocal distinguishes the caller from the agent, and type says which
stream it came from — userTranscript, agentTranscript or chatMessage.
The room’s state is not the agent’s state. isConnected goes true partway through
the join, while the agent is still connecting, pre-connect-buffering or
initializing — a UI keyed off isConnected alone tells the caller to speak to
somebody who is not listening yet. Wait for listening.
messages belongs to the live session. Once the call has ended, the stored
transcript — with every tool call, its output, the cost and the per-stage latency —
is on GET /v1/calls/{session_id}. See calls.
For a noisy problem, setTalqingBrowserLogLevel("debug") turns up the underlying
LiveKit client’s logging.
Next
Browser SDK reference
Every export with its full signature.
Frontend RPC
Building the tool operation that calls these handlers.
Vision and images
Which models can see, and the
vision_input.screenshare rules.