Browser SDK - Talqing

@talqing/sdk/browser is the React layer that joins a voice or video call from a web page. It wraps LiveKit so you never type livekit-client yourself, and it carries the pieces a Talqing call needs on top of a room: the token mint, userdata over RPC, images, screen share, the avatar track and frontend actions. This page is the reference. For the end-to-end integration — minting the token on your server, the first working component — read web calls and web call features.

Install

npm install @talqing/sdk

The browser layer is a second entrypoint of the same package, not a second install:

import { TalqingSessionProvider, useTalqingSession } from "@talqing/sdk/browser";

livekit-client and @livekit/components-react come with the package. react and react-dom are peer dependencies — ^18.3.1 or ^19 — and are used by this entrypoint only; the API client at @talqing/sdk renders nothing.

The three rules

One session is one call. useTalqingSession builds one Room and one token source. Start it when the component mounts and end it on the way out. To place a second call, mount a second session — render the call component while a call is up and drop it to hang up — and give it a key that changes per call. A finished session cannot be restarted. Its token names the room that call used, and the API refuses a second call rejoining it. start() on a spent session does not give you a new call. Screen share must start from a click. getDisplayMedia requires a transient user activation, so start() has to run inside a real user gesture. An agent can ask for a screen; it can never take one.

Joining a call

`useTalqingSession`

function useTalqingSession(options: {
  client: TalqingClient;
  agentId: string | null;
  roomOptions?: RoomOptions;
  callOptions?: TalqingCallOptions;
  onTokenDetails?: (details: CallTokenResponse) => void;
}): TalqingSession;

Builds the Room and the token source, and returns LiveKit’s session object — { room, isConnected, connectionState, start, end, waitUntilConnected, … }. callOptions is everything POST /v1/calls/token accepts except agent_id: contact_key, userdata, vars, agent_version, agent_override, agent_team. onTokenDetails receives the whole token response — including conversation_id, session_id, the minted contact_key and any warnings — the moment it is minted.

function CallSession({ agentId }: { agentId: string }) {
  const session = useTalqingSession({
    client: talqing,
    agentId,
    callOptions: { contact_key: "user-42" },
  });

  useEffect(() => {
    void session.start();
    return () => void session.end();
  }, []);

  return (
    <TalqingSessionProvider session={session}>
      <Call />
    </TalqingSessionProvider>
  );
}

Pass contact_key — your own id for the person — if the agent should recognise a returning caller. Each call mints its own key when you leave it out, so two calls from the same person arrive as two strangers and an agent set to carry past conversations has nothing to carry. roomOptions is LiveKit’s own RoomOptions, spread over the audio capture defaults below. Override audioCaptureDefaults there if you must.

`TalqingSessionProvider`

function TalqingSessionProvider(props: {
  session: TalqingSession;
  children: React.ReactNode;
  audio?: boolean; // default true
}): JSX.Element;

Puts the session in context — which is what every hook below except useTalqingImages reads — and mounts LiveKit’s audio renderer so the agent is audible. Pass audio={false} only if you are rendering the audio element yourself.

`useTalqingConnection`

function useTalqingConnection(): {
  isConnected: boolean;
  connectionState: ConnectionState;
  room: Room;
  start: (options?: SessionConnectOptions) => Promise<void>;
  end: () => Promise<void>;
};

The call itself, read from context. room is what useTalqingImages takes; end() hangs up.

The room’s state is not the agent’s state. isConnected goes true partway through the join, while the agent is still starting up. A button keyed off isConnected alone tells the caller to speak to somebody who is not listening yet — wait for useTalqingAgent().state === "listening".

`createTalqingTokenSource`

function createTalqingTokenSource(
  client: TalqingClient,
  agentId: string | null,
  options?: TalqingCallOptions,
  onTokenDetails?: (details: CallTokenResponse) => void,
): TokenSource;

What useTalqingSession builds internally. Reach for it only if you are managing your own Room. It mints once per source and caches the in-flight promise, which is load-bearing: POST /v1/calls/token files the session row the call is later read back from, and LiveKit asks its token source three times per call — to warm the connection, to connect, and once more as the call ends. Without the cache, the third mint filed a session nobody ever joined, which sat in the workspace’s call list forever. A failed mint is kept out of the cache so the surface can offer the call again. Freshness comes from a new source per call, never from re-minting inside one.

`TALQING_AUDIO_CAPTURE_DEFAULTS`

const TALQING_AUDIO_CAPTURE_DEFAULTS = {
  echoCancellation: true,
  noiseSuppression: true,
  autoGainControl: true,
};

Applied by useTalqingSession unless your roomOptions overrides audioCaptureDefaults. Echo cancellation is what stops the agent hearing its own speech through the caller’s microphone and interrupting itself — a call without it is broken in a way that sounds like a bad model, which is why it is a default rather than something each app has to remember.

Media

`useTalqingAgent`

function useTalqingAgent(): { state: AgentState; isFinished: boolean };

state is disconnected, connecting, pre-connect-buffering, initializing, idle, listening, thinking, speaking or failed. isFinished is true once the client has disconnected from the agent, expectedly or not.

`useTalqingMessages`

function useTalqingMessages(): {
  messages: ReceivedMessage[];
  send: (message: string, options?: SendTextOptions) => Promise<ReceivedChatMessage>;
  isSending: boolean;
};

The live transcript, and a way to type into a voice call — useful for a spelling, an order number, or an accessible path into an audio agent. messages belongs to the live session only; the stored transcript, with tool calls and cost, is on GET /v1/calls/{session_id}.

`setTalqingBrowserLogLevel`

function setTalqingBrowserLogLevel(
  level: "debug" | "info" | "warn" | "error" | "silent",
): void;

Sets the underlying LiveKit client’s log level. "debug" is the first move on a connection problem you cannot see.

Images

A caller attaches a photo mid-call and the agent sees it. The bytes go over a LiveKit byte stream addressed to the agent; the agent acknowledges over RPC, and the id send returns is the id that acknowledgement carries, so there is nothing to correlate by hand.

The agent deliberately does not speak when a photo lands. The caller is mid-conversation and about to say what it is for, so the image goes into the model’s view for their next turn.

`useTalqingImages`

function useTalqingImages(room: Room | null | undefined): {
  send: (file: File) => Promise<TalqingSentImage>;
  sent: TalqingSentImage[];
  clear: () => void;
};

Takes the Room rather than reading context, because a byte stream and an RPC handler are room-level things — pass useTalqingConnection().room, or the Room you built yourself. send validates, downscales, sends, and resolves once the bytes are on their way. The outcome arrives on sent, which is what a thumbnail should render. Each entry is a TalqingSentImage:

Field	Type	Meaning
`streamId`	`string`	The stream id, which the acknowledgement comes back under.
`name`	`string`	The original file name.
`previewUrl`	`string`	An object URL for the downscaled image. Revoked by `clear()`.
`status`	`"sending" \| "sent" \| "failed" \| "unknown"`
`error`	`string`	A sentence to show, set with `failed` or `unknown`.

unknown is the state people forget: the agent going away mid-upload, where no acknowledgement is ever coming. It is set after 30 seconds of silence, with “we did not hear back — the agent may not have received it”.

const { room } = useTalqingConnection();
const images = useTalqingImages(room);

<input
  type="file"
  accept="image/jpeg,image/png,image/webp"
  onChange={(e) => e.target.files?.[0] && void images.send(e.target.files[0])}
/>;

`talqingImageRejection`

function talqingImageRejection(file: File): string | null;

The sentence to show, or null when the file is fine. Refuses anything that is not JPEG, PNG or WebP (“that file is not a JPEG, PNG or WebP”) and anything over 10 MB (“images are 10 MB or smaller”). The server re-validates regardless — this exists so the common mistakes cost nothing to catch.

`talqingDownscaleImage`

function talqingDownscaleImage(file: File): Promise<Blob>;

Downscales to 1568 px on the longest edge and re-encodes — PNG stays PNG to keep transparency, everything else becomes JPEG at quality 0.85. It is also what normalizes whatever the OS handed the file input, so a Safari HEIC becomes a JPEG here. Throws if the image cannot be decoded. useTalqingImages.send calls it for you.

`talqingImageDataUrl`

function talqingImageDataUrl(file: File): Promise<string>;

Validate and downscale into the data: URL a text message carries — POST /v1/conversations/messages takes images: [{ data_url, filename }], and this produces the data_url. It doubles as a thumbnail src, so there is no object URL to create and revoke. Throws the sentence to show if the file is refused.

Image constants

Export	Value
`TALQING_IMAGE_TOPIC`	`"talqing.images"`
`TALQING_IMAGE_MAX_BYTES`	`10485760` (10 MB)
`TALQING_IMAGE_MAX_EDGE_PX`	`1568`

`useTalqingScreenShare`

function useTalqingScreenShare(): {
  enabled: boolean;
  pending: boolean;
  start: () => Promise<unknown>;
  stop: () => Promise<unknown>;
};

enabled is whether a screen is being shared right now; pending is true while the browser picker is open or the track is publishing. stop() ends it, and the agent is told it can no longer see on its next turn.

const screen = useTalqingScreenShare();

<button onClick={() => void screen.start()} disabled={screen.pending}>
  {screen.enabled ? "Stop sharing" : "Share my screen"}
</button>;

start() must be called from a user gesture. A browser will not open the share picker for a page that asks on its own.

The capture and publish options are fixed, and they are the reason a shared screen stays readable:

Export	Setting	Why
`TALQING_SCREEN_SHARE_CAPTURE`	`contentHint: "text"`	Tells the encoder to preserve spatial detail. No downstream quality setting recovers it once lost.
	`resolution: 1920×1080 at 1 fps`	Set explicitly rather than inherited, where the default is a hint and not a guarantee.
`TALQING_SCREEN_SHARE_PUBLISH`	`videoCodec: "h264"`	What keeps that hint. `livekit-client` forces `contentHint: "motion"` on a screen share published as vp9 or av1 — publishing on an SVC codec costs you legibility, and it is the first thing to check if the agent starts misreading identifiers.
	`screenShareEncoding: { maxFramerate: 1, maxBitrate: 1000000 }`	The frame-rate cap that actually binds. Every delivered frame is copied into the agent’s process before it can be dropped, so this is the biggest single lever on what a call costs. One frame a second is ample: the agent is handed the newest frame at the end of each turn.

Userdata

Read and write the live session’s userdata from the page, over RPC to the agent. See userdata for what it is.

`useTalqingUserdata`

function useTalqingUserdata(): {
  get: (keys?: string[], responseTimeout?: number) => Promise<JsonObject>;
  set: (patch: JsonObject, responseTimeout?: number) => Promise<{ ok: boolean; error?: string }>;
};

Bound to the connected room. Both throw Error("room is not connected") when there is no room.

const userdata = useTalqingUserdata();
await userdata.set({ cart_total: 4200, tier: "pro" });
const { tier } = await userdata.get(["tier"]);

`setTalqingUserdata` and `getTalqingUserdata`

function setTalqingUserdata(
  room: Room,
  patch: JsonObject,
  responseTimeout?: number, // default 3 seconds
): Promise<{ ok: boolean; error?: string }>;

function getTalqingUserdata(
  room: Room,
  keys?: string[],
  responseTimeout?: number, // default 3 seconds
): Promise<JsonObject>;

The same calls outside React, against a Room you hold. They find the agent participant and call talqing.client.userdata_set / talqing.client.userdata_get on it. With no agent connected, set resolves { ok: false, error: "agent participant is not connected" } while get throws — a write that did not land is a state you can render, a read that returned nothing is not. Omit keys on get to fetch the whole bag.

Frontend RPC

A tool’s frontend_rpc operation calls back into your page mid-call — to highlight a product, open a form, or move the user along. See frontend RPC for the tool side.

`useTalqingRpcHandlers`

function useTalqingRpcHandlers(
  handlers: Record<string, (data: TalqingRpcInvocation) => string | Promise<string>>,
  enabled?: boolean, // default true
): void;

Registers the talqing.frontend_rpc method on the room and dispatches by the envelope’s method. A "*" key catches everything unmatched — and when the wildcard handles a call, data.payload is the whole envelope (method and payload) rather than just the payload, so the handler can tell what it caught. An unknown method with no wildcard throws TalqingRpcError code 1404 back to the agent; anything else your handler throws goes back as code 1500 with its message. Each handler receives a TalqingRpcInvocation:

Field	Type
`payload`	`string` — JSON, which you parse
`responseTimeout`	`number \| undefined`
`callerIdentity`	`string \| undefined`

and must return a string (or a promise of one), which is what the agent receives.

useTalqingRpcHandlers({
  show_product: async ({ payload }) => {
    const { sku } = JSON.parse(payload) as { sku: string };
    setHighlighted(sku);
    return JSON.stringify({ shown: true });
  },
});

Registration is skipped until the room is connected, and unregistered on unmount. Pass enabled: false to hold off.

`useTalqingFrontendRpcs`

function useTalqingFrontendRpcs(enabled?: boolean): {
  actions: Record<string, { payload?: JsonValue; at: number }>;
  clear: () => void;
};

The declarative alternative: it registers a wildcard handler for you and keeps the latest payload per method name, with at as the timestamp it arrived. Always answers the agent "ok". Use it to render whatever the agent asked for without writing a handler per method.

`TalqingRpcError`

class TalqingRpcError extends Error {
  readonly code: number;
  readonly data?: string;
  constructor(code: number, message: string, data?: string);
}

Throw it from a handler to send a specific code back to the agent instead of the generic 1500.

Video

`useTalqingAvatarTrack`

function useTalqingAvatarTrack(): TrackReference | undefined;

Finds the video track to render for a video agent. It prefers the avatar worker’s camera track — the worker publishes on the agent’s behalf, which is how it is identified — and falls back through the worker’s screen share and then the agent’s own tracks. undefined until one is published.

`TalqingVideoTrack`

function TalqingVideoTrack(props: {
  track: TrackReference;
  className?: string;
}): JSX.Element;

Renders it at the publication’s own dimensions.

const track = useTalqingAvatarTrack();
return track ? <TalqingVideoTrack track={track} className="rounded-xl" /> : null;

Exported types

Type	What it is
`TalqingCallOptions`	Everything `POST /v1/calls/token` takes except `agent_id`.
`TalqingBrowserSessionOptions`	The argument to `useTalqingSession`.
`TalqingSession`	LiveKit’s session return type, as `useTalqingSession` gives it back.
`TalqingMessage`	One entry in `useTalqingMessages().messages`.
`TalqingVideoTrackRef`	The track reference `useTalqingAvatarTrack` returns and `TalqingVideoTrack` takes.
`TalqingSentImage`, `TalqingImageStatus`	An image in flight, and its four states.
`TalqingRpcHandler`, `TalqingRpcHandlers`, `TalqingRpcInvocation`	The frontend-RPC handler shapes.
`TalqingFrontendRpcEnvelope`	What the agent sends: `method` and an optional `payload`.

Web calls

The token, the join, and a complete integration.

Web call features

Images, screen share, userdata and frontend actions in context.

​Install

​The three rules

​Joining a call

​useTalqingSession

​TalqingSessionProvider

​useTalqingConnection

​createTalqingTokenSource

​TALQING_AUDIO_CAPTURE_DEFAULTS

​Media

​useTalqingAgent

​useTalqingMessages

​setTalqingBrowserLogLevel

​Images

​useTalqingImages

​talqingImageRejection

​talqingDownscaleImage

​talqingImageDataUrl

​Image constants

​Screen share

​useTalqingScreenShare

​Userdata

​useTalqingUserdata

​setTalqingUserdata and getTalqingUserdata

​Frontend RPC

​useTalqingRpcHandlers

​useTalqingFrontendRpcs

​TalqingRpcError

​Video

​useTalqingAvatarTrack

​TalqingVideoTrack

​Exported types

Web calls

Web call features

Install

The three rules

Joining a call

`useTalqingSession`

`TalqingSessionProvider`

`useTalqingConnection`

`createTalqingTokenSource`

`TALQING_AUDIO_CAPTURE_DEFAULTS`

Media

`useTalqingAgent`

`useTalqingMessages`

`setTalqingBrowserLogLevel`

Images

`useTalqingImages`

`talqingImageRejection`

`talqingDownscaleImage`

`talqingImageDataUrl`

Image constants

Screen share

`useTalqingScreenShare`

Userdata

`useTalqingUserdata`

`setTalqingUserdata` and `getTalqingUserdata`

Frontend RPC

`useTalqingRpcHandlers`

`useTalqingFrontendRpcs`

`TalqingRpcError`

Video

`useTalqingAvatarTrack`

`TalqingVideoTrack`

Exported types