# React SDK

> Reference for @talqing/react and @talqing/client: useTalqingSession, images, screen share, frontend RPC and TalqingChat.

`@talqing/react` is the React layer for joining a voice or video call from a web page. `@talqing/client` is the same call protocol as plain functions over a `livekit-client` `Room`, plus `TalqingChat` for text chats; `@talqing/react` re-exports all of it. Neither renames LiveKit: anything LiveKit can do, you can still do (see [the escape hatch](#the-escape-hatch)).

This page is the reference for every export. For the end-to-end integration, see [web calls](https://docs.talqing.com/channels/web-calls.md) and [web call features](https://docs.talqing.com/channels/web-call-features.md).

> **Note:**
> **Breaking changes in `@talqing/client` 0.7.0 and `@talqing/react` 0.14.0.** `getTalqingUserdata`, `setTalqingUserdata` and `useTalqingUserdata` are removed: the browser can no longer read or write the live session's userdata. In the generated types, `RecordingSpec.consent` and `consent_notice` are replaced by `notice`, and the `consent_withdrawn` recording state is gone.

## Install

```bash
npm install @talqing/react
```

Its peer dependencies are `@talqing/client`, `@talqing/sdk`, `livekit-client`, `@livekit/components-react`, `react` and `react-dom` (React 18.3 or 19). npm 7+ and pnpm install peers automatically; on Yarn, list them:

```bash
yarn add @talqing/react @talqing/client @talqing/sdk livekit-client @livekit/components-react
```

Without React: `npm install @talqing/client livekit-client`.

## Rules

- **One session is one call.** Start it on mount and end it on unmount. For another call, mount a new session with a `key` that changes per call.
- **A finished session cannot be restarted**: its token names that call's room, and the API refuses rejoining it.
- **Screen share must start from a click.** Browsers only open the share picker inside a user gesture.

## Joining a call

### The three ways to authorize a call

Exactly one per session; passing two is a type error.

| Branch | Use when |
| --- | --- |
| `{ token }` | Your server called `POST /v1/calls/token` and forwarded `server_url` and `participant_token` |
| `{ fetchToken }` | The page asks your endpoint, which mints the token. Called once, when the call starts |
| `{ client, agentId, callOptions }` | **Trusted first-party apps only**: mints from the browser with a `TalqingClient` that already holds an EDITOR credential (as the Talqing dashboard does) |

> **Warning:**
> **Never ship a personal access token to a browser.** Use `token` or `fetchToken` and mint on your server. `callOptions` exists only on the `client` branch because `contact_key` decides whose history the call resumes and `userdata` is merged onto that person's record; on the other branches those stay on your server.

### `useTalqingSession`

```tsx
function useTalqingSession(options: TalqingSessionOptions): TalqingSession;

type TalqingSessionOptions = { roomOptions?: RoomOptions } & (
  | { token: TalqingCallCredentials }
  | { fetchToken: () => Promise<TalqingCallCredentials> }
  | { client: TalqingClient; agentId: string | null; callOptions?: TalqingCallOptions }
);
```

Builds the `Room` and token source and returns LiveKit's session object (`room`, `isConnected`, `connectionState`, `start`, `end`, `waitUntilConnected`, …) plus `warnings: string[]`.

```tsx
import { useEffect } from "react";
import { TalqingSessionProvider, useTalqingSession } from "@talqing/react";

function CallSession() {
  const session = useTalqingSession({
    fetchToken: async () => {
      const response = await fetch("/api/talqing-token", { method: "POST" });
      return await response.json(); // { server_url, participant_token }
    },
  });

  useEffect(() => {
    void session.start();
    return () => void session.end();
  }, []);

  return (
    <TalqingSessionProvider session={session}>
      <Call />
    </TalqingSessionProvider>
  );
}
```

| Option | Meaning |
| --- | --- |
| `callOptions` | Everything `POST /v1/calls/token` accepts except `agent_id`: `contact_key`, `userdata`, `vars`, `secrets`, `agent_version`, `agent_override`, `agent_team`, `region`, … **Memoize it**: a new object identity rebuilds the token source |
| `roomOptions` | LiveKit `RoomOptions`, spread over [`TALQING_AUDIO_CAPTURE_DEFAULTS`](#talqing_audio_capture_defaults) |

`warnings` are the token response's plan warnings (an unreachable team member, a recording mismatch across a handoff). They are filled only on the `client` branch; with the other branches your server received them in the token response.

### `TalqingSessionProvider`

```tsx
function TalqingSessionProvider(props: {
  session: TalqingSession;
  children: React.ReactNode;
  audio?: boolean; // default true
}): JSX.Element;
```

Puts the session in context (every hook below except `useTalqingImages` reads it) and renders LiveKit's `RoomAudioRenderer` so the agent is audible. `audio={false}` only if you render audio yourself.

### `useTalqingConnection`

```tsx
function useTalqingConnection(): {
  isConnected: boolean;
  connectionState: ConnectionState;
  room: Room;
  start: (options?: SessionConnectOptions) => Promise<void>;
  end: () => Promise<void>;
};
```

The call from context. `end()` hangs up; `room` is the way down to raw `livekit-client`.

> **Warning:**
> `isConnected` turns true while the agent is still starting. Before telling the caller to speak, wait for `useAgent().state === "listening"`.

### `createTalqingTokenSource`

```tsx
function createTalqingTokenSource(
  fetchToken: () => Promise<TalqingCallCredentials>,
): TokenSourceFixed;
```

From `@talqing/client`; what `useTalqingSession` uses internally. It calls `fetchToken` **at most once per source** (a failed attempt is not cached), because every `POST /v1/calls/token` files a call record and LiveKit asks its token source several times per call. Use a new source per call.

### `TALQING_AUDIO_CAPTURE_DEFAULTS`

```tsx
const TALQING_AUDIO_CAPTURE_DEFAULTS = {
  echoCancellation: true,
  noiseSuppression: true,
  autoGainControl: true,
};
```

Applied unless `roomOptions.audioCaptureDefaults` overrides it. Without echo cancellation the agent hears itself through the caller's speakers and interrupts itself.

## The agent, the transcript and the video

Re-exported unchanged from `@livekit/components-react` so one import covers a call.

| Export | What it is |
| --- | --- |
| `useAgent()` | The agent's state as a union narrowed by `state`: `disconnected`, `connecting`, `pre-connect-buffering`, `initializing`, `idle`, `listening`, `thinking`, `speaking`, `failed` (with `failureReasons`). Also `isConnected`, `canListen`, `isFinished`, `isPending` and the agent's tracks |
| `useSessionMessages()` | `{ messages, send(text), isSending }`: the live transcript, and typing into a voice call. The stored transcript is `GET /v1/calls/{session_id}` |
| `VideoTrack` | Renders one `TrackReference`; pair with [`useTalqingAvatarTrack`](#usetalqingavatartrack) |

`setLogLevel("debug")` from `livekit-client` turns on client logging for connection problems.

## Images

The caller attaches a photo mid-call; it is sent to the agent over a LiveKit byte stream, and the agent acknowledges it. The agent does not speak when an image lands; it sees it on the caller's next turn.

### `useTalqingImages`

```tsx
function useTalqingImages(room: Room | null | undefined): {
  send: (file: File) => Promise<TalqingSentImage>;
  sent: TalqingSentImage[];
  clear: () => void;
};
```

Takes the room explicitly; pass `useTalqingConnection().room`. `send` validates, downscales and sends, resolving once the bytes are on their way, and throws a displayable message if the file is refused or the agent has not joined. Outcomes arrive on `sent`:

| `TalqingSentImage` field | Meaning |
| --- | --- |
| `streamId` | Stream id the acknowledgement comes back under |
| `name` | Original file name |
| `previewUrl` | Object URL of the downscaled image; revoked by `clear()` and on unmount |
| `sentAt` | Epoch ms when handed to the room |
| `status` | `"sending"`, `"sent"`, `"failed"`, or `"unknown"` (no acknowledgement within 30 s) |
| `error` | A sentence to show, with `failed` or `unknown` |

```tsx
const { room } = useTalqingConnection();
const images = useTalqingImages(room);

<input
  type="file"
  accept="image/jpeg,image/png,image/webp"
  onChange={(e) => e.target.files?.[0] && void images.send(e.target.files[0])}
/>;
```

Without React: `sendTalqingImage(room, file)` and `onTalqingImageResult(room, handler)`, which returns an unregister function.

### Image helpers and constants

| Export | Behaviour |
| --- | --- |
| `talqingImageRejection(file)` | The message to show, or `null`. Refuses non-JPEG/PNG/WebP and files over 10 MB |
| `talqingDownscaleImage(file)` | `Promise<Blob>`: longest edge to 1,568 px; PNG stays PNG, everything else becomes JPEG at 0.85 |
| `talqingImageDataUrl(file)` | Validates and downscales a photo into a `data:` URL. Throws the refusal message |
| `TALQING_IMAGE_TOPIC` | `"talqing.images"` |
| `TALQING_IMAGE_MAX_BYTES` | `10485760` |
| `TALQING_IMAGE_MAX_EDGE_PX` | `1568` |
| `TALQING_IMAGE_ACK_TIMEOUT_MS` | `30000` |

### File helpers

| Export | Behaviour |
| --- | --- |
| `talqingFileRejection(file)` | The message to show, or `null`. Refuses an empty file or one over 20 MB; any type is accepted |
| `talqingFileDataUrl(file)` | A `data:` URL for a chat message's `files: [{ data_url, filename }]`. JPEG, PNG and WebP are downscaled first; everything else is sent as is. Throws the refusal message |
| `TALQING_FILE_MAX_BYTES` | `20971520` |
| `TALQING_MAX_FILES_PER_MESSAGE` | `4` |

## Screen share

### `useTalqingScreenShare`

```tsx
function useTalqingScreenShare(): {
  enabled: boolean;  // a screen is being shared
  pending: boolean;  // picker open or track publishing
  start: () => Promise<unknown>;  // call from a click
  stop: () => Promise<unknown>;
};
```

```tsx
const screen = useTalqingScreenShare();

<button onClick={() => void screen.start()} disabled={screen.pending}>
  {screen.enabled ? "Stop sharing" : "Share my screen"}
</button>;
```

The agent must have `vision_input.screenshare` enabled to see it. Capture and publish settings are fixed and exported for `room.localParticipant.setScreenShareEnabled(true, capture, publish)`:

| Export | Value |
| --- | --- |
| `TALQING_SCREEN_SHARE_CAPTURE` | `{ resolution: { width: 1920, height: 1080, frameRate: 1 }, contentHint: "text" }` |
| `TALQING_SCREEN_SHARE_PUBLISH` | `{ videoCodec: "h264", screenShareEncoding: { maxBitrate: 1000000, maxFramerate: 1 } }` |

Keep `h264`: `livekit-client` forces `contentHint: "motion"` on vp9 and av1 screen shares, which blurs text the agent needs to read.

## Userdata

`talqingAgentParticipant(room)` returns the agent's `RemoteParticipant`, or `undefined` before it joins.

## Frontend RPC

A tool's [`frontend_rpc`](https://docs.talqing.com/tools/frontend-rpc.md) operation calls a method on your page mid-call and waits for a string reply.

### `useTalqingRpcHandlers`

```tsx
function useTalqingRpcHandlers(handlers: TalqingRpcHandlers, enabled?: boolean): void;

type TalqingRpcHandlers = Record<string, (data: TalqingRpcInvocation) => string | Promise<string>>;
interface TalqingRpcInvocation { payload: string; responseTimeout?: number; callerIdentity?: string }
```

Dispatches by the envelope's `method`; `payload` is the method's payload as a JSON string. A `"*"` handler catches unmatched methods and receives the **whole envelope** (`{ method, payload }`) as its payload. An unmatched method with no wildcard returns RPC error `1404` to the agent; anything a handler throws returns `1500` with its message. Handlers register once the room is connected and unregister on unmount. **Memoize `handlers`**: a new object re-registers.

```tsx
const handlers = useMemo(() => ({
  show_product: async ({ payload }: TalqingRpcInvocation) => {
    const { sku } = JSON.parse(payload) as { sku: string };
    setHighlighted(sku);
    return JSON.stringify({ shown: true });
  },
}), []);
useTalqingRpcHandlers(handlers);
```

Without React: `registerTalqingRpcHandlers(room, handlers)` returns an unregister function.

### `useTalqingFrontendRpcs`

```tsx
function useTalqingFrontendRpcs(enabled?: boolean): {
  actions: Record<string, { payload?: JsonValue; at: number }>;
  clear: () => void;
};
```

Registers a wildcard handler that records the latest payload per method (with `at` in epoch ms) and always answers `"ok"`. Use it to render what the agent asked for without a handler per method.

### `TalqingRpcError`

```tsx
class TalqingRpcError extends Error {
  constructor(code: number, message: string, data?: string);
}
```

Throw it from a handler to return a specific error code instead of `1500`.

## Video

### `useTalqingAvatarTrack`

```tsx
function useTalqingAvatarTrack(): TrackReference | undefined;
```

The track to render for a video agent: the avatar worker's camera, then its screen share, then the agent's own camera or screen share. `undefined` until one is published.

```tsx
const track = useTalqingAvatarTrack();
return track ? <VideoTrack trackRef={track} className="rounded-xl" /> : null;
```

## Text chat from a browser

### `TalqingChat`

From `@talqing/client`; no React or LiveKit needed. Your server starts the chat (`POST /v1/chats`) and mints a chat token (`POST /v1/chats/{chat_id}/token`, valid 1 hour); the page holds only that token, which opens that one chat.

```tsx
import { TalqingChat } from "@talqing/client";

const chat = new TalqingChat({
  fetchToken: async () => (await fetch("/api/chat-token", { method: "POST" })).json(),
});

const { events, result } = chat.send("Where is my order?");
for await (const event of events) {
  if (event.event === "assistant.delta") render(event.text);
}
const turn = await result; // TalqingChatTurn
```

| Member | Behaviour |
| --- | --- |
| `new TalqingChat({ fetchToken })` | `fetchToken` returns `{ chat_id, token, expires_at, api_url }`, the `POST /v1/chats/{chat_id}/token` response unchanged. Called again a minute before expiry |
| `send(text, { files? })` | Streams one message. `events` yields `assistant.started`, `assistant.delta`, `assistant.completed`, `item.created`, `turn.result`; `result` resolves to the turn (`status`, `items`, `chat_status`, …). Either may be used alone. `files` takes `{ data_url, filename? }` from [`talqingFileDataUrl`](#file-helpers) |
| `items()` | The chat so far, oldest first. Only what the person can see, never tool calls |
| `end()` | Ends the chat; later messages are refused |

API errors throw `TalqingChatError` with `.status`, `.message` and `.errors`. See [text conversations](https://docs.talqing.com/channels/text-conversations.md).

## The escape hatch

This layer only adds. Take `useTalqingConnection().room` and use `livekit-client` directly for anything not covered:

```tsx
import { ConnectionQuality, RoomEvent } from "livekit-client";

const { room } = useTalqingConnection();
useEffect(() => {
  const onQuality = (quality: ConnectionQuality) => setBars(quality);
  room.on(RoomEvent.ConnectionQualityChanged, onQuality);
  return () => void room.off(RoomEvent.ConnectionQualityChanged, onQuality);
}, [room]);
```

Every other `@livekit/components-react` hook and component (`useLocalParticipant`, `useTrackToggle`, `StartMediaButton`, …) works inside `TalqingSessionProvider`. `livekit-client` and `@livekit/components-react` are peer dependencies so that the `Room` you receive is the same class as your own imports.

## Exported types

| Type | What it is |
| --- | --- |
| `TalqingCallCredentials` | `{ server_url, participant_token }` |
| `TalqingSessionOptions`, `TalqingCallOptions`, `TalqingSession` | `useTalqingSession`'s argument, `callOptions`, and its return (LiveKit's `UseSessionReturn` plus `warnings`) |
| `TalqingSentImage`, `TalqingImageStatus` | An image in flight and its four states |
| `TalqingRpcHandler`, `TalqingRpcHandlers`, `TalqingRpcInvocation`, `TalqingFrontendRpcEnvelope` | Frontend RPC shapes; the envelope is `{ method, payload? }` |
| `TalqingChatCredentials`, `TalqingChatTurn`, `TalqingChatItem`, `TalqingChatEvent`, `TalqingChatFile` | `TalqingChat` shapes |
| `TalqingChatAttachment` = `TalqingChatImage` \| `TalqingChatDocument` \| `TalqingChatFileAttachment` | A file on an item, by `kind` |
| `JsonObject`, `JsonValue`, `JsonPrimitive` | Free-form JSON |
| `AgentState`, `ReceivedMessage`, `TrackReference`, `UseAgentReturn`, `UseSessionReturn` | LiveKit's, re-exported |

- [Web calls](https://docs.talqing.com/channels/web-calls.md): Minting the token and a complete integration.
- [Web call features](https://docs.talqing.com/channels/web-call-features.md): Images, screen share and frontend actions in context.