Turning a Personal Website Into an AI Assistant

Most personal websites are read-only. They present projects, articles, and work history. The visitor still has to find the right page and work out which details matter.

I wanted mine to do more. Someone should be able to ask, "What has Joseph built with AI?" and get an answer based on my actual projects. They should also be able to switch to voice, interrupt the assistant, and keep talking without opening another application.

The finished interface looks like a chat window. Behind it are two conversation paths, a shared set of website tools, usage controls, and a contact flow that never lets the model send an email on its own.

Here is how those paths work, including the parts that took more thought than the model call.

The assistant needed a narrow job

My first question was: what should the assistant be allowed to do?

Maya is a conversational interface to my personal website. She can explain my projects, experience, writing, and ways to contact me. She refers to me in the third person. If the available data does not support an answer, she says so.

One assistant profile holds those rules. It contains the shared identity and factual limits, then adds instructions for the current mode. Voice needs shorter responses and should not read URLs aloud. Text can use Markdown and return more detail.

The application has two model connections, so I keep this contract in one place. Two copied prompts would eventually drift.

lib/assistant-profile.js
  shared identity and factual boundaries
  text conversation instructions
  realtime voice instructions
  current page context

I also pass the current page as context. If a visitor opens the assistant while reading an article and asks, "What is this post about?", Maya can resolve "this post" to the article on screen.

A long prompt was not enough

I could have placed every project and article inside the system instructions. It would have worked for a prototype and become awkward to maintain.

A large prompt would also send irrelevant content with most questions. A visitor asking about frontend work does not need the full text of every backend article.

I exposed a small set of tools. They retrieve profile data, search projects, read articles, and prepare a contact message. The model chooses a tool; the server implements it and validates every argument.

One tool looks like this:

const searchProjects = tool({
  description: "Find Joseph's projects relevant to a visitor's question.",
  inputSchema: z.object({
    query: z.string().trim().min(1).max(120)
  }),
  execute: async ({ query }) => findProjects(query)
});

The model can query website content through this function. It has no direct access to the filesystem, database, or email provider.

{
  "query": "AI"
}

The response UI renders tool results separately from the assistant's prose. I tell the model not to repeat titles and links already visible in the interface. Before adding that instruction, project cards were often followed by a paragraph saying the same thing.

Generative UI instead of longer answers

Structured output gave me a better option than turning every result into prose. The application already knows the shape of each tool result, so it can render a useful interface for it.

Article searches render as a compact list. A résumé request returns a card with a direct PDF link. Photo requests use an approved image grid. The contact tool has its own review card, shown later in this article.

These components also run inside the live assistant. The model selects the tool. Application code decides which interface appears and which actions are available.

Streaming the text conversation

Typed chat goes through a server route built around the AI SDK. The route validates incoming messages, builds the assistant instructions, registers the approved tools, and streams the result.

export async function POST(request) {
  const input = await validateAssistantRequest(request);
 
  const result = streamText({
    model: openai(TEXT_MODEL),
    system: buildAssistantInstructions(input.pageContext),
    messages: await convertToModelMessages(input.messages),
    tools: createAssistantTools()
  });
 
  return result.toUIMessageStreamResponse();
}

Streaming makes the first response appear sooner. It also changes the frontend state model. During a response, the renderer may receive partial text, a tool request, or a completed tool result. It cannot treat every message as a finished string.

The route handles abuse controls too. It checks message shape and size, applies rate limits, and derives a safety identifier before calling the model.

Recorded voice starts as an array of chunks

The first voice path is push-to-record. It uses the browser's MediaRecorder API and works like a voice note.

MediaRecorder emits dataavailable events while it records. I collect those chunks in a ref because recording state does not need to rerender the component.

const chunksRef = useRef([]);
 
recorder.addEventListener("dataavailable", (event) => {
  if (event.data.size > 0) {
    chunksRef.current.push(event.data);
  }
});

When recording stops, the chunks become one Blob:

recorder.addEventListener("stop", async () => {
  const audio = new Blob(chunksRef.current, {
    type: recorder.mimeType
  });
 
  chunksRef.current = [];
  await transcribe(audio);
});

The transcription endpoint checks that an audio file exists, rejects oversized uploads, and sends a valid file to the transcription model. The returned text enters the typed conversation path.

Reusing that path means recorded voice inherits its tools, rate limits, renderer, and conversation history.

Live voice is a different system

Push-to-record follows a request and response cycle. Live voice keeps a connection open.

For live voice, the browser creates an RTCPeerConnection, adds the microphone track, and opens a data channel for Realtime API events.

const peerConnection = new RTCPeerConnection();
const mediaStream = await navigator.mediaDevices.getUserMedia({
  audio: true
});
 
for (const track of mediaStream.getAudioTracks()) {
  peerConnection.addTrack(track, mediaStream);
}
 
const eventChannel =
  peerConnection.createDataChannel("oai-events");

The audio track carries speech. The data channel carries transcripts, assistant output, interruptions, errors, and function calls.

The browser creates an SDP offer and sends it through a server-created Realtime session. After applying the remote description, it can send audio over the peer connection.

const offer = await peerConnection.createOffer();
await peerConnection.setLocalDescription(offer);
 
const answer = await startRealtimeSession(offer.sdp);
 
await peerConnection.setRemoteDescription({
  type: "answer",
  sdp: answer
});

My server handles session creation and tool execution. WebRTC carries the audio, so the application server never proxies a continuous media stream.

Sharing tools between text and voice

The text SDK executes registered tools inside its response loop. Realtime reports function calls as data-channel events, which required a small bridge.

When the voice model requests a tool, the browser sends its name and arguments to a dedicated server route. The route runs the validated implementation used by text chat, then the browser returns the result to the Realtime conversation.

eventChannel.send(
  JSON.stringify({
    type: "conversation.item.create",
    item: {
      type: "function_call_output",
      call_id: callId,
      output: JSON.stringify(toolResult)
    }
  })
);
 
eventChannel.send(
  JSON.stringify({ type: "response.create" })
);

This leaves me with two transport adapters and one source of truth for website data and validation.

Voice needs lifecycle controls

Real-time sessions can remain open after someone walks away. A connection can also drop halfway through a response, before the browser reports a clean end.

The application treats a voice conversation as a lifecycle:

claim allowance
  -> start session
  -> connect peer
  -> exchange audio and events
  -> end or disconnect
  -> complete allowance record

The server signs the allowance state, so accounting does not depend on a duration reported by the browser. Separate start and end routes record the session. Idle timers close forgotten conversations. Client cleanup stops microphone tracks and closes the data channel and peer connection.

This extra work has a practical purpose: voice sessions cost more than ordinary page requests, and an active microphone needs a clear end state.

The model does not send email

Maya can prepare a contact draft for a visitor.

The interface renders the draft as a contact card. The visitor can review and edit the name, address, and message before submitting it. A separate endpoint validates the request, checks for spam and duplicate submissions, and sends the email.

I deliberately left delivery out of the assistant tool. Sending requires an explicit click, and the model never receives general access to the email provider.

I use the same split for other sensitive actions: the model prepares the request, and application code authorizes it.

What took the most time

The model calls were straightforward. The harder work was keeping text and voice consistent while managing browser audio, session accounting, and user-approved email delivery. That integration took more time than the AI itself.