Anthropic Claude - Response Prefill

Anthropic suggests the same three measures in their documentation, as well as a new one.

For their model Claude you can specify the first few tokens the assistant should generate and it will continue from there – in other words, you can pre-fill the response.

const response = client.messages.create(
    model="claude-3-5-sonnet-20240620",
    max_tokens=1024,
    messages=[
         {"role": "user", "content": `
            Generate a JSON object with keys for 'name', 'age', and 'occupation'.
            All values should be strings.
            Input: {${user_string}}
         `},
         {"role": "assistant", "content": "{"}  # Prefill here
    ]
)

This drastically increases the likelihood of the response starting with JSON. Just make sure to prepend the pre-filled characters again to your response before parsing.

Prompting, Parsing & Runtime Errors

But even being this specific in the prompt does not guarantee a usable response. In order to catch any run-time errors, we need to be prepared for the following scenarios:

1. Response is not a valid JSON
   • response is prefixed with text e.g. Here is your response formatted as JSON: {...}
   • response is larger than the context window and therefore an incomplete json {"name": "John Doe", "age": "30", "occupati
   • generated json response contains unescaped characters
2. validate that the JSON actually has the structure we expect
   • values are missing
   • keys are renamed
   • arrays are formatted as objects

In any of those cases, we could try to run the prompt again, creating additional cost and processing time, but still doesn’t guarantee a parsable response.

In my personal experience, I have seen all of the cases above occur, albeit not very frequently. However, as it is possible that they occur, we need to be prepared for it… or even better use one of the following more robust solutions.

Example: Schemas with ZOD

One of the most popular libraries for defining, stringifying, and parsing JSON schemas is Zod. Zod is typescript first, which means any schemas defined with zod are typed and can e.g. be used when parsing to safely access specific attributes of a JSON response. Zod can not create a schema based on the interfaces or classes you have defined in your typescript application.

Defining such a schema is trivial.

import { z } from "zod";

const UserSchema = z.object({
  id: z.string().uuid(),
  name: z.string().min(2).max(50),
  age: z.number().int().positive().max(120),
  email: z.string().email(),
  isActive: z.boolean(),
  tags: z.array(z.string()).min(1).max(5),
  preferences: z.object({
    theme: z.enum(["light", "dark", "system"]),
    notifications: z.boolean(),
  }),
  createdAt: z.date(),
});

Example: Schema in GPT-4o

For GPT-4o using the schema is quite simple. We just need to specify the correct response format and attach the schema. We do not need to explain the schema, the data attributes etc or expected output format in the prompt itself unless we want to.

import OpenAI from "openai";
import { zodResponseFormat } from "openai/helpers/zod";

const unstructuredUserInformation = "";

const completion = await openai.beta.chat.completions.parse({
  model: "gpt-4o-2024-08-06",
  messages: [
    { role: "system", content: "Extract the user information." },
    { role: "user", content: unstructuredUserInformation },
  ],
  response_format: zodResponseFormat(UserSchema, "user"),
});

const user = completion.choices[0].message.parsed;
console.log(user.name);

The attributes of the user can now safely be accessed and the likelihood of errors has been significantly reduced.

Example: Schema in Llama

For Mistral (using the Together AI API), we can use the same Zod schema we defined earlier. We just need to convert it to a JSON schema format using the zod-to-json-schema library. The process is straightforward:

const together = new Together();
const jsonSchema = zodToJsonSchema(UserSchema, "UserSchema");

const mistralExtract = await together.chat.completions.create({
  messages: [
    {
      role: "system",
      content: "Extract the user information from the given text.",
    },
    { role: "user", content: unstructuredUserInformation },
  ],
  model: "mistralai/Mixtral-8x7B-Instruct-v0.1",
  response_format: { type: "json_object", schema: jsonSchema },
});

Example: Schema in Gemini

Google’s Gemini model uses a slightly different approach for schema definition. Instead of using Zod directly, we need to translate our schema into Google’s FunctionDeclarationSchemaType format. Here’s how we can adapt our UserSchema for Gemini:

const genAI = new GoogleGenerativeAI("YOUR_API_KEY");
const model = genAI.getGenerativeModel({
  model: "gemini-1.5-pro",
  generationConfig: {
    responseMimeType: "application/json",
    responseSchema: {
      type: FunctionDeclarationSchemaType.OBJECT,
      properties: {
        id: { type: FunctionDeclarationSchemaType.STRING },
        name: { type: FunctionDeclarationSchemaType.STRING },
        age: { type: FunctionDeclarationSchemaType.NUMBER },
        email: { type: FunctionDeclarationSchemaType.STRING },
        isActive: { type: FunctionDeclarationSchemaType.BOOLEAN },
        tags: {
          type: FunctionDeclarationSchemaType.ARRAY,
          items: { type: FunctionDeclarationSchemaType.STRING },
        },
        preferences: {
          type: FunctionDeclarationSchemaType.OBJECT,
          properties: {
            theme: { type: FunctionDeclarationSchemaType.STRING },
            notifications: { type: FunctionDeclarationSchemaType.BOOLEAN },
          },
        },
        createdAt: { type: FunctionDeclarationSchemaType.STRING },
      },
    },
  },
});

const geminiResult = await model.generateContent(
  "Extract user information from: " + unstructuredUserInformation
);
const geminiUser = JSON.parse(geminiResult.response.text());
console.log("Gemini result:", geminiUser.name);