Building an AI receipt scanner with Next.js, the Vercel AI SDK and Supabase

The shape of the system

• Chat componentA client component where the user types a question or attaches a receipt image.
• Chat routeA route handler at app/api/chat that calls streamText from the ai package with the openai('gpt-4o-mini') model from @ai-sdk/openai, allows up to 30 seconds, and streams plain text back with toTextStreamResponse().
• Server actionsFunctions in app/actions/ai.ts that read and write Supabase using the signed-in user's identity.
• DatabaseSupabase PostgreSQL with Row Level Security. Session cookies are refreshed in proxy.ts, which is Next.js 16's replacement for middleware, by calling supabase.auth.getUser().

Step 1: the photo goes to the model

The browser accepts JPEG, PNG or WebP images up to 5 MB, reads the file with FileReader as a base64 data URL, and posts it to the chat route along with the message history. The route appends one extra user message with two parts, the image and an instruction:

{
  role: 'user',
  content: [
    { type: 'image', image: image },
    { type: 'text', text: 'This is a receipt image. Extract the amount, category, description, date and payment method. Ask me for any missing information.' },
  ],
}

Step 2: structured data through a marker

The system prompt tells the model to answer with a friendly message and also embed a JSON marker. A receipt with several line items produces an array, with one entry per item, and fields the model cannot see are left as empty strings and listed as missing.

[EXTRACT_RECEIPT: {"amount": "25.50", "category": "food", "description": "Coffee at Starbucks", "date": "2024-11-15", "paymentMethod": "", "missingFields": ["paymentMethod"]}]

Why a marker

The client strips the marker out of the displayed text, parses the JSON, and opens a confirmation dialog. The user sees a readable summary and a form, never raw JSON. Markers work with a plain text stream and are easy to build, but they depend on the model following the format. The Vercel AI SDK can also return schema-validated objects, which is the natural next step for stricter output.

Step 3: a person confirms

Reading numbers from a photo is never perfect, and a wrong amount in a finance app is worse than a missing one, so nothing is saved by the model alone. The confirmation form lists every extracted item with editable amount, category, description, date and payment method. Missing fields are highlighted, the first one gets focus, and a payment method is required for every item. Only after the user confirms does the client post the result to /api/add-expense.

Step 4: the server does not trust the model

The add-expense route calls a server action that treats its input as untrusted, even though it came from the app's own model. It takes the user's identity from the Supabase session, never from the request body, and validates every field before inserting:

const { data: { user } } = await supabase.auth.getUser()
if (!user) return { error: 'Not authenticated' }

if (!amount || amount <= 0) return { error: 'Invalid amount. Please provide a positive number.' }
if (!/^\d{4}-\d{2}-\d{2}$/.test(date)) return { error: 'Invalid date format. Use YYYY-MM-DD.' }
if (!allowedCategories.includes(normalizedCategory)) return { error: 'Invalid category.' }

Step 5: the database is the last line of defence

Row Level Security is enabled on the expenses table, with a separate policy for select, insert, update and delete. Each one compares the signed-in user to the row's owner, so even a bug in application code cannot read or change another user's rows through these tables.

create policy "Users can view own expenses"
  on public.expenses
  for select
  using (auth.uid() = user_id);

create policy "Users can insert own expenses"
  on public.expenses
  for insert
  with check (auth.uid() = user_id);

Constraints and indexes

The schema repeats the validation as CHECK constraints (a positive amount, and fixed lists of categories and payment methods), so bad data is rejected even if it gets past the application. A trigger keeps an updated-at column current and another creates a profile row when someone signs up. Indexes cover user, date, category and the common user-and-date query.

Answering questions about spending

For text questions, the chat route looks for simple patterns in the message. A request to list expenses fetches the latest ones, words such as total, month or week fetch statistics for that period, and a category name fetches that category's totals. A short summary is added to the system prompt, so the model answers from the user's real numbers.

The server actions behind this compute the total, average, count, per-category breakdown and top category. All of them filter by the signed-in user, and the raw expense list is capped at the latest 100 rows. This is retrieval by rules and not model tool-calling. It is predictable and cheap, but only as smart as the patterns.

Guardrails around the assistant

• ServerAt most 50 messages per conversation, 2,000 characters per message, a blocklist of sensitive words, and a finance-topic check. Image messages skip the topic check.
• ClientA 500 ms minimum gap between messages, plus image type and size checks before upload.
• System promptPersonal finance only, no investment advice, no acting on passwords or card numbers, and always ask before adding an expense.

Limitations

• The marker format depends on the model following instructions. Schema-validated structured output would be stricter.
• The keyword guardrails are blunt. A message containing the phrase credit card is rejected as sensitive even when the user only means a payment method, and the topic check is a keyword list and not a classifier.
• Intent detection uses regular expressions, so an unusual phrasing gets no spending context.
• Amounts are shown in dollars and the app is single-currency. Multi-currency support is listed as future work.

What is next

The project's roadmap lists budgets, recurring expenses, CSV and PDF export, multi-currency support, shared expenses and a mobile app. On the AI side, moving from markers to schema-validated output and replacing regex intent detection with model tool-calling would make the assistant sturdier.