Experiment Notes: A Blog Chatbot and the OpenAI API
Adding an OpenAI chatbot to a personal blog: one Next.js API route and a system prompt were enough, and RAG was dropped for cost and complexity.
Joohwan Yu · Founder, Full Stack Developer
Toris · Published
Contents
Q. Did a personal blog chatbot need RAG?
No. One Next.js API Route and a system prompt were enough, and I dropped RAG because the quality gain was small compared with the embedding cost and complexity.
This post is a record of an experiment, not an introduction to a finished product. While adding an OpenAI API chatbot to a personal blog, I sum up what I tried to verify, which design I chose, where I backed off, and what I learned along the way.
Problem Definition: What I Tried to Verify
There were two main questions I wanted to answer.
- What is the minimum setup needed to add an LLM chatbot to a personal-blog-scale service? Can it be done inside the Next.js app the blog already runs on, with no separate backend?
- How much machinery does it take to reflect the blog’s content in the answers? Is a prompt alone enough, or do I need a retrieval layer like RAG?
In other words, the goal of this project was not a “good chatbot” but finding the right level of complexity for an LLM feature at personal-project scale.
Background: How Does an LLM Produce an Answer?
Before getting into the design, I worked out for myself what happens behind the API. Without knowing what goes on between input and output, you can’t estimate cost and limits.
Generative AI and LLMs
- Generative AI is a branch of AI that creates new content such as conversation, stories, images, video, and music.
- An LLM (large language model) is the technology within it that handles understanding and generating text.
From Input to Output
- Receive natural-language input
- Split the context into tokens
- Pick the next token
- Keep repeating step 3
- Stop generating at the right point
- Output natural language
The core is step 3, “picking the next token.” Internally it goes through the following process.
token array → embedding vector array → (complex computation) → context vector
→ probability distribution of the next token → next token selection
What “Training” Means
- It learns how to turn the input language into a single context vector. This means mapping an appropriate embedding vector to each token and updating the countless weights in the computation.
- It learns how to compute the next token’s probability distribution from that context vector. This means updating the matrix that turns a context vector into a per-token probability distribution.
ChatGPT’s Three Layers
- Pre-Training: learning to understand natural language and predict tokens well
- Fine-Tuning & RLHF: adjusting it to follow human instructions well
- Prompting: writing a default context so anyone can use it easily
The practical conclusion I took from this was one thing. API call cost is ultimately proportional to the number of tokens, and what you put in the context is itself cost design. This view carries straight into the RAG decision later.
Design: Starting with One Next.js API Route
The answer to the first question (the minimum setup) was simple. Since the blog runs on Next.js (Pages Router), a single /api handler could be the whole chatbot backend.
- The client sends an array of conversation messages by POST
- The server handler prepends a system prompt and calls the OpenAI Chat Completions API
- It appends the response message to the array and returns it
The handler I wrote at the time is below.
import type { NextApiRequest, NextApiResponse } from 'next';
import OpenAI from 'openai';
import type { ChatCompletionMessageParam } from 'openai/resources/index.mjs';
const openai = new OpenAI({
apiKey: process.env.NEXT_PUBLIC_OPENAI_API,
organization: process.env.NEXT_PUBLIC_ORGANIZATION_API
});
type CompletionsResponse = {
messages: ChatCompletionMessageParam[];
};
export default async function handler(
req: NextApiRequest,
res: NextApiResponse<CompletionsResponse>
) {
if (req.method !== 'POST') return res.status(405).end();
const messages = req.body.messages as ChatCompletionMessageParam[];
const response = await openai.chat.completions.create({
messages: [
{
role: 'system',
content:
'This chatbot is a dedicated chatbot that answers development questions about this blog in good faith.'
},
...messages
],
model: 'gpt-3.5-turbo'
});
messages.push(response.choices[0].message);
res.status(200).json({ messages });
}
The structure itself was as minimal as intended. Looking at it again now, though, there is one clear flaw. I used the NEXT_PUBLIC_ prefix on the API key environment variables. In Next.js, this prefix exposes the value in the client bundle, so it is a name you must never use for a server-only secret. Even if the value was only used inside the server handler, the naming itself is an accident waiting to happen, and it remains one of the more valuable lessons of this experiment, worth more than the code.
The Second Question: Trying and Dropping RAG
To reflect the blog’s content in the answers, I first tried to introduce RAG (Retrieval-Augmented Generation). The design was to embed the blog posts, store them in a vector database, and search for content related to the user’s question to provide as context.
Problems That Surfaced Along the Way
1. API cost
- Embedding the whole blog’s content incurred a considerable API cost.
- Pushing search results into the context for every user question increased tokens per call, and costs came out higher than expected.
- At personal-project traffic levels this extra cost was a burden, and the improvement in answer quality was small for the cost.
2. Over-engineering
- Given the usage and the complexity actually needed for a personal blog chatbot, RAG was a solution bigger than the problem.
- Simple prompt engineering alone gave answer quality sufficient for the purpose.
- Bringing in a vector DB and a retrieval pipeline raised system complexity and maintenance burden together.
The Decision to Drop It
On both cost and complexity, I judged that RAG exceeded this project’s real needs, so I dropped it and simplified to a basic prompt engineering approach. It was less a failure than getting the answer to the second verification question. At personal-blog scale, a prompt alone is enough.
Limitations
I also state clearly what this experiment did not cover.
- Because answers depend on the model’s pre-trained knowledge and the system prompt, I can’t guarantee responses at the level of accurately citing the blog’s latest posts. This is a limitation I deliberately accepted when dropping RAG.
- Operational mechanisms like request authentication and usage limits were outside the scope of the experiment. They are the first thing needed if this were to grow into a public service.
- As with the environment variable naming problem mentioned above, there were points where the goal of a minimal setup led me to skip checking security practices.
What I Learned
- Adopt technology based on real need relative to cost and complexity. With LLMs in particular, context design is cost design, and if you add a retrieval layer on the mere expectation that “it will probably be more accurate,” the cost arrives first.
- A simple solution may suit a small project better. The minimal setup of one API Route plus a system prompt was the right answer at this scale.
- It is better to avoid over-engineering, start from the minimal solution that fits the real need, and expand once the need is proven. If I had started with a prompt and looked at RAG once its shortfall was proven, rather than building RAG first and then dropping it, I would have saved the embedding cost.
- Secret management is not minimal even in a minimal setup. I left a note as a code review item that a single environment variable name can lead to an exposure incident.
Boiled down to one sentence, the conclusion of this experiment is this. A technology’s complexity and cost must balance with the project’s real needs, and that balance point lies on a simpler side than you’d expect.
Advertisement