Long memory for an LLM chat, without a vector database

An LLM doesn't remember anything between requests. Whatever it seems to remember is whatever you send with the next message.

That's easy for short chats: send the whole history. It breaks for long ones. A chat that runs for weeks doesn't fit in the context window, and sending all of it every time costs more with every message.

The usual answer is embeddings and a vector database. For a one-on-one chat, a simpler setup goes a long way: a few layers of context, each with a fixed size, built fresh for every request. This is the setup I use.

The layers

Every request gets one system message, built in this order, followed by the recent messages:

Each layer answers a different question. Recent messages give the flow of the conversation. The summary gives the story. Memories give the facts. Earlier lines bring back the exact words when they matter again.

1. A budget for recent messages

Walk back from the newest message and stop when the budget is full. If a single message is longer than the whole budget, keep its end, since that's the part the reply will answer.

function trimToBudget(turns: Turn[], maxChars = 8000, maxTurns = 40): Turn[] {
  const kept: Turn[] = [];
  let used = 0;
  for (let i = turns.length - 1; i >= 0 && kept.length < maxTurns; i--) {
    const text = turns[i].content;
    if (used + text.length > maxChars) {
      if (kept.length === 0) kept.unshift({ ...turns[i], content: text.slice(-maxChars) });
      break;
    }
    used += text.length;
    kept.unshift(turns[i]);
  }
  return kept;
}

Counting characters instead of tokens keeps this the same for every model and provider. You don't need a tokenizer for each one, and a character budget is a safe stand-in.

2. Extract memories in the background

Memories are short facts: "Has an interview on Friday", "Her brother is called Arjun". A background job updates them:

• after 12 new messages,
• or when the chat has been quiet for 20 seconds,
• or when the app goes to the background,

whichever comes first.

The job sends the same model the current memories, the summary and the messages since the last update, and asks for JSON only:

{"memories": ["Has an interview on Friday", "..."], "summary": "..."}

The prompt asks for lasting facts only (people and their names, work or studies, plans and dates, likes and dislikes, important events), one short sentence each, with changed facts updated and duplicates merged.

Don't trust the reply blindly:

• Read the JSON even if the model wrapped it in a code fence or added text around it.
• If it still isn't valid, ask once more with "Reply with valid JSON only." added.
• Trim every item, cut it at a length limit, drop empty items and duplicates, and cap the list.
• If either key is missing, throw the whole reply away and try again after the next message.
• Replace the old memories and summary in one database transaction.

One rule matters a lot: users can write their own memories, and the model never touches them. They're sent marked as fixed, the prompt says never to change, repeat or return them, and the clean-up step drops anything that repeats one. A memory list is easier to trust when the facts users typed stay exactly as typed.

3. A rolling summary for everything older

The same job rewrites the summary: the story so far, under 200 words, in the past tense, covering what happened and how the user has been feeling. The model merges the old summary with the new messages and keeps important older events.

Memories hold facts. The summary holds the arc. Between them, a conversation from three weeks ago still shapes today's reply in a few hundred words.

4. Recall: keyword RAG with IDF

Memories and the summary lose the exact words. If the user says "the bank called back!", the model should see the old line where they said which bank interview it was.

You can find that line without embeddings:

1. Split the new message into words. Drop words shorter than 3 letters and a list of common words.
2. For each older message outside the recent window, add up a weight for every word it shares with the new message.
3. Weight each word by how rare it is in this chat (inverse document frequency), so "bank" counts for much more than "today".
4. Keep the top four, put them back in time order, and label each with its date and who said it.

function findRecall(older: Message[], query: string, max = 4): Message[] {
  const wanted = [...new Set(words(query))];
  if (wanted.length === 0) return [];
  const docs = older.map((m) => ({ m, set: new Set(words(m.content)) }));
  const weight = new Map(
    wanted.map((w) => [w, Math.log(1 + docs.length / (1 + docs.filter((d) => d.set.has(w)).length))]),
  );
  return docs
    .map((d) => ({ m: d.m, score: wanted.reduce((n, w) => n + (d.set.has(w) ? weight.get(w)! : 0), 0) }))
    .filter((d) => d.score > 0)
    .sort((a, b) => b.score - a.score)
    .slice(0, max)
    .map((d) => d.m)
    .sort((a, b) => a.createdAt - b.createdAt);
}

The common-word list matters more than it looks. In a chat that mixes English, Hinglish and Hindi, words like "kya", "hai" and "nahi" appear everywhere and would match every line. Put them in the list for every language your users write in.

In personal chats, the words that matter are usually rare ones: names, places, exams, companies. That's exactly what IDF rewards. You need no embedding model, no index and nothing extra to host, and the same input always gives the same lines.

It has limits. It won't match synonyms ("job" and "work"), and it misses different spellings of the same word, which happens a lot in Hinglish. For a large knowledge base, or users who paraphrase a lot, embeddings are the better tool.

Tell the model how to use what it gets:

Lines from earlier in your chats that may matter now (older than the messages that follow):
...
Use them only if they fit what the user is saying now. Don't quote them or say you looked them up.

5. Treat everything users can edit as data

The persona, the memories, the summary, the earlier lines and any custom instructions are all text that users (or the model) wrote. If any of it lands in the prompt as plain text, it can pose as a rule.

So before it goes in:

• Strip control characters.
• Turn line breaks into spaces, so it can't start a new rule line.
• Replace < and > with look-alike characters, so it can't close a section or open a new one.
• Put each piece inside its own clearly marked section, like <memories>…</memories>.

And keep the order fixed: your rules first, the user's content in the middle, your reminder last.

6. End with a short reminder

The last lines of the system message weigh the most, so the final layer repeats only what matters most: stay in character, answer in the language and script of the user's last message, and keep it short.

Language drift is the reason this layer exists. If a persona usually talks in Hinglish, the model tends to drift into Hinglish even when the user writes in Devanagari. So a one-line note goes at the very end, where it carries the most weight: "Their last message is in Devanagari, so write your reply in Devanagari."

The whole thing in one sentence

Send the recent messages within a budget, a summary for the story, a short list of facts for what must not be forgotten, and a few old lines found by rare words, with your rules on both ends and everything users wrote escaped as data.