~/tools / Embeddings and search (RAG) / RAG Citation Checker
RAG Citation Checker — Check source numbers and quotes in a RAG answer against the sources you sent
Check the citations in a RAG answer against the sources you sent: every source number must exist, every quote should appear in its source, and the tool lists sentences without a source number, sources that were never cited, and answers that were cut off. Reads Ollama /api/chat responses (including streamed ones), OpenAI-compatible responses and plain text. Nothing leaves your browser.
The example answer is written by hand to show a quote that is not in its source.
4 sources (a JSON array of strings). Answer read as an Ollama /api/chat response, in the form of JSON with citations.
Tokens in the prompt (prompt_eval_count): 306
- The quote of citation #2 is not in source [4].
Citations
| Source | Quote | Result |
|---|---|---|
| [2] | hnsw.ef_search is 40 by default | Found as written |
| [4] | 4 bytes per dimension plus 8 | Not in this source |
This only checks that the numbers and quotes match the sources. It cannot tell whether the answer is correct or whether a source really supports a sentence.
Response format from Ollama's API reference and source code, checked on Ollama 0.40.11 on 2026-11-05. Ollama API: /api/chat
Everything is done in your browser. The tool connects to nothing.
When a model answers from numbered sources, the citations can be checked mechanically. Paste the same sources you sent, in the same format as in the RAG Prompt Builder (a JSON array of strings or objects, or text separated by --- lines or blank lines; they are numbered from [1] in the same way), and paste the answer. The answer can be the whole response from Ollama's /api/chat (the tool takes message.content; a streamed response with one JSON object per line is joined), an OpenAI-compatible response (choices[0].message.content), or just the text. If the answer is JSON with an answer and a list of citations, as the RAG Prompt Builder asks for, the tool checks each citation: the source number must be one of the sources, and the quote should appear in that source, either exactly or after ignoring differences in spaces, letter case, full-width characters and curly quotes. If the answer is text, the tool collects the numbers in square brackets, such as [2] or [1, 3], reports numbers that do not match a source, and lists the sentences that have no number. In both cases it lists the sources that were never cited and warns when there are no citations at all, or when done_reason is "length", which means that the answer was cut off by num_predict. What it cannot do is judge whether the answer is right. In our test with qwen2.5:7b, an answer quoted the correct rule from its source word for word and still gave a wrong number, and another answer said it did not know but cited an unrelated source. A citation that passes here only shows that the quoted text is in the source.
The core logic of this tool is open source under the MIT License. github.com/seigo5555/toolkitbox-lib/src/rag-citations.ts
Use Cases
- Check that a local model's answer only cites sources that were really sent, with quotes that are really in them
- Find sentences in an answer that have no source, before you show the answer to users
- Compare prompts or models on the same questions by how often their citations hold
Usage
- Paste the sources you sent with the question, in the same order.
- Paste the answer: the whole /api/chat response, an OpenAI-compatible response, or the text of the answer.
- Read the table of citations, the sentences without a source number and the sources that were not cited.
FAQ
A quote is marked as not found. Is the answer wrong?
Not necessarily. The model may have paraphrased the source instead of copying it. But a quote that is not in the source cannot be checked, so treat that statement as unsupported until you have read the source.
All citations pass. Is the answer correct?
Not necessarily. The tool only checks that the numbers and quotes match the sources. In our test, the model quoted the correct rule (4 times the dimensions plus 8 bytes) and still answered 1552 bytes instead of 1,544 for a 384-dimension vector.
Why must the sources be in the same order?
The numbers in the answer refer to the order in which the sources were sent. If the order differs, [2] points to a different text and the check is meaningless.
How does the tool split the answer into sentences?
At line breaks, after . ! ? followed by a space, and after the full-width 。！？. A number that comes right after the end of a sentence, such as "… here. [1]", counts for that sentence. Abbreviations such as e.g. can split a sentence in two.
What does done_reason "length" mean?
In Ollama's source code, it is the reason given when the answer reached the token limit (num_predict) instead of stopping by itself. The end of the answer, and possibly its citations, is missing.
Is my data sent anywhere?
No. The check runs in your browser. The tool never connects to Ollama or to any other service.
Related articles
- Store and search Ollama embeddings in PostgreSQL with pgvectorKeep Ollama embeddings in PostgreSQL with pgvector, search them with HNSW or IVFFlat, and pass the results to /api/chat so that the answer cites its sources.
- Embeddings with Ollama: the basics for search and RAGMake embeddings with Ollama's /api/embed: the models and their sizes, truncate and dimensions, unit-length vectors and cosine similarity, from the official docs.