RAG Text Chunker — Chia văn bản cho RAG và dựng yêu cầu Ollama /api/embed

Chạy trên một máy kết nối được tới Ollama. Phản hồi có một vector cho mỗi chunk, theo thứ tự này.

curl http://localhost:11434/api/embed -d '{
  "model": "embeddinggemma",
  "input": [
    "Embeddings turn text into numeric vectors. You can store them in a vector database and search them with cosine similarity, or use them in a RAG pipeline.\n\nThe length of each vector depends on the embedding model. Use the same model when you index your documents and when you embed a question.\n\n",
    "Ollama'\''s /api/embed endpoint takes a single string or an array of strings. By default, input that is longer than the model'\''s context window is truncated. Set truncate to false to get an error instead.\n\n",
    "Split long documents into smaller chunks before you embed them, so that each chunk fits in the context window and a search can return the part that answers the question."
  ],
  "truncate": false
}'

Các chunk

3 chunk. Chunk dài nhất có 294 ký tự.

1. #0 · 294 ký tự · bắt đầu tại vị trí 0

   Embeddings turn text into numeric vectors. You can store them in a vector database and search them with cosine similarity, or use them in a RAG pipeline. The length of each vector depends on the embedding model. Use the same model when you index your documents and when you embed a question.

2. #1 · 202 ký tự · bắt đầu tại vị trí 294

   Ollama's /api/embed endpoint takes a single string or an array of strings. By default, input that is longer than the model's context window is truncated. Set truncate to false to get an error instead.

3. #2 · 169 ký tự · bắt đầu tại vị trí 496

   Split long documents into smaller chunks before you embed them, so that each chunk fits in the context window and a search can return the part that answers the question.

{
  "model": "embeddinggemma",
  "input": [
    "Embeddings turn text into numeric vectors. You can store them in a vector database and search them with cosine similarity, or use them in a RAG pipeline.\n\nThe length of each vector depends on the embedding model. Use the same model when you index your documents and when you embed a question.\n\n",
    "Ollama's /api/embed endpoint takes a single string or an array of strings. By default, input that is longer than the model's context window is truncated. Set truncate to false to get an error instead.\n\n",
    "Split long documents into smaller chunks before you embed them, so that each chunk fits in the context window and a search can return the part that answers the question."
  ],
  "truncate": false
}