# Add Collection
Source: https://docs.zeroentropy.dev/api-reference/collections/add-collection
/api-reference/openapi.json post /collections/add-collection
Adds a collection.
If the collection already exists, a `409 Conflict` status code will be returned.
# Delete Collection
Source: https://docs.zeroentropy.dev/api-reference/collections/delete-collection
/api-reference/openapi.json post /collections/delete-collection
Deletes a collection.
A `404 Not Found` status code will be returned, if the provided collection name does not exist.
# Get Collection List
Source: https://docs.zeroentropy.dev/api-reference/collections/get-collection-list
/api-reference/openapi.json post /collections/get-collection-list
Gets a complete list of all of your collections.
# Add Document
Source: https://docs.zeroentropy.dev/api-reference/documents/add-document
POST /documents/add-document
Adds a document to a given collection.
A status code of `201 Created` will be returned if a document was successfully added. A status code of `409 Conflict` will be returned if the given collection already has a document with the same path.
If `overwrite` is given a value of `true`, then a status code of `200 OK` will be returned if a document was overwritten (Rather than a status code of `409 Conflict`).
When a document is inserted, it can take time to appear in the index. Check the `/status/get-status` endpoint to see progress.
# Delete Document
Source: https://docs.zeroentropy.dev/api-reference/documents/delete-document
post /documents/delete-document
Deletes a document
A `404 Not Found` status code will be returned, if the provided collection name or document path does not exist.
# Get Document Info
Source: https://docs.zeroentropy.dev/api-reference/documents/get-document-info
post /documents/get-document-info
Retrieves information about a specific document. The request parameters define what information you would like to receive.
A `404 Not Found` will be returned if either the collection name does not exist, or the document path does not exist within the provided collection.
# Get Document Info List
Source: https://docs.zeroentropy.dev/api-reference/documents/get-document-info-list
post /documents/get-document-info-list
Retrives a list of document metadata information that matches the provided filters.
The documents returned will be sorted by path in lexicographically ascending order. `path_gt` can be used for pagination, and should be set to the path of the last document returned in the previous call.
A `404 Not Found` will be returned if either the collection name does not exist, or the document path does not exist within the provided collection.
# Get Page Info
Source: https://docs.zeroentropy.dev/api-reference/documents/get-page-info
/api-reference/openapi.json post /documents/get-page-info
Retrieves information about a specific page. The request parameters define what information you would like to receive.
A `404 Not Found` will be returned if either the collection name does not exist, or the document path does not exist within the provided collection.
# Update Document
Source: https://docs.zeroentropy.dev/api-reference/documents/update-document
/api-reference/openapi.json post /documents/update-document
Updates a document. This endpoint is atomic.
Currently both `metadata` and `index_status` are supported.
- When updating with a non-null `metadata`, the document must have `index_status` of `indexed`. After this call, the document will have an `index_status` of `not_indexed`, since the document will need to reindex with the new metadata.
- When updating with a non-null `index_status`, setting it to `not_parsed` or `not_indexed` requires that the document must have `index_status` of `parsing_failed` or `indexing_failed`, respectively.
A `404 Not Found` status code will be returned, if the provided collection name or document path does not exist.
# Embed
Source: https://docs.zeroentropy.dev/api-reference/models/embed
/api-reference/openapi.json post /models/embed
Embeds the provided input text with ZeroEntropy embedding models.
The results will be returned in the same order as the text provided. The embedding is such that queries will have high cosine similarity with documents that are relevant to that query.
Organizations will, by default, have a ratelimit of `500000` bytes-per-minute. Ratelimits are refreshed every 15 seconds. If this is exceeded, requests will be throttled into `latency: "slow"` mode, up to `5000000` bytes-per-minute. If even this is exceeded, you will get a `429` error.
The "bytes" used by a request is calculated as `sum(150 + s.encode('utf-8') for s in input)`. Note a baseline overhead of `150` bytes. The maximum per-request payload size is `5000000` bytes.
To increase your ratelimits, subscribe to a higher tier on the [ZeroEntropy dashboard](https://dashboard.zeroentropy.dev/billing). Any payments made for subscriptions in a calendar month will be deducted from your usage charges for that month.
To request even higher ratelimits, please contact [founders@zeroentropy.dev](mailto:founders@zeroentropy.dev) or message us on [Discord](https://go.zeroentropy.dev/discord) or [Slack](https://go.zeroentropy.dev/slack)!
# Rerank
Source: https://docs.zeroentropy.dev/api-reference/models/rerank
/api-reference/openapi.json post /models/rerank
Reranks the provided documents, according to the provided query.
The results will be sorted by descending order of relevance. For each document, the index and the score will be returned. The index is relative to the documents array that was passed in. The score is the query-document relevancy determined by the reranker model. The results will be returned in descending order of relevance.
Organizations will, by default, have a ratelimit of `500,000` bytes-per-minute (BPM) and `1000` requests-per-minute (RPM). Ratelimits are refreshed every 15 seconds. If this is exceeded, requests will be throttled into `latency: "slow"` mode, up to `5000000` bytes-per-minute. If even this is exceeded, you will get a `429` error.
The "bytes" used by a request is calculated as `sum(150 + query.encode('utf-8') + d.encode('utf-8') for d in documents)`. Note a baseline overhead of `150` bytes, and that the query bytes are included for each document, as rerankers are cross-encoders. The maximum per-request payload size is `5000000` bytes.
To increase your ratelimits, subscribe to a higher tier on the [ZeroEntropy dashboard](https://dashboard.zeroentropy.dev/billing). Any payments made for subscriptions in a calendar month will be deducted from your usage charges for that month.
To request even higher ratelimits, please contact [founders@zeroentropy.dev](mailto:founders@zeroentropy.dev) or message us on [Discord](https://go.zeroentropy.dev/discord) or [Slack](https://go.zeroentropy.dev/slack)!
# Top Documents
Source: https://docs.zeroentropy.dev/api-reference/queries/top-documents
post /queries/top-documents
Get the top K documents that match the given query
# Top Pages
Source: https://docs.zeroentropy.dev/api-reference/queries/top-pages
/api-reference/openapi.json post /queries/top-pages
Get the top K pages that match the given query
# Top Snippets
Source: https://docs.zeroentropy.dev/api-reference/queries/top-snippets
post /queries/top-snippets
Get the top K snippets that match the given query.
You may choose between coarse and precise snippets. Precise snippets will average ~200 characters, while coarse snippets will average ~2000 characters. The default is coarse snippets. Use the `precise_responses` parameter to adjust.
# Get Status
Source: https://docs.zeroentropy.dev/api-reference/status/get-status
post /status/get-status
Gets the current indexing status across all documents.
If a collection name is passed in, it will get the indexing status of only the documents within that collection. Otherwise, it will show the cumulative status across all of your collections.
A `404 Not Found` status code will be returned, if a collection name was provided, but it does not exist.
# Architecture
Source: https://docs.zeroentropy.dev/architecture
ZeroEntropy's system architecture
## System Architecture
ZeroEntropy is built with the purpose of bringing advanced document intelligence to your knowledge base. We've designed our retrieval system to solve common failures found in native hybrid search implementations.
### Ingestion Architecture
![]()
![]()
### Query Architecture
![]()
![]()
### Core Components
1. **Document Processing Pipeline**
* Handles document ingestion and parsing
* Supports a variety document formats (PDF, DOCX, PPT, TXT, etc.)
* Supports complex diagrams found in medicine, manufacturing, and deep tech.
* Correctly parses the hierarchical structure found in legal, healthcare, and other industries.
* Uses LLMs to tag the data, as if you had hired thousands of SEO engineers to manually annotate your corpus.
2. **Data Storage**
* Document raw data is stored in object storage, along with images for PDF/DOCX/PPT pages.
* Document metadata is stored in PostgreSQL.
* The document ingestion pipeline stores vector data in [turbopuffer](https://turbopuffer.com/), keyword data in [ParadeDB](https://www.paradedb.com/) BM25 indices, collection dictionaries in S3 with the [BK-tree](https://en.wikipedia.org/wiki/BK-tree) data structure.
3. **Query Processing Engine**
* Interprets natural language queries without any special syntax required.
* Uses LLM-in-the-loop to automatically generate potential keywords, semantic searches, and to make a final review of everything retrieved before making a final decision on exactly what is most important and relevant to your query.
### Security & Performance
* End-to-end encryption for data in transit and at rest.
* On-Prem deployment available for enterprise users, as easy-to-use docker images.
# Core Concepts
Source: https://docs.zeroentropy.dev/core-concepts
Understand the core concepts behind ZeroEntropy — our models and end-to-end search engine.
ZeroEntropy gives you full control over search quality — from the **models** that power retrieval and reranking, to the **architecture** that runs complete search pipelines.
***
## Models
ZeroEntropy offers state-of-the-art models for retrieval and reranking:
1. **`zerank-2` and `zerank-1-small`**
Cross-encoder rerankers that can dramatically improve the ordering of your search results.
They work on top of any vector or hybrid retriever, boosting precision by recognizing subtle semantic distinctions across complex domains — from legal and financial documents to code and scientific text.
2. **`zembed-1`**
A high-quality embedding model optimized for retrieval across all domains and languages -- faster and more cost-efficient than frontier alternatives.
Learn more about model specs, performance, and examples in the [**Models section →**](/models)
***
## End-to-End Search Engine
Beyond individual models, ZeroEntropy provides an **end-to-end search engine** that integrates retrieval, reranking, and query orchestration into a single unified API.
You can build custom indexes, control document granularity, and query at the level of collections, documents, pages, or snippets.
See the [**Architecture section →**](/architecture) for a detailed breakdown of how all components connect.
## Data Concepts
1. **Collections:** Independent datastores for your documents — ideal for multi-tenant or multi-dataset setups.
2. **Documents:** Core indexing units. Each document can include metadata for [document-level filtering](/metadata-filtering).
3. **Pages:** Ordered segments within a document, preserving contextual flow (e.g., PDF pages or sequential messages in a conversation).
## Query Modes
1. **Top-K Documents** — retrieve the most relevant documents for a query.
2. **Top-K Pages** — retrieve the most relevant pages within documents.
3. **Top-K Snippets** — fine-grained retrieval for short, high-precision results (≈200–2000 characters), with built-in reranking
***
## Examples
* **Files:** PDFs or .docx uploads are automatically parsed into pages.
* **Conversations:** Treat each Slack channel as a document and each message as a page to preserve message order.
* **CSVs:** Each row (e.g., SKU) can be indexed as a document for structured retrieval.
***
By understanding these concepts, you can combine ZeroEntropy’s **models** and **search architecture** to build the most accurate and customizable retrieval systems available.
Next: explore the [**API Reference →**](/api-reference/)
# Embed
Source: https://docs.zeroentropy.dev/examples/embed
Inference ZeroEntropy's embedding model zembed-1
#### Embed queries and text
`zembed-1` is the default embedding model used in zsearch, ZeroEntropy's search engine.
You can also call the embedding model directly and plug it into the vector database of your choice using the `/models/embed` endpoint or directly through the SDKs.
```python Python theme={null}
from zeroentropy import ZeroEntropy
zclient = ZeroEntropy()
query = "What is Retrieval Augmented Generation?"
documents = [
"RAG combines retrieval with generation by conditioning the LLM on external documents.",
"Retrieval-Augmented Generation is a machine learning technique introduced by Meta AI in 2020.",
"It uses reinforcement learning to generate music sequences.",
"RAG can improve factual accuracy by grounding answers in retrieved evidence.",
"Transformers are a type of deep learning architecture."
]
# Embed the query
query_response = zclient.models.embed(
model="zembed-1",
input=query,
input_type="query",
)
# Embed the documents
docs_response = zclient.models.embed(
model="zembed-1",
input=documents,
input_type="document",
)
```
```typescript TypeScript theme={null}
import ZeroEntropy from 'zeroentropy';
const zclient = new ZeroEntropy();
const query = "What is Retrieval Augmented Generation?";
const documents = [
"RAG combines retrieval with generation by conditioning the LLM on external documents.",
"Retrieval-Augmented Generation is a machine learning technique introduced by Meta AI in 2020.",
"It uses reinforcement learning to generate music sequences.",
"RAG can improve factual accuracy by grounding answers in retrieved evidence.",
"Transformers are a type of deep learning architecture."
];
// Embed the query
const queryResponse = await zclient.models.embed({
model: "zembed-1",
input: query,
input_type: "query",
});
// Embed the documents
const docsResponse = await zclient.models.embed({
model: "zembed-1",
input: documents,
input_type: "document",
});
```
#### Compute similarity
Use cosine similarity to rank documents by relevance to the query.
```python Python theme={null}
import numpy as np
query_embedding = np.array(query_response.results[0].embedding)
doc_embeddings = np.array([d.embedding for d in docs_response.results])
# Cosine similarity
similarities = doc_embeddings @ query_embedding / (
np.linalg.norm(doc_embeddings, axis=1) * np.linalg.norm(query_embedding)
)
for i in np.argsort(similarities)[::-1]:
print(f"{similarities[i]:.4f} {documents[i][:80]}")
```
```typescript TypeScript theme={null}
function cosineSimilarity(a: number[], b: number[]): number {
const dot = a.reduce((sum, ai, i) => sum + ai * b[i], 0);
const normA = Math.sqrt(a.reduce((sum, ai) => sum + ai * ai, 0));
const normB = Math.sqrt(b.reduce((sum, bi) => sum + bi * bi, 0));
return dot / (normA * normB);
}
const queryEmbedding = queryResponse.results[0].embedding as number[];
const similarities = docsResponse.results.map((d, i) => ({
score: cosineSimilarity(queryEmbedding, d.embedding as number[]),
text: documents[i],
}));
similarities.sort((a, b) => b.score - a.score);
similarities.forEach(s => console.log(`${s.score.toFixed(4)} ${s.text.slice(0, 80)}`));
```
#### Configuring embedding parameters
You can customize the embedding output with additional parameters:
* **`dimensions`**: Output dimensionality. For `zembed-1`, the available options are: 2560 (default), 1280, 640, 320, 160, 80, 40. Lower dimensions reduce storage cost at the expense of accuracy.
* **`encoding_format`**: `"float"` (default) or `"base64"`. Base64 is significantly more efficient for transfer.
* **`latency`**: `"fast"` for subsecond inference, `"slow"` for higher throughput. Omit to let the API choose automatically.
```python Python theme={null}
response = zclient.models.embed(
model="zembed-1",
input="What is RAG?",
input_type="query",
dimensions=320,
encoding_format="float",
latency="fast",
)
```
```typescript TypeScript theme={null}
const response = await zclient.models.embed({
model: "zembed-1",
input: "What is RAG?",
input_type: "query",
dimensions: 320,
encoding_format: "float",
latency: "fast",
});
```
The embedding will return a list of floats (or a base64 string) that represent the chunk of text embedded.
You can read more about available embedding models in the [Models](/models) section.
You can read more about how to pick the right parameters, such as embedding size, on [our blog](https://zeroentropy.dev/blog).
# Query
Source: https://docs.zeroentropy.dev/examples/query
Query your documents using ZeroEntropy
After uploading some documents to your collection, you can start querying them.
Below are some examples of how to do this using the ZeroEntropy SDK for Python and TypeScript.
#### Query for top k documents
In the [examples/upload](/examples/upload) section, we uploaded a document with the text "My favorite apple is the Granny Smith.".
```python Python theme={null}
from zeroentropy import ZeroEntropy
zclient = ZeroEntropy()
# Assume you have already added documents to the collection "default"
response = zclient.queries.top_documents(
collection_name="default",
query="What is the best apple?",
k=1,
)
print(response.results)
```
```typescript TypeScript theme={null}
import { ZeroEntropy } from 'zeroentropy'
const zclient = new ZeroEntropy()
// Assume you have already added documents to the collection "default"
async function queryTopDocuments() {
const response = await zclient.queries.topDocuments({
collection_name: "default",
query: "What is the best apple?",
k: 1,
})
console.log(response.results)
}
queryTopDocuments()
```
#### Query for top k pages
In the [examples/upload](/examples/upload) section, we uploaded a document with two pages of text about apples and search.
```python Python theme={null}
from zeroentropy import ZeroEntropy
zclient = ZeroEntropy()
# Assume you have already added documents to the collection "pages"
response = zclient.queries.top_pages(
collection_name="pages",
query="What is the best apple?",
k=1,
include_content=True,
)
print(response.results)
```
```typescript TypeScript theme={null}
import { ZeroEntropy } from 'zeroentropy'
const zclient = new ZeroEntropy()
// Assume you have already added documents to the collection "pages"
async function queryTopPages() {
const response = await zclient.queries.topPages({
collection_name: "pages",
query: "What is the best apple?",
k: 1,
include_content: true,
})
console.log(response.results)
}
```
#### Query for top k snippets with metadata filtering
In the [examples/upload](/examples/upload) section, we uploaded a pdf which was an arxiv paper about RAG evaluation in the "pdfs" collection.
The pdf was uploaded with the metadata `"arxiv"` and `"research"`.
```python Python theme={null}
from zeroentropy import ZeroEntropy
zclient = ZeroEntropy()
# Assume you have already added documents to the collection "pdfs"
response = zclient.queries.top_snippets(
collection_name="pdfs",
query="What is Retrieval Augmented Generation?",
k=1,
filter={
"list:tags": {
"$in": ["arxiv"]
}
},
precise_responses=True,
reranker="zerank-2", # Use our Reranker as a post-processing step
)
print(response.results)
```
```typescript TypeScript theme={null}
import { ZeroEntropy } from 'zeroentropy'
const zclient = new ZeroEntropy()
// Assume you have already added documents to the collection "pdfs"
async function queryTopSnippets() {
const response = await zclient.queries.topSnippets({
collection_name: "pdfs",
query: "What is Retrieval Augmented Generation?",
k: 1,
filter: {
"list:tags": {
"$in": ["arxiv"]
}
},
precise_responses: true,
reranker: "zerank-2", // Use our Reranker as a post-processing step
})
console.log(response.results)
}
queryTopSnippets()
```
For more information on metadata filtering, see the [metadata filtering](/metadata-filtering) section.
# Rerank
Source: https://docs.zeroentropy.dev/examples/rerank
Rerank your search results using ZeroEntropy
#### Rerank an existing query
You can also rerank results that you’ve already retrieved using the `/models/rerank` endpoint or directly through the SDKs.\
This is useful when you already have a list of candidate documents (from BM25, hybrid, or vector search) and want to reorder them by semantic relevance.
```python Python theme={null}
from zeroentropy import ZeroEntropy
zclient = ZeroEntropy()
# Example: reranking 5 retrieved snippets
query = "What is Retrieval Augmented Generation?"
documents = [
"RAG combines retrieval with generation by conditioning the LLM on external documents.",
"Retrieval-Augmented Generation is a machine learning technique introduced by Meta AI in 2020.",
"It uses reinforcement learning to generate music sequences.",
"RAG can improve factual accuracy by grounding answers in retrieved evidence.",
"Transformers are a type of deep learning architecture."
]
response = zclient.models.rerank(
model="zerank-2",
query=query,
documents=documents,
)
# Each document will include a score and be sorted by relevance
for doc in response.results:
print(doc)
```
```typescript TypeScript theme={null}
import { ZeroEntropy } from 'zeroentropy'
const zclient = new ZeroEntropy()
const query = "What is Retrieval Augmented Generation?"
const documents = [
"RAG combines retrieval with generation by conditioning the LLM on external documents.",
"Retrieval-Augmented Generation is a machine learning technique introduced by Meta AI in 2020.",
"It uses reinforcement learning to generate music sequences.",
"RAG can improve factual accuracy by grounding answers in retrieved evidence.",
"Transformers are a type of deep learning architecture."
]
async function rerankDocuments() {
const response = await zclient.models.rerank({
model: "zerank-2",
query,
documents,
})
console.log(response.results)
}
rerankDocuments()
```
The reranker will return a sorted list of documents with confidence scores indicating their semantic relevance to the query.
You can read more about available reranker models in the [Models](/models) section.
# Set Up
Source: https://docs.zeroentropy.dev/examples/setup
Setting up ZeroEntropy
## Creating a new API Key
Start by creating an API Key on the dashboard, which you can use to store your API key and track your usage.
You can then store the API Key and export it as an environment variable in your development environment.
```bash MacOS/Linux theme={null}
export ZEROENTROPY_API_KEY="your_api_key"
```
```powershell Windows theme={null}
setx ZEROENTROPY_API_KEY "your_api_key"
```
In order to use ZeroEntropy, you can simply install the official ZeroEntropy package for [Python](https://pypi.org/project/zeroentropy/) or [TypeScript / JavaScript](https://www.npmjs.com/package/zeroentropy) to get started quickly.
```python Python theme={null}
pip install zeroentropy
```
```typescript TypeScript theme={null}
npm install zeroentropy
```
# Upload
Source: https://docs.zeroentropy.dev/examples/upload
Examples of uploading documents to ZeroEntropy
After creating an API Key and exporting it, you can start uploading documents to your collections.
Below are some examples of how to do this using the ZeroEntropy SDK for Python and TypeScript.
#### Upload a text file
```python Python theme={null}
from datetime import datetime
from zeroentropy import ZeroEntropy
zclient = ZeroEntropy()
# Add a new collection
response = zclient.collections.add(collection_name="default")
# Add a document to the collection
response = zclient.documents.add(
collection_name="default",
path="docs/document.txt",
content={
"type": "text",
"text": "My favorite apple is the Granny Smith.",
},
metadata={
"timestamp": datetime.now().isoformat(),
"list:tags": ["tag 1", "tag 2"],
}
)
print(response.message)
```
```typescript TypeScript theme={null}
import { ZeroEntropy } from 'zeroentropy';
const zclient = new ZeroEntropy();
// Add a document to a new collection
async function addDocument() {
try {
// Add a new collection
await zclient.collections.add({
collection_name: "default",
});
console.log("Collection 'default' created successfully.");
// Add a document to the collection
const response = await zclient.documents.add({
collection_name: "default",
path: "docs/document.txt",
content: {
type: "text",
text: "My favorite apple is the Granny Smith.",
},
metadata: {
timestamp: new Date().toISOString(),
"list:tags": ["tag 1", "tag 2"],
}
});
console.log(response.message);
} catch (error) {
console.error("Error:", error);
}
}
addDocument();
```
#### Upload a PDF file
```python Python theme={null}
import requests
import base64
from datetime import datetime
from zeroentropy import ZeroEntropy
zclient = ZeroEntropy()
# Create new collection
response = zclient.collections.add(
collection_name="pdfs"
)
document = requests.get(
"https://arxiv.org/pdf/2408.10343.pdf"
)
# Convert to base64
base64_content = base64.b64encode(document.content).decode('utf-8')
response = zclient.documents.add(
collection_name="pdfs",
path="docs/document.pdf",
content={
"type": "auto",
"base64_data": base64_content,
},
metadata={
"timestamp": datetime.now().isoformat(),
"list:tags": ["arxiv", "research"],
}
)
print(response.message)
```
```typescript TypeScript theme={null}
import axios from 'axios';
import { ZeroEntropy } from 'zeroentropy';
const zclient = new ZeroEntropy();
// Add a document to a new collection
async function addPdf() {
try {
// Add a new collection
await zclient.collections.add({
collection_name: "pdfs",
});
console.log("Collection 'pdfs' created successfully.");
// Fetch the document
const documentResponse = await axios.get('https://arxiv.org/pdf/2408.10343.pdf', {
responseType: 'arraybuffer'
});
// Convert document to Base64
const base64Content = Buffer.from(documentResponse.data).toString('base64');
// Add a document to the collection
const response = await zclient.documents.add({
collection_name: "pdfs",
path: "docs/document.pdf",
content: {
type: "auto",
base64_data: base64Content,
},
metadata: {
timestamp: new Date().toISOString(),
"list:tags": ["arxiv", "research"],
}
});
console.log(response.message);
} catch (error) {
console.error("Error:", error);
}
}
addPdf();
```
#### Upload documents with pages
```python Python theme={null}
from zeroentropy import ZeroEntropy
zclient = ZeroEntropy()
response = zclient.collections.add(collection_name="pages")
# Upload text with pages, for TopK pages queries
response = zclient.documents.add(
collection_name="pages",
path="docs/document_pages.txt",
content={
"type": "text-pages",
"pages": [
"page 1 content: My favorite apple is the Granny Smith.",
"page 2 content: Search is a fun problem to work on.",
],
},
)
print(response.message)
```
```typescript TypeScript theme={null}
import { ZeroEntropy } from 'zeroentropy'
const zclient = new ZeroEntropy()
// Add a document with pages to a new collection
async function addTextPages() {
try {
// Add a new collection
await zclient.collections.add({
collection_name: "pages",
});
console.log("Collection 'pages' created successfully.");
// Upload text with pages for TopK page queries
const response = await zclient.documents.add({
collection_name: "pages",
path: "docs/document_pages.txt",
content: {
type: "text-pages",
pages: [
"page 1 content",
"page 2 content",
],
},
});
console.log(response.message);
} catch (error) {
console.error("Error:", error);
}
}
addTextPages();
```
# Introduction
Source: https://docs.zeroentropy.dev/introduction
Welcome to the ZeroEntropy documentation.
## Introduction
ZeroEntropy develops state-of-the-art AI models for information retrieval — including **rerankers**, **embedders**, and **end-to-end retrieval pipelines**.
Our technology powers intelligent search systems that can index, search, and retrieve documents with exceptional precision.
## What is Agentic Retrieval?
**Agentic Retrieval** refers to a retrieval system that actively determines the optimal strategy to find information based on the context of a query.
Unlike traditional systems, it **mimics human reasoning** — selecting and combining retrieval techniques dynamically, and improving through feedback and learning over time.
## Key Features
* **Model Access:** Combine our cutting-edge `zerank-2` reranker and `zembed-1` embeddings to supercharge the accuracy of any search pipeline.
* **Document Indexing:** Seamlessly add and manage documents in a fast, secure, and scalable environment.
* **Advanced Querying:** Retrieve the most relevant documents, pages, or snippets with fine-grained control.
* **Security:** All API calls are encrypted, with optional on-premises deployment within your own VPC.
## Getting Started
To begin using the ZeroEntropy API:
1. **Get Your API Key** — Visit the [dashboard](https://dashboard.zeroentropy.dev) to obtain your API key.
2. **Set Up Your Environment** — Follow our [Quickstart Guide](/quickstart) to send your first query.
3. **Explore the API** — See the [API Reference](/api-reference/) for detailed endpoints and usage examples.
Need help? Reach out at [founders@zeroentropy.dev](mailto:founders@zeroentropy.dev) or join our [Slack community](https://go.zeroentropy.dev/slack) for direct support.
# Metadata Filtering
Source: https://docs.zeroentropy.dev/metadata-filtering
Filtering queries by document metadata
Often, you will want to attach metadata information to each document. Then, when sending queries, you may want to filter documents based on that metadata information. ZeroEntropy supports query-time metadata filtering via a comprehensive metadata query language.
## Metadata Specification
Document metadata must be of the type `dict[str, str | list[str]]`. For example, you could attach the following JSON object as document metadata,
```python theme={null}
{
"timestamp": "2024-12-12T20:00:45",
"author": "Nicholas Pipitone",
"language": "en",
"list:tags": ["Artificial Intelligence", "Technology", "Documentation"],
"list:write-permissions": ["admin", "author"],
"list:read-permissions": ["all"]
}
```
Note that attribute names must be alphanumeric (hyphens and underscores are also allowed). And, when an attribute is a list of strings, it must be prefixed with `list:`.
## Metadata Filtering
#### Basic Usage
In order to filter, you can use the operators `$eq`, `$ne`, `$gt`, `$gte`, `$lt`, `$lte`. These operators represent "equals", "not equals", "greater than", "greater than or equal to", "less than", and "less than or equal to", respectively. Here is an example of a few filters,
```python theme={null}
# For "language" == "en"
results = await zclient.queries.top_snippets(
collection_name="default",
query="I'm looking for documents about apples",
k=5,
filter={
"language": {
"$eq": "en"
}
},
)
# For "timestamp" > (1 day ago)
from datetime import datetime, timedelta
results = await zclient.queries.top_snippets(
collection_name="default",
query="I'm looking for documents about apples",
k=5,
filter={
"timestamp": {
"$gt": (datetime.now() - timedelta(days=1)).isoformat()
}
},
)
```
You can check whether or not a string attribute matches a list using `$in` and `$nin` for "in" and "not in" operations.
```python theme={null}
# For "language" in ["en", "es"]
results = await zclient.queries.top_snippets(
collection_name="default",
query="I'm looking for documents about apples",
k=5,
filter={
# `true` if "language" is set to either "en" OR "es"
"language": {
"$in": ["en", "es"]
}
},
)
```
If you provide a filter query and a document does not contain that attribute, then that attribute will be considered `null` for that document. In other words, `$eq`, `$gt`, `$gte`, `$lt`, `$lte` will all always evaluate to `false`. But, `$neq` will always evaluate to `true`, because `null` is not equal to any string. Be careful to not have any typos in your query attribute name, or you may not match any documents!
#### Lists of Strings
When using a "list of strings" metadata attribute, the attribute name must start with `list:`. For example, you can set `list:tags` to be a list of tags for a blog article document.
List of strings can only be used with the operators `$in`, `$nin`. These operators will execute "set intersection". Meaning, `a in b` is `true` if and only if `a` and `b` have **at least** one element in common. Here are a few examples,
```python theme={null}
# Upload two blog posts, one about tech, and the other about food.
await zclient.documents.add(
collection_name="default",
path="ai_blog.txt",
content={
"type": "text",
"text": "This is a blog post about artificial intelligence."
},
metadata={
"list:tags": ["blog", "tech"]
}
)
await zclient.documents.add(
collection_name="default",
path="food_blog.txt",
content={
"type": "text",
"text": "This is a blog post about food."
},
metadata={
"list:tags": ["blog", "food"]
}
)
await zclient.documents.add(
collection_name="default",
path="empty.txt",
content={
"type": "text",
"text": "This is an empty file with no tags."
},
metadata={} # Omission is equivalent to `list:tags` being an empty array
)
# This will only match `ai_blog.txt`
results = await zclient.queries.top_snippets(
collection_name="default",
query="I'm looking for documents about apples",
k=5,
filter={
# Only `true` if "list:tags" contains EITHER "tech" OR "finance" (or both)
"list:tags": {
"$in": ["tech", "finance"]
}
},
)
# This will only match `empty.txt`
results = await zclient.queries.top_snippets(
collection_name="default",
query="I'm looking for documents about apples",
k=5,
filter={
# Only `true` if "list:tags" contains NEITHER "tech" NOR "food"
"list:tags": {
"$nin": ["tech", "food"]
}
},
)
```
When sending query filters, do not forget that "list of string" attributes must start with `list:`! If you accidentally query for `tags`, then you will not find any results. You must query for `list:tags`.
#### Boolean Operators
If you want to combine filters, you can use `$and`, `$or` as boolean operators. These boolean operators will take in an array of filters. They can also be used recursively to create a tree of boolean logic.
```python theme={null}
# For "language" == "en" && "timestamp" > (1 day ago)
results = await zclient.queries.top_snippets(
collection_name="default",
query="I'm looking for documents about apples",
k=5,
filter={
"$and": [
{
"language": {
"$eq": "en"
}
},
{
"timestamp": {
"$gt": (datetime.now() - timedelta(days=1)).isoformat()
}
}
]
},
)
# For
# "author" == "Nicholas Pipitone"
# or ("language" == "en" and "timestamp" > (1 day ago))
from datetime import datetime, timedelta
results = await zclient.queries.top_snippets(
collection_name="default",
query="I'm looking for documents about apples",
k=5,
filter={
"$or": [
{
"author": {
"$eq": "Nicholas Pipitone"
}
},
{
"$and": [
{
"language": {
"$eq": "en"
}
},
{
"timestamp": {
"$gt": (datetime.now() - timedelta(days=1)).isoformat()
}
}
]
}
]
},
)
```
# Models
Source: https://docs.zeroentropy.dev/models
Using Embedding and Reranking Models Developed by ZeroEntropy
## Embeddings
Embedding models are neural networks that encode information into representative vectors that can be used for tasks like semantic retrieval, clustering, and recommender systems.
#### zembed-1
`zembed-1` is ZeroEntropy's flagship, state-of-the-art, open-weight, multilingual embedding model.
You can read more about its performance in [this blog post](https://www.zeroentropy.dev/articles/introducing-zembed-1-the-worlds-best-multilingual-text-embedding-model).
`zembed-1` is the default embedding model used in [zsearch](/zsearch), ZeroEntropy's search engine.
There are multiple ways to use `zembed-1`:
* Calling the [models/embed](/api-reference/models/embed) API endpoint, which is available via the Python and Node SDKs.
* Downloading the weights from [HuggingFace](https://huggingface.co/zeroentropy/zembed-1) and self-hosting the model.
* On the AWS Marketplace through SageMaker.
```python Python theme={null}
from zeroentropy import ZeroEntropy
zclient = ZeroEntropy()
response = zclient.models.embed(
model="zembed-1",
input_type="query", # "query" or "document"
input="What is retrieval augmented generation?", # string or list[str]
dimensions=2560, # 2560 (default), 1280, 640, 320, 160, 80, or 40
encoding_format="float", # "float" or "base64"
latency="fast", # "fast" or "slow"; omit for auto
)
```
```javascript Javascript theme={null}
import ZeroEntropy from 'zeroentropy';
const zclient = new ZeroEntropy();
const response = await zclient.models.embed({
model: "zembed-1",
input_type: "query", // "query" or "document"
input: "What is retrieval augmented generation?", // string or string[]
dimensions: 2560, // optional: 2560, 1280, 640, 320, 160, 80, or 40
encoding_format: "float", // "float" or "base64"
latency: "fast", // "fast" or "slow"; omit for auto
});
```
There are three parameters you can configure when using `zembed-1`:
* **Latency mode:** Control the trade-off between latency and throughput based on your use case.
* **Embedding type:** Specify whether you are embedding a query or a passage to take advantage of asymmetrical retrieval.
* **Embedding size:** Choose an output dimension from the available options: 2560 (default), 1280, 640, 320, 160, 80, or 40.
Higher-dimension embeddings yield greater accuracy at the cost of increased storage.
To read more about how to think about these trade-offs, you can refer [to our blog](https://zeroentropy.dev/blog).
For guidance on how to inference the model, you can check out the [examples](/examples/embed), as well as [our cookbook here](https://github.com/zeroentropy-ai/zcookbook/tree/main/guides/zembed_quickstart).
## Rerankers
Rerankers are [cross-encoder](https://sbert.net/docs/package_reference/cross_encoder/cross_encoder.html) neural networks that can boost the accuracy of any search system. You can read more about what rerankers are and when they are most useful in [this blog post](https://www.zeroentropy.dev/blog/what-is-a-reranker-and-do-i-need-one).
#### zerank-2 and zerank-1
`zerank-2` is our flagship state-of-the-art reranker, you can read more about its performance at [this blog post](https://www.zeroentropy.dev/articles/zerank-2-advanced-instruction-following-multilingual-reranker).
`zerank-1` and `zerank-1-small` are our [first generation](https://www.zeroentropy.dev/blog/announcing-zeroentropys-first-reranker) of SOTA rerankers.
All our rerankers can be called using:
* Using the [models/rerank](/api-reference/models/rerank) API endpoint, which is callable via the Python and Node SDKs.
* By passing in the `reranker` query parameter into [top-snippets](/api-reference/queries/top-snippets)
* Downloading from our [HuggingFace](https://huggingface.co/zeroentropy/models) and self-hosting the models.
* On the [AWS Marketplace](https://aws.amazon.com/marketplace/pp/prodview-o7avk66msiukc) through SageMaker.
* `zerank-1-small` is also available on [Baseten](https://www.baseten.co/library/zerank-1-small/).
We've open-sourced `zerank-1-small` under an Apache 2.0 license, and it is also available through [HuggingFace](https://huggingface.co/zeroentropy/zerank-1-small) and Baseten.
Our flagship model `zerank-2` can be downloaded from [HuggingFace](https://huggingface.co/zeroentropy/zerank-2) under a non-commercial license. To use in a commercial setting, contact us at [founders@zeroentropy.dev](mailto:founders@zeroentropy.dev) and we'll get you a license ASAP!
#### Using the ZeroEntropy SDK
```python Python theme={null}
# Create an API Key at https://dashboard.zeroentropy.dev
# pip install zeroentropy
from zeroentropy import ZeroEntropy
# Initialize the ZeroEntropy client (reads ZEROENTROPY_API_KEY from env)
zclient = ZeroEntropy()
response = zclient.models.rerank(
model="zerank-2",
query="What is 2+2?",
documents=[
"4",
"The answer is definitely 1 million.",
],
)
print(response.model_dump_json(indent=4))
```
```javascript Javascript theme={null}
// Create an API Key at https://dashboard.zeroentropy.dev
// npm install zeroentropy
import ZeroEntropy from 'zeroentropy'; // or: const { ZeroEntropy } = require('zeroentropy');
// Initialize the ZeroEntropy client (reads ZEROENTROPY_API_KEY from env)
const zclient = new ZeroEntropy();
const response = await zclient.models.rerank({
model: 'zerank-2',
query: 'What is 2+2?',
documents: [
'4',
'The answer is definitely 1 million.',
]
});
console.log(JSON.stringify(response, null, 2));
```
#### Using [top-snippets](/api-reference/queries/top-snippets)
When querying for [/top-snippets](/api-reference/queries/top-snippets) from a ZeroEntropy collection, you can easily apply the reranker and get a significantly better ranking. Scores from a reranker are deterministic and more readily interpretable, which is another benefit over just hybrid search.
```python Python theme={null}
from zeroentropy import ZeroEntropy
zclient = ZeroEntropy()
# Assuming you have already added documents to the collection "pdfs"
response = zclient.queries.top_snippets(
collection_name="pdfs",
query="What is Retrieval Augmented Generation?",
k=10,
reranker="zerank-2", # All K results will be reranked using our reranker.
)
print(response.results)
```
```javascript Javascript theme={null}
import { ZeroEntropy } from 'zeroentropy'
const zclient = new ZeroEntropy()
// Assuming you have already added documents to the collection "pdfs"
const response = await zclient.queries.topSnippets({
collection_name: "pdfs",
query: "What is Retrieval Augmented Generation?",
k: 10,
reranker: "zerank-2", // All K results will be reranked using our reranker.
})
console.log(response.results)
```
## Ratelimiting and Pricing
#### Rate limits
Rate limits apply to both our [embed](/api-reference/models/embed) and [rerank](/api-reference/models/rerank) endpoints, and they depend on your subscription tier. On the free tier, each API key is limited to `500,000 UTF-8 bytes per minute` on the default latency mode `"fast"`. Higher tiers start at \$50/mo and unlock significantly higher throughput — upgrade on your [billing page](https://dashboard.zeroentropy.dev/billing).
| Tier | Latency Mode | TPM | RPM |
| :------------- | :----------: | -------------------------: | ---: |
| **Free** | `"fast"` | 500,000 UTF-8 bytes | 100 |
| **Free** | `"slow"` | 5,000,000 UTF-8 bytes | 100 |
| **Starter** | `"fast"` | 2,000,000 UTF-8 bytes | 500 |
| **Starter** | `"slow"` | 20,000,000 UTF-8 bytes | 500 |
| **Teams** | `"fast"` | 25,000,000 UTF-8 bytes | 1000 |
| **Teams** | `"slow"` | 250,000,000 UTF-8 bytes | 1000 |
| **Enterprise** | `"fast"` | 150,000,000 UTF-8 bytes | 2000 |
| **Enterprise** | `"slow"` | 15,000,000,000 UTF-8 bytes | 2000 |
All tiers share a maximum per-request payload size of `5,000,000 bytes`.
A reranker request consumes bytes based on the number of documents and the total length of the input. The formula is:
```
Total bytes = 150
+ len(query.encode("utf-8"))
+ len(document.encode("utf-8"))
```
This is calculated per document, so the query is counted once for each document you pass in.
For example, if you send a request with 10 documents, the total usage is:
```
10 × len(query.encode("utf-8"))
+ ∑ len(document_i.encode("utf-8")) for i in 1…10
```
An embedding request consumes bytes based on the total length of the input being embedded, whether it is a document or a query.
If you exceed your `"fast"` rate limit:
* Your requests will still be served.
* However, they will be throttled to `"slow"` mode, which has significantly higher throughput but much higher latency.
* You may experience several seconds of latency per request.
* In this degraded mode, throughput scales with your tier's `"slow"` mode limit (see the table above).
To get a `429` error instead of falling back, set `latency="fast"` explicitly in your request.
#### Pricing
Our usage-based pricing is simple and transparent, for both our model endpoints and the Search API.
| Model | Price per 1000 Tokens | Price per 1M Tokens |
| :----------------- | :-------------------: | ------------------: |
| **zembed-1** | \$0.000050 | \$0.050 |
| **zerank-2** | \$0.000025 | \$0.025 |
| **zerank-1** | \$0.000025 | \$0.025 |
| **zerank-1-small** | \$0.000025 | \$0.025 |
| Search API Feature | Price |
| :----------------- | -------------------: |
| **OCR** | \$1.75 / 1,000 pages |
| **Indexing** | \$0.50 / MB |
| **Storage** | \$0.10 / MB / month |
| **Queries** | \$1.50 / TB queried |
## Deployment Options
All of our models are open-weight and available through different deployment options. For help choosing the right option for your use case, [reach out to our team](mailto:contact@zeroentropy.dev).
The fastest way to get started. Fully managed infrastructure with no deployment overhead.
* **SDKs**: [Python](https://github.com/zeroentropy-ai/zeroentropy-python) | [Node](https://github.com/zeroentropy-ai/zeroentropy-node)
* **Authentication**: API key via dashboard. All requests authenticated over TLS. SSO SAML through Okta available for enterprise customers.
* **Regions available**: US-East, US-West, Europe.
* **Rate limits**: You can refer to the [rate limits](/models#rate-limits) shown above.
* **Latency**: We benchmarked our models latency in this [open-source repository](https://github.com/zeroentropy-ai/benchmark-api).
* **Status Page**: Visit our [Status Page](https://status.zeroentropy.dev) to monitor Uptime.
```python Python theme={null}
pip install zeroentropy
```
```javascript JavaScript theme={null}
npm install zeroentropy
```
**Your data is never used for model training.** \
MSA, DPA, and BAA available on request. \
See our [Trust Portal](https://trust.delve.co/zeroentropy) for SOC 2 Type II, Pentest, and other compliance documentation.
Deploy on [AWS SageMaker](https://aws.amazon.com/marketplace/seller-profile?id=seller-nurj4uavxb4z2) through Public or Private offers. Use your existing AWS committed spend and EDPs.
**Recommended instances**:
| Model | Instance | Throughput | Latency (p50) |
| ------------- | --------------- | --------------- | ------------- |
| zerank-2 | ml.g6e.48xlarge | 0.5 QPS | \~1500ms |
| zerank-2 | p5.48xlarge | 5 QPS | \~700ms |
| zerank-2-nano | ml.g6e.48xlarge | 0.5 queries/sec | \~500ms |
| zerank-2-nano | ml.g6e.48xlarge | 5 queries/sec | \~500ms |
**Private offers** are available for volume pricing, custom SLAs, and BAAs. [Contact us](mailto:contact@zeroentropy.dev) to scope your deployment or get help selecting the right instance configuration.
Models run entirely within your AWS account. No data leaves your VPC. MSA, DPA, and BAA available on request. \
See our [Trust Portal](https://trust.delve.co/zeroentropy) for compliance documentation.
Deploy through Public or Private offers on the [Azure Marketplace](https://marketplace.microsoft.com/en-us/product/zeroentropy.zeroentropy-zerank-2?tab=Overview). Use your existing Azure committed spend (MACC).
**Private offers** are available for volume pricing, custom SLAs, and BAAs. [Contact us](mailto:contact@zeroentropy.dev) to discuss your requirements.
Models run entirely within your Azure tenant. No data leaves your environment. All data encrypted in transit and at rest. MSA, DPA, and BAA available on request. See our [Trust Portal](\[link]) for compliance documentation.
Our models are open-weight on [HuggingFace](https://huggingface.co/zeroentropy). Obtain a commercial license from us and run them on your own infrastructure.
**Licensing**: Commercial use requires a license. [Reach out](mailto:contact@zeroentropy.dev) for terms, pricing, and enterprise support/SLAs for self-hosted deployments.
Full control over your data and infrastructure. \
MSA, DPA, and BAA available on request. See our [Trust Portal](https://trust.delve.co/zeroentropy) for compliance documentation.
# Quickstart
Source: https://docs.zeroentropy.dev/quickstart
Getting Started using the ZeroEntropy API
### Create and Export your API Key
Start by creating an API Key on the dashboard, which you can use to store your API key and track your usage.
You can then store the API Key and export it as an environment variable in your development environment.
```bash MacOS/Linux theme={null}
export ZEROENTROPY_API_KEY="your_api_key"
```
```powershell Windows theme={null}
setx ZEROENTROPY_API_KEY "your_api_key"
```
New accounts start on the free tier with rate limits sufficient for prototyping. Higher tiers start at \$50/mo — upgrade on the [billing page](https://dashboard.zeroentropy.dev/billing) or see [rate limits](/models#rate-limits) for details.
### Getting Started
After checking out the [Core Concepts](/core-concepts), you'll be ready to use the API. We offer many different ways to access our API:
1. Using our official SDKs for [Python](https://pypi.org/project/zeroentropy/) and [TypeScript / JavaScript](https://www.npmjs.com/package/zeroentropy) as shown below.
```python Python theme={null}
pip install zeroentropy
```
```typescript TypeScript theme={null}
npm install zeroentropy
```
2. Using our [interactive API Reference](/api-reference/). Simply drop your API Key into the "Authorization" button and use our interactive API to try it out.
* You can copy example queries using cURL, Python, Javascript into your development environment.
3. Using [https://go.zeroentropy.dev/openapi.json](https://go.zeroentropy.dev/openapi.json) to access the API through an API platform such as Thunder Client or Postman.
* For example, in Postman, go to File -> Import, and then paste `https://go.zeroentropy.dev/openapi.json` into the prompt. You'll have to set the `bearerToken` variable to your API Key.
4. Using our [SwaggerUI](https://api.zeroentropy.dev/v1/docs) interface, simply drop your API Key into "Authorize" button in the top-right corner.
If you need support for EU-based datacenters for compliance, you can create an EU-based API Key using our [EU Dashboard](https://eu-dashboard.zeroentropy.dev/). Note the subdomain of `eu-dashboard.` rather than the US-based `dashboard.`. Similarly, you will want to set the `base_url` to
`https://eu-api.zeroentropy.dev/v1`.
Using the EU Dashboard and EU API Endpoints will ensure that all data is fully processed and stored within the EU.
* When using our [interactive API Reference](/api-reference/), you can select
`https://eu-api.zeroentropy.dev/v1` in the dropdown menu at the top of each API request.
* We also have a [SwaggerUI](https://eu-api.zeroentropy.dev/v1/docs) interface for our EU API, using the `eu-api.` subdomain.
### Send your First Query
```python Python theme={null}
from zeroentropy import ZeroEntropy
import time
zclient = ZeroEntropy()
# Create a collection
collection = zclient.collections.add(collection_name="default")
# Add a text file to the collection
document = zclient.documents.add(
collection_name="default",
path="docs/document.txt",
content={
"type": "text",
"text": "My favorite apple is the Granny Smith.",
},
)
# Wait until the document is indexed
while True:
status = zclient.documents.get_info(collection_name="default", path="docs/document.txt")
if status.document.index_status == "indexed":
print("Document is indexed.")
break
time.sleep(1)
# Query the collection
response = zclient.queries.top_documents(
collection_name="default",
query="What is the best apple?",
k=1,
)
print(response.results)
```
```typescript TypeScript theme={null}
import { ZeroEntropy } from 'zeroentropy'
const zclient = new ZeroEntropy()
// Create a collection
const collection = await zclient.collections.add({
collection_name: "default",
})
// Add a text file to the collection
const document = await zclient.documents.add({
collection_name: "default",
path: "docs/document.txt",
content: {
type: "text",
text: "My favorite apple is the Granny Smith.",
},
})
// Wait until the document is indexed
let indexed = false;
while (!indexed) {
const status = await zclient.documents.getInfo({
collection_name: "default",
path: "docs/document.txt",
});
if (status.document.index_status === "indexed") {
console.log("Document is indexed.");
indexed = true;
} else {
await new Promise(resolve => setTimeout(resolve, 1000));
}
}
// Query the collection
const response = await zclient.queries.topDocuments({
collection_name: "default",
query: "What is the best apple?",
k: 1,
})
console.log(response.results)
```
### Prompt your LLM to use ZeroEntropy
Click this card to copy a prompt for using ZeroEntropy with your LLM. You can save it as a text file and reference it in tools like Cursor.
```text copy theme={null}
ZeroEntropy SDK Helper
### Description:
ZeroEntropy is a state-of-the-art retrieval API for documents, pages, snippets and reranking.
It provides low-latency, high-accuracy search over your private corpus via a simple Python SDK.
ZeroEntropy can be installed using:
• Python: pip install zeroentropy
• Node.js: npm install zeroentropy
### Client Usage
from zeroentropy import ZeroEntropy
client = ZeroEntropy(api_key="your_api_key")
Auth & Configuration:
• ENV VARS read by SDK:
ZEROENTROPY_API_KEY
Missing key triggers authentication error on instantiation.
Instantiate:
from dotenv import load_dotenv
load_dotenv()
from zeroentropy import AsyncZeroEntropy, ConflictError, HTTPStatusError
zclient = AsyncZeroEntropy() # picks up ENV VARS
### SDK Structure:
All methods are async, grouped under:
zclient.collections
zclient.documents
zclient.status
zclient.queries
zclient.models
Each method returns structured responses defined by pydantic.BaseModel.
### Collections
• client.collections.add(collection_name: str) -> None
Always specify a collection name using client.collections.add(collection_name="my_collection")
If the collection already exists, it will be throw an error, so you need to check if the collection exists first.
• client.collections.get_list() -> List[str]
• client.collections.delete(collection_name: str) -> None
### Documents
• client.documents.add(collection_name: str, path: str, content, metadata: dict = None, overwrite: bool = False) -> None
The add method already handles parsing for PDFs etc. The content dict can take the following formats:
content={"type":"auto", "base64_data":"my_document.pdf"} for a PDF, content={"type":"text", "text":"my_document.pdf"} for a text file, and content={"type":"text-pages", "pages":[ "page 1 content", "page 2 content"]} for pages of text.
If the document already exists, it will be throw an error, so you need to check if the document exists first.
• client.documents.get_info(collection_name: str, path: str, include_content: bool = False) -> DocumentResponse
• client.documents.get_info_list(collection_name: str, limit: int = 1024, path_gt: Optional[str] = None) -> List[DocumentGetInfoListResponse]
• client.documents.update(collection_name: str, path: str, metadata: Optional[dict]) -> UpdateDocumentResponse
• client.documents.delete(collection_name: str, path: Union[str, List[str]]) -> DocumentDeleteResponse
### Queries
• client.queries.top_documents(collection_name: str, query: str, k: int, filter: Optional[dict] = None, include_metadata: bool = False, latency_mode: str = "low") -> List[DocumentRetrievalResponse]
• client.queries.top_pages(collection_name: str, query: str, k: int, filter: Optional[dict] = None, include_content: bool = False, latency_mode: str = "low") -> List[PageRetrievalResponse]
• client.queries.top_snippets(collection_name: str, query: str, k: int, filter: Optional[dict] = None, precise_responses: bool = False) -> List[SnippetResponse]
### Status
• client.status.get_status(collection_name: Optional[str] = None) -> StatusGetStatusResponse
### Models
• client.models.embed(input: Union[str, List[str]], input_type: "query" | "document", model: str, dimensions: Optional[int] = None, encoding_format: "float" | "base64" = "float", latency: Optional["fast" | "slow"] = None) -> ModelEmbedResponse
• client.models.rerank(documents: List[str], model: str, query: str, top_n: Optional[int] = None) -> ModelRerankResponse
Common Patterns:
1 Collections
try:
await zclient.collections.add(collection_name="my_col")
except ConflictError:
pass
names = (await zclient.collections.get_list()).collection_names
await zclient.collections.delete(collection_name="my_col")
2 Documents
# Add text
await zclient.documents.add(
collection_name="col",
path="doc.txt",
content={"type":"text","text":text},
metadata={"source":"notes"},
)
# Add PDF via OCR
b64 = base64.b64encode(open(path,"rb").read()).decode()
await zclient.documents.add(
collection_name="col",
path="doc.pdf",
content={"type":"auto","base64_data":b64},
metadata={"type":"pdf"},
)
# Add CSV lines
for i,line in enumerate(open(path).read().splitlines()):
await zclient.documents.add(
collection_name="col",
path=f"{path}_{i}",
content={"type":"text","text":line},
metadata={"type":"csv"},
)
# Delete
await zclient.documents.delete(collection_name="col", path="doc.txt")
# Get info (with optional content)
info = await zclient.documents.get_info(
collection_name="col",
path="doc.txt",
include_content=True
)
3 Update & Pagination
# Update metadata or force re-index
await zclient.documents.update(
collection_name="col",
path="doc.txt",
metadata={"reviewed":"yes"},
)
# List documents with pagination
resp = await zclient.documents.get_info_list(
collection_name="col",
limit=100,
path_gt="doc_009.txt",
)
for doc in resp.documents:
print(doc.path, doc.index_status)
# Per-page info
page = await zclient.documents.get_page_info(
collection_name="col",
path="doc.pdf",
page_index=2,
include_content=True,
)
4 Pure Parsing (OCR helper)
pages = await zclient.documents.parse(
base64_data=b64
)
# returns list of page strings without indexing
5 Status (overall or per-collection)
status_all = await zclient.status.get_status()
status_col = await zclient.status.get_status(collection_name="col")
6 Queries
# Top K documents (k≤2048), with filter, reranker, latency_mode
docs = await zclient.queries.top_documents(
collection_name="col",
query="find insight",
k=5,
filter={"type":{"$ne":"csv"}},
include_metadata=True,
reranker="zerank-2",
latency_mode="low",
)
# Top K pages (k≤1024), include_content, latency_mode
pages = await zclient.queries.top_pages(
collection_name="col",
query="overview",
k=3,
include_content=True,
latency_mode="high",
)
# Top K snippets (k≤128), precise or coarse
snips = await zclient.queries.top_snippets(
collection_name="col",
query="key method",
k=5,
precise_responses=True,
include_document_metadata=True,
reranker="zerank-2",
)
### Expected Response Models
All responses return structured BaseModel objects as follows:
1. DocumentGetInfoResponse
Used in get_info()
python
class DocumentGetInfoResponse(BaseModel):
document: Document
class Document(BaseModel):
id: str # UUID of the document
collection_name: str
path: str
file_url: str # URL to download raw document
size: int # Raw document size in bytes
metadata: Dict[str, Union[str, List[str]]] # Metadata key-value pairs
index_status: str # Enum: "not_parsed", "parsing", "not_indexed", "indexing", "indexed", "parsing_failed", "indexing_failed"
num_pages: Optional[int] = None # Can be null
content: Optional[str] = None # Null unless `include_content=True`
2. DocumentUpdateResponse
Used in update()
python
class DocumentUpdateResponse(BaseModel):
message: Optional[str] = None # "Success!"
3. DocumentRetrievalResponse
Used in top_documents()
python
class DocumentRetrievalResponse(BaseModel):
results: List[Response]
class Response(BaseModel):
path: str
metadata: Optional[Dict[str, Union[str, List[str]]]] = None # Null if `include_metadata=False`
score: float # Relevancy score
4. PageRetrievalResponse
Used in top_pages()
python
class PageRetrievalResponse(BaseModel):
results: List[Response]
class Response(BaseModel):
path: str # Document path
page_index: int # 0-indexed page number
score: float # Relevancy score
content: Optional[str] = None # Null if `include_content=False`
5. SnippetResponse
Used in top_snippets()
python
class SnippetResponse(BaseModel):
results: List[Response]
class Response(BaseModel):
path: str
start_index: int # Start character index of snippet
end_index: int # End character index of snippet
page_span: List[int] # (start_page, end_page) index range
content: Optional[str] = None # Snippet text
score: float # Relevancy score
6. StatusGetStatusResponse
Used in status.get_status()
python
class StatusGetStatusResponse(BaseModel):
num_documents: int # Total document count
num_parsing_documents: int # Documents still being parsed
num_indexing_documents: int # Documents currently being indexed
num_indexed_documents: int # Successfully indexed documents
num_failed_documents: int # Documents that failed
num_indexed_bytes: int # Total UTF-8 encoded bytes indexed
7. ParseDocumentResponse
Used in parse_document()
python
class ParseDocumentResponse(BaseModel):
pages: List[str] # List of extracted page contents
8. ModelEmbedResponse
Used in embed()
python
class ModelEmbedResponse(BaseModel):
results: List[Result]
usage: Usage
class Result(BaseModel):
embedding: Union[List[float], str] # float array or base64 string
class Usage(BaseModel):
total_bytes: int # For ratelimiting
total_tokens: int # For billing
9. ModelRerankResponse
Used in rerank()
python
class ModelRerankResponse(BaseModel):
results: List[Result]
actual_latency_mode: str # "fast" or "slow"
e2e_latency: float # Total time in seconds
inference_latency: float
total_bytes: int
total_tokens: int
class Result(BaseModel):
index: int # Index in original documents array
relevance_score: float # 0.0 to 1.0
All query endpoints support JSON filters:
### Additional Notes
- LLM should always use these BaseModels when generating SDK-based responses.
- Metadata Filtering
- Document metadata is always dict[str, str | list[str]].
- Filters support operators: $eq, $ne, $gt, $gte, $lt, $lte (for equality and range queries).
- Responses will always match these structures unless otherwise stated.
- Always include imports and load_dotenv().
- Use proper async/await structure with asyncio.run().
- Respect parameter limits and JSON‐filter syntax.
- Override base_url when targeting EU cluster.
- Wrap calls in try/except to handle ConflictError and HTTPStatusError.
### Complete Async Example:
import asyncio
from dotenv import load_dotenv
from zeroentropy import AsyncZeroEntropy, ConflictError, HTTPStatusError
import base64
load_dotenv()
zclient = AsyncZeroEntropy()
async def main():
try:
await zclient.collections.add(collection_name="my_col")
except ConflictError:
pass
text = "Hello ZeroEntropy"
await zclient.documents.add(
collection_name="my_col",
path="hello.txt",
content={"type":"text","text":text},
metadata={"lang":"en"},
)
status = await zclient.status.get_status(collection_name="my_col")
print("Indexed:", status.num_indexed_documents)
docs = await zclient.queries.top_documents(
collection_name="my_col",
query="Hello",
k=1,
include_metadata=True,
)
print(docs.results)
if __name__ == "__main__":
asyncio.run(main())
```
# Search Engine
Source: https://docs.zeroentropy.dev/zsearch
Using `zsearch`, the search engine developed by ZeroEntropy
## zsearch
`zsearch` is ZeroEntropy's end-to-end search engine, abstracting away data processing from OCR and chunking, to embedding and storing, to querying and reranking.
## Index
#### Add documents to a collection
When you add a document to a collection in `zsearch`, it goes through a fully managed ingestion pipeline:
1. **Parse:** Binary files (PDF, DOCX, PPT, images, etc.) are OCR'd and converted to text. Plain text and CSV inputs skip this step.
2. **Chunk:** The parsed text is split into chunks at multiple granularities: coarse (\~2000 chars) and fine (\~200 chars), optimized for retrieval.
3. **Embed:** Each chunk is embedded using `zembed-1`, ZeroEntropy's state-of-the-art multilingual embedding model, and stored in our vector index.
When you call [add-document](/api-reference/documents/add-document), documents are automatically added to a collection with a unique path (like a filepath). ZeroEntropy supports three content types:
* `text`: Plain text content.
* `text-pages` / `text-pages-unordered`: Pre-paginated text (array of strings). Use unordered for data like CSVs where pages are independent.
* `auto`: Binary files (PDF, DOCX, PPT, etc.) encoded as base64. ZeroEntropy handles OCR and parsing automatically.
Set `overwrite: true` to upsert (atomically replace if the path already exists).
#### Custom Chunking
If you want control over how your data is chunked, use the `text-pages` content type. Each string in the pages array becomes its own page in the index, letting you define chunk boundaries yourself. Use `text-pages-unordered` when pages are independent (e.g. CSV rows, FAQ entries).
See [examples](/examples/upload) for detailed walkthroughs of different ingestion strategies.
#### Using zembed-1 as a standalone
You can also call `zembed-1` directly via the [embed endpoint](/api-reference/models/embed) and plug it in to a vector database of your choice. See [Models](/models) for more details.
## Query
There are three granularity levels for querying your indexed data: documents, pages, and snippets. All query endpoints accept a natural language query, a collection\_name, and a `k` parameter controlling how many results to return.
All query endpoints support [metadata filtering](/metadata-filtering) via the optional `filter` parameter.
#### Top Documents
Returns the top K most relevant documents for a given query. Useful when you want to identify which documents are relevant without needing sub-document granularity.
Note that `top-documents` only returns document paths, not contents. Document contents are accessible using the [Get Document Info endpoint](/api-reference/documents/get-document-info).
Use latency\_mode: "high" if you need higher throughput at the cost of higher latency (default is "low").
```python Python theme={null}
from zeroentropy import ZeroEntropy
zclient = ZeroEntropy()
response = zclient.queries.top_documents(
collection_name="contracts",
query="What are the payment terms?",
k=5,
include_metadata=True,
)
```
```javascript Javascript theme={null}
import ZeroEntropy from 'zeroentropy';
const zclient = new ZeroEntropy();
const response = await zclient.queries.topDocuments({
collection_name: "contracts",
query: "What are the payment terms?",
k: 5,
include_metadata: true,
reranker: "zerank-2", // optional
});
for (const doc of response.results) {
console.log(`${doc.path} (score: ${doc.score})`);
}
```
#### Top Pages
Returns the top K most relevant pages. Ideal for page-level retrieval over PDFs, DOCX, or documents ingested with text-pages content type. \
Set include\_content to true to return the full text of each page. A **URL to an image** of the page will also be provided in the results.
#### Top Snippets
Returns the top K most relevant text snippets. This is the most granular query type. \
Each snippet includes the exact character range (start\_index, end\_index) and page\_span within the source document.\
You can choose between coarse snippets (averaging \~2000 characters, default) and precise snippets (averaging \~200 characters) using the precise\_responses parameter. \
Pass a reranker, such as `zerank-2` for even better ranking.
```python Python theme={null}
from zeroentropy import ZeroEntropy
zclient = ZeroEntropy()
response = zclient.queries.top_snippets(
collection_name="pdfs",
query="What is Retrieval Augmented Generation?",
k=10,
reranker="zerank-2",
precise_responses=True,
)
for snippet in response.results:
print(f"{snippet.path} [pages {snippet.page_span}] (score: {snippet.score})")
print(snippet.content)
```
```javascript Javascript theme={null}
import ZeroEntropy from 'zeroentropy';
const zclient = new ZeroEntropy();
const response = await zclient.queries.topSnippets({
collection_name: "pdfs",
query: "What is Retrieval Augmented Generation?",
k: 10,
reranker: "zerank-2",
precise_responses: true,
});
for (const snippet of response.results) {
console.log(`${snippet.path} [pages ${snippet.page_span}] (score: ${snippet.score})`);
console.log(snippet.content);
}
```
## Data Management
zsearch organizes data into collections, each containing documents. Think of collections as databases and documents as records.
#### Collections
Create, list, and delete collections. Collection names are strings up to 1024 UTF-8 bytes.
```python Python theme={null}
from zeroentropy import ZeroEntropy
zclient = ZeroEntropy()
# Create a collection
zclient.collections.add(collection_name="contracts")
# List all collections
response = zclient.collections.get_list()
print(response.collection_names)
# Delete a collection
zclient.collections.delete(collection_name="contracts")
```
```javascript Javascript theme={null}
import ZeroEntropy from 'zeroentropy';
const zclient = new ZeroEntropy();
// Create a collection
await zclient.collections.add({ collection_name: "contracts" });
// List all collections
const response = await zclient.collections.getList();
console.log(response.collection_names);
// Delete a collection
await zclient.collections.delete({ collection_name: "contracts" });
```
#### Documents
After adding a document to a collection, it takes time to parse and index.
Use the [Get Document Info](/api-reference/documents/get-document-info) endpoint to track progress. \\
Each document response includes file\_url for downloading the raw file, index\_status for tracking processing state, raw content, and num\_pages (null if still parsing or unsupported filetype). \\
You can delete one or more documents by path. We support batch deletion of up to 64 paths at once.
```python Python theme={null}
from zeroentropy import ZeroEntropy
zclient = ZeroEntropy()
# Delete a single document
zclient.documents.delete(
collection_name="contracts",
path="contracts/acme-nda.txt",
)
# Batch delete
response = zclient.documents.delete(
collection_name="contracts",
path=["old/doc1.txt", "old/doc2.txt", "old/doc3.txt"],
)
print(response.deleted_paths) # paths that were actually found and deleted
```
```javascript Javascript theme={null}
import ZeroEntropy from 'zeroentropy';
const zclient = new ZeroEntropy();
// Delete a single document
await zclient.documents.delete({
collection_name: "contracts",
path: "contracts/acme-nda.txt",
});
// Batch delete
const response = await zclient.documents.delete({
collection_name: "contracts",
path: ["old/doc1.txt", "old/doc2.txt", "old/doc3.txt"],
});
console.log(response.deleted_paths);
```
More examples can be found [here](/examples/setup).