Knowledge & RAG

The knowledge service gives agents access to tenant-owned documents and wiki pages. Documents are parsed, chunked, embedded, and searched with vector or hybrid retrieval. Wiki pages are Markdown records addressed by topic slug.

The service runs on port 8007 and is exposed through the gateway at /v1/knowledge/*.

Current Access Model

Knowledge is tenant-scoped and also has per-item visibility controls:

Visibility	Who can read
private	Owner and admins.
shared	Tenant users. If audience_tags are set, only users in those audiences.
restricted	Users listed in user_grants, users in audience_tags, owner, and admins.

Item metadata includes owner_user_id, creator_user_id, visibility_mode, audience_tags, user_grants, tags, and archive status. Admin roles can administer items; normal users can read only items allowed by policy.

Document uploads through the simple file/text endpoints create private items by default. Airbyte ingestion can set visibility_mode, audience_tags, user_grants, owner_user_id, and confirm_tenant_wide.

Document Pipeline

1. Parse the uploaded file or raw text.
2. Split text into overlapping chunks.
3. Ask the configured LLM for a short context prefix per chunk.
4. Embed context_prefix + chunk with the configured embedding model.
5. Store vectors and metadata in Qdrant.
6. Store item ownership and visibility metadata in PostgreSQL.

If contextualization fails, ingestion falls back to a generic context prefix so the document can still be indexed.

Search uses hybrid retrieval when SPARSE_EMBEDDING_ENABLED=true; otherwise it falls back to dense vector search.

Document API

Upload a File

curl http://localhost:8080/v1/knowledge/ingest \
  -F "file=@./report.pdf" \
  -F "title=Q1 Report" \
  -F "tenant_id=default"

Abbreviated response:

{
  "document_id": "a3f2c1d4-5678-9abc-def0-1234567890ab",
  "title": "Q1 Report",
  "chunks_indexed": 23
}

Ingest Text

curl http://localhost:8080/v1/knowledge/ingest/text \
  -F "title=Runbook Notes" \
  -F "content=Deployment steps..." \
  -F "tenant_id=default"

Ingest an Airbyte Batch

curl http://localhost:8080/v1/knowledge/ingest/airbyte \
  -H "Content-Type: application/json" \
  -d '{
    "tenant_id": "default",
    "visibility_mode": "restricted",
    "audience_tags": ["engineering"],
    "records": [
      {"title": "Release Notes", "content": "Full text...", "metadata": {}}
    ]
  }'

Search

curl http://localhost:8080/v1/knowledge/search \
  -H "Content-Type: application/json" \
  -d '{
    "tenant_id": "default",
    "query": "deployment rollback steps",
    "limit": 5
  }'

Response:

{
  "results": [
    {
      "content": "Rollback uses the previous image tag...",
      "source": "a3f2c1d4-5678-9abc-def0-1234567890ab",
      "score": 0.87,
      "document_title": "Deployment Runbook",
      "chunk_index": 3,
      "retrieval_rationale": "Matched chunk 3 from Deployment Runbook"
    }
  ]
}

Search emits a rag_chunks_retrieved turn event when the request is part of an agent turn.

Document and wiki list/detail responses also include governance fields such as knowledge_item_id, visibility_mode, audience_tags, user_grants, tags, owner_user_id, creator_user_id, archived_at, and last_error where applicable.

List and Manage Documents

curl "http://localhost:8080/v1/knowledge/documents?tenant_id=default"

curl "http://localhost:8080/v1/knowledge/documents/{document_id}?tenant_id=default"

curl "http://localhost:8080/v1/knowledge/documents/{document_id}/content?tenant_id=default"

curl -X POST "http://localhost:8080/v1/knowledge/documents/{document_id}/reindex?tenant_id=default"

curl -X DELETE "http://localhost:8080/v1/knowledge/documents/{document_id}?tenant_id=default"

Wiki API

Wiki pages are Markdown records in MinIO with policy metadata tracked by the knowledge metadata store.

curl http://localhost:8080/v1/knowledge/wiki \
  -H "Content-Type: application/json" \
  -d '{
    "tenant_id": "default",
    "topic": "deployment-runbook",
    "title": "Deployment Runbook",
    "summary": "Production deployment steps",
    "tags": ["ops"],
    "content": "# Deployment Runbook\n\n1. Build images.\n2. Deploy.\n3. Verify."
  }'

curl "http://localhost:8080/v1/knowledge/wiki/deployment-runbook?tenant_id=default"

curl "http://localhost:8080/v1/knowledge/wiki?tenant_id=default"

curl -X DELETE "http://localhost:8080/v1/knowledge/wiki/deployment-runbook?tenant_id=default"

Audience Administration

Audience tags are tenant-local groups used by shared and restricted items.

curl "http://localhost:8080/v1/knowledge/admin/audiences?tenant_id=default"

curl -X PUT "http://localhost:8080/v1/knowledge/admin/audiences/engineering?tenant_id=default" \
  -H "Content-Type: application/json" \
  -d '{"description": "Engineering audience"}'

curl -X PUT "http://localhost:8080/v1/knowledge/admin/audiences/engineering/members" \
  -H "Content-Type: application/json" \
  -d '{"tenant_id": "default", "audience_tag": "engineering", "user_ids": ["alice"]}'

Agent Tools

The agent runtime registers these tools when the knowledge service is available:

Tool	Purpose
knowledge_search	Search readable internal documents for the current user.
wiki_read	Read a wiki page by topic slug if the user can read it.
wiki_write	Create or update a wiki page as the current user.

Agents are instructed to prefer knowledge_search over web search for internal or company-specific questions.

Configuration

Variable	Default	Description
QDRANT_HOST	localhost	Qdrant host.
QDRANT_PORT	6333	Qdrant port.
MINIO_ENDPOINT	minio:9000	Object store endpoint for wiki and file content.
EMBEDDING_MODEL	sentence-transformers/all-MiniLM-L6-v2	Embedding model.
EMBEDDING_DIMS	384	Embedding vector size.
LLM_BASE_URL	http://inference-router:8004/v1	Contextual chunking LLM endpoint.
LLM_MODEL	default	Contextual chunking model.
SPARSE_EMBEDDING_ENABLED	true	Enable hybrid dense+sparse search.
SEARCH_SCORE_THRESHOLD	0.5	Minimum score returned from search.
KNOWLEDGE_PORT	8007	Knowledge service port.