HarborRAG

Running HarborRAG from a checkout

This page is for people working on HarborRAG itself, or running the durable Temporal-based ingestion mode. If you just want to use HarborRAG on your documents, follow the Quick Start instead - it needs no clone.

Two stages: a five-minute check that needs no Docker or credentials, then the full local stack (data services, Temporal, worker, API) for durable ingestion and the HTTP surface.

Requirements

Python 3.12 or newer
uv
Docker Engine with Compose v2 (the full stack only)
Roughly 520 MB of disk for the workspace sync, plus several GB for the full stack’s images and model cache

Explore a checkout without services

1. Clone and sync

git clone https://github.com/cbtw-apac/HarborRAG.git
cd HarborRAG
uv sync --all-packages --extra dev

2. Confirm the CLI

uv run harborrag --help

You should see five commands, grouped by purpose: init (Setup), doctor (Operations), chat (Chat), retrieve (Retrieval), and ingest (Ingestion, itself a subgroup). This works without Temporal or any data service.

harborrag doctor checks the checkout’s catalogs and the local services, so most lines fail until step 6 starts them; add --temporal after step 7 to check the workflow service.

3. Parse a document

The parser registry picks a parser family by suffix and MIME type, then the family routes to a concrete engine. Run this from the checkout root - source_uri is repository-relative:

uv run python - <<'PY'
import asyncio

from harborrag_adapters.parsers import HarborParserFactory, ParseRequest


async def main() -> None:
    registry = HarborParserFactory().create_registry()
    result = await registry.parse_request(
        ParseRequest(
            source_uri="docs/getting-started/README.md",
            filename="README.md",
            mime_type="text/markdown",
        )
    )
    print(result.parser_name, result.engine_name, len(result.text))


asyncio.run(main())
PY

Expect something like markup markdown 663 - the family, the engine, and the extracted character count.

4. Inspect the catalogs

The checked-in YAML files load through the runtime loaders, which is the fastest way to see what a fresh checkout is configured to do:

LOCAL_SOURCE_PATH=docs uv run python -c "
from harborrag_runtime.config import load_connector_catalog
c = load_connector_catalog('config/connectors.yaml')
print('enabled connectors:', c.names(enabled_only=True))
"

uv run python -c "
from harborrag_runtime.config import load_parser_catalog
c = load_parser_catalog('config/parsers.yaml')
print('enabled parsers:', c.names(enabled_only=True))
"

Three connections are enabled: harborrag-workspace (local files, no credentials), confluence-main, and jira-main. Only the first works without credentials, and it is the one the rest of this guide uses.

That is the service-free check. You have a working install and a working parser. Continue below for real ingestion, or jump to Configuration to point HarborRAG at your own sources.

Run the full stack

5. Create the `env/` folder

Every credential lives in an ignored env/ folder that you create once. One command creates all seven files from their templates, generates a random MCP bearer token, and sets mode 0600:

scripts/deployment/dev.sh bootstrap

Now fill in the values that have no safe default. The first one is mandatory - Compose will not start without it:

openssl rand -hex 32     # use the output as HARBORRAG_SECRETS_ENCRYPTION_KEY

# env/.env.database
HARBORRAG_SECRETS_ENCRYPTION_KEY=<the value above>
POSTGRES_PASSWORD=<a strong password, replacing change-me>

# env/.env.models - config/models.yaml requires all six
HARBOR_CHAT_PROVIDER=openai
HARBOR_CHAT_MODEL=gpt-4o-mini
HARBOR_CHAT_API_KEY=<your key>
HARBOR_EMBED_PROVIDER=openai
HARBOR_EMBED_MODEL=text-embedding-3-small
HARBOR_EMBED_API_KEY=<your key>

Everything else has a working local default. See Configuration for what each file holds and how to keep it outside the checkout.

Check the model catalog resolves before starting anything:

uv run --env-file env/.env.models \
  python -m harborrag_adapters.models explain config/models.yaml --family chat

References expand eagerly, so this fails immediately on a missing or empty variable - which is exactly what you want to find out now rather than mid-ingestion.

6. Start the services

scripts/deployment/dev.sh up

That starts, in order: the data stack (PostgreSQL, Qdrant, FalkorDB, Redis, MinIO), the Temporal server with its schema, namespace, and UI, the ingestion workers, and the control-plane API.

On a clean checkout up bootstraps env/ and then stops so you can review the credentials. Run it again after editing them.

The first worker build installs the selected parser and model dependencies and can take several minutes. Downloaded Hugging Face assets persist in the shared harborrag-model-cache volume and are reused after the container is replaced.

Prefer to bring things up one at a time?

Command	Starts
`dev.sh data`	data services only - creates the shared network, so it goes first
`dev.sh temporal`	Temporal server, schema, namespace, UI - never a worker
`dev.sh worker`	the ingestion worker only
`dev.sh api`	the API only
`dev.sh up --no-worker`	everything except the worker
`dev.sh down`	stop everything; `--volumes` also discards the data

Add --build to up, worker, or api after changing source, dependencies, or baked worker configuration.

If up fails with required variable HARBORRAG_SECRETS_ENCRYPTION_KEY is missing a value, go back to step 5 - Compose treats an empty value as unset.

7. Verify

HARBORRAG_TEMPORAL_TARGET=localhost:7233 \
  uv run harborrag doctor --json --temporal

doctor only contacts Temporal when --temporal is passed; without the flag the target above is read but never used, and the temporal check reports skip.

The Temporal UI is at http://localhost:8080, and dev.sh api prints the API health URL (http://127.0.0.1:8000/api/v1/health by default).

8. Ingest something

export HARBORRAG_TEMPORAL_TARGET=localhost:7233

uv run harborrag ingest start \
  --connector-id harborrag-workspace \
  --tenant tenant-1 \
  --wait

--connector-id names a connection from config/connectors.yaml, not a source type - --connector-id local fails with the list of valid IDs. --connector is an accepted alias. --wait shows the same inline progress block as ingest run; ingest watch RUN_ID attaches it to a run started elsewhere.

Save the run ID and drive it:

uv run harborrag ingest status RUN_ID --json
uv run harborrag ingest wait RUN_ID --json
uv run harborrag ingest watch RUN_ID      # inline progress until the run settles
uv run harborrag ingest pause RUN_ID
uv run harborrag ingest resume RUN_ID
uv run harborrag ingest cancel RUN_ID

Re-running the same ingestion reports documents as unchanged. That is the admission fast path, not an error - it skips parsing, chunking, and encoding. Use --force-reprocess to override it; see Ingestion modes.

9. Query it

uv run harborrag retrieve "deployment requirements" \
  --tenant tenant-1 \
  --top-k 5 \
  --include-content

retrieve runs hybrid dense + sparse search with graph observation enabled. Use --lane dense|sparse|hybrid to pin one lane and --no-graph to skip graph expansion.

Chat is retrieval-grounded and makes a real provider request, so it may incur charges:

uv run harborrag chat \
  "Explain HarborRAG in one paragraph." \
  --tenant tenant-1 \
  --json

The JSON response contains a generated session ID. Pass it back with --session on later calls to keep conversation context. See the Chat guide for the HTTP surface.

10. Connect an MCP client

scripts/deployment/mcp.sh --check

That performs a real MCP handshake and prints the eleven advertised retrieval tools without opening provider connections. For normal use, point your MCP client at scripts/deployment/mcp.sh; for a browser playground, run scripts/deployment/mcp.sh --http and open http://127.0.0.1:8010/. See MCP Tools.

Run the tests

uv run pytest packages/harborrag-core/tests   # one package, fast
uv run pytest                                 # everything
uv run make coverage                          # with the 90% gate

--extra dev covers most of the suite, but several packages gate optional adapters behind their own extras. To run it the way CI does:

uv sync --all-packages --all-extras
uv run pytest

Something not working?

Troubleshooting - setup, configuration, provider, and quality-gate failures
Installation - pip, editable installs, PDF/OCR backends, platform notes

Continue

Configuration - point HarborRAG at your own sources
CLI Reference - every command and flag
Python SDK - use it as a library
Chat - retrieval-grounded chat over HTTP and CLI
Architecture - before contributing code
Deployment - the full local topology and the production boundary

This site is open source. Improve this page.