msjidandClaude Sonnet 5
771280cde6
RAG + multi-agent (retrieval, policy-compliance, drafting) support assistant for a fictional shipping company, built in Rust with Qdrant for vector storage and a hand-rolled Anthropic tool-calling client (no Rust pydantic-ai equivalent exists). A deterministic supervisor orchestrates the agents and runs every response through a rule-based guardrail pipeline with an append-only audit log. Includes a 44-example golden dataset (normal/injection/policy- violation/hallucination-bait/out-of-scope) and a regression harness that scores retrieval, generation, and guardrail correctness separately, plus a write-up of the failure modes and trade-offs. Co-Authored-By: Claude Sonnet 5 <[email protected]>
99 lines
4.1 KiB
Markdown
99 lines
4.1 KiB
Markdown
# Maritime/Logistics Support Assistant
|
|
|
|
A RAG + multi-agent support assistant for a fictional shipping company
|
|
(Meridian Sea Logistics), built with a safety/eval layer as the headline
|
|
feature rather than an afterthought. Written in Rust.
|
|
|
|
## Why this exists
|
|
|
|
This is a portfolio piece modeled on real maritime-logistics support work,
|
|
scoped to be publishable: its own fictional company, own documents, own
|
|
IP. The interesting part isn't the RAG pipeline (that's table stakes) --
|
|
it's the guardrail and regression-eval layer wrapped around it, and the
|
|
trade-offs documented in [`docs/SAFETY_WRITEUP.md`](docs/SAFETY_WRITEUP.md).
|
|
|
|
## Architecture
|
|
|
|
Three narrow agents behind a deterministic supervisor -- the supervisor is
|
|
plain Rust control flow, not another LLM call, so *what's allowed through*
|
|
is never itself something the model can be talked out of:
|
|
|
|
```
|
|
query -> guardrail.pre_check (rule-based, blocks prompt injection)
|
|
-> RetrievalAgent (Qdrant vector search -> LLM relevance filter)
|
|
-> DraftingAgent (tool-calling: lookup_shipment, escalate_to_human)
|
|
-> PolicyAgent (LLM compliance check against acceptable_use.md)
|
|
-> guardrail.post_check (rule-based PII/injection scan + policy verdict)
|
|
-> answer, audit record
|
|
```
|
|
|
|
Every LLM call asks for a schema-validated structured output (via a forced
|
|
tool-call, see `src/llm/client.rs`) rather than free text -- there's no
|
|
Rust equivalent of pydantic-ai, so this is a small hand-rolled agent loop
|
|
against the raw Anthropic Messages API.
|
|
|
|
The crate is organized as a dependency-injected module tree (see
|
|
`src/cli.rs`, the composition root) -- no global/static singletons
|
|
anywhere; every component is constructed explicitly and passed to its
|
|
collaborators through the constructor.
|
|
|
|
```
|
|
src/
|
|
config.rs Settings, loaded once in cli.rs
|
|
models/ domain types + structured agent I/O + eval types
|
|
knowledge/ the fictional company's markdown knowledge base
|
|
data/ seed shipment records
|
|
embeddings/ EmbeddingProvider trait + local fastembed impl
|
|
retrieval/ chunking, Qdrant wrapper, reference extraction
|
|
tools/ shipment lookup + human escalation (as LLM tools)
|
|
llm/ Anthropic client, tool trait, agentic loop
|
|
agents/ retrieval / policy / drafting agents + supervisor
|
|
guardrails/ rule-based scanner, audit log, decision pipeline
|
|
eval/ golden dataset loader, scoring, regression harness
|
|
cli.rs composition root + subcommands
|
|
```
|
|
|
|
## Running it
|
|
|
|
Requires a Qdrant instance and an Anthropic API key. Embeddings run
|
|
locally via `fastembed` (ONNX Runtime) -- the model downloads once on
|
|
first use, then runs fully offline.
|
|
|
|
```bash
|
|
docker run -p 6334:6334 qdrant/qdrant
|
|
cp .env.example .env # fill in ANTHROPIC_API_KEY
|
|
|
|
cargo run -- ingest # chunk + embed + index the knowledge base
|
|
cargo run -- ask "Where is my container MSCU7654321?"
|
|
cargo run -- eval # run the golden dataset, print the report
|
|
```
|
|
|
|
## Testing
|
|
|
|
```bash
|
|
cargo test # pure-logic unit + integration tests, no network required
|
|
cargo clippy --all-targets
|
|
```
|
|
|
|
The `eval` command exercises the live model and Qdrant; `cargo test`
|
|
deliberately does not, so CI-style checks don't need API keys.
|
|
|
|
## The safety/eval layer
|
|
|
|
- **Golden dataset** (`data/golden_dataset.jsonl`, 44 examples): normal
|
|
queries, prompt-injection attempts, policy-violation requests,
|
|
hallucination bait (asking about shipments that don't exist), and
|
|
out-of-scope requests.
|
|
- **Regression harness** (`src/eval/harness.rs`): scores retrieval
|
|
accuracy, generation accuracy, and guardrail correctness *separately*
|
|
per example and per category, so a failing run points at where the
|
|
failure originated instead of just "test 17 failed."
|
|
- **Guardrail pipeline** (`src/guardrails/`): a fast rule-based scan runs
|
|
before and after the LLM agents, independent of what the policy agent
|
|
concludes, and every decision -- allow or block -- is written to an
|
|
append-only audit log.
|
|
|
|
See [`docs/SAFETY_WRITEUP.md`](docs/SAFETY_WRITEUP.md) for the failure
|
|
modes this was designed for, the trade-offs made, and what would need to
|
|
change for real production scale.
|