139 lines
6.1 KiB
Markdown
139 lines
6.1 KiB
Markdown
# Maritime/Logistics Support Assistant
|
|
|
|
A RAG + multi-agent support assistant for a fictional shipping company
|
|
(Meridian Sea Logistics), built with a safety/eval layer as the headline
|
|
feature rather than an afterthought. Written in Rust.
|
|
|
|
## Why this exists
|
|
|
|
This is a portfolio piece modeled on real maritime-logistics support work,
|
|
scoped to be publishable: its own fictional company, own documents, own
|
|
IP. The interesting part isn't the RAG pipeline (that's table stakes) --
|
|
it's the guardrail and regression-eval layer wrapped around it, and the
|
|
trade-offs documented in [`docs/SAFETY_WRITEUP.md`](docs/SAFETY_WRITEUP.md).
|
|
|
|
## Architecture
|
|
|
|
Three narrow agents behind a deterministic supervisor:
|
|
|
|
```mermaid
|
|
flowchart TD
|
|
Q["Customer Query"] --> REGEX{{"Regex injection scan"}}
|
|
REGEX -- clean --> SEM{{"Embed query -> cosine similarity<br/>vs. known injection examples"}}
|
|
SEM -- score < threshold --> RET["Retrieval Agent<br/>vector search + LLM relevance filter"]
|
|
SEM -- score >= threshold --> JUDGE{{"LLM Injection Judge"}}
|
|
JUDGE -- not injection --> RET
|
|
JUDGE -- injection --> FINAL
|
|
REGEX -- match --> FINAL["Final Answer + Citations -> Customer"]
|
|
RET <--> QD[("Qdrant Vector Store<br/>chunked KB docs")]
|
|
RET --> DRAFT["Drafting Agent<br/>tool-calling loop"]
|
|
DRAFT -- lookup_shipment --> SHIP[("Shipment Repository")]
|
|
DRAFT -- escalate_to_human --> ESC[("Escalation Desk")]
|
|
DRAFT --> POL["Policy-Compliance Agent"]
|
|
POL --> DOCS[("acceptable_use.md +<br/>tracking_policy.md")]
|
|
POL --> POST{{"Guardrail: Post-check<br/>PII/injection scan + policy verdict"}}
|
|
POST -- allow / escalate --> FINAL
|
|
REGEX -.-> AUDIT[("Audit Log<br/>JSONL, append-only")]
|
|
JUDGE -.-> AUDIT
|
|
POST -.-> AUDIT
|
|
|
|
classDef agent fill:#eef2ff,stroke:#4f46e5,color:#312e81
|
|
classDef guardrail fill:#fff7ed,stroke:#ea580c,color:#7c2d12
|
|
classDef store fill:#f1f5f9,stroke:#64748b,color:#334155
|
|
classDef audit fill:#f0fdf4,stroke:#16a34a,color:#14532d
|
|
classDef io fill:#f8fafc,stroke:#334155,color:#0f172a
|
|
|
|
class RET,DRAFT,POL agent
|
|
class REGEX,SEM,JUDGE,POST guardrail
|
|
class QD,SHIP,ESC,DOCS store
|
|
class AUDIT audit
|
|
class Q,FINAL io
|
|
```
|
|
|
|
The pre-check is three layers, cheapest first: a regex scan catches fixed
|
|
wordings for free; if that's clean, the query is embedded and compared
|
|
against a small corpus of known injection examples so paraphrases that
|
|
dodge the regexes still get caught; only if that similarity score clears a
|
|
threshold does the pipeline pay for an LLM call to judge intent, since
|
|
embedding similarity alone is too noisy to block a customer on. See
|
|
`src/guardrails/semantic_guardrail.rs` and `src/guardrails/injection_judge.rs`.
|
|
|
|
The supervisor (`src/agents/supervisor.rs`) is what runs this graph -- plain
|
|
Rust control flow, not another LLM call, so *what's allowed through* is
|
|
never itself something the model can be talked out of. The dashed edges are
|
|
the audit trail: every guardrail decision is logged, not just the blocked
|
|
ones.
|
|
|
|
Every LLM call asks for a schema-validated structured output (via a forced
|
|
tool-call, see `src/llm/client.rs`) rather than free text -- there's no
|
|
Rust equivalent of pydantic-ai, so this is a small hand-rolled agent loop
|
|
against the raw Anthropic Messages API.
|
|
|
|
The crate is organized as a dependency-injected module tree (see
|
|
`src/cli.rs`, the composition root) -- no global/static singletons
|
|
anywhere; every component is constructed explicitly and passed to its
|
|
collaborators through the constructor.
|
|
|
|
```
|
|
src/
|
|
config.rs Settings, loaded once in cli.rs
|
|
models/ domain types + structured agent I/O + eval types
|
|
knowledge/ the fictional company's markdown knowledge base
|
|
data/ seed shipment records
|
|
embeddings/ EmbeddingProvider trait + local fastembed impl
|
|
retrieval/ chunking, Qdrant wrapper, reference extraction
|
|
tools/ shipment lookup + human escalation (as LLM tools)
|
|
llm/ Anthropic client, tool trait, agentic loop
|
|
agents/ retrieval / policy / drafting agents + supervisor
|
|
guardrails/ rule-based scanner, audit log, decision pipeline
|
|
eval/ golden dataset loader, scoring, regression harness
|
|
cli.rs composition root + subcommands
|
|
```
|
|
|
|
## Running it
|
|
|
|
Requires a Qdrant instance and an Anthropic API key. Embeddings run
|
|
locally via `fastembed` (ONNX Runtime) -- the model downloads once on
|
|
first use, then runs fully offline.
|
|
|
|
```bash
|
|
docker run -p 6334:6334 qdrant/qdrant
|
|
cp .env.example .env # fill in ANTHROPIC_API_KEY
|
|
|
|
cargo run -- ingest # chunk + embed + index the knowledge base
|
|
cargo run -- ask "Where is my container MSCU7654321?"
|
|
cargo run -- eval # run the golden dataset, print the report
|
|
```
|
|
|
|
## Testing
|
|
|
|
```bash
|
|
cargo test # pure-logic unit + integration tests, no network required
|
|
cargo clippy --all-targets
|
|
```
|
|
|
|
The `eval` command exercises the live model and Qdrant; `cargo test`
|
|
deliberately does not, so CI-style checks don't need API keys.
|
|
|
|
## The safety/eval layer
|
|
|
|
- **Golden dataset** (`data/golden_dataset.jsonl`, 44 examples): normal
|
|
queries, prompt-injection attempts, policy-violation requests,
|
|
hallucination bait (asking about shipments that don't exist), and
|
|
out-of-scope requests.
|
|
- **Regression harness** (`src/eval/harness.rs`): scores retrieval
|
|
accuracy, generation accuracy, and guardrail correctness *separately*
|
|
per example and per category, so a failing run points at where the
|
|
failure originated instead of just "test 17 failed."
|
|
- **Guardrail pipeline** (`src/guardrails/`): a fast rule-based scan runs
|
|
before and after the LLM agents, independent of what the policy agent
|
|
concludes; the pre-check also embeds the query and escalates
|
|
high-similarity matches against known injection phrasings to an LLM
|
|
judge before blocking, catching paraphrases the regexes miss without
|
|
paying for an LLM call on every query. Every decision -- allow or
|
|
block -- is written to an append-only audit log.
|
|
|
|
See [`docs/SAFETY_WRITEUP.md`](docs/SAFETY_WRITEUP.md) for the failure
|
|
modes this was designed for, the trade-offs made, and what would need to
|
|
change for real production scale.
|