Case study
What the KLOD check found in precommitEU
I ran the klod-check skill on my own compliance scanner before asking anyone else to. Here is the report, what it got right, and the one gap it found.
Before asking anyone to run the KLOD check on their software, I ran it on mine. precommitEU is a scanner I maintain that reads source code and reports likely breaches of eight EU regulations, from the GDPR to the AI Act. Version 0.2.1 runs small language models on the developer's own machine and sends nothing anywhere. That makes it a useful first case, because it is AI software with no hosted supplier at all, and I wanted to see whether the check would say something sensible rather than invent a problem.
The check took one session with a coding assistant. It read 7,725 lines of Python, the tests, the GitHub Action, the documentation and a demo codebase, then wrote klod_report.md at the root of the repository. I committed the report as it came out, and the full text is at the end of this post. What follows is my reading of it.
Zero hosted dependencies, so no level
The first thing the check does is classify what it is looking at. precommitEU is a product, not a library: a command-line tool, a GitHub Action and a local web UI. Its one capability is compliance scanning, and every model call goes to a llama-server process that the scanner starts itself on the loopback address, through an HTTP opener that ignores proxy settings so that a corporate proxy cannot quietly receive your code.
KLOD's levels, L0 to L3, exist for work that stops when a supplier stops. A model you run from weights you hold is not a supplier, so the report gives no level and says so in the headline. That is the correct answer, and I was glad the check did not stretch to award something. A tool like this has nothing to declare under the specification.
What it gets instead is a review under principle 6, the one about keeping people in control of AI that runs inside your software. In the report that is a five-row table: four controls in place, one gap.
The scanner fails closed
The most interesting property the report picked out is one I had built without naming it. precommitEU has no way to detect anything without its model. There is no regex fallback pretending to be a scan. When the model is missing, the run stops with a configuration error before it reads a single file. When llama-server cannot start, the run fails. When a file cannot be analysed, the scan is marked incomplete, and in CI it exits with code 3, so an incomplete scan can never be mistaken for a clean pass.
That is the right shape for a gate. The alternative, a scanner that degrades quietly and reports "no findings" because it could not look, would be worse than no scanner. The check traced this through the code and the tests and wrote it down as evidenced. I would not have thought to document it as a continuity property, but it is one: when the AI is unavailable, the tool says so, and the team is back where it was before the tool existed, reviewing the change by hand.
The gap
That last sentence is also where the check found the gap. The human path exists in practice, since people reviewed code for these rules before the scanner and can do so again. The regulation knowledge the model uses is not locked inside it: each pack ships a readable summary of the relevant articles, 387 lines for the GDPR alone, which a reviewer can use directly. But nowhere in the documentation does it say what a team should do when the scanner cannot run.
The report's first action is one paragraph in the CI guide: a failed or skipped scan means a person reviews the change against the regulation summaries, and exit 3 must block the merge. I will add it. It is a small fix, and it is exactly the kind of thing that never gets written down because everyone assumes it is obvious, until the day the runner image changes and nobody remembers what the gate was protecting.
Where precommitEU did well
Four properties came out of the review evidenced, with file and line references I could check:
- Nothing leaves the machine. Every model request goes to a loopback
llama-serverthe scanner starts and stops itself, and the HTTP opener drops proxy settings so a corporate proxy never sees the code. The security policy treats any egress as a critical bug. - The models cannot act. Their tools read files inside the scanned directory and nothing else. The only effect a finding has is an exit code that the operator chooses to enforce.
- People have the last word, with a paper trail. A confirmed finding can be accepted with a recorded reason that stays in the report and never changes the exit code on its own.
- The knowledge is inspectable. The regulation summaries and the case files the models work from are plain Markdown and JSONL in the package, so a reviewer can read exactly what the scanner was taught to look for.
None of these earns a KLOD level, because there is no supplier to lose. They are what principle 6 asks of any AI that runs inside software, and the report found them already in place.
Two things the report kept separate
The report lists two installation-time downloads in its inventory: Hugging Face for the model bundle and GitHub releases for the llama.cpp binary in the Action. An existing installation keeps working without either, the Action caches the bundle, and the bundle ships checksums so a mirrored copy can be verified. The check recorded these as installation-time, not runtime, and they count toward nothing. That distinction matters. Conflating "we download the model once" with "we call a supplier on every request" would make the whole method useless.
It also noted that the models only read and report. Their tools are confined to the scanned directory, the only effect a finding has is an exit code the operator chooses to enforce, and a person can accept a finding with a reason that stays in the report. That is the action boundary principle 6 asks for, and it was already there.
What I take from it
Running the check on my own tool was the fastest way to learn what it feels like from the other side. Two things stood out. The report is specific: every claim carries a file and line range, and a script confirmed each of the 35 ranges exists before I published it. And it is short and restrained: a result line, five bullets, tables, no level where none applies, no finding where the evidence only supports an observation, and a clear line between what was inspected and what was not. Tests were read, not run. Model weights were not inspected. No server was started. The report says all of that in 78 lines.
If you maintain software that uses AI, hosted or local, the check is one command away once the skills are installed:
apm install AlexandruGirlea/keepthelightson#v0.1.2
Then ask your assistant to run klod-check, or use the Spec Kit and OpenSpec integrations I wrote about last time. If your report finds something I have not thought about, I would like to read it.
The full report
As written on 12 September 2026 against revision 26baea4. The same file is committed in the precommitEU repository.
- Project: precommitEU 0.2.1 (pyproject.toml:10-12)
- Git origin: ssh://github.com/AlexandruGirlea/precommiteu.git
- Revision: 26baea4ad37721a9a7888e4d26bf43cb167956b9
- Model: claude-fable-5-1 (source: Claude Code host system prompt)
- Report timestamp: 2026-09-12T14:00:42+03:00 (Europe/Bucharest)
Result: PASS. No dependence on an external AI provider. 0 dependent capabilities. Coverage: complete.
Summary
- Classification: product, a command-line scanner with a GitHub Action and a local web UI (pyproject.toml:67; action.yml:1-9; src/precommiteu/ui/server.py:17-18).
- No business work depends on a hosted model. Compliance scanning runs on language models the operator holds, served by
llama-serverprocesses the scanner starts on the loopback address (src/precommiteu/llama_server.py:156-186; src/precommiteu/model_factory.py:51-67). - No gap against the numbered clauses. One human-control gap: the documentation never says what a team does when the scanner cannot run (see Human control).
- First action: add one paragraph to docs/ci.md stating that a failed or skipped scan means a person reviews the change against the regulation summaries, and that exit 3 must block the merge. Owner: maintainer.
- Assessed against specification 0.1.0.
AI inventory
| AI use | Where | Workflow | Type | Counts toward a level |
|---|---|---|---|---|
| Base model on a local llama-server, used by the validator and the orchestrator | src/precommiteu/scan.py:713-727; src/precommiteu/model_factory.py:118-135 | Compliance scanning | Self-run | No |
| Per-regulation LoRA detector adapter on a second local llama-server | src/precommiteu/scan.py:751-764; src/precommiteu/llama_server.py:175-176 | Compliance scanning | Self-run | No |
| Model bundle downloaded once from Hugging Face, then verified by checksum | src/precommiteu/ui/install.py:15-29, 66-78; action.yml:90-101 | Installation, before any scan | Installation-time | No |
| llama.cpp release binary downloaded by the GitHub Action | action.yml:61-70 | Installation in CI | Installation-time | No |
| OpenAI client inside the demo codebase written for the scanner to find | example-local-ui/demo_ai_act_violations/api/candidate_chat.py:6-10 | None, a scan target | Sample | No |
No development-time AI in CI (.github/workflows/ci.yml, release.yml). No hosted model API, gateway or provider SDK on any runtime path.
Capabilities
| Capability | Supplier | Declared | Supported | Blocking gap |
|---|---|---|---|---|
| None | No hosted AI supplier; models run locally from held weights | None | No level applies | None |
Portability: not applicable. For information, model access goes through one adapter class (src/precommiteu/model_factory.py:30-49) to an OpenAI-compatible loopback endpoint (src/precommiteu/llama_server.py:299), and model files are chosen by configuration (src/precommiteu/defaults.py:22-35).
Findings
No findings against the numbered clauses.
Human control (principle 6)
| Control | Status | Evidence | Next action |
|---|---|---|---|
| AI-off switch that works without the supplier, with the mode visible | In place | Every run is a deliberate invocation; --dry-run lists files without loading a model (src/precommiteu/cli.py:238-243); a missing model stops the run before any scan (src/precommiteu/defaults.py:8-14; docs/troubleshooting.md:87-93); a missing adapter continues in a labelled degraded mode (src/precommiteu/scan.py:738-748) |
None |
| Mode enforced where actions are accepted, kept across restarts | In place | The models only read and report: tools are confined to sandbox roots (src/precommiteu/src/tools/sandbox.py:19-30) and the grep worker is read-only with a timeout (src/precommiteu/src/tools/read_tools.py:264-283); findings act only through an exit code the operator selects (src/precommiteu/cli.py:513-517) | None |
| Late results and queued AI actions rejected after switch-off | In place | A stopped scan exits 130 with partial results (src/precommiteu/cli.py:509-511); the ledger records only files analysed end to end, so unfinished files are scanned again (src/precommiteu/scan.py:90-99); servers are terminated when the scan ends (src/precommiteu/llama_server.py:121-138) and stale ones are killed by the UI (src/precommiteu/ui/runner.py:33-44) | None |
| Manual work usable: inputs, status, permissions, controls | Gap | The scanner fails closed: an unanalysed file is reported and, under --ci, exits 3 so an incomplete scan never reads as a clean pass (src/precommiteu/cli.py:224-227, 498-514). The regulation knowledge a reviewer needs is in readable files (src/precommiteu/regulations/gdpr/regulations_summary.md:1-8 and five more packs). No page tells a team to review by hand when the scanner cannot run |
Add the paragraph named in the summary to docs/ci.md |
| Restart needs a recorded human decision | In place | Each scan is started by a person or a CI job; a finding can be accepted only with a recorded reason that stays in the report (docs/suppressions.md:11-18; docs/reports.md:45) | None |
Installation-time note: a fresh install needs Hugging Face for the model bundle and, in the Action, GitHub releases for the llama.cpp binary. An installed copy works without any network service (SECURITY.md:17-19; docs/install.md:270); the Action caches the bundle (action.yml:83-88).
Coverage
| Area | Status | Note |
|---|---|---|
| src/precommiteu/ (7,725 lines) | Inspected | Self-run detector, validator and orchestrator models |
| src/precommiteu/regulations/ | Inspected | Prompts and readable regulation summaries; no model calls |
| src/precommiteu/ui/ | Inspected | Bundle download, preflight checks, local scan runner |
| tests/ (1,199 lines) | Inspected | Fallback, output guard and ledger tests; read, not run |
| action.yml, .github/workflows/ | Inspected | Installation-time downloads; no development-time AI |
| docs/, README.md, SECURITY.md, CHANGELOG.md | Inspected | Exit codes, suppressions, air-gapped install |
| example-local-ui/ | Inspected | Sample code with a hosted AI client, by design |
| pyproject.toml, uv.lock | Inspected | No AI SDK dependencies; huggingface_hub optional for the UI |
| dist/, .venv/, tool caches | Excluded | Built artefacts of earlier versions and local environments |
Evidence to obtain
- None. A recorded CI run of the three failure modes (missing model, missing engine, interrupted scan) would turn inspected tests into observed behaviour.
Scope and limits
- Repository review, not a drill. No service was interrupted and no provider was called.
- Tests were read, not run. Model weights, the Hugging Face bundle and the deployed behaviour of the Action were not inspected.
- Working tree clean.
- Previous scan: none (first scan).
