Delivery case: turning 44 warehouse-project pitfalls into a tool AI assistants can call

I turned the pitfalls I have seen on warehouse and logistics projects into a library, then built two things on top of it: a five-minute project self-check, and a set of tools that AI assistants such as Claude or Cursor can call directly.

This is how it was built, what broke, and how I check whether it can be trusted.

1. Context and constraints

Since 2019 I have been taking warehouse and logistics systems live, and I have watched the same pitfalls come back again and again:

• the location master is half filled in on go-live day;
• the automated warehouse was bought as a black box, so after acceptance every change to an outbound rule becomes a change request;
• nobody can say who will be on site the night of the cutover.

That knowledge lived in three places: my implementation handbook, my go-live checklist and my judgment cards. It could help people, but only if they read it.

I wanted two kinds of people to be able to use it:

• the customer-side project manager, a few months from go-live, who wants to know where the project will break;
• the implementation consultant, who already works with an AI assistant and wants it to answer “which pitfall is this symptom, and what do I do about it”.

Four constraints:

1. No company can be identifiable. Every pitfall comes from a real project, so everything is de-identified.
2. A card only counts once I have reviewed it myself. Nothing machine-assembled goes live unreviewed.
3. Chinese and English. Both sites use it.
4. One person maintains it. It cannot be a system that needs daily care.

2. The calls I made

Call 1: the model does not do the scoring.

A customer needs a conclusion they can audit: the same answers must give the same result today and tomorrow. So the self-check’s scoring rules are fixed in code:

• the project phase and scope (automated warehouse or not, AGVs, MES integration) select the questions that apply;
• an answer of “no” or “don’t know” means the linked pitfall is present;
• results are ranked by severity, then by distance from the current phase, and the top three become the advice.

Every rule has unit tests. With zero hits the tool does not pad the list to three. If every answer is “don’t know”, it lists the documents to request from the vendor first.

Call 2: the model does what it is good at, which is reading what people say.

There is no language model inside the tools. They are exposed over MCP, the standard interface for AI assistants to call external tools, so the user’s own assistant does the work:

• the assistant understands what the user means;
• the library supplies an answer it can back up: every pitfall cites the handbook chapter or checklist item it comes from.

The model cannot invent the answer, and users do not pay for my model calls.

Call 3: one master copy.

Pitfalls, questions, stations and scoring rules live in a single code package. The website, the account service and the MCP service all read from it, and every copy carries a version number and a content hash. If they disagree, the tests fail.

3. What I built

Pitfall cards (one Markdown file each; only reviewed cards are compiled)
   │  compile: de-identification check + schema validation
   ▼
Pitfall package (the master copy): 21 stations · 5 lines · 44 pitfalls · 30 self-check questions · scoring rules
   ├─▶ Website: home-page module, self-check intro page, sample health-check report
   ├─▶ Account service: phased questionnaire, result page, one-click booking for a project health check
   └─▶ MCP service: 3 tools
         · wms_list_questions   list the self-check questions that apply
         · wms_risk_scan        scan answers for risk (same scoring as the web self-check)
         · wms_symptom_lookup   describe a symptom on site, get the 3 most likely pitfalls

Symptom lookup (wms_symptom_lookup) is deterministic too, with no model call:

• the sentence is split into terms, and each term is weighted by how distinctive it is;
• a term counts more when it matches a pitfall’s synonyms or keywords, and less when it only matches the title or the symptom text;
• if the best score is below the threshold, the answer is “no match”. It would rather say nothing than make something up.

The Chinese site went live first in October 2026 (10-09), the English site followed (10-13).

4. What broke, and how I found it

1. The Chinese page showed the English site’s endpoint.

• Symptom: while preparing the Chinese tools, I found three endpoint addresses on the Chinese /mcp/ page pointing at the English domain.
• Cause: the tool-catalogue snapshot was pulled from the English account service, and validation only accepted the English domain, so the Chinese page displayed the wrong address while passing every check.
• Fix: shipped on its own, without waiting for the next release. The snapshot is now split by language, the Chinese page reads the copy from the China account service, and two checks were added: one validates the snapshot’s language, the other checks the endpoint’s domain at build time.

2. “Verified locally”, but 403 from outside.

• Symptom: when I added a visit counter to the self-check intro page, the deploy script reported a successful verification, yet the counter URL on the English staging site returned 403 from outside and the count never moved.
• Cause: staging has an address allow-list as well as a password. The script removed the password requirement but did not allow the address. Its verification ran from the server itself, which is on the allow-list, so it could not see the problem. It also turned out the script was reading the host name from the wrong line of the config.
• Fix: on sites like this the script now also allows the counter address, reads the host name only from the right directive; three tests were added and one updated.
• In hindsight: the Chinese staging site had the same configuration, and the bug had been there for days. Nobody had looked from outside.

Lesson: verify from where the user stands. Verifying from an allowed location will never show you the problem.

3. One label in the evaluation set was wrong.

• Symptom: in the English evaluation set, “Developers test once” was labelled as a hit for one pitfall.
• Cause: an independent review found nothing in the original Chinese card to support that match. An extra phrase in the English synonym list had produced it.
• Fix: relabelled as “should not match”, with the change recorded in the dataset notes.

Lesson: the evaluation set needs review too. With a wrong label, a good-looking metric means nothing.

5. How I verify it

The acceptance bar lives in the tests: at least 80% of expected matches appear in the top three, and at most 20% false positives when nothing should match.

The threshold was calibrated on labelled examples: among phrases that should match, the lowest top score was 19.5; among phrases that should not, the highest top score was 16.9. The threshold is 18.

Results from a re-run on 2026-11-04 (5 test files, 82 passed, 1 skipped):

Evaluation	Should match	Should not match
Chinese	59 / 59 in the top three	0 / 10 false positives
English, calibration set	29 / 29	0 / 11 false positives
English, held-out set (not used for tuning)	14 / 14	0 / 10 false positives
English, descriptions over 200 characters	2 / 3	0 / 8 false positives

What these numbers do and do not show:

• the labelled examples were written by an AI agent from the pitfall cards; they are not real users’ words;
• 100% more likely means the examples are too easy than that the tool is this accurate in the field;
• long descriptions only hit 2 out of 3, the weakest point today: when a user writes a long paragraph of background, the key signal gets diluted.

When a real user’s phrasing gets no match, it is de-identified and logged. Once there are enough of these, they will feed new synonyms and a bigger evaluation set.

6. Results

• Both languages are live: the web self-check, the sample report and the 3 MCP tools all work.
• The library, website, account service and MCP service are driven by one package, with a version number you can check.
• There is little real usage data yet, so this post makes no claims about impact. Once enough real lookups have accumulated, I will write up the hit rate on real phrasing.

7. What I would do differently

1. Build the evaluation set from 20 real user sentences first, then write synonyms. Writing synonyms first and then generating examples turns into setting your own exam.
2. Always verify from outside. Pitfall 2 shows that every “verification” has to start from where the user is.
3. Handle long descriptions separately. Pull the symptom sentences out of the paragraph first, then match, instead of scoring the whole block.

Want to try it on your own project? Take the five-minute self-check or read the sample health-check report.