Why AI plans break your rules

The setup

A fictional harbour, 11:45 on a working day. Four vessels need decisions. One has waited at anchor since dawn for a berth that has been empty all morning. One is ready to leave after a repair. A deep-draft ship can only use the inner channel inside a two-hour tide window. A car carrier is due mid-afternoon. The harbour has two tugs and two pilots, and the forecast says the wind will close the pilot boarding station from 15:00.

The data is the kind a port already holds: the berth plan, ship positions, the tide table, the tug and pilot rosters, the weather. Before any model ran, we wrote down what a good harbour planner would conclude, so the answer could not drift to fit the results.

What happened

We gave a current, capable model all of that data and the harbour’s rules in plain language, and asked for the plan: who moves when, with which pilot and tugs, and why. We ran it six times. Every one of the six first plans broke at least one hard rule.

None of these plans looked wrong. Each came with clear reasons and cited the right sources. The errors were in the arithmetic of the day: both tugs booked for one ship at noon while a second ship also needed one; a car carrier sent to board at 15:45, after the station had closed.

A fluent plan that quietly breaks a rule is worse than no plan, because it looks finished.

What fixed it

Two things, both outside the model. First, check the plan against the rules in plain code: tug counts, pilot overlaps, the boarding station’s hours, the tide window, the berth plan. That takes milliseconds, and it does not get tired at four in the afternoon.

When the check told the model exactly which rule it had broken, it produced a plan that passed within one or two more attempts, in all three runs where we tried it. But in one of them, a revision introduced a new problem while fixing an old one: a pilot booked on two movements at once. Fixing by conversation works, as long as every new version is checked again.

Second, let code compute the parts that are arithmetic, and keep the model for what it does well: reading the situation, explaining the plan in the planner’s words, and drafting the message to the ship. In the three runs built that way, the checked plan held every rule.

Outside a harbour

Any plan with hard constraints has the same shape. A staff roster with certifications and rest rules. A delivery schedule with vehicle limits. A loan decision with policy thresholds. A maintenance window with safety interlocks. A model will write a plausible plan for every one of them. The question is what checks it before anyone acts.

That check, grounded in your data and your rules, explained and kept on the record, is the reasoning layer at its simplest.