Evolving a generated project

A generated project is not finished when it first runs. The agent kit's pet store benchmark (benchmarks/petstore) first had a shop built from an empty folder, then gave each finished shop a change request, and looked at where the work left the DSL. The measurements were made with Claude Code and the ElastiCORE agent kit, one run per arm, by the authors, on one Windows PC; each result was built, started and called by a separate Claude Code agent against a checklist.

The change request

On top of a working shop (task-2.ko.md, given in Korean):

1. Stock. Every item has a stock quantity (0 or more, required). An order takes each line's quantity off; if any line lacks stock the whole order is refused (409, with the item's name). Cancelling puts the stock back. The administrator can change stock.
2. Reviews. A signed-in member reviews an item: a rating (1-5), a text (1-1000 characters), the time the server sets. One review per member and item (the second is 409). Only the writer edits or deletes it (the administrator may delete). Anyone lists an item's reviews, newest first; a review shows the writer's login and no other personal data.
3. A courier API. When the administrator ships an order, the courier's API is called and the tracking number is stored on the order. The courier's address is a setting; a fake courier inside the server answers by default. If the courier fails, shipping fails and the order stays PLACED (502).
4. BI measures. Revenue (total without cancelled orders), order count, average order and member count as an Apache Ossie semantic model; the administrator's sales summary returns the average order too.
5. Korean error messages for Accept-Language: ko, English otherwise.

The results

Three arms - Java hexagonal, Java DDD, Kotlin hexagonal - each on a copy of its finished shop, committed first so the change reads as a diff; graded with checklist-2.md (20 items):

Arm	Score	Time	Cost	Turns
Java hexagonal	20 / 20	13 min 24 s	$1.67	56
Java DDD	19.5 / 20	14 min 11 s	$2.02	70
Kotlin hexagonal	20 / 20	13 min 58 s	$1.77	70

Regenerating each changed project on a copy wrote nothing and reported no SKIP(user-modified): the hand code sat in owned files only.

Hand code that stayed, as business logic: stock taken on order and put back on cancel, the reviewer's login stored on the review, the courier call itself.

What it showed, and what changed

Seen	Now (ElastiCORE 3.0.261022-rc1, kit 0.4.0)
all three searched the long guides for metrics: and ossie-export	on the kit's quick reference
built-in validation messages in the machine's language next to English DSL messages; each arm set spring.web.locale or wrote ValidationMessages.properties	one language per answer: Accept-Language, else spring.web.locale, never the machine's
a hand rule in an update() override was bypassed by PUT /batch (an admin edited a member's review)	batchUpdate calls update(); beforeSave is the place for write rules
a review could be moved to another item	@updatable(false) on a reference works
the TypeScript client kept the first model (elcore3ts is a task of its own)	typescriptClient = true, and a warning when the client is older than the model
a courier failure needed a hand-written exception and handler for 502	PortCallException, 502
an unknown enum value answered Spring's default body	problem+json with fieldErrors for every request error
count(Customer) exported as a bare COUNT(*)	the metric names its dataset
name search kept case	@search(like) ignores case in the v3 bundles (hexagonal, DDD, Kotlin, -Pv3)

Before that: building the shop

The first task (task.ko.md): a catalog, members who sign in, orders with a lifecycle and prices the server works out, an administrator's report, a static page - graded with 24 items.

Round 1 (before the fixes, kit 0.3.0): all three arms 23.5 of 24; 8 min 06 s (Java hexagonal), 8 min 40 s (Java DDD), 30 min 53 s (Kotlin hexagonal). Every arm searched the guides for value objects, @login/@access, states: and @snapshot/@compute, opened the runtime jar with javap, and wrote a sign-in controller, an existence check for referenced rows, the 409 for a refused state step and the order time by hand. Those became @signin, requireReferences (404), the 409, and @readonly @default(now) in ElastiCORE; the kit got a shop recipe and a table of the runtime classes. Writing the recipe also found two ElastiCORE bugs - an optional embedded address with required fields refused every order without it, and a model file that did not parse was dropped while the build succeeded - both fixed.

Round 2 (after the fixes, kit 0.4.0):

Arm	Score	Time	Cost	Turns
Java hexagonal	24 / 24	7 min 25 s	$0.86	28
Java DDD	23 / 24	8 min 49 s	$1.01	37
Kotlin hexagonal	24 / 24	9 min 14 s	-	-

No arm searched the long guides for the shop's features or wrote a sign-in controller or a role check. Hand code left: the start-up data, the report endpoint, the event listener, VAT and discount in the owned service.

Reading the numbers

The kit's shop recipe was written after round 1 and is close to this task, so round 2 measures the kit with a recipe for the task's domain. One run per arm; the task, checklists, runners and analysis scripts are in the benchmark folder.

The workflow on your own project

1. Write the change request in business terms.
2. Change the YAML; regenerate and compile; fix every DSL warning.
3. Write what the DSL cannot say in owned files only (beforeSave, a command override, a small controller).
4. Run it and call it - the refusals included.
5. Report the change in business terms; regenerate once more to prove nothing hand-edited is in a generated file.

The agent kit does exactly this with an AI assistant: see Change requests.