Contract test and benchmarks

Keeping up with ElastiCORE

• The kit holds no copy of the DSL documentation - only the one-page reference and the recipes. The full guides are those every generation copies into the project's src/main/resources/blueprint/, from the ElastiCORE version the project uses; the skill sends Claude there for everything else. When a project upgrades ElastiCORE, Claude reads the new guides. (The guides are in Korean; the kit's own pages are in English.)
• What the kit relies on is tested against ElastiCORE. tests/contract/check.py builds the new-project template, the worked example and a 2.x-generator project with the newest ElastiCORE 3 release from Maven Central, and generates, edits, breaks, restores and starts them. It checks the file markers, both generators' status lines and warning headers, the guide names, the Gradle tasks and what the started application answers - and the board, library and shop recipes end to end (the shop's contracts S1-S12: sign-up, sign-in, access, the lifecycle's 409, ...). A failure names the assumption that no longer holds and where the kit states it.
• CI runs it on every change and every Monday, so a change in ElastiCORE the kit has not followed shows up without a commit to the kit. The hook script is tested on Linux, macOS and Windows; the opencode adapter has its own tests.

python3 tests/contract/check.py                                   # JDK 17 and Gradle 8.14+ on the PATH
ELASTICORE_VERSION=3.0.261015-alpha1 python3 tests/contract/check.py   # a given release

A version ending in -SNAPSHOT is also looked up in the local Maven repository, so a build of ElastiCORE published with publishToMavenLocal can be checked before its release.

Benchmark: the same task with and without the kit

benchmarks/library - three agents got a library loan API (books with a 13-digit ISBN, members, loans, a 409 when no copy is left, a return operation) once as written and once with the kit and "build it with ElastiCORE". A separate Claude Code agent that took no part in the runs built, started and called every project against the same 21 checks. One run per arm, run by the authors (2026-10-20/21, one Windows PC, Gradle builds inside the times, kit 0.3.0, Codex CLI 0.163.0).

Agent	Without the kit	With the kit
Claude Code, Sonnet	4 min 41 s, 20.5 of 21	2 min 18 s, 20.5 of 21
Codex CLI, gpt-5.6-sol	7 min 13 s, 21 of 21	3 min 45 s, 21 of 21
opencode, a small local model (qwen3.8-flash-next-a5b)	no final report in four attempts (best: 19.5 of 21, no README)	11 min 42 s, 21 of 21

Run	Model calls	Context read	Output tokens	Cost
Claude Code, without	16	657k	26.0k	$0.48
Claude Code, with	10	437k	7.5k	$0.26
Codex, without	29	1,000k	14.6k	(subscription)
Codex, with	24	859k	9.5k	(subscription)

Without the kit an agent writes every entity, DTO, repository, service, controller and error handler by hand; with it, the same agent writes one 27-line YAML, the service methods and the one controller the recipe gives.

How to read it

The kit's library recipe was written from this task after the first round, so the with-kit runs had a recipe for the task's domain. The first run, with kit 0.2.0 and no such recipe, took Claude Code 7 min 47 s for 20.5 - slower than without the kit - and is the number for a task the kit has no recipe for. What that run still had to write in Java became five DSL additions in ElastiCORE (@readonly, @search(isnull), @pattern('regex', 'message'), a required reference by its key, ConflictException).

Benchmark: the pet store

benchmarks/petstore - Claude Code with the kit, three arms (Java hexagonal, Java DDD, Kotlin hexagonal), graded by a separate agent. The point was not the score but where the agent left the DSL - the long guides it searched, the jar it opened, the Java it wrote - to fix ElastiCORE or the kit there.

	Java hexagonal	Java DDD	Kotlin hexagonal
Round 1 - building the shop (24 items, before the fixes)	23.5, 8 min 06 s	23.5, 8 min 40 s	23.5, 30 min 53 s
Round 2 - after the fixes	24, 7 min 25 s, $0.86	23, 8 min 49 s, $1.01	24, 9 min 14 s
Phase 2 - the change request (20 items)	20, 13 min 24 s, $1.67	19.5, 14 min 11 s, $2.02	20, 13 min 58 s, $1.77

What changed because of it, in ElastiCORE and in the kit: Evolving a generated project.

Limits and tips

• The kit's pages are for the 3.0 rule engine with hexagonal or ddd; in a 2.x project the hooks work, the reference and recipes do not apply.
• The hooks see the edit tools and Gradle through Bash and PowerShell; an edit through a script is not stopped.
• Of a failed command Claude Code passes on only the first 10,000 characters - in a very large project warnings may be beyond what the hook sees.
• Codex CLI gets the skill only (no hooks). In Codex and opencode on Windows build with the Gradle daemon off (--no-daemon, or GRADLE_OPTS=-Dorg.gradle.daemon=false): otherwise opencode's PowerShell call does not return, and the codex process does not end after its answer.
• Never bootRun from a tool's shell; stop the server by its process id.