morluto/rea

Fork 0

mirror of https://github.com/morluto/rea.git synced 2026-11-05 06:45:22 +00:00

[GH-ISSUE #8951] [Bug] environment-build demotes ID sorting when existing creation clocks are missing #7866

New issue

Open

opened 2026-11-05 02:37:35 +00:00 by bsakaguchi · 1 comment

bsakaguchi commented

2026-11-05 02:37:35 +00:00

Owner

Originally created by @coygeek on GitHub (Nov 5, 2026).
Original GitHub issue: https://github.com/morluto/rea/issues/8951

Before submitting

I searched the open and closed issues for a related report.
I removed credentials, tokens, private paths, and sensitive target data.

Area

Evidence, artifacts, or reconstruction

Observed behavior

A capture whose existing collection has ascending non-v4 string IDs and sparse created_at fields produces a served create plan with insert: {"rule":"last"}. The create response supplies a generated created_at clock. Filling the missing existing clocks with ascending timestamps, or omitting clocks from both existing and created entities, produces a served sorted insertion by ID instead.

This affects actual serving. Through the unmodified environment-serve CLI and HTTP MCP, three successful post_items calls append generated IDs out of ascending order in the sparse case. The complete-clock and no-clock controls retain ascending ID order after the same calls. I observed this on both the baseline source runtime and published 6.8.0.

Reproduction

From a built checkout, using Node 24.22.1, run the following. It constructs ordinary HAR 1.2 fixtures, runs the real CLI, and projects the resulting capture Evidence. No REA source edits or helper calls are required.

set -eu
repo="$PWD"
proof="$(mktemp -d)"
mkdir -p "$proof/home"
cd "$proof"
export HOME="$proof/home"
python3 - <<'PY'
import json
from pathlib import Path
for mode in ['sparse', 'complete', 'no-clock']:
    ids = ['6ad47b491023b50001087124', '6ad47b491023b50001087125', '6ad47b491023b50001087126']
    members = [dict(id=i, name=n) for i, n in zip(ids, ['Delta', 'Alpha', 'Charlie'])]
    if mode != 'no-clock':
        members[0]['created_at'] = '2026-10-01T09:00:00Z'
    if mode == 'complete':
        members[1]['created_at'] = '2026-10-01T10:00:00Z'
        members[2]['created_at'] = '2026-10-01T11:00:00Z'
    created = dict(id='6ad47b491023b50001087130', name='Echo')
    if mode != 'no-clock':
        created['created_at'] = '2026-10-17T18:00:04Z'
    rows = [('GET', '/api/items', members, None)]
    rows += [('GET', '/api/items/' + m['id'], m, None) for m in members]
    rows += [('POST', '/api/items', created, {'name': 'Echo'}), ('GET', '/api/items/' + created['id'], created, None)]
    entries = []
    for i, (method, path, body, sent) in enumerate(rows):
        text = json.dumps(body)
        request = dict(method=method, url='http://app.example' + path, httpVersion='HTTP/1.1', cookies=[], headers=[], queryString=[], headersSize=-1, bodySize=0 if sent is None else len(json.dumps(sent)))
        if sent is not None:
            request['postData'] = dict(mimeType='application/json', text=json.dumps(sent))
        entries.append(dict(startedDateTime=f'2026-10-17T18:00:0{i}.000Z', time=5, cache={}, timings=dict(send=0, wait=5, receive=0), request=request, response=dict(status=201 if method == 'POST' else 200, statusText='OK', httpVersion='HTTP/1.1', cookies=[], headers=[], redirectURL='', headersSize=-1, bodySize=len(text), content=dict(size=len(text), mimeType='application/json', text=text))))
    Path(mode + '.har').write_text(json.dumps(dict(log=dict(version='1.2', creator=dict(name='fixture', version='1'), entries=entries))))
    request = dict(source=dict(kind='evidence-file', path=str(Path(mode + '-bundle/envs/items/evidence.json').resolve())), view=dict(kind='environment-state', template_rules=['after-uid-literal', 'opaque-token', 'sibling-variance', 'prefix-variance']))
    Path(mode + '.request.json').write_text(json.dumps(request))
PY
for mode in sparse complete no-clock; do
  node "$repo/scripts/rea.mjs" environment-build "$mode.har" "$mode-bundle" --name items --json > "$mode.build.json"
  node "$repo/scripts/rea.mjs" inspect-analysis-view "$mode.request.json" --json > "$mode.state.json"
done
python3 - <<'PY'
import json
for mode in ['sparse', 'complete', 'no-clock']:
    state = json.load(open(mode + '.state.json'))['normalized_result']
    plan = next(w for w in state['writes'] if w['tool'] == 'post_items')
    print(mode, json.dumps({k: plan[k] for k in ['served', 'insert', 'checked_record_ordinals']}))
PY

Observed output:

sparse {"served": true, "insert": {"rule": "last"}, "checked_record_ordinals": [5]}
complete {"served": true, "insert": {"rule": "sorted", "field": "id", "direction": "ascending"}, "checked_record_ordinals": [5]}
no-clock {"served": true, "insert": {"rule": "sorted", "field": "id", "direction": "ascending"}, "checked_record_ordinals": [5]}

I also ran the equivalent three-member fixtures on the installed published package. Each sparse case produced the same served last plan. A further fixture containing a final whole list in ascending ID order still retained last when the captured new ID was larger than all existing IDs.

Expected behavior

Missing or incomparable existing clock values should not establish that IDs contradict creation order. Keep the consistent identity sort eligible at its normal priority unless actual comparable clock values show an inversion, or the IDs have the explicitly recognized random UUID form.

Expected source: randomIdentities at the tested revision defines the non-v4 case by every creation clock showing another order. Its insertCandidates contract puts random identities after append. A missing value establishes neither order nor an inversion.

Impact and scope

P2: bounded incorrect inference changes the live local environment's entity order. The verified class is generated non-sequence, non-v4 identities, inferred creation clocks, and omitted existing clock values that cannot establish inversion. Existing integer sequence IDs bypass this check. V4 UUIDs legitimately receive the random classification.

All 18 HTTP MCP creates across the two runtimes and three fixtures succeeded. Sparse environments became unsorted; both controls remained sorted. No real application's capture prevalence, broader task failure rate, data destruction, or security effect was established. A practical workaround preserving sparse recorded data was not established. An all-clocks-absent variant is withheld by published 6.8.0 and is not needed to establish the reported served failure.

REA version

Baseline source: c3647ade1e557c4179a8b825a37e0c2193cd2526, package metadata 6.8.0. Also reproduced with npm-published rea-agents@6.8.0, which npm view rea-agents version returned as the latest published release. The relevant inference and simulation files are unchanged on fetched main 71f680d7c1aa7a45ba0325fe87b104744b0c105f; main was inspected, not separately run.

Runtime and platform

Node v24.22.1, macOS 27.2, arm64. No native analysis provider is involved.

Target and artifact context

Small synthetic HAR 1.2 captures of an ordinary JSON collection API: an initial list, item reads, a create response, and an item read confirming that create. Names are deliberately not monotonic, so they do not introduce an alternate scalar sort.

Interface

CLI

The learned environment was additionally exercised through its real HTTP MCP server and data plane.

Earliest failure stage

Transport or result projection

The first observable defect is the projected environment-state create plan's insert rule.

Relevant output or evidence

The CLI reproduction above returns exit 0 for all builds and projections. The sparse plan is served and confirmed by read ordinal 5, yet selects last instead of the eligible ascending ID sort.

For the served check, I started the generated manifest with environment-serve <manifest.json> --evidence <evidence.json> --port 0 --json, connected with the MCP client, called post_items three times with {"body":{"name":"Echo"}}, and fetched data/get from the environment's data plane. On both runtimes, the sparse fixture's resulting entity IDs were:

6ad47b491023b50001087124
6ad47b491023b50001087125
6ad47b491023b50001087126
e6c1d4f1586aae4e13052cff
1c78555ff498d7bfe90f2721
dedb2c6a6923260792ca0e3c

The second generated ID belongs before every initial ID but was appended. Complete-clock and no-clock controls remained sorted after three creates each. These observations establish served ordering for these fixtures, not a real application incident. Each listener shut down with handled signal status 143.

The existing focused tests also passed: npm run test:focused -- src/domain/analysisView/environmentStateWriteExplain.test.ts src/domain/analysisView/environmentStateWrites.test.ts — 250 tests in two files.

Typed outcome or recovery expected

Preserve the eligible {"rule":"sorted","field":"id","direction":"ascending"} insertion plan for the sparse fixture. Missing creation-clock comparisons should supply no positive evidence that the identity is random.

Analysis or suspected cause

Source-derived reasoning: compareValues returns undefined for missing values. randomIdentities converts it to positive one with ?? 1, satisfying its inversion test. buildCreate passes clocks inferred from the created entity while comparing existing members, which need not contain those clocks. The false random classification moves identity sorting after append.

A later whole-list validator can retry insertion candidates when placement fails; it does not universally restore the sort. Item reads do not establish list placement, and a captured create falling last agrees with append. Actual serving subsequently consumes the selected plan in environmentStateWriteAnswer. No code changes were made.

Additional context

I searched open and closed issues and PRs for the helper name, creation clocks, missing clocks, sparse fields, identity sorting, random identities, and insertion. No exact duplicate was found. The closed submillisecond timestamp sorting issue #6525 concerns a different comparator precision defect. Merged PR #7600 adds later-list sort learning; that mechanism is present in the tested code and does not prevent this sparse-clock case.

Originally created by @coygeek on GitHub (Nov 5, 2026). Original GitHub issue: https://github.com/morluto/rea/issues/8951 ### Before submitting - [x] I searched the open and closed issues for a related report. - [x] I removed credentials, tokens, private paths, and sensitive target data. ### Area Evidence, artifacts, or reconstruction ### Observed behavior A capture whose existing collection has ascending non-v4 string IDs and sparse `created_at` fields produces a served create plan with `insert: {"rule":"last"}`. The create response supplies a generated `created_at` clock. Filling the missing existing clocks with ascending timestamps, or omitting clocks from both existing and created entities, produces a served `sorted` insertion by ID instead. This affects actual serving. Through the unmodified `environment-serve` CLI and HTTP MCP, three successful `post_items` calls append generated IDs out of ascending order in the sparse case. The complete-clock and no-clock controls retain ascending ID order after the same calls. I observed this on both the baseline source runtime and published 6.8.0. ### Reproduction From a built checkout, using Node 24.22.1, run the following. It constructs ordinary HAR 1.2 fixtures, runs the real CLI, and projects the resulting capture Evidence. No REA source edits or helper calls are required. ```sh set -eu repo="$PWD" proof="$(mktemp -d)" mkdir -p "$proof/home" cd "$proof" export HOME="$proof/home" python3 - <<'PY' import json from pathlib import Path for mode in ['sparse', 'complete', 'no-clock']: ids = ['6ad47b491023b50001087124', '6ad47b491023b50001087125', '6ad47b491023b50001087126'] members = [dict(id=i, name=n) for i, n in zip(ids, ['Delta', 'Alpha', 'Charlie'])] if mode != 'no-clock': members[0]['created_at'] = '2026-10-01T09:00:00Z' if mode == 'complete': members[1]['created_at'] = '2026-10-01T10:00:00Z' members[2]['created_at'] = '2026-10-01T11:00:00Z' created = dict(id='6ad47b491023b50001087130', name='Echo') if mode != 'no-clock': created['created_at'] = '2026-10-17T18:00:04Z' rows = [('GET', '/api/items', members, None)] rows += [('GET', '/api/items/' + m['id'], m, None) for m in members] rows += [('POST', '/api/items', created, {'name': 'Echo'}), ('GET', '/api/items/' + created['id'], created, None)] entries = [] for i, (method, path, body, sent) in enumerate(rows): text = json.dumps(body) request = dict(method=method, url='http://app.example' + path, httpVersion='HTTP/1.1', cookies=[], headers=[], queryString=[], headersSize=-1, bodySize=0 if sent is None else len(json.dumps(sent))) if sent is not None: request['postData'] = dict(mimeType='application/json', text=json.dumps(sent)) entries.append(dict(startedDateTime=f'2026-10-17T18:00:0{i}.000Z', time=5, cache={}, timings=dict(send=0, wait=5, receive=0), request=request, response=dict(status=201 if method == 'POST' else 200, statusText='OK', httpVersion='HTTP/1.1', cookies=[], headers=[], redirectURL='', headersSize=-1, bodySize=len(text), content=dict(size=len(text), mimeType='application/json', text=text)))) Path(mode + '.har').write_text(json.dumps(dict(log=dict(version='1.2', creator=dict(name='fixture', version='1'), entries=entries)))) request = dict(source=dict(kind='evidence-file', path=str(Path(mode + '-bundle/envs/items/evidence.json').resolve())), view=dict(kind='environment-state', template_rules=['after-uid-literal', 'opaque-token', 'sibling-variance', 'prefix-variance'])) Path(mode + '.request.json').write_text(json.dumps(request)) PY for mode in sparse complete no-clock; do node "$repo/scripts/rea.mjs" environment-build "$mode.har" "$mode-bundle" --name items --json > "$mode.build.json" node "$repo/scripts/rea.mjs" inspect-analysis-view "$mode.request.json" --json > "$mode.state.json" done python3 - <<'PY' import json for mode in ['sparse', 'complete', 'no-clock']: state = json.load(open(mode + '.state.json'))['normalized_result'] plan = next(w for w in state['writes'] if w['tool'] == 'post_items') print(mode, json.dumps({k: plan[k] for k in ['served', 'insert', 'checked_record_ordinals']})) PY ``` Observed output: ```text sparse {"served": true, "insert": {"rule": "last"}, "checked_record_ordinals": [5]} complete {"served": true, "insert": {"rule": "sorted", "field": "id", "direction": "ascending"}, "checked_record_ordinals": [5]} no-clock {"served": true, "insert": {"rule": "sorted", "field": "id", "direction": "ascending"}, "checked_record_ordinals": [5]} ``` I also ran the equivalent three-member fixtures on the installed published package. Each sparse case produced the same served `last` plan. A further fixture containing a final whole list in ascending ID order still retained `last` when the captured new ID was larger than all existing IDs. ### Expected behavior Missing or incomparable existing clock values should not establish that IDs contradict creation order. Keep the consistent identity sort eligible at its normal priority unless actual comparable clock values show an inversion, or the IDs have the explicitly recognized random UUID form. Expected source: [`randomIdentities` at the tested revision](https://github.com/morluto/rea/blob/c3647ade1e557c4179a8b825a37e0c2193cd2526/src/domain/analysisView/environmentStateWriteExplain.ts#L703-L731) defines the non-v4 case by every creation clock showing another order. Its [`insertCandidates` contract](https://github.com/morluto/rea/blob/c3647ade1e557c4179a8b825a37e0c2193cd2526/src/domain/analysisView/environmentStateWriteExplain.ts#L733-L764) puts random identities after append. A missing value establishes neither order nor an inversion. ### Impact and scope P2: bounded incorrect inference changes the live local environment's entity order. The verified class is generated non-sequence, non-v4 identities, inferred creation clocks, and omitted existing clock values that cannot establish inversion. Existing integer sequence IDs bypass this check. V4 UUIDs legitimately receive the random classification. All 18 HTTP MCP creates across the two runtimes and three fixtures succeeded. Sparse environments became unsorted; both controls remained sorted. No real application's capture prevalence, broader task failure rate, data destruction, or security effect was established. A practical workaround preserving sparse recorded data was not established. An all-clocks-absent variant is withheld by published 6.8.0 and is not needed to establish the reported served failure. ### REA version Baseline source: `c3647ade1e557c4179a8b825a37e0c2193cd2526`, package metadata 6.8.0. Also reproduced with npm-published `rea-agents@6.8.0`, which `npm view rea-agents version` returned as the latest published release. The relevant inference and simulation files are unchanged on fetched main `71f680d7c1aa7a45ba0325fe87b104744b0c105f`; main was inspected, not separately run. ### Runtime and platform Node v24.22.1, macOS 27.2, arm64. No native analysis provider is involved. ### Target and artifact context Small synthetic HAR 1.2 captures of an ordinary JSON collection API: an initial list, item reads, a create response, and an item read confirming that create. Names are deliberately not monotonic, so they do not introduce an alternate scalar sort. ### Interface CLI The learned environment was additionally exercised through its real HTTP MCP server and data plane. ### Earliest failure stage Transport or result projection The first observable defect is the projected environment-state create plan's `insert` rule. ### Relevant output or evidence The CLI reproduction above returns exit 0 for all builds and projections. The sparse plan is served and confirmed by read ordinal 5, yet selects `last` instead of the eligible ascending ID sort. For the served check, I started the generated manifest with `environment-serve <manifest.json> --evidence <evidence.json> --port 0 --json`, connected with the MCP client, called `post_items` three times with `{"body":{"name":"Echo"}}`, and fetched `data/get` from the environment's data plane. On both runtimes, the sparse fixture's resulting entity IDs were: ```text 6ad47b491023b50001087124 6ad47b491023b50001087125 6ad47b491023b50001087126 e6c1d4f1586aae4e13052cff 1c78555ff498d7bfe90f2721 dedb2c6a6923260792ca0e3c ``` The second generated ID belongs before every initial ID but was appended. Complete-clock and no-clock controls remained sorted after three creates each. These observations establish served ordering for these fixtures, not a real application incident. Each listener shut down with handled signal status 143. The existing focused tests also passed: `npm run test:focused -- src/domain/analysisView/environmentStateWriteExplain.test.ts src/domain/analysisView/environmentStateWrites.test.ts` — 250 tests in two files. ### Typed outcome or recovery expected Preserve the eligible `{"rule":"sorted","field":"id","direction":"ascending"}` insertion plan for the sparse fixture. Missing creation-clock comparisons should supply no positive evidence that the identity is random. ### Analysis or suspected cause Source-derived reasoning: [`compareValues`](https://github.com/morluto/rea/blob/c3647ade1e557c4179a8b825a37e0c2193cd2526/src/domain/analysisView/environmentStateWriteExplain.ts#L666-L681) returns `undefined` for missing values. `randomIdentities` converts it to positive one with `?? 1`, satisfying its inversion test. [`buildCreate`](https://github.com/morluto/rea/blob/c3647ade1e557c4179a8b825a37e0c2193cd2526/src/domain/analysisView/environmentStateWriteCreate.ts#L420-L442) passes clocks inferred from the created entity while comparing existing members, which need not contain those clocks. The false random classification moves identity sorting after append. A later whole-list validator can retry insertion candidates when placement fails; it does not universally restore the sort. Item reads do not establish list placement, and a captured create falling last agrees with append. Actual serving subsequently consumes the selected plan in [`environmentStateWriteAnswer`](https://github.com/morluto/rea/blob/c3647ade1e557c4179a8b825a37e0c2193cd2526/src/application/environment/environmentStateWriteAnswer.ts#L620-L635). No code changes were made. ### Additional context I searched open and closed issues and PRs for the helper name, creation clocks, missing clocks, sparse fields, identity sorting, random identities, and insertion. No exact duplicate was found. The closed [submillisecond timestamp sorting issue #6525](https://github.com/morluto/rea/issues/6525) concerns a different comparator precision defect. Merged [PR #7600](https://github.com/morluto/rea/pull/7600) adds later-list sort learning; that mechanism is present in the tested code and does not prevent this sparse-clock case.

bsakaguchi commented

2026-11-05 02:37:35 +00:00

Author

Owner

@xu91102 commented on GitHub (Nov 5, 2026):

I reproduced this independently on fresh upstream 71f680d7c1aa7a45ba0325fe87b104744b0c105f (Node 24.22.1, macOS arm64) before making any implementation changes.

Using four ordinary HAR fixtures through the compiled public environment-build and inspect-analysis-view CLI commands (8 successful CLI calls):

Existing creation clocks	Served `post_items` insertion
Only the first existing member has `created_at`	`last`
All existing clocks increase with the IDs	`sorted`, `id`, ascending
Neither existing nor created members have clocks	`sorted`, `id`, ascending
Existing clocks genuinely decrease as IDs increase	`sorted`, `created_at`, descending

All four writes were served and checked against capture ordinal 5; all original HAR SHA-256 digests remained unchanged. This independently confirms the sparse-clock plan defect and retains a genuine inverse-clock control. This probe did not yet exercise live HTTP serving, a real captured application, other hosts, or performance.

I read the relevant production changes from #7600: their later whole-list sort learning is already present in this baseline and does not change the observed result. Searches for this issue, randomIdentities, sparse clocks and environment sort/clock work found no public same-root PR at the time checked. I am preparing a focused fix at the existing identity-order evidence owner, preserving UUID-v4 handling, genuine inversions, and existing write validation.

@xu91102 commented on GitHub (Nov 5, 2026): I reproduced this independently on fresh upstream `71f680d7c1aa7a45ba0325fe87b104744b0c105f` (Node 24.22.1, macOS arm64) before making any implementation changes. Using four ordinary HAR fixtures through the compiled public `environment-build` and `inspect-analysis-view` CLI commands (8 successful CLI calls): | Existing creation clocks | Served `post_items` insertion | | --- | --- | | Only the first existing member has `created_at` | `last` | | All existing clocks increase with the IDs | `sorted`, `id`, ascending | | Neither existing nor created members have clocks | `sorted`, `id`, ascending | | Existing clocks genuinely decrease as IDs increase | `sorted`, `created_at`, descending | All four writes were served and checked against capture ordinal 5; all original HAR SHA-256 digests remained unchanged. This independently confirms the sparse-clock plan defect and retains a genuine inverse-clock control. This probe did not yet exercise live HTTP serving, a real captured application, other hosts, or performance. I read the relevant production changes from #7600: their later whole-list sort learning is already present in this baseline and does not change the observed result. Searches for this issue, `randomIdentities`, sparse clocks and environment sort/clock work found no public same-root PR at the time checked. I am preparing a focused fix at the existing identity-order evidence owner, preserving UUID-v4 handling, genuine inversions, and existing write validation.

No milestone

No project

No assignees

1 participant

Notifications

Due date

The due date is invalid or out of range. Please use the format "yyyy-mm-dd".

No due date set.

Dependencies

No dependencies set.

Reference

morluto/rea#7866

No description provided.

Rows
Columns