Compare commits

...

2 Commits

Author SHA1 Message Date
bymyself
0f285aaefe fix: address review — deterministic dates, use collection length 2026-03-28 00:59:17 -07:00
bymyself
04b4f24164 spike: evaluate FakerJS for deterministic test data generation 2026-03-28 00:56:05 -07:00
6 changed files with 711 additions and 526 deletions

View File

@@ -121,6 +121,7 @@
},
"devDependencies": {
"@eslint/js": "catalog:",
"@faker-js/faker": "catalog:",
"@intlify/eslint-plugin-vue-i18n": "catalog:",
"@lobehub/i18n-cli": "catalog:",
"@nx/eslint": "catalog:",

1142
pnpm-lock.yaml generated

File diff suppressed because it is too large Load Diff

View File

@@ -7,6 +7,7 @@ catalog:
'@astrojs/vue': ^5.0.0
'@comfyorg/comfyui-electron-types': 0.6.2
'@eslint/js': ^9.39.1
'@faker-js/faker': ^10.4.0
'@formkit/auto-animate': ^0.9.0
'@iconify-json/lucide': ^1.1.178
'@iconify/json': ^2.2.380

View File

@@ -0,0 +1,34 @@
# FakerJS Evaluation
## Seed Stability
- Are outputs identical across runs with same seed? **Partially** — all fields except `faker.date.recent()` are identical. Date-based generators use `new Date()` as a reference point, so they shift between runs even with the same seed.
- Do other fields change when schema changes? **Yes** — inserting a new faker call before existing calls shifts the PRNG sequence, causing all downstream fields to produce different values. This is the single-stream RNG problem.
## Schema Mutation Test Results
Adding a field (`author: faker.person.fullName()`) before `type` in a 5-record array:
- Record 0: `name` and `size` stable, `type` changed
- Records 14: `name`, `size`, and `type` all changed
This confirms that faker uses a single global PRNG stream. Any insertion/deletion of a faker call shifts all subsequent outputs.
## Pros
- Rich API for realistic test data (names, files, paths, numbers, lorem text)
- Seeded output is deterministic within a single schema version (non-date fields)
- Good for generating bulk data (e.g., 50 items for infinite scroll testing)
- Lightweight, well-maintained, no native dependencies
- Already available via pnpm catalog
## Cons
- **Schema fragility**: Adding, removing, or reordering faker calls in a generator shifts all downstream values. This will invalidate screenshot goldens whenever fixture schemas change.
- **Date instability**: `faker.date.recent()`, `faker.date.past()`, etc. use wall-clock time as a reference, making them non-deterministic even with a seed. Must use `faker.date.between()` with fixed boundaries instead.
- **No per-field isolation**: Unlike property-based testing libraries, faker has a single PRNG stream. There is no built-in way to isolate individual fields from schema changes.
- **Mitigation complexity**: Workarounds exist (per-record re-seeding, `faker.seed(baseIndex + i)` per item) but add boilerplate and reduce readability.
## Recommendation
- **Use** for generating bulk test data in unit/integration tests where screenshot stability is not required (e.g., testing infinite scroll behavior, list rendering, search/filter logic).
- **Do not use** for screenshot/golden-image test fixtures — schema changes will cascade into visual diffs. For those, prefer hand-crafted or static JSON fixtures.
- **If adopted**, establish these conventions:
1. Always use `faker.date.between({ from: '2024-01-01', to: '2024-12-31' })` instead of `faker.date.recent()`.
2. Consider per-record seeding (`faker.seed(BASE_SEED + index)`) to isolate records from each other when schema stability matters.
3. Pin the `@faker-js/faker` major version to avoid cross-version PRNG changes.

View File

@@ -0,0 +1,32 @@
/* eslint-disable no-console */
import { faker } from '@faker-js/faker'
// Original schema
faker.seed(12345)
const original = Array.from({ length: 5 }, () => ({
name: faker.system.fileName(),
size: faker.number.int({ min: 1000, max: 10000000 }),
type: faker.helpers.arrayElement(['image', 'model', 'checkpoint']),
}))
// Modified schema — extra field inserted BEFORE 'type'
faker.seed(12345)
const modified = Array.from({ length: 5 }, () => ({
name: faker.system.fileName(),
size: faker.number.int({ min: 1000, max: 10000000 }),
author: faker.person.fullName(), // NEW FIELD inserted before 'type'
type: faker.helpers.arrayElement(['image', 'model', 'checkpoint']),
}))
console.log('=== Schema mutation stability test ===')
for (let i = 0; i < original.length; i++) {
const nameMatch = original[i].name === modified[i].name
const sizeMatch = original[i].size === modified[i].size
const typeMatch = original[i].type === modified[i].type
console.log(
`Record ${i}: name=${nameMatch ? 'STABLE' : 'CHANGED'} size=${sizeMatch ? 'STABLE' : 'CHANGED'} type=${typeMatch ? 'STABLE' : 'CHANGED'}`
)
if (!nameMatch) console.log(` name: "${original[i].name}" -> "${modified[i].name}"`)
if (!sizeMatch) console.log(` size: ${original[i].size} -> ${modified[i].size}`)
if (!typeMatch) console.log(` type: "${original[i].type}" -> "${modified[i].type}"`)
}

27
temp/scripts/faker-poc.ts Normal file
View File

@@ -0,0 +1,27 @@
/* eslint-disable no-console */
import { faker } from '@faker-js/faker'
// Seed for determinism
faker.seed(12345)
// Generate 10 mock asset records
const assets = Array.from({ length: 10 }, () => ({
name: faker.system.fileName(),
size: faker.number.int({ min: 1000, max: 10000000 }),
type: faker.helpers.arrayElement(['image', 'model', 'checkpoint']),
path: faker.system.filePath(),
modified: faker.date.between({ from: '2024-01-01', to: '2024-12-31' }).toISOString(),
}))
console.log('Assets:', JSON.stringify(assets, null, 2))
// Generate 50 items for infinite scroll
faker.seed(12345) // Re-seed to test stability
const scrollItems = Array.from({ length: 50 }, (_, i) => ({
id: i,
title: faker.lorem.words(3),
description: faker.lorem.sentence(),
}))
console.log(`\nGenerated ${scrollItems.length} scroll items`)
console.log('First 3:', JSON.stringify(scrollItems.slice(0, 3), null, 2))