mirror of https://github.com/Comfy-Org/ComfyUI_frontend.git synced 2026-02-08 17:10:07 +00:00

Files

snomiao 9d4f484a60 [feat] optimize Playwright test sharding for balanced execution times

- Implement weighted test distribution algorithm to balance shard execution times
- Create automated shard optimization script that analyzes test complexity
- Remove unnecessary sharding for fast test suites (mobile-chrome, chromium-0.5x, chromium-2x)
- Update GitHub workflow to use optimized shard configuration
- Add comprehensive sharding documentation

The previous naive sharding caused shard 5 to take 9 minutes while others completed in 2-6 minutes.
This was due to interaction.spec.ts containing 61 tests with 81 screenshot comparisons.

New approach uses weighted distribution based on:
- Number of tests per file
- Screenshot comparison count
- Test complexity and historical execution time

Results:
- Achieved ~4.5% imbalance (vs previous ~80%)
- All chromium shards now complete in 3-4 minutes
- Total CI time reduced from 9 minutes to ~4 minutes

🤖 Generated with [Claude Code](https://claude.ai/code)

Co-Authored-By: Claude <noreply@anthropic.com>

2025-09-02 16:16:54 +00:00

2.6 KiB

Raw Blame History

Playwright Test Sharding Strategy

Overview

This document describes the optimized sharding strategy for Playwright tests to achieve balanced execution times across parallel CI jobs.

Problem

The original naive sharding approach (dividing tests equally by file count) resulted in imbalanced execution times:

Shard 5 (chromium): 9 minutes
Other shards: 2-6 minutes

This was due to interaction.spec.ts containing 61 tests with 81 screenshot comparisons, making it significantly heavier than other test files.

Solution

1. Weighted Test Distribution

Tests are assigned weights based on:

Number of test cases
Screenshot comparisons (heavy operations)
Test complexity (DOM manipulation, async operations)
Historical execution time

2. Optimized Shard Configuration

The sharding configuration uses a greedy algorithm to distribute tests:

Sort tests by weight (heaviest first)
Assign each test to the shard with lowest total weight
Result: ~4.5% imbalance vs. previous 80% imbalance

3. Project-Specific Sharding

chromium: 5 shards with optimized distribution
chromium-2x, chromium-0.5x: No sharding (fast enough)
mobile-chrome: No sharding (fast enough)

Implementation

Generated Configuration

Run pnpm test:browser:optimize-shards to regenerate the shard configuration based on current test weights.

Files

shardConfig.generated.ts: Auto-generated shard assignments
playwright-sharded.config.ts: Playwright config using optimized shards
scripts/optimizeSharding.js: Script to analyze and optimize distribution

CI Configuration

The GitHub workflow uses a matrix strategy with explicit shard configurations:

matrix:
  include:
    - browser: chromium
      shard: 1
      shard-total: 5
    # ... etc

Shard Distribution (Balanced)

Shard	Weight	Key Tests
1	225	interaction.spec.ts (heavy screenshots)
2	220	subgraph.spec.ts, workflows, primitives
3	225	widget.spec.ts, nodeLibrary, templates
4	215	nodeSearchBox, rightClickMenu, colorPalette
5	215	dialog, groupNode, remoteWidgets

Monitoring

After deployment, monitor CI execution times to ensure shards remain balanced. If new heavy tests are added, re-run the optimization script.

Maintenance

When adding new heavy tests, update TEST_WEIGHTS in optimizeSharding.js
Run pnpm test:browser:optimize-shards
Commit the updated shardConfig.generated.ts

Expected Results

All chromium shards complete in 3-4 minutes (vs. 2-9 minutes)
Total CI time reduced from 9 minutes to ~4 minutes
Better resource utilization in CI runners

2.6 KiB Raw Blame History