Comfy-Org/desktop

Fork 0

mirror of https://github.com/Comfy-Org/desktop.git synced 2026-11-04 23:47:43 +00:00

[GH-ISSUE #900] [DevTask] Separate e2e tests - install & post-install #248

New issue

Closed

opened 2026-10-31 22:10:34 +00:00 by bsakaguchi · 6 comments

bsakaguchi commented

2026-10-31 22:10:34 +00:00

Owner

Originally created by @webfiltered on GitHub (Feb 14, 2025).
Original GitHub issue: https://github.com/Comfy-Org/desktop/issues/900

End-to-end tests currently clean state on exit when the test option is set. There is no separation or grouping of install tests and

Tests that need to test the install process should all start from a blank state, and always dispose their own changes
Tests for post-install desktop features should share a single install process, which disposes only changes made by the test

Playwright project, or simply a different setup / config file & test directory may be used for this.

┆Issue is synchronized with this Notion page by Unito

Originally created by @webfiltered on GitHub (Feb 14, 2025). Original GitHub issue: https://github.com/Comfy-Org/desktop/issues/900 End-to-end tests currently clean state on exit when the test option is set. There is no separation or grouping of install tests and - [x] Tests that need to test the install process should all start from a blank state, and always dispose their own changes - [ ] Tests for post-install desktop features should share a single install process, which disposes only changes made by the test Playwright project, or simply a different setup / config file & test directory may be used for this. ┆Issue is synchronized with this [Notion page](https://www.notion.so/Issue-900-DevTask-Separate-e2e-tests-install-post-install-19a6d73d365081c7be8ede44c6f1d95e) by [Unito](https://www.unito.io)

bsakaguchi closed this issue

2026-10-31 22:10:34 +00:00

bsakaguchi commented

2026-10-31 22:10:35 +00:00

Author

Owner

@christian-byrne commented on GitHub (Feb 14, 2025):

Separate setups with their own workflows is cleanest but would be wasteful. What about just 2 projects with tests selected by @tags in test descriptions, and configured such that the post-install projects runs second?

@christian-byrne commented on GitHub (Feb 14, 2025): Separate setups with their own workflows is cleanest but would be wasteful. What about just 2 projects with tests selected by `@tags` in test descriptions, and configured such that the post-install projects runs second?

bsakaguchi commented

2026-10-31 22:10:35 +00:00

Author

Owner

@webfiltered commented on GitHub (Feb 14, 2025):

I need to clean up the test classes still - forgot to move them out of tests/integration. Everything apart from the spec files and setup / config would be shared.

Does Playwright guarantee projects will be run in a certain order? My local Playwright ran tests in a different order to CI. I think it might look like it has an order but just isn't guaranteed anywhere (version / env change could cause flakiness, etc).

Also I thought the projects + @tags thing was designed to do multiple runs of the same thing with different browsers. I had a look at it a day or two ago but it didn't seem right. As in, it looked possible, just kind of hacky compared to just dropping the spec files into different subdirs.

@webfiltered commented on GitHub (Feb 14, 2025): I need to clean up the test classes still - forgot to move them out of `tests/integration`. Everything apart from the spec files and setup / config would be shared. Does Playwright guarantee projects will be run in a certain order? My local Playwright ran tests in a different order to CI. I think it might look like it has an order but just isn't guaranteed anywhere (version / env change could cause flakiness, etc). Also I thought the projects + @tags thing was designed to do multiple runs of the same thing with different browsers. I had a look at it a day or two ago but it didn't seem right. As in, it looked possible, just kind of hacky compared to just dropping the spec files into different subdirs.

bsakaguchi commented

2026-10-31 22:10:35 +00:00

Author

Owner

@christian-byrne commented on GitHub (Feb 14, 2025):

Does Playwright guarantee projects will be run in a certain order?

Yep, there is the option to specify projects as being dependencies of one another to support our exact use case: https://playwright.dev/docs/test-projects#dependencies

"When working with tests that have a dependency, the dependency will always run first and once all tests from this project have passed, then the other projects will run in parallel."

Also I thought the projects + @tags thing was designed to do multiple runs of the same thing with different browsers. I had a look at it a day or two ago but it didn't seem right. As in, it looked possible, just kind of hacky compared to just dropping the spec files into different subdirs.

It does feel hacky. Splitting based on file/subdir could be better. I was thinking there might be tests that want to be run in both pre-install and post-install e.g., the window open/close tests.

@christian-byrne commented on GitHub (Feb 14, 2025): > Does Playwright guarantee projects will be run in a certain order? Yep, there is the option to specify projects as being dependencies of one another to support our exact use case: https://playwright.dev/docs/test-projects#dependencies "When working with tests that have a dependency, the dependency will always run first and once all tests from this project have passed, then the other projects will run in parallel." > Also I thought the projects + `@tags` thing was designed to do multiple runs of the same thing with different browsers. I had a look at it a day or two ago but it didn't seem right. As in, it looked possible, just kind of hacky compared to just dropping the spec files into different subdirs. It does feel hacky. Splitting based on file/subdir could be better. I was thinking there might be tests that want to be run in both pre-install and post-install e.g., the window open/close tests.

bsakaguchi commented

2026-10-31 22:10:35 +00:00

Author

Owner

@webfiltered commented on GitHub (Feb 14, 2025):

Excellent. Four projects, three subdirs. One project for a shared test dir. One that is just the one file - setup for post-install tests. It will use the uv cache so will take ~30sec. Less brittle that way.

More test files is good - especially as we can only hope to parallelise by sharding (at a much later date).

@webfiltered commented on GitHub (Feb 14, 2025): Excellent. Four projects, three subdirs. One project for a shared test dir. One that is just the one file - setup for post-install tests. It will use the uv cache so will take ~30sec. Less brittle that way. More test files is good - especially as we can only hope to parallelise by sharding (at a much later date).

bsakaguchi commented

2026-10-31 22:10:35 +00:00

Author

Owner

@christian-byrne commented on GitHub (Feb 14, 2025):

So the purpose of having a setup for post-install project is that the installation tests will (in the future) teardown installation files?

@christian-byrne commented on GitHub (Feb 14, 2025): So the purpose of having a `setup for post-install` project is that the installation tests will (in the future) teardown installation files?

bsakaguchi commented

2026-10-31 22:10:35 +00:00

Author

Owner

@christian-byrne commented on GitHub (Feb 14, 2025):

How about this?

  projects: [
    {
      name: 'install',
      testMatch: ['install/**/*.spec.ts', 'shared/**/*.spec.ts'],
      use: {
        screenshot: 'only-on-failure',
        trace: 'on-first-retry',
        video: 'on-first-retry',
      },

    },
    {
      name: 'post-install-setup',
      testMatch: ['post-install-setup/**/*.spec.ts'],
      dependencies: ['install'],
    },
    {
      name: 'post-install',
      testMatch: ['post-install/**/*.spec.ts', 'shared/**/*.spec.ts'],
      dependencies: ['post-install-setup', 'install'],
      use: {
        screenshot: 'only-on-failure',
        trace: 'on-first-retry',
        video: 'on-first-retry',
      },
    },
  ],

@christian-byrne commented on GitHub (Feb 14, 2025): How about this? ```typescript projects: [ { name: 'install', testMatch: ['install/**/*.spec.ts', 'shared/**/*.spec.ts'], use: { screenshot: 'only-on-failure', trace: 'on-first-retry', video: 'on-first-retry', }, }, { name: 'post-install-setup', testMatch: ['post-install-setup/**/*.spec.ts'], dependencies: ['install'], }, { name: 'post-install', testMatch: ['post-install/**/*.spec.ts', 'shared/**/*.spec.ts'], dependencies: ['post-install-setup', 'install'], use: { screenshot: 'only-on-failure', trace: 'on-first-retry', video: 'on-first-retry', }, }, ], ```

bsakaguchi referenced this issue

2026-10-31 22:16:18 +00:00

[PR #248] [MERGED] Install Python using uv. #723

No milestone

No project

No assignees

1 participant

Notifications

Due date

The due date is invalid or out of range. Please use the format "yyyy-mm-dd".

No due date set.

Dependencies

No dependencies set.

Reference

Comfy-Org/desktop#248

No description provided.

Rows
Columns