Files
ComfyUI_frontend/apps/website
Christian Byrne 167a1e6a0c fix(website): override X-Robots-Tag to index,follow on production deployment (#11831)
## P0 SEO fix — entire marketing site is currently `noindex`

A deep-audit sweep on **2026-05-02** confirmed that every page on
`comfy.org` (home, `/about`, `/pricing`, `/customers`, `/careers`, …) is
being served with:

```
x-robots-tag: noindex
```

This hides the entire marketing site from Google and other search
engines.

### Root cause

`comfy.org` reaches the Astro marketing site via a Cloudflare Worker
reverse proxy
([`comfy-router`](https://github.com/Comfy-Org/comfy-router)) that does:

```js
fetch('https://website-frontend-comfyui.vercel.app/...')
```

Per Vercel's KB article [Avoiding duplicate-content with vercel.app
URLs](https://vercel.com/kb/guide/avoiding-duplicate-content-with-vercel-app-urls):

> By default, Vercel adds an `X-Robots-Tag: noindex` HTTP header to all
deployments hosted on `vercel.app` URLs. […] This header tells search
engines like Google not to index these deployment URLs.

Because `website-frontend-comfyui.vercel.app` is **not** registered as a
Production custom domain on the Vercel project, Vercel applies that
header — and our Worker faithfully forwards it back to `comfy.org`
clients (and Googlebot).

### Fix

Vercel's documented workaround for proxied `*.vercel.app` setups is to
override the header via `vercel.json` ([blog post by Dan
Denney](https://www.dandenney.com/posts/front-end-dev/fixing-x-robots-nofollow-with-vercel/)):

```json
"headers": [
  {
    "source": "/(.*)",
    "has": [
      { "type": "host", "value": "website-frontend-comfyui.vercel.app" }
    ],
    "headers": [
      { "key": "X-Robots-Tag", "value": "index, follow" }
    ]
  }
]
```

### Why the `has` host filter

Critical scoping detail: without the `has` filter, the override would
also apply to PR-preview deployments at
`comfy-website-preview-pr-N.vercel.app`. We want previews to **stay**
`noindex` to avoid duplicate-content competition with production.

The filter pins the override to the production deployment hostname only.
The hostname matches `WEBSITE_ORIGIN` in `comfy-router/src/index.js`:

```js
const WEBSITE_ORIGIN = 'https://website-frontend-comfyui.vercel.app';
```

### Defense in depth

A parallel PR on `comfy-router` (TBD) will also strip `X-Robots-Tag:
noindex` at the Worker layer, so the public `comfy.org` response is
correct even if a future Vercel project change re-introduces the
upstream header.

### Verification (after merge + Vercel production deploy)

```bash
# Production should no longer be noindex
curl -sI https://comfy.org/ | grep -i x-robots-tag
curl -sI https://comfy.org/about | grep -i x-robots-tag
curl -sI https://comfy.org/pricing | grep -i x-robots-tag
# Expect: empty output, OR "x-robots-tag: index, follow"

# Direct Vercel production hostname
curl -sI https://website-frontend-comfyui.vercel.app/ | grep -i x-robots-tag
# Expect: "x-robots-tag: index, follow"

# Preview deploys must stay noindex (proves the host filter works)
curl -sI https://comfy-website-preview-pr-<N>.vercel.app/ | grep -i x-robots-tag
# Expect: "x-robots-tag: noindex"
```

### Pre-merge state (for the record)

```
$ curl -sI https://comfy.org/ | grep -iE 'x-robots-tag|x-served-by'
x-served-by: vercel-website
x-robots-tag: noindex
```

### Scope

Minimum delta — only `apps/website/vercel.json`. No other files touched.

┆Issue is synchronized with this [Notion
page](https://app.notion.com/p/PR-11831-fix-website-override-X-Robots-Tag-to-index-follow-on-production-deployment-3546d73d365081489b62e6633d25dfe5)
by [Unito](https://www.unito.io)

---------

Co-authored-by: Christian Byrne <christian@comfy.org>
Co-authored-by: GitHub Action <action@github.com>
2026-05-01 23:03:25 -07:00
..

@comfyorg/website

Marketing/brand website built with Astro + Vue.

Ashby careers integration

/careers and /zh-CN/careers are rendered from Ashby's public job board API at build time. Data flow:

  1. src/pages/careers.astro awaits fetchRolesForBuild() during the Astro build.
  2. src/utils/ashby.ts calls GET https://api.ashbyhq.com/posting-api/job-board/{board}?includeCompensation=false, validates the envelope and each posting with Zod (src/utils/ashby.schema.ts), and maps to the domain type in src/data/roles.ts.
  3. On any failure (network, HTTP 4xx/5xx, envelope schema drift), the fetcher falls back to the committed JSON snapshot at src/data/ashby-roles.snapshot.json.
  4. src/utils/ashby.ci.ts emits GitHub Actions annotations and a $GITHUB_STEP_SUMMARY block so stale fetches are visible on green builds.

Required environment variables

Both are build-time only. Never prefix with PUBLIC_ (Astro would inline that into the client bundle).

Name Purpose Default (when unset)
WEBSITE_ASHBY_API_KEY Ashby API key (Basic auth) Build uses the committed snapshot
WEBSITE_ASHBY_JOB_BOARD_NAME Ashby public job board slug Build uses the committed snapshot

CI wiring (manual step — required)

This repo's .github/workflows/*.yaml changes cannot be pushed by a GitHub App. A maintainer must apply the following edits once:

.github/workflows/ci-website-build.yaml — pass the env into the build step and run the unit tests before it:

jobs:
  build:
    runs-on: ubuntu-latest
    steps:
      - uses: actions/checkout@v6
      - name: Setup frontend
        uses: ./.github/actions/setup-frontend

      - name: Run website unit tests
        run: pnpm --filter @comfyorg/website test:unit

      - name: Build website
        env:
          WEBSITE_ASHBY_API_KEY: ${{ secrets.WEBSITE_ASHBY_API_KEY }}
          WEBSITE_ASHBY_JOB_BOARD_NAME: ${{ vars.WEBSITE_ASHBY_JOB_BOARD_NAME || 'comfy-org' }}
        run: pnpm --filter @comfyorg/website build

      - name: Verify API key is not leaked into build output
        env:
          WEBSITE_ASHBY_API_KEY: ${{ secrets.WEBSITE_ASHBY_API_KEY }}
        run: |
          set +x
          if [ -z "${WEBSITE_ASHBY_API_KEY:-}" ]; then
            echo "Secret not available in this run; skipping leak check."
            exit 0
          fi
          # grep -rlF prints only file paths (never match content).
          MATCHES=$(grep -rlF --exclude-dir=node_modules --null \
            -e "$WEBSITE_ASHBY_API_KEY" apps/website/dist/ 2>/dev/null \
            | tr '\0' '\n' || true)
          if [ -n "$MATCHES" ]; then
            echo "::error title=Ashby API key leaked into build output::$MATCHES"
            exit 1
          fi

.github/workflows/ci-vercel-website-preview.yaml — add the two env vars to the top-level env: block so vercel build (both deploy-preview and deploy-production jobs) sees them:

env:
  VERCEL_ORG_ID: ${{ secrets.VERCEL_WEBSITE_ORG_ID }}
  VERCEL_PROJECT_ID: ${{ secrets.VERCEL_WEBSITE_PROJECT_ID }}
  VERCEL_TOKEN: ${{ secrets.VERCEL_WEBSITE_TOKEN }}
  VERCEL_SCOPE: comfyui
  WEBSITE_ASHBY_API_KEY: ${{ secrets.WEBSITE_ASHBY_API_KEY }}
  WEBSITE_ASHBY_JOB_BOARD_NAME: ${{ vars.WEBSITE_ASHBY_JOB_BOARD_NAME || 'comfy-org' }}

The secret must also be added to the Vercel project environment (vercel env add WEBSITE_ASHBY_API_KEY … or via the Vercel UI) so that vercel build in the preview job has access to it.

Fork PRs do not exercise this path: ci-vercel-website-preview.yaml receives an empty VERCEL_TOKEN for forks and fails at vercel pull before the build runs. Fork-safe PR interactions (the preview-URL comment) are handled by pr-vercel-website-preview.yaml.

Refreshing the snapshot

When a maintainer wants to update the committed snapshot (e.g. after onboarding/offboarding roles):

WEBSITE_ASHBY_API_KEY=WEBSITE_ASHBY_JOB_BOARD_NAME=comfy-org \
  pnpm --filter @comfyorg/website ashby:refresh-snapshot
git commit apps/website/src/data/ashby-roles.snapshot.json

The script exits non-zero on any non-fresh outcome so stale/empty snapshots can't be accidentally committed.

HubSpot contact form

The contact page uses HubSpot's hosted form embed for the interest form:

<script
  src="https://js-na2.hsforms.net/forms/embed/developer/244637579.js"
  defer
></script>
<div
  class="hs-form-html"
  data-region="na2"
  data-form-id="94e05eab-1373-47f7-ab5e-d84f9e6aa262"
  data-portal-id="244637579"
></div>

The localized /zh-CN/contact page uses the same portal and script with form ID 6885750c-02ef-4aa2-ba0d-213be9cccf93.

This keeps submission handling, validation, anti-spam updates, and field configuration in HubSpot. The local implementation in src/components/contact/HubspotFormEmbed.vue only loads the hosted script and renders the documented embed container.

Scripts

  • pnpm dev — Astro dev server
  • pnpm build — production build to dist/
  • pnpm typecheckastro check
  • pnpm test:unit — Vitest unit tests
  • pnpm test:e2e — Playwright E2E tests (requires pnpm build first)
  • pnpm ashby:refresh-snapshot — refresh the committed careers snapshot