Nicolas
785d97ba9e
Le run 122 (branche avec l'isolation par conteneur du commit précédent) a
quand même échoué : sa base `postgres-122` était bien isolée et migrée
proprement, mais le run 123 (ancien ci.yml) tournait en parallèle sur le
même runner. Deux jobs `test` concurrents saturent le CPU (chacun entraîne
un modèle spaCy pour son intent-service *et* déroule toute la suite Mocha
DB) : un hook `resetDatabase()` dépasse alors le timeout Mocha de 10 s,
Mocha enchaîne sur le hook suivant pendant que le TRUNCATE + reseed du
précédent tourne encore, et les deux se percutent *dans la base d'un seul
run* (`deadlock detected`, `Unique constraint failed on (key)`,
intent-service affamé → F1 0.000).
L'isolation par base ne suffit donc pas ; il faut « un seul job test à la
fois », ce que `concurrency:` ferait nativement — indisponible ici
(Forgejo 9.0.3). À la place : un verrou via le seul point commun à tous
les jobs du runner, le démon docker. `docker network create ci-test-lock`
échoue si le nom est pris → le premier run le détient, les autres bouclent
jusqu'à libération (file d'attente, pas d'annulation). Libération en étape
`if: always()` gardée par un marqueur `/tmp` (un run qui a renoncé ne doit
pas supprimer le verrou du détenteur) + purge d'un verrou de plus de
~35 min (détenteur tué avant sa libération). `timeout-minutes` passe à 60
pour couvrir l'attente en file. L'isolation `postgres-${GITHUB_RUN_ID}`
reste en défense en profondeur (fenêtre de transition où d'autres branches
sont encore sur l'ancien fichier, sans verrou).
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
339 lines
16 KiB
YAML
339 lines
16 KiB
YAML
name: CI
|
|
|
|
# Every push, on every branch — not just main — so build breakage shows up
|
|
# on the first commit of a branch, not only once a PR targets main. No
|
|
# separate pull_request trigger: for a PR from a branch on this same repo,
|
|
# push already fires on that branch, and GitHub attaches the run to the PR
|
|
# by commit SHA regardless of which event triggered it — adding
|
|
# pull_request too would just run every job twice per push. (Re-add it,
|
|
# scoped to forks, only if this repo starts accepting fork PRs — push
|
|
# events from a fork never reach here.)
|
|
on:
|
|
push:
|
|
|
|
env:
|
|
# Only the `test` job ever opens a database connection, and it *overwrites*
|
|
# this value at runtime (see its "Start Postgres" step) with a per-run
|
|
# hostname. This entry stays for one reason: `build`'s `pnpm install` runs
|
|
# `prisma generate` (apps/api `postinstall`), which wants DATABASE_URL to
|
|
# be present and syntactically valid — it never dials the host, so the
|
|
# literal `postgres` here need not resolve anywhere.
|
|
#
|
|
# Why the `test` job runs Postgres as a plain `docker run` and not a
|
|
# `services:` block: this repo's Forgejo `act` runner does not keep a
|
|
# `services:` container as the *same* instance across a job's steps, so
|
|
# `migrate deploy` would land the schema in one Postgres and `pnpm --filter
|
|
# api test`'s `beforeAll` would then hit an empty one ("table … does not
|
|
# exist" right after "migrations successfully applied"). One long-lived
|
|
# container in the job, shared by every step, removes that failure mode.
|
|
DATABASE_URL: "postgresql://ci:ci@postgres:5432/batchcooking_ci?schema=public"
|
|
# Test-only secret, never used outside CI — real deployments must set their own.
|
|
JWT_SECRET: "ci-only-secret-not-used-anywhere-else-32chars+"
|
|
# Same reasoning as JWT_SECRET above — lets tech-step-worker.routes.test.ts
|
|
# exercise the success path (matching secret), not just the "unset"
|
|
# rejection every environment that doesn't set this gets by default.
|
|
INTERNAL_WORKER_SECRET: "ci-only-worker-secret-not-used-anywhere-else-32chars+"
|
|
# Same reasoning — lets admin-auth.test.ts exercise the admin login
|
|
# success path + the requireAdmin-guarded routes, not just the "no admin
|
|
# secret configured -> 401" path.
|
|
ADMIN_JWT_SECRET: "ci-only-admin-secret-not-used-anywhere-else-32chars+"
|
|
# Shared between the `test` job's own uvicorn step (below) and apps/api's
|
|
# IntentServiceClient — that step runs `uvicorn` as a plain background
|
|
# process (not a `services:` block or even a container) because CI can
|
|
# only pull a published image, not build
|
|
# services/tech-step-intent-service/Dockerfile from this checkout.
|
|
INTENT_SERVICE_BASE_URL: "http://localhost:8000"
|
|
INTENT_SERVICE_SECRET: "ci-only-intent-secret-not-used-anywhere-else-32chars+"
|
|
|
|
jobs:
|
|
# Five independent jobs, no needs: between them — each starts in parallel
|
|
# and reports as its own check, instead of the previous single chained
|
|
# "lint-and-test then e2e" pipeline.
|
|
lint:
|
|
runs-on: ubuntu-latest
|
|
steps:
|
|
- uses: https://github.com/actions/checkout@v4
|
|
|
|
- uses: pnpm/action-setup@v4
|
|
|
|
- uses: https://github.com/actions/setup-node@v4
|
|
with:
|
|
node-version: 22
|
|
cache: pnpm
|
|
|
|
- run: pnpm install --frozen-lockfile
|
|
- run: pnpm lint
|
|
|
|
test:
|
|
runs-on: ubuntu-latest
|
|
# The active job is ~16-19 min (runs 117-121); the rest of the budget is
|
|
# for time spent queued on the serialization lock below (a realistic
|
|
# pile-up is two, sometimes three, runs). Also a runaway ceiling: Mocha
|
|
# runs with no `--exit` (`apps/api/.mocharc.json`), so one leaked handle
|
|
# would otherwise hang the job until the runner's global timeout — a
|
|
# `test` job zombied ~1h once before being cancelled by hand. If a hung
|
|
# holder ever hits this ceiling its `if: always()` "Release the test-job
|
|
# lock" step still frees the queue (and the stale-lock sweep frees it
|
|
# sooner, at ~35 min).
|
|
timeout-minutes: 60
|
|
steps:
|
|
- name: Install curl and the docker CLI
|
|
run: |
|
|
apt-get update && apt-get install -y curl docker.io
|
|
|
|
# ── Serialize the `test` job across every concurrent run ─────────────
|
|
# GitHub's `concurrency:` is the native "only one `test` job at a time,
|
|
# across all branches", but this Forgejo (9.0.3) predates Actions
|
|
# `concurrency` entirely — it landed in Forgejo v14 / Gitea 1.26, and
|
|
# the key is silently ignored on anything older. The one resource every
|
|
# job on this runner shares is the docker daemon, so a uniquely-named
|
|
# docker object is the mutex: `docker network create` fails if the name
|
|
# is taken, so the first run to create `ci-test-lock` holds it and the
|
|
# others spin until it is freed — queue, don't cancel (the task's
|
|
# preferred `cancel-in-progress: false`: a half-finished `main` run
|
|
# shouldn't be killed by a feature push).
|
|
#
|
|
# Why serialize and not just give each run a private database (it gets
|
|
# one too, next step): two `test` jobs on this single runner also
|
|
# contend for CPU — each trains a spaCy model for its own
|
|
# intent-service *and* runs the full Mocha DB suite — and under that
|
|
# load a `resetDatabase()` hook overran Mocha's 10s timeout; Mocha then
|
|
# started the next file's hook while the timed-out one's TRUNCATE +
|
|
# reseed was still in flight, and the two collided inside one run's own
|
|
# database (`deadlock detected`, `Unique constraint failed on (key)` in
|
|
# `seedReferenceData`, intent-service starved to F1 0.000). Run 122
|
|
# failed exactly that way while run 123's `test` job overlapped it.
|
|
- name: Acquire the test-job lock
|
|
run: |
|
|
# Force-release a lock older than ~35 min (well past a normal
|
|
# ~19-min run): its holder was killed outright before the release
|
|
# step could run (SIGKILL skips `if: always()`), or is wedged.
|
|
# `timeout-minutes` on this job is the backstop for a holder that
|
|
# is merely slow. Fail safe: if the age can't be read, leave the
|
|
# lock alone rather than risk nuking a live holder's lock.
|
|
created="$(docker network inspect -f '{{.Created}}' ci-test-lock 2>/dev/null || true)"
|
|
if [ -n "$created" ]; then
|
|
# `.Created` renders as RFC3339 on newer daemons and as Go's
|
|
# "2006-01-02 15:04:05 -0700 MST" on older ones — strip a trailing
|
|
# timezone name so GNU `date -d` accepts either.
|
|
created="$(printf '%s' "$created" | sed 's/ [A-Za-z][A-Za-z]*$//')"
|
|
if created_epoch="$(date -d "$created" +%s 2>/dev/null)"; then
|
|
age=$(( $(date +%s) - created_epoch ))
|
|
if [ "$age" -gt 2100 ]; then
|
|
echo "test-job lock is ${age}s old — assuming abandoned, releasing it"
|
|
docker network rm ci-test-lock >/dev/null 2>&1 || true
|
|
fi
|
|
else
|
|
echo "could not read the test-job lock's age ('$created') — leaving it in place"
|
|
fi
|
|
fi
|
|
# Up to ~40 min of waiting (80 * 30s); past that the stale-release
|
|
# above would have fired for any real hang, so give up and let the
|
|
# re-run get through.
|
|
for i in $(seq 1 80); do
|
|
if docker network create ci-test-lock >/dev/null 2>&1; then
|
|
echo "test-job lock acquired"
|
|
# Marker checked by the release step so a job that gave up
|
|
# waiting (and never held the lock) can't delete another run's.
|
|
# /tmp persists across steps (same job container) and, unlike
|
|
# the workspace, isn't wiped by actions/checkout.
|
|
touch /tmp/ci-test-lock-held
|
|
exit 0
|
|
fi
|
|
echo "waiting for the test-job lock, held by another run… attempt ${i}"
|
|
sleep 30
|
|
done
|
|
echo "could not acquire the test-job lock within ~40 min" >&2
|
|
exit 1
|
|
|
|
# Postgres as a plain background container instead of a `services:`
|
|
# block — see this workflow's top-level `env.DATABASE_URL` comment for
|
|
# why. The runner mounts the docker socket and runs every job on one
|
|
# shared user-defined network, so a `--name` sibling resolves by name;
|
|
# the network is discovered from the job container itself rather than
|
|
# hardcoded.
|
|
#
|
|
# The container name still carries the run id (`postgres-${GITHUB_RUN_ID}`)
|
|
# even though the lock above already keeps `test` jobs from overlapping:
|
|
# defence in depth for the window where another branch is still on the
|
|
# pre-lock version of this file (its `test` job takes no lock and would
|
|
# collide on a bare `--name postgres`), and for a mis-fired stale-lock
|
|
# release. The `if: always()` teardown plus the age-gated reap below
|
|
# keep these per-run containers from piling up on the runner host.
|
|
- name: Start Postgres in the background
|
|
run: |
|
|
pg_host="postgres-${GITHUB_RUN_ID}"
|
|
|
|
# Best-effort reap of per-run Postgres containers left behind by a
|
|
# `test` job that was force-killed before its teardown step could
|
|
# run (SIGKILL skips `if: always()` steps). Age-gated well above
|
|
# this job's `timeout-minutes`, so a slower *concurrent* `test`
|
|
# job's container is never in range.
|
|
reap_before=$(( $(date +%s) - 7200 ))
|
|
for c in $(docker ps -aq --filter "name=postgres-"); do
|
|
started="$(docker inspect -f '{{.State.StartedAt}}' "$c" 2>/dev/null)" || continue
|
|
started_epoch="$(date -d "$started" +%s 2>/dev/null)" || continue
|
|
if [ "$started_epoch" -lt "$reap_before" ]; then
|
|
echo "reaping abandoned Postgres container $c (started $started)"
|
|
docker rm -f "$c" >/dev/null 2>&1 || true
|
|
fi
|
|
done
|
|
|
|
net="$(docker inspect "$(hostname)" \
|
|
--format '{{range $k, $_ := .NetworkSettings.Networks}}{{$k}} {{end}}' \
|
|
| awk '{print $1}')"
|
|
echo "job network: ${net:?could not determine the job container network}"
|
|
docker rm -f "$pg_host" >/dev/null 2>&1 || true
|
|
docker run -d --name "$pg_host" --network "$net" \
|
|
-e POSTGRES_USER=ci -e POSTGRES_PASSWORD=ci -e POSTGRES_DB=batchcooking_ci \
|
|
postgres:16-alpine
|
|
|
|
# Hand this run's hostname to every later step (prisma migrate
|
|
# deploy, pnpm --filter api test) — this line overrides the
|
|
# workflow-level env.DATABASE_URL, whose `postgres` host no longer
|
|
# resolves.
|
|
echo "DATABASE_URL=postgresql://ci:ci@${pg_host}:5432/batchcooking_ci?schema=public" >> "$GITHUB_ENV"
|
|
|
|
for _ in $(seq 1 60); do
|
|
docker exec "$pg_host" pg_isready -U ci -d batchcooking_ci && exit 0
|
|
sleep 2
|
|
done
|
|
echo "Postgres did not become ready in time" >&2
|
|
docker logs "$pg_host" || true
|
|
exit 1
|
|
|
|
- uses: https://github.com/actions/checkout@v4
|
|
|
|
- uses: pnpm/action-setup@v4
|
|
|
|
- uses: https://github.com/actions/setup-node@v4
|
|
with:
|
|
node-version: 22
|
|
cache: pnpm
|
|
|
|
- uses: https://github.com/astral-sh/setup-uv@v5
|
|
with:
|
|
python-version: "3.12"
|
|
enable-cache: true
|
|
|
|
# A `services:` block can only pull an already-published image — it
|
|
# can't build services/tech-step-intent-service/Dockerfile from this
|
|
# checkout. Running `uvicorn` as a plain background step instead: it
|
|
# keeps running for the rest of this job (steps in one job share the
|
|
# same process tree), and `pnpm --filter api test` below needs a real
|
|
# instance to talk to per this repo's "never mock an internal service"
|
|
# test convention — same "real dependency, started by the job itself"
|
|
# posture as the Postgres step above, not a mock HTTP server.
|
|
- name: Install services/tech-step-intent-service
|
|
working-directory: services/tech-step-intent-service
|
|
run: uv sync --frozen
|
|
- name: Start services/tech-step-intent-service in the background
|
|
working-directory: services/tech-step-intent-service
|
|
run: |
|
|
uv run uvicorn intent_service.main:app --host 0.0.0.0 --port 8000 &
|
|
# `/health` only returns 200 once this service has finished
|
|
# training itself from scratch (no model ever persisted to disk —
|
|
# see its own README) — measured at ~3min combined (fr+en) against
|
|
# the current ~74-technique corpus, down from ~930s once
|
|
# `spacy.TextCatBOW.v3`'s `length` stopped being left at its
|
|
# 262144 default (see `_TEXTCAT_MODEL_LENGTH` in
|
|
# `locale_pipeline.py`, and docker-compose.yml's healthcheck for
|
|
# the same reasoning). The timeout stays well above that: it's a
|
|
# ceiling for a slow/contended runner, not a target.
|
|
timeout 600 bash -c 'until curl -sf http://localhost:8000/health > /dev/null; do sleep 2; done'
|
|
|
|
- run: pnpm install --frozen-lockfile
|
|
- run: pnpm --filter api exec prisma migrate deploy
|
|
- run: pnpm --filter api test
|
|
|
|
# Release the serialization lock as early as possible so the next
|
|
# queued `test` job can start — `always()` so a failed or timed-out job
|
|
# frees it too (the stale-lock sweep in "Acquire the test-job lock"
|
|
# only covers a job killed outright). Only if this job actually holds
|
|
# it: a job that gave up waiting must not remove the holder's lock.
|
|
- name: Release the test-job lock
|
|
if: always()
|
|
run: |
|
|
if [ -f /tmp/ci-test-lock-held ]; then
|
|
docker network rm ci-test-lock >/dev/null 2>&1 || true
|
|
fi
|
|
|
|
# Remove this run's Postgres even when a step above failed or the job
|
|
# hit its `timeout-minutes` — with a per-run name nothing else ever
|
|
# reuses or cleans it, and a bare `docker run` sibling outlives the job
|
|
# container. `|| true`: teardown must never itself fail the job. A job
|
|
# killed outright (SIGKILL, no `always()` step) is caught by the
|
|
# age-gated reap in "Start Postgres" on a later run.
|
|
- name: Stop this run's Postgres
|
|
if: always()
|
|
run: docker rm -f "postgres-${GITHUB_RUN_ID}" >/dev/null 2>&1 || true
|
|
|
|
intent-service-test:
|
|
runs-on: ubuntu-latest
|
|
steps:
|
|
- uses: https://github.com/actions/checkout@v4
|
|
|
|
- uses: https://github.com/astral-sh/setup-uv@v5
|
|
with:
|
|
python-version: "3.12"
|
|
enable-cache: true
|
|
|
|
- name: Install services/tech-step-intent-service
|
|
working-directory: services/tech-step-intent-service
|
|
run: uv sync --frozen
|
|
- name: Run pytest
|
|
working-directory: services/tech-step-intent-service
|
|
run: uv run pytest -q
|
|
|
|
build:
|
|
runs-on: ubuntu-latest
|
|
steps:
|
|
- uses: https://github.com/actions/checkout@v4
|
|
|
|
- uses: pnpm/action-setup@v4
|
|
|
|
- uses: https://github.com/actions/setup-node@v4
|
|
with:
|
|
node-version: 22
|
|
cache: pnpm
|
|
|
|
- run: pnpm install --frozen-lockfile
|
|
- run: pnpm build
|
|
|
|
e2e:
|
|
runs-on: ubuntu-latest
|
|
steps:
|
|
- uses: https://github.com/actions/checkout@v4
|
|
|
|
- name: Install Xvfb and Cypress dependencies
|
|
run: |
|
|
apt-get update
|
|
apt-get install -y xvfb libgtk2.0-0 libgtk-3-0 libgbm-dev libnotify-dev libnss3 libxss1 libasound2 libxtst6 xauth dbus-x11
|
|
|
|
- uses: pnpm/action-setup@v4
|
|
|
|
- uses: https://github.com/actions/setup-node@v4
|
|
with:
|
|
node-version: 22
|
|
cache: pnpm
|
|
|
|
- name: Cache Cypress binary
|
|
uses: https://github.com/actions/cache@v4
|
|
with:
|
|
path: ~/.cache/Cypress
|
|
key: cypress-${{ runner.os }}-${{ hashFiles('pnpm-lock.yaml') }}
|
|
|
|
- run: pnpm install --frozen-lockfile
|
|
# pnpm install doesn't reliably trigger Cypress's postinstall binary
|
|
# download (see apps/web's cypress caveat in the README) — install it
|
|
# explicitly so `cypress run` finds it.
|
|
- run: pnpm --filter web exec cypress install
|
|
- name: Run E2E tests
|
|
env:
|
|
HOST: "0.0.0.0"
|
|
run: pnpm --filter web e2e
|
|
# No dev server needed here — Cypress spins up its own Vite dev
|
|
# server internally for component testing (see cypress.config.ts's
|
|
# `component.devServer`), unlike `e2e` above which needs the real app
|
|
# running first.
|
|
- run: pnpm --filter web cy:run:component
|