fix(tech-steps): calibre le seuil F1 sur une vraie execution et corrige un bug de comptage

Docker etant redevenu disponible dans cette session, j'ai pu lancer pour de
vrai la suite Mocha d'apps/api (334/334, y compris les tests Phase 1/2
qui n'avaient pu etre executes precedemment) ainsi que les scripts de la
Phase 5 contre une vraie base de test.

- tech-step-eval-dataset.ts : corrige un vrai bug d'auteur - "Take the
  plates..." collisionnait avec le synonyme anglais enregistre "plates"
  (technique plate), invalidant ce cas negatif. Remplace par "dishes".
- tech-step-eval-runner.ts : F1 reel mesure = 0.815 (33 TP / 9 FP / 6 FN).
  Documente ce chiffre et les vraies erreurs de classification decouvertes
  (ex: "Blanchissez les haricots verts..." classifie a tort comme "peel")
  - des faiblesses reelles du classifieur que ce harness est cense
  detecter, pas a masquer en ajustant le jeu de test.
- retrain-tech-steps.ts : le script loggait `appliedIds.length`/
  `rejectedIds.length` (ce qui a ete demande) au lieu du `count` reel
  retourne par `updateMany` (ce qui a vraiment ete modifie) - un id
  inexistant faisait afficher un faux succes. Decouvert en executant le
  script pour de vrai avec des ids partiellement invalides.

Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>

apps/api/src/lib/recipe-matching/tech-step-eval-dataset.ts

@@ -19,22 +19,20 @@
  * real DB ids and back by `tech-step-eval-runner.ts`, this file only ever
  * deals in stable keys so it doesn't need DB access to author or read.
  *
- * Known limit of this dataset: most cases anchor on a technique's own
+ * Known limit of this dataset, confirmed against a real run (see
- * registered synonym (verb form), which `_classifyClause` always falls
+ * `MIN_OVERALL_F1`'s own doc comment, `tech-step-eval-runner.ts`): most
- * back to labeling correctly even when the intent classifier itself isn't
+ * cases anchor on a technique's own registered synonym, but `_classifyClause`
- * confident (see `tech-step-matcher.ts`'s doc comment, point 3) — so this
+ * only falls back to that anchor when the intent classifier's own score is
- * dataset mainly measures precision (wrong/duplicate matches, false
+ * *below* `CONFIDENCE_THRESHOLD` — a confidently *wrong* whole-clause
- * positives from a synonym overlapping another technique's vocabulary) and
+ * classification (e.g. "Blanchissez les haricots verts..." scoring
- * breadth of coverage across all ~26 techniques, not the classifier's
+ * confidently as `peel` despite the correct `blanch` anchor) overrides the
- * ability to recognize a technique described without ever naming it
+ * anchor just as readily as a confidently *right* one does, so this
- * (`tech-step-matcher.test.ts` already covers a few of those specific,
+ * dataset genuinely does measure real classifier failures, not just a
- * verified cases at the unit level — e.g. "jusqu'à ce que le beurre ait
+ * synthetic floor. A handful of such real mismatches are expected and
- * disparu dans la poêle" for `melt`). Extending this dataset with more
+ * intentionally left uncorrected here (see `MIN_OVERALL_F1`'s doc comment)
- * paraphrase-only cases is valuable future work, but each one needs to be
+ * — fixing the classifier's actual behavior on them is corpus work for a
- * verified against a real trained classifier before being added (a wrong
+ * future change, not something to hide by loosening this dataset's own
- * expected label here fails the regression gate for the wrong reason) —
+ * expectations to match whatever it currently outputs.
- * see this feature's plan document for the current gap in this session's
- * ability to run the classifier locally.
  */
 
 export interface TechStepEvalCase {
@@ -240,7 +238,7 @@ export const TECH_STEP_EVAL_DATASET: TechStepEvalCase[] = [
     expectedKeys: [],
   },
   {
-    description: "Take the plates and glasses out of the cupboard.",
+    description: "Take the dishes and glasses out of the cupboard.",
     locale: "en",
     expectedKeys: [],
   },

apps/api/src/lib/recipe-matching/tech-step-eval-runner.ts

@@ -8,12 +8,24 @@ import {
 import { techStepClassifier } from "./tech-step-matcher.js";
 
 /**
- * Provisional floor (not a target) both `runTechStepEvalSuite`'s
+ * Regression floor (not a target) both `runTechStepEvalSuite`'s consumers
- * consumers gate on — see `test/recipe-matching/tech-step-eval.test.ts`'s
+ * gate on — exported from here (not defined separately in each consumer)
- * own doc comment for the full reasoning behind this specific value and
+ * so the CI regression gate and `scripts/retrain-tech-steps.ts`'s
- * when to tighten it. Exported from here (not defined separately in each
- * consumer) so the CI regression gate and `scripts/retrain-tech-steps.ts`'s
  * pre-backfill gate can never silently drift to different thresholds.
+ *
+ * Calibrated against a real run: the classifier trained on the corpus as
+ * of this constant's introduction scored **0.815** aggregate F1
+ * (33 TP / 9 FP / 6 FN) against `TECH_STEP_EVAL_DATASET` — `0.8` leaves a
+ * small margin below that for run-to-run noise while still catching a
+ * real regression (not a floor picked blind before ever running this
+ * suite — see this feature's plan document for that earlier state). The
+ * mismatches this run surfaced (e.g. "Blanchissez les haricots verts..."
+ * misclassified as `peel`, a handful of anchor-less sentences expected to
+ * match nothing instead scoring confidently as some technique) are real,
+ * known classifier weaknesses — evidence this harness is doing its job,
+ * not something to quietly paper over by loosening the dataset's own
+ * expectations. Improving them is corpus work for a future change, gated
+ * by this same suite.
  */
 export const MIN_OVERALL_F1 = 0.8;
 

apps/api/src/scripts/retrain-tech-steps.ts

@@ -70,18 +70,22 @@ async function retrainTechSteps(): Promise<void> {
   console.info(`Backfilled ${changed}/${total} step(s).`);
 
   if (appliedIds.length > 0) {
-    await prisma.techStepTrainingSuggestion.updateMany({
+    // `updateMany`'s own `count` (rows actually matched/updated), not
+    // `appliedIds.length` (what was merely *asked for*) — an id that
+    // doesn't exist (typo, already-processed id) would otherwise log a
+    // success count that silently doesn't match what really changed.
+    const { count } = await prisma.techStepTrainingSuggestion.updateMany({
       where: { id: { in: appliedIds } },
       data: { status: "applied" },
     });
-    console.info(`Marked ${appliedIds.length} suggestion(s) as applied.`);
+    console.info(`Marked ${count}/${appliedIds.length} suggestion(s) as applied.`);
   }
   if (rejectedIds.length > 0) {
-    await prisma.techStepTrainingSuggestion.updateMany({
+    const { count } = await prisma.techStepTrainingSuggestion.updateMany({
       where: { id: { in: rejectedIds } },
       data: { status: "rejected" },
     });
-    console.info(`Marked ${rejectedIds.length} suggestion(s) as rejected.`);
+    console.info(`Marked ${count}/${rejectedIds.length} suggestion(s) as rejected.`);
   }
 }
 

apps/api/test/recipe-matching/tech-step-eval.test.ts

@@ -12,22 +12,9 @@ import { resetDatabase } from "../../test-support/reset-db.js";
  * `TechStepTrainingSuggestion`, see `scripts/retrain-tech-steps.ts`) must
  * keep this suite green. Runs {@link runTechStepEvalSuite} (the real
  * trained classifier against `tech-step-eval-dataset.ts`) and asserts the
- * aggregate F1 doesn't fall below {@link MIN_OVERALL_F1}.
+ * aggregate F1 doesn't fall below {@link MIN_OVERALL_F1} — see that
- *
+ * constant's own doc comment (`tech-step-eval-runner.ts`) for the real run
- * `MIN_OVERALL_F1` (`tech-step-eval-runner.ts`) is a provisional floor,
+ * it was calibrated against.
- * not a target: most of the dataset's cases are built around a
- * technique's own registered synonym, which `_classifyClause` always
- * resolves correctly via its NER-anchor fallback even when the intent
- * classifier itself scores under `CONFIDENCE_THRESHOLD` (see
- * `tech-step-matcher.ts`'s doc comment, point 3) — so a healthy run should
- * land well above this floor. It's set low enough to tolerate the residual
- * uncertainty in a dataset authored without being able to run it against a
- * live trained classifier first (no local Postgres was reachable in the
- * session that introduced this file — see this feature's plan document).
- * Once this suite has actually run once (locally or in CI) and produced
- * real numbers, tighten that constant to just below the observed F1, so a
- * real future regression still fails loudly instead of hiding under a
- * floor that's too forgiving.
  */
 
 describe("tech-step-eval", () => {