Nicolas
53d415fddb
Une seule feature livree en une seule PR, en 5 phases : - Phase 1 : enrichit le corpus NLP (tech-step-training-data.ts) et ajoute un harness d'evaluation (precision/rappel/F1) avec un jeu de test etiquete - la premiere metrique objective de qualite pour ce classifieur. - Phase 2 : schema Prisma (StepTechStepCorrection, TechStepTrainingSuggestion) + endpoints utilisateur (POST/GET corrections, ouverts a tout viewer, pas seulement l'auteur) + endpoints internes /internal/tech-steps/* proteges par secret partage (requireInternalWorker). - Phase 3 : UI de highlight/correction cote web (selection de texte -> association a une technique, ou clic sur un highlight existant pour le corriger/supprimer) - verifiee via Cypress (component + e2e, en Chrome reel). - Phase 4 : worker LLM autonome (services/tech-step-llm-worker, hors du monorepo pnpm comme experiments/llm-tech-step-poc) qui audite les clauses a faible confiance et transforme les corrections utilisateur en suggestions d'entrainement, sans jamais toucher le chemin interactif. - Phase 5 : script retrain-tech-steps.ts (gate de regression F1 + backfill) et list-pending-training-suggestions.ts pour la revue humaine avant application au corpus. Verification effectuee cette session : tsc/biome sur l'ensemble du repo, build complet (pnpm build), suite Cypress complete (component 39/39, e2e 75/76 - le seul echec est preexistant et sans rapport, cote recipe-form.feature/ingredient-picker), tests unitaires du worker (6/6) et son install/typecheck reels contre node-llama-cpp. Les tests Mocha d'apps/api (Phases 1 et 2) n'ont pas pu etre executes dans cette session (pas de Postgres local disponible) - a lancer avant merge. Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
50 lines
2.2 KiB
TypeScript
50 lines
2.2 KiB
TypeScript
import { expect } from "chai";
|
|
import { prisma } from "../../src/db/prisma.js";
|
|
import {
|
|
MIN_OVERALL_F1,
|
|
runTechStepEvalSuite,
|
|
} from "../../src/lib/recipe-matching/tech-step-eval-runner.js";
|
|
import { resetDatabase } from "../../test-support/reset-db.js";
|
|
|
|
/**
|
|
* Regression gate for `TECH_STEP_TRAINING_DATA` — every change to that
|
|
* corpus (including a maintainer applying suggestions from
|
|
* `TechStepTrainingSuggestion`, see `scripts/retrain-tech-steps.ts`) must
|
|
* keep this suite green. Runs {@link runTechStepEvalSuite} (the real
|
|
* trained classifier against `tech-step-eval-dataset.ts`) and asserts the
|
|
* aggregate F1 doesn't fall below {@link MIN_OVERALL_F1}.
|
|
*
|
|
* `MIN_OVERALL_F1` (`tech-step-eval-runner.ts`) is a provisional floor,
|
|
* not a target: most of the dataset's cases are built around a
|
|
* technique's own registered synonym, which `_classifyClause` always
|
|
* resolves correctly via its NER-anchor fallback even when the intent
|
|
* classifier itself scores under `CONFIDENCE_THRESHOLD` (see
|
|
* `tech-step-matcher.ts`'s doc comment, point 3) — so a healthy run should
|
|
* land well above this floor. It's set low enough to tolerate the residual
|
|
* uncertainty in a dataset authored without being able to run it against a
|
|
* live trained classifier first (no local Postgres was reachable in the
|
|
* session that introduced this file — see this feature's plan document).
|
|
* Once this suite has actually run once (locally or in CI) and produced
|
|
* real numbers, tighten that constant to just below the observed F1, so a
|
|
* real future regression still fails loudly instead of hiding under a
|
|
* floor that's too forgiving.
|
|
*/
|
|
|
|
describe("tech-step-eval", () => {
|
|
beforeEach(async () => {
|
|
await resetDatabase();
|
|
});
|
|
|
|
after(async () => {
|
|
await prisma.$disconnect();
|
|
});
|
|
|
|
it(`scores at least ${MIN_OVERALL_F1} aggregate F1 against the labeled evaluation set`, async () => {
|
|
const { overall, byKey } = await runTechStepEvalSuite();
|
|
|
|
expect(
|
|
overall.f1,
|
|
`aggregate F1 ${overall.f1.toFixed(3)} (precision ${overall.precision.toFixed(3)}, recall ${overall.recall.toFixed(3)}) fell below the ${MIN_OVERALL_F1} floor — per-technique breakdown: ${JSON.stringify(byKey)}`,
|
|
).to.be.at.least(MIN_OVERALL_F1);
|
|
});
|
|
});
|