batchCooking/apps/api/src/lib/recipe-matching/ingredient-matcher.ts
Nicolas 9f68c144f3 feat(api): remplace la détection des tech steps par un pipeline NLP (node-nlp)
Le matching par regex ne généralisait jamais au-delà de son propre
vocabulaire — une étape décrivant la fonte du beurre comme "jusqu'à ce
que le beurre ait disparu dans la poêle" ne contient aucun verbe sur
lequel une regex pourrait s'ancrer, alors que le sens est sans
ambiguïté.

Nouveau pipeline en 3 étapes (TechStepClassifierService, node-nlp
4.27.0 — la 5.x est encore alpha, non retenue) :
1. NER (entités enum) trouve les mentions candidates + leur position
   exacte, à partir de listes de synonymes (tech-step-training-data.ts)
   plutôt que de regex écrites à la main. ner.threshold: 1 (exact,
   après normalisation) — le défaut à 0.8 faisait matcher "faire" (verbe
   auxiliaire omniprésent) contre "frire" par pure proximité de chaîne.
2. La description est découpée en clauses autour de ces candidats
   (splitIntoClauses, pure/testable sans modèle).
3. Le NlpManager classe chaque clause individuellement, entraîné sur
   des phrases qui n'emploient jamais le verbe de la technique — c'est
   ce qui apporte la compréhension du sens. En dessous de
   CONFIDENCE_THRESHOLD (0.65, ajusté empiriquement), retombe sur la
   technique impliquée par l'ancre NER plutôt que d'abandonner un match
   clairement ancré sur un mot-clé.

TechStepMapping (table de regex par technique/locale) supprimée —
migration 20260821130000_drop_tech_step_mapping — plus aucune table
n'est interrogée à l'exécution, les données de matching vivent en code.
TECH_STEPS (reference-seed-data.ts) simplifié en simple liste de uid,
les mappings ayant disparu.

Deux pièges trouvés en construisant ce pipeline, corrigés à la source :
- db/prisma.ts construisait PrismaClient sans importer config/env.ts —
  un run de test isolé pouvait faire gagner la course au .env interne
  de Prisma (dev) contre .env.test. Fixé en important config/env.js en
  tout premier, pour effet de bord.
- NlpManager a autoSave/autoLoad: true par défaut — persiste le modèle
  entraîné dans model.nlp et le recharge au lieu de ré-entraîner au
  prochain démarrage. Les deux désactivés explicitement (sinon un
  modèle obsolète masquerait silencieusement toute mise à jour du
  corpus/seuil) ; model.nlp ajouté au .gitignore en garde-fou.

apps/api/src/db/prisma.ts, recipe.service.ts, sources.service.ts et
recipe-translation.ts adaptés à la matching async (le classifieur
entraîné remplace le couple loadTechStepMappingRules+matchTechStepSpans
synchrone) ; server.ts appelle techStepClassifier.warmUp() avant
d'accepter du trafic (le tout premier appel réel à
NlpManager.process() charge les ressources par langue de node-nlp,
plusieurs secondes).

Vérifié : tsc --noEmit, biome check (0 erreur), build complet des 6
packages, 308 tests API (dont un test-support/reset-db.ts corrigé —
référençait encore tech_step_mapping dans son TRUNCATE).

Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
2026-08-21 15:29:09 +02:00

218 lines
10 KiB
TypeScript

import {
INGREDIENT_LABEL_SYNONYMS_EN,
INGREDIENT_LABELS_EN,
UNIT_LABELS_EN,
} from "@batch-cooking/shared";
import { prisma } from "../../db/prisma.js";
import { normalizeText } from "./tech-step-matcher.js";
/**
* Resolves the free-text `name`/`unit`/`quantity` a `RecipeSourceAdapter`
* lifts from an English-language source (`ParsedRecipeIngredient`) against
* our own `Ingredient`/`Unit` reference catalogs — the ingredient-side
* counterpart to `tech-step-matcher.ts`'s technique detection, built for
* the same reason: an English source's raw text has no idea our catalogs
* even exist.
*
* Unlike tech steps (regex mappings hand-authored per technique),
* ingredient/unit labels are plain hand-written English text
* (`INGREDIENT_LABELS_EN`/`UNIT_LABELS_EN`, `packages/shared`) — matching
* them against arbitrary free text (extra adjectives, plurals, "large diced
* yellow onion" for a catalog entry that's just "Onion") needs its own,
* lighter algorithm: word-tokenize both sides, naively stem for plurals,
* then look for the catalog phrase's tokens as a contiguous run inside the
* ingredient text's tokens. The longest (most specific) matching catalog
* phrase wins, same "specificity resolves overlaps" idea as
* `matchTechSteps`, just without weights (there's no need to rank two
* *unrelated* ingredients — only a phrase against its own substrings, e.g.
* "chicken breast" beating bare "chicken").
*
* `matchIngredientName`/`matchUnit`/`extractQuantity` are pure, so they're
* unit-testable without a database (see `test/ingredient-matcher.test.ts`);
* `loadIngredientCatalog`/`loadUnitCatalog` are the only DB-touching pieces,
* meant to be fetched once per request and reused across every ingredient
* line, the same "don't requery per item" convention
* `tech-step-matcher.ts`'s `TechStepClassifierService` follows for its own
* one-time training pass.
*/
/** One `Ingredient` row trimmed to what {@link matchIngredientName} needs, alongside its English matching label. */
export interface IngredientMatchEntry {
ingredientId: number;
/** English label from `INGREDIENT_LABELS_EN`, e.g. `"Chicken breast"` — matched against free text, never displayed. */
label: string;
}
/** One `Unit` row trimmed to what {@link matchUnit} needs, alongside its accepted English spellings. */
export interface UnitMatchEntry {
unitId: number;
/** Accepted spellings from `UNIT_LABELS_EN`, e.g. `["tbsp", "tbs", "tablespoon", "tablespoons"]`. */
synonyms: string[];
}
/**
* Naive English stemmer — strips a plural suffix so "tomato"/"tomatoes",
* "onion"/"onions", "cherry"/"cherries" compare equal after stemming.
* Deliberately not a real linguistic stemmer: it only has to be
* *consistent* (the same word always stems the same way) since both sides
* of every comparison go through it, not linguistically correct on its
* own — see the module doc comment.
*/
function stemWord(word: string): string {
if (word.endsWith("ies") && word.length > 4) return `${word.slice(0, -3)}y`;
if (word.endsWith("es") && word.length > 3) return word.slice(0, -2);
if (word.endsWith("s") && !word.endsWith("ss") && word.length > 3) return word.slice(0, -1);
return word;
}
/** Lowercases, strips diacritics (via `normalizeText`) and splits `text` into stemmed word tokens — empty/non-letter runs are dropped, not kept as empty tokens. */
function tokenize(text: string): string[] {
return normalizeText(text)
.split(/[^a-z]+/)
.filter((word) => word.length > 0)
.map(stemWord);
}
/** Whether `needle` appears as a contiguous run inside `haystack`, at any starting position. */
function containsSubsequence(haystack: string[], needle: string[]): boolean {
if (needle.length === 0 || needle.length > haystack.length) return false;
for (let start = 0; start <= haystack.length - needle.length; start++) {
if (needle.every((word, i) => haystack[start + i] === word)) return true;
}
return false;
}
/**
* Resolves free-text `name` (a `ParsedRecipeIngredient.name`, e.g. `"large
* diced yellow onions"`) to the best-matching `Ingredient` in `catalog`, or
* `null` if nothing matches. Among every catalog entry whose label's words
* all appear as a contiguous run in `name`, the one with the most words
* wins (most specific — "chicken breast" over bare "chicken"); ties break
* on the lowest `ingredientId`, for a deterministic result independent of
* catalog order.
*/
export function matchIngredientName(name: string, catalog: IngredientMatchEntry[]): number | null {
const nameTokens = tokenize(name);
if (nameTokens.length === 0) return null;
let best: { ingredientId: number; tokenCount: number } | null = null;
for (const entry of catalog) {
const labelTokens = tokenize(entry.label);
if (!containsSubsequence(nameTokens, labelTokens)) continue;
if (
best === null ||
labelTokens.length > best.tokenCount ||
(labelTokens.length === best.tokenCount && entry.ingredientId < best.ingredientId)
) {
best = {
ingredientId: entry.ingredientId,
tokenCount: labelTokens.length,
};
}
}
return best?.ingredientId ?? null;
}
/**
* Resolves free-text `unitText` (e.g. `"tbsp"`, `"Cups"`) to the matching
* `Unit` in `catalog`, or `null` if nothing matches. A unit is a single
* word by convention (see `UNIT_LABELS_EN`), so this is a whole-token
* equality check (after stemming/normalizing), not the substring search
* `matchIngredientName` does — `"cup"` shouldn't match inside an unrelated
* longer word.
*/
export function matchUnit(unitText: string, catalog: UnitMatchEntry[]): number | null {
const tokens = tokenize(unitText);
if (tokens.length === 0) return null;
const firstToken = tokens[0];
for (const entry of catalog) {
if (entry.synonyms.some((synonym) => stemWord(normalizeText(synonym)) === firstToken)) {
return entry.unitId;
}
}
return null;
}
/** What {@link extractQuantity} pulls out of a leading numeric expression, alongside what's left of the string after it. */
export interface ExtractedQuantity {
quantity: number | null;
/** `rawText` with the leading quantity (and any separating whitespace) removed — `rawText` unchanged if none was found. */
remainder: string;
}
// Leading "1 1/2", "1/2", "1.5", "1,5" or "2" (optionally followed by a
// hyphenated range like "2-3", in which case only the first number counts —
// good enough for a best-effort quantity, not meant to model ranges.
const LEADING_QUANTITY_PATTERN = /^(\d+)\s+(\d+)\/(\d+)|^(\d+)\/(\d+)|^(\d+(?:[.,]\d+)?)/;
/**
* Pulls a leading quantity off `rawText` (e.g. `"1 1/2 cups flour"` ->
* `{ quantity: 1.5, remainder: "cups flour" }`), supporting a plain
* integer/decimal, a simple fraction (`"1/2"`), or a mixed number
* (`"1 1/2"`). `quantity: null` (remainder === rawText, trimmed) when
* `rawText` doesn't start with a recognizable number — e.g. `"salt to
* taste"`, which has none.
*/
export function extractQuantity(rawText: string): ExtractedQuantity {
const trimmed = rawText.trim();
const match = LEADING_QUANTITY_PATTERN.exec(trimmed);
if (!match) return { quantity: null, remainder: trimmed };
let quantity: number;
if (match[1] !== undefined) {
// Mixed number: "1 1/2".
quantity = Number(match[1]) + Number(match[2]) / Number(match[3]);
} else if (match[4] !== undefined) {
// Simple fraction: "1/2".
quantity = Number(match[4]) / Number(match[5]);
} else {
// Plain integer/decimal: "2" or "1.5"/"1,5" — group 6 is guaranteed
// defined here (the only remaining alternative in the pattern), `?? ""`
// is just satisfying the indexed-access type, not a real fallback.
quantity = Number((match[6] ?? "").replace(",", "."));
}
return { quantity, remainder: trimmed.slice(match[0].length).trim() };
}
/** Loads the full `Ingredient` catalog as {@link IngredientMatchEntry}s — one entry per key with an authored English label (see `INGREDIENT_LABELS_EN`), plus one extra entry per alternate wording (`INGREDIENT_LABEL_SYNONYMS_EN`, e.g. "Vanilla pod" alongside "Vanilla bean" — see issue #54) sharing the same `ingredientId`; `matchIngredientName` doesn't need to know synonyms exist, it just sees more candidate labels for the same ingredient. An ingredient with no English label yet is silently skipped, never a matching target. Meant to be fetched once per request and reused across every ingredient line, not re-queried per line. */
export async function loadIngredientCatalog(): Promise<IngredientMatchEntry[]> {
try {
const ingredients = await prisma.ingredient.findMany({
select: { id: true, key: true },
});
const catalog: IngredientMatchEntry[] = [];
for (const ingredient of ingredients) {
const label = INGREDIENT_LABELS_EN[ingredient.key];
if (label === undefined) continue;
catalog.push({ ingredientId: ingredient.id, label });
for (const synonym of INGREDIENT_LABEL_SYNONYMS_EN[ingredient.key] ?? []) {
catalog.push({ ingredientId: ingredient.id, label: synonym });
}
}
return catalog;
} catch (err) {
// Rethrown as-is — the caller (`sources.service.ts`/`recipe-translation.ts`)
// already handles/logs failures centrally; this function just isn't
// allowed a bare `await` per the repo's async/try-catch convention.
throw err;
}
}
/** Loads the full `Unit` catalog as {@link UnitMatchEntry}s — one entry per key with authored English synonyms (see `UNIT_LABELS_EN`); a unit with none yet is silently skipped. Meant to be fetched once per request, same reasoning as {@link loadIngredientCatalog}. */
export async function loadUnitCatalog(): Promise<UnitMatchEntry[]> {
try {
const units = await prisma.unit.findMany({
select: { id: true, key: true },
});
const catalog: UnitMatchEntry[] = [];
for (const unit of units) {
const synonyms = UNIT_LABELS_EN[unit.key];
if (synonyms !== undefined) catalog.push({ unitId: unit.id, synonyms });
}
return catalog;
} catch (err) {
throw err; // see loadIngredientCatalog()'s catch comment above
}
}