What Changed in v5
entity_matches_llama31_backup holds the 13,429 ai_confirmed/ai_rejected match records; ai_queue_llama31_backup holds the queue rows with scores. All future analysis can compare models directly via SQL JOIN on source_type+source_id+company_id.
existing_conf values stored in ai_queue. Original methods were inferred from the 10 distinct confidence values produced by known pipeline formulas (exact 0.90, LEI 0.92/0.828, DBA 0.855, fuzzy 0.675/0.75/0.81). All 13,429 ai_queue items were reset to status='pending'.
ecosystem.config.cjs updated with OLLAMA_MODEL=mistral:latest, process renamed ai-queue-mistral. The complete queue was processed under PM2. Result: 1,694 confirmed (12.6%) and 11,731 rejected (87.4%) — a dramatic divergence from LLaMA's 61.0% confirm rate.
Headline KPIs — Mistral vs LLaMA 3.1
Interpretation context: v5 reflects Mistral as the active model — 17,565 effective matches total (5,165 TM + 12,400 patent unchanged). The LLaMA 3.1 state (24,055 effective) is preserved in entity_matches_llama31_backup and can be restored at any time. Patent matches are untouched in both versions.
Cross-Model Agreement Matrix
50.0% agreement rate overall. The 6,599 LLaMA-only confirms is the critical region — these are items where the models disagree most strongly and represent the core question of whether corporate-family linkage is in scope.
Agreement Breakdown by Original Confidence Tier
Each row represents one of the 10 distinct confidence values produced by the matching pipeline. Inferred original match method shown in parentheses.
The gap is consistent across all tiers but is most extreme on cross-country tiers (0.90 exact, 0.675 fuzzy). Mistral confirms ≤21% in every single tier; LLaMA never goes below 49%. The 0.75 fuzzy (no-country) tier is the only one where Mistral reaches 31.8% — no country mismatch to penalize.
Behavioral Analysis — What Each Model Confirms
Mistral philosophy: strict legal identity. Confirms only when the holder name and company name are the same legal entity with minor textual variation — punctuation differences, trailing commas, missing period after abbreviation, obvious single-character typos, or a Cyrillic name that explicitly contains the Latin transliteration in parentheses. Country mismatch is treated as a strong contra-indicator. Score output is near-binary: 0.8225 (most confirms) or 0.85 for slightly higher confidence; all rejects score 0.
LLaMA 3.1 philosophy: corporate family linkage. Confirms when the holder is a subsidiary, regional branch, product division, or rebranded entity of the target company — even if the names are structurally different. "Swedish Match Brasil SA" → "Swedish Match AB" was confirmed. "Imperial Tobacco Ventures Limited" → "Imperial Tobacco Limited" was confirmed. Score for confirmed items was uniformly 0.795 (92%) or 0.85 (8%); all rejects score 0.
Representative Examples — LLaMA Confirmed / Mistral Rejected
| Holder Name | Company Name | Country | Original Conf | Assessment |
|---|---|---|---|---|
| Imperial Tobacco Ventures Limited | Imperial Tobacco Limited | SM | 0.92 (LEI) | Corporate family |
| Swedish Match Lighters B.V. | Swedish Match AB | MV | 0.92 (LEI) | Subsidiary → parent |
| SWEDISH MATCH DO BRASIL SA | Swedish Match AB | BI | 0.92 (LEI) | Regional → global |
| AV Cannabis Inc. | VIVO Cannabis, Inc. | TD | 0.92 (LEI) | Acquired entity |
| C.B. FLEET COMPANY, INCORPORATED. | C. B. Fleet Company, Inc. | ST | 0.90 (exact) | Same entity |
| DR. REDDY'S LABORATORIES LTD | Dr.Reddy's Laboratories, Ltd. | GY | 0.90 (exact) | Same entity |
| SPECTRUM CANNABIS CANADA LTD | Spectrum Cannabis Canada, Ltd. | TD | 0.90 (exact) | Same entity |
| SHENZHEN JIANAN TECHNOLOGY CO.,LIMITED | Shenzhen Jianan Technology Co., Ltd. | CC | 0.90 (exact) | Same entity |
| Hanfama Vertriebs GmbH | Hanfama Vertriebs GmbH | BY | 0.90 (exact) | Near-identical |
| TOA INDUSTRY. CO., LTD. | Toa Industry Co., Ltd. | ID | 0.90 (exact) | Same entity |
Mistral false-negative problem: Many items LLaMA confirmed and mistral rejected are clearly the same legal entity — "C.B. Fleet Company, Incorporated." vs "C. B. Fleet Company, Inc." is an obvious match. Mistral's refusal here appears to be driven by the country mismatch (holder filed in country ST, company is US-based). The model interprets a different-country filing as a different legal entity rather than recognizing this as the same company filing in multiple jurisdictions.
Representative Examples — Mistral Confirmed / LLaMA Rejected
| Holder Name | Company Name | Country | Mistral Score | Assessment |
|---|---|---|---|---|
| Shenzhen Quawins Technology Co.,ltd. | Shenzhen Quawins Technology Co., Ltd. | CC | 0.8225 | Near-literal (punctuation) |
| Shenzhen Yocan Technology Co., Ltd | Shenzhen Yocan Technology Co., Ltd. | CC | 0.8225 | Near-literal (trailing dot) |
| Japan Tabacco Inc. | Japan Tobacco Inc. | ID | 0.8225 | Single typo (cc→c) |
| Perfetti Van Melle SpA | Perfetti Van Melle S.p.A. | GY | 0.8225 | Abbreviation format |
| Carmans Fine Foods Pty Ltd | Carman's Fine Foods Pty, Ltd | BB | 0.8225 | Apostrophe variant |
| Реемтсма Сігареттенфабрікен ГмбХ (Reemtsma Cigarettenfabriken GmbH) | Reemtsma Cigarettenfabriken GmbH | TF | 0.8225 | Cyrillic + Latin form |
Mistral's 108 unique confirms are predominantly Shenzhen Chinese technology companies and other near-literal name pairs where the fuzzy score was marginally below LLaMA's acceptance threshold. LLaMA rejected these while confirming far looser corporate family matches — revealing an inconsistency in LLaMA's evaluation criteria: it was lenient on name similarity for corporate families but strict on literal-format variants.
AI Score Distribution
Mistral Score Distribution
3 distinct non-zero score values. The 0.8225 cluster dominates (Chinese company near-exact + punctuation variants). Near-binary output — no gradation in rejections.
LLaMA 3.1 Score Distribution
Also effectively 3 score values. The 0.795 cluster dominates (91.9% of accepts). Also near-binary — but with a far larger accept bucket.
Both models produce near-binary outputs — score granularity is extremely low (3 distinct values each, concentrated in one bucket). Neither model uses the score range continuously. This means the AI_MIN_SCORE=0.50 threshold is effectively irrelevant; the real decision is binary yes/no, with the score serving as a label rather than a calibrated confidence measure. The difference between models is entirely in where they draw the yes/no line.
Current Match Method Breakdown — v5 (Mistral)
Trademark Holders (1,262,112 source)
Effective: 5,165 rows · 3,721 unique holders · Total rows incl. audit: 16,896
Patent Applicants (1,479,367 source) — unchanged
Total: 12,400 rows · 9,736 unique applicants · Patent AI queue not yet run
v4 (LLaMA) → v5 (Mistral) Comparison
| Dimension | v4 · LLaMA 3.1 | v5 · Mistral | Notes |
|---|---|---|---|
| Model | llama3.1:8b | mistral:latest | |
| TM uncertain queue size | 13,429 items | 13,429 items | Identical input |
| AI Confirmed | 8,188 (61.0%) | 1,694 (12.6%) | −79.3% |
| AI Rejected | 5,241 (39.0%) | 11,731 (87.4%) | +123.9% |
| Processing errors | 0 | 13 (4 new) | Minor |
| TM effective matches | 11,655 | 5,165 | −55.7% |
| TM unique holders matched | 9,344 | 3,721 | −60.2% |
| Total effective (TM + patent) | 24,055 | 17,565 | −27.0% |
| Score range (confirmed) | 0.74–0.85 | 0.795–0.85 | Both near-binary |
| Avg score (confirmed) | 0.7995 | 0.8264 | Mistral scores higher |
| Corporate family links | ~5,000+ est. | ~0 | Excluded by design |
| Both models agree | 50.0% (6,718 / 13,425) | Low agreement | |
| Patent matches | 12,400 (unchanged) | 12,400 (unchanged) | Not affected |
Which model is correct? Neither model is universally right — they answer different questions. Mistral answers: "Is this holder the same legal entity as the company?" LLaMA answers: "Is this holder in the same corporate group as the company?" Both are valid use cases. For export to an entity graph, LLaMA's broader linkage is typically more useful. For strict identity deduplication, Mistral's conservative approach is safer. Recommendation: treat the 1,586 both-confirmed items as high-confidence, and re-evaluate the 6,599 LLaMA-only items with a targeted review pass.
Script Coverage — Canonical TM Holders
Script distribution is unchanged from v4. AI model swap does not affect the source data or the non-AI matching pipeline.
332,342 non-Latin holders (26.3%) remain effectively unmatched. Cyrillic holders that embed the Latin transliteration in parentheses are now better served by Mistral (which correctly identifies "Реемтсма ГмбХ (Reemtsma GmbH)" as the same entity). Pure-Cyrillic without Latin annotation still requires a transliteration pipeline.
Recommended Next Steps
Key decision point: The 6,599 LLaMA-only confirmed items (corporate-family links) need a policy decision before export. Options: (A) restore LLaMA results for the disputed region; (B) keep Mistral (strict identity only); (C) run a third model or human review pass on the 6,599; (D) export the 1,586 both-confirmed as highest confidence, separately flag the 6,599. The LLaMA backup tables make all of these options available without re-running any pipeline.
| Priority | Action | Expected Impact | Est. Time |
|---|---|---|---|
| Immediate | Policy decision: use Mistral (strict) or LLaMA (family) for the 6,599 disputed TM matches | Defines export scope: 1,694 vs 8,188 TM AI-confirmed rows available | Decision only — no code |
| Immediate | Export high-confidence matches (both-confirmed + exact + DBA ≥ 0.95) for API write-back | ~5,100 production-ready entity links with Mistral; ~11,600 with LLaMA | Export script — 1 hr |
| High | Build + run patent AI queue (P2+P3 uncertain fuzzy, 4,517 rows) | Review patent fuzzy matches; expect ~50–60% survive under LLaMA, fewer under Mistral | ~2–4 hrs depending on model |
| High | Fix AI queue to exclude lei matches from uncertain track (or provide LEI code as context) |
Prevents AI from rejecting LEI-matched pairs based on name alone — both models affected | Code change — build-ai-queue.js |
| Medium | Run AI queue P4 (no-match candidates, ~998K) — choose model based on policy decision above | Expected 1–3% yield = 10K–30K new links from unmatched holders/applicants | ~30 days at 32/min · ~4 days at 4× |
| Medium | CJK transliteration pipeline (Hanzi → Pinyin) | Opens matching for 187,564 CJK holders (14.9% of TM canonical pool) | Pipeline development + re-run |
| Low | Deduplicate upstream company entries (Nestlé ×2, Philip Morris ×3, JTI ×2, PepsiCo ×2) | Prevents double-count before write-back; affects ~10 companies | Manual review |