WIMS · Research & Analysis

Entities Mapper — Implementation Plan

Rule-based + Fuzzy + AI matching pipeline  ·  TM ↔ Companies ↔ Patents ↔ Products
1

API Findings

Base URL & Auth

Host: https://wims-api.winglegroup.com   —   PHP 7.4 / Symfony / API Platform (Hydra / JSON-LD)

Key Entity Counts

Companies
13,739
/api/companies
Brands
17,332
/api/brands
Products
78,719
/api/products
Patents
2,807,127
/api/patents
Trademarks (TM items)
4,545,489
/api/trademark_items
Trademark Holders
4,552,327
/api/trademark_holders
Patent Applicants (raw)
6,532,620
/api/patent_applicants
Unique Applicant Names
1,479,130
/api/patents/applicants

Existing Relationships (partial coverage)

The API already exposes relationship fields, but they are mostly empty — e.g. patent.companiesAmount = 0 for the vast majority of records:

🔑

Core problem: TrademarkHolder.name = "Philip Morris Products S.A." and PatentApplicant.name = "Philip Morris Products Sa" both carry no foreign key to the Companies table. This pipeline's entire purpose is to bridge that gap with confidence-scored matches.

2

Architecture

Matching Pipeline

Step 1 Data Collection
  • Companies (13K) + DBAs + reg numbers + URLs
  • TM Holders (4.5M unique names)
  • Patent Applicants (1.5M unique names)
  • Products (79K) + brands + manufacturers
  • Brands (17K) + ownerCompanies
↓
Step 2 Entity Normalization
  • Strip legal suffixes (Inc, LLC, S.A., Ltd, GmbH…)
  • Lowercase + remove punctuation / diacritics
  • Tokenize + sort tokens
  • Build inverted index of companies by normalized name
↓
Step 3 Rule-based Matching confidence 0.90 – 1.0
  • Exact match on normalized name
  • DBA / alias match
  • Registration number match
  • Domain / URL match
↓
Step 4 Fuzzy Matching confidence 0.60 – 0.89
  • Token sort ratio ≥ 85 (fuzzball)
  • Longest common prefix matching
  • Bigram / trigram overlap score
  • Dedupe & rank candidates per entity
↓
Step 5 AI Relationship Scoring confidence 0.30 – 0.69
  • Ollama (local) — llama3.1:8b or mistral:7b
  • Prompt: "Are these two names the same company?"
  • Context injection: country, address, category
  • Only run on unmatched or borderline candidates
↓
Step 6 Output & Manual Review Queue
  • Tables: entity_matches, review_queue
  • JSON exports by connection type
  • Manual review queue for conf < 0.75

Connection Types (priority order)

1
TM → Company
Source: TrademarkHolder.name, country  →  Target: Company.name, .dbas, .registrationNumber
2
Patent → Company
Source: Applicant.name, country  →  Target: Company.name, .dbas, .registrationNumber
3
TM → Product
Source: TrademarkItem.name  →  Target: Product.name, Brand.name
4
Patent → Product
Source: via Company link + applicant name  →  Target: Product.manufacturers, Product.patents
5
TM → Patent
Source: shared Company + brand/product context  →  Target: Applicant.name ≈ Holder.name
6
Company → Product
Already partially exists — extend coverage
7
Brand Ecosystem Mapping
Source: Brand.ownerCompanies + TM name  →  Target: cross-entity brand mapping
3

Technical Stack

💡

Why not PostgreSQL / Elasticsearch? The company index is only 13,739 records — SQLite FTS5 handles all lookups in milliseconds. External services add deployment complexity with no measurable gain for a pipeline tool.

🔒

Why Ollama over cloud AI? All data stays on-premises, zero API cost, works fully offline. llama3.1:8b fits in 8 GB VRAM or runs on CPU.

4

Matching Logic Detail

Name Normalization

Applied uniformly to every entity name before any comparison:

"Philip Morris Products S.A."
  → lowercase:        "philip morris products s.a."
  → remove suffixes:  "philip morris products"
  → remove punct:     "philip morris products"
  → token sort:       "morris philip products"
  → stored as         normalized_name

Legal suffix strip list: inc llc corp ltd limited sa s.a. plc p.l.c. gmbh ag bv nv srl sas pty co company international group holding holdings

Confidence Scoring Model

Exact normalized match
1.00
Registration number
0.98
DBA / alias match
0.95
Fuzzy score 95 – 100
0.90
Fuzzy score 85 – 94
0.75
Fuzzy score 70 – 84
0.55
AI says "yes"
0.30 – 0.85
Multipliers applied on top of base score (capped at 1.0):
  • Same country match  →  × 1.05
  • Different country  →  × 0.90
  • Matched via DBA / alias  →  × 1.02

Filtering Strategy for Large Datasets

Rather than downloading all 4.5 M TM holders row-by-row:

  1. 1
    Stream unique holder names (deduplicated in SQLite as they arrive) — reducing ~4.5 M rows to ~1 M distinct strings.
  2. 2
    For each unique name, run the matching pipeline against the 13,739-company index (held in memory / FTS5).
  3. 3
    Write matches to SQLite; skip non-matching holders — they belong to entities outside the WIMS company scope.
📉

This reduces the effective working set from 4.5 M raw records → ~1 M unique applicants / ~1 M unique holders with a single streaming pass.

5

Key Risks & Mitigations

Risk Impact Mitigation
API rate limiting High — could block data collection entirely 100 ms delay between pages; exponential backoff on HTTP 429
Token expiry (1 h TTL) Medium — pipeline breaks mid-run Auto-refresh via refresh_token before each batch request
Name ambiguity (139 "Philip Morris" variants found) High — wrong company linked Always surface top-3 candidates with scores; human review for confidence < 0.75
Fuzzy false positives (e.g. "BAT" → wrong company) Medium — pollutes match quality Short names (< 5 chars) require stricter threshold (≥ 0.95)
Ollama model hallucination Low — minor score distortion AI used as tie-breaker only; never sole basis for a high-confidence match
Large dataset memory pressure Low — OOM crash on weak machines Stream and process in chunks of 1,000; write to SQLite incrementally
6

Summary

Total entities to match
~6 M TM holders + applicants → 13,739 companies
MVP connection types
4  (TM→Company, Patent→Company, TM→Product, Patent→Product)
Phase 2 connections
+ 3  (TM→Patent, Company→Product, Brand ecosystem)
Tech stack
Node.js + SQLite + fuzzball + Ollama
Phase 1
Rule-based matching
Phase 2
Fuzzy matching + extended connections
Phase 3
AI scoring + brand ecosystems
Phase 2 enhancements (future)
pgvector / Qdrant · Scheduled re-matching · Confidence calibration · Webhook integration
7

Next Implementation Steps — post v2 TM run · 2026-06-01

✅

TM→Company done (no AI): 407,304 matches · 76.4% high-confidence · avg 0.9224 · pharma false-positives eliminated. Stats report: stats-report.html

Step 1 — Patent Applicant Matching (match:patents)

Goal: run the same pipeline (exact → DBA → LEI → fuzzy → transliteration) against the 1,479,130 unique patent applicant names. The patent_applicants table is currently empty — this is the highest-leverage remaining work.

  • Collect unique applicant names via /api/patents/applicants pagination
  • Normalize + detect script (reuse TM holder pipeline)
  • Run exact → DBA → LEI → fuzzy → transliteration in order
  • Write results to entity_matches with source_type = 'patent_applicant'
  • Expected volume: ~50–150K matches at similar precision to TM run

Step 2 — Expand Transliteration Coverage

Goal: 1,005,798 non-Latin TM holders (22% of total) currently match at <2%. Extending transliteration to four remaining high-volume scripts is the single highest-leverage precision improvement short of AI.

ScriptHolder CountCurrent Match RateApproach
Arabic102,4710.05%ALA-LC romanization table or any-Latin ICU transform
Thai103,1870.20%Royal Thai General System transliteration
Hebrew5,2210.02%SBL / ISO 259 romanization
Hangul (Korean)204,4610.02%Revised Romanization of Korean (RRK)

Estimated yield: 1–5% match rate across these scripts → 4,000–20,000 additional high-confidence links.

Step 3 — Company Deduplication

Goal: upstream API contains duplicate company records (Nestlé ×2, Philip Morris ×3, PepsiCo ×2, JTI ×2). These inflate match counts and would create double-links on API write-back. Must be resolved before any export.

  • Query entity_matches for companies sharing the same normalized_name — identify canonical IDs
  • Merge match rows: reassign target_id of duplicates to the canonical ID
  • Flag duplicates in a company_duplicates lookup table for downstream use
  • Do not delete from the companies table — API IDs may be load-bearing

Step 4 — LEI Bridge Expansion

Goal: only 74 of 13,762 companies have a confirmed LEI via Wikidata. The GLEIF golden copy (3,075,680 records) is already loaded — a direct name-match pass between lei_records.legal_name and companies.name can confirm LEIs for hundreds more.

  • Run normalized-name exact + fuzzy match between lei_records and companies
  • Store confirmed LEIs in companies.lei (new column)
  • Re-run the LEI bridge step against TM holders — expected uplift from 54,627 toward 200K+

Step 5 — AI Scoring Pass (Ollama)

Goal: 88,543 matches sit in the uncertain band (0.60–0.89). AI scoring should elevate true positives and discard false positives in this band — it was designed as a tie-breaker, not a primary matcher.

  • Populate ai_queue from entity_matches WHERE confidence BETWEEN 0.60 AND 0.89
  • Priority ordering: priority = 1 - confidence (most uncertain first)
  • Run run:ai-queue against Ollama (llama3.1:8b) — inject country + category context
  • Update entity_matches.confidence and match_method for confirmed/rejected entries
  • Run steps 1–4 above first — more stable input = better AI signal

Step 6 — API Write-back (Export)

Goal: push 311,208 high-confidence (≥0.90) TM→Company links back to the WIMS API to populate the empty TrademarkHolder → Company foreign-key relationships.

  • Complete Step 3 (deduplication) first — do not write duplicate target IDs
  • Batch PATCH requests to /api/trademark_holders/{id} with company FK
  • Rate-limit to 100 ms/request; auto-refresh token on 401
  • Log each write; support --dry-run mode for verification
  • After TM write-back: repeat for Patent→Company once Step 1 is complete