Meilisearch vs pg_bigm — JERA file-mode run

Fuzzy TM and potential-term TB search replacement comparison

View per-chunk raw results

Queries came from five files’ chunks (get_chunks2), not from dump self-hits:

This is the replacement question: Meili is what runpod still searches; pg_bigm is what honyaku-api batch_fuzzy_* now does. After the search, TM uses the same Python rerank on both sides (Levenshtein / RapidFuzz, drop < 0.40, keep 2). Potential-term TB does not rerank — engine top 2 only.

Headline

pg_bigm is faster. It is close enough on TM near-exact counts, weaker on which row is #1, and clearly weaker on potential-term TB.

Translation memory

What both do well

• When a chunk is a real TM sentence, both often recover a near-exact Japanese match. Near-exact #1 is 430 (Meili) vs 422 (PG) — almost the same.
• After Python rerank, rank-1 similarity is high on both (0.94 vs 0.88). The second-stage scorer is what the pipeline actually keeps; engine ranking is only the candidate pool.

Where they disagree (the 50% that do not share #1)

Kept-list overlap histogram: 312 queries share 0 IDs, 456 share 1, only 62 share both kept IDs.

That is not “PG finds nothing.” Breakdown of kept lists:

• Both have hits: 548
• Both empty after rerank: 197
• PG has hits, Meili has none: 85
• Meili has hits, PG has none: 0

So after the shared Python filter, pg_bigm is not the side that goes empty more often. Meili’s raw top 10 includes more rows that then fail the 0.40 threshold (269 Meili queries dropped to zero kept vs 172 PG). Raw search empties: Meili 13, PG 25. PG misses a few more at search time; Meili retrieves more junk that rerank throws away.

Raw overlap is only 1.71 of 10. The two engines do not propose the same candidate set. Meili ranks across indexed fields with its own ranking. pg_bigm ranks bigm_similarity on one column (ja when target_lang is en). Different pool → different #2 even when #1 text is equivalent.

Shorter queries disagree more (disagree mean length 23 chars vs agree 33). Tiny fragments like タイトル案： produce two equally mediocre 0.67 hits with swapped order.

test-ppt.pptx is an outlier: 11 / 136 TM top-1 agree (8%). Most of those chunks have no kept TM after rerank (Meili empty 111, PG empty 85). Slide text is a poor TM query shape for both engines, not a pg_bigm-only failure. The long JERA article files sit around 50–88% top-1 agree.

What pg_bigm lacks on TM (vs Meili)

1. Candidate ranking is trigram similarity on ja only. Meili can use the whole document (and its ranking model). Short or punctuated queries dilute bigrams.
2. No ranking score on the TM endpoint, so you cannot compare engine scores until Python rerank. (TB does return _rankingScore.)
3. Duplicate Meili IDs vs one PG row inflate “disagreement” without a worse translation.
4. Raw top-10 overlap is low — replacement is not a drop-in for “the same 10 neighbors,” only for “after rerank, often the same near-exact sentence.”

What it does not lack here: ability to hit near-exact TM on real sentences, or speed (faster than Meili on this batch).

Potential terms → fuzzy termbase

This is the weaker replacement.

Potential terms are short (mean ~5 characters). Fuzzy TB is only the engine’s top 2. Meili _rankingScore vs PG bigm_similarity are not the same number, but they are what each side uses.

What Meili does better

• Same #1 only 30% — the two lists are often different people.
• Near-exact #1: Meili 109 / 606 vs PG 44 / 606. Meili is much more likely to put the actual term (or a very close ja) first.
• Rank-1 score: 0.70 vs 0.43. PG’s #1 is, on average, a looser bigram neighbor.
• Shared IDs: 332 / 606 share nothing. Full overlap (2/2) only 76 times.

Where pg_bigm is “more complete” but worse

Kept empties:

• Meili empty: 126
• PG empty: 54
• PG-only hits: 88
• Meili-only hits: 16

pg_bigm returns something more often. Those extras are frequently weak matches (low bigm_similarity), which is why average PG rank-1 is 0.43 and near-exact is 44. For potential-term review, a false neighbor is noisier than an empty list.

=% / bigm_similarity on 1–2 character Japanese terms is a known weak spot: there are few trigrams, so many terms look alike. Meili’s tokenizer/ranking still separates them better.

Meili TB search hits all searchable attributes. PG TB search uses ja only when source_lang is ja. English-only or mixed potential terms are disadvantaged on PG.

No Python rerank on this path, so engine error is the product error. TM’s second scorer cannot save TB.

Practical takeaway for the Meili → pg_bigm swap

Latency is a win (TM ~1.8×, TB ~9× faster here). Correctness for TB is not.

Files to inspect: index.html (per-chunk kept + raw TM lists), compare.csv (kept vs *_raw_* columns).