Accuracy

The result

Read the larger test as well. It uses 1,000 documents with generated values, and on sentence wordings the rules were not tuned on it gives about 83%, lower than the 86% to 92% on the small hand-written sets below.

On a set of examples written before the latest round of fixes, with no rule changed after seeing its results, and re-measured with today's rules on 3 November 2026, the hosted service, which was running the small language model (check /health for the current one), hid 31 of 37 sensitive values (83.8%). With the large language model, which the tables below use, Jnvoy hid 34 of 37 sensitive values (91.9%) and left 20 of 21 harmless words (95.2%) readable. That is a good result for a first pass and it is not a guarantee: do not rely on any redaction tool as your only control.

A value counts as hidden only if neither the whole value nor any recognisable part of it (a surname, the last digits of a number) is still visible in the redacted text. These are first-pass results: the optional second check is off on the hosted service.

Which language model. The figures above use the large English model (en_core_web_lg). A server that has only the small model (en_core_web_sm) hides fewer names: on the fresh held-out set it hid 31 of 37 (83.8%) instead of 34. One name that the small model labels as a place is now left alone by the weak name cues, because those cues are now stopped by the model's place labels as they were always meant to be. The hosted service reports which model it is running at /health, under detector.spacy_model.

Speed on these short texts (35 to 50 characters each) was 16 to 32 milliseconds each in a small cloud container. We have not measured longer texts or the hosted service under load.

A larger test: 1,000 documents with generated values

The sets above were written by the person who wrote the detector, so they flatter it. To see how much, we built larger sets where the values (names, emails, phone numbers, addresses, card numbers, bank numbers, dates of birth, IP addresses, keys and passwords) are generated by Faker, a public library, in formats from several countries. We still wrote the sentences around them, so this is not an independent evaluation, only a harder and larger one. The sets are made from fixed seeds, so anyone can regenerate them.

Result. On sentence wordings the rules were not tuned on, Jnvoy hid 835 of 1,009 sensitive values (82.8%; the likely range is 80.3 to 85.0). With the small language model that the hosted service runs it hid 80.9% (range 78.3 to 83.2). Roughly one sensitive value in six was still visible afterwards. The first two sets score about 12 points higher because the fixes were written against their wordings; that gap is how much of the improvement is familiarity and not general ability.

By type of data, on the unseen set

Harmless words kept readable. Jnvoy kept 92.0% (89.5 to 94.0). Presidio with its default settings kept 43.0% (39.0 to 47.1), because it hides dates, places and organisations by default. Set up the way Jnvoy is, with dates, places, nationalities and organisations switched off, it kept 91.2% (88.6 to 93.2) and hid 57.9% of the sensitive values; without a date type it cannot see a birth date at all. So most of the gap in harmless words kept comes from settings, not from better detection. An earlier version of this page left Presidio's organisation type switched on in that comparison, which made Jnvoy's lead look larger than it is.

Where Jnvoy is ahead, behind, or too close to call

• Clearly ahead of Presidio's defaults (the ranges do not overlap): API_KEY (100.0% against 16.1%), ADDRESS (50.0% against 22.2%), PASSWORD (60.5% against 20.9%).
• Clearly behind: none.
• Too close to call at this sample size (the ranges overlap, so a difference in either direction could be chance): PERSON (78.0% against 78.9%), PHONE (94.7% against 93.0%), POSTCODE (41.1% against 34.4%), DOB (100.0% against 98.6%), CARD (92.1% against 88.9%).

The comparison is with Presidio on its own, using the same language model, and is not any product's configuration. Overall, Jnvoy hid 82.8% against 70.5% for Presidio's defaults on this set; of the values only one of them hid, Jnvoy hid 167 and Presidio 43.

What round one of the fixes changed, measured on the development set

• Names in capital letters (after a cue such as "Dear" or "Patient:"): 8 of 72 hidden before, 54 of 72 now. "Surname, First" names: 30 of 65 before, 50 of 65 now.
• Passwords written "pwd secret": 0 of 18 before, 18 of 18 now.
• Card numbers of 12, 15 and 19 digits and the newer 2-series Mastercards, when a word such as "card" or "visa" is next to them: cards overall 77 of 101 to 101 of 101; by length, 12-digit 0 of 7 to 7 of 7, 15-digit 12 of 15 to 15 of 15, 19-digit 0 of 7 to 7 of 7.
• Phone numbers and street addresses after a label such as "phone", "WhatsApp", "ship to" or "lives at", in any layout: phones 163 of 180 to 178 of 180, street addresses 69 of 179 to 164 of 179, postcodes 48 of 179 to 152 of 179.
• A bug in an older rule: a weak name cue ("to", "from", "with") followed by an unfamiliar place, such as "Ann Arbor", was being redacted as a person, because the check that should have stopped it looked for the wrong labels. It now works, and a test covers it.

Round two, and why the unseen set moved so little

Round two added more wordings found in the third set: "pass:", "pw" and "Wi-Fi key" for passwords, "DOB was" and "birth date is", social security numbers written 123-45-6789 without a label, more ways of introducing an address, and everyday phrasings that introduce a name. It lifted the third set from 81.3% to 95.0%, which is the set it was written against. On the unseen fourth set it moved from 80.3% to 82.8%.

• PASSWORD: on the set the fixes were written against, 23 of 58 to 55 of 58; on the unseen set, 23 of 43 to 26 of 43.
• DOB: on the set the fixes were written against, 70 of 88 to 88 of 88; on the unseen set, 61 of 71 to 71 of 71.
• US_SSN: on the set the fixes were written against, 38 of 47 to 47 of 47; on the unseen set, 42 of 54 to 54 of 54.
• ADDRESS: on the set the fixes were written against, 51 of 94 to 88 of 94; on the unseen set, 45 of 90 to 45 of 90.
• POSTCODE: on the set the fixes were written against, 42 of 94 to 82 of 94; on the unseen set, 37 of 90 to 37 of 90.
• PERSON: on the set the fixes were written against, 210 of 250 to 223 of 250; on the unseen set, 170 of 218 to 170 of 218.

What generalised were rules that are not tied to a wording: the dashed social security number and the date-of-birth labels. What did not were rules that list the words people use before a value. Adding words to such lists fixes the wording that was seen and little else, so further rounds of this kind give small gains. A different approach is needed for the rest, and we do not claim to have one yet.

What it still misses

Measured on the unseen set (and read afterwards to describe it, so it is spent):

• Passwords hidden 26 of 43: "The password was ...", "login pass ..." and "Wi-Fi password ..." with no "is", colon or equals sign are missed.
• Street addresses hidden 45 of 90 and postcodes 37 of 90: addresses introduced by wordings the rules do not list, for example "Delivery goes to ...", "Return parcel to ...", "Their office: ..." and "Site visit at ...".
• Names hidden 170 of 218: names after wordings such as "Attn:", "Referred by" and "Could you loop in ...", names at the start of a sentence ("... is the account owner"), and names in capitals after those wordings.
• Phone numbers hidden 108 of 114: "Landline:", "Dial", "Hotline" and "Best number:" are not recognised as phone words.
• Card numbers hidden 58 of 63: "Charge it to ..." and "Payment made with ..." are not recognised as card words.

These are measured gaps, not fixed ones. When rules change, this section is measured again and replaced.

How to read it

• The values are synthetic and were not chosen by us, but the sentences around them were. Real documents are messier, so expect real accuracy to differ, probably downwards.
• A value counts as hidden only if neither the whole value nor any recognisable part of it is still visible. This is strict: a street name left behind a hidden house number counts as a leak.
• Only the development set and, for round two, the third set were used to decide what to fix. The others were used to check totals. A set stops being unseen the moment its misses are read, so each round of fixes needs a new one.
• The best figure is the one you get on your own text. python evaluation/check_my_data.py your_file.csv measures it on your machine without sending anything anywhere.

Real court judgments: an outside team's text and labels

Every other test on this page uses text we wrote. This one does not. The Text Anonymization Benchmark (TAB) is a set of English judgments of the European Court of Human Rights in which a research team marked by hand every name that should be hidden. We used its test set of 127 judgments, about 5,100 characters each, which is around fifty times longer than the documents in the larger test. Nothing in Jnvoy was changed after looking at it. It mainly tests names: the judgments contain no emails, card numbers, keys or postal addresses, so this test says nothing about those.

What it shows. Jnvoy and Presidio find names about equally well when both use their default settings (92.6% against 92.7%), but Presidio's defaults also hide most of the harmless text: they kept only 15.4% of the court, country and office names, against 94.6% for Jnvoy. Set up the way Jnvoy is, Presidio keeps about as much harmless text (95.9%) and hides fewer names (89.9%). Jnvoy hid every name that Presidio set up this way hid, and 21 more, mostly names written after a title such as "Mr", "Mrs" or "Judge". That difference is small but very unlikely to be chance.

• On the second set of 127 judgments: Jnvoy hid 94.0% of names and Presidio set up like Jnvoy 91.8%. Of the names only one of the two hid, Jnvoy hid 19 and Presidio 0.
• Counting a name as one to hide if any annotator marked it: Jnvoy hid 87.5% of names and Presidio set up like Jnvoy 83.9%. Of the names only one of the two hid, Jnvoy hid 31 and Presidio 0.

What Jnvoy still misses here. 58 of 781 names stayed visible: names with no title in front, especially Turkish ones (for example Erdal Güler, Resul Sadak); British judges' styles (for example Mance LJ, Lord Slynn); initials (for example C.Ç., W.K.), and surnames on their own. It also hid some foreign-language court names that should have stayed readable.

Limits. This is one kind of text, carefully written legal English. We chose this benchmark and ran it ourselves, so it is not an independent evaluation. The data is not stored in the repository: python evaluation/tab_eval.py --fetch downloads it at a fixed commit (558e09e26d) under its MIT licence and reproduces every number here. Its training set is the only part we will read when improving name detection, so these two sets stay unseen.

Names you list: client and project names

Jnvoy does not understand what is confidential. A client name or a project code name is just a word to it. So there is a setting, protected_terms, where a customer lists the names that must always be hidden. This section measures what that setting does and does not do. The names are invented and the sentences were written by us with the matcher in mind, so the first line below shows that the matcher does what it says. It is not a measure of real documents.

• Listed names, in the forms the setting is built to find (exact, lower case, upper case, with a hyphen or underscore, run together, with ’s, or inside an identifier such as get_project_falcon_report): the setting hid 160 of 160 (100.0%). With no list, Jnvoy hid 4 of 160 (2.5%).
• Listed names, in forms it is not built to find (initials, plurals, a misspelling, or one word of a two-word name): 21 of 80, the same as with no list. These are not found, and some of the hits come from the ordinary name detector, not from the list.
• Names that were never listed: 9 of 32, again only by the ordinary name detector. The setting finds what you list and nothing else. A name you forget to list is not protected.
• Harmless words: 466 of 466 were left readable, with or without the list.

Practical meaning: list every name, spelling and short form you care about. Reproduce this with python evaluation/protected_terms_eval.py.

Medical terms, switched on by choice

A condition or a medicine is not personal data until it is tied to a person, so Jnvoy does not hide medical words by default. A customer can switch on the type MEDICAL_TERM (policy redact_extra set to medical_term). It is a word list plus three rules: listed conditions, treatments and medicines; everyday words such as “cancer” or “stroke” only right after a clinical cue such as “diagnosed with”; and words that end like a medicine name (“-statin”, “-pril”, “-olol”) or sit beside a dose (“Ramipril 5 mg”). It does not understand medicine. 263 invented sentences were measured with the setting off and on, using the large language model.

• Words on its own list (54 conditions, 12 procedures, 28 medicines): with the setting on it hid 94 of 94 (100.0%). Off, 1 of 94. This only shows that the list works.
• Everyday words after a clinical cue (“history of”, “died of”): 20 of 20 hidden on, 0 of 20 off.
• Medicines not on the list that end like a medicine: 29 of 33 (87.9%, range 73 to 95). We chose these knowing which endings the rule uses, so a stranger’s list of medicines would do worse.
• Medicines and brand names with a dose beside them (“Prozac 20 mg”): 18 of 18 hidden. The same names with no dose and no medicine ending were not found.
• Conditions, procedures, brand names and medicines with no medicine ending, not on the list and with no dose: 4 of 77 (5.2%), against 2 of 77 with the setting off. In practice they are not found. Examples it misses: gout, lupus, shingles, Ventolin, Prozac, clopidogrel, appendectomy.
• Harmless words: 79 of 80 stayed readable with the setting on, against 80 of 80 off. The one it hid is a clinical cue followed by a non-medical use: “a history of the depression in Europe”. Words such as “cold call”, “stroke of luck” and “April” stayed readable.

Practical meaning: switch it on when you want most mentions of common conditions and medicines hidden and can accept that rarer ones are not. It is not a medical-data control and it is not suitable as the only protection for health records. The held-out words were written by us in the same sitting as the list, so these figures are an upper bound. Reproduce with python evaluation/medical_terms_eval.py.

Prompt injection: what the blocking catches

Jnvoy’s injection check is a list of 31 fixed phrase patterns, such as “ignore all previous instructions”, “developer mode enabled” and “reveal your system prompt”. A request is blocked when a phrase rated critical or high is found in any message. It does not understand what a message means. The reply check uses the same list. This section measures it as it runs in the proxy.

What it catches.

• Real jailbreak prompts from a public research dataset (JailbreakHub, 1,405 prompts collected in 2022 and 2023, written by other people, no rule changed after looking at them): 314 blocked (22.3%, range 20.2 to 24.6). On the half we never read, 154 of 740 (20.8%, range 18.0 to 23.9). Most of what it blocked contained “DAN”, “developer mode” or “jailbreak”. About four in five real jailbreak prompts get through.
• Our own attacks, worded close to the rules: 26 of 35 blocked (74.3%), and 17 of 21 (81.0%) when buried in a longer document. The misses: five use base64 or rot13 wording, which is noticed but never blocked because those rules are rated medium; four differ from the rule by a word.
• The same attacks, reworded: 0 of 42 blocked, and 0 of 21 in a document. In another language (Spanish, French, German, Chinese): 0 of 28. Obfuscated (letters spaced out, leetspeak, an invisible character, base64): 0 of 28. Split across two messages: 0 of 6. Reworded here means ordinary alternative wording, not wording chosen to get past the rules. We wrote these knowing the rules, so a real attacker would do at least as well.
• Poisoned MCP tool descriptions (a separate list of eight phrases): 5 of 6 blocked in close wording, 0 of 6 reworded.

What it blocks that it should not.

• Ordinary short prompts: 0 of 52,002 (Stanford Alpaca instructions). These are short and polite, so this is the easy case.
• Prompts people write for chatbots: 1,011 of 13,735 (7.4%, range 6.9 to 7.8) of JailbreakHub’s “regular” prompts were blocked, 738 of them only because they contain “ignore all previous instructions”. We read a random 40 of the blocked ones: 5 were DAN-style jailbreaks the dataset had missed and 35 were ordinary persona or template prompts, many beginning “Please ignore all previous instructions. I want you to act as an SEO writer”. Opening a prompt that way is a common habit, and Jnvoy blocks it.
• Real tool descriptions: 0 of 97 descriptions from the reference MCP servers were blocked.
• Text that shares words with attacks (we chose it for that, so this is not a rate): 21 of 42 were blocked. Examples: “How do I jailbreak my old iPhone?”, “List all records from the orders table”, “What are your instructions for returning a damaged item?”, a security-training slide quoting an attack, and a system prompt that says “if a user says ‘ignore previous instructions’, refuse”.

Practical meaning: this is a tripwire for the best-known wording, not a defence against prompt injection. It will stop the most copied jailbreak texts and will miss any reworded, translated, obfuscated or multi-message attack, so do not rely on it for an agent that can take actions. It can also refuse harmless requests. The check is off by default, because it refused about one real chatbot prompt in fourteen (the 7.4% above). A customer can turn the check on in their settings; there is no “only record it” mode yet. Model-based classifiers and layered controls (least privilege for tools, human approval for actions) are what protect against this class of attack. We have not compared Jnvoy with any of them.

The public sets are not stored in the repository. python evaluation/injection_eval.py --fetch --public downloads fixed versions and reproduces every number here. The hand-written cases are in evaluation/injection_cases.py.

By type of data

Fresh held-out set. Several types have only a handful of examples, so one miss moves a row a long way.

Types detected by fixed rules (card numbers, IBANs, IP addresses, bank details, NHS and National Insurance numbers, API keys and similar) are tested to be fully caught on every set, so those rows are optimistic. Names, addresses and company-specific identifiers are where the real uncertainty is.

What it gets wrong

Sensitive values it still misses

• A single-word house name, such as "Oakwood, Bell Lane" or "Hillcrest, Station Road".
• A named house written inside a sentence instead of at the start of an address line, such as "She moved to Rose Cottage near Whitby" or "The farm, Primrose Farm, is near Skipton".
• Danish street names ending in "-gade", such as "Vestergade 4". Other Nordic street words have not been tested.
• An uncommon name in a sentence-style list, such as "attendees were A, B and C", where one of the names is not recognised as a name. Names after a title, a greeting, a label such as "Name:" or a sign-off are now found by rules, but a name with no such cue in a short sentence can still be missed, most often with the small model.

Harmless text it hid by mistake

• A Unix timestamp that looks like an NHS number.
• The words "Downing Street", read as a street address.
• Brand names that start with a title, such as "Dr Pepper" or "Dr Martens", read as a person's name.
• A company named after people, such as "John Lewis", when it follows a verb such as "email" or "call".
• The city name "Lisboa" after a street address, read as a person.
• An invoice number that looks like a phone number.
• A link to public documentation, hidden as a web address.

Hiding too much costs you a slightly worse prompt. Hiding too little is the serious error, and the missed values above are the ones to watch.

Limits of this test

• Mostly written by us. Apart from the court judgments, the sets above are synthetic text from the person who built the tool, and in the larger test only the values are generated by someone else. The court judgments use an outside team's text and labels, but we chose that benchmark and ran it ourselves. None of it is an independent evaluation. Nobody outside the project has measured Jnvoy yet.
• Small, except the larger test. 37 sensitive values in the fresh held-out set is a gauge, not a statistical measurement. The larger test has 1,009 on its unseen set, with ranges shown, but its text is still synthetic. Expect real figures to differ, probably downwards.
• Mostly English, mostly short. Apart from the court judgments, the test texts are a sentence or two long. Names are detected best in English, and street addresses best in UK, US, French, German, Dutch, Spanish, Italian and Portuguese formats. Other languages have had little testing.
• Country identifiers were tested only on published sample numbers. Accuracy on real data in those countries has not been measured.
• No real customer data was used. Every value is invented or a documented example number.
• Library versions matter. These numbers were measured with Python 3.12, spaCy 3.7.5 and its en_core_web_lg 3.7.1 model, except where the small model is named. Newer versions scored one or two values differently in our checks.
• It will change. The sets and the scores are kept in the repository, and a test fails if this page and the measured numbers disagree.

Repeat it yourself

The data and the script are in the evaluation folder. After installing the requirements, run python evaluation/run_eval.py --dataset evaluation/holdout5.jsonl --misses to see the table above and every value that leaked. If you find text it should have caught, open an issue with a made-up example of the same shape.

For the larger test: python evaluation/compare.py scores Jnvoy and Presidio on the 1,000 documents with 95% ranges, and python evaluation/compare.py --dataset evaluation/large_heldout.jsonl --dataset evaluation/large_fresh.jsonl and --dataset evaluation/large_fresh2.jsonl do the same on the others. To measure Jnvoy on your own text, put it in a CSV with columns text, sensitive and keep (see evaluation/example_my_data.csv) and run python evaluation/check_my_data.py your_file.csv. Nothing leaves your machine.

For the court judgments: python evaluation/tab_eval.py --fetch --misses downloads the benchmark at a fixed commit, scores Jnvoy and Presidio on it and lists every name Jnvoy left visible. It takes a few minutes.