How to Evaluate a Grooming Detection Classifier

If you are evaluating a grooming or CSAM detection classifier, the single most useful skill is knowing which numbers to trust and which ones can be true and misleading at the same time. Vendors will show you accuracy, precision, recall, F1, and often a headline like "99% accurate." Some of these numbers are stable across deployments. Others depend entirely on how common the target behavior is in the data they were measured on. Grooming is rare in any real message stream, and that rarity is exactly where naive metrics break.

This is a practical guide to reading those numbers honestly, written for buyers rather than model builders.

Start with the metrics that do not move

Lead your evaluation with metrics that do not depend on how common the positive class is.

Recall, also called sensitivity or the true positive rate, is the share of actual grooming cases the model catches. Recall = TP / (TP + FN). It depends only on the positive cases, so it does not shift when prevalence shifts.

False positive rate is the share of benign cases the model wrongly flags. FPR = FP / (FP + TN). It also depends only on its own class.

ROC AUC summarizes the tradeoff between recall and false positive rate across every threshold. Google's Machine Learning Crash Course gives the interpretation worth memorizing: AUC is the probability that the model ranks a randomly chosen positive example above a randomly chosen negative one. 1.0 is perfect, 0.5 is a coin flip. AUC is a reasonable prevalence-independent summary of ranking quality.

These three are your foundation because they describe the model itself, not the population you happen to run it on.

The base-rate trap

Now the metric that will mislead you: precision. Precision = TP / (TP + FP), the share of flags that are correct. It sounds like exactly what you want to know, and it is, but it is not a fixed property of the model. Precision depends on prevalence. As the Wikipedia entry on precision and recall states, recall depends only on positive cases and is unaffected by the class ratio, but precision is not. Lower the share of positives in the data and precision falls, even though the model has not changed.

The Wikipedia article on the base rate fallacy states the mechanism directly: when prevalence is lower than the test's false positive rate, even tests with a very low false positive rate in any individual case will produce more false positives than true positives overall. The reason is arithmetic. There are so many more negatives than positives that even a small false positive percentage, applied to that huge pool, swamps the true positives.

The worked example in that article is worth internalizing. At 2% prevalence with a 5% false positive rate, only about 20 of 69 positive results are real. That is roughly 29% precision from a test many people would call "95% accurate." Nothing is wrong with the test. The population is just mostly negative.

Grooming detection lives at the far end of this. A recent academic grooming classifier evaluated on the PANC dataset described its own test set as highly imbalanced, 11,802 normal conversations against 69 grooming conversations. In that same work, a baseline model reported perfect recall of 1.000 but precision of only 0.188, a vivid picture of a model that catches everything and floods reviewers with false positives. The stronger model reached 0.819 precision, but only after the authors built a more balanced 10-to-1 test set to measure on. Which brings us to the two rules that follow.

Two rules for reading precision

Never accept a precision number without the base rate it was measured at. "94% precision" is not a fact about the model. It is a fact about the model on a specific dataset with a specific prevalence. Ask what that prevalence was. If the vendor cannot tell you, the number is uninterpretable.

Insist on evaluation at realistic prevalence. Grooming is a fraction of a percent of real traffic. A precision figure measured on a balanced 50-50 test set, or even a rebalanced 10-to-1 set, will be far higher than what you will see in production. Rebalancing the test set is standard practice in research, and it is not dishonest when disclosed, but it inflates precision relative to your deployment. Ask to see performance on a low-prevalence set that resembles your actual stream.

Thresholds, PR curves, and the tradeoff you control

Precision and recall trade off against each other, and you set where. Raising the decision threshold cuts false positives and raises precision, at the cost of missing more real cases. Lowering it does the reverse. There is no single correct point. There is a point that fits your review capacity and your tolerance for missed cases.

Because of this, ask for a precision-recall curve, not just a single operating point, and not only a ROC curve. As both scikit-learn's documentation and Google's crash course note, precision-recall curves are the more informative view when classes are very imbalanced, which is always the case here. A model can have an impressive ROC AUC while its real-world precision at a usable recall is poor.

Calibration, robustness, and review

Calibration. Many tools return a confidence score. A score of 0.8 should mean something. A well-calibrated classifier, in scikit-learn's definition, is one where among the cases it scored near 0.8, about 80% are actually positive. Ask whether scores are calibrated and to see a reliability diagram. If they are not, you cannot use the score to set thresholds or prioritize a queue with any confidence.

Multilingual and adversarial robustness. Offenders adapt. They misspell, use codewords, and switch languages. Ask how the model performs outside English and how it holds up against deliberate evasion, because a model measured only on clean English will degrade in the wild.

Human review is not optional. At grooming's prevalence, some rate of false positives is unavoidable no matter how good the model is. That is the base-rate math, not a defect. So the model's job is to prioritize a queue for people, not to act on its own. Confirm the vendor's output is built to route to human reviewers, and remember that any downstream CyberTipline report is made by the platform and reviewed by people. It is never automatic.