Enum memchr::memmem::Prefilter

Prefilter controls whether heuristics are used to accelerate searching.

A prefilter refers to the idea of detecting candidate matches very quickly, and then confirming whether those candidates are full matches. This idea can be quite effective since it’s often the case that looking for candidates can be a lot faster than running a complete substring search over the entire input. Namely, looking for candidates can be done with extremely fast vectorized code.

The downside of a prefilter is that it assumes false positives (which are candidates generated by a prefilter that aren’t matches) are somewhat rare relative to the frequency of full matches. That is, if a lot of false positives are generated, then it’s possible for search time to be worse than if the prefilter wasn’t enabled in the first place.

Another downside of a prefilter is that it can result in highly variable performance, where some cases are extraordinarily fast and others aren’t. Typically, variable performance isn’t a problem, but it may be for your use case.

The use of prefilters in this implementation does use a heuristic to detect when a prefilter might not be carrying its weight, and will dynamically disable its use. Nevertheless, this configuration option gives callers the ability to disable prefilters if you have knowledge that they won’t be useful.