ENH: spatial: remove redundant weight summing from hamming distance

[DavidM603]

Instead of re-summing all the weights each time a pair of inputs is compared, sum the weights once and re-use the result. When I measured locally, trimming this extra work allowed about 25% more throughput on chars and 5% on doubles. I'll make this a draft to start, and seek review if the ci benchmarks corroborate.

Reference issue

n/a

What does this implement/fix?

Moderate performance improvements to existing functionality.

Additional information

n/a

[tylerjereddy]

So far it doesn't seem to cross the threshold for the current benchmarks via i.e., asv continuous -e -b ".*Xdist.*" main hamming_distance_speedup locally on x86_64 Linux.

That said, 25 % for a specific data type might be promising. If you find that this data type is not covered by benchmarks/benchmarks/spatial.py, which I think focuses on float types, it would probably be fine to either expand those benchmarks to include the new type, or add another benchmark to cover the case. I believe string types may only be of interest for a subset of the distance metrics, so may need a small separate benchmark in there I suppose.

XdistWeighted may be the closest benchmark to what you are touching at the moment?

[DavidM603]

So far it doesn't seem to cross the threshold for the current benchmarks via i.e., asv continuous -e -b ".*Xdist.*" main hamming_distance_speedup locally on x86_64 Linux.

That said, 25 % for a specific data type might be promising. If you find that this data type is not covered by benchmarks/benchmarks/spatial.py, which I think focuses on float types, it would probably be fine to either expand those benchmarks to include the new type, or add another benchmark to cover the case. I believe string types may only be of interest for a subset of the distance metrics, so may need a small separate benchmark in there I suppose.

XdistWeighted may be the closest benchmark to what you are touching at the moment?

I took a look at the benchmarking code in that file and found something on line 293. All of the distance measures are using 3 dimensions, but for some metrics that's a few orders of magnitude smaller than the ordinary use and doesn't really provide a good sample. For example, in XdistWeighted.time_pdist the largest sample size for hamming distance (100 x 3) runs in 291us, but the C part alone clocks in around 12us at that scale, so ~95% of that bench is spent in wrappers. I'd like to take on adding larger, but reasonable, benchmarks for the metrics that warrant higher dimensions. Maybe later I could add benches for the metrics that take non-double types too.