⚡️ Faster implementation of internal function `runIdToFrequency` #3551

dubzzz · 2023-01-15T14:40:50Z

While it does not seem to be that much harmful in terms of performances at first glance, due to how often we call it, the helper runIdToFrequency tends to be visible for tests not relying on complex generators.

As an example on a property like:

fc.assert(fc.property(fc.constant(1), (_unused) => true));

The helper was initially (before that change) responsible for 4% of the time.

With the current optimization we drastically drop the cost of it as visible in this benchmark:

┌─────────┬──────────────────────────────┬───────────────────────┬───────────────────────┬───────────────────────┬────────────────────┐
│ (index) │             Name             │         Mean          │          P75          │          P99          │        RME         │
├─────────┼──────────────────────────────┼───────────────────────┼───────────────────────┼───────────────────────┼────────────────────┤
│    0    │         'reference'          │ 0.001976805371799113  │ 0.0019000000320374966 │ 0.003399999812245369  │ 0.6809282462222994 │
│    1    │           'log10'            │ 0.002102162820645359  │ 0.0018000002019107342 │ 0.003599999938160181  │ 2.759354493296883  │
│    2    │      'log10 hardcoded'       │ 0.0018829548354208382 │ 0.0018000002019107342 │ 0.0032999999821186066 │ 0.2409024975359869 │
│    3    │   'log10 hardcoded multi'    │ 0.002231573096880576  │ 0.0027000000700354576 │ 0.0035000001080334187 │ 3.699387223537233  │
│    4    │ 'log10 hardcoded multi fast' │ 0.0002505374557727212 │ 0.0002999999560415745 │ 0.0005000000819563866 │  2.22637949700103  │
└─────────┴──────────────────────────────┴───────────────────────┴───────────────────────┴───────────────────────┴────────────────────┘

The implementations used in the benchmarks are:

const safeMathFloor = Math.floor;
const safeMathLog = Math.log;
const runIdToFrequencyReference = (runId) => 2 + safeMathFloor(safeMathLog(runId + 1) / safeMathLog(10));

const log10 = safeMathLog(10);
const runIdToFrequencyLog10 = (runId) => 2 + safeMathFloor(safeMathLog(runId + 1) / log10);

const runIdToFrequencyLog10Hard = (runId) => 2 + safeMathFloor(safeMathLog(runId + 1) / 2.302585092994046);

const runIdToFrequencyLog10HardMulti = (runId) => 2 + safeMathFloor(safeMathLog(runId + 1) * 0.43429448190325176);

const runIdToFrequencyLog10HardMultiFast = (runId) => 2 + ~~(safeMathLog(runId + 1) * 0.43429448190325176);

Category:

Potential impacts:

While it does not seem to be that much harmful in terms of performances at first glance, due to how often we call it, the helper `runIdToFrequency` tends to be visible for tests not relying on complex generators. As an example on a property like: ```ts fc.assert(fc.property(fc.constant(1), (_unused) => true)); ``` The helper was initially (before that change) responsible for 4% of the time. With the current optimization we drastically drop the cost of it as visible in this benchmark: ```txt ┌─────────┬──────────────────────────────┬───────────────────────┬───────────────────────┬───────────────────────┬────────────────────┐ │ (index) │ Name │ Mean │ P75 │ P99 │ RME │ ├─────────┼──────────────────────────────┼───────────────────────┼───────────────────────┼───────────────────────┼────────────────────┤ │ 0 │ 'reference' │ 0.001976805371799113 │ 0.0019000000320374966 │ 0.003399999812245369 │ 0.6809282462222994 │ │ 1 │ 'log10' │ 0.002102162820645359 │ 0.0018000002019107342 │ 0.003599999938160181 │ 2.759354493296883 │ │ 2 │ 'log10 hardcoded' │ 0.0018829548354208382 │ 0.0018000002019107342 │ 0.0032999999821186066 │ 0.2409024975359869 │ │ 3 │ 'log10 hardcoded multi' │ 0.002231573096880576 │ 0.0027000000700354576 │ 0.0035000001080334187 │ 3.699387223537233 │ │ 4 │ 'log10 hardcoded multi fast' │ 0.0002505374557727212 │ 0.0002999999560415745 │ 0.0005000000819563866 │ 2.22637949700103 │ └─────────┴──────────────────────────────┴───────────────────────┴───────────────────────┴───────────────────────┴────────────────────┘ ``` The implementations used in the benchmarks are: ```js const safeMathFloor = Math.floor; const safeMathLog = Math.log; const runIdToFrequencyReference = (runId) => 2 + safeMathFloor(safeMathLog(runId + 1) / safeMathLog(10)); const log10 = safeMathLog(10); const runIdToFrequencyLog10 = (runId) => 2 + safeMathFloor(safeMathLog(runId + 1) / log10); const runIdToFrequencyLog10Hard = (runId) => 2 + safeMathFloor(safeMathLog(runId + 1) / 2.302585092994046); const runIdToFrequencyLog10HardMulti = (runId) => 2 + safeMathFloor(safeMathLog(runId + 1) * 0.43429448190325176); const runIdToFrequencyLog10HardMultiFast = (runId) => 2 + ~~(safeMathLog(runId + 1) * 0.43429448190325176); ```

codesandbox-ci · 2023-01-15T14:42:51Z

This pull request is automatically built and testable in CodeSandbox.

To see build info of the built libraries, click here or the icon next to each commit SHA.

Latest deployment of this branch, based on commit 3b0106d:

Sandbox	Source
Vanilla	Configuration
@fast-check/examples	Configuration

dubzzz · 2023-01-15T14:53:02Z

Benchmark run against https://pkg.csb.dev/dubzzz/fast-check/commit/c000985a/fast-check at https://github.com/dubzzz/fast-check-benchmarks/actions/runs/3923961600

dubzzz · 2023-01-15T14:58:16Z

Code: [...Array(100)].map((_,i) => runIdToFrequency(i)).join(', ');

Before:
2, 2, 2, 2, 2, 2, 2, 2, 2, 3, 3, 3, 3, 3, 3, 3, 3, 3, 3, 3, 3, 3, 3, 3, 3, 3, 3, 3, 3, 3, 3, 3, 3, 3, 3, 3, 3, 3, 3, 3, 3, 3, 3, 3, 3, 3, 3, 3, 3, 3, 3, 3, 3, 3, 3, 3, 3, 3, 3, 3, 3, 3, 3, 3, 3, 3, 3, 3, 3, 3, 3, 3, 3, 3, 3, 3, 3, 3, 3, 3, 3, 3, 3, 3, 3, 3, 3, 3, 3, 3, 3, 3, 3, 3, 3, 3, 3, 3, 3, 4

After:
2, 2, 2, 2, 2, 2, 2, 2, 2, 2, 3, 3, 3, 3, 3, 3, 3, 3, 3, 3, 3, 3, 3, 3, 3, 3, 3, 3, 3, 3, 3, 3, 3, 3, 3, 3, 3, 3, 3, 3, 3, 3, 3, 3, 3, 3, 3, 3, 3, 3, 3, 3, 3, 3, 3, 3, 3, 3, 3, 3, 3, 3, 3, 3, 3, 3, 3, 3, 3, 3, 3, 3, 3, 3, 3, 3, 3, 3, 3, 3, 3, 3, 3, 3, 3, 3, 3, 3, 3, 3, 3, 3, 3, 3, 3, 3, 3, 3, 3, 3

dubzzz · 2023-01-15T15:02:20Z

Before: (frequency value => when it appears for the first time)
2 => 0
3 => 9
4 => 99
5 => 1000
6 => 9999
7 => 99999

After:
2 => 0
3 => 10
4 => 100
5 => 1000
6 => 10000
7 => 100000

codecov · 2023-01-15T15:20:57Z

Codecov Report

Merging #3551 (3197aa9) into main (f213e00) will not change coverage.
The diff coverage is n/a.

❗ Current head 3197aa9 differs from pull request most recent head 3b0106d. Consider uploading reports for the commit 3b0106d to get more accurate results

@@           Coverage Diff           @@
##             main    #3551   +/-   ##
=======================================
  Coverage   95.47%   95.47%           
=======================================
  Files         213      213           
  Lines        5895     5895           
  Branches     1094     1094           
=======================================
  Hits         5628     5628           
  Misses        267      267

Flag	Coverage Δ
unit-tests	`95.47% <ø> (ø)`
unit-tests-14.x-Linux	`95.45% <ø> (-0.02%)`	⬇️
unit-tests-16.x-Linux	`95.47% <ø> (ø)`
unit-tests-18.x-Linux	`95.47% <ø> (ø)`

Flags with carried forward coverage won't be shown. Click here to find out more.

Impacted Files	Coverage Δ
...ages/fast-check/src/check/property/IRawProperty.ts	`100.00% <ø> (ø)`

Help us with your feedback. Take ten seconds to tell us how you rate us. Have a feature suggestion? Share it here.

dubzzz · 2023-01-15T15:21:39Z

With our last approximation of log10, we finally get:
2 => 0
3 => 9
4 => 99
5 => 999
6 => 9999
7 => 99999
8 => 999999

dubzzz · 2023-01-15T15:23:47Z

Measurements of the first benchmark:

Algorithm;3.5.0;main;extra
Property(fc.boolean());7493.866827039029;7817.802385704138;7657.485375205835
Property(fc.integer());7256.729528320608;7769.382320709278;7615.785545975474
Property(fc.maxSafeInteger());7077.5875832304555;7433.679857222167;7332.969052652293
Property(fc.float());6702.424262454433;6991.364936083257;6876.2439431781
Property(fc.double());5410.919158309458;5481.512893671747;5362.895635894777
Property(fc.bigInt());2250.7953131838754;2744.5702871351045;2722.071461409716
Property(fc.char());7206.290010629806;7490.067189924747;7354.6703374272165
Property(fc.string());3659.703384822588;3697.548870741788;3765.5124070556803
Property(fc.string({ minLength: 0, maxLength: 500, size: 'max' }));226.0072534103605;232.2292479109433;232.80266449480533
Property(fc.string({ minLength: 0, maxLength: 25_000, size: 'max' }));3.2389599070732373;3.3034470151527766;3.3415445018713843
Property(fc.array(fc.integer()));4697.18846434163;4956.644254579077;5029.064767426614
Property(fc.array(fc.integer(), { minLength: 0, maxLength: 500, size: 'max' }));324.8703579471513;371.32650933630873;372.1640632106248
Property(fc.array(fc.integer(), { minLength: 0, maxLength: 25_000, size: 'max' }));5.122612750286871;5.881797097086375;5.852833897383533
Property(fc.uniqueArray(fc.integer()));4160.22762443837;4412.891789311261;4432.6190775526375
Property(fc.uniqueArray(fc.integer(), { minLength: 0, maxLength: 500, size:'max' }));213.4280774301873;241.7043764203018;242.27808328422861
Property(fc.anything());417.16770765600404;412.5106602138352;427.5208475971921
Property(fc.constant('a'));8329.702419225961;8440.372753733363;8165.923031963672
Property(fc.constantFrom('a', 'b', 'c'));8131.8813894623545;8471.682292901078;8284.875150376758
Property(fc.option(fc.integer()));6942.5387379144595;7188.676325656557;7016.542257009179
Property(fc.oneof(fc.ascii(), fc.hexa()));6668.669511116389;6813.415952110652;6696.676588919574
Property(fc.oneof(fc.ascii()@w=1, fc.hexa()@w=2));6676.572030735864;6845.67857302604;6698.869376127208
Property(fc.tuple(fc.ascii(), fc.hexa()));5613.384546502257;5779.939219997913;5756.741154290548
Property(fc.record({ascii: fc.ascii(), hexa: fc.hexa()}));3845.94907710702;3877.6602608015273;3831.753468073679
Property(<tree-with-letrec>);3933.224189100185;3983.4018292509495;4050.1312083932858
Property(<comment-generator-letrec>);1995.1048457766792;2000.3802470832977;2022.7964724684525
Property(fc.emailAddress());70.58196548774367;71.09024470622404;71.66512257281506
Property(fc.webUrl());130.98808371094188;130.0477439226024;127.2283721575602
Property(fc.integer().filter(() => true));7276.54137628167;7647.229991765686;7604.180277514843
Property(fc.integer().map(n => n));7302.48169728566;7545.658167510117;7466.204249283315
Property(fc.integer().chain(() => fc.integer()));6439.9591455650625;6728.099701241813;6573.060305078654
Property(fc.integer().noBias());7853.825647968435;8261.84433136545;8053.874092380704
Property(fc.integer().noShrink());7306.275385159062;7836.049821517382;7683.615917262529

Which means:

Algorithm;3.5.0;main;extra
Property(fc.boolean());-;=;-
Property(fc.integer());-;=;-
Property(fc.maxSafeInteger());-;=;=
Property(fc.float());-;=;=
Property(fc.double());=;=;=
Property(fc.bigInt());--;=;=
Property(fc.char());-;=;=
Property(fc.string());=;=;=
Property(fc.string({ minLength: 0, maxLength: 500, size: 'max' }));-;=;=
Property(fc.string({ minLength: 0, maxLength: 25_000, size: 'max' }));=;=;=
Property(fc.array(fc.integer()));-;=;=
Property(fc.array(fc.integer(), { minLength: 0, maxLength: 500, size: 'max' }));--;=;=
Property(fc.array(fc.integer(), { minLength: 0, maxLength: 25_000, size: 'max' }));-;=;=
Property(fc.uniqueArray(fc.integer()));-;=;+
Property(fc.uniqueArray(fc.integer(), { minLength: 0, maxLength: 500, size:'max' }));--;=;=
Property(fc.anything());=;=;=
Property(fc.constant('a'));-;=;-
Property(fc.constantFrom('a', 'b', 'c'));-;=;-
Property(fc.option(fc.integer()));-;=;-
Property(fc.oneof(fc.ascii(), fc.hexa()));-;=;=
Property(fc.oneof(fc.ascii()@w=1, fc.hexa()@w=2));-;=;-
Property(fc.tuple(fc.ascii(), fc.hexa()));-;=;-
Property(fc.record({ascii: fc.ascii(), hexa: fc.hexa()}));-;=;-
Property(<tree-with-letrec>);=;=;=
Property(<comment-generator-letrec>);=;=;=
Property(fc.emailAddress());=;=;=
Property(fc.webUrl());=;=;-
Property(fc.integer().filter(() => true));-;=;=
Property(fc.integer().map(n => n));-;=;=
Property(fc.integer().chain(() => fc.integer()));-;=;=
Property(fc.integer().noBias());-;=;-
Property(fc.integer().noShrink());-;=;-

dubzzz · 2023-01-15T15:25:49Z

New benchmark against https://pkg.csb.dev/dubzzz/fast-check/commit/3b0106dc/fast-check is https://github.com/dubzzz/fast-check-benchmarks/actions/runs/3924089963

dubzzz · 2023-01-15T16:47:31Z

Last benchmark:

Algorithm;3.5.0;main;extra
Property(fc.boolean());771.286381887558;778.1170279160449;766.6870216216012
Property(fc.integer());766.4942131280219;760.1812131466888;770.8908942678032
Property(fc.maxSafeInteger());736.720825081606;725.3988210122144;745.9552900241403
Property(fc.float());697.4642114079002;706.4830335686443;721.4141543351901
Property(fc.double());546.2217881981767;567.5237412502428;563.7827835430841
Property(fc.bigInt());223.54348017007536;271.8855800383519;272.6948876290017
Property(fc.char());763.9928703051208;767.6255435274942;773.2123803021044
Property(fc.string());395.7510651583536;414.77515055323806;405.2161402770733
Property(fc.string({ minLength: 0, maxLength: 500, size: 'max' }));22.04395225599041;23.904426423596515;22.806599050214484
Property(fc.string({ minLength: 0, maxLength: 25_000, size: 'max' }));0.3134166888612203;0.3305766230808069;0.3240909873928782
Property(fc.array(fc.integer()));513.2152887636059;534.1285937811496;527.6774859973233
Property(fc.array(fc.integer(), { minLength: 0, maxLength: 500, size: 'max' }));36.50884584887855;40.86501577802647;39.45393536519161
Property(fc.array(fc.integer(), { minLength: 0, maxLength: 25_000, size: 'max' }));0.5848011277127306;0.6420293480269664;0.6254242999549446
Property(fc.uniqueArray(fc.integer()));443.38182950883487;467.95841796584455;456.72413283004977
Property(fc.uniqueArray(fc.integer(), { minLength: 0, maxLength: 500, size:'max' }));24.24541343391667;26.522807411821944;25.461641351329682
Property(fc.anything());44.29234381526829;45.522261197539265;44.283531178883266
Property(fc.constant('a'));853.5541120117953;858.6849537428322;865.2076709245398
Property(fc.constantFrom('a', 'b', 'c'));835.9176960957369;840.5051277660338;848.5436419199276
Property(fc.option(fc.integer()));733.8760221036456;731.084006851782;732.8467691120053
Property(fc.oneof(fc.ascii(), fc.hexa()));689.0852918737312;684.9543174385589;687.0055384789035
Property(fc.oneof(fc.ascii()@w=1, fc.hexa()@w=2));698.532104380543;691.2480371219679;707.9467118168799
Property(fc.tuple(fc.ascii(), fc.hexa()));587.2136629609873;612.8736532198766;608.0090950511469
Property(fc.record({ascii: fc.ascii(), hexa: fc.hexa()}));395.77493309162816;406.03978181529664;405.5237294793104
Property(<tree-with-letrec>);412.4350688320461;425.9575108028365;421.1847814158811
Property(<comment-generator-letrec>);228.3838775936629;227.32464467243076;226.14632534461813
Property(fc.emailAddress());32.99443464238736;33.93257774892593;33.05182085791688
Property(fc.webUrl());65.39610801107297;67.40305152015263;64.57680308571425
Property(fc.integer().filter(() => true));758.5621305437155;748.7809090248934;763.2605672926582
Property(fc.integer().map(n => n));743.628966714598;747.4914551839219;765.3451569428826
Property(fc.integer().chain(() => fc.integer()));671.6981956325105;673.6236385953611;670.461602026331
Property(fc.integer().noBias());802.3738359511236;799.1355442657764;787.2098948255265
Property(fc.integer().noShrink());764.5671748679748;757.0168405325716;759.9081066249

Algorithm;3.5.0;main;extra
Property(fc.boolean());-;=;-
Property(fc.integer());=;=;=
Property(fc.maxSafeInteger());+;=;+
Property(fc.float());=;=;+
Property(fc.double());-;=;=
Property(fc.bigInt());--;=;=
Property(fc.char());=;=;=
Property(fc.string());-;=;-
Property(fc.string({ minLength: 0, maxLength: 500, size: 'max' }));-;=;-
Property(fc.string({ minLength: 0, maxLength: 25_000, size: 'max' }));-;=;-
Property(fc.array(fc.integer()));-;=;-
Property(fc.array(fc.integer(), { minLength: 0, maxLength: 500, size: 'max' }));--;=;-
Property(fc.array(fc.integer(), { minLength: 0, maxLength: 25_000, size: 'max' }));-;=;-
Property(fc.uniqueArray(fc.integer()));-;=;-
Property(fc.uniqueArray(fc.integer(), { minLength: 0, maxLength: 500, size:'max' }));-;=;-
Property(fc.anything());=;=;=
Property(fc.constant('a'));-;=;+
Property(fc.constantFrom('a', 'b', 'c'));-;=;+
Property(fc.option(fc.integer()));+;=;=
Property(fc.oneof(fc.ascii(), fc.hexa()));+;=;=
Property(fc.oneof(fc.ascii()@w=1, fc.hexa()@w=2));=;=;+
Property(fc.tuple(fc.ascii(), fc.hexa()));-;=;=
Property(fc.record({ascii: fc.ascii(), hexa: fc.hexa()}));-;=;=
Property(<tree-with-letrec>);-;=;=
Property(<comment-generator-letrec>);=;=;=
Property(fc.emailAddress());-;=;=
Property(fc.webUrl());-;=;-
Property(fc.integer().filter(() => true));+;=;+
Property(fc.integer().map(n => n));=;=;+
Property(fc.integer().chain(() => fc.integer()));=;=;=
Property(fc.integer().noBias());=;=;-
Property(fc.integer().noShrink());=;=;=

versions

c000985

dubzzz added 2 commits January 15, 2023 15:06

update snaps

3197aa9

better approximation for log10

f8b8502

regenerate snaps

3b0106d

dubzzz merged commit 078eb14 into main Jan 15, 2023

dubzzz deleted the faster-run-to-freq branch January 15, 2023 15:29

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

GitHub Sponsors

⚡️ Faster implementation of internal function `runIdToFrequency` #3551

⚡️ Faster implementation of internal function `runIdToFrequency` #3551

dubzzz commented Jan 15, 2023 •

edited

Loading

codesandbox-ci bot commented Jan 15, 2023 •

edited

Loading

dubzzz commented Jan 15, 2023

dubzzz commented Jan 15, 2023

dubzzz commented Jan 15, 2023

codecov bot commented Jan 15, 2023 •

edited

Loading

dubzzz commented Jan 15, 2023

dubzzz commented Jan 15, 2023 •

edited

Loading

dubzzz commented Jan 15, 2023

dubzzz commented Jan 15, 2023

⚡️ Faster implementation of internal function runIdToFrequency #3551

⚡️ Faster implementation of internal function runIdToFrequency #3551

Conversation

dubzzz commented Jan 15, 2023 • edited Loading

codesandbox-ci bot commented Jan 15, 2023 • edited Loading

dubzzz commented Jan 15, 2023

dubzzz commented Jan 15, 2023

dubzzz commented Jan 15, 2023

codecov bot commented Jan 15, 2023 • edited Loading

Codecov Report

dubzzz commented Jan 15, 2023

dubzzz commented Jan 15, 2023 • edited Loading

dubzzz commented Jan 15, 2023

dubzzz commented Jan 15, 2023

⚡️ Faster implementation of internal function `runIdToFrequency` #3551

⚡️ Faster implementation of internal function `runIdToFrequency` #3551

dubzzz commented Jan 15, 2023 •

edited

Loading

codesandbox-ci bot commented Jan 15, 2023 •

edited

Loading

codecov bot commented Jan 15, 2023 •

edited

Loading

dubzzz commented Jan 15, 2023 •

edited

Loading