Improve HTTP probe gap between targets for automatic discovery

[manugarg]

Copied from google/cloudprober#575, originally filed by @benner.

Improve HTTP probe gap between targets for automatic discovery

Current behavior (interval_between_targets_msec) is perfect for static target list but target list are updating dynamically (e. g. file_targets) all probes if there are changes has tendency to move to interval beginning.

I spot this possibility from code and made short experiment:

• 1 probe
• 6 targets in file
• interval_between_targets_msec: 10000
• interval_msec: 60000

In beginning if there are no changes everything works as expected (10s between probes):

benner=> SELECT LAG(time, 1) OVER () - time from (SELECT DISTINCT(time) FROM m ORDER BY time) AS m LIMIT 10;
     ?column?     
------------------
 
 -00:00:10.47214
 -00:00:09.223467
 -00:00:10.22748
 -00:00:10.386196
 -00:00:10.057331
 -00:00:09.633533
 -00:00:10.472111
 -00:00:09.223475
 -00:00:10.227483
(10 rows)

Then I manually removed some targets from file end (latest targets to probe in whole interval) and adding it again after update/refresh time. After repeating this few times I'm getting all probes in almost same time:

benner=> SELECT LAG(time, 1) OVER () - time from (SELECT DISTINCT(time) FROM m ORDER BY time) AS m LIMIT 10 OFFSET 50;
     ?column?     
------------------
 -00:00:09.536079
 -00:00:00.097285
 -00:00:00.320518
 -00:00:00.08567
 -00:00:49.960433
 -00:00:09.536071
 -00:00:00.097328
 -00:00:00.320667
 -00:00:00.085556
 -00:00:49.960377
(10 rows)