Expected 'add_job' to be called once. Called 2 times.

[brennie]

We have an intermittent failure in the Cirrus testsuite:

=================================== FAILURES ===================================
_____________________ test_fetch_schedule_recipes_failure ______________________

self = <MagicMock name='scheduler.add_job' id='281473687159760'>
args = (<function fetch_schedule_recipes at 0xffffb3338720>, 'interval')
kwargs = {'max_instances': 1, 'max_retries': 3, 'seconds': 30}
msg = "Expected 'add_job' to be called once. Called 2 times.\nCalls: [call(<function fetch_schedule_recipes at 0xffffb333872...\n call(<function fetch_schedule_recipes at 0xffffb3338720>, 'interval', seconds=30, max_instances=1, max_retries=3)]."

    def assert_called_once_with(self, /, *args, **kwargs):
        """assert that the mock was called exactly once and that that call was
        with the specified arguments."""
        if not self.call_count == 1:
            msg = ("Expected '%s' to be called once. Called %s times.%s"
                   % (self._mock_name or 'mock',
                      self.call_count,
                      self._calls_repr()))
>           raise AssertionError(msg)
E           AssertionError: Expected 'add_job' to be called once. Called 2 times.
E           Calls: [call(<function fetch_schedule_recipes at 0xffffb3338720>, 'interval', seconds=30, max_instances=1, max_retries=3),
E            call(<function fetch_schedule_recipes at 0xffffb3338720>, 'interval', seconds=30, max_instances=1, max_retries=3)].

/usr/local/lib/python3.11/unittest/mock.py:950: AssertionError

During handling of the above exception, another exception occurred:

scheduler_mock = <MagicMock name='scheduler' id='281473688277648'>
remote_setting_mock = <MagicMock name='remote_setting' id='281473687089616'>
exception = Exception('some error')

    @pytest.mark.asyncio
    async def test_fetch_schedule_recipes_failure(
        scheduler_mock, remote_setting_mock, exception
    ):
        remote_setting_mock.fetch_recipes.side_effect = exception
    
        await fetch_schedule_recipes()
    
        remote_setting_mock.fetch_recipes.assert_called_once()
    
        # Check that a job was added to the scheduler to retry after 30 seconds
>       scheduler_mock.add_job.assert_called_once_with(
            fetch_schedule_recipes,
            "interval",
            seconds=30,
            max_instances=1,
            max_retries=3,
        )
E       AssertionError: Expected 'add_job' to be called once. Called 2 times.
E       Calls: [call(<function fetch_schedule_recipes at 0xffffb3338720>, 'interval', seconds=30, max_instances=1, max_retries=3),
E        call(<function fetch_schedule_recipes at 0xffffb3338720>, 'interval', seconds=30, max_instances=1, max_retries=3)].

tests/test_main.py:175: AssertionError
------------------------------ Captured log setup ------------------------------
ERROR    cirrus.experiment_recipes:experiment_recipes.py:48 Failed to fetch recipes: HTTPConnectionPool(host='kinto', port=8888): Max retries exceeded with url: /v1/buckets/main/collections/nimbus-web-experiments/records (Caused by NameResolutionError("<urllib3.connection.HTTPConnection object at 0xffffb3368890>: Failed to resolve 'kinto' ([Errno -2] Name or service not known)"))
ERROR    cirrus.main:main.py:234 Failed to fetch recipes: HTTPConnectionPool(host='kinto', port=8888): Max retries exceeded with url: /v1/buckets/main/collections/nimbus-web-experiments/records (Caused by NameResolutionError("<urllib3.connection.HTTPConnection object at 0xffffb3368890>: Failed to resolve 'kinto' ([Errno -2] Name or service not known)"))
------------------------------ Captured log call -------------------------------
ERROR    cirrus.main:main.py:234 Failed to fetch recipes: some error

• https://app.circleci.com/pipelines/github/mozilla/experimenter/38759/workflows/3f9492f3-db74-4211-ba3b-b25a47193a20/jobs/185854
• https://app.circleci.com/pipelines/github/mozilla/experimenter/38951/workflows/b77d4542-16a2-4547-bce4-f5a8008fb698/jobs/187680

┆Issue is synchronized with this Jira Task