Skip to content
Permalink

Comparing changes

Choose two branches to see what’s changed or to start a new pull request. If you need to, you can also or learn more about diff comparisons.

Open a pull request

Create a new pull request by comparing changes across two branches. If you need to, you can also . Learn more about diff comparisons here.
base repository: box/flaky
Failed to load repositories. Confirm that selected base ref is valid, then try again.
Loading
base: v3.7.0
Choose a base ref
...
head repository: box/flaky
Failed to load repositories. Confirm that selected head ref is valid, then try again.
Loading
compare: v3.8.0
Choose a head ref
  • 9 commits
  • 35 files changed
  • 9 contributors

Commits on Aug 22, 2020

  1. Update Python versions (#169)

    * Test on all supported versions
    
    * Drop support for EOL Python 3.4
    
    * Drop support for EOL Python 2.7
    
    * Upgrade Python syntax with pyupgrade --py3-plus
    
    * Drop support for EOL Python 2.7
    
    * Remove duplicate line
    hugovk authored Aug 22, 2020
    Copy the full SHA
    32fd411 View commit details

Commits on Apr 14, 2021

  1. Verified

    This commit was created on GitHub.com and signed with GitHub’s verified signature.
    Copy the full SHA
    1e8c954 View commit details

Commits on Sep 1, 2023

  1. Verified

    This commit was created on GitHub.com and signed with GitHub’s verified signature.
    Copy the full SHA
    23e6d9e View commit details
  2. Verified

    This commit was created on GitHub.com and signed with GitHub’s verified signature.
    Copy the full SHA
    b9eb0c7 View commit details
  3. GitHub Action to run tox (#193)

    Because Travis CI is no longer free to open source projects.
    cclauss authored Sep 1, 2023

    Verified

    This commit was created on GitHub.com and signed with GitHub’s verified signature.
    Copy the full SHA
    8de75b5 View commit details

Commits on Sep 6, 2023

  1. Remove nose support (#181)

    The nose test runner hasn't been actively developed since 2016.
    
    Co-authored by mcepl@suse.com
    stratakis authored Sep 6, 2023

    Verified

    This commit was created on GitHub.com and signed with GitHub’s verified signature.
    Copy the full SHA
    23198fe View commit details

Commits on Sep 21, 2023

  1. Verified

    This commit was created on GitHub.com and signed with GitHub’s verified signature.
    Copy the full SHA
    69c297e View commit details

Commits on Mar 4, 2024

  1. Simplify flaky dependency (#197)

    * Remove dependency on genty external package.
    
    Whole parametrization of tests can be done now easily with subTest
    from the standard library.
    
    * External package mock is not needed anymore.
    mcepl authored Mar 4, 2024

    Verified

    This commit was created on GitHub.com and signed with GitHub’s verified signature.
    Copy the full SHA
    b61e722 View commit details

Commits on Mar 10, 2024

  1. Fix incompatibilities with newer dependencies (#201)

    This commit fixes an incompatibility with pytest 8.1.1,
    and  some with python 3.12.
    
    This commit also fixes up some pycodestyle and
    pylint problems that snuck in with some earlier commits.
    
    Fixes #199 and #198
    ---------
    
    Co-authored-by: Jonathan Karlsen <JONAK@equinor.com>
    Jeff-Meadows and jonathan-eq authored Mar 10, 2024

    Verified

    This commit was created on GitHub.com and signed with GitHub’s verified signature.
    Copy the full SHA
    2bb1514 View commit details
42 changes: 42 additions & 0 deletions .github/workflows/tox.yml
Original file line number Diff line number Diff line change
@@ -0,0 +1,42 @@
name: tox
on: [push, pull_request]
jobs:
tox-jobs:
strategy:
fail-fast: false
matrix:
job: [coverage, pycodestyle, pylint, readme]
runs-on: ubuntu-latest
steps:
- uses: actions/checkout@v3
- uses: actions/setup-python@v4
with:
python-version: 3.x
cache: 'pip' # Only use if there are requirements.txt files
- run: pip install --upgrade pip
- run: pip install tox
- run: tox -e ${{ matrix.job }}
# TODO: Add coveralls token
#- if: matrix.job == 'coverage'
# run: |
# pip install coveralls
# coveralls

tox:
strategy:
fail-fast: false
# max-parallel: 4
matrix:
os: [ubuntu-latest] # [macos-latest, ubuntu-latest, windows-latest]
python: ['3.8', '3.9', '3.10', '3.11', 'pypy3.10']
runs-on: ${{ matrix.os }}
steps:
- uses: actions/checkout@v3
- uses: actions/setup-python@v4
with:
python-version: ${{ matrix.python }}
cache: 'pip' # Only use if there are requirements.txt files
- run: python3 --version && python --version
- run: pip install --upgrade pip
- run: pip install tox
- run: tox -e py
1 change: 0 additions & 1 deletion .gitignore
Original file line number Diff line number Diff line change
@@ -24,7 +24,6 @@ pip-log.txt
# Unit test / coverage reports
.coverage
.tox
nosetests.xml

# Translations
*.mo
13 changes: 6 additions & 7 deletions .pylintrc
Original file line number Diff line number Diff line change
@@ -16,7 +16,8 @@

# List of plugins (as comma separated values of python modules names) to load,
# usually to register additional checkers.
#load-plugins=
load-plugins=pylint.extensions.no_self_use



[MESSAGES CONTROL]
@@ -38,10 +39,8 @@

# C0111 => Missing docstring
# W0108 => unnecessary lambda
# W0142 => Used * or ** magic
# R0921 => Abstract class not referenced
# R0205 => Useless object inheritence: codebase still supports 2.7
disable=I, C0111, W0108, W0142, R0921, R0205
disable=I, C0111, W0108, R0205


[REPORTS]
@@ -54,7 +53,7 @@ output-format=text
# Put messages in a separate file for each module / package specified on the
# command line instead of printing them on stdout. Reports (if any) will be
# written in a file name "pylint_global.[txt|html]".
files-output=no
# files-output=no

# Tells whether to display a full report or only the messages
reports=no
@@ -74,7 +73,7 @@ msg-template={module}:{line}:{column}: [{msg_id}({symbol}), {obj}] {msg}
[BASIC]

# List of builtins function names that should not be used, separated by a comma
bad-functions=map,filter,apply,input
# bad-functions=map,filter,apply,input

# Regular expression which should only match correct module names
module-rgx=(([a-z_][a-z0-9_]*)|([A-Z][a-zA-Z0-9]+))$
@@ -259,4 +258,4 @@ int-import-graph=

# Exceptions that will emit a warning when being caught. Defaults to
# "Exception"
overgeneral-exceptions=Exception
overgeneral-exceptions=builtins.Exception
24 changes: 0 additions & 24 deletions .travis.yml

This file was deleted.

2 changes: 2 additions & 0 deletions HISTORY.rst
Original file line number Diff line number Diff line change
@@ -6,6 +6,8 @@ Release History
Upcoming
++++++++

- nose support has been removed.

3.7.0 (2020-07-07)
++++++++++++++++++

16 changes: 4 additions & 12 deletions README.rst
Original file line number Diff line number Diff line change
@@ -4,16 +4,16 @@ flaky
.. image:: http://opensource.box.com/badges/stable.svg
:target: http://opensource.box.com/badges

.. image:: https://travis-ci.org/box/flaky.svg?branch=master
:target: https://travis-ci.org/box/flaky
.. image:: https://github.com/box/flaky/actions/workflows/tox.yml/badge.svg?branch=master&event=push
:target: https://github.com/box/flaky/actions/workflows/tox.yml

.. image:: https://img.shields.io/pypi/v/flaky.svg
:target: https://pypi.python.org/pypi/flaky

About
-----

Flaky is a plugin for nose or pytest that automatically reruns flaky tests.
Flaky is a plugin for pytest that automatically reruns flaky tests.

Ideally, tests reliably pass or fail, but sometimes test fixtures must rely on components that aren't 100%
reliable. With flaky, instead of removing those tests or marking them to @skip, they can be automatically
@@ -118,12 +118,6 @@ It can also be used to incur a delay between test retries:
Activating the plugin
~~~~~~~~~~~~~~~~~~~~~

Like any nose plugin, flaky can be activated via the command line:

.. code-block:: console
nosetests --with-flaky
With pytest, flaky will automatically run. It can, however be disabled via the command line:

.. code-block:: console
@@ -152,7 +146,7 @@ Pass ``--max-runs=MAX_RUNS`` and/or ``--min-passes=MIN_PASSES`` to control the b
is specified. Flaky decorators on individual tests will override these defaults.


*Additional usage examples are in the code - see test/test_nose/test_nose_example.py and test/test_pytest/test_pytest_example.py*
*Additional usage examples are in the code - see test/test_pytest/test_pytest_example.py*

Installation
------------
@@ -169,8 +163,6 @@ Compatibility

Flaky is tested with the following test runners and options:

- Nosetests. Doctests cannot be marked flaky.

- Py.test. Works with ``pytest-xdist`` but not with the ``--boxed`` option. Doctests cannot be marked flaky.


3 changes: 0 additions & 3 deletions flaky/__init__.py
Original file line number Diff line number Diff line change
@@ -1,4 +1 @@
# coding: utf-8

from __future__ import unicode_literals
from .flaky_decorator import flaky
53 changes: 24 additions & 29 deletions flaky/_flaky_plugin.py
Original file line number Diff line number Diff line change
@@ -1,22 +1,17 @@
# coding: utf-8

from __future__ import unicode_literals

from io import StringIO
from traceback import format_exception

from flaky import defaults
from flaky.names import FlakyNames
from flaky.utils import ensure_unicode_string


class _FlakyPlugin(object):
class _FlakyPlugin:
_retry_failure_message = ' failed ({0} runs remaining out of {1}).'
_failure_message = ' failed; it passed {0} out of the required {1} times.'
_not_rerun_message = ' failed and was not selected for rerun.'

def __init__(self):
super(_FlakyPlugin, self).__init__()
super().__init__()
self._stream = StringIO()
self._flaky_success_report = True
self._had_flaky_tests = False
@@ -42,9 +37,9 @@ def _log_test_failure(self, test_callable_name, err, message):
"""
formatted_exception_info = ''.join(format_exception(*err)).replace('\n', '\n\t').rstrip()
self._stream.writelines([
ensure_unicode_string(test_callable_name),
ensure_unicode_string(message),
ensure_unicode_string(formatted_exception_info),
str(test_callable_name),
str(message),
str(formatted_exception_info),
'\n',
])

@@ -113,7 +108,7 @@ def _should_handle_test_error_or_failure(self, test):
:param test:
The test that has raised an error
:type test:
:class:`nose.case.Test` or :class:`Function`
:class:`Function`
:return:
True, if the test needs to be rerun; False, otherwise.
:rtype:
@@ -135,7 +130,7 @@ def _will_handle_test_error_or_failure(self, test, name, err):
:param test:
The test that has raised an error
:type test:
:class:`nose.case.Test` or :class:`Function`
:class:`Function`
:param name:
The name of the test that has raised an error
:type name:
@@ -164,7 +159,7 @@ def _handle_test_error_or_failure(self, test, err):
:param test:
The test that has raised an error
:type test:
:class:`nose.case.Test` or :class:`Function`
:class:`Function`
:param err:
Information about the test failure (from sys.exc_info())
:type err:
@@ -209,7 +204,7 @@ def _should_rerun_test(self, test, name, err):
:param test:
The test that has raised an error
:type test:
:class:`nose.case.Test` or :class:`Function`
:class:`Function`
:param name:
The test name
:type name:
@@ -233,7 +228,7 @@ def _mark_test_for_rerun(self, test):
:param test:
The test that has raised an error or succeeded
:type test:
:class:`nose.case.Test` or :class:`Function`
:class:`Function`
"""
raise NotImplementedError # pragma: no cover

@@ -258,7 +253,7 @@ def _handle_test_success(self, test):
:param test:
The test that has raised an error
:type test:
:class:`nose.case.Test` or :class:`Function`
:class:`Function`
:return:
True, if the test will be rerun; False, if the test runner should handle it.
:rtype:
@@ -280,7 +275,7 @@ def _handle_test_success(self, test):

if self._flaky_success_report:
self._stream.writelines([
ensure_unicode_string(name),
str(name),
' passed {} out of the required {} times. '.format(
passes,
min_passes,
@@ -409,7 +404,7 @@ def _copy_flaky_attributes(cls, test, test_class):
:param test:
The test that is being prepared to run
:type test:
:class:`nose.case.Test`
:class:`Function`
"""
test_callable = cls._get_test_callable(test)
if test_callable is None:
@@ -432,7 +427,7 @@ def _get_flaky_attribute(test_item, flaky_attribute):
:param test_item:
The test method from which to get the attribute
:type test_item:
`callable` or :class:`nose.case.Test` or :class:`Function`
`callable` or :class:`Function`
:param flaky_attribute:
The name of the attribute to get
:type flaky_attribute:
@@ -454,7 +449,7 @@ def _set_flaky_attribute(test_item, flaky_attribute, value):
:param test_item:
The test callable on which to set the attribute
:type test_item:
`callable` or :class:`nose.case.Test` or :class:`Function`
`callable` or :class:`Function`
:param flaky_attribute:
The name of the attribute to set
:type flaky_attribute:
@@ -474,7 +469,7 @@ def _increment_flaky_attribute(cls, test_item, flaky_attribute):
:param test_item:
The test callable on which to set the attribute
:type test_item:
`callable` or :class:`nose.case.Test` or :class:`Function`
`callable` or :class:`Function`
:param flaky_attribute:
The name of the attribute to set
:type flaky_attribute:
@@ -490,7 +485,7 @@ def _has_flaky_attributes(cls, test):
:param test:
The test that is being prepared to run
:type test:
:class:`nose.case.Test` or :class:`Function`
:class:`Function`
:return:
:rtype:
`bool`
@@ -506,7 +501,7 @@ def _get_flaky_attributes(cls, test_item):
:param test_item:
The test callable from which to get the flaky related attributes.
:type test_item:
`callable` or :class:`nose.case.Test` or :class:`Function`
`callable` or :class:`Function`
:return:
:rtype:
`dict` of `unicode` to varies
@@ -526,7 +521,7 @@ def _add_flaky_test_failure(cls, test, err):
:param test:
The flaky test on which to update the flaky attributes.
:type test:
:class:`nose.case.Test` or :class:`Function`
:class:`Function`
:param err:
Information about the test failure (from sys.exc_info())
:type err:
@@ -587,7 +582,7 @@ def _get_test_callable(cls, test):
:param test:
The test that has raised an error or succeeded
:type test:
:class:`nose.case.Test` or :class:`pytest.Item`
:class:`pytest.Item`
:return:
The test declaration, callable and name that is being run
:rtype:
@@ -603,7 +598,7 @@ def _get_test_callable_name(test):
:param test:
The test that has raised an error or succeeded
:type test:
:class:`nose.case.Test` or :class:`pytest.Item`
:class:`pytest.Item`
:return:
The name of the test callable that is being run by the test
:rtype:
@@ -619,7 +614,7 @@ def _make_test_flaky(cls, test, max_runs=None, min_passes=None, rerun_filter=Non
:param test:
The test in question.
:type test:
:class:`nose.case.Test` or :class:`Function`
:class:`Function`
:param max_runs:
The value of the FlakyNames.MAX_RUNS attribute to use.
:type max_runs:
@@ -636,9 +631,9 @@ def _make_test_flaky(cls, test, max_runs=None, min_passes=None, rerun_filter=Non
Information about the test failure (from sys.exc_info())
- name (`unicode`):
The test name
- test (:class:`nose.case.Test` or :class:`Function`):
- test (:class:`Function`):
The test that has raised an error
- plugin (:class:`FlakyNosePlugin` or :class:`FlakyPytestPlugin`):
- plugin (:class:`FlakyPytestPlugin`):
The flaky plugin. Has a :prop:`stream` that can be written to in
order to add to the Flaky Report.
:type rerun_filter:
Loading