Run unittests automatically on github #796
Merged
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
christian-intra2net
force-pushed
the
unittest-automation
branch
from
11930ee
to
d9746f4
Compare
|
For testing I have merged this branch into my fork's master, results are all green: |
christian-intra2net
force-pushed
the
unittest-automation
branch
from
d9746f4
to
a536e8b
Compare
christian-intra2net
force-pushed
the
unittest-automation
branch
from
5ddf8b0
to
77872a7
Compare
|
Py 3.5 is not available any more. Since we need reliable testing, just keep the test for latest py2, py3 and pypy3.9 . Also fix tests on py2 and skip some for PyPy. Add |
Use github actions for Continuous Integration
Could not find a way to say "newest pypy"; "pypy3" or "pypy3.x" did not work
Save computing time by failing immediately if any test fails. If we return early from one VM, the others are stopped, as well, so this saves lots of parallel time.
Want this to check why requirements installation does not happen as expected
(1) Cannot open a temp file for writing from 2 python processes
simultaneously
(2) Need proper environment setup for Popen (thanks to metatoaster's
reply on stackoverflow https://stackoverflow.com/a/72845540/4405656)
(3) Need different os-separator, so replace hard-coded '/' with
os.path.join or os.sep
(4) Minor cleanup of imports & whitespace in test_issue_*.py
There are still 2 tests that should be adapted to Windows, but those are
lower-prio in my eyes.
Also, unittest run on my windows VM creates many resource warnings due to
non-closed file descriptors. Should address that some (future) day.
Do not assume a known length of output when checking it. Only assume that meta information about oletool used is first. Ignore warnings and messages that come after that to identify the actual result we want to check.
Encountered an example where a 3rd-party library issued a warning that messed up the json output. Create test to reproduce this.
We set logging.captureWarnings to True and use our logging framework for python's builtin warnings logger. Warnings from that logger will have "type=warning" per default (as opposed to "type=msg").
Warning message includes source file and line number, hope this is not dependent on python version..
Need to make sure that json output is formatted since e.g. warning messages
are created like this:
log.warning('%s', actual_message)
so without proper formatting the message is just "%s".
Also test that exception info is added as usual.
Use a standard logging.Formatter to do %-formatting and other stuff (i.e. the regular formatter job) before converting to Json.
If there is an error in log-handling (e.g. wrong args to logger call) do not fail or raise but produce something helpful
Availability of python versions in cached runner images is subject to change and specification in https://raw.githubusercontent.com/actions/python-versions/main/versions-manifest.json did not help
Pylint is just wrong in a few cases (admittedly: those are hard), so can
safely ignore these warnings.
Also ignore errors in thirdparty, this is not really our code.
Also remove a few old "pylint silencers" ("# pylint: disable=...") that
are not used any more in v3.10 of pylint
I do not see how I could keep the code clean for both for pylint2 and pylint3 at the same time, so do not even try.
Decrypting test samples "on the fly" using a generator causes trouble when removing the temp file, PyPy/Windows thinks the file is still being used.
christian-intra2net
force-pushed
the
unittest-automation
branch
from
77872a7
to
e9d2543
Compare
This was removed by github in June 2023
|
Github has removed the py2-runners, so do not request them any more. There are options to replace this with docker-images, but would need to find out how to adapt solutions mentioned in actions/setup-python#672 to our case. |
decalage2
approved these changes
Apr 17, 2024
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
People start relying on oletools, so we should provide a minimal amount of stability and reliability.
With this PR I propose to run the unittests for every commit to master and every pull request to master. The unittests will be started as github actions, they will run on various python versions on Windows, Mac and Linux, using CPython and PyPy.
Since we use msoffcrypt-tools, which uses cryptography, which produces warnings for python2, I could only get the unittests working by merging another branch of mine (extending the log-helper to capture warnings). Also, quite a few unittests needed fixing so they can run on windows.