Strange report for integration tests

[aaronzhangSema4]

Hi, I am a new user of pybuilder. I encountered a strange behavior for integration test reports. My integration test module is test_integration.py. After running "pyb run_integration_tests", I see "BUILD FAILED"

[INFO]  Running integration test test_integration
[INFO]  Running integration test test_integration.cpython-37
[ERROR] Integration test failed: ***/src/integrationtest/python/__pycache__/test_integration.cpython-37.pyc
------------------------------------------------------------
BUILD FAILED - KeyError: 'exception' (site-packages/pybuilder/plugins/python/test_plugin_helper.py:62)
------------------------------------------------------------
Build finished at 2020-08-05 14:17:45
Build took 37 seconds (37973 ms)

However, when I check target/reports/integrationtests/test_integration.err, there is no error:

......................
----------------------------------------------------------------------
Ran 22 tests in 6.273s

OK

In target/reports/integrationtests/test_integration.cpython-37.err, it did report an error:

Traceback (most recent call last):
  File "/**/src/integrationtest/python/test_integration.py", line 9, in <module>
    print(resource_dir)
FileNotFoundError: [Errno 2] No such file or directory: '/**/python/resource/required_tables.csv'

That missing file is actually found by python at the correct address '/**/resource/required_tables.csv' (no '/python').

Not sure what is the cause.

[aaronzhangSema4]

I removed that file opening line, but it still reports the FileNotFoundError

[arcivanov]

Thanks for the report. Could you post the full build log with -X -v?

[arcivanov]

@aaronzhangSema4 bump?

[arcivanov]

/**/python/resource/required_tables.csv'
The leading slash concerns me.

[aaronzhangSema4]

[DEBUG] Registered Python environment 'system': CPython version 3.7.4.final.0 on darwin in ['/Users/xzhang/.local/share/virtualenvs/js-magnitud-C2QM3xGh/bin/python3']
PyBuilder version 0.12.7
Build started at 2020-08-05 17:06:20
------------------------------------------------------------
[DEBUG] Loading project module from ./build.py
[DEBUG] Setting up plugins VEnv at '/Users/xzhang/git/js-magnitud/.pybuilder/plugins/cpython-3.7.4.final.0'
[DEBUG] Registered Python environment 'pybuilder': CPython version 3.7.4.final.0 on darwin in ['/Users/xzhang/git/js-magnitud/.pybuilder/plugins/cpython-3.7.4.final.0/bin/python3']
[DEBUG] Patched __import__ system to defer plugin loading
[DEBUG] Loading plugin 'python.core'
[DEBUG] Trying to load direct plugin 'python.core', module 'pybuilder.plugins.python.core_plugin'
[DEBUG] Found direct plugin 'python.core', module 'pybuilder.plugins.python.core_plugin'
[DEBUG] Loading plugin 'copy_resources'
[DEBUG] Trying to load direct plugin 'copy_resources', module 'pybuilder.plugins.copy_resources_plugin'
[DEBUG] Found direct plugin 'copy_resources', module 'pybuilder.plugins.copy_resources_plugin'
[DEBUG] Loading plugin 'python.unittest'
[DEBUG] Trying to load direct plugin 'python.unittest', module 'pybuilder.plugins.python.unittest_plugin'
[DEBUG] Found direct plugin 'python.unittest', module 'pybuilder.plugins.python.unittest_plugin'
[DEBUG] Loading plugin 'python.integrationtest'
[DEBUG] Trying to load direct plugin 'python.integrationtest', module 'pybuilder.plugins.python.integrationtest_plugin'
[DEBUG] Found direct plugin 'python.integrationtest', module 'pybuilder.plugins.python.integrationtest_plugin'
[DEBUG] Loading plugin 'python.flake8'
[DEBUG] Trying to load direct plugin 'python.flake8', module 'pybuilder.plugins.python.flake8_plugin'
[DEBUG] Found direct plugin 'python.flake8', module 'pybuilder.plugins.python.flake8_plugin'
[DEBUG] Loading plugin 'python.coverage'
[DEBUG] Trying to load direct plugin 'python.coverage', module 'pybuilder.plugins.python.coverage_plugin'
[DEBUG] Found direct plugin 'python.coverage', module 'pybuilder.plugins.python.coverage_plugin'
[DEBUG] Loading plugin 'python.distutils'
[DEBUG] Trying to load direct plugin 'python.distutils', module 'pybuilder.plugins.python.distutils_plugin'
[DEBUG] Found direct plugin 'python.distutils', module 'pybuilder.plugins.python.distutils_plugin'
[DEBUG] Loading plugin 'core'
[DEBUG] Trying to load direct plugin 'core', module 'pybuilder.plugins.core_plugin'
[DEBUG] Found direct plugin 'core', module 'pybuilder.plugins.core_plugin'
[DEBUG] Loading plugin 'analysis'
[DEBUG] Trying to load direct plugin 'analysis', module 'pybuilder.plugins.analysis_plugin'
[DEBUG] Found direct plugin 'analysis', module 'pybuilder.plugins.analysis_plugin'
[DEBUG] Found task 'clean' with dependencies []
[DEBUG] Registering task 'clean'
[DEBUG] Found task 'compile_sources' with dependencies [prepare]
[DEBUG] Registering task 'compile_sources'
[DEBUG] Registering initializer 'init'
[DEBUG] Found task 'install' with dependencies [package, publish(optional)]
[DEBUG] Registering task 'install'
[DEBUG] Found task 'package' with dependencies [compile_sources, run_unit_tests(optional)]
[DEBUG] Registering task 'package'
[DEBUG] Found task 'prepare' with dependencies []
[DEBUG] Registering task 'prepare'
[DEBUG] Found task 'print_module_path' with dependencies []
[DEBUG] Registering task 'print_module_path'
[DEBUG] Found task 'print_scripts_path' with dependencies []
[DEBUG] Registering task 'print_scripts_path'
[DEBUG] Found task 'publish' with dependencies [package, verify(optional)]
[DEBUG] Registering task 'publish'
[DEBUG] Found task 'run_integration_tests' with dependencies [package]
[DEBUG] Registering task 'run_integration_tests'
[DEBUG] Found task 'run_unit_tests' with dependencies [compile_sources]
[DEBUG] Registering task 'run_unit_tests'
[DEBUG] Found task 'verify' with dependencies [run_integration_tests(optional)]
[DEBUG] Registering task 'verify'
[DEBUG] Found task 'prepare' with dependencies []
[DEBUG] Registering task 'prepare'
[DEBUG] Registering initializer 'init_python_directories'
[DEBUG] Found task 'package' with dependencies []
[DEBUG] Registering task 'package'
[DEBUG] Registering initializer 'init_copy_resources_plugin'
[DEBUG] Found task 'package' with dependencies []
[DEBUG] Registering task 'package'
[DEBUG] Found action coverage_init
[DEBUG] Registering action 'coverage_init'
[DEBUG] Registering initializer 'init_test_source_directory'
[DEBUG] Found task 'run_unit_tests' with dependencies []
[DEBUG] Registering task 'run_unit_tests'
[DEBUG] Found action coverage_init
[DEBUG] Registering action 'coverage_init'
[DEBUG] Registering initializer 'initialize_integrationtest_plugin'
[DEBUG] Found task 'run_integration_tests' with dependencies []
[DEBUG] Registering task 'run_integration_tests'
[DEBUG] Found task 'analyze' with dependencies [prepare]
[DEBUG] Registering task 'analyze'
[DEBUG] Found action assert_flake8_is_executable
[DEBUG] Registering action 'assert_flake8_is_executable'
[DEBUG] Registering initializer 'initialize_flake8_plugin'
[DEBUG] Found task 'analyze' with dependencies [run_unit_tests]
[DEBUG] Registering task 'analyze'
[DEBUG] Found task 'coverage' with dependencies [verify]
[DEBUG] Registering task 'coverage'
[DEBUG] Registering initializer 'init_coverage_properties'
[DEBUG] Found task 'prepare' with dependencies []
[DEBUG] Registering task 'prepare'
[DEBUG] Registering late dependency of task 'publish' on [coverage(optional)]
[DEBUG] Resolved late dependency of task 'publish' on [coverage(optional)]
[DEBUG] Found action build_binary_distribution
[DEBUG] Registering action 'build_binary_distribution'
[DEBUG] Registering initializer 'initialize_distutils_plugin'
[DEBUG] Found task 'install' with dependencies []
[DEBUG] Registering task 'install'
[DEBUG] Found action set_description
[DEBUG] Registering action 'set_description'
[DEBUG] Found task 'upload' with dependencies []
[DEBUG] Registering task 'upload'
[DEBUG] Found action write_manifest_file
[DEBUG] Registering action 'write_manifest_file'
[DEBUG] Found action write_setup_script
[DEBUG] Registering action 'write_setup_script'
[DEBUG] Have loaded plugins python.core, copy_resources, python.unittest, python.integrationtest, python.flake8, python.coverage, python.distutils, core, analysis
[DEBUG] Registering initializer 'set_properties'
[DEBUG] Adding 'prepare' as a dependency of task 'compile_sources'
[DEBUG] Adding 'package' as a dependency of task 'install'
[DEBUG] Adding 'publish(optional)' as a dependency of task 'install'
[DEBUG] Adding 'compile_sources' as a dependency of task 'package'
[DEBUG] Adding 'run_unit_tests(optional)' as a dependency of task 'package'
[DEBUG] Adding 'package' as a dependency of task 'publish'
[DEBUG] Adding 'verify(optional)' as a dependency of task 'publish'
[DEBUG] Adding 'coverage(optional)' as a dependency of task 'publish'
[DEBUG] Adding 'package' as a dependency of task 'run_integration_tests'
[DEBUG] Adding 'compile_sources' as a dependency of task 'run_unit_tests'
[DEBUG] Adding 'run_integration_tests(optional)' as a dependency of task 'verify'
[DEBUG] Adding 'prepare' as a dependency of task 'analyze'
[DEBUG] Adding 'run_unit_tests' as a dependency of task 'analyze'
[DEBUG] Adding 'verify' as a dependency of task 'coverage'
[DEBUG] Adding before action 'coverage_init' for task 'prepare'
[DEBUG] Adding before action 'coverage_init' for task 'prepare'
[DEBUG] Adding after action 'assert_flake8_is_executable' for task 'prepare'
[DEBUG] Adding before action 'build_binary_distribution' for task 'publish'
[DEBUG] Adding after action 'set_description' for task 'prepare'
[DEBUG] Adding after action 'write_manifest_file' for task 'package'
[DEBUG] Adding after action 'write_setup_script' for task 'package'
[DEBUG] Verbose output enabled.

[DEBUG] Executing initializer 'init' from 'pybuilder.plugins.core_plugin'
[DEBUG] Executing initializer 'init_python_directories' from 'pybuilder.plugins.python.core_plugin'
[DEBUG] Executing initializer 'init_copy_resources_plugin' from 'pybuilder.plugins.copy_resources_plugin'
[DEBUG] Executing initializer 'init_test_source_directory' from 'pybuilder.plugins.python.unittest_plugin'
[DEBUG] Executing initializer 'initialize_integrationtest_plugin' from 'pybuilder.plugins.python.integrationtest_plugin'
[DEBUG] Executing initializer 'initialize_flake8_plugin' from 'pybuilder.plugins.python.flake8_plugin'
[DEBUG] Executing initializer 'init_coverage_properties' from 'pybuilder.plugins.python.coverage_plugin'
[DEBUG] Executing initializer 'initialize_distutils_plugin' from 'pybuilder.plugins.python.distutils_plugin'
[DEBUG] Executing initializer 'set_properties' from 'build'
[DEBUG] Project properties: 
                       __coverage_config : None
                         __covered_tasks : None
               _coverage_config_prefixes : {}
                         _coverage_tasks : []
                                 basedir : /Users/xzhang/git/js-magnitud
                     copy_resources_glob : ['**/resource/**']
                   copy_resources_target : $dir_dist
     coverage_allow_non_imported_modules : None
  coverage_branch_partial_threshold_warn : 0
          coverage_branch_threshold_warn : 0
                    coverage_break_build : False
                    coverage_concurrency : ['thread']
                          coverage_debug : []
                     coverage_exceptions : []
                           coverage_fork : None
                           coverage_name : js-magnitude
                 coverage_reload_modules : None
                  coverage_reset_modules : False
                    coverage_source_path : $dir_source_main_python
                 coverage_threshold_warn : 70
                                dir_dist : $dir_target/dist/js-magnitude-1.0.dev0
                        dir_dist_scripts : scripts
                        dir_install_logs : $dir_logs/install_dependencies
                                dir_logs : $dir_target/logs
                             dir_reports : $dir_target/reports
       dir_source_integrationtest_python : src/integrationtest/python
                  dir_source_main_python : src/main/python
                 dir_source_main_scripts : src/main/scripts
              dir_source_unittest_python : src/unittest/python
                              dir_target : target
                   distutils_classifiers : ['Development Status :: 3 - Alpha', 'Programming Language :: Python']
               distutils_command_options : None
                      distutils_commands : ['sdist', 'bdist_wheel', 'bdist_wheel']
               distutils_console_scripts : None
         distutils_description_overwrite : False
                  distutils_entry_points : None
              distutils_fail_on_warnings : False
  distutils_issue8876_workaround_enabled : False
            distutils_readme_description : False
                   distutils_readme_file : README.md
           distutils_readme_file_convert : False
          distutils_readme_file_encoding : None
              distutils_readme_file_type : None
           distutils_readme_file_variant : None
                distutils_setup_keywords : None
             distutils_summary_overwrite : False
               distutils_upload_register : False
             distutils_upload_repository : None
         distutils_upload_repository_key : None
                   distutils_upload_sign : False
          distutils_upload_sign_identity : None
          distutils_upload_skip_existing : False
                distutils_use_setuptools : True
                      distutils_zip_safe : True
                      flake8_break_build : False
                 flake8_exclude_patterns : None
                    flake8_extend_ignore : None
                           flake8_ignore : None
                 flake8_include_patterns : None
                  flake8_include_scripts : False
             flake8_include_test_sources : False
                   flake8_max_complexity : None
                  flake8_max_line_length : 120
        install_dependencies_constraints : constraints_file
    install_dependencies_extra_index_url : None
          install_dependencies_index_url : None
install_dependencies_insecure_installation : []
       install_dependencies_trusted_host : None
            install_dependencies_upgrade : False
  integrationtest_additional_commandline : 
  integrationtest_additional_environment : {}
          integrationtest_always_verbose : False
            integrationtest_breaks_build : True
      integrationtest_cpu_scaling_factor : 4
               integrationtest_file_glob : test_*
             integrationtest_file_suffix : None
     integrationtest_inherit_environment : True
                integrationtest_parallel : False
              integrationtest_python_env : test
                             pip_verbose : 0
                           refresh_venvs : False
                            remote_debug : 0
                   unittest_breaks_build : True
                    unittest_file_suffix : None
                    unittest_module_glob : test_*
                     unittest_python_env : build
                         unittest_runner : (<function init_test_source_directory.<locals>.<lambda> at 0x10864d3b0>, '_make_result')
             unittest_test_method_prefix : None
                              venv_clean : False
                       venv_dependencies : {}
                              venv_names : ['build', 'test']
                                 verbose : True
[DEBUG] Execution plan is clean, prepare, compile_sources, run_unit_tests, analyze, package, run_integration_tests, verify, coverage, publish
[INFO]  Building js-magnitude version 1.0.dev0
[INFO]  Executing build in /Users/xzhang/git/js-magnitud
[INFO]  Going to execute tasks: clean, analyze, run_integration_tests, verify, publish
[DEBUG] Executing task 'clean'
[DEBUG] Executing subtask of 'clean' from 'pybuilder.plugins.core_plugin'
[INFO]  Removing target directory /Users/xzhang/git/js-magnitud/target
[DEBUG] Executing task 'prepare'
[DEBUG] Executing action 'coverage_init' from 'pybuilder.plugins.python.unittest_plugin' before task
[DEBUG] Executing action 'coverage_init' from 'pybuilder.plugins.python.integrationtest_plugin' before task
[DEBUG] Executing subtask of 'prepare' from 'pybuilder.plugins.core_plugin'
[DEBUG] Creating target directory /Users/xzhang/git/js-magnitud/target
[DEBUG] Creating reports directory /Users/xzhang/git/js-magnitud/target/reports
[DEBUG] Inspecting package coverage
[DEBUG] Package coverage is added to the list of installation constraints
[DEBUG] Package coverage has a non-exact version specifier and will be updated
[DEBUG] Inspecting package flake8
[DEBUG] Package flake8 is added to the list of installation constraints
[DEBUG] Package flake8 has a non-exact version specifier and will be updated
[DEBUG] Inspecting package pypandoc
[DEBUG] Package pypandoc is added to the list of installation constraints
[DEBUG] Package pypandoc has a non-exact version specifier and will be updated
[DEBUG] Inspecting package setuptools
[DEBUG] Package setuptools is added to the list of installation constraints
[DEBUG] Package setuptools has a non-exact version specifier and will be updated
[DEBUG] Inspecting package twine
[DEBUG] Package twine is added to the list of installation constraints
[DEBUG] Package twine has a non-exact version specifier and will be updated
[DEBUG] Inspecting package unittest-xml-reporting
[DEBUG] Package unittest-xml-reporting is added to the list of installation constraints
[DEBUG] Package unittest-xml-reporting has a non-exact version specifier and will be updated
[DEBUG] Inspecting package wheel
[DEBUG] Package wheel is added to the list of installation constraints
[DEBUG] Package wheel has a non-exact version specifier and will be updated
[INFO]  Processing plugin packages 'coverage~=5.2' to be installed with {'upgrade': True}
[INFO]  Processing plugin packages 'flake8~=3.7' to be installed with {'upgrade': True}
[INFO]  Processing plugin packages 'pypandoc~=1.4' to be installed with {'upgrade': True}
[INFO]  Processing plugin packages 'setuptools>=38.6.0' to be installed with {'upgrade': True}
[INFO]  Processing plugin packages 'twine>=1.15.0' to be installed with {'upgrade': True}
[INFO]  Processing plugin packages 'unittest-xml-reporting~=3.0.2' to be installed with {'upgrade': True}
[INFO]  Processing plugin packages 'wheel>=0.34.0' to be installed with {'upgrade': True}
[DEBUG] Executing command: '/Users/xzhang/git/js-magnitud/.pybuilder/plugins/cpython-3.7.4.final.0/bin/python3' '-m' 'pip.__main__' 'install' '--upgrade' '--upgrade-strategy' 'only-if-needed' 'coverage~=5.2' 'flake8~=3.7' 'pypandoc~=1.4' 'setuptools>=38.6.0' 'twine>=1.15.0' 'unittest-xml-reporting~=3.0.2' 'wheel>=0.34.0'
[DEBUG] Executing subtask of 'prepare' from 'pybuilder.plugins.python.core_plugin'
[DEBUG] Creating log directory '/Users/xzhang/git/js-magnitud/target/logs/install_dependencies'
[INFO]  Creating target 'build' VEnv in '/Users/xzhang/git/js-magnitud/target/venv/build/cpython-3.7.4.final.0'
[DEBUG] Registered Python environment 'build': CPython version 3.7.4.final.0 on darwin in ['/Users/xzhang/git/js-magnitud/target/venv/build/cpython-3.7.4.final.0/bin/python3']
[DEBUG] Inspecting package click
[DEBUG] Package click is not installed and will be installed
[DEBUG] Inspecting package pandas
[DEBUG] Package pandas is not installed and will be installed
[DEBUG] Inspecting package parameterized
[DEBUG] Package parameterized is not installed and will be installed
[DEBUG] Inspecting package psycopg2-binary
[DEBUG] Package psycopg2-binary is not installed and will be installed
[DEBUG] Inspecting package pybuilder
[DEBUG] Package pybuilder is not installed and will be installed
[INFO]  Processing dependency packages 'click' to be installed with {}
[INFO]  Processing dependency packages 'pandas' to be installed with {}
[INFO]  Processing dependency packages 'parameterized' to be installed with {}
[INFO]  Processing dependency packages 'psycopg2-binary' to be installed with {}
[INFO]  Processing dependency packages 'pybuilder' to be installed with {}
[DEBUG] Executing command: '/Users/xzhang/git/js-magnitud/target/venv/build/cpython-3.7.4.final.0/bin/python3' '-m' 'pip.__main__' 'install' '-c' '/Users/xzhang/git/js-magnitud/target/venv/build/cpython-3.7.4.final.0/constraints_file' 'click' 'pandas' 'parameterized' 'psycopg2-binary' 'pybuilder'
[INFO]  Creating target 'test' VEnv in '/Users/xzhang/git/js-magnitud/target/venv/test/cpython-3.7.4.final.0'
[DEBUG] Registered Python environment 'test': CPython version 3.7.4.final.0 on darwin in ['/Users/xzhang/git/js-magnitud/target/venv/test/cpython-3.7.4.final.0/bin/python3']
[DEBUG] Inspecting package click
[DEBUG] Package click is not installed and will be installed
[DEBUG] Inspecting package pandas
[DEBUG] Package pandas is not installed and will be installed
[DEBUG] Inspecting package parameterized
[DEBUG] Package parameterized is not installed and will be installed
[DEBUG] Inspecting package psycopg2-binary
[DEBUG] Package psycopg2-binary is not installed and will be installed
[DEBUG] Inspecting package pybuilder
[DEBUG] Package pybuilder is not installed and will be installed
[INFO]  Processing dependency packages 'click' to be installed with {}
[INFO]  Processing dependency packages 'pandas' to be installed with {}
[INFO]  Processing dependency packages 'parameterized' to be installed with {}
[INFO]  Processing dependency packages 'psycopg2-binary' to be installed with {}
[INFO]  Processing dependency packages 'pybuilder' to be installed with {}
[DEBUG] Executing command: '/Users/xzhang/git/js-magnitud/target/venv/test/cpython-3.7.4.final.0/bin/python3' '-m' 'pip.__main__' 'install' '-c' '/Users/xzhang/git/js-magnitud/target/venv/test/cpython-3.7.4.final.0/constraints_file' 'click' 'pandas' 'parameterized' 'psycopg2-binary' 'pybuilder'
[DEBUG] Executing subtask of 'prepare' from 'pybuilder.plugins.python.coverage_plugin'
[INFO]  Requested coverage for tasks: pybuilder.plugins.python.unittest_plugin:run_unit_tests, pybuilder.plugins.python.integrationtest_plugin:run_integration_tests
[DEBUG] Executing action 'assert_flake8_is_executable' from 'pybuilder.plugins.python.flake8_plugin' before task
[DEBUG] Checking if flake8 is executable.
[DEBUG] Verifying command: 'flake8' '--version'
[DEBUG] Executing action 'set_description' from 'pybuilder.plugins.python.distutils_plugin' before task
[DEBUG] Executing task 'compile_sources'
[DEBUG] Executing subtask of 'compile_sources' from 'pybuilder.plugins.core_plugin'
[DEBUG] Executing task 'run_unit_tests'
[DEBUG] Executing subtask of 'run_unit_tests' from 'pybuilder.plugins.core_plugin'
[DEBUG] Executing subtask of 'run_unit_tests' from 'pybuilder.plugins.python.unittest_plugin'
[INFO]  Running unit tests
[INFO]  Executing unit tests from Python modules in /Users/xzhang/git/js-magnitud/src/unittest/python
[DEBUG] Including files matching 'test_*'
[DEBUG] starting test_from_env (test_dbconn.TestDbConn)
[DEBUG] starting test_from_property_file (test_dbconn.TestDbConn)
[DEBUG] starting test_parse_pipeline_file (test_pipeline.TestPipelineLoader)
[DEBUG] starting test_parse_sql_collection_file (test_pipeline.TestPipelineLoader)
[DEBUG] starting test_sql_list_for_extraction (test_pipeline.TestPipelineLoader)
[INFO]  Executed 5 unit tests
[INFO]  All unit tests passed.
[DEBUG] Executing task 'analyze'
[DEBUG] Executing subtask of 'analyze' from 'pybuilder.plugins.python.flake8_plugin'
[INFO]  Executing flake8 on project sources.
[DEBUG] Executing command: 'flake8' '--max-line-length=120' 'src/main/python'
[WARN]  flake8: src/main/python/queryenginefactory.py:15:38: W292 no newline at end of file
[WARN]  flake8: src/main/python/sqlpipeline.py:7:121: E501 line too long (376 > 120 characters)
[WARN]  flake8: src/main/python/sqlpipeline.py:22:5: E303 too many blank lines (2)
[WARN]  flake8: src/main/python/sqlpipeline.py:68:89: W292 no newline at end of file
[WARN]  flake8: src/main/python/queryengine.py:26:121: E501 line too long (129 > 120 characters)
[WARN]  flake8: src/main/python/queryengine.py:58:121: E501 line too long (121 > 120 characters)
[WARN]  flake8: src/main/python/queryengine.py:68:27: W292 no newline at end of file
[WARN]  flake8: src/main/python/app.py:24:121: E501 line too long (153 > 120 characters)
[WARN]  flake8: src/main/python/app.py:74:121: E501 line too long (153 > 120 characters)
[WARN]  flake8: src/main/python/app.py:108:1: W391 blank line at end of file
[WARN]  flake8: src/main/python/dbconnfactory.py:36:77: E202 whitespace before ')'
[WARN]  flake8: src/main/python/jsqueryengine.py:12:121: E501 line too long (123 > 120 characters)
[WARN]  flake8: src/main/python/jsqueryengine.py:97:46: W292 no newline at end of file
[WARN]  flake8 found 13 warning(s).
[DEBUG] Executing subtask of 'analyze' from 'pybuilder.plugins.analysis_plugin'
[DEBUG] Executing task 'package'
[DEBUG] Executing subtask of 'package' from 'pybuilder.plugins.core_plugin'
[DEBUG] Executing subtask of 'package' from 'pybuilder.plugins.python.core_plugin'
[DEBUG] Creating directory /Users/xzhang/git/js-magnitud/target/dist/js-magnitude-1.0.dev0
[INFO]  Building distribution in /Users/xzhang/git/js-magnitud/target/dist/js-magnitude-1.0.dev0
[DEBUG] Copying module/ package queryenginefactory.py
[DEBUG] Copying module/ package sqlpipeline.py
[DEBUG] Copying module/ package __init__.py
[DEBUG] Copying module/ package __pycache__
[DEBUG] Copying module/ package queryengine.py
[DEBUG] Copying module/ package app.py
[DEBUG] Copying module/ package dbconnfactory.py
[DEBUG] Copying module/ package jsqueryengine.py
[INFO]  Copying scripts to /Users/xzhang/git/js-magnitud/target/dist/js-magnitude-1.0.dev0/scripts
[DEBUG] Executing subtask of 'package' from 'pybuilder.plugins.copy_resources_plugin'
[INFO]  Copying resources matching '**/resource/**' from /Users/xzhang/git/js-magnitud to /Users/xzhang/git/js-magnitud/target/dist/js-magnitude-1.0.dev0
[DEBUG] Copying resource src/unittest/resource/test_database.properties
[DEBUG] Copying resource src/unittest/resource/required_tables.csv
[DEBUG] Copying resource src/unittest/resource/queries.json
[DEBUG] Copying resource src/unittest/resource/pipeline1.json
[DEBUG] Copying resource src/integrationtest/resource/required_tables.csv
[DEBUG] Copying resource src/main/resource/sql/query_library.json
[DEBUG] Copying resource src/main/resource/sql/required_tables.csv
[DEBUG] Copying resource src/main/resource/sql/magnitude-pipeline-dev-withoutPSA.json
[DEBUG] Copying resource src/main/resource/sql/magnitude-pipeline-dev.json
[DEBUG] Executing action 'write_manifest_file' from 'pybuilder.plugins.python.distutils_plugin' before task
[DEBUG] No data to write into MANIFEST.in
[DEBUG] Executing action 'write_setup_script' from 'pybuilder.plugins.python.distutils_plugin' before task
[INFO]  Writing setup.py as /Users/xzhang/git/js-magnitud/target/dist/js-magnitude-1.0.dev0/setup.py
[DEBUG] Executing task 'run_integration_tests'
[DEBUG] Executing subtask of 'run_integration_tests' from 'pybuilder.plugins.core_plugin'
[DEBUG] Executing subtask of 'run_integration_tests' from 'pybuilder.plugins.python.integrationtest_plugin'
[DEBUG] Running integration tests sequentially
[INFO]  Running integration test test_integration
[DEBUG] Executing command: '/Users/xzhang/git/js-magnitud/target/venv/test/cpython-3.7.4.final.0/bin/python3' '/Users/xzhang/git/js-magnitud/src/integrationtest/python/test_integration.py'
[INFO]  Running integration test test_integration.cpython-37
[DEBUG] Executing command: '/Users/xzhang/git/js-magnitud/target/venv/test/cpython-3.7.4.final.0/bin/python3' '/Users/xzhang/git/js-magnitud/src/integrationtest/python/__pycache__/test_integration.cpython-37.pyc'
[ERROR] Integration test failed: /Users/xzhang/git/js-magnitud/src/integrationtest/python/__pycache__/test_integration.cpython-37.pyc
File /Users/xzhang/git/js-magnitud/target/reports/integrationtests/test_integration.cpython-37:

File /Users/xzhang/git/js-magnitud/target/reports/integrationtests/test_integration.cpython-37.err:
    Traceback (most recent call last):
      File "/Users/xzhang/git/js-magnitud/src/integrationtest/python/test_integration.py", line 3, in <module>
        from dbconnfactory import DbConnFactory
    ModuleNotFoundError: No module named 'src.main.python'
[INFO]  Executed 2 integration tests.
Traceback (most recent call last):
  File "/Users/xzhang/.local/share/virtualenvs/js-magnitud-C2QM3xGh/lib/python3.7/site-packages/pybuilder/cli.py", line 468, in main
    environments=options.environments, tasks=arguments)
  File "/Users/xzhang/.local/share/virtualenvs/js-magnitud-C2QM3xGh/lib/python3.7/site-packages/pybuilder/reactor.py", line 234, in build
    execution_summary = self.build_execution_plan(tasks, execution_plan)
  File "/Users/xzhang/.local/share/virtualenvs/js-magnitud-C2QM3xGh/lib/python3.7/site-packages/pybuilder/reactor.py", line 273, in build_execution_plan
    reactor=self)
  File "/Users/xzhang/.local/share/virtualenvs/js-magnitud-C2QM3xGh/lib/python3.7/site-packages/pybuilder/execution.py", line 399, in execute_execution_plan
    summaries.append(self.execute_task(task, **kwargs))
  File "/Users/xzhang/.local/share/virtualenvs/js-magnitud-C2QM3xGh/lib/python3.7/site-packages/pybuilder/execution.py", line 346, in execute_task
    task.execute(self.logger, kwargs, _executable=_executable)
  File "/Users/xzhang/.local/share/virtualenvs/js-magnitud-C2QM3xGh/lib/python3.7/site-packages/pybuilder/execution.py", line 185, in execute
    executable.execute(argument_dict)
  File "/Users/xzhang/.local/share/virtualenvs/js-magnitud-C2QM3xGh/lib/python3.7/site-packages/pybuilder/execution.py", line 101, in execute
    self.callable(*arguments)
  File "/Users/xzhang/.local/share/virtualenvs/js-magnitud-C2QM3xGh/lib/python3.7/site-packages/pybuilder/plugins/python/integrationtest_plugin.py", line 75, in run_integration_tests
    reports_processor.write_report_and_ensure_all_tests_passed()
  File "/Users/xzhang/.local/share/virtualenvs/js-magnitud-C2QM3xGh/lib/python3.7/site-packages/pybuilder/plugins/python/test_plugin_helper.py", line 54, in write_report_and_ensure_all_tests_passed
    raise BuildFailedException("%d of %d integration tests failed." % (self.tests_failed, self.tests_executed))
pybuilder.errors.BuildFailedException: 1 of 2 integration tests failed.
------------------------------------------------------------
BUILD FAILED - 1 of 2 integration tests failed. (site-packages/pybuilder/plugins/python/test_plugin_helper.py:54)
------------------------------------------------------------
Build finished at 2020-08-05 17:07:55
Build took 94 seconds (94169 ms)

[aaronzhangSema4]

Sorry for the delay. I posted the error messages above. I did a lot of testing. It seem the error always comes from .pyc, and from the use of the integration test plugin. I now moved everything into unittest (without any code changes), and everything runs properly.

[arcivanov]

@aaronzhangSema4 Thanks! Could you please post your build.py as well?

[arcivanov]

[DEBUG] Executing command: '/Users/xzhang/git/js-magnitud/target/venv/test/cpython-3.7.4.final.0/bin/python3' '/Users/xzhang/git/js-magnitud/src/integrationtest/python/pycache/test_integration.cpython-37.pyc'

PYC????

[aaronzhangSema4]

I have no idea what it is about... I was using pure python.

[arcivanov]

Could you post your build.py please?

[aaronzhangSema4]

I will try to reproduce the issue with a toy project in the coming days. Thanks.

[arcivanov]

Thanks!