WIP PyInstaller progress

[uellue]

Current status: libertem-server starts up, but doesn't find the static files where they are expected

Fixes #39

Contributor Checklist:

• I have added or updated my entry in the creators.json file
• I have added a changelog entry for my contribution
• I have added/updated documentation for all user-facing changes
• I have added/updated test cases
• I have included the rebuilt production build of the client (only if changes were made to the GUI)

Reviewer Checklist:

• /azp run libertem.libertem-data passed
• No import of GPL code from MIT code

[uellue]

The opentelemetry hook won't be required in the future and should be deleted, probably: fixed properly in pyinstaller/pyinstaller-hooks-contrib#725

Thx @rokm for the help and the quick fix!

[codecov]

Codecov Report

All modified and coverable lines are covered by tests ✅

Project coverage is 65.76%. Comparing base (92f14c3) to head (e20a249).

Additional details and impacted files

@@            Coverage Diff             @@
##           master    #1628      +/-   ##
==========================================
- Coverage   69.85%   65.76%   -4.09%     
==========================================
  Files         325      325              
  Lines       27251    27251              
  Branches     3136     3136              
==========================================
- Hits        19036    17922    -1114     
- Misses       7646     8823    +1177     
+ Partials      569      506      -63     

☔ View full report in Codecov by Sentry.
📢 Have feedback on the report? Share it here.

[rokm]

This should be unnecessary, as existing hook for dask should collect that file.

[rokm]

This could be handled in more portable way by creating a hook for distributed that calls collect_data_files('distributed').

[rokm]

And to avoid a future issue report - if your code uses a package that in turn uses multiprocessing (I think dask.distributed / distributed fits the bill), make sure to call multiprocessing.freeze_support() at the start of your program. Otherwise, use of multiprocessing will likely result in endless program spawn loop. See https://pyinstaller.org/en/stable/common-issues-and-pitfalls.html#multi-processing

[sk1p]

Current status: libertem-server starts up, but doesn't find the static files where they are expected

Do they end up in the site-packages in the installation? Is the relative path correct (whats the working directory when this code runs?), or does it maybe need a os.path.join(os.path.dirname(__file__), '..', '..', 'client') or similar?

[rokm]

If relative paths are given like that, PyInstaller assumes they are relative to the spec file (in order to factor out current working directory during the build).

I think part of the problem here is that the contents of that client directory are collected into libertem directory, as instructed. The target directory should be libertem\web\client, as it is when the package is installed.

But if the build process here assumes that libertem is installed (in non-editable mode) in the python environment prior to the build (which I suspect is the case, otherwise PyInstaller will have no way of finding libertem modules...), then the data files could be simply collected by hook-libertem.py:

from PyInstaller.utils.hooks import collect_data_files
datas = collect_data_files('libertem')

The second part of the problem seems to be that in frozen application, tornado.template.Loader receives relative root_directory (libertem\web), which it incorrectly resolves to absolute path using current working directory. When running unfrozen, that root_directory is already a correct absolute path. Will need to dig around a bit more to see what exactly is going wrong...

[rokm]

The second part of the problem seems to be that in frozen application, tornado.template.Loader receives relative root_directory (libertem\web), which it incorrectly resolves to absolute path using current working directory. When running unfrozen, that root_directory is already a correct absolute path. Will need to dig around a bit more to see what exactly is going wrong...

Ah yes, tornado is trying to infer the path from co_filename of code objects, which are anonymized relative paths in PyInstaller builds.

I suppose the easiest way for you to work around this insanity is to set template_path in the application settings, similarly to how it is done for static_path. In this case, the value should be os.path.dirname(__file__).

[uellue]

@rokm thank you for the pointers, much appreciated!