Python to deb conversion fails

[me21]

I tried to convert Python package to deb but the process crashed. Here's the output:

debian@evolve-azarubkin:~/evolve_numpy$ fpm
Loading flags from rc file .fpm {:level=>:warn}
Additional options: -f -a armhf --python-bin python3 --verbose -s python -t deb -n evolve_numpy -v 1.21.4 {:level=>:warn}
Additional arguments: numpy {:level=>:warn}
Setting workdir {:workdir=>"/tmp", :level=>:info}
Trying to download {:package=>"numpy", :level=>:info}
Looking in indexes: https://pypi.python.org/simple, https://www.piwheels.org/simple {:level=>:info}
Collecting numpy==1.21.4 {:level=>:info}
  Using cached https://files.pythonhosted.org/packages/fb/48/b0708ebd7718a8933f0d3937513ef8ef2f4f04529f1f66ca86d873043921/numpy-1.21.4.zip {:level=>:info}
  Saved /tmp/package-python-build-6750e16279e2fd042de8fe1cf7aafaec4cf39922736acf6dfc52e8d2e2a9/numpy-1.21.4.zip {:level=>:info}
  Installing build dependencies: started {:level=>:info}
  Installing build dependencies: still running... {:level=>:info}
  Installing build dependencies: still running... {:level=>:info}
  Installing build dependencies: still running... {:level=>:info}
  Installing build dependencies: still running... {:level=>:info}
  Installing build dependencies: still running... {:level=>:info}
  Installing build dependencies: still running... {:level=>:info}
  Installing build dependencies: still running... {:level=>:info}
  Installing build dependencies: still running... {:level=>:info}
  Installing build dependencies: still running... {:level=>:info}
  Installing build dependencies: still running... {:level=>:info}
  Installing build dependencies: still running... {:level=>:info}
  Installing build dependencies: still running... {:level=>:info}
  Installing build dependencies: still running... {:level=>:info}
  Installing build dependencies: finished with status 'done' {:level=>:info}
Successfully downloaded numpy {:level=>:info}
Traceback (most recent call last):
        10: from /usr/local/bin/fpm:23:in `<main>'
         9: from /usr/local/bin/fpm:23:in `load'
         8: from /var/lib/gems/2.5.0/gems/fpm-1.15.1/bin/fpm:7:in `<top (required)>'
         7: from /var/lib/gems/2.5.0/gems/clamp-1.0.1/lib/clamp/command.rb:133:in `run'
         6: from /var/lib/gems/2.5.0/gems/fpm-1.15.1/lib/fpm/command.rb:591:in `run'
         5: from /var/lib/gems/2.5.0/gems/clamp-1.0.1/lib/clamp/command.rb:68:in `run'
         4: from /var/lib/gems/2.5.0/gems/fpm-1.15.1/lib/fpm/command.rb:367:in `execute'
         3: from /var/lib/gems/2.5.0/gems/fpm-1.15.1/lib/fpm/command.rb:367:in `each'
         2: from /var/lib/gems/2.5.0/gems/fpm-1.15.1/lib/fpm/command.rb:368:in `block in execute'
         1: from /var/lib/gems/2.5.0/gems/fpm-1.15.1/lib/fpm/package/python.rb:98:in `input'
/var/lib/gems/2.5.0/gems/fpm-1.15.1/lib/fpm/package/python.rb:177:in `download_if_necessary': Unexpected directory layout after `pip download ...`. This might be an fpm bug? The directory is /tmp/package-python-build-6750e16279e2fd042de8fe1cf7aafaec4cf39922736acf6dfc52e8d2e2a9 (RuntimeError)

Here's my .fpm file:

-f
-a armhf
--python-bin python3
--verbose
-s python
-t deb
-n evolve_numpy
-v 1.21.4

Also, the mentioned directory inside /tmp doesn't exist after the program is terminated.

[me21]

The thing is a zip file is downloaded while the code expected .tar.gz.

[me21]

After applying a fix in python.rb similar to bugfood@839d542, the archive extracts and fpm fails further:

fetching package metadata {:setup_cmd=>"env PYTHONPATH=/var/lib/gems/2.5.0/gems/fpm-1.15.1/lib/fpm/package:$PYTHONPATH python3 setup.py --command-packages=pyfpm get_metadata --output=/tmp/package-python-build-3ed2fb11ec8950b8ffe2244d76bc3ba2328fa1b6f9d5e68c4c73c2257c06/metadata.json", :level=>:info}
Running from numpy source directory. {:level=>:info}
Traceback (most recent call last): {:level=>:info}
  File "setup.py", line 448, in <module> {:level=>:info}
    setup_package() {:level=>:info}
  File "setup.py", line 421, in setup_package {:level=>:info}
    run_build = parse_setuppy_commands() {:level=>:info}
  File "setup.py", line 357, in parse_setuppy_commands {:level=>:info}
    raise RuntimeError("Unrecognized setuptools command: {}".format(args)) {:level=>:info}
RuntimeError: Unrecognized setuptools command: ['--command-packages=pyfpm', 'get_metadata', '--output=/tmp/package-python-build-3ed2fb11ec8950b8ffe2244d76bc3ba2328fa1b6f9d5e68c4c73c2257c06/metadata.json'] {:level=>:info}
Process failed: /bin/bash failed (exit code 1). Full command was:["/bin/bash", "-c", "env PYTHONPATH=/var/lib/gems/2.5.0/gems/fpm-1.15.1/lib/fpm/package:$PYTHONPATH python3 setup.py --command-packages=pyfpm get_metadata --output=/tmp/package-python-build-3ed2fb11ec8950b8ffe2244d76bc3ba2328fa1b6f9d5e68c4c73c2257c06/metadata.json"] {:level=>:error}

[me21]

Okay, this is because numpy's setup.py checks for the commands passed to it and doesn't recognize get_metadata and fails. This could be overridden by adding --force argument to that call (

fpm/lib/fpm/package/python.rb

Line 234 in b085edc

"setup.py --command-packages=pyfpm get_metadata --output=#{tmp}"

), but I don't know if it's going to influence other packages conversion. It has a side effect of setting run_build variable in numpy's setup.py to True.

Probably another command line argument can be created like --python-get-metadata-extra-params

Nevertheless, the build fails later again, but this is an issue with numpy: numpy/numpy#21038

during RTL pass: expand {:level=>:info}
numpy/core/src/umath/loops_trigonometric.dispatch.c.src: In function ‘FLOAT_sin_NEON_VFPV4’: {:level=>:info}
numpy/core/src/umath/loops_trigonometric.dispatch.c.src:202:20: internal compiler error: in convert_move, at expr.c:218 {:level=>:info}
 NPY_NO_EXPORT void NPY_CPU_DISPATCH_CURFX(FLOAT_@func@) {:level=>:info}
 {:level=>:info}
                    ^ {:level=>:info}
########### EXT COMPILER OPTIMIZATION ########### {:level=>:info}
Please submit a full bug report, {:level=>:info}

I don't know how to use the workaround they suggest in fpm.

Why does it build numpy==1.21.4 in the first place when .whl is available for that version?..