BUG: numpy 1.26.4 does not find MKL at runtime

[AgilentGCMS]

Describe the issue:

I recently installed numpy 1.26.4 from source on a machine that has intel oneapi compilers (icx/icpx) and MKL libraries. I installed with

export PKG_CONFIG_PATH=/ford1/local/intel/oneapi/mkl/2022.1.0/lib/pkgconfig:$PKG_CONFIG_PATH # this is where my .pc files are
CC=icx CXX=icpx spin build -- -Dallow-noblas=false -Dblas-order=mkl,openblas -Dlapack-order=mkl,openblas -Dblas=mkl-dynamic-lp64-iomp -Dlapack=mkl-dynamic-lp64-iomp

and during compilation, it told me that it had found the correct shared library,

Test features "SSE SSE2 SSE3" : Supported
Test features "SSSE3" : Supported
Test features "SSE41" : Supported
Test features "POPCNT" : Supported
Test features "SSE42" : Supported
Test features "AVX" : Supported
Test features "F16C" : Supported
Test features "FMA3" : Supported
Test features "AVX2" : Supported
Test features "AVX512F" : Supported
Test features "AVX512CD" : Supported
Test features "AVX512_KNL" : Supported
Test features "AVX512_KNM" : Unsupported due to Arguments "-msse, -msse2, -msse3, -mssse3, -msse4.1, -mpopcnt, -msse4.2, -mavx, -mf16c, -mfma, -mavx2, -mno-mmx, -mavx512f, -mavx512cd, -mavx512er, -mavx512pf, -mavx5124fmaps, -mavx5124vnniw, -mavx512vpopcntdq" are not supported
Test features "AVX512_SKX" : Supported
Test features "AVX512_CLX" : Supported
Test features "AVX512_CNL" : Supported
Test features "AVX512_ICL" : Supported
Test features "AVX512_SPR" : disabled due to AVX512_SPR is disabled due to "intel-llvm compiler does not support it"
Configuring npy_cpu_dispatch_config.h using configuration
Message:
CPU Optimization Options
  baseline:
    Requested : min
    Enabled   : SSE SSE2 SSE3
  dispatch:
    Requested : max -xop -fma4
    Enabled   : SSSE3 SSE41 POPCNT SSE42 AVX F16C FMA3 AVX2 AVX512F AVX512CD AVX512_KNL AVX512_SKX AVX512_CLX AVX512_CNL AVX512_ICL

Library m found: YES
Run-time dependency mkl-dynamic-lp64-iomp found: YES 2022.1
Message: BLAS symbol suffix:

And yet, at runtime it seems to be using openblas and not MKL.

Reproduce the code example:

import numpy
numpy.show_runtime()

[{'numpy_version': '1.26.4',
  'python': '3.11.9 (main, Apr  4 2024, 11:00:55) [Clang 14.0.0 (icx '
            '2022.1.0.20220316)]',
  'uname': uname_result(system='Linux', node='gs6101-trident.ndc.nasa.gov', release='4.18.0-513.24.1.el8_9.x86_64', version='#1 SMP Thu Mar 14 14:20:09 EDT 2024', machine='x86_64')},
 {'simd_extensions': {'baseline': ['SSE', 'SSE2', 'SSE3'],
                      'found': ['SSSE3',
                                'SSE41',
                                'POPCNT',
                                'SSE42',
                                'AVX',
                                'F16C',
                                'FMA3',
                                'AVX2',
                                'AVX512F',
                                'AVX512CD',
                                'AVX512_SKX'],
                      'not_found': ['AVX512_KNL',
                                    'AVX512_KNM',
                                    'AVX512_CLX',
                                    'AVX512_CNL',
                                    'AVX512_ICL']}},
 {'architecture': 'SkylakeX',
  'filepath': '/home/sbasu1/packages/lib/python3.11/site-packages/numpy.libs/libopenblas64_p-r0-0cf96a72.3.23.dev.so',
  'internal_api': 'openblas',
  'num_threads': 8,
  'prefix': 'libopenblas',
  'threading_layer': 'pthreads',
  'user_api': 'blas',
  'version': '0.3.23.dev'}]

Error message:

No response

Python and NumPy Versions:

$ python -c 'import sys, numpy; print(numpy.__version__); print(sys.version)'
1.26.4
3.11.9 (main, Apr  4 2024, 11:00:55) [Clang 14.0.0 (icx 2022.1.0.20220316)]

Runtime Environment:

[{'numpy_version': '1.26.4',
'python': '3.11.9 (main, Apr 4 2024, 11:00:55) [Clang 14.0.0 (icx '
'2022.1.0.20220316)]',
'uname': uname_result(system='Linux', node='gs6101-trident.ndc.nasa.gov', release='4.18.0-513.24.1.el8_9.x86_64', version='#1 SMP Thu Mar 14 14:20:09 EDT 2024', machine='x86_64')},
{'simd_extensions': {'baseline': ['SSE', 'SSE2', 'SSE3'],
'found': ['SSSE3',
'SSE41',
'POPCNT',
'SSE42',
'AVX',
'F16C',
'FMA3',
'AVX2',
'AVX512F',
'AVX512CD',
'AVX512_SKX'],
'not_found': ['AVX512_KNL',
'AVX512_KNM',
'AVX512_CLX',
'AVX512_CNL',
'AVX512_ICL']}},
{'architecture': 'SkylakeX',
'filepath': '/home/sbasu1/packages/lib/python3.11/site-packages/numpy.libs/libopenblas64_p-r0-0cf96a72.3.23.dev.so',
'internal_api': 'openblas',
'num_threads': 8,
'prefix': 'libopenblas',
'threading_layer': 'pthreads',
'user_api': 'blas',
'version': '0.3.23.dev'}]

Context for the issue:

No response

[AgilentGCMS]

I've explicitly tried to exclude openblas by excluding it from lapack-order and blas-order as follows

-Dblas-order=mkl -Dlapack-order=mkl

but it still builds and links against openblas.

[AgilentGCMS]

Update: I realized that I was testing a different build of numpy than I was building. However, for the life of me I can't figure out how to install into a specific folder with spin build. It always installs into build-install, and Google has failed me on this problem.

[r-devulap]

Not sure how to install into a specific directory, but you can use NUMPY_SITE env variable to pick that version of numpy:

export NUMPY_SITE=$(realpath build-install/usr/lib/python*/site-packages/)

[ngoldbaum]

I use pip to install a development build of numpy if I need to use it in a python environment that depends on numpy. Something like:

python -m pip install -v . --no-build-isolation -Cbuilddir=build -C'compile-args=-v' -C'setup-args=-Dbuildtype=debug'

This will build a new install of numpy in the build folder (same as spin, and spin can re-use this install) and then install it to site-packages as normal. You also don't need to pass the compile-args or setup-args, see the meson-python docs for more about passing arguments to meson.

spin is really only for working on numpy itself. If whatever you're testing only needs numpy, you can use e.g. spin python or spin ipython to directly use the version of numpy spin builds.

[AgilentGCMS]

@ngoldbaum Let's say I've already built numpy with spin build. Is there a way to install it to a prefix of my choice? I could use the NUMPY_SITE environment variable, but I'm trying to install for multiple people, so I want to avoid kludges like setting an environment variable for all users that point to my home directory.

[mattip]

If you are providing a package for multiple people, then you should be producing a wheel and publishing that wheel for them to install

[ngoldbaum]

You can pass arguments to meson from a pip invocation. So, to pass the MKL options, you'd do:

pip install python -m pip install -v . --no-build-isolation -C'setup-args=-Dblas-order=mkl' -C'setup-args=-Dlapack-order=mkl'

Also see https://meson-python.readthedocs.io/en/latest/how-to-guides/meson-args.html?highlight=setup-args#passing-arguments-to-meson.

[AgilentGCMS]

If you are providing a package for multiple people, then you should be producing a wheel and publishing that wheel for them to install

You have the wrong idea. This is a high performance computing system. I want multiple users on the same machine to use this from a single location, and I don't want that location to be my home directory where I build stuff. I certainly don't want them to install wheels individually. I've built hundreds of packages on *nix systems, and most allow specification of a "prefix" in which to install binaries, libraries, headers, documentation, etc. I should be able to do this for python packages as well, or there's something broken about how python thinks packages should be managed.

[rgommers]

And yet, at runtime it seems to be using openblas and not MKL.

I don't think that is possible. Once there is a found: YES in the build log, that is what is used - no other BLAS libraries are even searched for.

This part of the show_runtime output:

  'filepath': '/home/sbasu1/packages/lib/python3.11/site-packages/numpy.libs/libopenblas64_p-r0-0cf96a72.3.23.dev.so',

shows that the numpy you are importing is coming from a PyPI wheel, which has openblas64_... vendored. So the package you built against MKL probably works, but isn't installed or is shadowed by another numpy installed into python3.11/site-packages.

Run-time dependency mkl-dynamic-lp64-iomp found: YES 2022.1

It probably doesn't matter here, but if you still have problems after removing the other numpy, then this MKL version may be too old. MKL is a complete mess in how it's distributed, and broken in so many ways when trying to use anything but MKL SDL in older versions (SDL was the only thing that the numpy.distutils supported, via direct linking to mkl_rt.so) - so I suggest using at least the version we use in CI (2023.2, see

numpy/.github/workflows/linux_blas.yml

Line 278 in fba4fc1

mkl:

).

[rgommers]

I certainly don't want them to install wheels individually. I've built hundreds of packages on *nix systems, and most allow specification of a "prefix" in which to install binaries, libraries, headers, documentation, etc. I should be able to do this for python packages as well, or there's something broken about how python thinks packages should be managed.

This is perfectly fine, that is a supported and important use case. I think though that it has to be set up in a way that users cannot install other Python packages themselves with something like the --user flag of pip, because that has the problem that you can get two numpy versions in the same environment. Something like that could be happening here.

[rgommers]

shows that the numpy you are importing is coming from a PyPI wheel,

Ah you figured this out already. Then to go on to the actual use case:

• I'd ignore spin, it is only useful when trying to contribute to numpy. It should never be used for the kind of HPC system install you're trying to do.
• Same for editable installs - not useful unless you're trying to contribute to numpy itself.

For MKL: do you want to link to a central MKL install, either in say /usr/lib or in a custom location like /opt/intel/? If the latter, then please ensure that opt/intel/ is on the library search path.

If you want the numpy install to be self-contained and vendor MKL, then you have to run auditwheel to actually do that vendoring.

[AgilentGCMS]

@rgommers Thanks for commenting and explaining, and I apologize for my meandering bugfixing method. As you correctly surmised, I have since figured out that my initial complain of not finding MKL at runtime was because I was loading the wrong numpy, i.e., not the numpy that had successfully compiled against MKL. This in turn was because I was expecting spin build to install into python's "global" prefix, i.e., the folder tree where the python binary was located, and not to build-install inside the numpy source tree. Why do/did I use spin build? Because I find meson build frustrating when debugging build errors; it rarely tells me what actually went wrong, and it always points to a non-existent log file, like so:

A full log can be found at /tmp/.../build/meson-logs/meson-log.txt

I cannot tell you how many hours I've spent hunting down these supposedly helpful meson log files that don't exist. I've also found that pip install takes me down that same logfile-hunting journey when a build fails, probably because it uses meson build. That's why I use spin build, at least I can figure out from the output what went wrong. This is not my complain alone, a Google search of "missing meson log file" yields several discussion posts such as this.

Anyway, my current situation is

• I have MKL installed in a global location /opt/intel/...
• I have built python from source and installed it in a custom prefix /opt/local/.../python/3.11.6 which I want multiple users to be able to use. This python is built with Intel compilers.
• I don't have a mechanism to prevent users from installing packages into their own PYTHONPATH and loading (say) the wrong numpy and scipy, but it's understood that if they do that they're on their own. My responsibility is to provide a python+numpy+scipy that works system-wide and uses the MKL that's installed in /opt/intel.
• I have finally figured out how to build numpy to use the MKL on our system,

export PKG_CONFIG_PATH=/opt/intel/oneapi/mkl/2022.1.0/lib/pkgconfig:$PKG_CONFIG_PATH
CC=icx CXX=icpx spin build -- -Dallow-noblas=false -Dblas-order=mkl -Dlapack-order=mkl -Dblas=mkl-static-lp64-iomp -Dlapack=mkl-static-lp64-iomp

• My trouble now comes when I try to make this numpy build available to all users by putting it in /opt/local/.../python/3.11.6, because the above step only installs into build-install. Right now I rsync from build-install into the location I want. That can't be the recommended way.
• If someone call tell me what the recommended way is of building numpy as above and installing it into /opt/local, I will follow that. If it didn't include hours of frustration with meson build (see above), that would be an added bonus.

[rgommers]

I cannot tell you how many hours I've spent hunting down these supposedly helpful meson log files that don't exist. [...] This is not my complain alone, a Google search of "missing meson log file" yields several discussion posts such as this.

This is an annoying problem indeed, but it's easy to resolve: add -Cbuild-dir=build to your pip invocation, and you'll find the logs under ./build/. Because this is indeed a recurring problem for users, I just wrote more extensive docs for that: mesonbuild/meson-python#616

Right now I rsync from build-install into the location I want. That can't be the recommended way.

Indeed, it is not. I suggest to change your build command to (first line is unchanged):

export PKG_CONFIG_PATH=/opt/intel/oneapi/mkl/2022.1.0/lib/pkgconfig:$PKG_CONFIG_PATH
CC=icx CXX=icpx python -m pip -Cbuild-dir=build -Csetup-args=-Dallow-noblas=false -Cetup-args=-Dblas-order=mkl -Cetup-args=-Dlapack-order=mkl -Cetup-args=-Dblas=mkl-static-lp64-iomp -Csetup-args=-Dlapack=mkl-static-lp64-iomp

(I don't think the *-order args should be needed, but I kept it unchanged from your working variant to be sure)

That should be all. In case it builds fine but doesn't work at runtime because an MKL .so goes missing (I can't remember what the OneAPI activation scripts do exactly), then do export LDFLAGS=-Wl,-rpath=/opt/intel/path/to/libdir.

[AgilentGCMS]

@rgommers I tried your python -m pip command (after correcting the obvious -Cetup-args typos to -Csetup-args), but now I see something confusing.

CC=icx CXX=icpx python -m pip install -v . -Cbuild-dir=build -Csetup-args=-Dallow-noblas=false -Csetup-args=-Dblas-order=mkl -Csetup-args=-Dlapack-order=mkl -Csetup-args=-Dblas=mkl-dynamic-lp64-iomp -Csetup-args=-Dlapack=mkl-dynamic-lp64-iomp

This failed with the error

  Installing build dependencies ... done
  Running command Getting requirements to build wheel
  Getting requirements to build wheel ... done
  Running command pip subprocess to install backend dependencies
  Collecting ninja>=1.8.2
    Using cached ninja-1.11.1.1-py2.py3-none-manylinux1_x86_64.manylinux_2_5_x86_64.whl.metadata (5.3 kB)
  Collecting patchelf>=0.11.0
    Using cached patchelf-0.17.2.1-py2.py3-none-manylinux_2_5_x86_64.manylinux1_x86_64.musllinux_1_1_x86_64.whl.metadata (3.3 kB)
  Using cached ninja-1.11.1.1-py2.py3-none-manylinux1_x86_64.manylinux_2_5_x86_64.whl (307 kB)
  Using cached patchelf-0.17.2.1-py2.py3-none-manylinux_2_5_x86_64.manylinux1_x86_64.musllinux_1_1_x86_64.whl (425 kB)
  Installing collected packages: patchelf, ninja
  Successfully installed ninja-1.11.1.1 patchelf-0.17.2.1
  Installing backend dependencies ... done
  Running command Preparing metadata (pyproject.toml)

  ('\x1b[31m',)meson-python: error: Could not find ninja version 1.8.2 or newer.
  error: subprocess-exited-with-error

  × Preparing metadata (pyproject.toml) did not run successfully.
  │ exit code: 1
  ╰─> See above for output.

  note: This error originates from a subprocess, and is likely not a problem with pip.
  full command: /home/sbasu1/packages/bin/python /home/sbasu1/packages/lib/python3.11/site-packages/pip/_vendor/pyproject_hooks/_in_process/_in_process.py prepare_metadata_for_build_wheel /tmp/tmpmlzt6l82
  cwd: /home/sbasu1/Downloads/sources/numpy-1.26.4
  Preparing metadata (pyproject.toml) ... error
error: metadata-generation-failed

× Encountered error while generating package metadata.
╰─> See above for output.

note: This is an issue with the package mentioned above, not pip.
hint: See above for details.

However, the error about ninja version cannot be true, because

$ $ ninja --version
1.11.1.git.kitware.jobserver-1

On the other hand, spin build from the same source tree builds fine.

[AgilentGCMS]

(I don't think the *-order args should be needed, but I kept it unchanged from your working variant to be sure)

They're probably not needed, but I want things to fail if MKL is not found instead of silently using openblas.

[rgommers]

However, the error about ninja version cannot be true, because

That is very strange. I don't know what is going on there, I've never seen that before. The build log also shows ninja being downloaded right above where it fails. Some kind of pip bug maybe (not likely though). If spin build works, then disabling build isolation should definitely work as well - can you add --no-build-isolation to the pip invocation?

They're probably not needed, but I want things to fail if MKL is not found instead of silently using openblas.

It will. Using -Dblas says "use this exact blas" and will disable auto-detection (if not, that is a bug). I verified with:

$ pip install . -Csetup-args=-Dblas=nonsense -Csetup-args=-Dallow-noblas=false --no-build-isolation
...
      Run-time dependency nonsense found: NO (tried pkgconfig, framework and cmake)

      ../numpy/meson.build:185:4: ERROR: Problem encountered: No BLAS library detected! Install one, or use the `allow-noblas` build option (note, this may be up to 100x slower for some linear algebra operations).

[AgilentGCMS]

So, the --no-build-isolation flag seemed to do the trick. After ensuring that mkl-static-lp64-iomp.pc was in my PKG_CONFIG_PATH, I issued the following:

CC=icx CXX=icpx python -m pip install --no-build-isolation -v . -Cbuild-dir=build -Csetup-args=-Dallow-noblas=false -Csetup-args=-Dblas-order=mkl -Csetup-args=-Dlapack-order=mkl -Csetup-args=-Dblas=mkl-static-lp64-iomp -Csetup-args=-Dlapack=mkl-static-lp64-iomp

which successfully compiled numpy and installed it in the same folder tree/prefix when my python binary is located. Yay, thanks @rgommers! I'm OK closing this ticket unless you want otherwise.

[rgommers]

Great, thanks @AgilentGCMS. Yes, let's close it, since things work for you now.

If you have problems again with BLAS/MKL/Meson, please feel free to ping me directly on a new issue.