installing tensorflow 2.11.0 on windows using poetry fails to install tensorflow-intel dependency

[twerkmeister]

Click to expand!

Issue Type

Bug

Source

binary

Tensorflow Version

2.11.0

Custom Code

No

OS Platform and Distribution

Windows

Mobile device

No response

Python version

3.9

Bazel version

No response

GCC/Compiler version

No response

CUDA/cuDNN version

No response

GPU model and memory

No response

Current Behaviour?

After installing tensorflow 2.11.0 on windows using poetry, importing tensorflow fails with ModuleNotFoundError: No module named 'tensorflow'. You can see on the pypi page of 2.11 that all the windows wheels are empty (only 1.9kb) and so are the arm packages as reported here

Installing with pip works as it seems to pick up on a tensorflow-intel dependency from somewhere (which then also contains tensorflow). We saw something like that in the Metadata file of the tensorflow wheel ~ however that seems not to be enough to give poetry that info.

Standalone code to reproduce the issue

mkdir poetry-test
cd poetry-test
poetry init
poetry add tensorflow
python -c "import tensorflow"

-> ModuleNotFoundError: No module named 'tensorflow'

Relevant log output

No response

[mohantym]

Hi @twerkmeister !
Thanks for sharing your observation on installing Tensorflow 2.11 on Windows through poetry.

@SuryanarayanaY !
Could you look at this issue.

Thank you!

[mihaimaruseac]

CC @nitins17 @learning-to-play

[SuryanarayanaY]

@twerkmeister ,
Could you please check now and close the issue if resolved.

Thankyou!

[twerkmeister]

Hey @SuryanarayanaY, any idea how I can test this right now? Since this is about the wheel thats published on pypi, I would need to new wheel with the change to be published, right? So I'll probably have to wait until the next micro release ~

[mihaimaruseac]

You cannot test right now with the 2.11. You could try nightly or wait until 2.12

[google-ml-butler]

This issue has been automatically marked as stale because it has no recent activity. It will be closed if no further activity occurs. Thank you.

[twerkmeister]

not stale - will test with nightly -

[twerkmeister]

Can confirm that, for example, in the nightly version 2.12.0.dev20221212 tensorflow-nightly-intel is picked up as an additional requirement. However, anything after 20121212 failed bc it resolved to requiring tf-nightly-cpu-aws:

$ poetry add tf-nightly==2.12.0.dev20221213
Because tf-nightly (2.12.0.dev20221213) depends on tf-nightly-cpu-aws (2.12.0-dev20221213) which doesn't match any versions, tf-nightly is forbidden.
So, because poetry-install-tf depends on tf-nightly (2.12.0.dev20221213), version solving failed.

[sachinprasadhs]

This behavior could be due to the recent collaborator build process for windows by Intel from Tensorflow 2.11 release.
tensorflow-intel/tf-nightly-intel dependency is built when user is on windows machine which installs the TensorFlow package built by Intel.

Similarly, 'tensorflow-cpu-aws'/'tf-nightly-cpu-aws' is built when TensorFlow package built by AWS if the user is running Linux on an Aarch64 machine.

we will look into the dependency issues. Thanks!

[eyadmba]

I'm seeing the same issue. Same environment as OP.

I worked around the issue by just installing tensorflow using pip, and the only thing it did actually install was tensorflow_intel-2.11.0-cp39-cp39-win_amd64.whl

[mihaimaruseac]

That is the case from TF 2.11 onwards: Windows builds are being done by Intel, so the actual code comes from tensorflow_intel wheels.

The above applies for CPU builds. GPU builds should be the Linux ones, via WSL-2.

See https://blog.tensorflow.org/2022/09/announcing-tensorflow-official-build-collaborators.html

[Mazyod]

Another issue that arises from metadata mismatch between different wheels based on platform is described in the issue referenced below.

A TL;DR version is that poetry also relies on wheels having the same metadata even for wheels that don’t match the current system/arch when using a legacy repository (e.g. Nexus, artifactory, devpi, pypiserver…). It simply downloads the first wheel and expects it to have the same metadata.

Would very much appreciate it if the fix was applied the other way around, too, if that makes sense for this project. (i.e. full dependencies listed for win and other variations).

As it’s not part of the standard, it may not make sense to make such a change. I can’t judge either way.

python-poetry/poetry#4000 (comment)

[dimbleby]

As it’s not part of the standard

just dropping by to say that I think this is a misunderstanding of something I wrote in that other issue. PEP508 environment markers are very much part of the standard and are encouraged as the way to specify platform-dependent requirements

ie requirements like

foo>=1 ; sys_platform == "win32"
bar>=1 ; sys_platform == "darwin"

are nowadays preferred over putting differing requirements in the windows / macos distributions.

[Dyskal]

hi, by the #58770 pull request, this should be fixed, but doesn't the if condition only work for linux environments ?

[mbbyn]

hi, by the #58770 pull request, this should be fixed, but doesn't the if condition only work for linux environments ?

Since poetry doesn't use the inferred environment to download the wheel for metadata collection, it could download the linux wheel even in Windows, if the linux wheel is the default.

There is a small poetry plugin that patches poetry to download the wheel for the inferred environment, and it solved our tensorflow issue. It is an extremely hacky plugin that is prone to break ... "It is what it is".

https://github.com/mazyod/poetry-legacy-index

[sachinprasadhs]

@twerkmeister , Could you please consider the above comment and let us know if that helps you. Thank you.

[github-actions]

This issue is stale because it has been open for 7 days with no activity. It will be closed if no further activity occurs. Thank you.

[github-actions]

This issue was closed because it has been inactive for 7 days since being marked as stale. Please reopen if you'd like to work on this further.

[google-ml-butler]

Are you satisfied with the resolution of your issue?
Yes
No

[RoelantStegmann]

Still had this issue, also with tensorflow-cpu.

[sachinprasadhs]

@mraunak , Could you please take a look into this. Thanks!

[mraunak]

Hi @RoelantStegmann, could you please share the error you are facing while installing tensorflow-cpu using poetry? Is it Linux wheels getting installed along with Window's wheel

[RoelantStegmann]

I get the same error with:

mkdir poetry-test
cd poetry-test

conda remove --name poetry-test --all -y
conda create --name poetry-test python=3.10 -y
conda activate poetry-test
pip install poetry

poetry init  # this requires manual input 

poetry config virtualenvs.create false
poetry add tensorflow-io-gcs-filesystem==0.31.0  # this one doesn't install, so I have to add it manually
poetry add tensorflow-cpu==2.12.0 
python -c "import tensorflow as tf"  # fails

How to verify that it's because of the wheel confusion? :) It seems related to tensorflow/io#1617

[mraunak]

Hi @RoelantStegmann, please follow the steps given below and let me know if you face any issue further
Steps to install TensorFlow through poetry.docx

[evansdoe]

@dimbleby try:

poetry add tensorflow-intel