import PIL.Image double quotes for Core version: "9.2.0" on win-64 for python 3.7 and python 3.8

[skupr-anaconda]

What did you do?

I'm building Pillow 9.2.0 from the source code and after import PIL.Image on win-64 for python 3.7 and 3.8 tests failed

What did you expect to happen?

import PIL.Image on win-64 for python 3.7 and python 3.8 succeeds

What actually happened?

TEST START: output-artifacts\win-64\pillow-9.2.0-py37hdc2b20a_1.tar.bz2
Renaming work directory 'C:\ci\pillow_1657789265718\work' to 'C:\ci\pillow_1657789265718\work_moved_pillow-9.2.0-py37hdc2b20a_1_win-64'
shutil.move(work)=C:\ci\pillow_1657789265718\work, dest=C:\ci\pillow_1657789265718\work_moved_pillow-9.2.0-py37hdc2b20a_1_win-64)
Collecting package metadata (repodata.json): ...working... done
Solving environment: ...working... done
DEBUG:urllib3.connectionpool:https://repo.anaconda.com:443 "GET /pkgs/main/win-64/libwebp-1.2.2-h2bbff1b_0.conda HTTP/1.1" 200 673648
DEBUG:urllib3.connectionpool:https://repo.anaconda.com:443 "GET /pkgs/main/win-64/lz4-c-1.9.3-h2bbff1b_1.conda HTTP/1.1" 200 134941
DEBUG:urllib3.connectionpool:https://repo.anaconda.com:443 "GET /pkgs/main/win-64/zstd-1.5.2-h19a0ad4_0.conda HTTP/1.1" 200 521566
DEBUG:urllib3.connectionpool:https://repo.anaconda.com:443 "GET /pkgs/main/win-64/libtiff-4.2.0-he0120a3_1.conda HTTP/1.1" 200 771675

## Package Plan ##

  environment location: C:\ci\pillow_1657789265718\_test_env


The following NEW packages will be INSTALLED:

    ca-certificates: 2022.4.26-haa95532_0                                                                             
    certifi:         2022.6.15-py37haa95532_0                                                                         
    freetype:        2.10.4-hd328e21_0                                                                                
    jpeg:            9e-h2bbff1b_0                                                                                    
    libpng:          1.6.37-h2a8f88b_0                                                                                
    libtiff:         4.2.0-he0120a3_1                                                                                 
    libwebp:         1.2.2-h2bbff1b_0                                                                                 
    lz4-c:           1.9.3-h2bbff1b_1                                                                                 
    openssl:         1.1.1q-h2bbff1b_0                                                                                
    pillow:          9.2.0-py37hdc2b20a_1     file:///C:/ci/containers/00000f7bgd7/tmp/build/80754af9/output-artifacts
    pip:             22.1.2-py37haa95532_0                                                                            
    python:          3.7.13-h6244533_0                                                                                
    setuptools:      61.2.0-py37haa95532_0                                                                            
    sqlite:          3.38.5-h2bbff1b_0                                                                                
    tk:              8.6.12-h2bbff1b_0                                                                                
    vc:              14.2-h21ff451_1                                                                                  
    vs2015_runtime:  14.27.29016-h5e58377_2                                                                           
    wheel:           0.37.1-pyhd3eb1b0_0                                                                              
    wincertstore:    0.2-py37haa95532_2                                                                               
    xz:              5.2.5-h8cc25b3_1                                                                                 
    zlib:            1.2.12-h8cc25b3_2                                                                                
    zstd:            1.5.2-h19a0ad4_0                                                                                 

Preparing transaction: ...working... done
Verifying transaction: ...working... done
Executing transaction: ...working... done
C:\ci\pillow_1657789265718\_test_env\lib\site-packages\PIL\Image.py:121: RuntimeWarning: The _imaging extension was built for another version of Pillow or PIL:
Core version: "9.2.0"
Pillow version: 9.2.0
  warnings.warn(str(v), RuntimeWarning)
Traceback (most recent call last):
  File "C:\ci\pillow_1657789265718\test_tmp\run_test.py", line 5, in <module>
    import PIL.Image
  File "C:\ci\pillow_1657789265718\_test_env\lib\site-packages\PIL\Image.py", line 104, in <module>
    "The _imaging extension was built for another version of Pillow or PIL:\n"
ImportError: The _imaging extension was built for another version of Pillow or PIL:
Core version: "9.2.0"
Pillow version: 9.2.0
Tests failed for pillow-9.2.0-py37hdc2b20a_1.tar.bz2 - moving package to C:\ci\broken
WARNING:conda_build.build:Tests failed for pillow-9.2.0-py37hdc2b20a_1.tar.bz2 - moving package to C:\ci\broken
set PREFIX=C:\ci\pillow_1657789265718\_test_env
set SRC_DIR=C:\ci\pillow_1657789265718\test_tmp

(base) %SRC_DIR%>call "%SRC_DIR%\conda_test_env_vars.bat" 

(base) %SRC_DIR%>set "CONDA_SHLVL="   && 

(base) %SRC_DIR%>conda activate "%PREFIX%" 

(%PREFIX%) %SRC_DIR%>IF 0 NEQ 0 exit /B 1 

(%PREFIX%) %SRC_DIR%>"%PREFIX%\python.exe" -s "%SRC_DIR%\run_test.py" 
import: 'PIL'
import: 'PIL.Image'

(%PREFIX%) %SRC_DIR%>IF 1 NEQ 0 exit /B 1 
TESTS FAILED: pillow-9.2.0-py37hdc2b20a_1.tar.bz2

What are your OS, Python and Pillow versions?

• OS: win-64 (build tool: vs2017_win-64: 19.16.27032.1-hb4161e2_3)
• Python: (CPython) 3.7 and 3.8
• Pillow: 9.2.0

I think the bug is here:

Pillow/setup.py

Lines 849 to 856 in e42d8b0

	if (
	sys.platform == "win32"
	and sys.version_info < (3, 9)
	and not (PLATFORM_PYPY or PLATFORM_MINGW)
	):
	defs.append(("PILLOW_VERSION", f'"\\"{PILLOW_VERSION}\\""'))
	else:
	defs.append(("PILLOW_VERSION", f'"{PILLOW_VERSION}"'))

Because if I use defs.append(("PILLOW_VERSION", f'"{PILLOW_VERSION}"')) without double quotes for python 3.7 and 3.8 on win-64 it succeeded

[hugovk]

This was first added in edc8d86 "Fix to ensure that define is interpreted as a string on windows " in PR #2517.

And then adjusted to only apply for Python < 3.9 in PR #4633.

If I make this change, the CI fails for Python 3.7 and 3.8:

• https://ci.appveyor.com/project/hugovk/pillow/builds/44169790
• https://github.com/hugovk/Pillow/actions/runs/2669338214

[radarhere]

You're installing Pillow through conda. Perhaps https://github.com/conda-forge/pillow-feedstock is a better place for this issue?

[skupr-anaconda]

If I make this change, the CI fails for Python 3.7 and 3.8:

• https://ci.appveyor.com/project/hugovk/pillow/builds/44169790
• https://github.com/hugovk/Pillow/actions/runs/2669338214

I try to check if it might be a conda-build or our CI system bug because the latest build for v9.0.1 was successful and import PIL.Image worked

[skupr-anaconda]

@hugovk The last version of pillow we built (v9.0.1, https://repo.anaconda.com/pkgs/main/win-64/pillow-9.0.1-py37hdc2b20a_0.tar.bz2) was built using conda-build-3.21.7,
Looking at the conda-build changelog for v3.21.9, it was noticed some changes to support the new setuptools >=60.0

I updated the recipe meta.yaml with:

    - setuptools <60  # [win and (py==37 or py==38)]
    - setuptools      # [not (win and (py==37 or py==38))]

And now it succeeded on win for py37-py38, see AnacondaRecipes/pillow-feedstock#10

[radarhere]

AnacondaRecipes/pillow-feedstock#10 has been merged.

So are you saying that this is actually a problem between conda-build and setuptools? While Pillow may be triggering the issue, that is where the real problem lies?

[skupr-anaconda]

I guess there are some major changes in setuptools >=60 that parses version strings differently than they were before.

[hugovk]

Yes, confirming a problem with setuptools 60 on Windows for CPython 3.7 and 3.8.

python3 -m pip install -U "setuptools<=59" (hugovk@0c17344):

setuptools 59

• https://github.com/hugovk/Pillow/actions/runs/2675896444
• All pass:
  • ✅ 3.7: setuptools upgraded 47.1.0 -> 59.8.0
  • ✅ 3.8: setuptools upgraded 56.0.0 -> 59.8.0
  • ✅ 3.9: setuptools upgraded 58.1.0 -> 59.8.0
  • ✅ 3.10: setuptools upgraded 58.1.0 -> 59.8.0
  • ✅ 3.11: setuptools upgraded 58.1.0 -> 59.8.0
  • ✅ PyPy3.7: setuptools upgraded 47.1.0 -> 59.8.0
  • ✅ PyPy3.8: setuptools upgraded 58.1.0 -> 59.8.0

setuptools 60

python3 -m pip install -U "setuptools<=60" (hugovk@f66b4f2):

• https://github.com/hugovk/Pillow/actions/runs/2675788756
• CPython 3.7 and 3.8 fail, CPython 3.9-3.11 and PyPy3.7-3.8 pass
  • ❌ 3.7: setuptools upgraded 47.1.0 -> 60.10.0
  • ❌ 3.8: setuptools upgraded 56.0.0 -> 60.10.0
  • ✅ 3.9: setuptools upgraded 58.1.0 -> 60.10.0
  • ✅ 3.10: setuptools upgraded 58.1.0 -> 60.10.0
  • ✅ 3.11: setuptools upgraded 58.1.0 -> 60.10.0
  • ✅ PyPy3.7: setuptools upgraded 47.1.0 -> 60.10.0
  • ✅ PyPy3.8: setuptools upgraded 58.1.0 -> 60.10.0

[radarhere]

I've created PR #6450 to resolve this.

[hugovk]

Thanks, looks good!

@skupr-anaconda please could you also test the patch?

[skupr-anaconda]

@hugovk Thanks, it resolves the issue, see AnacondaRecipes/pillow-feedstock#11

[hugovk]

Great, we'll merge it and it'll be in the next quarterly release in October. 🚀

[bartoli]

Just in case it happens again for another reason, shouldn't Image.py also be modified so that

if __version__ != getattr(core, "PILLOW_VERSION", None):

becomes

if __version__ != getattr(core, "PILLOW_VERSION", None).strip('"'):

?

[radarhere]

I can't think of what "another reason" would be, so I personally find that idea a bit too defensive. I doubt setuptools would change this behaviour again any time soon.

Considering that setuptools 60 was released in December, I'm actually surprised that it's taken this long for an issue to be raised about this. Perhaps the number of Windows users who upgrade their setuptools but don't upgrade to the latest version of Python is small.

And in two years and two months, Python 3.7 and 3.8 will both be EOL, so this problem will go away entirely.

[cgohlke]

The "fix" for this issue breaks the builds for Python 3.7 and 3.8 on my system with setuptools 65.5.0:

src/_imaging.c(4128): error C2440: 'type cast': cannot convert from 'double' to 'char *'
src/_imaging.c(4128): error C2143: syntax error: missing ';' before 'constant'
src/_imaging.c(4128): error C2224: left of '.dev0' must have struct/union type
error: command 'C:\\Program Files (x86)\\Microsoft Visual Studio\\2019\\Community\\VC\\Tools\\MSVC\\14.29.30133\\bin\\HostX86\\x64\\cl.exe' failed with exit status 2

Reverting #6450 works.

[hugovk]

Strange, I couldn't reproduce with setuptools 59-65, I wonder what the difference is.

@cgohlke Does #6437 (comment) fix it for you? Maybe we do need that "another reason" after all!

[cgohlke]

Found it: my build system has SETUPTOOLS_USE_DISTUTILS=stdlib set.

[hugovk]

Good find! That was introduced in setuptools 60 and this distutils vendoring thing must be the root of this problem.

https://setuptools.pypa.io/en/latest/deprecated/distutils-legacy.html

We'd better get a fix into today's release and add a check for this env var in the same place.

[hugovk]

Please see PR #6695. Some testing on Windows would be appreciated, and hopefully we can include a fix in the release. Thanks!

[radarhere]

And in two years and two months, Python 3.7 and 3.8 will both be EOL, so this problem will go away entirely.

Thanks to #7230, the double quotes have now been removed in Pillow 10.0.0.