Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Test: Default Alias in Base Class #4581

Draft
wants to merge 6 commits into
base: master
Choose a base branch
from

Conversation

ax3l
Copy link
Collaborator

@ax3l ax3l commented Mar 20, 2023

Description

This tests that template type aliases for base class templates are resolved to the same registered types.

This shows problems for me on Windows with both MSVC and Clang (clang-cl in CMake).

E       TypeError: Unable to convert function return value to a Python type! The signature was
E       	() -> S<DefaultAllocator>

X-ref: #4319 (comment)

Suggested changelog entry:

This tests that template type aliases for base class templates are
resolved to the same registered types.
@ax3l ax3l force-pushed the topic-template-alias-base branch from c27f62d to e009a87 Compare March 21, 2023 05:30
@ax3l ax3l requested a review from rwgk March 21, 2023 05:51
// this returns S<DefaultAllocator> even though we register
// the type S<std::allocator>
// MSVC and Clang on Windows failed here to detect the registered type
S<> make_S() { return S<>{}; }
Copy link
Collaborator

@rwgk rwgk Mar 21, 2023

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This is basically stress-testing the compiler+library: can it figure out that S<DefaultAllocator> and S<std::allocator> are the same?

Question 1: Is it difficult to not have that stress-test baked into your downstream project? Either, by consistently avoiding the alias, or consistently using the alias.

Question 2: What happens if you patch internals.h similar to #4319?

If you decide to experiment more here, could you please not force push? That makes it more difficult for me to follow.

Copy link
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This is basically stress-testing the compiler+library: can it figure out that S and Sstd::allocator are the same?

Correct.

Question 1: Is it difficult to not have that stress-test baked into your downstream project? Either, by consistently avoiding the alias, or consistently using the alias.

This is somewhat tricky in my case, especially across library/module interfaces. By consistently using one or the other, one has to hope a user does never use the other possible option of the alias.

Question 2: What happens if you patch internals.h similar to #4319?

Will try and push here.

If you decide to experiment more here, could you please not force push? That makes it more difficult for me to follow.

Will do! I just force-pushed initially yesterday when I realized the initial test draft was not yet complete/buggy.

Copy link
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Pushed patch :)

Copy link
Collaborator Author

@ax3l ax3l Mar 22, 2023

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@rwgk I don't understand your patch yet:
The comment says, GNU's libstdc++ is good (#if defined(__GLIBCXX__)) - all other implementations need the self-made hash.

Your patch says LLVM's libc++ is not good (#if !defined(_LIBCPP_VERSION)) - but all other implementations can use the standard hashes.

Copy link
Collaborator

@rwgk rwgk Mar 22, 2023

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Yes: the CI for #4319 that ran yesterday (all successful) supports "LLVM's libc++ is not good (#if !defined(_LIBCPP_VERSION)) - but all other implementations can use the standard hashes."

But it looks like your added test doesn't work universally any way you tried, is that a correct summary?

Copy link
Collaborator Author

@ax3l ax3l Mar 22, 2023

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Nice!

My test works on Linux & Windows with libstdc++.

ax3l and others added 3 commits March 21, 2023 17:07
@rwgk
Copy link
Collaborator

rwgk commented Mar 22, 2023

My test works on Linux & Windows with libstdc++.

Below are the CI results 1. Failing, 2. Successful. It looks like a Clangs are failing. Which are Linux & libstdc++?


CI / 🐍 3.6 • windows-2022 • x64 (pull_request) Failing after 6m
CI / 🐍 3.9 • windows-2022 • x64 (pull_request) Failing after 6m
CI / 🐍 3.10 • windows-2022 • x64 (pull_request) Failing after 6m
CI / 🐍 3.11 • windows-2022 • x64 (pull_request) Failing after 11m
CI / 🐍 pypy-3.7 • windows-2022 • x64 (pull_request) Failing after 13m
CI / 🐍 pypy-3.8 • windows-2022 • x64 (pull_request) Failing after 8m
CI / 🐍 pypy-3.9 • windows-2022 • x64 (pull_request) Failing after 9m
CI / 🐍 3.6 • macos-latest • x64 (pull_request) Failing after 9m
CI / 🐍 3.9 • macos-latest • x64 (pull_request) Failing after 8m
CI / 🐍 3.10 • macos-latest • x64 (pull_request) Failing after 7m
CI / 🐍 3.11 • macos-latest • x64 (pull_request) Failing after 8m
CI / 🐍 pypy-3.7 • macos-latest • x64 (pull_request) Failing after 8m
CI / 🐍 pypy-3.8 • macos-latest • x64 (pull_request) Failing after 9m
CI / 🐍 pypy-3.9 • macos-latest • x64 (pull_request) Failing after 10m
CI / 🐍 3.6 • windows-2019 • x64 -DPYBIND11_FINDPYTHON=ON (pull_request) Failing after 7m
CI / 🐍 3.9 • windows-2019 • x64 (pull_request) Failing after 9m
CI / 🐍 3 • Clang 3.6 • C++11 • x64 (pull_request) Failing after 41s
CI / 🐍 3 • Clang 3.7 • C++11 • x64 (pull_request) Failing after 50s
CI / 🐍 3 • Clang 3.9 • C++11 • x64 (pull_request) Failing after 54s
CI / 🐍 3 • Clang 7 • C++11 • x64 (pull_request) Failing after 50s
CI / 🐍 3 • Clang 9 • C++11 • x64 (pull_request) Failing after 43s
CI / 🐍 3 • Clang dev • C++11 • x64 (pull_request) Failing after 4m
CI / 🐍 3 • Clang 5 • C++14 • x64 (pull_request) Failing after 4m
CI / 🐍 3 • Clang 10 • C++17 • x64 (pull_request) Failing after 37s
CI / 🐍 3 • Clang 11 • C++20 • x64 (pull_request) Failing after 4m
CI / 🐍 3 • Clang 12 • C++20 • x64 (pull_request) Failing after 6m
CI / 🐍 3 • Clang 13 • C++20 • x64 (pull_request) Failing after 4m
CI / 🐍 3 • Clang 14 • C++20 • x64 (pull_request) Failing after 4m
CI / 🐍 3 • Clang 15 • C++20 • x64 (pull_request) Failing after 4m
CI / 🐍 3 • CentOS7 / PGI 22.9 • x64 (pull_request) Failing after 15m
CI / 🐍 3 • ICC latest • x64 (pull_request) Failing after 9m
CI / 🐍 3 • centos:7 • x64 (pull_request) Failing after 2m
CI / 🐍 3.6 • MSVC 2019 • x86 (pull_request) Failing after 6m
CI / 🐍 3.7 • MSVC 2019 • x86 -DCMAKE_CXX_STANDARD=14 (pull_request) Failing after 8m
CI / 🐍 3.8 • MSVC 2019 • x86 -DCMAKE_CXX_STANDARD=17 (pull_request) Failing after 7m
CI / 🐍 3.9 • MSVC 2019 • x86 -DCMAKE_CXX_STANDARD=20 (pull_request) Failing after 7m
CI / 🐍 3.8 • MSVC 2019 (Debug) • x86 -DCMAKE_CXX_STANDARD=17 (pull_request) Failing after 6m
CI / 🐍 3.9 • MSVC 2019 (Debug) • x86 -DCMAKE_CXX_STANDARD=20 (pull_request) Failing after 8m
CI / 🐍 3.9 • MSVC 2022 C++20 • x64 (pull_request) Failing after 11m
CI / 🐍 3.10 • windows-latest • clang-latest (pull_request) Failing after 10m
CI / macos-latest • brew install llvm (pull_request) Failing after 1m


CI / 🐍 3.6 • ubuntu-20.04 • x64 -DPYBIND11_FINDPYTHON=ON -DCMAKE_CXX_FLAGS="-D_=1" (pull_request) Successful in 11m
CI / 🐍 3.9 • ubuntu-20.04 • x64 (pull_request) Successful in 10m
CI / 🐍 3.10 • ubuntu-20.04 • x64 (pull_request) Successful in 11m
CI / 🐍 3.11 • ubuntu-20.04 • x64 (pull_request) Successful in 10m
CI / 🐍 pypy-3.7 • ubuntu-20.04 • x64 (pull_request) Successful in 13m
CI / 🐍 pypy-3.8 • ubuntu-20.04 • x64 -DPYBIND11_FINDPYTHON=ON (pull_request) Successful in 15m
CI / 🐍 pypy-3.9 • ubuntu-20.04 • x64 (pull_request) Successful in 13m
CI / 🐍 3.9-dbg (deadsnakes) • Valgrind • x64 (pull_request) Successful in 15m
CI / 🐍 3.11 (deadsnakes) • x64 (pull_request) Successful in 4m
CI / 🐍 3.10 • CUDA 11.7 • Ubuntu 22.04 (pull_request) Successful in 9m
CI / 🐍 3 • GCC 7 • C++11• x64 (pull_request) Successful in 4m
CI / 🐍 3 • GCC 7 • C++17• x64 (pull_request) Successful in 5m
CI / 🐍 3 • GCC 8 • C++14• x64 (pull_request) Successful in 5m
CI / 🐍 3 • GCC 8 • C++17• x64 (pull_request) Successful in 5m
CI / 🐍 3 • GCC 10 • C++17• x64 (pull_request) Successful in 5m
CI / 🐍 3 • GCC 11 • C++20• x64 (pull_request) Successful in 6m
CI / 🐍 3 • GCC 12 • C++20• x64 (pull_request) Successful in 5m
CI / 🐍 3 • almalinux:8 • x64 (pull_request) Successful in 5m
CI / 🐍 3 • almalinux:9 • x64 (pull_request) Successful in 4m
CI / 🐍 3.7 • Debian • x86 • Install (pull_request) Successful in 5m
CI / Documentation build test (pull_request) Successful in 1m
CI / 🐍 3 • windows-latest • mingw64 (pull_request) Successful in 19m
CI / 🐍 3 • windows-latest • mingw32 (pull_request) Successful in 27m

@ax3l
Copy link
Collaborator Author

ax3l commented Mar 22, 2023

Yes, the passing ones are with GCC and the Windows GCC port aka mingw32/64 ones.

Definitely curious issue...

@rwgk
Copy link
Collaborator

rwgk commented Mar 22, 2023

Yes, the passing ones are with GCC and the Windows GCC port aka mingw32/64 ones.

Definitely curious issue...

Oh, sorry I got libstdc++ and libc++ mixed up in my mind between November and now. I just re-learned:

  • libstdc++ is GNU (usually comes with GCC)
  • libc++ is LLVM (usually comes with Clang)

(I'm not sure this is strictly correct, but hopefully close enough.)

@ax3l
Copy link
Collaborator Author

ax3l commented Mar 22, 2023

What I find curious is the following: technically, all STLs should work if we always use our has implementation. But they do not, some fail. See last commit.

I wonder if there is something on top of your libc++ fix that we need to address here.

@rwgk
Copy link
Collaborator

rwgk commented Mar 22, 2023

I wonder if there is something on top of your libc++ fix that we need to address here.

Misunderstanding? PR #4319 doesn't have a fix for libc++, essentially it just changes the behavior for Windows.

My test under PR #4319 exercises the behavior for types in the unnamed namespace. Unconditionally using the t.name()-based comparisons (your last commit) breaks that on all platforms.

Your test is related but different.

My current understanding:

test                        libstdc++   libc++  MSVC   t.name()
unnamed namespace (#4319)    ok          fail    ok     fail everywhere
template template (#4581)    ok          fail    fail   some fail, some don't?

@ax3l
Copy link
Collaborator Author

ax3l commented Mar 22, 2023

I think the issue with the test I have here is actually rooted somewhere in the function dispatch logic:
https://github.com/pybind/pybind11/blob/v2.10.4/include/pybind11/pybind11.h#L668-L1126

@rwgk
Copy link
Collaborator

rwgk commented Mar 22, 2023

I think the issue with the test I have here is actually rooted somewhere in the function dispatch logic: https://github.com/pybind/pybind11/blob/v2.10.4/include/pybind11/pybind11.h#L668-L1126

Hm... My guess is that the cast_out::cast() call is failing.
I'd add prints to the make_S() C++ function to be sure it gets there, and then after the cast_out::cast() call to see what we got.

@ax3l
Copy link
Collaborator Author

ax3l commented Mar 23, 2023

Yep, this seems to be the location.

@rwgk
Copy link
Collaborator

rwgk commented Mar 24, 2023

A few observations:

I checked out your branch and made the .cpp code like this:

S<std::allocator> make_S() {
    std::cout << "in make_S()" << std::endl;
    return S<std::allocator>{};
}

That passes the test. (No surprise.)

But with

S<> make_S() {
    std::cout << "in make_S()" << std::endl;
    return S<std::allocator>{};
}

I see a compiler error:

clang++ -o pybind11/tests/test_template_alias_base.os -c -std=c++17 -fPIC -fvisibility=hidden -O0 -g -Wall -Wextra -Wconversion -Wcast-qual -Wdeprecated -Wundef -Wnon-virtual-dtor -Wunused-result -Werror -isystem /usr/include/python3.10 -isystem /usr/include/eigen3 -DPYBIND11_STRICT_ASSERTS_CLASS_HOLDER_VS_TYPE_CASTER_MIX -DPYBIND11_ENABLE_TYPE_CASTER_ODR_GUARD_IF_AVAILABLE -DPYBIND11_TEST_BOOST -Ipybind11/include -I/usr/local/google/home/rwgk/forked/pybind11/include -I/usr/local/google/home/rwgk/clone/pybind11/include /usr/local/google/home/rwgk/forked/pybind11/tests/test_template_alias_base.cpp
/usr/local/google/home/rwgk/forked/pybind11/tests/test_template_alias_base.cpp:20:12: error: no viable conversion from returned value of type 'S<template allocator>' to function return type 'S<(default) template DefaultAllocator>'
    return S<std::allocator>{};
           ^~~~~~~~~~~~~~~~~~~
/usr/local/google/home/rwgk/forked/pybind11/tests/test_template_alias_base.cpp:13:8: note: candidate constructor (the implicit copy constructor) not viable: no known conversion from 'S<std::allocator>' to 'const S<> &' for 1st argument
struct S : public Base<Allocator> {};
       ^
/usr/local/google/home/rwgk/forked/pybind11/tests/test_template_alias_base.cpp:13:8: note: candidate constructor (the implicit move constructor) not viable: no known conversion from 'S<std::allocator>' to 'S<> &&' for 1st argument
struct S : public Base<Allocator> {};
       ^
1 error generated.

Going back to return S<>{}; and running with this diff

diff --git a/include/pybind11/detail/type_caster_base.h b/include/pybind11/detail/type_caster_base.h
index 0b710d7e..941db900 100644
--- a/include/pybind11/detail/type_caster_base.h
+++ b/include/pybind11/detail/type_caster_base.h
@@ -928,6 +928,7 @@ public:
     // polymorphic_type_hook). If the instance isn't derived, returns the base version.
     static std::pair<const void *, const type_info *> src_and_type(const itype *src) {
         const auto &cast_type = typeid(itype);
+  printf("\nLOOOK typeid.name()=%s clean=%s %s:%d\n", cast_type.name(), detail::clean_type_id(cast_type.name()).c_str(), __FILE__, __LINE__); fflush(stdout);
         const std::type_info *instance_type = nullptr;
         const void *vsrc = polymorphic_type_hook<itype>::get(src, instance_type);
         if (instance_type && !same_type(cast_type, *instance_type)) {
diff --git a/include/pybind11/pybind11.h b/include/pybind11/pybind11.h
index fb8a75c0..c1f3fb44 100644
--- a/include/pybind11/pybind11.h
+++ b/include/pybind11/pybind11.h
@@ -1364,6 +1364,7 @@ protected:

         auto &internals = get_internals();
         auto tindex = std::type_index(*rec.type);
+  printf("\nLOOOK rec.type->name()=%s clean=%s %s:%d\n", rec.type->name(), detail::clean_type_id(rec.type->name()).c_str(), __FILE__, __LINE__); fflush(stdout);
         tinfo->direct_conversions = &internals.direct_conversions[tindex];
         if (rec.module_local) {
             get_local_internals().registered_types_cpp[tindex] = tinfo;

I see:

LOOOK rec.type->name()=4BaseISaE clean=Base<std::allocator> /usr/local/google/home/rwgk/forked/pybind11/include/pybind11/pybind11.h:1367
LOOOK rec.type->name()=1SISaE clean=S<std::allocator> /usr/local/google/home/rwgk/forked/pybind11/include/pybind11/pybind11.h:1367
LOOOK typeid.name()=1SI16DefaultAllocatorE clean=S<DefaultAllocator> /usr/local/google/home/rwgk/forked/pybind11/include/pybind11/detail/type_caster_base.h:931

With this addditional diff

diff --git a/tests/test_template_alias_base.cpp b/tests/test_template_alias_base.cpp
index f5d48f9d..475334fa 100644
--- a/tests/test_template_alias_base.cpp
+++ b/tests/test_template_alias_base.cpp
@@ -25,4 +25,7 @@ TEST_SUBMODULE(template_alias_base, m) {
     py::class_<S<std::allocator>, Base<std::allocator>>(m, "S_std").def(py::init());

     m.def("make_S", &make_S);
+
+    m.def("typeid_S_empty_eq_S_std_allocator", []() { return typeid(S<>) == typeid(S<std::allocator>); });
+    m.def("type_index_S_empty_eq_S_std_allocator", []() { return std::type_index(typeid(S<>)) == std::type_index(typeid(S<std::allocator>)); });
 }
diff --git a/tests/test_template_alias_base.py b/tests/test_template_alias_base.py
index 2cae121d..284fbf52 100644
--- a/tests/test_template_alias_base.py
+++ b/tests/test_template_alias_base.py
@@ -9,3 +9,11 @@ def test_can_create_variable():
 def test_can_return_variable():
     v = m.make_S()
     print(v)
+
+
+def test_typeid_eq():
+    assert m.typeid_S_empty_eq_S_std_allocator()
+
+
+def test_type_index_eq():
+    assert m.type_index_S_empty_eq_S_std_allocator()

and clang (clang 14):

______________________________________________________________ test_typeid_eq ______________________________________________________________

    def test_typeid_eq():
>       assert m.typeid_S_empty_eq_S_std_allocator()
E       assert False
E        +  where False = <built-in method typeid_S_empty_eq_S_std_allocator of PyCapsule object at 0x7f19ffa41530>()
E        +    where <built-in method typeid_S_empty_eq_S_std_allocator of PyCapsule object at 0x7f19ffa41530> = m.typeid_S_empty_eq_S_std_allocator


test_template_alias_base.py:15: AssertionError
____________________________________________________________ test_type_index_eq ____________________________________________________________

    def test_type_index_eq():
>       assert m.type_index_S_empty_eq_S_std_allocator()
E       assert False
E        +  where False = <built-in method type_index_S_empty_eq_S_std_allocator of PyCapsule object at 0x7f19ffa43ed0>()
E        +    where <built-in method type_index_S_empty_eq_S_std_allocator of PyCapsule object at 0x7f19ffa43ed0> = m.type_index_S_empty_eq_S_std_allocator


test_template_alias_base.py:19: AssertionError

But everything including those last two pass with gcc (gcc 12).

@rwgk
Copy link
Collaborator

rwgk commented Mar 24, 2023

I think this just boils down to the behaviors for:

typeid(S<>) == typeid(S<std::allocator>)
std::string(typeid(S<>).name()) == std::string(typeid(S<std::allocator>).name()) 

With gcc both are true.

With clang and MSVC both are false.

The main question: What does the C++ standard say about this, if anything?

Related detail: is the clang compiler error compatible with the C++ standard?

But for all practical purposes: I don't think there is an alternative to consistently using S<> or S<std::allocator> in the code you want to bind with pybind11.

@rwgk
Copy link
Collaborator

rwgk commented Mar 24, 2023

Actually, you could do something like:

py::class_<Base<std::allocator>>(m, "B_std").def(py::init());
if (!py::detail::get_type_info(typeid(Base<>)))) {
    py::class_<Base<>>(m, "B_std_").def(py::init());
}

But that'll get ugly with the derived class S<>.
I'd rather not do it.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
Projects
None yet
Development

Successfully merging this pull request may close these issues.

None yet

2 participants