tests: add test that validates the callmap in the current runtime

[SamMousa]

Test for #8101

Reflection doesn't always work correctly for built-in functions, especially those defined by extensions. It's also going to be less specific than any psalm specific type like numeric-string, int<0, max>, class-string, etc. I can already see that many of those are false positives, so using a generated callmap is definitely not possible.

If you want to add that snippet to the bin directory and do a manual comparison to see which are actual problems though that would be great!

I'm aware that more specific return types are won't be validated by this tester; this is still a WIP.
The idea is that we can at least fail on things that are guaranteed to be wrong.

[AndrolGenhald]

Might be worth trying to scrape the documentation like #8000 as well. If reflection and documentation disagree we can see whether we need to open a documentation issue or ignore reflection for that case.

[AndrolGenhald]

Looking good so far! A lot of these we'll have to look at the documentation or test to see if we need to add a change to one of the version diffs or if the callmap was always incorrect and the historical one needs changed.

[SamMousa]

Yeah I'm liking it as well. Just an FYI regarding your comments on the change maps.
My time is limited and I prefer not to spend it on old PHP versions. Since the fixes fix what is currently broken in PHP8.1, to me this is already an improvement from what we have today (where it works on some older versions but definitely not on 8.1).

So I hope someone else is willing to do the manual fixes for the other version diffs. Note that this test code can also be run on PHP8 to identify issues with the callmap (which can then be fixed in the diff).

[orklah]

Since the fixes fix what is currently broken in PHP8.1, to me this is already an improvement

It is. Unfortunately, the way callmap works is that changing callmap is not enough. Callmap.php should always match with another Callmap file in which the signature was introduced. So you're forced to either pick the correct version, or to change Callmap_historical (which will break signatures for old versions that didn't change)

[SamMousa]

Unfortunately, the way callmap works is that changing callmap is not enough. Callmap.php should always match with another Callmap file in which the signature was introduced.

When running on the current version though, the historical file and diffs are not used. They are only used in downgrading the callmap (if i understand correctly). So this will work just fine on php 8.1 and fixing it to work on old version is something someone else will hopefully have motivation for.

Note that many (100+ of the fixes in the callmap) changes were already broken to begin with for older PHP versions, we just didn't know.

[SamMousa]

Is there some internal psalm function that will tell me if 2 types are compatible? Currently I'm using my own poor man's parser, but im thinking it probably already exists somewhere else.

[orklah]

You can use UnionTypeComparator::isContainedBy to check if an union is contained completely in another. Type;;parseString should give you an union from a string

[SamMousa]

@orklah to get this ready for merged I've chosen the following approach:

1. Revert most fixes to the callmap.
2. Add lots if exclusions to the new test.

If you'd be so kind to add a release label then all tests will pass.

Currently this the test file has ~2355 tests (in my docker container that has all possible php extensions enabled), of which 1167 are skipped.
This means that we have verified that ~1180 functions in the callmap are correct.

Next steps are to incrementally remove functions and prefixes from the ignore list and fix their definitions in the callmap & diff files. This can be done by anyone looking for simple improvements.
Also we should add the procedure that anyone making a PR that changes the callmap removes the corresponding functions from the ignore list.

[orklah]

Cool! Could you open an issue explaining how to tackle a function so we can redirect contributors to it when needed?

[AndrolGenhald]

We should probably update the callmap guide.