New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Handle Empty Array for IN and NOT IN queries. #8035
Conversation
@@ -73,6 +75,10 @@ public function getArguments() | |||
*/ | |||
public function __toString() | |||
{ | |||
if (count($this->arguments) === 0) { | |||
return $this->name . '(NULL)'; |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
wouldn't this return all elements where the value is null for nullable columns? This seems wrong.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
@beberlei No,
Select foo from bar where foo IN (NULL)
Return always 0 row. I tried it multiple times.
foo IN (A, B)
mean foo = A or foo = B
foo IN (NULL)
mean foo = NULL
foo = NULL
always returns no result, instead of foo IS NULL
.
And as explained here: #8033 (comment), this is already the query created when using
$query->andWhere("$alias.foo IN (:bar)");
$query->setParameter('bar', []);
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
AFAIK NULL
is handled that way in all commonly known databases like e.g. PostgreSQL, MySQL, SQLite...
https://dbfiddle.uk/?rdbms=sqlite_3.8&fiddle=ac349cc3ca597516e307d690e7f19d90
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
I saw this nice comic on Twitter about this the other day :)
— 🔎Julia Evans🔍 (@b0rk
) October 17, 2019
@@ -299,7 +299,7 @@ public function testNotInLiteralExpr() | |||
|
|||
public function testNotInExprForEmptyArray() | |||
{ | |||
self::assertEquals('u.id NOT IN(FALSE)', (string) $this->_expr->notIn('u.id', [])); | |||
self::assertEquals('u.id NOT IN(NULL)', (string) $this->_expr->notIn('u.id', [])); |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
u.id NOT IN(NULL)
would result in an empty result too.
https://dbfiddle.uk/?rdbms=sqlite_3.8&fiddle=0097a9450b4e56c142e64a480b745614
this contradicts with your expectations described in your issue #8033 (comment)
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Yeah, I know.
It's still an improvement since
- IN (NULL) works correclty
- NOT IN (NULL) wont return an error, when NOT IN () does.
And in the next comment #8033 (comment), I explained that
$query->andWhere("foo.bar NOT IN (:baz)")->setParameter('baz', []);
Is actually translated as foo.bar NOT IN (NULL)
.
So my PR make the behaviour of $query->expr()->notIn()
more consistent with $query->andWhere("foo.bar NOT IN (:baz)")->setParameter('baz', []);
.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
We can improve the behaviour of the NOT IN query with an empty array, but It'll need a lot a more code changes and some discussions, so I recommend to split this from this PR, which was only looking for more consistency.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
If you want to split it, then you'll have to remove the NOT IN
changes. What do you want to change for supporting this case?
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
If you want to split it, then you'll have to remove the
NOT IN
changes.
I disagree. Actually if you write
$query->andWhere($queryBuilder->expr()->notIn('foo.bar', ':baz'))
$query->setParameter('baz', []);
There is no exception and this is translated as foo.bar NOT IN (NULL)
.
But if you write directly
$query->andWhere($queryBuilder->expr()->notIn('contractOperation.type', []))
You get an exception. But there is absolutely no reason to behave differently from the first case.
So it's still an improvement to have a consistent behaviour for notIn('foo.bar', ':baz'))
and notIn('foo.bar', []))
.
I improve both the IN
case and the NOT IN
case.
So I don't think I should remove the NOT IN
changes.
I don't know what I have to change to completely supports the NOT IN case and if it's possible. It should handle correctly the case
$query->andWhere('foo.bar NOT IN :baz')
$query->setParameter('baz', []);
$query->andWhere($queryBuilder->expr()->notIn('foo.bar', ':baz'))
$query->setParameter('baz', []);
$query->andWhere($queryBuilder->expr()->notIn('foo.bar', []))
And this seems for me like a lot of work/rework.
Is this PR perfect and fixing everything ? No
Is this PR better than the actual situation ? Yes
@beberlei @SenseException Is this PR ok for you ? :) |
Please kindly squash your commits together. If you don't, we'll try to remember to do it for you but it's best if you save us this trouble. How to do that?
|
Done @greg0ire :) |
Is this a good idea? It set off my footgun sense and |
I'm just calling for consistency. With this PR There is no reason for having the first syntax crashing when the others don't. I prefer to have my app blowing up for a real good reason, and this is not a valid one. |
Hi @beberlei @SenseException @greg0ire ; What is needed to make a decision about this PR ? :) I had in mind to add consistency, in order to have the same behaviour for
which was working, and
which was previously returning an exception. You can prefer the opposite behaviour (returning an exception) but either way I think there is something to do about this non-consistency. |
Hi @SenseException. My main goal was to add consistency, in order to have the same behaviour for
and
|
I'm not satisfied with leaving |
I understand. What would you expect as query ?
Ok. I thought the @greg0ire Since you approved the PR, would you agree to merge it ? |
@@ -73,6 +75,10 @@ public function getArguments() | |||
*/ | |||
public function __toString() | |||
{ | |||
if (count($this->arguments) === 0) { | |||
return $this->name . '(NULL)'; |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Wait so IN
is considered a function 🤔 🤔 🤔 ? I mean it has a left operand…
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Yes, that's what I discovered 😅
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
This makes it a bit scary… touching such a generic method looks like it could have unforeseen repercussions.
I gave it more thought, and I can't see why people would want to use |
The real use case I met, was using a variable, a const, a property, a select list of choice from the user, ...
Is really more convenient than
Plus, it needs less SQL-specific knowledge. When I searched for other ORM behaviour with empty array, it seems that they're generally trying to handle the query. For example: https://www.bennadel.com/blog/3288-its-safe-to-use-empty-arrays-with-the-in-operator-in-sequelize.htm |
I felt it would be something like that, what I don't understand (or rather, remember, I probably encountered this myself in the past) is how and why do you let the user pick an empty baz in the first place? |
The user-choice was not my best example indeed. But I have others. Sometimes, we didn't succeed making only one request and then we did something like.
Maybe the request was possible using only one query and we're not good enough with sql/doctrine, but this is still a possible use case. Sometimes, the array list is coming from the user property
And it was possible for us to have no category for a user. You might say that I can do:
But that's not always possible. Some function asked to return the non-executed query. |
Ah ok, that's why… IMO in that case we should not even attempt to execute a query since we know it should always return an empty result set. We should definitely throw, that should push the user to do more controls before attempting to create the query. And explanation like "This user has no categories" is way better than an empty list that will not explain why nothing was found, isn't it? |
I don't understand how an exception would help for the case ?
If an exception is thrown, I won't be able to catch it... And if I user some filters on my list which lead me to a I met few hundred of developers, none of them was checking for an empty array before doing
This is not necessarily the best developers we can found and they doesn't talk for every other developers, but it still showed me that checkin the empty array case is not natural/obvious/necessary/... for everyone. |
IMO there should be a method that the developer can implement to decide whether to build the query at all, before even calling
But you can always know there will be no result in such cases, without running SQL, right? Whether to display "no result" or an explanation might depend on the context, and could be configurable, but in any case, there is no need to ask your RDBMS to execute that statement.
I think this is where we disagree, but I guess we could leave the possibility to implement a
With phpstan and psalm coming to maturity in the last years, there is a healthy trend of more input checking, and eliminating subtle bugs such as those that can arise when you support weird edge cases. This is a nice talk about what can happen when you try to support all the edge cases: https://www.destroyallsoftware.com/talks/wat |
@greg0ire If some rule should be added to decide if yes or no the query should be executed, it would be more useful to do it in doctrine, this way the developer won't have to make every time the check (this could lead more easily to duplicate code, errors, difficulty to debug...). I think you're taking a too much elaborate way of thinking the solution. There is a lot of beginners/junior developper using this library. From my experience I only saw three things:
I never saw a developper who wanted/tried/asked to handle the empty array case the way you describe. The fact that empty array are supported when using |
When you use an ORM you don't need to know about SQL at all to understand that asking for a contract operation which type is in Besides isn't there a bug in the UI/the js lib that posts the form with the filters, if we allow it to send
Why would you? I mean, sending an empty array inside To sum up my thoughts, doing so happens if:
|
That's where I disagree with you.
works. I expect the same If I compute by myself what's inside the Sometimes we didn't find other way than doing things like
You may say that performance are bad, I agree. And I have no reason to have a different UI if the pre-computed result is empty. The user doesn't have to know if I compute in sql or in php.
I gave you other example. You may compute the $array before using it in the query.
Not when the exception can occur in production and have huge impacts. In twig,
I met this case dozens of times.
|
Maybe you're right, I don't know. What I'm sure about is that I'm no longer approving of this PR. I won't block it, but I will let people that deal with that kind of situation more often than I do take care of it. |
I changed my mind: the implications of touching Func.php
are not 100% clear to me, and I feel that we should tell our users that using IN
with an empty array is not going to work instead of trying to make it work with NULL
. I might be wrong, because IIRC Doctrine sometimes uses IN
itself with identifiers obtained from somewhere else, but I can't remember.
I feel really bad about this @greg0ire. You're already the second one taking this position. This was approved here : doctrine/dbal#264 Ignoring this PR will still let a big inconsistency in the behaviour of the query builder for empty array... Having something who sometimes works sometimes not, is the worst situation. If you don't want to handle empty array, I strongly invite you to create a PR to revert both doctrine/dbal#264 and doctrine/DoctrineBundle#1144, and throw the exception you want instead. This non-consistent behaviour should not exist. |
doctrine/DoctrineBundle#1144 should be reverted after the exception I think might be good is thrown (meaning the bundle requires a version that has the exception). Otherwise, it will lead the profiler to show wrong information. |
I understand, and be happy to have other reviews. Thanks for the request-review you made. |
Hi @greg0ire. It seems that this problem does not interest a lot of people. |
I'd wait until the end of the lockdown. Some of us have families to take care of, and can't spend much time on open source during that time. |
My bad. I never saw it this way since, because of the lockdown, I can't be with my family and don't have much to do except open source 😅 Let's wait then ! |
Hi @greg0ire, friendly reminder. :) |
I think you should start with a rebase, looks like your build does not have runs for Github Actions that are now mandatory. |
Done :) |
@doctrine/team-orm please review |
Hi, @greg0ire. Can we do something about this PR ? :) |
I don't know. @SenseException can you maybe answer #8035 (comment) ? |
It's still like I wrote in #8035 (comment). A lot of thoughts and time were put into this PR, which I appreciate a lot, but I cannot become comfortable with the @greg0ire mentioned other arguments (especially #8035 (comment)) that makes me keep my position on this. In my DX, I don't want software to run code and have an unexpected result that needs to be debugged and costs developer time. SQL Queries with invalid arguments or with an expected empty result shouldn't be forwarded to the database anyway. In the end, you would still need an empty array check in an application if the empty result isn't the expected behaviour for a developer. Not having to run SQL queries has a positive effect on performance (depending on the projects) and the exception makes developers reconsider how they write project-code. I still would like to see more reviewers for this PR though. |
I agree with your arguments. Here, more than everything, I look for consistency. Currently, we have the following inconsistency
What about changing this to
This way we'll have consistency in both version of doctrine, and it'll go in your way if you prefer an exception. |
WDYT about this strategy @greg0ire @SenseException ? |
This matches with what I would expect for the next major release.
Either way, not throwing or throwing for all, wouldn't this be a BC break? Maybe that's what @greg0ire was referring to in #8035 (comment). This PR might be one that should target 3.0. |
Fix #8033