[Lock] MongoDbStore skim non-standard options from uri #37218

kralos · 2020-06-11T13:19:35Z

Q	A
Branch?	5.1
Bug fix?	yes
New feature?	no
Deprecations?	no
Tickets	Fix #37180
License	MIT
Doc PR

Don't allow non-standard connection uri options reach MongoDB\Client

nicolas-grekas · 2020-06-11T13:27:06Z

src/Symfony/Component/Lock/Store/MongoDbStore.php

+            parse_str($parsedUrl['query'], $query);
+        }
+
+        if (null === ($this->options['collection'] ?? null) && isset($query['collection'])) {


I know this was already the case before your PR, but the precedence looks wrong to me:
the DSN should have higher precedence over options (I just double-checked, that's how the Cache component does also).

/cc @jderusse maybe we should review all DSN parsing logic to ensure the precedence is consistent everywhere
/cc @Nyholm that's a good case for the DSN component - formalizing this behavior :)

👍 not only in cache and lock.

messenger Doctrine:

$configuration += $options + $query + static::DEFAULT_OPTIONS;

messenger Redis:

parse_str($parsedUrl['query'], $options);

messenger Sqs

$configuration = ['buffer_size'] = $options['buffer_size'] ?? (int) ($query['buffer_size'] ?? self::DEFAULT_OPTIONS['buffer_size']),

messenger amqp

$amqpOptions = array_replace_recursive([], $options, $parsedQuery);

Bad news, we need to fix this. Up for a PR?

@nicolas-grekas In order to change the precedence, I think we should detect dsn / options where a precedence change would cause an effective option change and instead trigger_deprecation for now. Then swap the precedence in symfony 6.0. Do you want me to add this detection / trigger_deprecation in this PR for Lock\Store\MongoDbStore only or should we make a list of ALL dsn / options precedence affected classes and do them in a separate PR?

I've added the precedence change since you guys seem to be classing this as a bug rather than a deprecation in: #37268

There is no way to deprecate this behavior properly. Also this will very unlikely hit anyone. This precedence issue lessens the capabilities of the ops side of things. Basically, having the options take over the DSN means that the source needs to change (the options) instead of some env vars. That's a management issue at this point, and this change fixes a process issue: the env vars should always win to give control to the ops, which are the ones that know better since they're the ones in touch with the effective infra.

TL;DR, this is a bugfix.

…derusse) This PR was squashed before being merged into the 5.1 branch. Discussion ---------- [Lock][Messenger] Fix precedence of DSN options for 5.1 | Q | A | ------------- | --- | Branch? | 5.1 | Bug fix? | yes | New feature? | no | Deprecations? | no | Tickets | #37218 (comment) | License | MIT | Doc PR | N/A This PR fix précédence of DSN options over constructor options in all component on branch 5.1 Commits ------- 9670e9f [Lock][Messenger] Fix precedence of DSN options for 5.1

This PR was merged into the 4.4 branch. Discussion ---------- [Messenger] Fix precedence of DSN options for 4.4 | Q | A | ------------- | --- | Branch? | 4.4 | Bug fix? | yes | New feature? | no | Deprecations? | no | Tickets | #37218 (comment) | License | MIT | Doc PR | N/A This PR fix précédence of DSN options over constructor options in all component on branch 4.4 Commits ------- 992205a Fix precendence in 4.4

This PR was merged into the 4.4 branch. Discussion ---------- [Messenger] Fix precedence of DSN options for 4.4 | Q | A | ------------- | --- | Branch? | 4.4 | Bug fix? | yes | New feature? | no | Deprecations? | no | Tickets | symfony/symfony#37218 (comment) | License | MIT | Doc PR | N/A This PR fix précédence of DSN options over constructor options in all component on branch 4.4 Commits ------- 992205a759 Fix precendence in 4.4

nicolas-grekas · 2020-06-18T18:45:15Z

Can you please rebase now that the priority is fixed?

jderusse · 2020-06-19T10:22:49Z

src/Symfony/Component/Lock/Store/MongoDbStore.php

+    private function skimUri(string $uri): string
+    {
+        if (false === $parsedUrl = parse_url($uri)) {
+            throw new InvalidArgumentException(sprintf('The given MongoDB Connection URI "%s" is invalid.', $mongo));


$mongo is not defined

jderusse · 2020-06-19T10:24:00Z

src/Symfony/Component/Lock/Store/MongoDbStore.php

+    /**
+     * Extract default database and collection from given connection uri and remove collection querystring.
+     */
+    private function skimUri(string $uri): string


I'm not comfortable with this method, a lot of complex logic to extract parameter from URI mixing parse_str and strpos.

Givent the format of the initial DSN is defined by symfony, wouldn't it be safer to build the mongo URI instead?

the reason i chose to use strpos to re-assemble was to simplify the re-assembly. there is no reverse of parse_str in php. see https://www.php.net/manual/en/function.parse-url.php#106731

I'm trying not to get too involved with touching the URI. What would you suggest that is simpler?

jderusse · 2020-07-01T20:04:49Z

@nicolas-grekas LGTM from a lock point of view. I just have concerns about complexity of url generator #37218 (comment) . If you think it's ok you can merge it.

fabpot · 2020-08-16T14:54:59Z

Thank you @kralos.

jmikola · 2020-08-17T16:51:39Z

src/Symfony/Component/Lock/Store/MongoDbStore.php

+                $prefix = rtrim($prefix, '?');
+            }
+            $suffix = substr($uri, $queryStringPos + \strlen($parsedUrl['query']));
+            $uri = $prefix.$newQuery.$suffix;


@kralos: I only just came across this PR based on a notification in #37180. I understand what this function is attempting to do, but using parse_str may silently corrupting a connection string with repeated readPreferenceTags keys (a valid use case, as explained in the URI options spec). This comment on PHP.net goes into more detail about that.

I think it would be preferable to capture the URI option with a regular expression and then, if anything was found, strip it from the returned string. While collection names have their own restrictions, for purposes of URI parsing I think it'd be best to use something like /collection=([^&]*)/i.

see #37864 / #37868

This PR was merged into the 4.4 branch. Discussion ---------- [Messenger] Fix precedence of DSN options for 4.4 | Q | A | ------------- | --- | Branch? | 4.4 | Bug fix? | yes | New feature? | no | Deprecations? | no | Tickets | symfony/symfony#37218 (comment) | License | MIT | Doc PR | N/A This PR fix précédence of DSN options over constructor options in all component on branch 4.4 Commits ------- 992205a759 Fix precendence in 4.4

carsonbot added Status: Needs Review Lock Bug labels Jun 11, 2020

kralos changed the base branch from master to 5.1 June 11, 2020 13:19

nicolas-grekas reviewed Jun 11, 2020

View reviewed changes

This was referenced Jun 13, 2020

[Messenger] Fix precedence of DSN options for 4.4 #37268

Merged

[Lock][Messenger] Fix precedence of DSN options for 5.1 #37269

Merged

kralos marked this pull request as ready for review June 15, 2020 06:05

kralos requested a review from xabbuh as a code owner June 15, 2020 06:24

kralos force-pushed the 37180-lock-mongodbstore-uri branch 2 times, most recently from e06956a to 599c128 Compare June 15, 2020 06:48

nicolas-grekas added this to the 5.1 milestone Jun 18, 2020

kralos marked this pull request as draft June 18, 2020 23:08

kralos force-pushed the 37180-lock-mongodbstore-uri branch from 4df26a9 to 523bf2a Compare June 18, 2020 23:08

kralos marked this pull request as ready for review June 18, 2020 23:17

kralos requested review from nicolas-grekas and jderusse June 18, 2020 23:17

jderusse reviewed Jun 19, 2020

View reviewed changes

nicolas-grekas requested a review from jderusse June 28, 2020 15:34

fabpot approved these changes Aug 16, 2020

View reviewed changes

carsonbot added Status: Reviewed and removed Status: Needs Review labels Aug 16, 2020

[Lock] MongoDbStore skim non-standard options from uri

67912fa

fabpot force-pushed the 37180-lock-mongodbstore-uri branch from 79fee70 to 67912fa Compare August 16, 2020 14:54

fabpot merged commit 5f074cd into symfony:5.1 Aug 16, 2020

jmikola reviewed Aug 17, 2020

View reviewed changes

jmikola mentioned this pull request Aug 17, 2020

[Lock] MongoDbStore::skimUri may strip repeated query string keys #37864

Closed

kralos deleted the 37180-lock-mongodbstore-uri branch August 17, 2020 23:40

jmikola mentioned this pull request Aug 18, 2020

[Lock] MongoDbStore handle duplicate querystring keys in mongodb uri when stripping #37868

Merged

fabpot mentioned this pull request Aug 31, 2020

Release v5.1.4 #37999

Merged

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

[Lock] MongoDbStore skim non-standard options from uri #37218

[Lock] MongoDbStore skim non-standard options from uri #37218

kralos commented Jun 11, 2020

nicolas-grekas Jun 11, 2020

jderusse Jun 11, 2020

nicolas-grekas Jun 11, 2020

kralos Jun 11, 2020

kralos Jun 15, 2020

nicolas-grekas Jun 15, 2020

nicolas-grekas commented Jun 18, 2020

jderusse Jun 19, 2020

kralos Jun 22, 2020

jderusse Jun 19, 2020

kralos Jun 22, 2020

jderusse commented Jul 1, 2020

fabpot commented Aug 16, 2020

jmikola Aug 17, 2020

kralos Aug 18, 2020

[Lock] MongoDbStore skim non-standard options from uri #37218

[Lock] MongoDbStore skim non-standard options from uri #37218

Conversation

kralos commented Jun 11, 2020

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

nicolas-grekas commented Jun 18, 2020

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

jderusse commented Jul 1, 2020

fabpot commented Aug 16, 2020

Choose a reason for hiding this comment

Choose a reason for hiding this comment