ipv6 zone identifier mitigation conflicts with URL percentage escape

[tony84727]

Problem

When connecting to AWS IoT by "MQTT over WebSocket". To authenticate, we need to pre-sign the URL by AWS v4 signer. The signing process basically appends a bunch of query strings to the URL.
One of the query string parameters looks like this: X-Amz-Credential=keyid%2Fdate%2Fregion%2Fservice%2Faws4_req
It's encoded because the parameter contains slashes. It can be decoded to X-Amz-Credential=keyid/date/region/service/aws4_req

But the ipv6 zone identifier migration in ClientOptions#AddBroker, coming from bb7927e replaces all % in the URL to %25, which escapes all % in the URL even though the percentage sign itself is an escape.

paho.mqtt.golang/options.go

Line 147 in 72d5136

server = re.ReplaceAllLiteralString(server, "%25")

After the replacement, X-Amz-Credential=keyid%2Fdate%2Fregion%2Fservice%2Faws4_req becomes X-Amz-Credential=keyid%252Fdate%252Fregion%252Fservice%252Faws4_req, which will be decoded to X-Amz-Credential=keyid%2Fdate%2Fregion%2Fservice%2Faws4_req and the AWS will complain invalid credential.

Expected Behavior

url.Parse from Golang seems to have trouble with the zone identifier in the past. Here's the ticket

RFC6874 specifies how the zone identifier in the URI should be handled.
In section 2:

According to URI syntax [RFC3986], "%" is always treated as
an escape character in a URI, so, according to the established URI
syntax [RFC3986] any occurrences of literal "%" symbols in a URI MUST
be percent-encoded and represented in the form "%25". Thus, the
scoped address fe80::a%en1 would appear in a URI as
http://[fe80::a%25en1].

So perhaps the correct way is to remove the zone identifier mitigation and the users that intent to append the zone identifier in the URL should escape them properly before passing to ClientOptions#AddBroker

But I expect doing so will be a breaking change.

[MattBrittan]

Given that the RFC6874 states that % must be escaped I think we should roll back PR #319. I agree that this would be potentially breaking (but the original PR was also breaking). I considered altering the regex so it only changes the host portion of the URL but getting this right would require quite a bit of work/testing and I'm not sure that it would be beneficial given that the #319 appears to be a workaround for users passing non-compliant URLs in.

@alsm - what are your thoughts?

[MattBrittan]

Please try @master and let me know if this resolves the issues.

[tony84727]

Please try @master and let me know if this resolves the issues.

Yes, the master branch works. I can add the AWS pre-signed URL by AddBroker and successfully connect to the AWS IoT.

Thank you!