feat: added sigv4 filter #3070

Anurag252 · 2024-05-07T16:46:25Z

Example 1 with default body {}

actual sdk code

see https://go.dev/play/p/nPiTzwAl6Ut for a signed sample request with sdk.
response : -

Signed Request:
AWS4-HMAC-SHA256 Credential=AKID/19700101/us-east-1/dynamodb/aws4_request, SignedHeaders=accept;content-length;content-type;host;x-amz-date;x-amz-meta-other-header;x-amz-meta-other-header_with_underscore;x-amz-security-token;x-amz-target, Signature=0dabdcda8c405b6f3cdd34c7a63a4d1a701942bd78fcd535ccd49de33be2a229

skipper

./bin/skipper -inline-routes='r: * -> status(200) -> sigv4( "us-east-1","dynamodb", "false", "false", "false") -> "http://httpbin.org"'

curl -

curl -v  POST \
  http://localhost:9090/anything \
  -H 'X-Amz-Target: prefix.Operation' \
  -H 'Content-Type: application/x-amz-json-1.0' \
  -H 'X-Amz-Meta-Other-Header: some-value=!@#$%^&* (+)' \
  -H 'X-Amz-Meta-Other-Header_With_Underscore: some-value=!@#$%^&* (+)' \
  -H 'X-amz-Meta-Other-Header_With_Underscore: some-value=!@#$%^&* (+)' \
  -H 'x-amz-accesskey: AKID' \
  -H 'x-amz-secret: SECRET' \
  -H 'x-amz-session: SESSION' \
  -H 'x-amz-time: 1970-01-01T00:00:00Z' \
  -H 'Host: dynamodb.us-east-1.amazonaws.com' -d '{}'

Response from curl -

*   Trying 52.20.143.163:80...
* Connected to POST (52.20.143.163) port 80 (#0)
> POST / HTTP/1.1
> Host: dynamodb.us-east-1.amazonaws.com
> User-Agent: curl/8.1.2
> Accept: */*
> X-Amz-Target: prefix.Operation
> Content-Type: application/x-amz-json-1.0
> X-Amz-Meta-Other-Header: some-value=!@#$%^&* (+)
> X-Amz-Meta-Other-Header_With_Underscore: some-value=!@#$%^&* (+)
> X-amz-Meta-Other-Header_With_Underscore: some-value=!@#$%^&* (+)
> x-amz-accesskey: AKID
> x-amz-secret: SECRET
> x-amz-session: SESSION
> x-amz-time: 1970-01-01T00:00:00Z
> Content-Length: 2
> 
< HTTP/1.1 404 Not Found
< Date: Thu, 09 May 2024 04:37:17 GMT
< Content-Type: text/html
< Content-Length: 146
< Connection: keep-alive
< 
<html>
<head><title>404 Not Found</title></head>
<body>
<center><h1>404 Not Found</h1></center>
<hr><center>nginx</center>
</body>
</html>
* Connection #0 to host POST left intact
*   Trying 127.0.0.1:9090...
* Connected to localhost (127.0.0.1) port 9090 (#1)
> POST /anything HTTP/1.1
> Host: dynamodb.us-east-1.amazonaws.com
> User-Agent: curl/8.1.2
> Accept: */*
> X-Amz-Target: prefix.Operation
> Content-Type: application/x-amz-json-1.0
> X-Amz-Meta-Other-Header: some-value=!@#$%^&* (+)
> X-Amz-Meta-Other-Header_With_Underscore: some-value=!@#$%^&* (+)
> X-amz-Meta-Other-Header_With_Underscore: some-value=!@#$%^&* (+)
> x-amz-accesskey: AKID
> x-amz-secret: SECRET
> x-amz-session: SESSION
> x-amz-time: 1970-01-01T00:00:00Z
> Content-Length: 2
> 
< HTTP/1.1 200 OK
< Access-Control-Allow-Credentials: true
< Access-Control-Allow-Origin: *
< Connection: keep-alive
< Content-Length: 1071
< Content-Type: application/json
< Date: Thu, 09 May 2024 04:37:17 GMT
< Server: gunicorn/19.9.0
< 
{
  "args": {}, 
  "data": "{}", 
  "files": {}, 
  "form": {}, 
  "headers": {
    "Accept": "*/*", 
    "Accept-Encoding": "gzip", 
    "Authorization": "AWS4-HMAC-SHA256 Credential=AKID/19700101/us-east-1/dynamodb/aws4_request, SignedHeaders=accept;content-length;content-type;host;x-amz-date;x-amz-meta-other-header;x-amz-meta-other-header_with_underscore;x-amz-security-token;x-amz-target, Signature=0dabdcda8c405b6f3cdd34c7a63a4d1a701942bd78fcd535ccd49de33be2a229", 
    "Content-Length": "2", 
    "Content-Type": "application/x-amz-json-1.0", 
    "Host": "httpbin.org", 
    "User-Agent": "curl/8.1.2", 
    "X-Amz-Date": "19700101T000000Z", 
    "X-Amz-Meta-Other-Header": "some-value=!@#$%^&* (+)", 
    "X-Amz-Meta-Other-Header-With-Underscore": "some-value=!@#$%^&* (+),some-value=!@#$%^&* (+)", 
    "X-Amz-Security-Token": "SESSION", 
    "X-Amz-Target": "prefix.Operation", 
    "X-Amzn-Trace-Id": "Root=1-663c52fd-1c443e565cba09c323349703"
  }, 
  "json": {}, 
  "method": "POST", 
  "origin": "158.181.74.224", 
  "url": "http://httpbin.org/anything"
}

Example 2 with non-default body

skipper

./bin/skipper -inline-routes='r: * -> status(200) ->  sigv4( "us-east-1","dynamodb",  "false", "false", "false") -> "http://httpbin.org"'

curl

anuragdubey@Anurags-Air skipper % curl -v  POST \
  http://localhost:9090/anything \
  -H 'X-Amz-Target: prefix.Operation' \
  -H 'Content-Type: application/x-amz-json-1.0' \
  -H 'X-Amz-Meta-Other-Header: some-value=!@#$%^&* (+)' \
  -H 'X-Amz-Meta-Other-Header_With_Underscore: some-value=!@#$%^&* (+)' \
  -H 'X-amz-Meta-Other-Header_With_Underscore: some-value=!@#$%^&* (+)' \
  -H 'x-amz-accesskey: AKID' \
  -H 'x-amz-secret: SECRET' \
  -H 'x-amz-session: SESSION' \
  -H 'x-amz-time: 1970-01-01T00:00:00Z' \
  -H 'Host: dynamodb.us-east-1.amazonaws.com' -d '{key:value}'   
*   Trying 34.232.152.68:80...
* Connected to POST (34.232.152.68) port 80 (#0)
> POST / HTTP/1.1
> Host: dynamodb.us-east-1.amazonaws.com
> User-Agent: curl/8.1.2
> Accept: */*
> X-Amz-Target: prefix.Operation
> Content-Type: application/x-amz-json-1.0
> X-Amz-Meta-Other-Header: some-value=!@#$%^&* (+)
> X-Amz-Meta-Other-Header_With_Underscore: some-value=!@#$%^&* (+)
> X-amz-Meta-Other-Header_With_Underscore: some-value=!@#$%^&* (+)
> x-amz-accesskey: AKID
> x-amz-secret: SECRET
> x-amz-session: SESSION
> x-amz-time: 1970-01-01T00:00:00Z
> Content-Length: 11
> 
< HTTP/1.1 404 Not Found
< Date: Thu, 09 May 2024 09:12:02 GMT
< Content-Type: text/html
< Content-Length: 146
< Connection: keep-alive
< 
<html>
<head><title>404 Not Found</title></head>
<body>
<center><h1>404 Not Found</h1></center>
<hr><center>nginx</center>
</body>
</html>
* Connection #0 to host POST left intact
*   Trying 127.0.0.1:9090...
* Connected to localhost (127.0.0.1) port 9090 (#1)
> POST /anything HTTP/1.1
> Host: dynamodb.us-east-1.amazonaws.com
> User-Agent: curl/8.1.2
> Accept: */*
> X-Amz-Target: prefix.Operation
> Content-Type: application/x-amz-json-1.0
> X-Amz-Meta-Other-Header: some-value=!@#$%^&* (+)
> X-Amz-Meta-Other-Header_With_Underscore: some-value=!@#$%^&* (+)
> X-amz-Meta-Other-Header_With_Underscore: some-value=!@#$%^&* (+)
> x-amz-accesskey: AKID
> x-amz-secret: SECRET
> x-amz-session: SESSION
> x-amz-time: 1970-01-01T00:00:00Z
> Content-Length: 11
> 
< HTTP/1.1 200 OK
< Access-Control-Allow-Credentials: true
< Access-Control-Allow-Origin: *
< Connection: keep-alive
< Content-Length: 1125
< Content-Type: application/json
< Date: Thu, 09 May 2024 09:12:03 GMT
< Server: gunicorn/19.9.0
< 
{
  "args": {}, 
  "data": "{key:value}", 
  "files": {}, 
  "form": {}, 
  "headers": {
    "Accept": "*/*", 
    "Accept-Encoding": "gzip", 
    "Authorization": "AWS4-HMAC-SHA256 Credential=AKID/19700101/us-east-1/dynamodb/aws4_request, SignedHeaders=accept;content-length;content-type;host;x-amz-date;x-amz-meta-other-header;x-amz-meta-other-header_with_underscore;x-amz-security-token;x-amz-target, Signature=8ff94a60e916faca5238078181d0858d37945640ce2e967ab8361b639252593c", 
    "Content-Length": "11", 
    "Content-Type": "application/x-amz-json-1.0", 
    "Host": "dynamodb.us-east-1.amazonaws.com", 
    "User-Agent": "curl/8.1.2", 
    "X-Amz-Date": "19700101T000000Z", 
    "X-Amz-Meta-Other-Header": "some-value=!@#$%^&* (+)", 
    "X-Amz-Meta-Other-Header-With-Underscore": "some-value=!@#$%^&* (+),some-value=!@#$%^&* (+)", 
    "X-Amz-Security-Token": "SESSION", 
    "X-Amz-Target": "prefix.Operation", 
    "X-Amzn-Trace-Id": "Root=1-663c9363-20d7b8ee264d1bfa56373aa7"
  }, 
  "json": null, 
  "method": "POST", 
  "origin": "158.181.74.224", 
  "url": "http://dynamodb.us-east-1.amazonaws.com/anything"
}
* Connection #1 to host localhost left intact

actual sdk code

https://go.dev/play/p/Ixx-ozxeWfv
Response

Signed Request:
AWS4-HMAC-SHA256 Credential=AKID/19700101/us-east-1/dynamodb/aws4_request, SignedHeaders=accept;content-length;content-type;host;x-amz-date;x-amz-meta-other-header;x-amz-meta-other-header_with_underscore;x-amz-security-token;x-amz-target, Signature=8ff94a60e916faca5238078181d0858d37945640ce2e967ab8361b639252593c

Signed-off-by: Anurag252 <Anuragsgsits@gmail.com>

Anurag252 · 2024-05-09T09:06:19Z

filters/sigv4/signer/httpsigner.go

+		signedURLMsg = fmt.Sprintf(logSignedURLMsg, request.Request.URL.String())
+	}
+	logger := log.WithContext(ctx.Request().Context())
+	logger.Logf(log.DebugLevel, logSignInfoMsg, request.CanonicalString, request.StringToSign, signedURLMsg)


This is only for debugging and only logs when options.LogSigning is true. This is also taken directly from SDK code here

Signed-off-by: Anurag252 <Anuragsgsits@gmail.com>

AlexanderYastrebov

This is quite a lot of new code and I am concerned about supporting this.

@szuecs should we also consider

adding AWS sdk as a dependency
copy-paste AWS signer code (vendor-in)
use AWS sdk with a build tag to exclude for lib users by default

filters/filters.go

AlexanderYastrebov · 2024-05-13T18:57:51Z

filters/signer/internal/cache.go

+	return cred
+}
+
+func deriveKey(secret, service, region string, t SigningTime) []byte {


Do we need all this caching complexity from the start?
Maybe we may have a benchmark to see how long does it take to calculate these four hmacs ?

Do we need all this caching complexity from the start? Maybe we may have a benchmark to see how long does it take to calculate these four hmacs ?

Thanks for the review. I copied this entirely from sdk. My approach was to introduce very little over what is available from sdk, just enough to independently carve out signing functionality . I would be open to do as you suggested of course.

filters/signer/internal/uri.go

filters/signer/sigv4/sigv4.go

AlexanderYastrebov · 2024-05-13T19:47:33Z

filters/signer/internal/cache.go

+
+func lookupKey(service, region string) string {
+	var s strings.Builder
+	s.Grow(len(region) + len(service) + 3)


copied this from here.

~~A quick test reveals that / takes up 3 bytes . I suspect that there is some char escaping taking place . I can confirm after checking the source for Marshal~~

I am not entirely sure. A rune typically occupies 1 to 4 bytes, but looking at code here , there is an optimization to check if it fits lesser number of bytes. For '/' rune, uint32 value comes out to be 47 which is < rune1Max and may only need one byte.

filters/signer/internal/cache.go

AlexanderYastrebov · 2024-05-13T19:49:14Z

filters/signer/internal/cache.go

+}
+
+// SigningKeyDeriver derives a signing key from a set of credentials
+type SigningKeyDeriver struct {


Why some types/functions are exported and some not?

Made Get un-exported. So now only SigningKeyDeriver type and its function DeriveKey is exported , since they are used in httpsigner.go

szuecs · 2024-05-14T12:14:45Z

This is quite a lot of new code and I am concerned about supporting this.

@szuecs should we also consider
* adding AWS sdk as a dependency

* copy-paste AWS signer code (vendor-in)

* use AWS sdk with a build tag to exclude for lib users by default

Good question, I thought it makes sense to not put AWS SDK into dependency because this is really huge.
The contributor copies basically code (option 2) that won't be able to be changed anyways into the repository, because AWS can't really change the SDK signing code, because if so they likely need to change validation, too and then you would break half the internet that uses AWS SDK to have to update.

Adding an AWS build tag would be something we can do to have this feature as a more "experimental" implementation.

Signed-off-by: Anurag252 <Anuragsgsits@gmail.com>

Anurag252 · 2024-05-28T09:18:14Z

Added review comments except the unresolved open discussions.

keeping or disabling caching
+3 to store lookup key in cache
incorrect exported functions/types

will add changes for these once I have more information

feat: added sigv4 filter

1721d33

Signed-off-by: Anurag252 <Anuragsgsits@gmail.com>

szuecs added the minor no risk changes, for example new filters label May 7, 2024

github-advanced-security bot found potential problems May 7, 2024

View reviewed changes

Anurag252 added 4 commits May 7, 2024 23:36

refactor: moved creds to headers

1873c8b

Signed-off-by: Anurag252 <Anuragsgsits@gmail.com>

refactor: dir structure changes and function signature

9824f89

Signed-off-by: Anurag252 <Anuragsgsits@gmail.com>

chore: error handling

850eeba

Signed-off-by: Anurag252 <Anuragsgsits@gmail.com>

chore: cleanup and removal of host headers

851a300

Signed-off-by: Anurag252 <Anuragsgsits@gmail.com>

Anurag252 marked this pull request as ready for review May 9, 2024 09:23

AlexanderYastrebov reviewed May 13, 2024

View reviewed changes

Anurag252 added 3 commits May 24, 2024 21:51

chore: review comments

752b4ae

Signed-off-by: Anurag252 <Anuragsgsits@gmail.com>

docs: updated docs

ea94e5e

Signed-off-by: Anurag252 <Anuragsgsits@gmail.com>

chore: review comments

056e8c3

Signed-off-by: Anurag252 <Anuragsgsits@gmail.com>

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

feat: added sigv4 filter #3070

feat: added sigv4 filter #3070

Anurag252 commented May 7, 2024 •

edited

Anurag252 May 9, 2024

AlexanderYastrebov left a comment

AlexanderYastrebov May 13, 2024

Anurag252 May 14, 2024

AlexanderYastrebov May 13, 2024

Anurag252 May 14, 2024 •

edited

Anurag252 May 14, 2024 •

edited

AlexanderYastrebov May 13, 2024

Anurag252 May 24, 2024

szuecs commented May 14, 2024

Anurag252 commented May 28, 2024

feat: added sigv4 filter #3070

Are you sure you want to change the base?

feat: added sigv4 filter #3070

Conversation

Anurag252 commented May 7, 2024 • edited

Example 1 with default body {}

actual sdk code

skipper

curl -

Example 2 with non-default body

skipper

curl

actual sdk code

Anurag252 May 9, 2024

Choose a reason for hiding this comment

AlexanderYastrebov left a comment

Choose a reason for hiding this comment

AlexanderYastrebov May 13, 2024

Choose a reason for hiding this comment

Anurag252 May 14, 2024

Choose a reason for hiding this comment

AlexanderYastrebov May 13, 2024

Choose a reason for hiding this comment

Anurag252 May 14, 2024 • edited

Choose a reason for hiding this comment

Anurag252 May 14, 2024 • edited

Choose a reason for hiding this comment

AlexanderYastrebov May 13, 2024

Choose a reason for hiding this comment

Anurag252 May 24, 2024

Choose a reason for hiding this comment

szuecs commented May 14, 2024

Anurag252 commented May 28, 2024

Anurag252 commented May 7, 2024 •

edited

Anurag252 May 14, 2024 •

edited

Anurag252 May 14, 2024 •

edited