StrictHttpFirewall rejects request when HtmlUnit WebClient is called with encoded URL

[thuri]

Affects: 5.3.13

---

I'm trying to run a unit test on my spring boot project using the HtmlUnit Webclient.
The test does a POST Request submitting form data to the Controller which will create a database entry and send a redirect to the client. The response will contain a URL in the Location header which will be encoded if necessary.

Everything works fine until the HtmlUnit Webclient tries to follow the redirect and the org.springframework.security.web.firewall.StrictHttpFirewall.getFirewalledRequest(HttpServletRequest) method is called which prevents the request from being processed. It complains about the % in the URL (see stack trace).

But when running the application and using an actual browser the problem does not occur. Searching the web I found #16067 which first looked familiar.

So I wrote a small project to reproduce the issue, and I think that the problem with HtmlUnit WebClient is a little bit different.

When running the webClientTestStringWithEncoding test method and debugging into org.springframework.security.web.firewall.StrictHttpFirewall#rejectedBlocklistedUrls(HttpServletRequest), one can see that request.getServletPath() still contains the encoded path while the called method decodedUrlContains seems to expect that it has been decoded.

If you run the mockMvcTestURI test method and debug the same line you can see that request.getServletPath() is empty but request.getPathInfo() contains an decoded path. request.getRequestURI() contains the encoded path in both cases.

In case you're wondering why the first test uses the webclient and the other one the mockMvc: I didn't find a way to pass either the encoded URL or the unencoded version (with spaces and an Umlaut) to the HtmlUnit Webclient (as you can see by the other test methods)

Perhaps the problem lies in the org.springframework.test.web.servlet.htmlunit.HtmlUnitRequestBuilder.buildRequest(ServletContext) where the servletPath is set on the request and should be decoded.

I think I can work around that issue in my test but would be happy if you could have a look at this.

Thanks in advance,
Michael

java.io.IOException: org.springframework.security.web.firewall.RequestRejectedException: The request was rejected because the URL contained a potentially malicious String "%"
	at org.springframework.test.web.servlet.htmlunit.MockMvcWebConnection.getResponse(MockMvcWebConnection.java:152)
	at org.springframework.test.web.servlet.htmlunit.MockMvcWebConnection.getResponse(MockMvcWebConnection.java:134)
	at org.springframework.test.web.servlet.htmlunit.DelegatingWebConnection.getResponse(DelegatingWebConnection.java:79)
	at com.gargoylesoftware.htmlunit.WebClient.loadWebResponseFromWebConnection(WebClient.java:1596)
	at com.gargoylesoftware.htmlunit.WebClient.loadWebResponse(WebClient.java:1518)
	at com.gargoylesoftware.htmlunit.WebClient.getPage(WebClient.java:493)
	at com.gargoylesoftware.htmlunit.WebClient.getPage(WebClient.java:413)
	at com.gargoylesoftware.htmlunit.WebClient.getPage(WebClient.java:548)
	at com.gargoylesoftware.htmlunit.WebClient.getPage(WebClient.java:529)
	at io.gitlab.thuri.spring.htmlunit.TestWithEncodedUriIssue.webClientTestStringWithEncoding(TestWithEncodedUriIssue.java:63)
	// cut junit and eclipse stacktrace entries
Caused by: org.springframework.security.web.firewall.RequestRejectedException: The request was rejected because the URL contained a potentially malicious String "%"
	at org.springframework.security.web.firewall.StrictHttpFirewall.rejectedBlocklistedUrls(StrictHttpFirewall.java:463)
	at org.springframework.security.web.firewall.StrictHttpFirewall.getFirewalledRequest(StrictHttpFirewall.java:429)
	at org.springframework.security.web.FilterChainProxy.doFilterInternal(FilterChainProxy.java:196)
	at org.springframework.security.web.FilterChainProxy.doFilter(FilterChainProxy.java:183)
	at org.springframework.web.filter.DelegatingFilterProxy.invokeDelegate(DelegatingFilterProxy.java:358)
	at org.springframework.web.filter.DelegatingFilterProxy.doFilter(DelegatingFilterProxy.java:271)
	at org.springframework.mock.web.MockFilterChain.doFilter(MockFilterChain.java:134)
	at org.springframework.web.filter.RequestContextFilter.doFilterInternal(RequestContextFilter.java:100)
	at org.springframework.web.filter.OncePerRequestFilter.doFilter(OncePerRequestFilter.java:119)
	at org.springframework.mock.web.MockFilterChain.doFilter(MockFilterChain.java:134)
	at org.springframework.web.filter.FormContentFilter.doFilterInternal(FormContentFilter.java:93)
	at org.springframework.web.filter.OncePerRequestFilter.doFilter(OncePerRequestFilter.java:119)
	at org.springframework.mock.web.MockFilterChain.doFilter(MockFilterChain.java:134)
	at org.springframework.web.filter.CharacterEncodingFilter.doFilterInternal(CharacterEncodingFilter.java:201)
	at org.springframework.web.filter.OncePerRequestFilter.doFilter(OncePerRequestFilter.java:119)
	at org.springframework.mock.web.MockFilterChain.doFilter(MockFilterChain.java:134)
	at org.springframework.test.web.servlet.MockMvc.perform(MockMvc.java:199)
	at org.springframework.test.web.servlet.htmlunit.MockMvcWebConnection.getResponse(MockMvcWebConnection.java:149)
	... 78 more

[rstoyanchev]

HtmlUnitRequestBuilder does not seem to support well URLs with encoded paths, including paths that are not encoded but become encoded when HtmlUnit's WebClient prepares a java.net.URL. The problem, as you pointed out, is that the servletPath is supposed to be decoded, but HtmlUnitRequestBuilder essentially uses the full path minus any contextPath, and without further decoding.

One reason this might not have failed as much is that Boot apps have alwaysUseFullPath set by default so effectively the servletPath is not used for mapping purposes, so only Spring Security's firewall is affected.

I think this can be corrected. I don't see another alternative, and hopefully it shouldn't cause issues since the servletPath is not often used for mapping purposes and in any case it should be decoded.