plan to submit a fuzz driver to OSS-Fuzz

[secsys-go]

We are developing a new technique to automatically transform your official unit tests into fuzz drivers. And our auto-generated fuzz driver successfully found your project's bugs #1967 #1968. Now we plan to submit this fuzz driver to OSS-Fuzz Project. We hope to obtain your approval. Thanks a lot.

The fuzz driver's source code is followed. We are appreciated if you could provide us with some suggestions to improve our fuzz driver's quality.

package iris_test

import (
         "testing"
         "github.com/kataras/iris/v12"
         "github.com/kataras/iris/v12/httptest"
)

func FuzzTestUseRouterParentDisallow(f *testing.F) {
	f.Add("no_userouter_allowed", "always", "_2", "_3", "/index", "/", "/user")
	f.Fuzz(func(t *testing.T, data1 string, data2 string, data3 string, data4 string, data5 string, data6 string, data7 string) {
			app := iris.New()
			app.UseRouter(func(ctx iris.Context) {
					ctx.WriteString(data2)
					ctx.Next()
			})
			app.Get(data5, func(ctx iris.Context) {
					ctx.WriteString(data1)
			})

			app.SetPartyMatcher(func(ctx iris.Context, p iris.Party) bool {
					// modifies the PartyMatcher to not match any UseRouter,
					// tests should receive the handlers response alone.
					return false
			})

			app.PartyFunc(data6, func(p iris.Party) { // it's the same instance of app.
					p.UseRouter(func(ctx iris.Context) {
							ctx.WriteString(data3)
							ctx.Next()
					})
					p.Get(data6, func(ctx iris.Context) {
							ctx.WriteString(data1)
					})
			})

			app.PartyFunc(data7, func(p iris.Party) {
					p.UseRouter(func(ctx iris.Context) {
							ctx.WriteString(data4)
							ctx.Next()
					})

					p.Get(data6, func(ctx iris.Context) {
							ctx.WriteString(data1)
					})
			})

			e := httptest.New(t, app)
			e.GET(data5)
			e.GET(data6)
			e.GET(data7)

	})
}

[kataras]

Hello @secsys-go,

Thanks for the update! I ran the example and it seems that it worked just fine, it returns:

no_userouter_allowed
no_userouter_allowed
no_userouter_allowed

I am wondering how did you know about this SetPartyMatcher method, it's a "new" one and not documented in our examples; just in test files. Good job :) Let me know if you need any further comment!

[secsys-go]

Hello @secsys-go,

Thanks for the update! I ran the example and it seems that it worked just fine, it returns:

no_userouter_allowed
no_userouter_allowed
no_userouter_allowed

We don't know how you ran this code. We used go test -fuzz to run this code and got the crash.

Here is the run command we successfully triggered the bug before:
go test github.com/kataras/iris/v12/core/router -fuzz ^\QFuzzTestUseRouterParentDisallow\E$ -run ^$$$$

However, to run this command, you need to modify package iris_test to package router_test and put this driver code into iris/core/router/. Then you need to cd to root directory of iris project and run this command. If you get crash, it will generate testdata/fuzz/FuzzTestUseRouterParentDisallow/ directory in the location of driver code, and in this directory, the inputs that let it crash will be stored here. Let me know if you encounter any troubles :)

I am wondering how did you know about this SetPartyMatcher method, it's a "new" one and not documented in our examples; just in test files. Good job :) Let me know if you need any further comment!

Actually we developed our fuzz driver by learning the unit tests, because we think unit tests represent a official usage of the APIs. We modifed the inputs of unit tests to fuzzed data, so we could obtain possible bugs.

[secsys-go]

We kindly recommend the adoption of fuzz drivers in this project to continuously enhance its robustness and reliability.

What is Fuzz Testing

Fuzz testing, also known as fuzzing, is a software testing technique that involves providing invalid, unexpected, or random data inputs to a program in order to discover vulnerabilities, bugs, or crashes. The goal of fuzz testing is to uncover errors or security flaws that may not be identified through traditional testing methods. To fuzz iris, we need to write a driver which carefully invokes iris's API and passes fuzz data to arguments.

Why We Need Fuzz Testing in iris

We have written a set of fuzz drivers to test iris's various APIs. These fuzz drivers have aided iris in discovering multiple bugs. Most importantly, some of fuzz drivers found bugs in old version, and meanwhile we discoverd other bugs in new version of iris using the same fuzz drivers again. It means that these fuzz drivers have the potential to continuously discover new bugs. The fuzz driver this issue represents is a good example. Last year we developed this fuzz driver and used it to successfully discover bug #1967 and #1968 . This time we fuzzed again using the same driver, and we found another bug #2215 . We believe our fuzz driver will probably find more bugs in the future, and we want to submit our fuzz driver to iris project in order to help iris find potential bugs.

Possible solutions

We suggest iris introduce fuzz test to continuously enhance its robustness and reliability. Now we would like to share our fuzz drivers with iris. Two possible ways could be considered.

1. We directly submit a PR to iris project.

2. If iris maintainers feel inconvenient to introduce fuzz driver code in the project now, we could also submit PR to OSS-Fuzz project which is widely used to continuously fuzz open source software and supported by Google. If iris maintainers consider this way is better, we kindly request supports from the maintainers including helping us review fuzz drivers code and giving us official approval of adding fuzz drivers of iris to OSS-Fuzz project.

[kataras]

Hello @secsys-go,

We don't know how you ran this code. We used go test -fuzz to run this code and got the crash.

Here is the run command we successfully triggered the bug before: go test github.com/kataras/iris/v12/core/router -fuzz ^\QFuzzTestUseRouterParentDisallow\E$ -run ^$$$$

However, to run this command, you need to modify package iris_test to package router_test and put this driver code into iris/core/router/. Then you need to cd to root directory of iris project and run this command. If you get crash, it will generate testdata/fuzz/FuzzTestUseRouterParentDisallow/ directory in the location of driver code, and in this directory, the inputs that let it crash will be stored here. Let me know if you encounter any troubles :)

I am wondering how did you know about this SetPartyMatcher method, it's a "new" one and not documented in our examples; just in test files. Good job :) Let me know if you need any further comment!

Actually we developed our fuzz driver by learning the unit tests, because we think unit tests represent a official usage of the APIs. We modifed the inputs of unit tests to fuzzed data, so we could obtain possible bugs.

Here's my full output:

PS C:\kataras\github\iris> go test github.com/kataras/iris/v12/core/router -fuzz ^\QFuzzTestUseRouterParentDisallow\E$ -run ^$$$$
fuzz: elapsed: 0s, gathering baseline coverage: 0/1 completed
fuzz: elapsed: 1s, gathering baseline coverage: 1/1 completed, now fuzzing with 20 workers
fuzz: elapsed: 3s, execs: 3448 (1149/sec), new interesting: 11 (total: 12)
fuzz: elapsed: 6s, execs: 18754 (5092/sec), new interesting: 37 (total: 38)
fuzz: elapsed: 9s, execs: 24549 (1933/sec), new interesting: 58 (total: 59)
fuzz: elapsed: 12s, execs: 32578 (2678/sec), new interesting: 78 (total: 79)
fuzz: elapsed: 15s, execs: 40607 (2678/sec), new interesting: 102 (total: 103)
fuzz: elapsed: 18s, execs: 52098 (3828/sec), new interesting: 124 (total: 125)
fuzz: elapsed: 21s, execs: 64060 (3987/sec), new interesting: 147 (total: 148)
fuzz: elapsed: 24s, execs: 78093 (4679/sec), new interesting: 171 (total: 172)
fuzz: elapsed: 27s, execs: 86414 (2770/sec), new interesting: 177 (total: 178)
fuzz: elapsed: 30s, execs: 93069 (2213/sec), new interesting: 185 (total: 186)
fuzz: elapsed: 33s, execs: 95871 (934/sec), new interesting: 189 (total: 190)
fuzz: elapsed: 34s, execs: 95871 (0/sec), new interesting: 189 (total: 190)
--- FAIL: FuzzTestUseRouterParentDisallow (33.76s)
    fuzzing process hung or terminated unexpectedly while minimizing: write |1: Insufficient system resources exist to complete the requested service.
    Failing input written to testdata\fuzz\FuzzTestUseRouterParentDisallow\b40be9ff2b906829
    To re-run:
    go test -run=FuzzTestUseRouterParentDisallow/b40be9ff2b906829
FAIL
exit status 1
FAIL    github.com/kataras/iris/v12/core/router 33.978s
PS C:\kataras\github\iris> cd core/router
PS C:\kataras\github\iris\core\router> go test -run=FuzzTestUseRouterParentDisallow/b40be9ff2b906829
PASS
ok      github.com/kataras/iris/v12/core/router 0.102s

Sometimes I feel so lonely here, you are open for any contribution, including direct PRs, this would be so great. And really it is my honor. I want to be honest, as always, the testing.F and -fuzz testing is something new for me, so I can't promise anything special, I would love to help you in Iris-specific work though.

[secsys-go]

Here's my full output:

Oh, we have already figured it out why you could not replicate our result. This is because you used the version that has fixed the bug in commit f955489. We found the bug in commit 24ba4e8. If you want to replicate bug #2215 , you need to roll back to 24ba4e8.

However, even in the current HEAD of main (commit 28f49cd), we also found somethings interesting.

Firstly, as described in your output, system resources ran out soon in fuzzing test. It is probably because some instances (maybe app or e) are not released or freed. If there are relevant methods to accomplish this, please let me know, and we will modify our driver code. But if there is no method to release these instances, we suggest maintainers develop them.

Secondly, we used the same fuzz driver to discover another possible bug in the version of current HEAD of main (commit 28f49cd), which was described in #2216 in detail.

Sometimes I feel so lonely here, you are open for any contribution, including direct PRs, this would be so great. And really it is my honor. I want to be honest, as always, the testing.F and -fuzz testing is something new for me, so I can't promise anything special, I would love to help you in Iris-specific work though.

Thank you for your trust. We have submit PR #2217 . We would also like to submit the fuzz driver to OSS-Fuzz, if it is what you are willing to do.