provide for cargo afl, but it is not complete. #352

xizheyin · 2024-03-07T15:35:06Z

Hi! I've added initial support for cargo afl, but it's not perfect. I'm not quite sure what the norms are for contributing code, and to what extent it can be merged.

Running cargo llvm-cov afl will run afl for 10 seconds (I'll modify it later to let the user set their own parameters for time, etc.) and get a report.

Signed-off-by: xizheyin <xizheyin@smail.nju.edu.cn>

taiki-e

Thanks for the PR!

I would like to use as criteria for merging this whether or not it works in my other projects that use cargo afl.

Currently it uses cargo afl with the following steps:

git clone https://github.com/taiki-e/parse-changelog
cd parse-changelog/fuzz
cargo afl build --release --features afl
cargo afl fuzz -i seeds/parse -o out/parse target/release/parse

If using llvm-cov would it be something like this?

# in fuzz directory
cargo llvm-cov afl --release --features afl -i seeds/parse -o out/parse -V 10 target/release/parse

Or exactly the same as cargo afl?

# in fuzz directory
cargo llvm-cov afl build --release --features afl
cargo llvm-cov afl fuzz -i seeds/parse -o out/parse -V 10 target/release/parse

taiki-e · 2024-03-09T03:10:23Z

src/main.rs

+        // prepare seeds
+        let seeds = vec!["example input 1", "sample data 2", "test case 3"];
+
+        for (index, seed) in seeds.into_iter().enumerate() {
+            let file_path = format!("{input_dir}seed_{index}.txt");
+            let path = Path::new(&file_path);
+            let mut file = fs::File::create(path)?;
+            file.write_all(seed.as_bytes())?;
+        }


I beliave these should not be defined on our end, but should respect what the user passes as arguments.

yeah, I would fix it,

taiki-e · 2024-03-09T03:16:59Z

src/main.rs

+        // set afl's loop count to 1000 to avoid AFL loop infinitely!
+        // AFL maybe exits a process several minutes later and produce a cov file.
+        cargo.set("AFL_FUZZER_LOOPCOUNT", "20")?;
+        cargo.arg("afl").arg("fuzz");
+        cargo.arg("-V").arg("10");
+        // set in and out
+        cargo.arg("-i").arg(input_dir);
+        cargo.arg("-o").arg(output_dir);


These probably have the same problem. Even if we define our own defaults, we need to respect them if they are specified by the user. Personally, I would prefer to report an error if an environment variable or argument is not set than to define our own on our end.

taiki-e · 2024-03-09T03:18:34Z

src/main.rs

+            cargo.arg("afl").arg("build");
+            cargo.arg("--target-dir").arg(cx.ws.target_dir.as_str());


This needs to propagate flags passed by the user, such as releases. We probably need to create a new function like test_or_run_args in cargo.rs.

taiki-e · 2024-03-09T03:48:33Z

cc @smoelius: Do you have any recommendations or opinions on the interface or behavior here? I guess you are the best person to ask about this because you are a cargo-afl maintainer and cargo-llvm-cov contributor:)

smoelius · 2024-03-09T10:33:12Z

cc @smoelius: Do you have any recommendations or opinions on the interface or behavior here? I guess you are the best person to ask about this because you are a cargo-afl maintainer and cargo-llvm-cov contributor:)

It's very kind of you to ask. :)

I think your proposed interface makes a lot of sense.

One thing I note about the current PR is that it contains code one would expect to find in cargo-afl, e.g., docs/cargo-llvm-cov-afl.txt.

FWIW, cargo afl subcommand will run cargo subcommand with AFL-instrumented binaries. I'm not sure if this makes your task easier, @xizheyin.

As an aside, someone recently asked for coverage on the afl.rs repo: rust-fuzz/afl.rs#465

xizheyin · 2024-03-09T15:18:26Z

Thanks for the advice, I'm working on my research project which requires getting afl coverage, due to time I'll finish my experiments before I refine the cargo llvm-cov afl, good luck!
By the way, how do I get the coverage of a library? Is there a suitable args? For example, my fuzz target uses url inside and I want to get the coverage of url. Until now, I could look at each file with llvm-cov report target/debug/coverage_test -instr-profile=target/coverage_test.profdata -Xdemangler=rustfilt, however, I am not sure about how to view this library as a whole.

xizheyin · 2024-03-14T07:24:17Z

@taiki-e
Sorry to interrupt again. If I have 10 different programs that all use a certain library and I have access to their profraws, is it possible to easily count the total coverage of this library by these ten programs?

taiki-e · 2024-03-17T17:39:17Z

Sorry to interrupt again. If I have 10 different programs that all use a certain library and I have access to their profraws, is it possible to easily count the total coverage of this library by these ten programs?

If it is possible to (temporarily) combine the crates to be tested into a single workspace to run tests, perhaps it would be easy to do by combining with --dep-coverage.

If not, it would be difficult with the current implementation since only artifacts related to the current workspace would be collected, but I think that could be handled by adjusting object_files (probably just making this if branch unconditional -- of course that is not a reasonable default behavior although), specifying the same target directory by CARGO_TARGET_DIR, and using --no-report + report pattern.

In any case, could you please open a separate issue for questions not directly related to this PR?

xizheyin · 2024-03-20T07:11:43Z

Thanks, I will open a issue if I have any question.

njelich · 2024-04-10T09:45:39Z

It seems sufficient to just add an "external tests" guide, but for AFL - this seems to work with the tutorial fuzzer in the AFL docs:

source <(cargo llvm-cov show-env --export-prefix)
cargo llvm-cov clean --workspace
cargo afl build
AFL_FUZZER_LOOPCOUNT=20 cargo afl fuzz -c - -V 10 -i in -o out target/debug/url-fuzz-target
cargo llvm-cov report --html

And it allows full customization without overhauling the enitrety of the llvm-cov args system.

xizheyin added 4 commits March 7, 2024 23:29

provide for cargo afl, but it is not complete.

e04ecef

Signed-off-by: xizheyin <xizheyin@smail.nju.edu.cn>

fix format issues

2b598e3

Signed-off-by: xizheyin <xizheyin@smail.nju.edu.cn>

fix format issues

4e491b0

Signed-off-by: xizheyin <xizheyin@smail.nju.edu.cn>

fix clippy issues

b5aec5b

Signed-off-by: xizheyin <xizheyin@smail.nju.edu.cn>

taiki-e requested changes Mar 9, 2024

View reviewed changes

Merge branch 'main' into cargo-afl

ba17458

This comment was marked as off-topic.

Sign in to view

smoelius mentioned this pull request Apr 11, 2024

Is there a way to see the code coverage? rust-fuzz/afl.rs#465

Open

njelich mentioned this pull request May 3, 2024

Add AFL coverage instructions #369

Open

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

provide for cargo afl, but it is not complete. #352

provide for cargo afl, but it is not complete. #352

xizheyin commented Mar 7, 2024

taiki-e left a comment •

edited

taiki-e Mar 9, 2024

xizheyin Mar 9, 2024

taiki-e Mar 9, 2024

taiki-e Mar 9, 2024

taiki-e commented Mar 9, 2024

smoelius commented Mar 9, 2024

xizheyin commented Mar 9, 2024

This comment was marked as off-topic.

This comment was marked as off-topic.

This comment was marked as off-topic.

xizheyin commented Mar 14, 2024

taiki-e commented Mar 17, 2024

xizheyin commented Mar 20, 2024

njelich commented Apr 10, 2024

		cargo.arg("afl").arg("build");
		cargo.arg("--target-dir").arg(cx.ws.target_dir.as_str());

provide for cargo afl, but it is not complete. #352

Are you sure you want to change the base?

provide for cargo afl, but it is not complete. #352

Conversation

xizheyin commented Mar 7, 2024

taiki-e left a comment • edited

Choose a reason for hiding this comment

taiki-e Mar 9, 2024

Choose a reason for hiding this comment

xizheyin Mar 9, 2024

Choose a reason for hiding this comment

taiki-e Mar 9, 2024

Choose a reason for hiding this comment

taiki-e Mar 9, 2024

Choose a reason for hiding this comment

taiki-e commented Mar 9, 2024

smoelius commented Mar 9, 2024

xizheyin commented Mar 9, 2024

This comment was marked as off-topic.

This comment was marked as off-topic.

This comment was marked as off-topic.

xizheyin commented Mar 14, 2024

taiki-e commented Mar 17, 2024

xizheyin commented Mar 20, 2024

njelich commented Apr 10, 2024

taiki-e left a comment •

edited