[Feature Request]: Introduce support for code coverage & enforcement (GSoC'24 4.1)

[BenHenning]

Is your feature request related to a problem? Please describe.

The team cannot currently monitor and enforce minimum code coverage (as a proxy for functional behavioral coverage) in automated tests.

Describe the solution you'd like

Suggested milestones

• Milestone 1: Introduce a new script to compute a per-unit code coverage percentage for a single file.

  • Deliverable 1: Introduce a new scripting utility that can run code coverage for a specific Bazel test target, interpret the results, and return a proto for data processing.
  • Deliverable 2: Introduce a new utility which can, given the proto from the deliverable 1 utility, generate a rich-text code coverage report in one of three formats:
    1. Markdown (for easy copying to GitHub)
    2. HTML (for easy local viewing)
  • Deliverable 3: Update the test exemption check script & its exemption format such that each file now has two possible states:
    1. Exempt from having a test file (the current exemption behavior).
    2. An override of the code coverage to meet a specific minimum (which can be 0%).

• Milestone 2: Integrate code coverage checking.

  • Deliverable 1: Introduce a new script that uses the utility from deliverable 1 to perform code coverage analysis for a single target and the utility from deliverable 2 to optionally generate and output a rich-text report file for the test run. This script should always output the computed code coverage for a given file, and have a configuration option to fail if it's below a specified threshold.
  • Deliverable 2: Introduce a new CI workflow which, similarly to the existing unit test workflow, runs a series of buckets for code coverage analysis using the new script from milestone 1. Report generation should be enabled & reports uploaded as artifacts. Code coverage should only run after tests are passing (on a per-bucket basis if possible).
  • Deliverable 3: Fix/replace the cancellation workflow to ensure re-runs of CI correctly and quickly terminate all existing workflows that are running.
  • Deliverable 4: Introduce a wiki page explaining how to use the code coverage tool, provide advice on how to write tests with good behavioral coverage, and explain the limitations of the code coverage tool (i.e. all the cases it does not correctly count coverage for a specific line).
  • Deliverable 5: File issues for all cases where the code coverage tool misses or incorrectly counts code coverage for future work.

Technical hints / guidance

Top-level components needed for code-coverage support

• coverage.proto: Describes the new scripting data structures needed for code coverage.
• CoverageRunner.kt: A new scripting utility for running code coverage for a selected target.
• CoverageReporter.kt: A new scripting utility for generating an HTML report of code coverage.
• RunCoverage.kt: A new script for running code coverage locally.
• scripts/src/java/org/oppia/android/scripts/testfile/TestFileCheck.kt: Existing utility that needs to be updated to include support for code coverage enforcement.
• scripts/src/java/org/oppia/android/scripts/proto/script_exemptions.proto: Existing exemptions definition that needs to be updated per the TestFileCheck changes.
• .github/workflows/unit_tests.yml: Existing CI workflow that needs to be updated in order to support code coverage.
• .github/workflows/workflow_canceller.yml: Existing CI workflow that needs to be fixed.
• (Other files for wiki changes).

Some key technical notes

• This spec overview does not include details for the Yaml or wiki changes.
• The test file exemptions textproto file will need to be rebuilt. The simplest way to do this would be to update TestFileCheck to include a text proto regenerate argument (as used in other scripts).
• workflow_canceller.yml changes may required manually introducing support for workflow cancellation, or finding an available open source utility that works correctly for dynamic matrix jobs.
• unit_tests.yml will probably be extended to introduce a new job that blocks on the test run (in the same way as check_tests) except it (note that this may require a new script not spec'ed out in this overview):
  • 1. Intreprets the same matrix data as the main test run but omits the failing tests using the outputs from the test runs themselves.
  • 2. Starts a new test matrix that runs the RunCoverage.kt script for each affected file corresponding to each affected test. For example, if StateFragmentTest.kt is one of the affected tests, then RunCoverage would be run for StateFragment.kt (which would then correspond to StateFragmentTest per its behavior).
  • Note that the min enforced coverage for now should be something very modest, like 10%. This number will be increased in future changes outside the scope of this project (as the number may not be believable until we understand the limitations of what can be covered by JaCoCo).
• BazelClient.kt may need to be updated to include a function for running coverage on a specific target & output the standard output lines from that (which can then be parsed to get the report to generate the CoverageReport proto object).
• All new components require new corresponding test files. All updated components will need their tests updated based on the changes to those components.

Suggested files to add/change

coverage.proto:

message CoverageReport {
  string bazel_test_target = 1; // The Bazel test target that was run.
  repeated CoveredFile covered_file = 2; // Files with some coverage in this report.
}

message CoveredFile {
  string file_path = 1; // Relative to the project root.
  string file_sha1_hash = 2; // SHA-1 hash of the file content at the time of report (to guard against changes).
  repeated CoveredLine covered_line = 3; // Lines of code covered in the report.
}

message CoveredLine {
  int32 line_number = 1; // 0-starting line number of the covered line.
  Coverage coverage = 2; // Detected coverage.
  
  enum Coverage {
    FULL, // This line was fully covered by the test.
    PARTIAL, // This line was partially covered by the test, indicating some branches not being covered.
    NONE // This line was not executed during the test.
  }
}

CoverageReporter.kt:

class CoverageReporter {
  fun generateRichTextReport(report: CoverageReport, format: ReportFormat): String
  fun computeCoverageRatio(file: String, report: CoverageReport): Float // Returns in [0, 1].
  
  enum class ReportFormat {
    MARKDOWN,
    HTML
  }
}

CoverageRunner.kt:

class CoverageRunner {
  // Uses bazel coverage to run code coverage on the specified test target & interprets the results to generate and return a CoverageReport.
  fun runWithCoverageAsync(bazelTestTarget: String): Deferred<CoverageReport>
}

RunCoverage.kt:

// Usage:
//   bazel run //scripts:run_coverage -- <repo_root> <relative_test_file> <report_output_dir> [min_coverage=<int_perc>] [format=markdown,html]
//
// Examples:
//   bazel run //scripts:run_coverage -- $(pwd) utility/src/main/java/org/oppia/android/util/math/MathTokenizer.kt $(pwd)/reports
//   bazel run //scripts:run_coverage -- $(pwd) utility/src/main/java/org/oppia/android/util/math/MathTokenizer.kt $(pwd)/reports min_coverage=20
//   bazel run //scripts:run_coverage -- $(pwd) utility/src/main/java/org/oppia/android/util/math/MathTokenizer.kt $(pwd)/reports format=markdown
//   bazel run //scripts:run_coverage -- $(pwd) utility/src/main/java/org/oppia/android/util/math/MathTokenizer.kt $(pwd)/reports format=html
//   bazel run //scripts:run_coverage -- $(pwd) utility/src/main/java/org/oppia/android/util/math/MathTokenizer.kt $(pwd)/reports min_coverage=20 format=markdown,html
//
// The script:
// 1. Finds the corresponding test file specific to the file to be checked.
//   a. If the file has no test & isn't exempted, the script should fail.
//   b. If the file has no test & is exempted, the script should output as such.
// 2. Uses CoverageRunner to run the corresponding test file to generate a report.
// 3. Outputs code coverage percentage based on the report.
//   a. If the file has a code coverage percentage exemption, it's also outputted at this point.
// 4. If formats are specified, CoverageReporter should be used to generate reports for each specified format in the destination report_output_dir directory.
// 5. If min_coverage is specified, compares the computed code coverage with the minimum specified. The script should fail iff the file has less than the minimum specified coverage.
fun main(vararg args: String)

class RunCoverage {
  fun runCoverage(targetFile: String, outputFormats: List<CoverageReporter.ReportFormat>)
}

script_exemptions.proto:

// The old TestFileExemptions proto (for reference).
message TestFileExemptionsOld {
  repeated string exempted_file_path = 1;
}

message TestFileExemptions {
  repeated TestFileExemption test_file_exemption = 1;
  
  message TestFileExemption {
    string exempted_file_path = 1;
    oneof exemption_type {
      bool test_file_not_required = 2;
      int32 override_min_coverage_percent_required = 3;
    }
  }
}

Describe alternatives you've considered

No response

Additional context

This is the high-level tracking issue corresponding to https://github.com/oppia/oppia/wiki/Google-Summer-of-Code-2024#41-code-coverage-support-and-enforcement.

Note that this project includes work that relate to the following issues: #1497, #1726, #1727, and #1728.

[BenHenning]

NB: This issue is locked to ensure that it only contains updates for requirements. If you have questions regarding this project, please use its corresponding discussion category: https://github.com/oppia/oppia-android/discussions/categories/gsoc-q-a-4-1-code-coverage-support.

[BenHenning]

NB: The main issue comment has been updated to include the technical details directly rather than a link to a Gist (so that we can easily change this if needed). None of the technical requirements have actually changed.