Determine how to use jacoco plugin to get coverage for integration tests
#32
Comments
|
Removing from |
|
Hi - could you provide info on how to add jacoco code coverage to unit tests? I've tried but no success. Thanks. |
|
Hey @markguckian can you add what you have tried here? It might help find the right path to get this figured out. I'd like to get this working using Gradle's built in JaCoCo support, but I haven't figured it out yet. I think part of the issue is that general usage with JaCoCo/Gradle is that the input is compiled code and exec data and it will generate reports for that. With Jenkins Pipelines, the input is source code that gets compiled in a different fashion, so the out-of-the-box usage of JaCoCo does not work. I just get 0 coverage for the integration tests. Here is what I tried in my example project at commit import com.gradle.scan.plugin.BuildScanPlugin
import org.gradle.kotlin.dsl.version
import java.io.ByteArrayOutputStream
plugins {
`jacoco`
id("com.gradle.build-scan") version "1.11"
id("com.mkobit.jenkins.pipelines.shared-library") version "0.4.0"
id("com.github.ben-manes.versions") version "0.17.0"
}
jacoco {
toolVersion = "0.8.0"
}
val commitSha: String by lazy {
ByteArrayOutputStream().use {
project.exec {
commandLine("git", "rev-parse", "HEAD")
standardOutput = it
}
it.toString(Charsets.UTF_8.name()).trim()
}
}
buildScan {
setLicenseAgree("yes")
setLicenseAgreementUrl("https://gradle.com/terms-of-service")
link("GitHub", "https://github.com/mkobit/jenkins-pipeline-shared-library-example")
value("Revision", commitSha)
}
tasks {
"wrapper"(Wrapper::class) {
gradleVersion = "4.4.1"
}
"jacocoIntegrationTestReport"(JacocoReport::class) {
reports {
csv.setEnabled(true)
xml.setEnabled(true)
}
executionData(tasks["integrationTest"])
val main by java.sourceSets
sourceSets(main)
}
}
java {
sourceCompatibility = JavaVersion.VERSION_1_8
}
dependencies {
testImplementation("org.spockframework:spock-core:1.1-groovy-2.4")
testImplementation("org.assertj:assertj-core:3.9.0")
}
sharedLibrary {
coreVersion = "2.95"
groovyVersion = "2.4.12"
pluginDependencies(Action {
dependency("org.jenkinsci.plugins", "pipeline-model-api", "1.2.5")
dependency("org.jenkinsci.plugins", "pipeline-model-declarative-agent", "1.1.1")
dependency("org.jenkinsci.plugins", "pipeline-model-definition", "1.2.5")
dependency("org.jenkinsci.plugins", "pipeline-model-extensions", "1.2.5")
})
}Running the
I think the first step is to figure out what kind of instrumentation data is output by JaCoCo and then see if that can be mapped to the existing Gradle task. If you have any other suggestions please let me now! |
|
Another possibility is the forked |
|
I just got *.exec files created but no reports. (Only just started looking at Jenkins pipeline testing/gradle so excuse my lack of knowledge!) |
|
Don't worry about it @markguckian , glad to have some more eyes on this. Questions will hopefully help drive some better documentation. In the example above that I tried I did create the report task with Changing the above example from: val main by java.sourceSets
sourceSets(main)to val main by java.sourceSets
val integrationTest by java.sourceSets
sourceSets(main)And the report is:
Which shows that there is at least sort of coverage for some classes. My belief here is that JaCoCo isn't mapping the runtime compiled Groovy classes to the source code, either because Jenkins is doing something different or the interaction with Jacoco isn't right. I think some of what needs to happen for me or whoever wants to investigate this is:
I think the cloning https://github.com/mkobit/jenkins-pipeline-shared-library-example would be the best place to start. Any help here would be greatly appreciated! |
mkobit
changed the title
jacoco plugin to get coveragejacoco plugin to get coverage for integration tests
|
Some further investigation:
But, I don't really know if that is useful. There are multiple |
|
I think that is useful, and I think I'm getting closer. Adding in class dumping: tasks {
"wrapper"(Wrapper::class) {
gradleVersion = "4.4.1"
}
"integrationTest" {
extensions.findByType(JacocoTaskExtension::class.java)?.apply {
classDumpDir = file("$buildDir/jacoco/classpathdumps")
}
}
"jacocoIntegrationTestReport"(JacocoReport::class) {
reports {
csv.setEnabled(true)
xml.setEnabled(true)
}
executionData(tasks["integrationTest"])
val main by java.sourceSets
sourceDirectories = main.allSource
classDirectories = fileTree("$buildDir/jacoco/classpathdumps")
}
}And running Looking at the dumped classes, there is a So, I think we need some way to "merge" those classes. If I delete the duplicates of each one change the tasks {
"wrapper"(Wrapper::class) {
gradleVersion = "4.4.1"
}
"integrationTest" {
extensions.findByType(JacocoTaskExtension::class.java)?.apply {
classDumpDir = file("$buildDir/jacoco/classpathdumps")
}
}
"jacocoIntegrationTestReport"(JacocoReport::class) {
reports {
csv.isEnabled = true
xml.isEnabled = true
}
executionData(tasks["integrationTest"])
val main by java.sourceSets
sourceDirectories = main.allSource
classDirectories = fileTree("$buildDir/jacoco/classpathdumps").apply {
include("evenOrOdd.*")
include("doStuff.*")
include("**/*ExampleSrc.*")
}
}
}I finally get some coverage data!
I think I'm really close to cracking the shell here. I still need to figure out how to handle the duplicate classes and what not. I don't think http://www.eclemma.org/jacoco/trunk/doc/offline.html will help, because these are compiled by Jenkins itself. |
|
Very promising, thanks for the hard work @mkobit |
|
I opened a question on the forum at https://groups.google.com/forum/#!msg/jacoco/-zTg9Bt5964/dUliK8OsAwAJ |
|
After getting some feedback on the forum post, I think that some work would need to be done to support getting coverage of library code. Here is a summary of what I think would need to happen based on the information I have right now:
I think the implementation, as based on the suggestion from Marc Hoffman, will have to implement coverage merging using the JaCoCo APIs. Cloning the branch at https://github.com/mkobit/jenkins-pipeline-shared-library-example/tree/mk/coverage-investigation , running This is a bit more work than I originally hoped for with this issue, so I don't think I'll get to it anytime soon. Side Note: OpenClover might be another option (possibly attempted with https://github.com/bmuschko/gradle-clover-plugin) but I think the same problems will be encountered. |
|
I have some library functions in test {
systemProperty "pipeline.stack.write", project.getProperty("pipeline.stack.write")
useJUnitPlatform()
testLogging {
events "passed", "skipped", "failed"
}
jacoco {
classDumpDir = file("$buildDir/jacoco/classpathdumps")
}
finalizedBy jacocoTestReport
}
jacocoTestReport {
afterEvaluate {
classDirectories.from = fileTree(dir: "$buildDir/jacoco/classpathdumps",
includes: [
'helm.*',
'aws.*'
]
)
}
reports {
csv.enabled false
xml {
enabled true
destination file("${buildDir}/test-results/jacoco.xml")
}
html.destination file("${buildDir}/reports/jacoco")
}
}(notice that the list of classes is filtered with This might help others get a bit further with some tweaking. |
|
@js-timbirkett, can you please give full example of your Appreciate your help! |
See mkobit/jenkins-pipeline-shared-libraries-gradle-plugin#32 and jenkinsci/JenkinsPipelineUnit#103. Signed-off-by: dblock <dblock@amazon.com>
See mkobit/jenkins-pipeline-shared-libraries-gradle-plugin#32 and jenkinsci/JenkinsPipelineUnit#103. Signed-off-by: dblock <dblock@amazon.com>
* Added test coverage for Groovy code. Signed-off-by: dblock <dblock@amazon.com> * Enable coverage for JenkinsPipelineUnit tests. See mkobit/jenkins-pipeline-shared-libraries-gradle-plugin#32 and jenkinsci/JenkinsPipelineUnit#103. Signed-off-by: dblock <dblock@amazon.com> * Remove base Manifest class that generates duplicate execution dump. Signed-off-by: dblock <dblock@amazon.com>
|
@mkobit based in your research and comments I extended it to something that works really good for me. I got around the duplicated classes error by creating several Jacoco xml reports and then merging them. For this SonarQube was really useful since it accepts several Jacoco xml reports. See jenkinsci/JenkinsPipelineUnit#103 (comment) for more information |
For unit tests it is straight forward and just works out of the box with the JaCoCo plugin (
id 'jacoco'). Integration tests are more likely to be used to flex the execution, so coverage is important to have.The text was updated successfully, but these errors were encountered: