Improve plugin testing

[MarceloRuiz]

Expected Behavior

After using TestKit to build the test project, it's BuildResult object should have a reference to both the Project and Settings instances (maybe a read-only copies) to allow the user to query them regarding the expect outcome after applying the plugin under test. This would be something similar to what ProjectBuilder allows.

Current Behavior (optional)

No response

Context

TestKit is quite limited to provide useful information regarding the build, and the plugin testing experience is so limited that users decide to skip it altogether.
It is very time consuming to create tasks in the build and settings scripts to print some information to the console to later parse in order to verify expected effects of having a Plugin applied.

[ov7a]

We need more information to understand the request. Could you please provide more context about what you want to achieve?

If we receive a clear use case description, we may be able to suggest a workaround.

This issue will be closed after 7 days unless you provide more information.

---

As stated in the docs, you are supposed to test the logic by checking the build outputs (produced files and/or stdout/stderr). If you want to inspect the state of the project, you should stick with ProjectBuilder for now.

Can you describe in more detail what are you testing and why you have difficulties to do it by checking the build outputs, please?

[MarceloRuiz]

Thank you for getting back to me so quickly!

The most important part of this feature request is that there is currently no easy way to develop plugins doing TDD. TestKit is the approach Gradle recommends and like you pointed out the docs state it is only about functional testing, and they also say: "Over time, the TestKit will likely expand to facilitate other kinds of tests". This last part, in my opinion, is what is really needed.

Maybe I am completely wrong, but it is possible that a good portion of developers do not test their plugins and just hope for the best (a possible discussion topic for discuss.gradle.org that gradle should start with developers). The fact that there are so little good examples and tutorials regarding both developing plugins and testing them might be a good indication this is the case. Here is a good example (read the paragraph where tests are mentioned): tutorial about plugins

TestKit has limitations that force users to use hacks to test what they are interested on, and ProjectBuilder (although very intuitive to use with the help an any IDE) is also limited: you can't get a Settings object and it does not read build files to set the Project object. It is also not quite reliable for some use cases: here is an example of my most recent problem with ProjectBuilder

Ideally TestKit should let the user also control the build lifecycle: why do you need to get to the execution phase when you are interested in things that happen during the Initialization or Configuration phases complete? What if I need to be sure whatever I am doing only happens during the Configuration phase? What if I need to be sure a 3rd party plugin is doing what is supposed to do during a specific phase of the build?

Currently, if you have a Settings plugin and you do any type of customization, you are forced to create custom functions in order to print the information you are interested in to the console and then parse the output in the test. This is clearly a hack and not what testing should be about. The same goes for Project plugins.

Since plugins can manipulate Project and Settings objects in any way they want, it makes perfect sense to be able to retrieve those objects from TestKit before any task is executed: after all, those objects are created as part of the build process anyway. If a developer could alter the Settings and Project objects, what would be a good reason to prevent him/her to access those objects during testing?

The bottom line is that TestKit is not very user friendly to help test driving plugins, but it could be greatly improved potentially with the hopefully not-so-huge changes I am suggesting. If a developer could alter the Settings and Project objects, what would be a good reason to prevent him/her to access those objects during testing?

Again, thanks for taking the time to read an answer this topic!

[ov7a]

Thanks a lot for the reply! It has a lot of great points!

Indeed, our testing capabilities for plugins should be improved.

[MarceloRuiz]

Thank you for taking the time to review this. I think if we get a better testing framework we will make Gradle more accessible for everyone: testing is also a great tool to learn Gradle!