Suggest Java 21 in recommended Jenkinsfile

[basil]

Since Java 11 is likely to be EOL in October 2024, it might be a good time to start recommending Java 21 and Java 17 in the default test matrix rather than Java 17 and Java 11.

[basil]

As long as the majority of the top 200 plugins do not comply, this ticket remains open.

[jglick]

start recommending Java 21 and Java 17 in the default test matrix rather than Java 17 and Java 11

To centralize jenkinsci/workflow-api-plugin#299 (comment) + #651 (comment):

implementing a migration of this nature across the whole ecosystem involves [much] code change

Yes but we have a bot for this purpose, correct? (Though apparently not in the case of, e.g., jenkinsci/junit-attachments-plugin#99.) Maintainers can merge proposed PRs at their leisure. Anyway dropping 11 support from a plugin because core has dropped 11 support would be something you would do if and when you update jenkins.version past a certain point and pick up a plugin-pom update which drops 11 support, at least if we follow the model we used for Java 8.

it is unlikely for a bug to be present when running on 11 but not 17 or 21

That depends on the kind of regression. Using -release and running bytecode version checks we should be protected against accidentally using a 17+ library, or an API method added in 13, etc. However there are plenty of functional changes in Java that code may inadvertently start to rely upon, such as bugs fixed in newer versions but not backported to 11. Analogously, if a plugin advertises compatibility with core 2.361.x, we prefer to actually run tests against 2.361.x, not merely compile against it and then presume that test runs against 2.414.x will suffice (though of course we also want to run tests against newer deps, using PCT if nothing else).

At the very least, deliberately dropping Java 11 test coverage from most of the Jenkins ecosystem while continuing to notionally support it seems like a surprising decision that it would be best to communicate explicitly, for example in PRs proposing it.

[basil]

Yes but we have a bot for this purpose, correct?

Has automation helped us complete the HtmlUnit migration in jenkinsci/jenkins-test-harness#568? The presence or absence of automation is not the goal. The goal is completion of the migration for the majority of critical plugins within a reasonable amount of time.

Maintainers can merge proposed PRs at their leisure.

But the goal is completion of the migration for the majority of critical plugins within a reasonable amount of time. Relying on maintainers to merge PRs at their leisure does not achieve this goal.

That depends on the kind of regression.

No argument there, but I think this risk is tolerable. What is not tolerable for me is starting migrations without an intent to complete them for the majority of critical plugins within a reasonable amount of time.

[basil]

However there are plenty of functional changes in Java that code may inadvertently start to rely upon, such as bugs fixed in newer versions but not backported to 11. Analogously

I thought about this some more and think that this concern can be mitigated to a large degree by running PCT/ATH on Java 11 (LTS) and 21 (weekly). That would greatly decrease the risk described above while still allowing us to coalesce into one sweep the "add 21" and "remove 11" ecosystem migrations in plugin repositories.

[jtnord]

That depends on the kind of regression. Using -release and running bytecode version checks we should be protected against accidentally using a 17+ library, or an API method added in 13, etc. However there are plenty of functional changes in Java that code may inadvertently start to rely upon, such as bugs fixed in newer versions but not backported to 11. Analogously, if a plugin advertises compatibility with core 2.361.x, we prefer to actually run tests against 2.361.x, not merely compile against it and then presume that test runs against 2.414.x will suffice (though of course we also want to run tests against newer deps, using PCT if nothing else).

Adding to the mix that additionally IIUC as we not use the URLClassloader and not the ANTClassloader we support multi release libraries on our classpath. This means that it is the tests that detect the issue and not the compilation as it is not the bytecode of the plugin that is at all impacted, but use of external libraries.

[basil]

Again, this is not a question of whether or not there are any costs to not testing on Java 11. I already agreed that there are some costs. My point is that the costs of two separate migrations across critical plugins are far greater.

[jtnord]

cf: #651 (comment)

The previous comment appears to assume that you are doing that migration, however a plugin author can do that as and when their plugin bumps to a core requiring java 17 as a minimum. No staggering extra cost in incurred here, if an author is bumping core versions.

We have to stop assuming that plugin authors are going to do these migrations "someday." The fact is that they aren't, and these incomplete migrations are becoming massive technical debt that falls on whoever happens to be next working on some unrelated change. And it is turning people away from plugin development:

So that means what. if there are no migrations to a newer Jenkins version that needs java17 then what is the actual aim of doing this?

If a plugin is that critical and is not exercised by the ATH, or in the BOM then I have to question if it is that is the problem that needs to be solved.

My point is that the costs of two separate migrations across critical plugins are far greater.

And yet I am still missing the need for the first migration! it is certainly as clear as mud from this ticket

There is no need to remove java11 from the list in plugins that are using it today. tomorrow when they move to Jenkins x.abc that needs Java17 then they will have to "migrate" - but there is not an automated update Jenkins to this version migration anyway, and if you where thinking of doing it in the plugin-pom then Just bumping a Jenkins version does not actually solve any problems we have either - it does not remove use of deprecated APIs by replacing them with something else.

And by the above you are saying that they are not migrating - so that will never happen - so there is no second migration...

As long as the majority of the top 200 plugins do not comply, this ticket remains open.

This seems some arbitrary self imposed thing, without any reasoning as the the why here.
This ticket does not tell me what or why you are hoping to achieve this - merely restating "multiple migrations" - but as a plugin author I will say there was no need to migrate my Jenkinsfile and remove java11 to begin with.

The fact that you recommend this in the archetype I have zero issues with - the plugin is new, there is no contract that says the develoepr that runs the archetype has to accept what it produces without modification.

[jtnord]

BTW - adding java 21 and removing java 11 is being inconsistently applied in PRs

[basil]

The archetype represents our general template and codifies best practices for all Jenkins plugins. Differences from the archetype should be the exception rather than the rule and should represent unusual scenarios that justify an exception rather than a normative state of affairs. For example, there are known issues with Docker-based tests on Windows VMs that require an unusual Jenkinsfile; similarly, a particularly enthusiastic maintainer like Mark may choose to enable testing on additional lines (taking on the burden of keeping the Jenkinsfile up-to-date), and this is perfectly fine.

Plugins that begin to fall behind from the archetype, whether with regard to the plugin parent POM, plugin BOM, .github directory, or any other aspect of our best practices (such as stopping builds before test teardown to avoid directory deletion errors), require laborious effort to modernize. This unfinished work becomes technical debt throughout the ecosystem and interferes with unrelated projects, exploding the scope of otherwise simple projects from "implement an RFE" to "implement an RFE and deal with an unbounded number of unfinished and often poorly documented migrations started by others but never finished". Morever, when one party is obliged to complete a migration started by another party without any context (let alone consent), the task becomes much more difficult. Many people, both new and regular contributors alike, have given up rather than face these tasks.

For these reasons, it is important that those who begin migrations make a reasonable effort to complete them throughout a reasonable portion of the ecosystem within a reasonable amount of time. For migrations that affect end users, I think plugins with more than about 1,000 - 10,000 users should be covered. For migrations that only affect developers, I think the top 100-250 plugins should be covered. I think a reasonable upper bound on the amount of time for completing migrations is 6-9 months, depending on the task.

[basil]

I am not seeing any progress in the spreadsheet.

[MarkEWaite]

I've switched the plugins that I maintain so that they are only testing Java 21 and Java 17. I'm willing to accept the risk that Java 11 specific issues might arise.

The plugins passing their tests with Java 21 in the plugin bill of materials and in the acceptance test harness is great. I'd very much like to see Java 21 testing in more of the plugin infrastructure, even if it is purely the addition of Java 21 tests.

@jtnord since you're OK with updating the archetype to test Java 21 and not test Java 11, that seems like the change proposed in pull request #651 is a good one and can be used by plugin maintainers as they are ready to adopt. We'll need a new release of the plugin archetype. The 1.20 release was at the end of July and is still testing Java 11 and Java 17. Are you OK with a new release of the archetype?

@jglick, I believe that you have permission to release the archetype. Are you willing to do that release with the current configuration that tests with Java 17 and Java 21?

@basil the recommendation to reduce the differences between the archetype and implementations in plugins seems like a worthwhile step to take that won't be resolved by pull request #651 associated with this issue. If the maintainers of a specific plugin want to continue testing with Java 11, they can do that. If they want to reduce the differences between their plugin and the archetype, they can do that as well.

I'm willing to revise the plugins that I maintain so that they will use the Jenkinsfile from the archetype. Those plugins that benefit from the addition of the forkCount argument will continue to use it. Plugins that don't benefit from it will continue to not use the forkCount argument.

[jglick]

@jglick, I believe that you have permission to release the archetype. Are you willing to do that release

I can release whenever, though I think @NotMyFault also now has full release permissions.

[NotMyFault]

@jglick, I believe that you have permission to release the archetype. Are you willing to do that release

I can release whenever, though I think @NotMyFault also now has full release permissions.

Yeah, I've released a new version.

[MarkEWaite]

I'm willing to revise the plugins that I maintain so that they will use the Jenkinsfile from the archetype.

I've revised the plugins that I maintain to test Java 17 and Java 21 in the Jenkinsfile.

Java 21 has now been released by Oracle. I've confirmed that Java 21 builds Jenkins core and 15+ Jenkins plugins and passes tests in each of those cases. I'd like to make progress on testing Java 21 in the Jenkinsfile of each of the high use plugins that are identified in the spreadsheet.

@jglick are you willing to approve pull requests to plugins where you are a maintainer (like script-security) if they add Java 21 testing without removing Java 11? My stronger priority is to enable Java 21 testing for each plugin so that maintainers don't need to wait for the plugin bill of materials before they detect failures related to Java 21. That means that when Java 11 reaches end of life in October 2024, we'll need to sweep through each Jenkinsfile again to remove Java 11.

@jtnord same question for you. Are you willing to approve pull requests for plugins that you maintain so long as they only add Java 21 testing without removing Java 11?

If that's workable for the two of you, then I'll plan to submit pull requests to add Java 21 testing without removing Java 11 testing. I think that @NotMyFault will be willing to revise his pull requests to continue building Java 11.

[jtnord]

@jtnord same question for you. Are you willing to approve pull requests for plugins that you maintain so long as they only add Java 21 testing without removing Java 11?

I have no issues with that.

[jglick]

Happy to merge whatever you decide upon. (I am not the only maintainer in that case.)

[basil]

I see no reason why a maintainer should be pushing back on any PR that brings a plugin into conformance with the current archetype:

buildPlugin(
  useContainerAgent: true, // Set to `false` if you need to use Docker for containerized tests
  configurations: [
    [platform: 'linux', jdk: 21],
    [platform: 'windows', jdk: 17],
])

Although @jtnord expressed some concerns above, I responded to them point by point, and I received no further response from James. Therefore I have to assume that these concerns have been addressed and that we are to proceed making plugin Jenkinsfiles conform to the archetype. Am I missing something?

[jtnord]

Although @jtnord expressed some concerns above, I responded to them point by point, and I received no further response from James

I lost the will to get into an ongoing discussion that will not lead to any reasonable conclusion.
You and I fundamentally disagree on what the archetype is and represents and the benefit/trade-offs of updating plugins and removing the lower bound of supported version. I have accepted that, not everything has to end in an agreement.

Therefore I have to assume that these concerns have been addressed and that we are to proceed making plugin Jenkinsfiles conform to the archetype

as per my earlier response to Mark I did not agree where I am a maintainer that I would just approve these PRs that dropped Java11 though.

I am not preventing you or anyone else from making PRs to any plugin - including ones I maintainer - just setting the context for plugins that I maintain I would not be approving the PR as is. I may leave a suggestion in the PR that I review to add back java11 and then merge but I would not be approving something that removes Java 11 (fwiw I have already done this in a few plugins). This is my view as a maintainer - it is not a view that I am applying to the archetype or any other plugin in the ecosystem.

[basil]

While the project has a tradition of allowing maintainers to express strong disagreement (e.g., not all maintainers have accepted automated release), a maintainer who expresses strong disagreement is also obligated to do the work of implementing the alternative path. For example, maintainers who decline to accept automated release are obligated to do manual releases. Similarly, maintainers who decline to accept PRs implementing the Jenkinsfile recommended in the archetype are obligated to keep their Jenkinsfile up-to-date via other means. One cannot simultaneously express strong disagreement while not doing the work of implementing the alternative path — the right to express strong disagreement cannot negatively impact the work of others. For example, if I were to decline to accept automated release and decline to perform manual releases, I would be negatively impacting the work of others who need to get their changes released. But if I were to decline to accept automated release and accept to perform manual releases, I would not be negatively impacting the work of others.