Gradle module metadata for 2.13.2.1 references non-existent jackson-bom 2.13.2.1

[shakuzen]

Describe the bug
See:
https://repo1.maven.org/maven2/com/fasterxml/jackson/core/jackson-databind/2.13.2.1/jackson-databind-2.13.2.1.module

The Gradle module metadata for the 2.13.2.1 release contains:

{
  "group": "com.fasterxml.jackson",
  "module": "jackson-bom",
  "version": {
    "requires": "2.13.2.1"
  },
  "attributes": {
    "org.gradle.category": "platform"
  },
  "endorseStrictVersions": true
}

The jackson-bom version that should be resolved (via jackson-base) is 2.13.2.20220324. Everything is fine consuming jackson-databind from Maven, but from Gradle it is currently broken in 2.13.2.1 due to this wrong metadata which causes Gradle to try and fail to resolve jackson-bom:2.13.2.1 as seen in this error:

Could not find com.fasterxml.jackson:jackson-bom:2.13.2.1.

This can also be seen in this failed build scan.

Version information
Which Jackson version(s) was this for? 2.13.2.1

To Reproduce
If you have a way to reproduce this with:

Add a dependency on jackson-databind version 2.13.2.1 in a Gradle build and build the project. E.g.

dependencies {
    implementation 'com.fasterxml.jackson.core:jackson-databind:2.13.2.1'
}

Expected behavior
If reproduction itself needs further explanation, you may also add more details here.

Additional context
Add any other context about the problem here.

[melix]

The easiest fix would be to publish a 2.13.2.1 BOM (which, IMHO should be done in any case). Depending on your situation, you can:

1. write a component metadata rule to fix wrong metadata. This is only suitable if you are writing an application, not a library, because those rules are not propagated to consumers

dependencies {
    components {
        withModule("com.fasterxml.jackson.core:jackson-databind") {
            allVariants {
                withDependencies { 
                    removeAll {
                        it.name == "jackson-bom"
                    }
                }
            }
        }
    }
}

2. add a constraint like the following:

dependencies {
    constraints {
        api('com.fasterxml.jackson:jackson-bom') {
            version {
                strictly '[2.13.2, 2.13.3['
                prefer '2.13.2'
                reject '2.13.2.1'
            }
            because 'Jackson Databind references non existent BOM'
        }
    }
}

This, however, if published, would always prevent from getting 2.13.2.1 if it's released, so it's clearly not a great solution either.

[cowtowncoder]

Unfortunately 2.13.2.1 bom is not practical for the reason that there is no consistent set of all jars; only 2.13.2 exist for everything but jackson-databind. There might however be other micro-patches coming at a later point (jackson-core 2.13.2.1; jackson-databind 2.13.2.2etc) -- but we could only have one singlejackson-bom2.13.2.1. This is why there is BOM version like2.13.2.20220324that has whatever micro-patches (just 1 in this case) exist for2.13.2` at the time of release. BOM version will still be ordered as expected with respect to both full releases and micro-patch versions.
These are pretty special hot fix releases, usually only done after branch is closed although sometimes earlier, like here.

I hope above makes sense wrt what this scheme tries to accomplish. I realize it is not necessarily compatible with expectations of other tooling that tries to make sense of structure.

On plus side once there are enough other fixes there will be 2.13.3, full set, so this becomes less of an issue.

[laffer1]

I understand your perspective but the way that tools, particularly for CVEs work, you should just follow a pattern where these micro releases trigger a full build for each patch. This 2.13.2.1 would have contained this fix and the next one would generate 2.13.2.2 and so on. Having artifacts with dates at the end causes chaos.

[cowtowncoder]

@laffer1 That may be how other projects work but this is not the way Jackson works or (IMO of course) should work.

The trade-off is as follows: releasing a full set of versions for ~20 components from about dozen repositories takes multiple hours. I will not do this for ONE SINGLE change in ONE COMPONENT. Full sets are released when there are enough things to release.

But sometimes community wants a hot-fix; either for otherwise closed branch, or, less commonly, due to timing from an open branch. In these cases I can do a patch version of just that one component. This is quicker to do.

And I could leave it at that: let everything use existing patch-version jackson-bom like 2.13.2 and override specific micro-patch component (or components, in theory). But to me it seems reasonable to provide a special bom as well: one that sorts in version hierarchy as expected -- use of timemstamp should be absolutely fine! -- like so:

2.13.2 < 2.13.2.20220225 < 2.13.3

so it is always clear which version of jackson-bom is newer.
I could alternatively choose to use seemingly "easier" or "simpler" version like 2.13.2.1 but that might give the wrong idea that there is a full set of components with that version number, which is not the case.
I guess it's a matter of taste.

From user perspective I do not see an actual problem: they select the latest jackson-bom, and version does give clear indicate of the patch level (and suggests, if you are familiar with the project, that something else has micro patch).

I do realize that some specific tooling -- like Gradle -- might have some other assumptions, and I am happy to try to work with others to alleviate the problems.

But what I do not think is reasonable to ask is that one would not have differing version numbers in components included in a BOM: after all this is what version sets are all about! If every component everywhere had just one globally synchronized version number we wouldn't need BOMs (or version dependency management systems).

Of course if this is all too complicated, I could just stop releasing jackson-bom for micro-patches. But so far I have not heard users complaining.

[melix]

We can argue whether Jackson should be released as a unit or not, we can argue about the versioning scheme, but this is orthogonal to this issue. The problem to be fixed is that there's wrong published metadata, which is harmful. There is a module file which references a non-existent BOM. Currently it makes things fail, but if the BOM was published, then it would start "working".

So to me, either a new version of jackson-databind needs to be released referencing an existing BOM, or BOM version 2.13.2.1 needs to be released. Those are the only two options which would solve the problem. Personally, I think it's fine if you release components separately, but in this case I think it's reasonable to publish a BOM alongside, which lists all versions of all components. Note that not using the same version for all components prevents from using nice features like version alignment (when I worked for Gradle there were quite a few complaints about inconsistent Jackson versions for complex projects even using a BOM), but again, this is orthogonal to this feature.

There are no particular assumptions made by Gradle, it's quite flexible. The only requirement is to have correct metadata. When wrong metadata is encountered, there are ways to workaround this (fix metadata via rules), but unfortunately, this wouldn't propagate to consumers.

We should also think about ways to prevent this from happening in the future. I don't know the jackson release process enough to help there, but for Micronaut (which uses Gradle) we implemented for example a "BOM checker" which would guarantee that you don't publish a BOM referencing non existent dependencies. Something similar could be implemented to verify that a library doesn't reference non existent dependencies (including BOMs). However I think it wouldn't be as easy to do with Maven.

[jjohannes]

Sorry, I don't have time right now to get into the whole conversation here.

I think whether or not the versioning would be changed in the future, at first the metadata publishing for the current versioning approach should be fixed. I think this this is the right solution:
FasterXML/jackson-bom#52 (comment)

[laffer1]

The metadata is flat out wrong. That needs to be addressed.

As for the difficulty in releasing components, that sounds like a problem that can be solved with an automated pipeline and versioning.

[cowtowncoder]

@laffer1 I agree that Gradle metadata issue should be resolved (I assume this is the crux of the matter).
If it cannot be resolved I can also consider dropping inclusion of such metadata altogether, if it cannot be reliably generated; I don't know how widely it is being used at the moment. Ideally this is not necessary of course,
However: I would also like to point out that pom.xml definitions themselves seem workable to me: users have not reported issues in versions themselves, outside of GMM.

But I see no reason or benefit from changing Jackson's versioning itself, with respect to micro patches (hot fixes), nor have any plans for changes in this area. It just makes no sense to release dozens of jars for a single change in a single jar, no matter how things are automated (nor can automation typically resolve the problem of release notes etc etc etc).

That said, choice of version number for special BOMs (or possibly avoiding releasing them) is something I would consider. To me choices look like:

1. Keep things as they are and hope gradle tooling (maven plug-in) can resolve the problem
2. Stop releasing jackson-bom for micro patches (micro-patch would use minor version BOM, so jackson-databind 2.13.2.1 would refer to base / bom 2.13.2)
3. . Start releasing partially matching boms so that there is indeed jackson-bom version 2.13.2.1 with consistent set of components -- some at 2.13.2, other(s) at 2.13.2.1. But also leave potential "gaps" to allow monotonically increasing bom versions (2.13.2.2, 2.13.2.3) in case further micro-patches for some components are needed
4. Release timestamp-versioned jackson-bom BUT make jackson-databind:2.13.2.1 depend on jackson-base:2.13.2 to avoid "loop" -- and optionally (depends on discussion) ALSO release timestamp-versioned jackson-bom for users to use.

I added (4) as the most likely choice after realizing that jackson-base/jackson-bom combo (note: it's awkward bundling; changed for Jackson 3.0 but for backwards-compatibility stuck with for 2.x) is used for 2 different things:

1. For consistent set across Jackson components, with minimal work. This is via jackson-base (which has jackson-bom as parent)
2. For consistent external view of versions, outside of Jackson components. This is via jackson-bom

So far I had tried to make both use cases work the same (... for 2.x) but now that I think about it, approach (4) would separate use cases for case of micro patches. It seems to me this might be the way to avoid problems with metadata.

[cowtowncoder]

@melix (and @laffer1) I agree wrt metadata: it should NOT refer to non-existing BOM. That we are in agreement on.
So the question is how to get there. My explanation was intended to help consider possibilities, and outline parts I am most willing to change.

I added another note on FasterXML/jackson-bom#52 as well and I wonder if path forward would be what I propose there (which is same as what I added as option (4) for my comment above).

[cowtowncoder]

I think I know how to fix the immediate parent-pom-version-wrong problem, but have one final concern on whether endorseStrictVersions might cause problems. I just released:

1. jackson-databind:2.12.6.1 which declares jackson-base:2.12.6 as its parent pom: GMM includes jackson-bom:2.12.6 as a dependency. Should be fine, but bom does specify jackson-databind:2.12.6. Could this be problematic
2. jackson-bom:2.12.6.20220326 for users to use as BOM that provides proper version set. Not relevant wrt GMM any more.

It would be great if someone could see whether dependency to jackson-databind:2.12.6.1 now works as expected or not, before I go ahead and make similar changes to publish jackson-databind:2.13.2.2 to get working GMM info.

[jjohannes]

Thanks a lot @cowtowncoder and sorry for causing this trouble by missing this particular "micro version" detail when we first introduced Gradle Metadata to Jackson. I tested with jackson-databind:2.12.6.1 and everything works as expected:

1. BOM is not 'downgrading'
   Declare only implementation("com.fasterxml.jackson.core:jackson-databind:2.12.6.1")
   The BOM refers to 2.12.6, but Gradle uses 2.12.6.1
   Result in Build Scan
2. Dependencies are aligned to highest major.minor.patch
   Add e.g. implementation("com.fasterxml.jackson.dataformat:jackson-dataformat-protobuf:2.11.2")
   The version is aligned to 2.12.6 through the BOM
   Result in Build Scan.
3. Use the newest BOM directly
   Remove versions from dependencies (or keep them) and just use the newest BOM
   implementation(platform("com.fasterxml.jackson:jackson-bom:2.12.6.20220326"))
   All the latest 2.12.6.x versions are picked
   Result in Build Scan

(The endorseStrictVersions has no effect in the case of Jackson. It only matters if versions inside the "BOM" are marked as strict. Which is a Gradle-only concept that you could only use if you also publish the BOM with Gradle Metadata - which we don't do here.)

[cowtowncoder]

Thank you @jjohannes! It's more my fault wrt micro releases really as I knew about this case but did not think it through or ask questions. But at least we are solving the issue now I think; and can have both immediate improvements and longer term maintainable choices.

And so it looks like 2.12.6.1 of jackson-databind (and matching BOM) is good, and I can focus on 2.13.2(.2) and 2.14 branch. More on that in related jackson-bom issue.

[vishnuprakash-thoughtworks-sde]

Is it fixed on 2.13.2.1 ? I am still getting error Could not find com.fasterxml.jackson:jackson-bom:2.13.2.1

[yeikel]

If I use it like this:

implementation(platform("com.fasterxml.jackson:jackson-bom:2.13.2.20220328"))

it works as expected.

E.g. if I add

implementation("com.fasterxml.jackson.core:jackson-databind")

I get com.fasterxml.jackson.core:jackson-databind -> 2.13.2.2

This does not seem to work with maven or perhaps I am missing something

I am using

<jackson.bom.version>2.13.2.20220328</jackson.bom.version>

<dependency>
<groupId>com.fasterxml.jackson</groupId>
 <artifactId>jackson-bom</artifactId>
<version>${jackson.bom.version}</version>
<type>pom</type>
 <scope>import</scope>
</dependency>

But then I can't define it as :

<dependency>
<groupId>com.fasterxml.jackson.core</groupId>
<artifactId>jackson-databind</artifactId>
 </dependency>

Because Maven expects a version

'dependencies.dependency.version' for com.fasterxml.jackson.core:jackson-databind:jar is missing

At the same time, I can't use the bom version because they do not align as before

I could define two separate properties to manage the versions but that defeats the purpose of using the bom

In 2.13.2, I was using the same versions for both as they used to align

See https://github.com/reactiverse/vertx-maven-plugin/pull/391/files for an example

[WARNING] 'dependencies.dependency.scope' for com.fasterxml.jackson:jackson-bom:pom must be one of [provided, compile, runtime, test, system] but is 'import'. @ line 218, column 20
[ERROR] 'dependencies.dependency.version' for com.fasterxml.jackson.core:jackson-databind:jar is missing. @ line 221, column 21
 @ 
[ERROR] The build could not read 1 project -> [Help 1]
[ERROR]   
[ERROR]   The project io.reactiverse:vertx-maven-plugin:1.0-SNAPSHOT (/Users/yeikel/vertx-maven-plugin/pom.xml) has 1 error
[ERROR]     'dependencies.dependency.version' for com.fasterxml.jackson.core:jackson-databind:jar is missing. @ line 221, column 21
[ERROR] 
[ERROR] To see the full stack trace of the errors, re-run Maven with the -e switch.
[ERROR] Re-run Maven using the -X switch to enable full debug logging.
[ERROR] 
[ERROR] For more information about the errors and possible solutions, please read the following articles:

edit :

Never mind. The issue is that BOMs need to be imported under dependencyManagement and not dependencies

See : https://stackoverflow.com/questions/11778276/what-is-the-difference-between-pom-type-dependency-with-scope-import-and-wit

[chrisdennis]

This is great and all... but none of this works if a Gradle user has turned on failOnVersionConflict(). The ultimate problem here seems to be that Jackson wants to use micro-versions for security fixes to individual packages, but the current system of BOMs and intermodule dependency declarations that connect the Jackson modules doesn't account for this.

I believe there are two outstanding issues here:

1. The Gradle module metadata published with each modules links the modules back to the originally released 3-digit BOM version only. I think that should probably be a range: [2.13.2,2.13.3) to correctly capture that "aligned" could mean using a module that has no security fixes with a BOM that captures security fixes.
2. The intermodule dependencies are "fixed" and not version ranges. If security fixes are done using the fourth digit, then dependencies on central modules like jackson-core or Jackson-databind should capture that using appropriate ranges. Of course currently this is indirectly tied back to the parent POM which does the central dependency management for the whole system, which is also the BOM for the system - I fear that fact might make things yet more complicated.

I'm wondering if the 'correct' solution here is to remove the BOM from the POM hierarchy. At that point you have the ultimate parent POM capture the compatibility rules for the versions, and then an independent BOM catalog collect a specific set of artifacts that satisfy those rules?

[cowtowncoder]

@chrisdennis I tried to ask about whether this would be problematic but understood @jjohannes suggested things are fine. I think my question was too specific on one aspect of conflicts.

I agree that it seems that some aspect of version ranges would be useful; but I have no idea how to make that work, and version ranges in, say, Maven dependencies, are widely considered to be problematic (wrt repeatable builds for example).

Now; I am not sure how to proceed. Issue specifically affects component(s) with micro-patch: here jackson-databind it seems. There we have 2 choices of jackson-bom to depend on (and I don't see how we could avoid that dependency easily):

1. Depend on Correct date-time stamped jackson-bom: it does provide micro-patch version, everything properly versioned. I did this earlier, and if it wasn't for incorrect plug-in configuration (using project.version instead of project.parent.version) it'd work. I did NOT use that for 2.13.2.2 as I thought the alternative might be better
2. Depend on minor-only jackson-bom (like 2.13.2). I chose that for 2.13.2.2 based on discussions

Doing (1) would provide better set for jackson-databind (or whatever micro-patched), but of course transitive dependencies via other other components would still bring "base" version (2.13.2).
If that leads to conflicts, we are probably screwed? :-(

[jjohannes]

Depend on Correct date-time stamped jackson-bom

If this is possible with your release/versioning process, it would be the best solution. Then there would be no "conflict" ,

[cowtowncoder]

@jjohannes This is how 2.13.2.1 would have worked, if the generation had used Maven parent pom version (instead of project version). I can definitely go back to doing that, instead of older minor-version bom.
That works fine as long as I first release BOM, then make micro patch component use that as the parent.

Not sure how big a problem the use of "last patch version BOM" is for 2.12.6.1 / 2.13.2.1 / 2.13.2.2.
I hope there is a work-around from Gradle side. Otherwise I guess I can (if absolutely something users need) to do one more release round for 2.12.6.2 / 2.13.2.2. But would really like not doing that at this point.

[jjohannes]

That works fine as long as I first release BOM, then make micro patch component use that as the parent.

@cowtowncoder that sounds good. I would do it like that for the next micro-patch releases then.

To reiterate: The important bit is that the version of the component that references the a timestamp BOM is the same version that the timestamp BOM references back of that component. Then there won't be this "internal conflict".

The issue only occurs if users use the failOnVersionConflict feature of Gradle. Something that in my mind is outdated as it blocks you from using a lot of useful conflict resolution functionality of Gradle. That's why I did not think about it when I suggested the solution you used.

But of course, I should not make assumptions about all users. So thank you @chrisdennis for bringing this up.

Not sure how big a problem the use of "last patch version BOM" is for 2.12.6.1 / 2.13.2.1 / 2.13.2.2.

I think it is not a big problem. Users who use failOnVersionConflict will most likely have rules in their build to "exclude" dependencies or "force" dependency versions already. They could can add another "exclude" for the case where the BOM is biting them. Or do something like what was suggested earlier. And the problem will be gone sooner or later with the next Jackson releases. But that's only my take on this.

[cowtowncoder]

Ok thank you @jjohannes. This makes sense. I will then follow the original procedure, now that generation of GMM has been fixed to use parent pom reference (as well as allow optional override if need be).

[chrisdennis]

From my point-of-view this is "solved" in that I can work around it. The working around is a little un-orthodox but it works. I'm having to do a combination of component metadata rules to decouple the Jackson modules from the BOM version restrictions, and a dependency substitution on the jackson-databind module. I would classify this as more than I should have to do (even for a failOnVersionConflict() user), but fault for that can probably be split between Jackson and Gradle.

This isn't really the place for a discussion of the relative merits of the "optimistic version upgrades" versus "manual conflict resolution" when resolving a dependency graph, so I'll leave that discussion in the Gradle Slack.

[blaghed]

Wow, what a mess... I can't believe you considered this as closed, since "jackson-bom:2.13.2.1" is totally broken as part of this discussion.
Will there be any follow-ups to this, or is it just considered as done and people have to hack their projects to get a proper fix in?

[jjohannes]

If you have something to add here, it would be wonderful if you would describe the issue you are having with instructions on how to reproduce it. The problems known so far, and their solutions, are discussed and described above.

A comment like:

Wow, what a mess... I can't believe you considered this as closed, since "jackson-bom:2.13.2.1" is totally broken as part of this discussion.

Is not an accurate description of a problem. You might want to reconsider how you communicate - in general, but also in particular on an issue tracker of an open source project that is maintained by a single person in their free time.

The underlying issue is solved. Upcoming Jackson releases won't run into this again. That's why the issue is closed. As said above: Already published components/metadata cannot be changed.

[blaghed]

Fair enough.

So, my issue is that when simply using "jackson-bom:2.13.2.1" (a released version that is listed here: https://search.maven.org/artifact/com.fasterxml.jackson/jackson-bom), the related dependencies all fail, since there are no releases for almost any of the jackson components that match the "2.13.2.1" version.
"jackson-databind" itself was given a custom version as part of that release (it is 2.13.2.2 there), so it is pretty much the only dependency that works in that bom.

Do keep in mind that I am not complaining about trying to use jackson-databin 2.13.2.1, but rather that this discussion led the bom into a really poor direction, while only fixing "exactly" jackson-databind and nothing else.

[yawkat]

I think the combination jackson-bom 2.13.2 and jackson-databind 2.13.2.2 should work fine right now. It's what we use with micronaut.

The bom will be fixed in the next release.

[cowtowncoder]

@blaghed I am not sure why you think jackson-bom:2.13.2.1 -- which I only released based on these discussion, mind you, trying to help everyone -- is broken. It should include a set with 2.13.2 for everything else except for jackson-databind. It is a one-off and there should not be any more of such bom versions (there will be "X.Y.Z" and "X.Y.Z.FULLDATE" but no "X.Y.Z.N"); but this one was included because of problems with GMM generation that led to otherwise dangling reference.

I do not quite understand this part:

this discussion led the bom into a really poor direction, while only fixing "exactly" jackson-databind and nothing else.

partly because there was nothing else that could be fixed in retrospect about jackson-databind:2.13.2.1 release: Maven Central is immutable and we cannot change what was already published.

Having said that, use of

https://mvnrepository.com/artifact/com.fasterxml.jackson/jackson-bom/2.13.2.20220328

(which points to jackson-databind:2.13.2.2 and otherwise 2.13.2 components)

is still recommended.

[cowtowncoder]

Just one last note: there now actually is jackson-bom:2.13.2.1 released, at Maven Central:

https://mvnrepository.com/artifact/com.fasterxml.jackson/jackson-bom/2.13.2.1

so the issue of dangling reference should be resolved.
This release was done as a one-off since there was no way to change actual jackson-databind:2.13.2.1 to point to the original intended, date-marked BOM version.