Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Maven codegen Mojo #5319

Closed
wants to merge 3 commits into from
Closed

Conversation

juergen-albert
Copy link
Contributor

@juergen-albert juergen-albert commented Jul 15, 2022

This PR is meant to get some early feedback on this.

The Mojo provides support for the generate mechanism via external plugins. The general bnd generator mechanism reads the instruction and looks in the workspace if it finds a bundle that provides a properly named external plugin capability. Thus I need to provide a Repository containing the dependency with the generator.
At the moment I use the build dependencies as source for that similar to our Mojos that work on a bndrun. This seems a bit wrong, as this is actually a Plugin dependency and no direct dependency of my maven module. I tried another approach doing the same based on the plugin dependencies. There is a problem though. Plugin dependencies are part of the Mojos Classpath when bnd starts it. If I didn't read the code wrong, generate requires a repository and will load the resulting jar in a completly isolated classloader. This in theory should not cause any issues with the Mojos classloader, that already has this jar.
So much for the theory. Reality poses other issues however. The isolated nature of the external plugin requires an all in one jar, that packs every class it will ever need. In the case of my generator I use and thus package e.g. Promises in an older Version. Because my generator is now on the classpath, it can break the Mojo and causes:

java.lang.NoSuchMethodError: org.osgi.util.promise.PromiseFactory.toPromise()Ljava/util/stream/Collector;
    at aQute.bnd.repository.fileset.FileSetRepository.readFiles (FileSetRepository.java:81)
    at aQute.bnd.repository.fileset.FileSetRepository.getBridge (FileSetRepository.java:73)
    at aQute.bnd.repository.fileset.FileSetRepository.list (FileSetRepository.java:189)
    at aQute.bnd.maven.export.plugin.BndContainer.injectImplicitRepository (BndContainer.java:170)
    at aQute.bnd.maven.export.plugin.BndContainer.generate (BndContainer.java:146)
    at aQute.bnd.maven.export.plugin.GenerateMojo.execute (GenerateMojo.java:69)

The classloader hirachy looks as follows (my generator is url[1] and the correct promise jar is url{37]):

realm =    plugin>biz.aQute.bnd:bnd-generate-maven-plugin:6.4.0-SNAPSHOT
strategy = org.codehaus.plexus.classworlds.strategy.SelfFirstStrategy
urls[0] = file:/N:/git/bndtools/maven/bnd-generate-maven-plugin/target/it-repo/biz/aQute/bnd/bnd-generate-maven-plugin/6.4.0-SNAPSHOT/bnd-generate-maven-plugin-6.4.0-SNAPSHOT.jar
urls[1] = file:/N:/git/bndtools/maven/bnd-generate-maven-plugin/target/it-repo/org/geckoprojects/emf/org.gecko.emf.osgi.codegen/4.1.1.202202162308/org.gecko.emf.osgi.codegen-4.1.1.202202162308.jar
urls[2] = file:/N:/git/bndtools/maven/bnd-generate-maven-plugin/target/it-repo/org/osgi/org.osgi.annotation.bundle/1.0.0/org.osgi.annotation.bundle-1.0.0.jar
urls[3] = file:/N:/git/bndtools/maven/bnd-generate-maven-plugin/target/it-repo/org/osgi/org.osgi.annotation.versioning/1.0.0/org.osgi.annotation.versioning-1.0.0.jar
urls[4] = file:/N:/git/bndtools/maven/bnd-generate-maven-plugin/target/it-repo/org/eclipse/emf/org.eclipse.emf.ecore.xmi/2.16.0/org.eclipse.emf.ecore.xmi-2.16.0.jar
urls[5] = file:/N:/git/bndtools/maven/bnd-generate-maven-plugin/target/it-repo/org/eclipse/emf/org.eclipse.emf.ecore/2.25.0/org.eclipse.emf.ecore-2.25.0.jar
urls[6] = file:/N:/git/bndtools/maven/bnd-generate-maven-plugin/target/it-repo/org/eclipse/emf/org.eclipse.emf.common/2.23.0/org.eclipse.emf.common-2.23.0.jar
urls[7] = file:/N:/git/bndtools/maven/bnd-generate-maven-plugin/target/it-repo/org/eclipse/emf/org.eclipse.emf.codegen.ecore/2.28.0/org.eclipse.emf.codegen.ecore-2.28.0.jar
urls[8] = file:/N:/git/bndtools/maven/bnd-generate-maven-plugin/target/it-repo/org/eclipse/platform/org.eclipse.core.runtime/3.25.0/org.eclipse.core.runtime-3.25.0.jar
urls[9] = file:/N:/git/bndtools/maven/bnd-generate-maven-plugin/target/it-repo/org/eclipse/platform/org.eclipse.core.jobs/3.13.0/org.eclipse.core.jobs-3.13.0.jar
urls[10] = file:/N:/git/bndtools/maven/bnd-generate-maven-plugin/target/it-repo/org/eclipse/platform/org.eclipse.core.contenttype/3.8.100/org.eclipse.core.contenttype-3.8.100.jar
urls[11] = file:/N:/git/bndtools/maven/bnd-generate-maven-plugin/target/it-repo/org/eclipse/platform/org.eclipse.equinox.app/1.6.100/org.eclipse.equinox.app-1.6.100.jar
urls[12] = file:/N:/git/bndtools/maven/bnd-generate-maven-plugin/target/it-repo/org/eclipse/platform/org.eclipse.core.resources/3.17.0/org.eclipse.core.resources-3.17.0.jar
urls[13] = file:/N:/git/bndtools/maven/bnd-generate-maven-plugin/target/it-repo/org/eclipse/jdt/org.eclipse.jdt.core/3.30.0/org.eclipse.jdt.core-3.30.0.jar
urls[14] = file:/N:/git/bndtools/maven/bnd-generate-maven-plugin/target/it-repo/org/eclipse/jdt/org.eclipse.jdt.launching/3.19.600/org.eclipse.jdt.launching-3.19.600.jar
urls[15] = file:/N:/git/bndtools/maven/bnd-generate-maven-plugin/target/it-repo/org/eclipse/jdt/org.eclipse.jdt.debug/3.19.200/org.eclipse.jdt.debug-3.19.200.jar
urls[16] = file:/N:/git/bndtools/maven/bnd-generate-maven-plugin/target/it-repo/org/eclipse/platform/org.eclipse.core.variables/3.5.100/org.eclipse.core.variables-3.5.100.jar
urls[17] = file:/N:/git/bndtools/maven/bnd-generate-maven-plugin/target/it-repo/org/eclipse/emf/org.eclipse.emf.codegen/2.22.0/org.eclipse.emf.codegen-2.22.0.jar
urls[18] = file:/N:/git/bndtools/maven/bnd-generate-maven-plugin/target/it-repo/org/eclipse/platform/org.eclipse.debug.core/3.19.100/org.eclipse.debug.core-3.19.100.jar
urls[19] = file:/N:/git/bndtools/maven/bnd-generate-maven-plugin/target/it-repo/org/eclipse/platform/org.eclipse.core.expressions/3.7.100/org.eclipse.core.expressions-3.7.100.jar
urls[20] = file:/N:/git/bndtools/maven/bnd-generate-maven-plugin/target/it-repo/org/eclipse/platform/org.eclipse.core.filesystem/1.9.0/org.eclipse.core.filesystem-1.9.0.jar
urls[21] = file:/N:/git/bndtools/maven/bnd-generate-maven-plugin/target/it-repo/org/eclipse/platform/org.eclipse.equinox.common/3.16.100/org.eclipse.equinox.common-3.16.100.jar
urls[22] = file:/N:/git/bndtools/maven/bnd-generate-maven-plugin/target/it-repo/org/eclipse/platform/org.eclipse.equinox.registry/3.11.100/org.eclipse.equinox.registry-3.11.100.jar
urls[23] = file:/N:/git/bndtools/maven/bnd-generate-maven-plugin/target/it-repo/org/eclipse/platform/org.eclipse.osgi/3.18.0/org.eclipse.osgi-3.18.0.jar
urls[24] = file:/N:/git/bndtools/maven/bnd-generate-maven-plugin/target/it-repo/org/eclipse/platform/org.eclipse.text/3.11.0/org.eclipse.text-3.11.0.jar
urls[25] = file:/N:/git/bndtools/maven/bnd-generate-maven-plugin/target/it-repo/org/eclipse/platform/org.eclipse.core.commands/3.10.200/org.eclipse.core.commands-3.10.200.jar
urls[26] = file:/N:/git/bndtools/maven/bnd-generate-maven-plugin/target/it-repo/org/eclipse/platform/org.eclipse.equinox.preferences/3.10.1/org.eclipse.equinox.preferences-3.10.1.jar
urls[27] = file:/N:/git/bndtools/maven/bnd-generate-maven-plugin/target/it-repo/org/osgi/org.osgi.service.prefs/1.1.2/org.osgi.service.prefs-1.1.2.jar
urls[28] = file:/N:/git/bndtools/maven/bnd-generate-maven-plugin/target/it-repo/biz/aQute/bnd/aQute.libg/5.3.0/aQute.libg-5.3.0.jar
urls[29] = file:/N:/git/bndtools/maven/bnd-generate-maven-plugin/target/it-repo/org/osgi/osgi.core/6.0.0/osgi.core-6.0.0.jar
urls[30] = file:/N:/git/bndtools/maven/bnd-generate-maven-plugin/target/it-repo/biz/aQute/bnd/biz.aQute.bnd.maven/6.4.0-SNAPSHOT/biz.aQute.bnd.maven-6.4.0-SNAPSHOT.jar
urls[31] = file:/N:/git/bndtools/maven/bnd-generate-maven-plugin/target/it-repo/org/osgi/org.osgi.dto/1.0.0/org.osgi.dto-1.0.0.jar
urls[32] = file:/N:/git/bndtools/maven/bnd-generate-maven-plugin/target/it-repo/org/osgi/org.osgi.resource/1.0.0/org.osgi.resource-1.0.0.jar
urls[33] = file:/N:/git/bndtools/maven/bnd-generate-maven-plugin/target/it-repo/org/osgi/org.osgi.framework/1.8.0/org.osgi.framework-1.8.0.jar
urls[34] = file:/N:/git/bndtools/maven/bnd-generate-maven-plugin/target/it-repo/org/osgi/org.osgi.service.repository/1.1.0/org.osgi.service.repository-1.1.0.jar
urls[35] = file:/N:/git/bndtools/maven/bnd-generate-maven-plugin/target/it-repo/org/osgi/org.osgi.util.function/1.2.0/org.osgi.util.function-1.2.0.jar
urls[36] = file:/N:/git/bndtools/maven/bnd-generate-maven-plugin/target/it-repo/org/osgi/osgi.annotation/8.0.1/osgi.annotation-8.0.1.jar
urls[37] = file:/N:/git/bndtools/maven/bnd-generate-maven-plugin/target/it-repo/org/osgi/org.osgi.util.promise/1.2.0/org.osgi.util.promise-1.2.0.jar
urls[38] = file:/N:/git/bndtools/maven/bnd-generate-maven-plugin/target/it-repo/org/codehaus/plexus/plexus-utils/3.0.22/plexus-utils-3.0.22.jar
urls[39] = file:/N:/git/bndtools/maven/bnd-generate-maven-plugin/target/it-repo/biz/aQute/bnd/biz.aQute.bnd.util/6.4.0-SNAPSHOT/biz.aQute.bnd.util-6.4.0-SNAPSHOT.jar
urls[40] = file:/N:/git/bndtools/maven/bnd-generate-maven-plugin/target/it-repo/biz/aQute/bnd/biz.aQute.bndlib/6.4.0-SNAPSHOT/biz.aQute.bndlib-6.4.0-SNAPSHOT.jar
urls[41] = file:/N:/git/bndtools/maven/bnd-generate-maven-plugin/target/it-repo/org/osgi/org.osgi.util.tracker/1.5.4/org.osgi.util.tracker-1.5.4.jar
urls[42] = file:/N:/git/bndtools/maven/bnd-generate-maven-plugin/target/it-repo/org/osgi/org.osgi.service.log/1.3.0/org.osgi.service.log-1.3.0.jar
urls[43] = file:/N:/git/bndtools/maven/bnd-generate-maven-plugin/target/it-repo/biz/aQute/bnd/biz.aQute.resolve/6.4.0-SNAPSHOT/biz.aQute.resolve-6.4.0-SNAPSHOT.jar
urls[44] = file:/N:/git/bndtools/maven/bnd-generate-maven-plugin/target/it-repo/org/osgi/org.osgi.service.resolver/1.1.1/org.osgi.service.resolver-1.1.1.jar
urls[45] = file:/N:/git/bndtools/maven/bnd-generate-maven-plugin/target/it-repo/biz/aQute/bnd/biz.aQute.repository/6.4.0-SNAPSHOT/biz.aQute.repository-6.4.0-SNAPSHOT.jar
urls[46] = file:/N:/git/bndtools/maven/bnd-generate-maven-plugin/target/it-repo/org/osgi/org.osgi.service.coordinator/1.0.2/org.osgi.service.coordinator-1.0.2.jar
urls[47] = file:/N:/git/bndtools/maven/bnd-generate-maven-plugin/target/it-repo/biz/aQute/bnd/biz.aQute.bnd.embedded-repo/6.4.0-SNAPSHOT/biz.aQute.bnd.embedded-repo-6.4.0-SNAPSHOT.jar

Thus I decided to stick with the method of using the build depencies, even though this goes against the typical mechanism with the extended plugin classpath.

@rotty3000 @pkriens @bjhargrave
Any thoughts or ideas on that?

Additional things that are on the todo list:

  • More Tests
  • configuration via pom.xml (currently I read the bnd file, similar to the bnd-maven-plugin)
  • Documentation
  • Find a way to automatically attach the generate goal to the generate-sources phase
  • remove the test as eclipse project
  • rebase on master

@bjhargrave bjhargrave marked this pull request as draft July 15, 2022 14:38
@laeubi
Copy link
Contributor

laeubi commented Jul 26, 2022

  • Find a way to automatically attach the generate goal to the generate-sources phase

You need a make your plugin an extension and define a custom packaging type for this to work, or you need to use AbstractMavenLifecycleListener and hack something into the model in afterProjectsRead

@juergen-albert
Copy link
Contributor Author

You need a make your plugin an extension and define a custom packaging type for this to work, or you need to use AbstractMavenLifecycleListener and hack something into the model in afterProjectsRead

@laeubi Thanks heaps. I will have a look.

@juergen-albert
Copy link
Contributor Author

@rotty3000 @bjhargrave @timothyjward Guys, I need some help.

I still have some issue with the problem from above. Here the TLDR of my initial issue for context:

  • the generate plugin use the bnd code generator, which in turn uses the ExternalPlugins
  • We load the external plugin in an isolated Classloader, so a code generator need to live in an uber jar that brings all dependencies along with it
  • configuring the new generate plugin with a dependency section as follows would be the right way to go, to use the typical maven mechanism for such things
                     <plugin>
				<groupId>biz.aQute.bnd</groupId>
				<artifactId>bnd-generate-maven-plugin</artifactId>
				<version>${bnd.version}</version>
				<dependencies>
					<dependency>
						<groupId>org.geckoprojects.emf</groupId>
						<artifactId>org.gecko.emf.osgi.codegen</artifactId>
						<version>4.1.1.202202162308</version>
					</dependency>
				</dependencies>
				<executions>
					<execution>
						<phase>generate-sources</phase>
						<goals>
							<goal>bnd-generate</goal>
						</goals>
					</execution>
				</executions>
			</plugin>
  • if the codegen jar brings e.g. promises in an older version along, it polluts the classpath of the mojo (as my example does) and we gat e.g. a NoSuchMethodError

@timothyjward gave me the advice to use the maven-shade-plugin to make the mojo an uber jar and move all its packges to a save place. Said and done and it seems to do the rick. All depdendent classes are packed and live in new save package namespaces.

Now my problem:

We use the flatten-maven-plugin to normalize the pom. It uses MavenProject.getFile() to get the pom, reads and resolves it and writes it to .flattened-pom.xml. It even has a nice option so it can call mavenProject.setPomFile() to set it again. All fine and dandy so far.
The maven-shade-plugin also modifies the pom. I even need it to do that , so it removes the dependencies that are now included, so they don't polut the classpath. It however gives zero fucks about the newly set pom file, as it works directly with the model of the mavenProject and writes this result to the jar.
The integration test now fails, complaining that it can't resolve the parent pom which now stays in the included pom.xml can't be resolved as it does not know what the version {revision} means. I suspect that this would happen in normal maven as well.

My conclusion so far: The shade plugin and the flatten plugin are 100% incompatible and for some reason the share plugin causes a pom.xml that by the integration test can't be resolved and thus maybe maven in general (the it-respo it creates for the test seems valid).

Right now, my only option to make this work is by searching for the generate plugin in the normal module dependencies, which works fine. This however is actually wrong, as this is a dependency which would fall under the plugin management and not the module dependencies. Before nothing works I would go for this solution, but only reluctantly.

Any advice?

@laeubi
Copy link
Contributor

laeubi commented Jul 28, 2022

The maven-shade-plugin also modifies the pom. I even need it to do that , so it removes the dependencies that are now included, so they don't polut the classpath. It however gives zero fucks about the newly set pom file, as it works directly with the model of the mavenProject and writes this result to the jar.

maven plugins are open source so you probably want to propose a patch.

We use the flatten-maven-plugin to normalize the pom.
The maven-shade-plugin also modifies the pom. I

You most probably want to change execution order of the mojos then

  • configuring the new generate plugin with a dependency section as follows would be the right way to go, to use the typical maven mechanism for such things

I'm not sure if this is "typical" if you actually don't want ti on your mojos classpath, because the typical use for this is:

  1. You need to override a declared dependency of the mojo (e.g. with a never version)
  2. You want to enhance (like fragments in OSGi) the class-space of the mojo

Your description of the problem sounds not that this is what you want, but I'm not completely sure why you need the additional things, but if you just want to analyze them in some way, or pass them to some code, it would be more suitable to mention them in the configuration section of your mojo and just use the Maven resolver (something along the lines like this: https://github.com/eclipse/tycho/blob/3d62159014ee3613b5e6e1c8337ce3521535dc9b/tycho-core/src/main/java/org/eclipse/tycho/osgi/configuration/MavenDependenciesResolverConfigurer.java#L62) you can even resolve transitive ones and then you can get the artifacts file and do whatever you want with it.

@laeubi
Copy link
Contributor

laeubi commented Jul 28, 2022

I just noticed that the BND baseline mojo do something similar to look-up the basline artifact ...

@juergen-albert
Copy link
Contributor Author

maven plugins are open source so you probably want to propose a patch.
I know, but I wouldn't know where to start, as this would in parts result in kind of a rewrite of the one or the other.

You most probably want to change execution order of the mojos then
I thought the same, but it would not change anything, as they do there things independently and can't share anything, the way they are build.

The generate plugin in and of itself will not do anything. It can execute code generators provided by third party libraries. After a night of sleep, I would provide the mojos plugins via the configuration with something like:

<configuration>
  <externalPlugins>
    <dependency>
        ...
    </dependency> 
  </externalPlugins>
</configuration>

@juergen-albert juergen-albert force-pushed the maven_codegen branch 2 times, most recently from a123867 to ed6d824 Compare July 29, 2022 14:32
@juergen-albert juergen-albert changed the title [Draft] Maven codegen Mojo Maven codegen Mojo Jul 29, 2022
Signed-off-by: Juergen Albert <j.albert@data-in-motion.biz>
Signed-off-by: Juergen Albert <j.albert@data-in-motion.biz>
Signed-off-by: Juergen Albert <j.albert@data-in-motion.biz>
@juergen-albert
Copy link
Contributor Author

@bjhargrave I have split the commits as discussed. I hope this works better now. Can you have a look and remove the draft Marker? Thanks

@juergen-albert juergen-albert marked this pull request as ready for review September 5, 2022 14:34
@juergen-albert
Copy link
Contributor Author

okay, it seemed that I was able to remove the draft state myself...

@bjhargrave bjhargrave mentioned this pull request Sep 29, 2022
@bjhargrave
Copy link
Member

Replaced by #5385

@bjhargrave bjhargrave closed this Sep 29, 2022
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

None yet

3 participants