Build bnd styled bundle in PDE setup

[fipro78]

Hi,

I have a PDE project setup to create some Eclipse UI plug-ins. I wanted to add a bundle for some services with the new bnd style, so the OSGi metadata gets automatically generated. I added the requirements in the pde.bnd file, and everything resolves fine in the workspace. But the Tycho build fails at the compilation phase, because it is not able to resolve the dependencies. There are many compilation failures, starting from the import statements.

I use Tycho 4.0.7 with a pomless setup and have a target definition configured to resolve the dependencies of the project.

Does Tycho already support the new project setup with a pde.bnd file?
Do I need a special bundle to be included in the build to make it work?
Do I have to specify the dependencies in an additional way to make them resolveable by Tycho (e.g. a pom.xml file or entries in the build.properties)?

Greez,
Dirk

[laeubi]

Hi Dirk,

great to hear that this new feature is used/evaluated. Tycho should support this already out of the box, but it is still quite new so there might be some bugs lurking around or missing documentation or some cases that are not fully supported. Also as always you should not need to declare things in an alternative way, Tycho should pickup and discover things automatically.

You can also find my little demonstrator project where I have evaluated this new feature in Equinox here but it is a very limited case of course!

Maybe you can create a little project in the Tycho demo that shows the problem?

[fipro78]

@laeubi

Stripping things down to a very simple setup is always not that easy. But I will give it a try. I will need some time to create a really simple reproducer.

In the meantime I have updated my cookbook example from the Tycho branch and converted the service project there.

https://github.com/fipro78/e4-cookbook-basic-recipe/tree/tycho-pde-bnd

I also get a compile error there, but interestingly here it says

Access restriction: The type 'Event' is not API

while on the other bigger project I get tons of

The import org.eclipse cannot be resolved

for various imports of EMF, XText etc.

Hope that helps at the moment. I will try to setup a simpler example for the Tycho demo. Or would a maybe simplified version of that cookbook example also be ok?

[laeubi]

There is a conversion wizard? I thought there is no such wizard at the moment.

Yes see:

• Provide and action to convert MANIFEST-first PDE projects to PDE automatic manifest eclipse-pde/eclipse.pde#934

it seems I sadly missed to write a N&N entry for that, if you like you can try it out and maybe want to describe it here (users usually write better entries than developers):
https://github.com/eclipse-platform/www.eclipse.org-eclipse/blob/master/news/4.32/pde.html

It currently misses conversion of hand written ds components but beside from that it should hopefully lower the barrier for using automatic manifest generation.

[fipro78]

Ah, cutting edge 4.32 ... so not released yet ;)

I will try it out once I am done with my current work and see that I can provide some N&N. But for that, the build should at least work, so there is a good overall feeling about that cool new feature.

[laeubi]

Ah, cutting edge 4.32 ... so not released yet ;)

It was released with 2023-12 maybe something improved later with 2024-03 but should defiantly be release, I just missed a N&N entry :-\

[fipro78]

Oh, I didn't see it, as it is in the Plug-in Tools context menu. I actually never used that entry. I always searched in the Configure context menu, as this is the place where you typically find the "Convert to xxx" functions.

[laeubi]

@fipro78 if you think we better place it somewhere else just propose a PR :-)

[laeubi]

@fipro78 I now tried the example and it first failed with

org.apache.maven.lifecycle.LifecycleExecutionException: Cannot resolve dependencies of project org.fipro.eclipse.tutorial:org.fipro.eclipse.tutorial.inverter:eclipse-plugin:1.0.0-SNAPSHOT
 with context {osgi.os=win32, osgi.ws=win32, org.eclipse.update.install.features=true, osgi.arch=x86_64, org.eclipse.update.install.sources=true}
  Software being installed: org.fipro.eclipse.tutorial.inverter 1.0.0.qualifier
  Missing requirement: org.fipro.eclipse.tutorial.inverter 1.0.0.qualifier requires 'java.package; org.fipro.eclipse.tutorial.service.inverter [1.0.0,2.0.0)' but it could not be found

I fixed that by adding

Export-Package:  org.fipro.eclipse.tutorial.*

to the pde.bnd file, after that I also see the compile errors now:

[ERROR] Failed to execute goal org.eclipse.tycho:tycho-compiler-plugin:5.0.0-SNAPSHOT:compile (default-compile) on project org.fipro.eclipse.tutorial.service.inverter: Compilation failure: Compilation failure: 
[ERROR] e4-cookbook-basic-recipe-tycho-pde-bnd/org.fipro.eclipse.tutorial.service.inverter/src/org/fipro/eclipse/tutorial/service/inverter/impl/InverterServiceImpl.java:[9] 
[ERROR] 	import org.osgi.service.event.Event;
[ERROR] 	       ^^^^^^^^^^^^^^^^^^^^^^^^^^^^
[ERROR] Access restriction: The type 'Event' is not API (restriction on classpath entry 'repository/p2/osgi/bundle/org.osgi.service.event/1.4.1.202109301733/org.osgi.service.event-1.4.1.202109301733.jar')
[ERROR] e4-cookbook-basic-recipe-tycho-pde-bnd/org.fipro.eclipse.tutorial.service.inverter/src/org/fipro/eclipse/tutorial/service/inverter/impl/InverterServiceImpl.java:[10] 
[ERROR] 	import org.osgi.service.event.EventAdmin;
[ERROR] 	       ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
[ERROR] Access restriction: The type 'EventAdmin' is not API (restriction on classpath entry 'repository/p2/osgi/bundle/org.osgi.service.event/1.4.1.202109301733/org.osgi.service.event-1.4.1.202109301733.jar')
[ERROR] e4-cookbook-basic-recipe-tycho-pde-bnd/org.fipro.eclipse.tutorial.service.inverter/src/org/fipro/eclipse/tutorial/service/inverter/impl/InverterServiceImpl.java:[11] 
[ERROR] 	import org.osgi.service.event.EventConstants;
[ERROR] 	       ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
[ERROR] Access restriction: The type 'EventConstants' is not API (restriction on classpath entry 'repository/p2/osgi/bundle/org.osgi.service.event/1.4.1.202109301733/org.osgi.service.event-1.4.1.202109301733.jar')
[ERROR] e4-cookbook-basic-recipe-tycho-pde-bnd/org.fipro.eclipse.tutorial.service.inverter/src/org/fipro/eclipse/tutorial/service/inverter/impl/InverterServiceImpl.java:[17] 
[ERROR] 	private EventAdmin eventAdmin;
[ERROR] 	        ^^^^^^^^^^
[ERROR] Access restriction: The type 'EventAdmin' is not API (restriction on classpath entry 'repository/p2/osgi/bundle/org.osgi.service.event/1.4.1.202109301733/org.osgi.service.event-1.4.1.202109301733.jar')
[ERROR] e4-cookbook-basic-recipe-tycho-pde-bnd/org.fipro.eclipse.tutorial.service.inverter/src/org/fipro/eclipse/tutorial/service/inverter/impl/InverterServiceImpl.java:[25] 
[ERROR] 	data.put(EventConstants.EVENT_TOPIC, topic);
[ERROR] 	         ^^^^^^^^^^^^^^^^^^^^^^^^^^
[ERROR] Access restriction: The type 'EventConstants' is not API (restriction on classpath entry 'repository/p2/osgi/bundle/org.osgi.service.event/1.4.1.202109301733/org.osgi.service.event-1.4.1.202109301733.jar')
[ERROR] e4-cookbook-basic-recipe-tycho-pde-bnd/org.fipro.eclipse.tutorial.service.inverter/src/org/fipro/eclipse/tutorial/service/inverter/impl/InverterServiceImpl.java:[25] 
[ERROR] 	data.put(EventConstants.EVENT_TOPIC, topic);
[ERROR] 	                        ^^^^^^^^^^^
[ERROR] Access restriction: The field 'EventConstants.EVENT_TOPIC' is not API (restriction on classpath entry 'repository/p2/osgi/bundle/org.osgi.service.event/1.4.1.202109301733/org.osgi.service.event-1.4.1.202109301733.jar')
[ERROR] e4-cookbook-basic-recipe-tycho-pde-bnd/org.fipro.eclipse.tutorial.service.inverter/src/org/fipro/eclipse/tutorial/service/inverter/impl/InverterServiceImpl.java:[27] 
[ERROR] 	Event event = new Event(topic, data);
[ERROR] 	^^^^^
[ERROR] Access restriction: The type 'Event' is not API (restriction on classpath entry 'repository/p2/osgi/bundle/org.osgi.service.event/1.4.1.202109301733/org.osgi.service.event-1.4.1.202109301733.jar')
[ERROR] e4-cookbook-basic-recipe-tycho-pde-bnd/org.fipro.eclipse.tutorial.service.inverter/src/org/fipro/eclipse/tutorial/service/inverter/impl/InverterServiceImpl.java:[27] 
[ERROR] 	Event event = new Event(topic, data);
[ERROR] 	                  ^^^^^
[ERROR] Access restriction: The type 'Event' is not API (restriction on classpath entry 'repository/p2/osgi/bundle/org.osgi.service.event/1.4.1.202109301733/org.osgi.service.event-1.4.1.202109301733.jar')
[ERROR] e4-cookbook-basic-recipe-tycho-pde-bnd/org.fipro.eclipse.tutorial.service.inverter/src/org/fipro/eclipse/tutorial/service/inverter/impl/InverterServiceImpl.java:[27] 
[ERROR] 	Event event = new Event(topic, data);
[ERROR] 	                  ^^^^^
[ERROR] Access restriction: The constructor 'Event(String, Map<String,?>)' is not API (restriction on classpath entry 'repository/p2/osgi/bundle/org.osgi.service.event/1.4.1.202109301733/org.osgi.service.event-1.4.1.202109301733.jar')
[ERROR] e4-cookbook-basic-recipe-tycho-pde-bnd/org.fipro.eclipse.tutorial.service.inverter/src/org/fipro/eclipse/tutorial/service/inverter/impl/InverterServiceImpl.java:[29] 
[ERROR] 	eventAdmin.postEvent(event);
[ERROR] 	           ^^^^^^^^^
[ERROR] Access restriction: The method 'EventAdmin.postEvent(Event)' is not API (restriction on classpath entry 'repository/p2/osgi/bundle/org.osgi.service.event/1.4.1.202109301733/org.osgi.service.event-1.4.1.202109301733.jar')
[ERROR] 10 problems (10 errors)

I now changed

-buildpath: \
  org.osgi.service.event;version=latest

to

-buildpath: \
  org.osgi.service.event

and the build passes, so it looks like Tycho is not correctly considering the version attribute on buildpath entry.

[fipro78]

Everytime I do a Clean - Build of all projects in the workspace, the consumer bundle fails, stating it can not resolve the exported package. I need to only Clean - Build the consumer bundle to make it resolve in the IDE. Seems like a timely dependency resolution issue somehow. Because the generated MANIFEST.MF always looks the same.

[laeubi]

Seems that this information is not correctly considered in the pde-bnd integration?

Tycho generates a preliminary Manifest based on the sources to discover the required / provided packages, it seems I need to enhance this to take annotations into account...

[laeubi]

@fipro78 I have checked this and it is actually already supported, no idea why it caused issues before (maybe becuase of the faulty classpath) but now it seems all working fine for your example!

[fipro78]

Also inside the ide? There I get the issues on clean build, not while building with tycho.

[laeubi]

I first have focused on the Tycho part as you previously mentioned it work in PDE :-)
I'll now try to take a look at PDE what might causing this.

[laeubi]

I have looked into the IDE problem now and it seems some kind of update problem, what I do is:

1. Perform "Project > Clean" and clean all projects, the project shows red
2. Now I open the InverterPart and go to the InverterService filed and pres F3, eclipse correctly jumps to the InverterService source
3. If I now switch back to the InverterPart editor tab the error goes away

So I'm not sure if its a PDE Issue, a JDT issue or a combination. Interestingly the Plugin Dependencies always already show the org.fipro.eclipse.tutorial.service.inverter as a dependency so it is not that something is really missing.

[laeubi]

Also target platform state shows this correctly:

Interestingly I can "fix" the problem as well by deleting the manifest error markers, so it seems to be that the Manifestbuilder somehow gets in the way here.

[laeubi]

Actually any change on the project fixes the issue... also if I change the dependent project.

One can workaround it by using BUNDLE_ROOT_PATH= in org.fipro.eclipse.tutorial.service.inverter/.settings/org.eclipse.pde.core.prefs, what seems to happens here is that PDE is updating the model on resource changes. Now the file gets deleted and the model gets deleted (so far so good), then a new file is created, but because this happens in the same build no new resource change is triggered... strange enough the classpath is updated correctly....

[laeubi]

After a lot of debugging I found the cause to be that PDE sometimes set the classpath in a synchronous way, stupid enough this is decided whether or not a model is CHANGED even though the code says it should happen on Manifest changes.

In this case it now happens that as the manifest got deleted, this does not trigger a CHANGED but an ADDED event and therefore it is executed synchronous what has the effect that a classpath update does not trigger a project build because the workspace tree is locked! As soon as one changes it to be async processed everything works:

• Always run asnyc if the workspace tree is locked eclipse-pde/eclipse.pde#1261

[laeubi]

@fipro78 can you try out the latest SDK build if it fixes your issues?

[fipro78]

@laeubi

I just downloaded the latest SDK build and played around with my example. Inside the IDE it looks good now. So with regards to the PDE integration it looks good for the moment. I will give further feedback while playing around with the feature.

I also tried out the latest Tycho 4.0.8-SNAPSHOT to build. The version String seems to be interpreted now correctly. But the Tycho build still does not work correctly with the @Export annotation.

The Tycho build only succeeds if I add Export-Package: org.fipro.eclipse.tutorial.service.inverter;version="1.0.0" to the pde.bnd file. If I remove that statement and only have the package-info.java with the @Export annotation, I get the following error:

[ERROR] Cannot resolve project dependencies:
[ERROR]   Software being installed: org.fipro.eclipse.tutorial.inverter 1.0.0.qualifier
[ERROR]   Missing requirement: org.fipro.eclipse.tutorial.inverter 1.0.0.qualifier requires 'java.package; org.fipro.eclipse.tutorial.service.inverter [1.0.0,2.0.0)' but it could not be found

[laeubi]

Thanks for the feedback will need to investigate on the missing export here.
Can you check if

mvn clean
mvn clean install

works?

[fipro78]

I rarely use mvn clean install ... I typically only run mvn clean verify for building Eclipse projects. But the result stays the same.

And I just noticed that the resulting jar file (if I have the Export-Package statement in the pde.bnd) is invalid. It does not contain the MANIFEST.MF file that it shows in the IDE, e.g. it misses several Import-Package entries, the Service-Component entry and the Provide-Capability header. Also the OSGI-INF folder with the service component definitions is missing.

Am I missing something in the pde.bnd that might cause this?

[laeubi]

@fipro78 I really mean a single mvn clean and then any other (maybe mvn clean verify) I don't know why but on local tests noticed that sometimes there seems to be some old / intermediate manifest state lingering around.

[fipro78]

I did exactly this. First a mvn clean which succeeds, then a mvn clean verify which fails with the above error.

Adding the Export-Package to the pde.bnd the build always succeeds, but produces the incorrect bundle jar file.

[fipro78]

@laeubi
While I created the demo app with the current SDK build, I noticed a new issue. Maybe related to your latest changes. If you leave out the version from the bundles in the buildpath I sometimes get a "version=0 Not found"

I have created a demo similar to my cookbook example. One difference is that there is now a service API bundle that makes use of the pde.bnd mechanism, and a service impl bundles that makes use of it. The ui bundle is PDE style and consumes the service API.

The PR is here: #3845

If I only specify the export via package-info.java, the Tycho build fails, saying in the UI bundle that it cannot resolve tycho.demo.service.api. If I add Export-Package: tycho.demo.service.api;version="1.0.0" to the pde.bnd of the API bundle, the build succeeds.

But in any case, the application does not work, because the resources in the generated JAR files do not contain the expected resources. There are no service component definitions in an OSGI-INF folder, and also the included MANIFEST.MF is the one you can see in the IDE when opening the pde.bnd.
Also when I start the application from the product definition, I see exceptions saying the required service is not available.

[laeubi]

Thanks thats very useful, that way we can later enhance the demo to show new issues / problems. I hope to find time looking into it and fixing those in the next days.

[laeubi]

I have now analyzed this and it is an issue with the Bnd Analyzer that leads to the export annotations we discover to be thrown away, sadly Analyzer#getExportedByAnnotation is a private method so we can not easily fix that, I'll try to came up with a workaround.

PR is here:

• Synthesize a (bnd) clazz from the JDT model to support export annotation #3852

[laeubi]

I have just changed your demo that it automatically starts the impl bundle and then I can start the application:

[fipro78]

Oh sorry, I have too many workspaces open where I play around with this. Seems I missed it in this one. :(
I will add it to the pde.bnd and update the PR.

Should I force update to have a single commit or multiple commits?

[laeubi]

No Problem, single commit is fine! btw the fix is merged now and should shortly approach in the snapshots.

[laeubi]

@fipro78 thanks for providing a demo, I have now integrated it with the automatic test-suite and added some basic verification:

• Run the 'pde-automatic-manifest' demo as part of integration tests #3856

the snapshots should already be deployed, yo if you see any issues let me know (and of course enhance either the demo or the test assertions).

[fipro78]

@laeubi thanks for working on this.

I have tested it now locally with the pde-automatic-manifest demo.

I downloaded the latest I-build and imported the pde-automatic-manifest workspace. I trigger a "Clean - Build" process, and then I see a compile error, saying the package tycho.demo.service.api can not be resolved. If I only "Clean" the tycho.demo.service.impl project, the issue goes away. Probably still some issue with the order of the processing?

I then tried to run a Tycho build, and get the following error:

[ERROR] Failed to execute goal org.eclipse.tycho:tycho-compiler-plugin:4.0.8-SNAPSHOT:validate-classpath (default-validate-classpath) on project tycho.demo.inverter.ui: Execution default-validate-classpath of goal org.eclipse.tycho:tycho-compiler-plugin:4.0.8-SNAPSHOT:validate-classpath failed: org.osgi.framework.BundleException: Bundle tycho.demo.inverter.ui cannot be resolved:tycho.demo.inverter.ui [52]
[ERROR]   Unresolved requirement: Import-Package: tycho.demo.service.api; version="[1.0.0,2.0.0)"

Opening the generated api and impl jars, I still miss the correct MANIFEST.MF files and component xml files. I used the 4.0.8-SNAPSHOT. Are the fixes you applied backported, or do I need to test this with a 5.0.0-SNAPSHOT?

There is also a test failure for the 4.0.x, which show exactly the same error:
https://github.com/eclipse-tycho/tycho/actions/runs/9131948328/job/25115578492

[fipro78]

Interestingly it only fails on Windows do you have a chance to try it out under Linux/mac?

I just tried in on a Ubuntu WSL and the build succeeds. On Windows the failure happens on every build.

Seems to be an issue with the target path depth. I tried to copy the pde-automatic-manifest folder to a higher level folder (C:/tmp) and even the copy operation fails with the message that org.eclipse.ecf.filetransfer.httpclient5.feature path is too long. Seems we hit the path length issue on Windows :(

BTW, the build fails even then on Windows.

[laeubi]

Can you explain where exactly there is a path that is too long?

If I look at the output on the CI I see

[DEBUG] Generated preliminary manifest for tycho.demo:tycho.demo.service.api:eclipse-plugin:1.0.0-SNAPSHOT:
Manifest-Version: 1.0
Bundle-ManifestVersion: 2
Bundle-Name: Api
Bundle-SymbolicName: tycho.demo.service.api
Bundle-Version: 1.0.0.qualifier
Export-Package: tycho.demo.service.api;version="1.0.0"

So the exported package is actually generated as expected. Seems I need add some more debug for the final processing.

[laeubi]

The enhanced output reveled an issue with \ versus / resulting in:

[ERROR] Classes found in the wrong directory: {tycho\demo\service\api\package-info.class=tycho.demo.service.api.package-info, tycho\demo\service\api\InverterService.class=tycho.demo.service.api.InverterService}

so it seems rather trivial to replace them with something BND like more...

[fipro78]

Can you explain where exactly there is a path that is too long?

It was an assumption because the copy operation failed with that message. But it seems that is not the issue as you found out. 50/50 chance for Windows build is either path-too-long or slash-backslash issues. :)

Let me know if you have something I can test.

[laeubi]

Let me know if you have something I can test.

The fix is now merged and backported, so it finally should work also under windows!