Few questions about the project nature

[deadlocklogic]

Hi, I have few curious questions about this project, I kept wondering about.
While I find this project really interesting with a huge potential, there are few answered remarks/questions:

• mono is the owner of this repository, but the project isn't being actively developed. Is the development now in a stagnant/contributor oriented state? Or did they lose interest in it? Considering that this project is relatively old.
• Why many critical issues (memory leaks) regarding the C# generator hasn't been resolved (in issues and PR).

Is the working force the main problem?

Plans for future:

• Work towards a cleaner/better interface API
• Standardize the inner architecture of the project (which I am working on step by step)
• Spread the word when creating new generators (I will do after finishing a couple of them)
• Motivate scripting languages users/maintainers to create their own generators.
• Motivate the usage of the project as a tool for sanitization etc...
• When more contributors pour in, we can create a roadmap/contribution guide, so we can let the community help push the project even more: documentation, readme, tests, showcase etc...

I have more questions, some are technical but will keep them for later.

Binding generation should become an easy task, especially when so many helper libraries are available. SWIG is good overall but has poor support for many languages and harder to extend, which we will have advantage over.
Thanks.

[tritao]

Hi, I have few curious questions about this project, I kept wondering about. While I find this project really interesting with a huge potential, there are few answered remarks/questions:

To understand the project, we will have to look at the historical context. The project was started by me back in 2012 after trying to use both SWIG and the mono/CppInterop projects to bind a game engine and being disappointed by both. I forked CppInterop at the time and replaced the generator with Clang (it used GCC-XML) which resulted in pretty much rewriting 99.9% of the code.

I solved the problem I had in that summer, with the first backend, C++/CLI. Eventually the project got noticed and I started working at Xamarin at the time and was contracted to keep working on the project (named Cxxi at the time) and build it up to bind the Blackberry platform, which required building a new C# backend.

At this time I think it was renamed to CppSharp and Dimitar Dobrev joined the project as an open source contributor, he was binding Qt with his QtSharp project. Dobred quickly became a main contributor of the project, and was the main driver of most of the more complex features of CppSharp, like its template support. At this time I invited him as an admin, and we shared administrative burden of the project.

After Xamarin was bought by Microsoft, I had another stint at developing something related to this project (Embeddinator-4000), which benifited CppSharp a little bit too since it built upon it. But a little later that project was canceled by management, which left CppSharp again without any support.

To make matters worse, Dimitar Dobrev got hit by COVID during the pandemic and died suddenly and unexpectedly, another big hit to the project, his last words to me during his hospital bed were to keep our baby alive, this was how much he cared for this project.

Meanwhile, Jose Torres, an external contributor at the time, also helped a lot, fixing a ton of bugs and improving the packaging process, and has helped on some support work for external companies using the project.

• mono is the owner of this repository, but the project isn't being actively developed. Is the development now in a stagnant/contributor oriented state? Or did they lose interest in it? Considering that this project is relatively old.

mono being the owner of this repository is mostly an historical artifact, the project is de facto owned by me (and Dimitar, Jose), we could transfer it to another organization, but no clear benefits have occured to me for doing so, so the status quo has prevailed so far.

About development of the project, I think saying it's not actively developed is not strictly fair, we have just recently released a new version with VS2022 support and updating to Clang 16, which took some non trivial effort to do. Maybe the development pace is not as it once was, but it's still being developed.

• Why many critical issues (memory leaks) regarding the C# generator hasn't been resolved (in issues and PR).

Mainly because my focus has shifted from C++ and C# to other backends, and making sure those pending PRs are fully correct and do not break anything else takes days of work. And so far no one else has stepped up to the plate to fix them.

Is the working force the main problem?

Yes, for sure. Historically we have never had many contributors, despite being very friendly to contributors. We did participate in Google Summer of Code for some summers, which helped in getting new contributions, but also took a lot of time in mentoring the students.

Plans for future:

• Work towards a cleaner/better interface API
• Standardize the inner architecture of the project (which I am working on step by step)
• Spread the word when creating new generators (I will do after finishing a couple of them)
• Motivate scripting languages users/maintainers to create their own generators.
• Motivate the usage of the project as a tool for sanitization etc...
• When more contributors pour in, we can create a roadmap/contribution guide, so we can let the community help push the project even more: documentation, readme, tests, showcase etc...

Overall I agree with those. If you would like to help with that I am here to support, I can help but I won't be the main driver for new development at this time. I think there is value to having most generators included as part of the project though, so if we can figure out a way for external contributors to own official generators in the project tree, that would be great. We can change the governance structure of the project for instance, but for that interested people like you need to come up first, most people just disappear after they solve their original issue 😄

[deadlocklogic]

Wow! What a roller-coaster of an answer. I didn't expect such a heavily detailed answer.
Sorry for the loss of the guy, he was a top contributor and deserves his contribution to be alive.
I totally agree that many contributors are casual tourists who will leave as soon as they solve their issues or lose interest.
The irony is I am kinda reliving your own story, I am too building an engine/framework which needs bindings for plugins/scripting support, and found out that SWIG is a pain for many languages.
So I settled to use libclang and create my own generator, and when I had all the ideas planned I stumbled upon cppast which is a fantastic but now deprecated library, which took all hassle out of many libclang limitations.
My primary goal was to build a binding generator mainly for lua using sol2, and with many trial and error, refactoring I had working a solution which is very reliable.
My whole infrastructure was written in lua. So similarly to CppSharp after many iterations, I had a mechanism which generates the lua bindings for cppast already in place and part of the build system.😂
So my workflow was:

1. choose a library to bind
2. generate bindings
3. compile/test
4. apply needed changes
5. re-ititerate with previously binded library

After a while I decided to add support for python with pybind11, so I heavily refactored the project to be easily extended.
All was going smooth and fun until the maintainer of cppast decided to end support for the library, which was failing its tests with the latest versions of clang.
Here I panicked and had 2 main options:

1. fork the library myself and become the new maintainer
   • Pro: I am the main leader there, keeping the same development environment
   • Con: I am the main leader there 😂, more time to maintain
2. find another solution
   • Pro: I am not the main leader there
   • Con: too few alternatives, heavy migration expected

And the funny part is that my project structure, after doing a quick comparison, looked quite similar to CppSharp one, even though I never ever heard about the latter before!

• CppSharp uses Generator, in my case Package
• CppSharp uses CodeGenerator, in my case Processor
• CppSharp uses Pass, in my case Task
• CppSharp uses the clang AST, in my case libclang AST which are quite similar with minor differences in template representation

So CppSharp was the trivial solution, farewell to poor libclang's template support!
Now I migrated my infrastructure to a compatible API, I can start slowly the migration to C# in a PR.
The only requirements which concern me so far, and which I opened already are:

1. Implement UsingDecl
2. Missing template specialization parameters
3. CXXDeductionGuide
4. Using a precompiled header

Bonus:

• I can help with fixing issues wherever I can but so far my main obstacle is in native debugging in absence of llvm debug binaries.
• While analyzing the ASTConverter.cs file I noticed that the workflow is quite redundant: why not just bind the whole or even partial clang API to C# and build on top of it?

Appreciate it.

[deadlocklogic]

@tritao I have been wondering on the best way to add new declaration properties for new generators:

1. create a kind of database in the generator with query by declaration entries
2. make the Declaration class partial and add newer properties where needed

The option 2 is only available because it is a C# feature, while option 1 might be the universal solution but requires more engineering to afford newer generators.

I will try using a key/value dictionary-like structure with an auto caster Get method.

[tritao]

I think using CppSharp was a good call, it should save you a significant chunk of time. And it will be really nice to see Lua and Python binding generators, I have always wanted to add support for those languages! Would be great to see them inside the CppSharp repo if you would like to so.

The only requirements which concern me so far, and which I opened already are:

Which one is currently the most pressing for you?

• While analyzing the ASTConverter.cs file I noticed that the workflow is quite redundant: why not just bind the whole or even partial clang API to C# and build on top of it?

This is partly for historic reasons again.

The first iteration of CppSharp actually just bound some Clang APIs directly with C++/CLI and that worked great, but eventually when we wanted to support Linux we had to migrate over to C#. The easiest way to get there was with the current architecture. But over the years, we developed a new system to auto-generate some of that code. See the https://github.com/mono/CppSharp/blob/main/src/CppParser/Bootstrap/Bootstrap.cs project.

There is another problem with just binding native APIs, which is that we lose type information for identity checks when returning a base class. For instance, take this imaginary Clang API, Decl* GetDecl() { return new FunctionDecl(); }, when it gets to C#, Decl is FunctionDecl returns false, which is just broken. I think modern .NET/C# provides some way to hook into those object identity checks, which they added for COM, so maybe it could be solved. Either way, I don't see the current architecture as big a deal as its stable and rarely gives problems, its easy to mantain, and it's actually pretty nice that we get a fully managed AST layer after parsing.

I will try using a key/value dictionary-like structure with an auto caster Get method.

I think this is fine, interested to see what you come up with!

[deadlocklogic]

I think using CppSharp was a good call, it should save you a significant chunk of time. And it will be really nice to see Lua and Python binding generators, I have always wanted to add support for those languages! Would be great to see them inside the CppSharp repo if you would like to so.

I already begun the migration, and now I am pretty much familiar with CppSharp API. Soon you will see for yourself!

Which one is currently the most pressing for you?

Well not necessarily pressing, but as the project develops such parser related issues should be solved at some point. But 2, 3, 4 and currently a barrier which doesn't allow the usage of CppSharp with some libraries using PCH or C++17 deduction guide.
I am used to libclang which has the only advantage over CppSharp of rarely aborting the compilation even when parsing errors occur, in trade for not so accurate type resolving. CppSharp aborts completely if any errors occur. It is expected because otherwise the parsing metadata won't be accurate.
By the way, I was trying to use CppSharp to create tools/bindings to a C++ game engine called rbfx and I thought maybe its community may become interested in this project too (drive more contributors), but I was hit by these errors which won't help my case anytime soon.😂

For the rest, we can discuss things along the way and decide on what is best.

I think this is fine, interested to see what you come up with!

You won't be disappointed. I know I took my time but had to carefully refactor my primary project to obtain a matching API.

Just to note, the current project structuring via premake is a bit annoying. I wanted to create a new module for registration-based generators and add the corresponding ones but found out this could mess up the build system. So my best bet is to develop in the Generators folder and then decide on the better structure.

[deadlocklogic]

@tritao Long time, sorry for that.
In the past time, while I've been too busy, I have been experimenting with clang frontend directly without the C# layer which CppSharp uses. I've built something in the spirit of CppSharp still in early phase.
The main problem which made me think about moving from CppSharp:

1. Structure: the project focuses on decoupling its code from clang, hence it has its own parallel AST, which looks nice but really suffers when you need advanced precise semantic analysis already available on clang side.
2. Process time: the newly created AST is generated by visiting every other correspondent construct in clang AST, which is unnoticeable for small input, but almost impossible for larger libraries with many dependencies.
   Example: a library which I was trying to process took 2 hours and all of the memory (a 20GB RAM which froze my laptop), the funny part is that the very same library takes very few seconds to parse and process in the C++ frontend.
3. Advanced semantics: now most binding generators are satisfied with the generation of main C++ constructs, except templates which they generally ignore or poorly support. But in practice, most concrete template types can be instantiated easily using the frontend, and CppSharp ignores this matter while it is very important for binding generation for tools like SWIG.

Still, this doesn't mean that switching to C++ will solve everything there, I still have many questions/challenges which I am trying to navigate and find solutions for.
The project is nice, and as I told you before I have a very similar project based on clang cindex which suffers from inferior AST capabilities.
I hope using the C++ frontend could solve the missing pieces of the puzzle.