Generate Wt bindings

[LorenDB]

As referenced in #16, I'm hoping to be able to use this app to generate a Wt binding.

Some things to take note of:

• Wt makes extensive use of the STL.
  • The STL is mainly used for std::unique_ptr with a few std::strings and such thrown in. Probably nothing that can't be worked around with core.stdcpp.
• Wt uses Boost, but only internally.
• Wt has a lot of classes/headers.
• The Java version of Wt (Jwt) is (according to an Emweb employee) not so much a binding as it is a source translation from C++ to Java.

First off, I need to figure out how the configuration file is set up.

[Superbelko]

In order to generate bindings this tool requires some knowledge of the project written in form of JSON configuration description.

It tells the tool basic information about the source code to work on - where to look for extra headers, which compiler flags to use, and macro definitions that are often used as a basic conditional compilation and affects what symbols are present or omitted.

Let's first look at the most basic configuration: (more info can be found at QuickStart)

basic.json:

{
    "version": 1,
    "input": {
        "std": "c++11",
        "paths": ["../include"],
        "includes": ["../include/"],
        "system": [],
        "defines": [],
        "cflags": []
    },
    "output": {
        "path": "generated.d",
        "target": "dlang",
        "attr-nogc": false
    }
}

This configuration provides knowledge about the source code for which bindings needs to be generated. It defines an input(C++ source) and (produced) output.

Input

Input entry describes a set of headers("includes"), "system" headers (STL and compiler specific's, optional), macro defines and compiler flags (cflags).
In this category "paths" serves as a filter - It can be a file or a folder. Empty "paths" will produce no output at all.
It is used to control the scope of what is going to be emitted by bindings generator, currently this is the only way to remove unneeded symbols and simply works as whitelist.
(IIRC in latest versions it supports globbing)

"includes" tells the generator where to look for both project and 3rd party headers, it often have the same folder as paths as the root.

"defines" category takes macro definitions used by C++ preprocessor, it takes array of strings in "name=value" format where value is an optional value for the item, this is very important as it greatly affects resulting configuration, some projects won't even work if some defines is not set (e.g. PhysX with its platform configuration)

"cflags" C++ compiler flags (in clang format), in most cases affects compiled code but not definitions and so can be left empty

Output

At this moment there is few options, the most important one is target file "path" that will contain generated bindings. (at the moment of writing only single file output is supported)

"target" - currently only "dlang" target is supported

There is also set of optional attributes such as "skip-bodies" and "mangle-all" that can be useful in some cases. See QuickStart for more info.

Back to Wt

Let's take a look at Wt source
https://github.com/emweb/wt

From example folder we see that the core project consists of <Wt/*.h> includes, generally such includes resides in dedicated "include" folder at the project root, adding it to the includes and paths should solve the problem, however Wt in that regard is a bit messy as it keeps everything in one folder. Anyway...

In case of Wt public headers is actually directly under src/ folder

So the config will look like

wt.json:

{
    "version": 1,
    "input": {
        "std": "c++11",
        "paths": ["../src/Wt"], // Note that for sharing purposes you can use env variables, also note that it is limited to Wt subfolder
        "includes": ["../src"]
        "system": [],
        "defines": [], // <- will likely won't work without set of feature defines (the ones that makes up logical configuration in project)
        "cflags": []
    },
    "output": {
        "path": "generated.d",
        "target": "dlang",
        "attr-nogc": false
    }
}

It might work as is, however very likely you will have to grab defines from your build system cache and/or including compiler defines(for gcc for example gcc -dM -E will dump macros).

In case of cmake you can dig through CMakeCache.txt and probe files to find anything relevant.

example command-line for MSVS compiler

 cl /c /Zi /W3 /WX- /diagnostics:column /Od /Ob0 /D _MBCS /D WIN32 /D _WINDOWS /D "CMAKE_INTDIR=\"Debug\"" /Gm- /RTC1 /MDd /GS /fp:precise /Zc:wchar_t /Zc:forScope /Zc:inline 

here it adds several defines (staring with /D prefix, -D for other compilers): "_MBCS" "WIN32" "_WINDOWS", it can also has values "CMAKE_INTDIR="Debug""

From a real example, for PhysX I had to put these defines, but later commented out unneeded stuff

        "defines": [
            "_MSC_EXTENSIONS",
            "_INTEGRAL_MAX_BITS=64",
            //"_MSC_VER=1915",
            //"_MSC_FULL_VER=191526726",
            //"_MSC_BUILD=0",
            "_M_AMD64=100",
            "_M_X64=100",
            "_WIN64",
            "_WIN32",
            "_CPPRTTI",
            "_MT",
            "_CRT_SECURE_NO_DEPRECATE",
            "_CRT_NONSTDC_NO_DEPRECATE",
            "NDEBUG",
            //"PX_VC=15",
            //"PX_DEBUG=0",
            //"PX_CHECKED=1",
            //"PX_NVTX=1",
            //"PX_SUPPORT_PVD=1"
        ],

After invoking gentool wt.json it should produce D bindings for Wt, after that due to STL/Boost usage you'll probably have to do some manual fixes replacing some types/signatures.

I hope that helps.
Regarding the manual fixes however I can only tell after seeing actual result and error messages.

[Superbelko]

I just noticed that
cmake --build path --config Release
doesn't works on Linux (probably only ever works with Visual Studio builds), so to build release builds I have to do this when generating build files

cmake .. -DCMAKE_BUILD_TYPE=Release
cmake --build .

Using this will probably allow you to build C++ release build and use it until static linking will be fixed.

[LorenDB]

That unfortunately still crashes.

[Superbelko]

I did a quick fix for this memory issue, there is still few places though where it crashes (templates...) and it failing the tests, more patches coming soon (tm)

[LorenDB]

Wow, it's been a while since I looked at this...

Running gentool with the latest commits produces the following output: log.txt. This certainly is a step forward from what had been happening, at any rate!

[Superbelko]

At this point you will need to bump c++ std version in json config to at least c++17 or even c++20 otherwise compiler will not recognize that syntax.

I tried running it on Wt and I see there is some compile-time constructs like advanced decltype, template tuples, and SFINAE kicks-in that are not handled yet and so had to be fixed by hand, the rest of it looks ok though.

Of course there is always chance of minor name mangling mismatches in D compiler that'll give link errors. But in any case this is a good starting point for manual tweaks here and there and it should work, there is no other tool that does bulk of the job better than this anyway.