fatal error: 'memory' file not found

[jeromegn]

Note: I've successfully got bindgen to work under macos. I had to modify the scripts a bit, but I was able to compile clang and everything. I just couldn't get -ltinfo, I just removed it for now.

I don't think I'll succeed, but I'm trying to generate bindings for libv8.

One of the first things it includes is <memory>. Which is part of the stdlib. My .yml config looks like this:

module: V8
library: "%/ext/binding_{BINDING_PLATFORM}.a -lstdc++ -lv8_base -lv8_init -lv8_initializers -lv8_libbase -lv8_libplatform -lv8_libsampler -lv8_nosnapshot"

processors:
  # Graph-refining processors:
  - default_constructor # Create default constructors where possible
  - function_class # Turn OOP-y C APIs into real classes
  - inheritance # Mirror inheritance hierarchy from C++
  - copy_structs # Copy structures as marked
  - macros # Support for macro mapping
  - functions # Add non-class functions
  - filter_methods # Throw out filtered methods
  - extern_c # Directly bind to pure C functions
  - instantiate_containers # Actually instantiate containers
  - enums # Add enums
  # Preliminary generation processors:
  - crystal_wrapper # Create Crystal wrappers
  - virtual_override # Allow overriding C++ virtual methods
  - cpp_wrapper # Create C++ <-> C wrappers
  - crystal_binding # Create `lib` bindings for the C wrapper
  - sanity_check # Shows issues, if any

generators:
  # C++ generator
  cpp:
    # Output file path  (Mandatory)
    output: ext/my_bindings.cpp
    # Output file preamble  (Optional)
    preamble: |-
      #include "bindgen_helper.hpp"
  crystal:
    # You'll most likely only need the `output` option.
    output: src/v8/binding.cr
  
parser:
  # List of files to include.  Can be relative to search-paths.
  # This is the only required option:
  files:
    - stdlib.h
    - v8.h
  flags: [ "-x", "c++", "-std=c++11" ]

I get the following error:

$ crystal ./lib/bindgen/src/bindgen.cr -- --chdir $(pwd) v8.yml
In file included from /var/folders/wd/y9m3t11s6nx1nhyzqmhnxmr40000gn/T/bindgen.GKCo5a:3:
/usr/local/include/v8.h:21:10: fatal error: 'memory' file not found
#include <memory>
         ^~~~~~~~

How can I make this work? I tried adding memory.h to my files and it didn't work.

[Papierkorb]

Hello,

The parser.files section shouldn't need memory, as its a system header (Not memory.h, the C++ stdlib headers don't have any ending). That section is only needed to tell Bindgen (Actually the clang parser part) where to find the declarations. If a normal C++ program only needs to #include <v8.h>, then that's all you need too.

The error looks like the clang part doesn't find its own system headers. Make sure that you don't move (copy, rename, etc.) anything related to clang (Both in your systems clang and bindgen).

Assuming you didn't move files, you can dive into what's going on by setting the VERBOSE environment variable when running bindgen: VERBOSE=1 lib/bindgen/tool.sh v8.yml

That'll then proceed to tell you the (possibly long) command it's running to call out to the clang part. With that in hand, you can then check why it's not finding its system headers. That command should understand the standard clang arguments.

A quick try would be to add the path to your system includes using -I.

This is however an artifact of bindgen not having a MacOS port. Afaik, the clang on MacOS wants some additional include paths (compared to on Linux), which the build script doesn't know about. The build script for that is clang/find_clang.cr.

On that note, does it work at all? Is the test suite running (Standard crystal spec for that) fine?

[jeromegn]

This is what I get when I run it with VERBOSE=1

Runner command: /Users/jerome/projects/github.com/superfly/v8-cr/lib/bindgen/src/bindgen/parser/../../../clang/bindgen /var/folders/wd/y9m3t11s6nx1nhyzqmhnxmr40000gn/T/bindgen.tfMeLu -m "" -f "" -- -x c++ -std=c++11 -D__STDC_CONSTANT_MACROS -D__STDC_LIMIT_MACROS
In file included from /var/folders/wd/y9m3t11s6nx1nhyzqmhnxmr40000gn/T/bindgen.tfMeLu:2:
/usr/local/include/v8.h:21:10: fatal error: 'memory' file not found
#include <memory>
         ^~~~~~~~

I can't run using the lib/bindgen/tool.sh directly because it somehow is broken. Maybe just a macos thing.

Error: File ./lib/bindgen/src/bindgen.cr does not exist

[jeromegn]

As for the spec, I got this:

$ crystal spec
.................................................................................................................................................................................................................................................................................................................................In file included from /var/folders/wd/y9m3t11s6nx1nhyzqmhnxmr40000gn/T/bindgen.FBLx6t:1:
/Users/jerome/projects/github.com/superfly/v8-cr/lib/bindgen/spec/integration/arguments.cpp:1:10: fatal error: 'string' file not found
#include <string>
         ^~~~~~~~
arguments.cpp:22:32: error: no viable conversion from 'int' to 'std::string' (aka 'basic_string<char, char_traits<char>, allocator<char>
      >')
  return _self_->defaultString(str);
                               ^~~
/Applications/Xcode.app/Contents/Developer/Toolchains/XcodeDefault.xctoolchain/usr/include/c++/v1/string:763:5: note: candidate
      constructor not viable: no known conversion from 'int' to 'const std::__1::basic_string<char> &' for 1st argument
    basic_string(const basic_string& __str);
    ^
/Applications/Xcode.app/Contents/Developer/Toolchains/XcodeDefault.xctoolchain/usr/include/c++/v1/string:768:5: note: candidate
      constructor not viable: no known conversion from 'int' to 'std::__1::basic_string<char> &&' for 1st argument
    basic_string(basic_string&& __str)
    ^
/Applications/Xcode.app/Contents/Developer/Toolchains/XcodeDefault.xctoolchain/usr/include/c++/v1/string:778:31: note: candidate
      constructor not viable: no known conversion from 'int' to 'const value_type *' (aka 'const char *') for 1st argument
    _LIBCPP_INLINE_VISIBILITY basic_string(const value_type* __s);
                              ^
/Applications/Xcode.app/Contents/Developer/Toolchains/XcodeDefault.xctoolchain/usr/include/c++/v1/string:810:5: note: candidate
      constructor not viable: no known conversion from 'int' to 'initializer_list<value_type>' (aka 'initializer_list<char>') for 1st
      argument
    basic_string(initializer_list<value_type> __il);
    ^
../arguments.cpp:21:33: note: passing argument to parameter 'str' here
  int defaultString(std::string str = "Okay") {
                                ^
1 error generated.
Build step failed!
  Directory: /Users/jerome/projects/github.com/superfly/v8-cr/lib/bindgen/spec/integration/tmp
  Command: c++ -std=c++11 -c -o arguments.o arguments.cpp -I.. -Wall -Werror -Wno-unused-function
FIn file included from /var/folders/wd/y9m3t11s6nx1nhyzqmhnxmr40000gn/T/bindgen.f4tuQd:1:
/Users/jerome/projects/github.com/superfly/v8-cr/lib/bindgen/spec/integration/basic.cpp:1:10: fatal error: 'cstdio' file not found
#include <cstdio>
         ^~~~~~~~
.In file included from /var/folders/wd/y9m3t11s6nx1nhyzqmhnxmr40000gn/T/bindgen.uopdlq:1:
/Users/jerome/projects/github.com/superfly/v8-cr/lib/bindgen/spec/integration/c_only.cpp:1:10: fatal error: 'stdarg.h' file not found
#include <stdarg.h>
         ^~~~~~~~~~
.In file included from /var/folders/wd/y9m3t11s6nx1nhyzqmhnxmr40000gn/T/bindgen.3EH4zC:1:
/Users/jerome/projects/github.com/superfly/v8-cr/lib/bindgen/spec/integration/c_wrapper.cpp:1:10: fatal error: 'cstdlib' file not found
#include <cstdlib>
         ^~~~~~~~~
.In file included from /var/folders/wd/y9m3t11s6nx1nhyzqmhnxmr40000gn/T/bindgen.T6tOpl:1:
/Users/jerome/projects/github.com/superfly/v8-cr/lib/bindgen/spec/integration/containers.cpp:1:10: fatal error: 'vector' file not found
#include <vector>
         ^~~~~~~~
containers.cpp:9:10: error: no viable conversion from returned value of type 'std::vector<int>' to function return type 'int'
  return _self_->integers();
         ^~~~~~~~~~~~~~~~~~
containers.cpp:13:10: error: no viable conversion from returned value of type 'std::vector<std::string>' (aka 'vector<basic_string<char,
      char_traits<char>, allocator<char> > >') to function return type 'int'
  return _self_->strings();
         ^~~~~~~~~~~~~~~~~
containers.cpp:17:22: error: no viable conversion from 'int' to 'std::vector<double>'
  return _self_->sum(list);
                     ^~~~
/Applications/Xcode.app/Contents/Developer/Toolchains/XcodeDefault.xctoolchain/usr/include/c++/v1/vector:530:5: note: candidate
      constructor not viable: no known conversion from 'int' to 'initializer_list<value_type>' (aka 'initializer_list<double>') for 1st
      argument
    vector(initializer_list<value_type> __il);
    ^
/Applications/Xcode.app/Contents/Developer/Toolchains/XcodeDefault.xctoolchain/usr/include/c++/v1/vector:542:5: note: candidate
      constructor not viable: no known conversion from 'int' to 'const std::__1::vector<double, std::__1::allocator<double> > &' for 1st
      argument
    vector(const vector& __x);
    ^
/Applications/Xcode.app/Contents/Developer/Toolchains/XcodeDefault.xctoolchain/usr/include/c++/v1/vector:548:5: note: candidate
      constructor not viable: no known conversion from 'int' to 'std::__1::vector<double, std::__1::allocator<double> > &&' for 1st
      argument
    vector(vector&& __x)
    ^
../containers.cpp:14:34: note: passing argument to parameter 'list' here
  double sum(std::vector<double> list) {
                                 ^
3 errors generated.
Build step failed!
  Directory: /Users/jerome/projects/github.com/superfly/v8-cr/lib/bindgen/spec/integration/tmp
  Command: c++ -std=c++11 -c -o containers.o containers.cpp -I.. -Wall -Werror -Wno-unused-function
F...

Failures:

  1) the argument translation functionality works
     Failure/Error: tool.run!.should eq(0)

       Expected: 0
            got: 1

     # spec/integration/spec_helper.cr:72

  2) container instantiation feature works
     Failure/Error: tool.run!.should eq(0)

       Expected: 0
            got: 1

     # spec/integration/spec_helper.cr:72

Finished in 16.44 seconds
329 examples, 2 failures, 0 errors, 0 pending

Failed examples:

crystal spec spec/integration/arguments_spec.cr:4 # the argument translation functionality works
crystal spec spec/integration/containers_spec.cr:4 # container instantiation feature works

Pretty good!

[Papierkorb]

/Users/jerome/projects/github.com/superfly/v8-cr/lib/bindgen/src/bindgen/parser/../../../clang/bindgen /var/folders/wd/y9m3t11s6nx1nhyzqmhnxmr40000gn/T/bindgen.tfMeLu -m "" -f "" -- -x c++ -std=c++11 -D__STDC_CONSTANT_MACROS -D__STDC_LIMIT_MACROS

That whole thing there is the invocation command of the clang part. Instead of

/var/folders/wd/y9m3t11s6nx1nhyzqmhnxmr40000gn/T/bindgen.tfMeLu

you can also just give it the path to your v8.h. Run that command, the error output should be the same what you got before.

If you can get that command to run, you'll get bindgen to work. So that's what you want to focus on. You can check clang/find_clang.cr on how it dissects your systems clang to get the information it needs. The gist is this:

1. Create an empty file ending in .cpp, or use an existing .cpp file. You just need a file ending in .cpp.
2. Run clang -"###" the_file.cpp. With that invocation, clang will dump the internal process invocations.
3. In that output there'll be a line showing a clang command, like: "/usr/bin/clang-6.0" "-cc1" "-triple" .... That's what you're interested in. Copy it into an editor for easier reading.
4. That line will contain quite a few arguments. For now, copy all arguments that mention a path as value.
5. Get back to the clang/bindgen command from before, and append all those to the invocation. Fire it off.
6. Ideally your problem is now fixed. If it works you should see a JSON document, which could get quite large.

With that we could come up with what to do next.

[jeromegn]

Ok, adding the resource-dir option fixed it, but the JSON is full of empty objects:

$ ./lib/bindgen/clang/bindgen /usr/local/include/v8.h -m "" -f "" -- -x c++ -std=c++11 -D__STDC_CONSTANT_MACROS -D__STDC_LIMIT_MACROS -resource-dir /usr/local/Cellar/llvm/5.0.1/lib/clang/5.0.1
{"enums": {}, "classes": {}, "macros": []}

[Papierkorb]

With that invocation that's an expected output. Bindgen doesn't "grab everything", only what you want it to. Please check the TEMPLATE.yml for further information.

[jeromegn]

Oh nice, adding one class produced a big JSON output.

I had to change the runner.cr to pass this argument I need to make it work with the usage from the README. I'm not sure where I should add it in the build process so I don't have to change the bindgen package.

It seems to work, there are a lot of coupling between classes in v8, I'm trying to find a simple class to get a "win" 😄Unfortunately I'm mostly getting errors likes:

Runner command: /Users/jerome/projects/github.com/superfly/v8-cr/lib/bindgen/src/bindgen/parser/../../../clang/bindgen /var/folders/wd/y9m3t11s6nx1nhyzqmhnxmr40000gn/T/bindgen.DRxuMl -c v8::Locker -m "" -f "" -- -x c++ -std=c++11 -D__STDC_CONSTANT_MACROS -D__STDC_LIMIT_MACROS -resource-dir /usr/local/Cellar/llvm/5.0.1/lib/clang/5.0.1
Invalid alias name "v8::Locker" at V8::Binding::v8::Locker
Invalid alias name "v8::Isolate" at V8::Binding::v8::Isolate
Argument 1 has unreachable type v8::Isolate at V8::Binding#initialize(isolate)
Result type v8::Locker is unreachable at V8::Binding#initialize(isolate)
Argument 1 has unreachable type v8::Isolate at V8::Binding.IsLocked(isolate)
Argument 1 has unreachable type Binding::v8::Isolate at V8::Locker#initialize(isolate)
Argument 1 has unreachable type Binding::v8::Isolate at V8::Locker.IsLocked(isolate)
Result type Binding::v8::Locker is unreachable at V8::Locker#to_unsafe()
Argument 1 has unreachable type Binding::v8::Locker at V8::Locker#initialize(unwrap)
Found 9 errors.  Aborting.

Most types are going to be void pointers I believe. I tried inspiring from the qt5 shard you made, but I don't fully grasp how things are parsed yet.

I added a bunch of ignored types in the v8.yml and brought it down to 2 errors, but it's hard to know what to do here.

Result type Binding::v8::Locker is unreachable at V8::Locker#to_unsafe()
Argument 1 has unreachable type Binding::v8::Locker at V8::Locker#initialize(unwrap)
Found 2 errors.  Aborting.

I feel like it might be easiest to write a C bridge! Although Qt must've been a massive effort and you seem to have done it.

[Papierkorb]

I had to change the runner.cr to pass this argument I need to make it work with the usage from the README.

The README is fun and all, but the main documentation is in the TEMPLATE.yml. Don't try to guess all of the functionality just from qt.yml. The TEMPLATE.yml is the definitive authority on what works, and how it works. In your case, that'd be the parser.flags array. This is a temporary fix though to make it work on MacOS. A proper fix would incorporate this ideally into the clang/find_clang.cr so it'd just work™ on MacOS.

I don't know anything about v8 (beyond its existence), so I don't know how it's meant to be used. But I recommend adding the absolutely necessary classes, that you'll actually interact with, first. Ignore all other types, do so explicitly where necessary (You did this already it seems). And then iteratively add classes as you need them, cleaning up ignored types as you go.

Invalid alias name "v8::Locker" at V8::Binding::v8::Locker

Qt doesn't use namespacing, so that case may not be handled nicely yet. That error basically warns that bindgen (with that configuration) was about to generate a Crystal line like alias v8::Locker = Void. That wouldn't compile anyway. You can 1) just ignore the type explicitly, or 2) add the class to the classes list, or 3) rename the type by setting the binding_type and crystal_type (I think it was) fields in the type configuration to provide a sane name.

The CrystalBinding processor is responsible for generating these aliases in any case.

Argument 1 has unreachable type Binding::v8::Isolate at V8::Locker#initialize(isolate)

Is an artifact of the error above, but do have have explicitly ignored the locker type, while having it in the classes list? If so, then yeah it's unreachable, as it doesn't exist on the binding level. Locker by name, and the API snippets from above, make it seem like it's a RAII locker class? That'll be fun, I'd ignore it for now. RAII doesn't translate well to Crystal. You probably want to half-manually bind it in the long run to provide a Crystal native API on top of it instead of passing it through directly. I do this in a few places in Qt as well.

I feel like it might be easiest to write a C bridge!

Yeah, that's what bindgen does for you. Take a look in the generated .cpp file (Just remove the sanity_check processor for a moment). That's all the code that basically you would have to write. Add the Crystal API, too. If you don't fancy that, then you now understand why I built bindgen :P

However, if there is a good C bridge, you could bind to that instead. Bindgen also has facilities to deal with C libraries. Consult the TEMPLATE.yml on that.

[jeromegn]

Thanks, that helps!

Looks like it's working well. The TEMPLATE has great comments, but I didn't know what I didn't know before, meaning I didn't know what to make of a lot of the comments in there. Now they make more sense.

While I've got you here, I'm not sure how to add a container like v8::Local<T>. Here's the class definition for reference: https://v8docs.nodesource.com/node-9.3/de/deb/classv8_1_1_local.html

I've added:

containers:
  - class: v8::Local
    type: Sequential

classes:
  # ...
  v8::Local: Local

types:
  # ...
  v8::Local:
    crystal_type: Local
    binding_type: Local

This template can contain a lot of different types, most commonly it will be a v8::Value.

I got the error:

Runner command: /Users/jerome/projects/github.com/superfly/v8-cr/lib/bindgen/src/bindgen/parser/../../../clang/bindgen /var/folders/wd/y9m3t11s6nx1nhyzqmhnxmr40000gn/T/bindgen.Ww3X99 -c v8::Isolate::CreateParams -c v8::HeapStatistics -c v8::Local -e v8::MemoryPressureLevel -m "" -f "" -- -x c++ -std=c++11 -resource-dir /usr/local/Cellar/llvm/5.0.1/lib/clang/5.0.1 -D__STDC_CONSTANT_MACROS -D__STDC_LIMIT_MACROS
clang/bindgen failed to execute. (Exception)
  from lib/bindgen/src/bindgen/parser/runner.cr:38:11 in 'run'
  from lib/bindgen/src/bindgen/parser/runner.cr:45:9 in 'run_and_parse'
  from lib/bindgen/src/bindgen/tool.cr:130:7 in 'parse_cpp_sources'
  from lib/bindgen/src/bindgen/tool.cr:48:7 in 'run_steps'
  from lib/bindgen/src/bindgen/tool.cr:40:15 in 'run!'
  from lib/bindgen/src/bindgen.cr:58:1 in '__crystal_main'
  from /usr/local/Cellar/crystal-lang/0.24.1_2/src/crystal/main.cr:11:3 in '_crystal_main'
  from /usr/local/Cellar/crystal-lang/0.24.1_2/src/crystal/main.cr:112:5 in 'main_user_code'
  from /usr/local/Cellar/crystal-lang/0.24.1_2/src/crystal/main.cr:101:7 in 'main'
  from /usr/local/Cellar/crystal-lang/0.24.1_2/src/crystal/main.cr:135:3 in 'main'

Is there a way to get more info from the error?

[Papierkorb]

The template instantiation support is quite in its beginnings, as I simply didn't need it much with Qt (Qt was the driving factor behind bindgen after all). A sequential container instantiation expects the C++ class to respond like a sequential STL container. The Local<T> class there looks like a smart pointer? That won't cut it :|

While that feature would be neat, I don't have the time right now to work on it.

What you can do however is providing custom conversion code to go from a Local<Value> to a Value (Or whatever makes sense). Qt uses this to bind QString in C++-land to String in Crystal-land. Maybe that's a workable solution?

Oh and when you add Foo: Bar to classes, it'll automatically set the crystal_type and binding_type for you. So in your paste above, you can omit the whole v8::Local block.

In general, that Value class sounds like it wants to have a proper Crystal API (To e.g. marshal Crystal values easily). You can use QString as starting point. If it is a container for an (basically) arbitrary value like QVariant, then you could look there too.

Basically a fully custom conversion goes like this:

1. Write a plain-old C structure which can hold all important data to convert in both directions
2. Reproduce that same C structure, manually, in Crystal. You may put that struct into the generated lib Binding block by simply re-opening it in a non-generated file.
3. Write C/C++ code doing the conversion (C++ type -> that struct)
4. Write Crystal code handling the Crystal part (That struct -> Crystal type)
5. Tell the bindgen type system about all of that.

W.r.t. 5), Set the cpp_type to the name of your conversion structure, binding_type to Binding::YourConversionStructure, and crystal_type to the user-facing Crystal class (or struct). Add functionality to that latter class/struct as you like. Then use the from_cpp, to_cpp and the from_crystal, to_crystal type configuration to register your conversion functions. The QString type in Qt may be helpful for this as real-world example.

That's so complex that I don't think a wrapper generator could auto-generate a good API that people would like. Bindgen can help you get the tedious bits done, though for such complex cases, you'll end up writing a few lines.

[jeromegn]

Yes, I think a converter would do the trick.

I think there are issues with the v8 namespacing. If I don't manually write those types in addition to the classes, it doesn't seem to be able to reach them.

I wasn't expecting perfect bindings generation, this shard already surpasses my wildest dreams ;) I have a feeling it's generating the right structs and functions. We'll see once I can actually try it out (I can't until I have v8::Platform done at least, which is fairly contained, so it should be quick enough.)

Any tips for std::unique_ptr<T>? v8 seems to be using those quite a bit.

[Papierkorb]

Any tips for std::unique_ptr? v8 seems to be using those quite a bit.

Oh shit. Well, that's going to be nasty. First there's the issue with there not being proper template instantiation support that's going to bite you here. (Although I think for this having a "smart pointer" specialization as there is for sequential containers would be neat, as that idiom is all over the place in C++).

That aside, the bigger issue is mixing the ownership model of C++ (Well, std::unique_ptr in this case) with Crystals (Pure garbage collection). Going from C++ to Crystal, I think it'd be reasonable to transfer ownership (as std::unique_ptr suggests) to Crystals GC. But what about the other way around? There's really no notion of "ownership" in Crystal, as there is no need for it (Memory management-wise).

You'd have to rip out the object pointer from the Crystal container class, and then in the #finalize (And actually, in every other method) check if the @ptr is null or not. That's not only super nasty, prone to error, but also (and IMHO the biggest issue) surprising to the end-user.

We can fix/add stuff to deal with smart pointers, maybe even special rules for shared/unique smart pointers. But we can't fix the user. I'm honestly not sure how to properly bridge this.

Do you know how other bindings (maybe for other GC'd languages) deal with this?

[jeromegn]

There are basic golang bindings here (they work well): https://github.com/augustoroman/v8

As far as I understand, he wraps those pointers in its own pointers and then passes those back and forth. Go is GCed, but it doesn't do bindings the same way that Crystal does.

That's the C bridge I might decide to use. Except I may just use that package and golang instead of writing bindings at all!

[jeromegn]

Would using a C bridge similar to https://github.com/augustoroman/v8 help? I expect I'd run into the same problems with pointers getting freed.

However, it probably would be a simpler bridge given the nicer bindings API in Crystal.

[Papierkorb]

As long no one feels like implementing smart pointer support, using a wrapper that "hides" them will help. Bindgen has facilities to wrap C libraries back into classes.