Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Incorrect handling of TypedefFunction #454

Open
stensmo opened this issue Nov 16, 2023 · 9 comments
Open

Incorrect handling of TypedefFunction #454

stensmo opened this issue Nov 16, 2023 · 9 comments

Comments

@stensmo
Copy link

stensmo commented Nov 16, 2023

Started to use the Clang.jl generator, and at least on Windows, there are bugs in TypedefFunction (Clang.CLTypedefDecl). At line 75 in https://github.com/JuliaInterop/Clang.jl/tree/master/src/generator
/print.jl there is a call to tokenize which often fail (@Assert ptr_ref[] != C_NULL is triggered)

Tested on Windows and Julia 1.9.2.

Returning early from the method above (return on line 72), solves the issue (but does not include the typedefs defined). Uncommenting line 75-77 solves the issue for this case. Best solution is to check source_range = clang_getCursorExtent(node.cursor) and if the range is null, don't run line 75-77

Example to trigger the bug can be found below.

using Clang.Generators
using Pango_jll # replace this with your jll package

cd(@DIR)

show(Pango_jll.artifact_dir)

include_dir = normpath(Pango_jll.artifact_dir, "include/pango-1.0/")
clang_dir = joinpath(include_dir, "pango")

options = load_options(joinpath(@DIR, "generator.toml"))

# add compiler flags, e.g. "-DXXXXXXXXX"
args = get_default_args() # Note you must call this function firstly and then append your own flags
push!(args, "-I$include_dir")

headers = [joinpath(clang_dir, "pango.h")]
#headers = [joinpath(clang_dir, header) for header in readdir(clang_dir) if endswith(header, ".h")]
# there is also an experimental detect_headers function for auto-detecting top-level headers in the directory
#headers = detect_headers(clang_dir, args)

# create context
ctx = create_context(headers, args, options)

# run generator
build!(ctx)

[general]
library_name = "Pango"
output_file_path = "./Pango.jl"
module_name = "LibPango"
jll_pkg_name = "Pango_jll"
export_symbol_prefixes = ["pango"]

@Gnimuc
Copy link
Member

Gnimuc commented Nov 21, 2023

did the generator produce clang compilation errors?

@stensmo
Copy link
Author

stensmo commented Nov 21, 2023

No, the generator does not run until completion.

@Assert ptr_ref[] != C_NULL is triggered (Line 13 in token.jl). I tested this on Cairo_jll as well, and the same issue there. After fixing the bug, it almost works to generate working code for Cairo_jll (only small manual fixes of the generated code needed).

The issue is that clang_getCursorExtent(c) does not always return what you expect.

@Gnimuc
Copy link
Member

Gnimuc commented Nov 25, 2023

@stensmo is this library Linux only?

I run the script and get the following compile errors:

       ctx = create_context(headers, args, options)
[ Info: Found dependent header: /Users/gnimuc/.julia/artifacts/a79c326639dc580d203f5962eebbb66df348c9ce/include/pango-1.0/pango/pango-attributes.h
[ Info: Found dependent header: /Users/gnimuc/.julia/artifacts/a79c326639dc580d203f5962eebbb66df348c9ce/include/pango-1.0/pango/pango-font.h
[ Info: Found dependent header: /Users/gnimuc/.julia/artifacts/a79c326639dc580d203f5962eebbb66df348c9ce/include/pango-1.0/pango/pango-coverage.h
[ Info: Found dependent header: /Users/gnimuc/.julia/artifacts/a79c326639dc580d203f5962eebbb66df348c9ce/include/pango-1.0/pango/pango-version-macros.h
[ Info: Found dependent header: /Users/gnimuc/.julia/artifacts/a79c326639dc580d203f5962eebbb66df348c9ce/include/pango-1.0/pango/pango-features.h
[ Info: Found dependent header: /Users/gnimuc/.julia/artifacts/a79c326639dc580d203f5962eebbb66df348c9ce/include/pango-1.0/pango/pango-types.h
[ Info: Found dependent header: /Users/gnimuc/.julia/artifacts/a79c326639dc580d203f5962eebbb66df348c9ce/include/pango-1.0/pango/pango-gravity.h
[ Info: Found dependent header: /Users/gnimuc/.julia/artifacts/a79c326639dc580d203f5962eebbb66df348c9ce/include/pango-1.0/pango/pango-matrix.h
[ Info: Found dependent header: /Users/gnimuc/.julia/artifacts/a79c326639dc580d203f5962eebbb66df348c9ce/include/pango-1.0/pango/pango-script.h
[ Info: Found dependent header: /Users/gnimuc/.julia/artifacts/a79c326639dc580d203f5962eebbb66df348c9ce/include/pango-1.0/pango/pango-language.h
[ Info: Found dependent header: /Users/gnimuc/.julia/artifacts/a79c326639dc580d203f5962eebbb66df348c9ce/include/pango-1.0/pango/pango-bidi-type.h
[ Info: Found dependent header: /Users/gnimuc/.julia/artifacts/a79c326639dc580d203f5962eebbb66df348c9ce/include/pango-1.0/pango/pango-direction.h
[ Info: Found dependent header: /Users/gnimuc/.julia/artifacts/a79c326639dc580d203f5962eebbb66df348c9ce/include/pango-1.0/pango/pango-color.h
[ Info: Found dependent header: /Users/gnimuc/.julia/artifacts/a79c326639dc580d203f5962eebbb66df348c9ce/include/pango-1.0/pango/pango-break.h
[ Info: Found dependent header: /Users/gnimuc/.julia/artifacts/a79c326639dc580d203f5962eebbb66df348c9ce/include/pango-1.0/pango/pango-item.h
[ Info: Found dependent header: /Users/gnimuc/.julia/artifacts/a79c326639dc580d203f5962eebbb66df348c9ce/include/pango-1.0/pango/pango-context.h
[ Info: Found dependent header: /Users/gnimuc/.julia/artifacts/a79c326639dc580d203f5962eebbb66df348c9ce/include/pango-1.0/pango/pango-fontmap.h
[ Info: Found dependent header: /Users/gnimuc/.julia/artifacts/a79c326639dc580d203f5962eebbb66df348c9ce/include/pango-1.0/pango/pango-fontset.h
[ Info: Found dependent header: /Users/gnimuc/.julia/artifacts/a79c326639dc580d203f5962eebbb66df348c9ce/include/pango-1.0/pango/pango-engine.h
[ Info: Found dependent header: /Users/gnimuc/.julia/artifacts/a79c326639dc580d203f5962eebbb66df348c9ce/include/pango-1.0/pango/pango-glyph.h
[ Info: Found dependent header: /Users/gnimuc/.julia/artifacts/a79c326639dc580d203f5962eebbb66df348c9ce/include/pango-1.0/pango/pango-enum-types.h
[ Info: Found dependent header: /Users/gnimuc/.julia/artifacts/a79c326639dc580d203f5962eebbb66df348c9ce/include/pango-1.0/pango/pango-fontset-simple.h
[ Info: Found dependent header: /Users/gnimuc/.julia/artifacts/a79c326639dc580d203f5962eebbb66df348c9ce/include/pango-1.0/pango/pango-glyph-item.h
[ Info: Found dependent header: /Users/gnimuc/.julia/artifacts/a79c326639dc580d203f5962eebbb66df348c9ce/include/pango-1.0/pango/pango-layout.h
[ Info: Found dependent header: /Users/gnimuc/.julia/artifacts/a79c326639dc580d203f5962eebbb66df348c9ce/include/pango-1.0/pango/pango-tabs.h
[ Info: Found dependent header: /Users/gnimuc/.julia/artifacts/a79c326639dc580d203f5962eebbb66df348c9ce/include/pango-1.0/pango/pango-markup.h
[ Info: Found dependent header: /Users/gnimuc/.julia/artifacts/a79c326639dc580d203f5962eebbb66df348c9ce/include/pango-1.0/pango/pango-renderer.h
[ Info: Found dependent header: /Users/gnimuc/.julia/artifacts/a79c326639dc580d203f5962eebbb66df348c9ce/include/pango-1.0/pango/pango-utils.h
[ Info: Parsing headers...
/Users/gnimuc/.julia/artifacts/a79c326639dc580d203f5962eebbb66df348c9ce/include/pango-1.0/pango/pango-coverage.h:25:10: fatal error: 'glib-object.h' file not found
Context(...)

@Gnimuc
Copy link
Member

Gnimuc commented Nov 25, 2023

still missing glib when I switched to

julia> args = get_default_args("x86_64-linux-gnu")
5-element Vector{String}:
 "-isystem/Users/gnimuc/.julia/ar" ⋯ 56 bytes ⋯ "/x86_64-linux-gnu/4.8.5/include"
 "-isystem/Users/gnimuc/.julia/ar" ⋯ 62 bytes ⋯ "4-linux-gnu/4.8.5/include-fixed"
 "-isystem/Users/gnimuc/.julia/ar" ⋯ 42 bytes ⋯ "e3d045/x86_64-linux-gnu/include"
 "-isystem/Users/gnimuc/.julia/ar" ⋯ 55 bytes ⋯ "-linux-gnu/sys-root/usr/include"
 "--target=x86_64-unknown-linux-gnu"

@Gnimuc
Copy link
Member

Gnimuc commented Nov 25, 2023

you need to config glib's include dirs so clang can find where the glib-object.h is. if not, identifiers defined in glib can not be properly tokenized and you got the error.

@Gnimuc
Copy link
Member

Gnimuc commented Nov 25, 2023

julia> using Clang.Generators

julia> using Pango_jll

julia> using Glib_jll

julia> include_dir = normpath(Pango_jll.artifact_dir, "include/pango-1.0/")
"/Users/gnimuc/.julia/artifacts/a79c326639dc580d203f5962eebbb66df348c9ce/include/pango-1.0/"

julia> include_dir_glib = normpath(Glib_jll.artifact_dir, "include/glib-2.0")
"/Users/gnimuc/.julia/artifacts/d2ea466e9d726fe2dc83e2179336f2098f5724a2/include/glib-2.0"

julia> options = load_options(joinpath(@__DIR__, "generator.toml"))
Dict{String, Any} with 1 entry:
  "general" => Dict{String, Any}("output_file_path"=>"./Pango.jl", "module_name"=>"LibPango", "library_name"=>"Pango", "jll_pkg_name"=>"Pango_jll", "export_symbol_prefixes"=>["pango"])

julia> args = get_default_args("x86_64-linux-gnu")
5-element Vector{String}:
 "-isystem/Users/gnimuc/.julia/ar" ⋯ 56 bytes ⋯ "/x86_64-linux-gnu/4.8.5/include"
 "-isystem/Users/gnimuc/.julia/ar" ⋯ 62 bytes ⋯ "4-linux-gnu/4.8.5/include-fixed"
 "-isystem/Users/gnimuc/.julia/ar" ⋯ 42 bytes ⋯ "e3d045/x86_64-linux-gnu/include"
 "-isystem/Users/gnimuc/.julia/ar" ⋯ 55 bytes ⋯ "-linux-gnu/sys-root/usr/include"
 "--target=x86_64-unknown-linux-gnu"

julia> push!(args, "-I$include_dir")
6-element Vector{String}:
 "-isystem/Users/gnimuc/.julia/ar" ⋯ 56 bytes ⋯ "/x86_64-linux-gnu/4.8.5/include"
 "-isystem/Users/gnimuc/.julia/ar" ⋯ 62 bytes ⋯ "4-linux-gnu/4.8.5/include-fixed"
 "-isystem/Users/gnimuc/.julia/ar" ⋯ 42 bytes ⋯ "e3d045/x86_64-linux-gnu/include"
 "-isystem/Users/gnimuc/.julia/ar" ⋯ 55 bytes ⋯ "-linux-gnu/sys-root/usr/include"
 "--target=x86_64-unknown-linux-gnu"
 "-I/Users/gnimuc/.julia/artifact" ⋯ 30 bytes ⋯ "b66df348c9ce/include/pango-1.0/"

julia> push!(args, "-isystem$include_dir_glib")  # use -isystem for dependencies
7-element Vector{String}:
 "-isystem/Users/gnimuc/.julia/ar" ⋯ 56 bytes ⋯ "/x86_64-linux-gnu/4.8.5/include"
 "-isystem/Users/gnimuc/.julia/ar" ⋯ 62 bytes ⋯ "4-linux-gnu/4.8.5/include-fixed"
 "-isystem/Users/gnimuc/.julia/ar" ⋯ 42 bytes ⋯ "e3d045/x86_64-linux-gnu/include"
 "-isystem/Users/gnimuc/.julia/ar" ⋯ 55 bytes ⋯ "-linux-gnu/sys-root/usr/include"
 "--target=x86_64-unknown-linux-gnu"
 "-I/Users/gnimuc/.julia/artifact" ⋯ 30 bytes ⋯ "b66df348c9ce/include/pango-1.0/"
 "-isystem/Users/gnimuc/.julia/ar" ⋯ 34 bytes ⋯ "36f2098f5724a2/include/glib-2.0"

julia> ctx = create_context(joinpath(include_dir, "pango", "pango.h"), args, options)
[ Info: Found dependent header: /Users/gnimuc/.julia/artifacts/a79c326639dc580d203f5962eebbb66df348c9ce/include/pango-1.0/pango/pango-attributes.h
[ Info: Found dependent header: /Users/gnimuc/.julia/artifacts/a79c326639dc580d203f5962eebbb66df348c9ce/include/pango-1.0/pango/pango-font.h
[ Info: Found dependent header: /Users/gnimuc/.julia/artifacts/a79c326639dc580d203f5962eebbb66df348c9ce/include/pango-1.0/pango/pango-coverage.h
[ Info: Found dependent header: /Users/gnimuc/.julia/artifacts/a79c326639dc580d203f5962eebbb66df348c9ce/include/pango-1.0/pango/pango-version-macros.h
[ Info: Found dependent header: /Users/gnimuc/.julia/artifacts/a79c326639dc580d203f5962eebbb66df348c9ce/include/pango-1.0/pango/pango-features.h
[ Info: Found dependent header: /Users/gnimuc/.julia/artifacts/a79c326639dc580d203f5962eebbb66df348c9ce/include/pango-1.0/pango/pango-types.h
[ Info: Found dependent header: /Users/gnimuc/.julia/artifacts/a79c326639dc580d203f5962eebbb66df348c9ce/include/pango-1.0/pango/pango-gravity.h
[ Info: Found dependent header: /Users/gnimuc/.julia/artifacts/a79c326639dc580d203f5962eebbb66df348c9ce/include/pango-1.0/pango/pango-matrix.h
[ Info: Found dependent header: /Users/gnimuc/.julia/artifacts/a79c326639dc580d203f5962eebbb66df348c9ce/include/pango-1.0/pango/pango-script.h
[ Info: Found dependent header: /Users/gnimuc/.julia/artifacts/a79c326639dc580d203f5962eebbb66df348c9ce/include/pango-1.0/pango/pango-language.h
[ Info: Found dependent header: /Users/gnimuc/.julia/artifacts/a79c326639dc580d203f5962eebbb66df348c9ce/include/pango-1.0/pango/pango-bidi-type.h
[ Info: Found dependent header: /Users/gnimuc/.julia/artifacts/a79c326639dc580d203f5962eebbb66df348c9ce/include/pango-1.0/pango/pango-direction.h
[ Info: Found dependent header: /Users/gnimuc/.julia/artifacts/a79c326639dc580d203f5962eebbb66df348c9ce/include/pango-1.0/pango/pango-color.h
[ Info: Found dependent header: /Users/gnimuc/.julia/artifacts/a79c326639dc580d203f5962eebbb66df348c9ce/include/pango-1.0/pango/pango-break.h
[ Info: Found dependent header: /Users/gnimuc/.julia/artifacts/a79c326639dc580d203f5962eebbb66df348c9ce/include/pango-1.0/pango/pango-item.h
[ Info: Found dependent header: /Users/gnimuc/.julia/artifacts/a79c326639dc580d203f5962eebbb66df348c9ce/include/pango-1.0/pango/pango-context.h
[ Info: Found dependent header: /Users/gnimuc/.julia/artifacts/a79c326639dc580d203f5962eebbb66df348c9ce/include/pango-1.0/pango/pango-fontmap.h
[ Info: Found dependent header: /Users/gnimuc/.julia/artifacts/a79c326639dc580d203f5962eebbb66df348c9ce/include/pango-1.0/pango/pango-fontset.h
[ Info: Found dependent header: /Users/gnimuc/.julia/artifacts/a79c326639dc580d203f5962eebbb66df348c9ce/include/pango-1.0/pango/pango-engine.h
[ Info: Found dependent header: /Users/gnimuc/.julia/artifacts/a79c326639dc580d203f5962eebbb66df348c9ce/include/pango-1.0/pango/pango-glyph.h
[ Info: Found dependent header: /Users/gnimuc/.julia/artifacts/a79c326639dc580d203f5962eebbb66df348c9ce/include/pango-1.0/pango/pango-enum-types.h
[ Info: Found dependent header: /Users/gnimuc/.julia/artifacts/a79c326639dc580d203f5962eebbb66df348c9ce/include/pango-1.0/pango/pango-fontset-simple.h
[ Info: Found dependent header: /Users/gnimuc/.julia/artifacts/a79c326639dc580d203f5962eebbb66df348c9ce/include/pango-1.0/pango/pango-glyph-item.h
[ Info: Found dependent header: /Users/gnimuc/.julia/artifacts/a79c326639dc580d203f5962eebbb66df348c9ce/include/pango-1.0/pango/pango-layout.h
[ Info: Found dependent header: /Users/gnimuc/.julia/artifacts/a79c326639dc580d203f5962eebbb66df348c9ce/include/pango-1.0/pango/pango-tabs.h
[ Info: Found dependent header: /Users/gnimuc/.julia/artifacts/a79c326639dc580d203f5962eebbb66df348c9ce/include/pango-1.0/pango/pango-markup.h
[ Info: Found dependent header: /Users/gnimuc/.julia/artifacts/a79c326639dc580d203f5962eebbb66df348c9ce/include/pango-1.0/pango/pango-renderer.h
[ Info: Found dependent header: /Users/gnimuc/.julia/artifacts/a79c326639dc580d203f5962eebbb66df348c9ce/include/pango-1.0/pango/pango-utils.h
[ Info: Parsing headers...
/Users/gnimuc/.julia/artifacts/d2ea466e9d726fe2dc83e2179336f2098f5724a2/include/glib-2.0/glib/gtypes.h:34:10: fatal error: 'glibconfig.h' file not found
Context(...)

julia> build!(ctx)
[ Info: Processing header: /Users/gnimuc/.julia/artifacts/a79c326639dc580d203f5962eebbb66df348c9ce/include/pango-1.0/pango/pango.h
[ Info: Building the DAG...
┌ Warning: [CollectDependentSystemNode]: found symbols in the system headers: [:gunichar, :GQueue, :GList, :gpointer, :GBytes, :_GError, :_GTypeInstance, :_GList, :_GMarkupParseContext, :_GQueue, :gboolean, :_GSList, :gchar, :_GObject, :_GTypeClass, :GTypeInstance, :_GTypeClass, :GSList, :_GError, :_GTypeModule, :GString, :gint, :_GSList, :GTypeClass, :_GData, :_GBytes, :_GObject, :GType, :_GTypeModule, :_GQueue, :GData, :GDestroyNotify, :_GString, :GObject, :guchar, :GQuark, :_GTypeInstance, :GError, :_GString, :GTypeModule, :_GList, :GMarkupParseContext, :guint]
└ @ Clang.Generators ~/.julia/dev/Clang/src/generator/passes.jl:95
[ Info: Emit Julia expressions...
[ Info: [ProloguePrinter]: print to ./Pango.jl
[ Info: [GeneralPrinter]: print to ./Pango.jl
[ Info: [EpiloguePrinter]: print to ./Pango.jl
[ Info: Done!
Context(...)

@Gnimuc
Copy link
Member

Gnimuc commented Nov 25, 2023

[ Info: Parsing headers...
/Users/gnimuc/.julia/artifacts/d2ea466e9d726fe2dc83e2179336f2098f5724a2/include/glib-2.0/glib/gtypes.h:34:10: fatal error: 'glibconfig.h' file not found
Context(...)

Still get this compile error but it's about the dependency library glib and we are only attempting to generate symbols from pango, so it probably doesn't affect the output.

[ Info: Building the DAG...
┌ Warning: [CollectDependentSystemNode]: found symbols in the system headers: [:gunichar, :GQueue, :GList, :gpointer, :GBytes, :_GError, :_GTypeInstance, :_GList, :_GMarkupParseContext, :_GQueue, :gboolean, :_GSList, :gchar, :_GObject, :_GTypeClass, :GTypeInstance, :_GTypeClass, :GSList, :_GError, :_GTypeModule, :GString, :gint, :_GSList, :GTypeClass, :_GData, :_GBytes, :_GObject, :GType, :_GTypeModule, :_GQueue, :GData, :GDestroyNotify, :_GString, :GObject, :guchar, :GQuark, :_GTypeInstance, :GError, :_GString, :GTypeModule, :_GList, :GMarkupParseContext, :guint]
└ @ Clang.Generators ~/.julia/dev/Clang/src/generator/passes.jl:95

These warnings means that those symbols are used in the headers of pango, but they are not a part of pango, they are actually defined in its dependencies. So, the generated code is not ready to use, you need to either manually define those symbols in Julia or export them from a wrapper package like Glib.jl if exists.

@stensmo
Copy link
Author

stensmo commented Feb 20, 2024

Sorry for the late reply. A patched version of Clang.jl worked fine. I can submit the (super simple) patch if you want to. The patched version generated (as a test) all of the Cairo library (a small amount of code restructuring was needed, since some structs came out in the wrong order). I think Clang.jl is great, but the last bugs need to be ironed out.

@Gnimuc
Copy link
Member

Gnimuc commented Apr 8, 2024

Any feedback would be highly appreciated.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

No branches or pull requests

2 participants