Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Simple Reduce Statement Fails to Compile Inside Module #32

Open
mattcbro opened this issue Nov 6, 2019 · 19 comments
Open

Simple Reduce Statement Fails to Compile Inside Module #32

mattcbro opened this issue Nov 6, 2019 · 19 comments

Comments

@mattcbro
Copy link

mattcbro commented Nov 6, 2019

The following code snippet runs just fine in Julia 1.2, unless it's put into a module.

module DebugModule

using ForceImport
@force using Reduce.Algebra
using ReduceLinAlg


f(a,b) = (:x - a) * (:x - b)
g(a,b) = int( f(a,b), :x)

# comment this out and the module loads,  or simply copy and paste the code
# and it works
g0 = g(:a,:b) - sub( (:(x=b), ), g(:a,:b))

end # module

Now see what happens when I try to use this module.

julia> using DebugModule
[ Info: Recompiling stale cache file /home/matt/.julia/compiled/v1.2/DebugModule.ji for DebugModule [top-level]
ERROR: LoadError: Base.Meta.ParseError("invalid operator \"--\"")
Stacktrace:
 [1] #parse#1(::Bool, ::Bool, ::Bool, ::typeof(Base.Meta.parse), ::String, ::Int64) at ./meta.jl:129
 [2] #parse at ./tuple.jl:0 [inlined]
 [3] #parse#4(::Bool, ::Bool, ::typeof(Base.Meta.parse), ::String) at ./meta.jl:160
 [4] parse at ./meta.jl:160 [inlined]
 [5] |>(::String, ::typeof(Base.Meta.parse)) at ./operators.jl:854
 [6] #parse_expr#46(::Int64, ::typeof(Reduce.parse_expr), ::String, ::Reduce.RExpr) at /home/matt/.julia/packages/Reduce/4dmwd/src/parser.jl:317
 [7] (::getfield(Reduce, Symbol("#kw##parse_expr")))(::NamedTuple{(:be,),Tuple{Int64}}, ::typeof(Reduce.parse_expr), ::String, ::Reduce.RExpr) at ./none:0
 [8] #parse#76(::Int64, ::typeof(parse), ::Reduce.RExpr) at /home/matt/.julia/packages/Reduce/4dmwd/src/parser.jl:438
 [9] parse at /home/matt/.julia/packages/Reduce/4dmwd/src/parser.jl:438 [inlined]
 [10] convert at /home/matt/.julia/packages/Reduce/4dmwd/src/rexpr.jl:414 [inlined]
 [11] unfold_expr_force(::Symbol, ::Symbol, ::Expr, ::Symbol) at /home/matt/.julia/packages/Reduce/4dmwd/src/parser.jl:847
 [12] #unfold_expr#77(::Bool, ::typeof(Reduce.unfold_expr), ::Symbol, ::Symbol, ::Expr, ::Symbol) at /home/matt/.julia/packages/Reduce/4dmwd/src/parser.jl:806
 [13] unfold_expr(::Symbol, ::Symbol, ::Expr, ::Symbol) at /home/matt/.julia/packages/Reduce/4dmwd/src/parser.jl:801
 [14] unfold(::Symbol, ::Symbol, ::Expr, ::Symbol) at /home/matt/.julia/packages/Reduce/4dmwd/src/parser.jl:863
 [15] int(::Expr, ::Symbol) at /home/matt/.julia/packages/Reduce/4dmwd/src/parser.jl:793
 [16] g(::Symbol, ::Symbol) at /mnt/WorkSpace/projects/Maestro/GlucoseSource/DebugModule.jl:10
 [17] top-level scope at /mnt/WorkSpace/projects/Maestro/GlucoseSource/DebugModule.jl:14
 [18] include at ./boot.jl:328 [inlined]
 [19] include_relative(::Module, ::String) at ./loading.jl:1094
 [20] include(::Module, ::String) at ./Base.jl:31
 [21] top-level scope at none:2
 [22] eval at ./boot.jl:330 [inlined]
 [23] eval(::Expr) at ./client.jl:432
 [24] top-level scope at ./none:3
in expression starting at /mnt/WorkSpace/projects/Maestro/GlucoseSource/DebugModule.jl:14
ERROR: Failed to precompile DebugModule [top-level] to /home/matt/.julia/compiled/v1.2/DebugModule.ji.
Stacktrace:
 [1] compilecache(::Base.PkgId, ::String) at ./loading.jl:1253
 [2] _require(::Base.PkgId) at ./loading.jl:1013
 [3] require(::Base.PkgId) at ./loading.jl:911
 [4] require(::Module, ::Symbol) at ./loading.jl:906

julia> 

If I copy and paste the code into the julia REPL it works fine and the module loads if the last line,
g0 = g(:a,:b) - sub( (:(x=b), ), g(:a,:b)) is commented out. This looks like a bug to me.

@chakravala
Copy link
Owner

That is incorrect, it is not a bug, you need to use an __init__() function in your module like this

module DebugModule
    using ForceImport
    @force using Reduce.Algebra
    using ReduceLinAlg
    f(a,b) = (:x - a) * (:x - b)
    g(a,b) = int( f(a,b), :x)
    function __init__()
        global g0
        g0 = g(:a,:b) - sub( (:(x=b), ), g(:a,:b))
    end
end

When I load the module, it works:

julia> DebugModule.g0
:(-((((3b - 2x) * x - 3 * (2b - x) * a) * x + (3a - b) * b ^ 2)) / 6)

You can't use Reduce for symbolic computation until it is initialized first.

@mattcbro
Copy link
Author

mattcbro commented Nov 6, 2019

This is still odd to me since ultimately my use of g0 and what will follow are compile time calculations not run time calculations. My intention is to do some symbolic pre-processing and then compile functions derived from the result.

However init() is primarily designated for run time initialization; but declaring g0 as a const or a global does not make the error go away, and I know that I can declare global variables or constants inside modules. So this error still is odd to me. Perhaps you can explain it further? What do you mean by it needs to be initialized first? How does that init command initialize Reduce?

Is module compilation on it's own insufficient to do Reduce computation?

Thanks in advance for your help.

@chakravala
Copy link
Owner

In that case, here is how to get the situation you want:

module DebugModule
    using Reduce
    @force using Reduce.Algebra
    using ReduceLinAlg
    f(a,b) = (:x - a) * (:x - b)
    g(a,b) = int( f(a,b), :x)
    Reduce.Load()
    g0 = g(:a,:b) - sub( (:(x=b), ), g(:a,:b))
end

It is recommended to only use symbolic computation within your __init__() call or at run-time to generate new compiled code after your module is already loaded.

However, if you really, really want to, you can use Reduce.Load() to initialize Reduce on the first compilation pass of the package also. That should do what you wanted.

@mattcbro
Copy link
Author

mattcbro commented Nov 6, 2019

It's still not compiling on my rig. Am I doing something wrong? I'm on linux mint 19.2, using Julia 1.2. Adding in the Reduce.Load() command (copied and pasted from your example) still gives me this error:

julia> using DebugModule
[ Info: Precompiling DebugModule [top-level]
ERROR: LoadError: Base.Meta.ParseError("invalid operator \"--\"")
Stacktrace:
 [1] #parse#1(::Bool, ::Bool, ::Bool, ::typeof(Base.Meta.parse), ::String, ::Int64) at ./meta.jl:129
 [2] #parse at ./tuple.jl:0 [inlined]
 [3] #parse#4(::Bool, ::Bool, ::typeof(Base.Meta.parse), ::String) at ./meta.jl:160
 [4] parse at ./meta.jl:160 [inlined]
 [5] |>(::String, ::typeof(Base.Meta.parse)) at ./operators.jl:854
 [6] #parse_expr#46(::Int64, ::typeof(Reduce.parse_expr), ::String, ::Reduce.RExpr) at /home/matt/.julia/packages/Reduce/4dmwd/src/parser.jl:317
 [7] (::getfield(Reduce, Symbol("#kw##parse_expr")))(::NamedTuple{(:be,),Tuple{Int64}}, ::typeof(Reduce.parse_expr), ::String, ::Reduce.RExpr) at ./none:0
 [8] #parse#76(::Int64, ::typeof(parse), ::Reduce.RExpr) at /home/matt/.julia/packages/Reduce/4dmwd/src/parser.jl:438
 [9] parse at /home/matt/.julia/packages/Reduce/4dmwd/src/parser.jl:438 [inlined]
 [10] convert at /home/matt/.julia/packages/Reduce/4dmwd/src/rexpr.jl:414 [inlined]
 [11] unfold_expr_force(::Symbol, ::Symbol, ::Expr, ::Symbol) at /home/matt/.julia/packages/Reduce/4dmwd/src/parser.jl:847
 [12] #unfold_expr#77(::Bool, ::typeof(Reduce.unfold_expr), ::Symbol, ::Symbol, ::Expr, ::Symbol) at /home/matt/.julia/packages/Reduce/4dmwd/src/parser.jl:806
 [13] unfold_expr(::Symbol, ::Symbol, ::Expr, ::Symbol) at /home/matt/.julia/packages/Reduce/4dmwd/src/parser.jl:801
 [14] unfold(::Symbol, ::Symbol, ::Expr, ::Symbol) at /home/matt/.julia/packages/Reduce/4dmwd/src/parser.jl:863
 [15] int(::Expr, ::Symbol) at /home/matt/.julia/packages/Reduce/4dmwd/src/parser.jl:793
 [16] g(::Symbol, ::Symbol) at /mnt/WorkSpace/projects/Maestro/GlucoseSource/DebugModule.jl:7
 [17] top-level scope at /mnt/WorkSpace/projects/Maestro/GlucoseSource/DebugModule.jl:9
 [18] include at ./boot.jl:328 [inlined]
 [19] include_relative(::Module, ::String) at ./loading.jl:1094
 [20] include(::Module, ::String) at ./Base.jl:31
 [21] top-level scope at none:2
 [22] eval at ./boot.jl:330 [inlined]
 [23] eval(::Expr) at ./client.jl:432
 [24] top-level scope at ./none:3
in expression starting at /mnt/WorkSpace/projects/Maestro/GlucoseSource/DebugModule.jl:9
ERROR: Failed to precompile DebugModule [top-level] to /home/matt/.julia/compiled/v1.2/DebugModule.ji.
Stacktrace:
 [1] compilecache(::Base.PkgId, ::String) at ./loading.jl:1253
 [2] _require(::Base.PkgId) at ./loading.jl:1013
 [3] require(::Base.PkgId) at ./loading.jl:911
 [4] require(::Module, ::Symbol) at ./loading.jl:906

julia> 

@chakravala
Copy link
Owner

chakravala commented Nov 7, 2019

Alright, looks like Reduce.Load() only works from the REPL so I created Reduce.Preload() instead:

module DebugModule
    using Reduce
    @force using Reduce.Algebra
    using ReduceLinAlg
    f(a,b) = (:x - a) * (:x - b)
    g(a,b) = int( f(a,b), :x)
    Reduce.Preload()
    g0 = g(:a,:b) - sub( (:(x=b), ), g(:a,:b))
end

This is available in the most recent commit of Reduce and should work in precompilation now.

@mattcbro
Copy link
Author

mattcbro commented Nov 7, 2019

@chakravala Thanks so much for addressing this. Unfortunately when I tried to add the master branch of reduce, which contains the new function, I ran into some problems during the build. It kind of looks like I am missing an updated dependency which has the presumably linear algebra functions mateign and cofacter.

(v1.2) pkg> add Reduce#master
  Updating registry at `~/.julia/registries/General`
  Updating git-repo `https://github.com/JuliaRegistries/General.git`
   Cloning git-repo `https://github.com/chakravala/Reduce.jl.git`
  Updating git-repo `https://github.com/chakravala/Reduce.jl.git`
 Resolving package versions...
 Installed PositiveFactorizations ─ v0.2.3
 Installed GPUArrays ────────────── v1.0.4
 Installed QuadGK ───────────────── v2.1.1
 Installed DiffRules ────────────── v0.1.0
 Installed DataStructures ───────── v0.17.5
 Installed JuliaFormatter ───────── v0.1.19
 Installed LLVM ─────────────────── v1.3.2
 Installed Optim ────────────────── v0.19.4
 Installed Parsers ──────────────── v0.3.8
 Installed BinaryProvider ───────── v0.5.8
  Updating `~/.julia/environments/v1.2/Project.toml`
  [93e0c654] ↑ Reduce v1.2.3 ⇒ v1.2.4 #master (https://github.com/chakravala/Reduce.jl.git)
  Updating `~/.julia/environments/v1.2/Manifest.toml`
  [b99e7846] ↑ BinaryProvider v0.5.7 ⇒ v0.5.8
  [864edb3b] ↑ DataStructures v0.17.4 ⇒ v0.17.5
  [b552c78f] ↑ DiffRules v0.0.10 ⇒ v0.1.0
  [0c68f7d7] ↑ GPUArrays v1.0.3 ⇒ v1.0.4
  [98e50ef6] ↑ JuliaFormatter v0.1.17 ⇒ v0.1.19
  [929cbde3] ↑ LLVM v1.3.1 ⇒ v1.3.2
  [429524aa] ↑ Optim v0.19.3 ⇒ v0.19.4
  [69de0a69] ↑ Parsers v0.3.7 ⇒ v0.3.8
  [85a6dd25] ↑ PositiveFactorizations v0.2.2 ⇒ v0.2.3
  [1fd47b50] ↑ QuadGK v2.0.3 ⇒ v2.1.1
  [93e0c654] ↑ Reduce v1.2.3 ⇒ v1.2.4 #master (https://github.com/chakravala/Reduce.jl.git)
  Building Reduce → `~/.julia/packages/Reduce/C3TIu/deps/build.log`UndefVarError: mateign not defined

in expression starting at /mnt/WorkSpace/projects/Maestro/GlucoseSource/toplevel:1003
Revise.LogRecord(Warn, omitting call expression (getproperty)(Reduce.Algebra, mateign), lowered, Revise_c6e083f8, "/home/matt/.julia/packages/Revise/cbX7Q/src/lowered.jl", 210)UndefVarError: cofactor not defined
in expression starting at /mnt/WorkSpace/projects/Maestro/GlucoseSource/toplevel:1040
Revise.LogRecord(Warn, omitting call expression (getproperty)(Reduce.Algebra, cofactor), lowered, Revise_c6e083f8, "/home/matt/.julia/packages/Revise/cbX7Q/src/lowered.jl", 210)
(v1.2) pkg> 

@chakravala
Copy link
Owner

You are using Revise, you probably just need to quit the REPL and restart Julia to get rid of that problem.

@mattcbro
Copy link
Author

mattcbro commented Nov 8, 2019

Thanks for pointing that out. I was able to build it after restarting the REPL.

The use case of doing a bunch of symbolic calculations and then using the results to generate julia functions is a pretty common theme, with several web search hits. However when you want the function to return an array there appears to be a bit of a gotcha.

It's easy enough to generate an array of symbolic expressions say of the form x = [:a, :b, :(a+b)]. But you can not make a function out of this in the usual fashion say by doing @eval f(a,b) = $x. That will create a function that returns the original expression array without binding a and b. If however x = :[a ; b ; a+b], then @eval f(a,b) = $x works fine. The problem is, is that there is no programmatical way to transform [:a ; :b ; :(a+b)] to :[a; b ; a+b].

You can do y = :[$(x[1]) ; $(x[2]) ; $(x[3])] but y = :[$(x[i]) for i=1:3] won't work. i is not evaluated. I haven't yet found the write syntax to factor out a quote operation from an array or tuple.

@rjrosati
Copy link

rjrosati commented Nov 9, 2019

To quote arrays the way you describe, I've been doing this:

julia> x = [:a , :b]
2-element Array{Symbol,1}:
 :a
 :b

julia> arr = :([ $(x...) ])
:([a, b])

julia> 

@chakravala
Copy link
Owner

Another way is like this

julia> Expr(:vect,[:a,:b]...)
:([a, b])

Have a look at the output of dump to learn more:

julia> dump(:([a,b]))
Expr
  head: Symbol vect
  args: Array{Any}((2,))
    1: Symbol a
    2: Symbol b

@mattcbro
Copy link
Author

Thanks for the comments guys. I was beating my head against this wall for a while.

@mattcbro
Copy link
Author

mattcbro commented Aug 7, 2020

Well here we are at Julia 1.5 and Reduce.Preload() is no longer allowed inside a module. I'm getting this kind of error:

ERROR: LoadError: Evaluation into the closed module `Reduce` breaks incremental compilation because the side effects will not be permanent. This is likely due to some other module mutating `Reduce` with `eval` during precompilation - don't do this.
Stacktrace:
 [1] eval at ./boot.jl:331 [inlined]
 [2] eval at /home/matt/.julia/packages/Reduce/SMYnP/src/Reduce.jl:2 [inlined]
 [3] Preload() at /home/matt/.julia/packages/Reduce/SMYnP/src/Reduce.jl:370
 [4] top-level scope at /mnt/WorkSpace/projects/Maestro/PulseWave/SymbolicSpline.jl:7

The first 7 lines of the module are:

module SymbolicSpline
using Reduce
using ForceImport
@force using Reduce.Algebra
using ReduceLinAlg
Reduce.Preload()

Is there any work-around for this? I really liked the feature of being able to precompile symbolic code into a module.

Thanks once again.

@chakravala
Copy link
Owner

chakravala commented Aug 7, 2020

The problem is the definition of Preload() = (global rs=PSL(); global s; eval(s); atexit(()->kill(rs))), which assumes a global variable s that seems to make Julia v1.5 unhappy.

Prupose of s is to initialize various buffers and configurations needed after loading:

offs = ""
for o in offlist
    o != :nat && (offs = offs*"off $o; ")
end
write(rs.input,"off nat; $EOTstr;\n")
banner = readuntil(rs.output,EOT) |> String
readavailable(rs.output)
rcsl = occursin(" CSL ",banner)
if Sys.iswindows()
    banner = replace(banner,r"\r" => "")
    println(split(String(banner),'\n')[rcsl ? 1 : end-3])
else
    ReduceCheck(banner)
    println(split(String(banner),'\n')[rcsl ? 1 : end-3])
end
load_package(:rlfi)
offs |> RExpr |> rcall
rcall(R"on savestructr")
show(devnull,"text/latex",R"int(sinh(e**i*z),z)")
R"x" == R"x"
ListPrint(0)

Work around is to duplicate the code for s directly into the definition of Preload, which is rather inconvenient because it is less elegant, but I don't know of any other quick work around.

This should now work again on the latest commit. Let me know if you can think of a more elegant way to resolve the problem without literally duplicating the initialization code into the Preload definition.

PS, please do share the projects you are working on, I would like to see your work.

@mattcbro
Copy link
Author

mattcbro commented Aug 7, 2020

My "workaround" was going to be to create a script that ran Reduce and then render it's output as expression strings, which would then be written to another file, which would be included in a module. That approach is decidedly ugly, but it would certainly make all the module precompile errors go away.

@mattcbro
Copy link
Author

mattcbro commented Aug 7, 2020

I added the new #master version of the package and Pkg.status() shows I'm at version 1.2.8. However I'm still seeing this error namely:

julia> include("loadmimic.jl")
[ Info: Precompiling PulseAlgos [top-level]
Reduce (Free CSL version, revision 5286), 01-Mar-20 ...
ERROR: LoadError: Evaluation into the closed module `Reduce` breaks incremental compilation because the side effects will not be permanent. This is likely due to some other module mutating `Reduce` with `eval` during precompilation - don't do this.
Stacktrace:
 [1] eval at ./boot.jl:331 [inlined]
 [2] eval at /home/matt/.julia/packages/Reduce/MJFOl/src/Reduce.jl:2 [inlined]
 [3] |> at ./operators.jl:834 [inlined]
 [4] ==(::Reduce.RExpr, ::Reduce.RExpr) at /home/matt/.julia/packages/Reduce/MJFOl/src/rexpr.jl:527
 [5] Preload() at /home/matt/.julia/packages/Reduce/MJFOl/src/Reduce.jl:391
 [6] top-level scope at /mnt/WorkSpace/projects/Maestro/PulseWave/SymbolicSpline.jl:7
 [7] include(::Function, ::Module, ::String) at ./Base.jl:380
 [8] include(::Module, ::String) at ./Base.jl:368
 [9] top-level scope at none:2
 [10] eval at ./boot.jl:331 [inlined]
 [11] eval(::Expr) at ./client.jl:467
 [12] top-level scope at ./none:3
in expression starting at /mnt/WorkSpace/projects/Maestro/PulseWave/SymbolicSpline.jl:7
ERROR: LoadError: Failed to precompile SymbolicSpline [top-level] to /home/matt/.julia/compiled/v1.5/SymbolicSpline.ji.
Stacktrace:

The first 7 lines of the module are as before.

@chakravala
Copy link
Owner

This time it is due to my usage of eval in the following method:

function ==(r::RExpr, s::RExpr)
    n = expand(r).str
    m = expand(s).str
    l=length(n)
    llength(m) && (return false)
    b = true
    for j1:l
        b &= "if($(n[j]))=($(m[j]))then 1 else 0"|>rcall|>Meta.parse|>eval|>Bool
    end
    return b
end

However, I don't have time to immediately work on a fix, or to focus on all the other potential problems with this.

@chakravala chakravala reopened this Aug 7, 2020
@chakravala chakravala added the bug label Aug 7, 2020
@mattcbro
Copy link
Author

mattcbro commented Aug 7, 2020

I understand. Thanks for your help so far. I will try to find another approach. I do feel like an important use case for symbolic processing is the programmatic generation of code, especially optimized code. I've frequently used maple in that capacity.

@chakravala
Copy link
Owner

chakravala commented Aug 8, 2020

If my work is so important, maybe i should be getting paid. This is a bonus feature.

You don't even share your derived code with me.

Why would i care about helping you with a proprietary code base and you don't pay me?

@mattcbro
Copy link
Author

Sorry it's not my call. I'm under NDA and do not have either the authority to release my code nor hire you. My particular application though is quite trivial. I'm simply using symbolic processing to create splines with various boundary conditions. I could have used any symbolic tool for this. I just happen to like the integration with Julia that your tool provides. I hope you do find a worthy sponsor for your efforts.

Just FYI, the workaround of running your Reduce package as a preprocessor, then writing out the answers to a text file, worked like a charm.

@chakravala chakravala linked a pull request Sep 9, 2020 that will close this issue
@chakravala chakravala removed a link to a pull request Sep 9, 2020
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
None yet
Development

No branches or pull requests

3 participants