Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Base.Meta.ParseError("invalid operator \"--\"") when multiprocessing #23

Open
atthom opened this issue Jan 26, 2019 · 3 comments
Open

Comments

@atthom
Copy link

atthom commented Jan 26, 2019

Hello,

I'm using your package with a huge list of expression.
Basically, I'm doing this:

res = map(exp -> (exp, rcall(exp)), res)

It run correctly but it is taking forever and my CPU is barely used.

So I wanted to speed things up with multiprocessing. I tried with pmap, @threads, @spawn and @distributed and starting with julia -p 6 in deed.

Most of the time I get this error :

ERROR: LoadError: On worker 2:
Base.Meta.ParseError("invalid operator \"--\"")

Or no error but no increase in speed (and CPU < 10%).

Moreover, I have this error without doing any multiprocessing.
With the same code I can crash if I start julia ./script.jl
but I don't crash inside the REPL julia> include("GenerateFormula4.jl")
I'm using Windows 10.

I'm still new to Julia so I'm not sure what could cause the problem.

@chakravala
Copy link
Owner

The Reduce.jl package is not currently designed to handle multiprocessing, since communication is based on a single pipe communicating with another process. I've thought about specifically adding some feature to open multiple separate callable instances of the reduce process, but never tried it yet.

@atthom
Copy link
Author

atthom commented Jan 27, 2019

Thank you for your prompt reply.
I'll be glad to try this new feature.
Meanwhile I will try to find a workaround.

@chakravala
Copy link
Owner

Reduce.jl is fairly quick and responsive, but is limited in performance by the character stream interface. It's possible to create a faster interface for Reduce, but it would not be a simple task.

Note that you can also send multiple commands to Reduce at once (like an expression block).

@chakravala chakravala added this to To do in Implement Reduce via automation Jan 29, 2019
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
Development

No branches or pull requests

2 participants