Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Investigate pros/cons of multi-core #13

Open
missinglink opened this issue Jul 11, 2019 · 0 comments
Open

Investigate pros/cons of multi-core #13

missinglink opened this issue Jul 11, 2019 · 0 comments
Labels
question Further information is requested

Comments

@missinglink
Copy link
Member

missinglink commented Jul 11, 2019

The http server currently spawns one thread per core, however, only a single database connection is opened with sqlite compiled with SQLITE_THREADSAFE 0

What this basically means is that one core will run hot when querying with the others idle.
We should either develop or document a workflow which utilizes all cores.

A similar situation applies for imports, these are also single-threaded.

One completely valid solution to this issue is to simply say that the application is single-threaded but it can be scaled at the infrastructure level.

Eg. For imports:
it is possible to run multiple imports in parallel and then merge the databases into one, this has the con of taking some minutes to merge the files and the pro of not having to worry about threading, write locks, mutexes etc. which make the application code complex and prone to errors.

Eg. For queries:
it is possible to run multiple containers of the same image in a load-balancer setup, giving each container one CPU. The default read mode for the QueryService is mmap so the database pages are cached in the Kernel FileSystem Cache which is then shared between processes by the kernel; all this means that it does not require any additional RAM to generate additional caches for each container.

The simplest solution to this issue might be to just avoid it altogether and document how to achieve full CPU utilization while keeping the application single-core.

If that's the case, we can simplify the server/http.js code to also be single-threaded.

Related: #7

@missinglink missinglink added the question Further information is requested label Jul 11, 2019
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
question Further information is requested
Projects
None yet
Development

No branches or pull requests

1 participant