Save index in disk.

[v3ss0n]

Currently , it keeps indexes in Memory, that cause out of memory , swapping even a regular sized project . if we could keep indexes in disk it will be a lot more reliable.

My proposal is :

• json dump of indexes for each files
• each index file contains {'filepath' : [{index Objects}]}
• Look up related path from json indexes.

Thoughts?

UPDATE:

Atom have Localstore support and we can use IndexedDB , even better : Pouchdb.com , that way , we can query quickly without worrying about index files manually.

[acusti]

Yeah, saving the index somewhere is definitely necessary, and using json dumps feels like the path of least resistance. Are you suggesting a separate json file for each file’s symbol index? Or just a single index file? Or some combination (like one index per top-level directory)? I’m not clear on the architecture. Also, is there a localstorage-like alternative to directly writing files provided by Atom that we could use?

I was thinking it would be nice to persist the indexes for each project beyond it being open in Atom, but that gets hard with the fact that files can easily change while not open in Atom, invalidating the index, which would then mean having to save and compare checksums for the files to invalidate cached indexes or similar. So if we aren’t going to try to persist those indexes, I suppose the simplest/fastest storage mechanism would be best.

Another thought: if there are separate files containing the json dump for each file, that will facilitate an asynchronous process for indexing a project’s symbols, which is great, but it will mean having to write to and read from a lot of files, even after the index has been built.

[v3ss0n]

I never have experience of indexdb on atom , what do you think about it?
If atom supports IndexedDB , we can use http://pouchdb.com/ , with this we have very powerful DB features inside atom , no need to work with multiple Json file or just a single Json index file, and can query quickly!

var db = new PouchDB('dbname');

db.put({
  _id: 'hash_of_file_path?',
  symbols:  {symbols object},
});

[v3ss0n]

PouchDB is an embedded database engine inspired by CouchDB. In NW.js, it can be used as an abstraction over IndexedDB or WebSQL (via the Chromium implementations) or directly on LevelDB (via the node.js module)

We can use it , thats what i got from electron issue , which refrenced to nwjs use of IndexedDB , by zbenz

[v3ss0n]

electron/electron#897