Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Remove usage of pickle from objectdb and history #605

Open
4 of 9 tasks
lieryan opened this issue Dec 16, 2022 · 0 comments
Open
4 of 9 tasks

Remove usage of pickle from objectdb and history #605

lieryan opened this issue Dec 16, 2022 · 0 comments
Assignees

Comments

@lieryan
Copy link
Member

lieryan commented Dec 16, 2022

pickle is insecure and it's not very efficient for larger, more complex datasets.

This will be a multiple steps plan towards moving into sqlite-backed storage:

  1. Remove rarely used features
    1. Deprecate objectdb and history compression: This was never the default and disk space nowadays makes this unlikely to be an issue nowadays, maybe we can re-introduce compression when sqlite migration is complete (Remove .ropeproject/history and .ropeproject/objectdb compression #606)
    2. Deprecate importing of old format (objectdb.pickle and history.pickle): rope hasn't been generating this for decades (Remove _DataFiles._import_old_files() #607)
  2. Make ScopeInfo JSON-serializable: there are some data here that are python class instances, but they really doesn't need to be, we can just __getstate__()/__setstate__() them instead when de/serializing. Implement JSON DataFile serialization  #611
  3. Implement sqlite database schema versioning Automatically purge autoimport.db when there is schema change  #565
  4. These files contains a bunch of nested dictionaries, these can be translated to sqlite tables in a straightforward manner, we'll likely will still store JSON documents inside the sqlite tables instead of normalizing everything
  5. Make sure that the tables have appropriate indexes
  6. Remove write support for legacy objectdb and history files
  7. Remove read support for legacy objectdb and history files
@lieryan lieryan self-assigned this Dec 16, 2022
@lieryan lieryan changed the title Remove usage of pickle from objectdb and historydb Remove usage of pickle from objectdb and history Dec 16, 2022
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

No branches or pull requests

1 participant