Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

SQLite support #1615

Open
wants to merge 48 commits into
base: pluggable-datastores
Choose a base branch
from
Open

SQLite support #1615

wants to merge 48 commits into from

Conversation

DanCech
Copy link

@DanCech DanCech commented May 10, 2024

This PR adds support for using SQLite as a backend database.

Description

The SQLite storage implementation is largely based on the existing MySQL backend, with some slight differences:

  • migration 005 updated to issue separate ALTER TABLE statements for each column, as SQLite does not support multi-column ALTER statements
  • date-related SQL updated to use SQLite syntax
  • retry mechanism added for writes to avoid SQLITE_BUSY errors (this may not actually be needed with the busy_timeout setting, but is a backup in case the database layer does return SQLITE_BUSY when heavily loaded).
  • error handling for SQLite unique constraint errors

It uses the cgo-free https://modernc.org/sqlite driver.

Unless otherwise specified, it will use the WAL journal mode and specify a busy timeout of 500ms

jon-whit and others added 30 commits April 16, 2024 14:27
* fix: error handling leading to false positives

* test: add a failing test due to Check and ListObjects differences

* test: change test expectation for ListObjects assertion

* fix: union CheckFuncReducer and associated tests

* fix: intersection CheckFuncReducer

* fix: exclusion CheckFuncReducer

* fix: rewire the cycle detection mechanics to avoid error propagation

* test: add more edge case tests for cyclical evaluations

* test: rename new test that was poorly named

* chore: revert minor change that was unintended

* chore: tidy up changes and drop commented out code

* chore: minor touchups based on some feedback

* test: add test for CloneResolveCheckResponse

* test: add more tests for error around intersection

* test: more test assertions around cloned cached response

* test: add unit test to make sure we cache CycleDetected

* test: add more assertions around CycleDetected field

* chore: rename test names

* chore: fix datastore query count under reporting in intersection reducer
openfga#1543)

Co-authored-by: Maria Ines Parnisari <maria.inesparnisari@okta.com>
Signed-off-by: dependabot[bot] <support@github.com>
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
Co-authored-by: Jonathan Whitaker <jon.b.whitaker@gmail.com>
…o 26.0.2+incompatible (openfga#1553)

Signed-off-by: dependabot[bot] <support@github.com>
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
Signed-off-by: dependabot[bot] <support@github.com>
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
…n the dependencies group (openfga#1560)

Signed-off-by: dependabot[bot] <support@github.com>
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
….4.26 in the dependencies group (openfga#1562)

Signed-off-by: dependabot[bot] <support@github.com>
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
Co-authored-by: Jonathan Whitaker <jon.b.whitaker@gmail.com>
Signed-off-by: dependabot[bot] <support@github.com>
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
…penfga#1546)

Co-authored-by: Jonathan Whitaker <jon.b.whitaker@gmail.com>
@DanCech DanCech requested a review from a team as a code owner May 10, 2024 21:12
miparnisari and others added 8 commits May 10, 2024 15:22
…e dependencies group (openfga#1618)

Signed-off-by: dependabot[bot] <support@github.com>
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
…dates (openfga#1623)

Signed-off-by: dependabot[bot] <support@github.com>
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
…guration (openfga#1571)

Co-authored-by: Jonathan Whitaker <jon.b.whitaker@gmail.com>
Co-authored-by: Maria Ines Parnisari <maria.inesparnisari@okta.com>
@jon-whit jon-whit changed the base branch from main to pluggable-datastores May 29, 2024 16:00
@jon-whit
Copy link
Member

@DanCech I changed the base branch this PR is targeting because we're devising a strategy for how to handle community storage contributions without taking on the burden of the storage implementation longer term. We want to promote development, but the maintainer team can't practically support all of them and offer quality implementations if there are a lot of them we're having to maintain.

Consequently, we're thinking of an openfga/openfga-contrib repository structure and a plugin based mechanism that will allow the OpenFGA community to run OpenFGA with externally contributed storage implementations. We likely won't merge this into main and include it in an official release until we have a more well-defined design for that plugin ecosystem, and in the (hopefully near) future the SQLite implementation you provided herein will be one of the first community sponsored storage implementations in the contrib repo. See https://github.com/openfga/openfga-contrib/blob/main/storage/sqlite/sqlite.go

@DanCech
Copy link
Author

DanCech commented May 29, 2024

Thanks for the feedback @jon-whit , it looks like the base change has created quite a few conflicts. Is there anything I can do at this stage to help move things forward?

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

None yet

8 participants