-
Notifications
You must be signed in to change notification settings - Fork 163
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Optimising high dispatch and db query counts in broad model #1607
Comments
Hi @KlausVii, sorry for the delay!
If your model had relations with overlapping definitions, such as
Then we could introduce
This should not be the case - that flag is meant to increase the degree of concurrency that queries can have. It doesn't change the final outcome. A few things you can try:
In the short term we are planning to work on some performance improvements that could help your use case, (e.g. #1012) |
Hmm, we've done some hacks to bypass openfga for the problematic checks for now. Increasing the timeout was not an acceptable solution for us. The max conns is already quite high and the query was negatively affecting other checks. When are you planning on implementing this leopard index? Would that help? |
@KlausVii if you have some sample tuples to share that would be great. That being said, in the meantime, the problem here stems from bi-directional relationships between user and tenant. That is Starting with Shoot those tuples over though and we can see if there are some other options in the meantime. Thanks for the report! |
Hi @jon-whit , here's a sample store export with an example of a very expensive slow-store.txt (its actually yaml, but gh doesn't allow uploading those) To reproduce:
{
"grpc_method": "Check",
"tuple_key": {
"user": "user:f3659913-cde0-4aeb-b23f-dffb3176662b",
"relation": "can_view",
"object": "user:05d2c261-cec3-43d7-9b26-ef07b083f3ee"
},
"raw_response": {
"allowed": false,
"resolution": ""
},
"datastore_query_count": 829,
"dispatch_count": 852
} This is the worst case, where no relation to view exists, but even the more common successful one like this can lead to many db queries: {
"tuple_key": {
"user": "user:0904c37c-a06b-41dc-8c29-5d66540137e9",
"relation": "can_view",
"object": "user:05d2c261-cec3-43d7-9b26-ef07b083f3ee"
},
"raw_response": {
"allowed": true,
"resolution": ""
},
"datastore_query_count": 71,
"dispatch_count": 702
} |
Also is there indication to when the indexing will be implemented? |
Checklist
Description
We are experiencing high latency and failing Check queries in production using the auth model below. The problem seems to stem from a particular set of users who have relations with multiple tenants and projects. When a check is performed with one of these users as the object, we are seeing dispatch counts in the hundreds and db query counts over a 100.
This is causing queries to timeout and putting undue pressure on the db.
Is there anything we could do to make the model perform better while still maintaining the same intent of the model. For example we tried to substitute
owner_tenant
andowner_project
with[tenant#can_view_users, project#can_view_users]
, but this made the performance even worse.Another option I am aware of is decreasing the
OPENFGA_RESOLVE_NODE_BREADTH_LIMIT
, but I'm afraid that will lead to an unacceptable number of false negatives.Any ideas would be appreciated?
Expectation
I'd expect to be able check relations in a timely manner even for objects that have many relations.
Reproduction
owner_tenant
andowner_project
relationsuser[1] can_view user[0]
Store data
OpenFGA version
1.5.3
How are you running OpenFGA?
In Kubernetes
What datastore are you using?
Postgres
OpenFGA Flags
OPENFGA_CHECK_QUERY_CACHE_ENABLED, OPENFGA_CHECK_QUERY_CACHE_TTL=30s, OPENFGA_DATASTORE_MAX_OPEN_CONNS=500, OPENFGA_MAX_CONCURRENT_READS_FOR_CHECK=4000000, OPENFGA_MAX_CONCURRENT_READS_FOR_LIST_OBJECTS=4000000
Logs
The text was updated successfully, but these errors were encountered: