docs: Caveats of sharing a DB with multiple instances (#1743)

Signed-off-by: Charith Ellawala <charith@cerbos.dev>

docs/modules/configuration/pages/storage.adoc

@@ -205,6 +205,9 @@ The Postgres storage backend is one of the dynamic stores that supports adding o
 
 include::partial$cerbosctl.adoc[]
 
+[#postgres-db-sync]
+include::partial$db_sync.adoc[]
+
 NOTE: Unlike the SQLite3 driver, the tables and other database objects are not created automatically by the Cerbos Postgres driver. This is to minimize the privileges the Cerbos instance has on the Postgres installation. You must create the required tables using the provided script before configuring Cerbos to connect to the database.
 
 The driver configuration expects the connection details to be provided as connection URL. See link:https://www.postgresql.org/docs/current/libpq-connect.html#LIBPQ-CONNSTRING[Postgres connstring documentation] for more information. Use the `search_path` parameter to point to the schema containing the Cerbos tables.
@@ -237,8 +240,6 @@ storage:
       maxIdle: 5
 ----
 
-
-
 [#postgres-schema]
 === Database object definitions
 
@@ -257,6 +258,9 @@ The MySQL storage backend is one of the dynamic stores that supports adding or u
 
 include::partial$cerbosctl.adoc[]
 
+[#mysql-db-sync]
+include::partial$db_sync.adoc[]
+
 NOTE: Unlike the SQLite3 driver, the tables and other database objects are not created automatically by the Cerbos MySQL driver. This is to minimize the privileges the Cerbos instance has on the MySQL installation. You must create the required tables using the provided script before configuring Cerbos to connect to the database.
 
 The driver configuration expects the connection details to be provided as a DSN in the following form:
@@ -325,6 +329,7 @@ storage:
       maxIdle: 5
 ----
 
+
 [#mysql-schema]
 === Database object definitions
 
@@ -342,6 +347,9 @@ The SQL Server storage backend is one of the dynamic stores that supports adding
 
 include::partial$cerbosctl.adoc[]
 
+[#sqlserver-db-sync]
+include::partial$db_sync.adoc[]
+
 NOTE: Unlike the SQLite3 driver, the tables and other database objects are not created automatically by the Cerbos SQL Server driver. This is to minimize the privileges the Cerbos instance has on the SQL Server installation. You must create the required tables using the provided script before configuring Cerbos to connect to the database.
 
 The driver configuration expects the connection details to be provided as connection URL. See link:https://github.com/denisenkom/go-mssqldb#connection-parameters-and-dsn[SQL Server connstring documentation] for more information. Use the `database` parameter to point to the database containing the Cerbos tables.
@@ -374,6 +382,7 @@ storage:
       maxIdle: 5
 ----
 
+
 [#sqlserver-schema]
 === Database object definitions
 
@@ -396,7 +405,7 @@ You can configure the fallback error threshold and the fallback error window to
 ----
 storage:
   driver: "overlay"
-  overlay: 
+  overlay:
     baseDriver: postgres
     fallbackDriver: disk
     fallbackErrorThreshold: 5 # number of errors that occur within the fallbackErrorWindow to trigger failover

docs/modules/configuration/partials/db_sync.adoc

@@ -0,0 +1,10 @@
+[IMPORTANT]
+====
+
+Cerbos has an in-memory cache for holding compiled policy definitions to speed up the evaluation process. When a policy is removed or updated using the xref:api:admin_api.adoc#policy-management[Admin API] this cache is updated by the instance that handles the request. However, if you share the database with multiple Cerbos instances, the other instances won't be aware of the change and might still have the old policy definition cached in memory. There are several ways to handle this situation.
+
+- By default, the cache entries are stored indefinitely until there's memory pressure. You can set a maximum cache duration for entries by setting the `compile.cacheDuration` configuration value. This could help make all the Cerbos instances to become eventually consistent within a    time frame that's acceptable to you.
+- You can turn off caching completely by setting `compile.cacheSize` configuration to `0`. This would have a small performance hit but depending on your use case it could be negligible.
+- Invoke the xref:api:admin_api.adoc#store-management[`/admin/store/reload` API endpoint] on all the Cerbos instances whenever you make a change to your policies.
+
+====