Handling Persistent Storage Overflow with TanStack Query and persister

[frederikhors]

Describe the bug

I'm opening this because I think this is a very important issue without a final solution.

@hirbod opened #6788 and proposed #6788 (comment).

But I think we need more discussion on this.

How often does this bug happen?

Every time

TanStack Query version

5

[hirbod]

Thanks @frederikhors - I agree this needs more visibility.
For anybody looking for an idea/solution for now, this is what I call on every app start (native; or reload in terms of our SPA for web)

const buildKey = process.env.CODEBUILD_BUILD_ID
const buster = `whatever-your-like-${buildKey}`
const prefix = 'your-prefix-entry'

export const clearStalePersistedReactQueryData = () => {
  // this is needed to avoid issues with SSR rendering
  // prevents metro from build failing
  if (typeof window === 'undefined') return

  const persistentStoreTime = Date.now() - 1 * 24 * 60 * 60 * 1000 // Timestamp of 1 day

  storage.getAllKeys().forEach((entry) => {
    // we only want to clear the data for react-query with the prefix defined above
    if (!entry.startsWith(prefix)) {
      return
    }

    const rawValue = storage.getString(entry)
    if (!rawValue) return

    try {
      const value = JSON.parse(rawValue)
      
      // Check if the entry contains "buster" and if it matches the current buster
      // otherwise delete the entry
      if (value?.buster !== buster) {
        storage.delete(entry)
        return
      }

      if (value?.state?.dataUpdatedAt && value.state.dataUpdatedAt < persistentStoreTime) {
        // Delete the stale entry
        storage.delete(entry)
      } else {
        // Check if the entry contains "pages" and "pageParams"
        if (value.state.data && value.state.data.pages && value.state.data.pageParams) {
          const { pages, pageParams } = value.state.data

          // Check if the length of "pages" and "pageParams" is greater than 1
          if (pages.length > 1 && pageParams.length > 1) {
            // Slice "pages" and "pageParams" to keep only the first entry
            value.state.data.pages = pages.slice(0, 1)
            value.state.data.pageParams = pageParams.slice(0, 1)

            // Update the stored value with the modified data
            storage.set(entry, JSON.stringify(value))
          }
        }
      }
    } catch (error) {
      console.error(`Error parsing JSON for key ${entry}:`, error)
    }
  })
}

In case you're confused about the "pages" part. We just want to persist one page. maxPages is not working well (both problems with react-native and the persister)

[TkDodo]

we already have an issue for this:

• experimental_createPersister is not cleaning up after itself #7049