Make the cache size configurable

[dadrus]

First of all, thank you very much for the great project!

Currently, there is an option to configure the capacity of the cache in sense of the amount of items it should store via WithCapacity option. It does however not take the size of the items into account (by size I mean the memory used by the Item objects). In my application the size of the stored items can vary pretty much.

Typically, there is a strong need to configure the limits of a container, incl how much memory it should be able to use. Unfortunately, the above said option does not allow tuning the memory, the application making use of ttlcache, may use for cache purpose. And that makes the configuration of container limits pretty hard.

Do you see this as a viable option to add?

Thank you in advance.

p.s. The background for this FR is described in dadrus/heimdall#1250

[davseby]

Hi, an interesting idea. I'm not sure if we can compute the size of items ourselves without creating a fancy bytes computation method that may not work for everyone. Using unsafe's Sizeof doesn't seem to meet the requirements as well.

However, I think we could introduce a weight system. For example, when creating a cache a new option would be introduced.

func WithMaximumWeight[K comparable, V any](weight uint64, weightFn func(V) uint64) Option[K, V]

The user would be responsible for creating his own weight computation function which could be as flexible as one would want (e.g from computing the actual size in bytes to computing the weight using some sort of attribute, etc...).

@swithek thoughts?

[swithek]

However, I think we could introduce a weight system.

Yes, I agree. Besides, we might tackle more than one problem/use case with this kind of feature. However, one big issue with both of these approaches is that the values might grow in size/weight when no set/update/etc. operations are performed. I guess we would have to handle this with some kind of pub/sub (or notification) system, or just update the eviction queues when a new value is inserted/updated and not care about the value sizes at other times.

[dadrus]

Maybe one could do something similar rueidis (a redis client) is doing here: https://github.com/redis/rueidis/blob/cf567f7740252126f7932bbdbb4c6be1613e0cfb/lru.go#L235. In their case, the value type is actually known in advance.

[swithek]

@dadrus thanks for the example. I just wonder how we should approach the problem I outlined above, i.e., should we track dynamic size updates (the value can be modified by the user if they still have a reference to it) and make changes/evictions based on that or not?

[dadrus]

FMPOV, the implementation should only care about the values it has stored and not about changes done outside of it. So, my vote goes to "update the eviction queues when a new value is inserted/updated and not care about the value sizes at other times".