MMKV consumes too much unneeded size on file storage

[coldcode001]

When I save data to MMKV I also put data to a SharedPreferences on Android for comparison purpose, and I found the real MMKV file size is much bigger than the actual size.
For my case the MMKV file consumes 128KB while sharedprefs only occupies 9.3KB.

I found several related closed issues and simply read a bit the code, there's some logic to not do full writeback to file intentionally (for performance?), but it should not consume so much extra size on real file storage, for my case it's more than 10 times of the actual size, it's really too much.

Yep I noticed the API MMKV.trim() but it asks caller to call it after lots of deletions, it's difficult to detect such case if keys are removed in multiple sessions. What is even worse that it calls sync(true) inside, so it's not a good idea to ask user to call trim() at proper time.

MMKV can detect the deletions (with an inner counter?), or decide if it should write back to file by comparing its actual size and file size at proper time.
I think it should not be more than double the size of the actual data size, there are two good timing for this, loaded from file for the first time on cold start, or after tidying the KV data when allocated pages not big enough.

How do you think about this?

[lingol]

It's about performance. The smaller size of the file is, the much more frequent of full-write-back needed. And full-write-back is much much more expensive than a simple append. So no, 'no more than double the size of the actual data size' is not acceptable.

If one really really worries about file size, he will find a proper time to do the trim.

[coldcode001]

If the file size is not big, f.e. it's < 64KB it's ok to do like this, then we can prevent from expensive IO operations of full-write-back.
But along with the app used day after day, the MMKV file will increase to a considerable big size, which is not really necessary, is it really an issue for the lib to do this at proper time? f.e. when the data size changes from 128KB -> 256KB (and 256KB -> 512KB, and so on).

[lingol]

The lib won't know if it will ever need 256KB or 512KB again. Only the user knows the future usage.
One more information, SQLite also won't trim itself. The user has to vacuum manually.

[coldcode001]

For a MMKV instance we can introduce a threshold size for next auto trim, e.g. init it as max(128KB, nearest-power-of-two(actual size)) * 2, then when auto trim occurs, double the threshold size.
In this way the auto trim will not be triggered off too many times, but the file storage will be limited to an acceptable size, and also performs efficiently in most cases.

It's true for the case of SQLite, but MMKV is designed to be more like a SharedPreferences (Key-Value mappings).

If you insist on it's the responsibility of all apps using MMKV but not a reasonable improvement of the lib itself then it's OK.

[lingol]

MMKV won't increase to an unacceptable big size under normal usage. Before each growth, MMKV will always try to pack existing data to see if there's enough space for new commer.
However, if there's an unreasonable big value stored inside once, and if there's not enough space for that big value, MMKV might grow to an unexpected large size.

[lingol]

And there's no way for the lib to know if that big value is of reasonable size or not. Only the user knows.

[coldcode001]

I may need to clarify for my case the biggest value is less than 9.3KB, because there are about 10 keys and the total size is 9.3KB.
It's just a compromise proposal here to prevent the file size from unreasonable increasing.

In the end, if the file size is too large, then it also affects the performance on loading data from file, and affects the start up time.

[lingol]

I did mention about stored inside once and one big value. So the current key-value size doesn't really count, and the average size does not really count either.

[lingol]

2020-12-29 12:29:31.263 19692-19987/com.heytap.health I/MMKV: <MMKV_IO.cpp:367::ensureMemorySize> extending [DB_SP_SYNC_FILE] file size from 4096 to 65536, incoming size:4360, future usage:35120

You can check for logs that look like this.

[ramirobg94]

I have a similar issue, check this -> ammarahm-ed/react-native-mmkv-storage#286
maybe it is related, seems that the same key stores many values