Latest update of KBFS eating up a bunch of memory on macOS again.

[Naville]

I've seen previous report like #1529 , but I didn't experience similar issue until recently I upgraded my Keybase install when prompted to do so.
Activity Monitor always shows that the memory usage of kbfs is around 3.2~4.5GB. One time it even reached 32GB! (Yeah you read that correct, thirty-two GB of RAM, may Mac has only 16GB RAM)

[maxtaco]

Better to delete these and just do ‘keybase log send’. @strib will check them out.

…

On Sun, Nov 18, 2018 at 9:28 AM Zhang ***@***.***> wrote: Here are some logs Keybase.app.log <https://github.com/keybase/kbfs/files/2592841/Keybase.app.log> keybase.kbfs.log <https://github.com/keybase/kbfs/files/2592842/keybase.kbfs.log> keybase.service.log <https://github.com/keybase/kbfs/files/2592843/keybase.service.log> keybase.start.log <https://github.com/keybase/kbfs/files/2592844/keybase.start.log> keybase.updater.log <https://github.com/keybase/kbfs/files/2592845/keybase.updater.log> — You are receiving this because you are subscribed to this thread. Reply to this email directly, view it on GitHub <#1918 (comment)>, or mute the thread <https://github.com/notifications/unsubscribe-auth/AA05_7ycdFIP-E8VS-ZUcC5pss7vIX4qks5uwW6WgaJpZM4Yn9vn> .

[Naville]

Also I didn't do anything except overwritten one very large file with its newer local version, which is merely 700MB. And that issue persists long after the upload has finished

[Naville]

Better to delete these and just do ‘keybase log send’. @strib will check them out.
…

Cheers. My log id is : 527a0fc7f875635a10beca1c

[Naville]

So the problem is how can I just uninstall/disable kbfs while keeping everything else as-is? The RAM usage after a reboot is now 4.47GB+ and counting. This is driving me nuts

[strib]

Your local device is in a conflict state with the server, with respect to about 2 GB of data in one of your folders. Normally that should be fine and it should resolve the conflict without you needing to do anything, but it seems like KBFS is being frequently restarted before it can finish its work. Maybe due to excessive memory consumption, based on your description above? Right now our conflict resolution process doesn't handle large files in a memory-efficient way, so that might be the issue. In any case, it doesn't have anything to do with the latest update, but with the state of your local device.

To work around this for now, you should:

1. Make a local copy of any unsynced data from your local folder that you don't already have backed up. (This is data you only see on this device in the folder, but can't yet see from your other devices.)
2. Completely quit the Keybase app.
3. Run this command from the terminal:

mv "$HOME/Library/Application Support/Keybase/kbfs_journal/v1/0120f9b2ad24ee89f66c77ef9a04278b834d-2af519d97cb3c381" "$HOME/Library/Application Support/Keybase/kbfs_journal/v1/2af519d97cb3c381.bkp"

4. Restart Keybase.

At this point you should be able to copy the data back into the folder and proceed normally again. Let me know how it goes.

[strib]

(Note: edited the command above ^)

[Naville]

Followed your steps and rebooted and it now consistently uses RAM at around 140MB, which I guess is perfectly normal.
However my private local folder (aka /keybase/private/navillezhang/) is smaller than 2GB so I'm not sure where does the conflict comes from.

Regarding frequently restarted. I manually killed it one or two times, not sure if there are more restarts. My RAM was never fully used so I don't think RAM is the reason for potential restarts

[Naville]

As you can see here, adding up together private and public is around 1GB. On a side note, that 25.2MB file in private folder is the very large file I mentioned before which supposed to be 650-700MB.

I've previously uploaded earlier versions of the same file to kbfs and there was no issue whatsoever, each time overwritting the old file.

[strib]

Ok cool, it was probably the manual restarts then.

I can't tell from the logs exactly where the data usage was coming from. Maybe it was multiple versions of the 700 MB file you wrote? And that 2GB includes the encryption and padding overhead, so it's hard to be exact.

Anyway I'm glad it's working ok now, sorry for the bad experience. Close this out if things look ok, and let us know if you have more issues.

[Naville]

Maybe it was multiple versions of the 700 MB file you wrote?

Older versions shouldn't exist since last overwrite of the same file before this was around 2months ago and I clean reinstalled macOS since then.
It was a large SQL DB dump, and the upload causing the trouble was dumped from the DB which I've modified table structures. So my best guess is that the diff is too large (if kbfs was doing some kind of git-like text compare version control, this is probably the reason of the huge mem usage)