Don't generate a mutex for each Locked<T>, share one per object

[Al2Klimov]

This reduces RAM usage per object by sizeof(mutex)*(FIELDS-1).

The atomic_load(const std::shared_ptr<T>*) idea would, without doubt, also consume less RAM. However, it would scale worse, the more objects you have, with a fixed number of mutexes. Neither one SpinMutex per property, nor one std::mutex per object suffer from this limitation.

[julianbrost]

This misses part of what #10113 asked for:

1. Figure out how much of an effect this has on the total memory use of Icinga 2.

We have a vague "Icinga 2 uses more memory in newer versions", so how much does this affect the total memory use of the whole process? Might this be the answer or is it just a small puzzle piece?

Also, what made you choose adding that m_FieldsMutex and passing it around over what I suggested in #10113? I'm not a big fan of how random code (see the changes to lib/icingadb/ now has to access some internal attribute generated by mkclass. A nice aspect of the current implementation is that it's very easy to reason that it won't deadlock, all the magic happens here:

icinga2/lib/base/atomic.hpp

Lines 52 to 73 in 585b357

	template<typename T>
	class Locked
	{
	public:
	inline T load() const
	{
	std::unique_lock<std::mutex> lock(m_Mutex);
	return m_Value;
	}
	inline void store(T desired)
	{
	std::unique_lock<std::mutex> lock(m_Mutex);
	m_Value = std::move(desired);
	}
	private:
	mutable std::mutex m_Mutex;
	T m_Value;
	};

Whereas with this PR, exposing the underlying mutex, this gets more spread out in the code base.

[Al2Klimov]

Also, what made you choose adding that m_FieldsMutex and passing it around over what I suggested in #10113?

sharing the mutex between objects, this would reduce the memory requirements

The more fields, especially objects, share one mutex, the slower they can proceed. I prefer 1+ mutex per separate object.

I'm not a big fan of how random code (see the changes to lib/icingadb/ now has to access some internal attribute generated by mkclass. A nice aspect of the current implementation is that it's very easy to reason that it won't deadlock, all the magic happens here:

icinga2/lib/base/atomic.hpp

Lines 52 to 73 in 585b357

	template<typename T>
	class Locked
	{
	public:
	inline T load() const
	{
	std::unique_lock<std::mutex> lock(m_Mutex);
	return m_Value;
	}
	inline void store(T desired)
	{
	std::unique_lock<std::mutex> lock(m_Mutex);
	m_Value = std::move(desired);
	}
	private:
	mutable std::mutex m_Mutex;
	T m_Value;
	};

Whereas with this PR, exposing the underlying mutex, this gets more spread out in the code base.

You're right. One wrong move on the mutex and... 🙈

[julianbrost]

Why do you think a spinlock is a good idea here?

Let me quote cppreference.com for std::atomic_flag:

A spinlock mutex demo can be implemented in userspace using an atomic_flag. Do note that spinlock mutexes are extremely dubious in practice.

And we even had problems in Icinga 2 in the past due to a spinlock, see #8507/#8488. So I have doubts that would be a good idea this time.

[Al2Klimov]

Why do you think a spinlock is a good idea here?

std::mutex is 5 pointers large on x64, whereas std::atomic_flag at most one.

spinlock mutexes are extremely dubious in practice.

But the lock scope here is very small, isn't it? Not to mention how unlikely you'll hit the slow path at all in our case...

[julianbrost]

But the lock scope here is very small, isn't it?

Depends on what's inside. If it's an intrusive pointer, it just needs to increment the reference count, if it's a string though, it has to copy it, including allocating memory for it.

Not to mention how unlikely you'll hit the slow path at all in our case...

How did you conclude it's unlikely? #9364 was done for a reason, there were random crashes because. Crash means concurrent access, so that will go into the slow path. Also note that not every concurrent access resulted in a crash, so there will be more "hitting the slow path" than there were crashes.

[julianbrost]

Is this change hidden in a big rebase push related to this conversation? If so, please comment why you think this resolves it. Otherwise, what was the motivation for that change and how will you address this?

[Al2Klimov]

But the lock scope here is very small, isn't it?

Depends on what's inside. If it's an intrusive pointer, it just needs to increment the reference count, if it's a string though, it has to copy it, including allocating memory for it.

You're right. Especially the latter case can take a few more CPU cycles. So instead of wasting them on pure spinning in a waiting thread, the latter now says "after you" while something's locked. This shall bring the mutex holder in foreground and let it finish and unlock.

[julianbrost]

Please use a search engine of your choice to search for the keywords "userspace" and "spinlock". Seems like the majority isn't too positive about that idea. And as already said, Icinga 2 already had a negative experience with spinlocks itself. What makes you believe that now is a good time to use a spinlock in userspace?

By the way, Linus Torvalds' rant post (the one linked from cppreference.com) even has a follow-up from him on yield.

Apart from that, the current version of the PR seems to restrict the use of that suggested spinlock to just intrusive_ptr, so each string attribute would still use its own std::mutex. So the memory use will be somewhere in the middle (I haven't looked into the distribution of attribute types), but why do it that way if it can be done better?

The atomic_load(const std::shared_ptr<T>*) idea would, without doubt, also consume less RAM. However, it would scale worse, the more objects you have, with a fixed number of mutexes.

However, while there may be a large number of objects, the number of CPU cores and thereby threads we start is typically much smaller. And that actually limits the total contention. There won't be a thread for each single object trying to lock it.

[Al2Klimov]

However, while there may be a large number of objects, the number of CPU cores and thereby threads we start is typically much smaller. And that actually limits the total contention. There won't be a thread for each single object trying to lock it.

This sounds nice in theory, but I'd prefer a solution much more predicable, especially for large setups. I mean my one mutex per object idea.

[Al2Klimov]

I'm not a big fan of how random code (see the changes to lib/icingadb/ now has to access some internal attribute generated by mkclass. A nice aspect of the current implementation is that it's very easy to reason that it won't deadlock, all the magic happens here:

icinga2/lib/base/atomic.hpp

Lines 52 to 73 in 585b357

	template<typename T>
	class Locked
	{
	public:
	inline T load() const
	{
	std::unique_lock<std::mutex> lock(m_Mutex);
	return m_Value;
	}
	inline void store(T desired)
	{
	std::unique_lock<std::mutex> lock(m_Mutex);
	m_Value = std::move(desired);
	}
	private:
	mutable std::mutex m_Mutex;
	T m_Value;
	};

Whereas with this PR, exposing the underlying mutex, this gets more spread out in the code base.

I've fixed it. :)

[Al2Klimov]

And the cool "side" effect here is that the amount of mutexes scales linear with the amount of objects. So, in contrast to a central array of mutexes created at reload time, you guaranteed won't ever have too few mutexes.

[julianbrost]

Why do you think there would be a problem with "too few mutexes"? Like you'll have a more or less fixed number of threads started by Icinga 2, so that limits the concurrency and would be a good candidate for sizing the number of mutexes in a std::atomic/libatomic-like implementation.

I'm not opposed to the idea of using one mutex per object, however, the current PR introduces a quite strange interface with AtomicPseudoLocked where you have to specify a mutex which then is ignored and has to be specified in the custom get {{{ }}} implementations. What if mkclass would just emit the required locking code in the relevant getter methods instead? Like depending on the type, it will either be an atomic load or a lock + non-atomic load + unlock. (And similar for the setters of course.)

Nonetheless, I still don't think that would be necessary and having a global array of mutexes would suffice and keep the implementation simpler.

[Al2Klimov]

Why do you think there would be a problem with "too few mutexes"?

No. I don't. I don't know.

I don't know how many mutexes are enough – O(?).
I don't know the hashing algorithm mapping tons of Locked<> in memory to the O(?) mutexes in your scenario and how fair it "schedules".
And beyond the things which I know and the (above) ones I know that I don't know, there are things I don't know that I don't know.

The more radically you change a running system in an amount of time, the more likely you hit a (delayed) black swan. Like JSON-RPC crash one (v2.11.3), JSON-RPC crash two (v2.11.4), IcingaDB(?) OOM _{ref/NC/820479} (v2.?) ...

I'm not opposed to the idea of using one mutex per object,

👍

Btw. I'm neither opposed to yours.
I just lack the knowledge to implement it by myself properly enough to sleep well afterwards.

AtomicPseudoLocked where you have to specify a mutex which then is ignored and has to be specified in the custom get {{{ }}} implementations

No, yes and no.
AtomicPseudoLocked<T> just extends std::atomic<T> with two additional LockedMutex-compliant methods you can call if you wish. They indeed ignore the mutex _{(God bless compiler optimization!)} and call their (public) equivalents which don't take one. Custom get {{{ }}} implementations can also call the latter, not specifying any mutex, but all .ti files in the current diff override get {{{ }}} for strings. So they need the mutex and they actually lock it.

What if mkclass would just emit the required locking code in the relevant getter methods instead? Like depending on the type, it will either be an atomic load or a lock + non-atomic load + unlock.

mkclass is surely a good tool for stuff that would be significantly(?) more difficult in vanilla C++.
But from my experience I'd prefer "native solutions" (e.g vanilla C++) if applicable easily enough.