Planning: Limit ability of synthetic nodes to take up connection slots

[arya2]

Motivation

We want to discuss the impact of each potential fix on this vulnerability, and decide on 1-3 fixes that have the greatest impact.

Potential Fixes

This is the list of potential fixes from issue #7822:

• Increase the minimum number of connected peers to the initial target peerset size, so that Zebra always has a variety of peers, and the first/newest/fastest peers don't take up all the slots (also helps with Tracking: security: Limit ability of synthetic nodes to spread throughout network. Credit: Ziggurat Team #7824)
  

  zebra/zebra-network/src/peer_set/initialize.rs

  Line 913 in 7e7f989

  let should_always_dial = active_outbound_connections.update_count() == 0;

• Retry failed peers only if we've successfully connected to them, don't believe untrusted_last_seen

  zebra/zebra-network/src/meta_addr.rs

  Lines 639 to 640 in 7e7f989

  	Some(last_seen) => last_seen.saturating_elapsed(now) <= constants::MAX_RECENT_PEER_AGE,
  	None => false,

• Retry other failed peers (untrusted or no last seen time) if their last attempt is longer than MAX_PEER_ACTIVE_FOR_GOSSIP (or maybe just 10+ minutes?) but less than MAX_RECENT_PEER_AGE
• Make the outbound connection rate slower when there are lots of recent outbound connections #7852
• Zebra attempts new peer connections in a fixed, predictable order #1875
• If we're not getting any new blocks in the progress task, send a ChangePeerConnections request to the network to disconnect some peers
• Increase GET_ADDR_FANOUT to 2 or 3, to increase the diversity of the address book
• Reduce the TimerCrawl interval to increase the diversity of the address book

[arya2]

Is there a good way for Zebra to recognize and drop synthetic peers once they're connected?

[teor2345]

I don't think there is. Any algorithm that tries to do general detection of fake or bad peers can be defeated by this attack:

1. Start a fake peer.
2. Connect to the target peer.
3. Connect to another real peer.
4. When the target peer sends messages, proxy them to the real peer, and return its responses. Similarly, forward unsolicited messages from the real peer to the target peer.
5. Modify or introduce traffic as needed to attack the target peer.

Instead, if Zebra has a specific requirement of its peers, I suggest we rotate peers until we get peers that satisfy that requirement. For example, this suggested fix:

If we're not getting any new blocks in the progress task, send a ChangePeerConnections request to the network to disconnect some peers

[arya2]

When the target peer sends messages, proxy them to the real peer, and return its responses. Similarly, forward unsolicited messages from the real peer to the target peer.

Is it possible for the target to guess the likelihood that there's a proxy among a subset of peers when it's connected to the real peer by multiple proxies?

[teor2345]

When the target peer sends messages, proxy them to the real peer, and return its responses. Similarly, forward unsolicited messages from the real peer to the target peer.

Is it possible for the target to guess the likelihood that there's a proxy among a subset of peers when it's connected to the real peer by multiple proxies?

It might help to consider this attack as a thought experiment, rather than an actual practical attack. The underlying assumption is that it is possible to easily emulate and modify a real peer to get around any general rule about what peers should do.

For example, you can also emulate a peer by forking software, recording and replaying responses, or creating a minimal peer implementation that satisfies the general "good peer" rules.

And as a side note, creating those kinds of general rules about peer behaviour also makes it harder to write compatible protocol implementations. And it can open the network up to other attacks that induce "bad" behaviour in real peers.

[arya2]

Potential Fix: Increase the minimum number of connected peers to the initial target peerset size, so that Zebra always has a variety of peers, and the first/newest/fastest peers don't take up all the slots

[arya2]

This slows down how quickly an adversary could fill up the connection slots because the connectors are rate limited.

This is likely a required fix depending on how much time Zebra needs to confidently recognize and drop synthetic peers.

[teor2345]

Disadvantage: increases the load on the network from peers that aren't doing any work. This is a minor disadvantage, being vulnerable to attacks is worse.

This is likely a required fix depending on how much time Zebra needs to confidently recognize and drop synthetic peers.

Unless there is a required fix that we are sure will identify all synthetic peers, then this fix is required.

[arya2]

Potential Fix: Retry failed peers only if we've successfully connected to them, don't believe untrusted_last_seen

[arya2]

This is a required fix to prevent an easy exploit, there are alternatives, but I think we should do this one.

[teor2345]

Drawbacks: peers won't be retried if Zebra starts while disconnected from the network, or becomes disconnected while trying a lot of new peers. This is a major drawback that will significantly impact Zebra's connectivity.

Alternative: Instead, I suggest we retry peers without successful connections last.
Analysis Needed: Is this the existing implementation, or do we need to make changes?

[teor2345]

The existing behaviour is to order peers without successful connections last, with the standard 2 minute reconnection delay:

zebra/zebra-network/src/meta_addr.rs

Lines 1122 to 1132 in 193fd42

	// # Security
	//
	// Compare local times before untrusted gossiped times and services.
	// This gives malicious peers less influence over our peer connection
	// order.
	// If all local times are None, try peers that other peers have seen more recently
	let newer_untrusted_last_seen = self
	.untrusted_last_seen
	.cmp(&other.untrusted_last_seen)
	.reverse();

So do we need to change anything here?

[arya2]

Potential Fix: Retry other failed peers (untrusted or no last seen time) if their last attempt is longer than MAX_PEER_ACTIVE_FOR_GOSSIP (or maybe just 10+ minutes?) but less than MAX_RECENT_PEER_AGE

[arya2]

This may help in some cases where Zebra's nearly starved of peers, I'm not entirely sure, it depends on the other fixes, but it seems only tangentially relevant to me.

[teor2345]

Is this an alternative or a mitigation for the drawbacks of "Retry failed peers only if we've successfully connected to them"?

Benefits: Possible mitigation for other drawbacks.
Drawbacks: Delays could impact connectivity, we'd need to track another timer, hard to analyse.

Analysis Needed: Is the existing behaviour to have a delay, never retry, or retry last with no delay?

[teor2345]

The existing behaviour is to retry last with the standard 2 minute delay, see:
#7987 (reply in thread)

[teor2345]

I don't think we need to make any changes here.

[arya2]

Potential Fix:

• Make the outbound connection rate slower when there are lots of recent outbound connections #7852

[arya2]

This is likely a required fix depending on how much time Zebra needs to confidently recognize and drop synthetic peers.

[teor2345]

We already delay reconnections to the same IP for around 2 minutes, unless that IP is pushed out of the address book by new gossiped peers.

Benefits: Possible mitigation for rapid peer reconnections in a large address book.
Drawbacks: Duplicates existing behaviour but in a different way, we'd need to track another timer, hard to analyse.

Alternatives:

• Increase the size of the address book
• Increase the existing per-IP reconnection delay

Analysis Needed:

• Is the address book limit ever reached after our recent changes? How often, and how many peers are removed? Are they valid peers where a reconnection could cause performance issues or network load?
• Is the per-IP reconnection delay long enough? (2 minutes means there will be one connection per second if 120 peers behave this way.)

[teor2345]

I think the root cause/fix here is having diverse peers to connect to. Delaying the connection doesn't change which peer we will connect to next, so I'm not sure if this fix will help.

[teor2345]

There is an attack where an attacker tries to add lots of peers, fills the address book, and pushes their own IPs out of the address book, then adds their IPs again, and then has us reconnect within less than our 2 minute limit.

This attack isn't viable because:

• Zebra controls when peers are added to the address book, and limits the amount and frequency of peers being added
• Zebra limits outbound connection rates, it takes 8 minutes for Zebra to connect to a full address book, but the individual IP limit is 2 minutes

So it is actually better for the peer to stay in the address book.

[teor2345]

I don't think this change is guaranteed to help, and it might hurt security in other ways.

[arya2]

Potential Fix:

• Zebra attempts new peer connections in a fixed, predictable order #1875

[arya2]

Benefit: This practically prevents any attempt to predict which addresses are used by Zebra's outbound connector.

Drawbacks (from the issue):

this change risks connecting to slightly worse peers, or skipping some peers that would otherwise have been chosen for reconnection.

The drawback seems to outweigh the benefit, it's already very difficult to predict which addresses make it to the address book to be used by Zebra's outbound connector.

I think always caching some portion (in this case half) of the addresses from a GetAddr response, even if it's below half of the outbound connection limit would accomplish the same goal with a less-severe drawback in normal conditions.

[teor2345]

Since #7824, peer are chosen at random from gossiped addresses.

I think always caching some portion (in this case half) of the addresses from a GetAddr response, even if it's below half of the outbound connection limit would accomplish the same goal with a less-severe drawback in normal conditions.

We could make that randomness apply to more peer gossips, but whatever limit we choose, there will be some size of gossip that remote peers will be able to exploit. (In this case, 37 or less.)

But that wouldn't matter if we randomised reconnections themselves.

Once peers are in the address book, it is easy to predict or exploit their connection order, because we use a fixed ordering for reconnections, based on fields controlled by remote peers:

zebra/zebra-network/src/address_book.rs

Lines 590 to 602 in 193fd42

	/// Return an iterator over peers that are due for a reconnection attempt,
	/// in reconnection attempt order.
	pub fn reconnection_peers(
	&'_ self,
	instant_now: Instant,
	chrono_now: chrono::DateTime<Utc>,
	) -> impl Iterator<Item = MetaAddr> + DoubleEndedIterator + '_ {
	let _guard = self.span.enter();
	// Skip live peers, and peers pending a reconnect attempt.
	// The peers are already stored in sorted order.
	self.by_addr
	.descending_values()

And MetaAddr:

zebra/zebra-network/src/meta_addr.rs

Lines 1089 to 1171 in 193fd42

	// First, try states that are more likely to work
	let more_reliable_state = self.last_connection_state.cmp(&other.last_connection_state);
	// Then, try addresses that are more likely to be valid.
	// Currently, this prefers addresses with canonical Zcash ports.
	let more_likely_valid = self.peer_preference().cmp(&other.peer_preference());
	// # Security and Correctness
	//
	// Prioritise older attempt times, so we try all peers in each state,
	// before re-trying any of them. This avoids repeatedly reconnecting to
	// peers that aren't working.
	//
	// Using the internal attempt time for peer ordering also minimises the
	// amount of information `Addrs` responses leak about Zebra's retry order.
	// If the states are the same, try peers that we haven't tried for a while.
	//
	// Each state change updates a specific time field, and
	// None is less than Some(T),
	// so the resulting ordering for each state is:
	// - Responded: oldest attempts first (attempt times are required and unique)
	// - NeverAttempted...: recent gossiped times first (all other times are None)
	// - Failed: oldest attempts first (attempt times are required and unique)
	// - AttemptPending: oldest attempts first (attempt times are required and unique)
	//
	// We also compare the other local times, because:
	// - seed peers may not have an attempt time, and
	// - updates can be applied to the address book in any order.
	let older_attempt = self.last_attempt.cmp(&other.last_attempt);
	let older_failure = self.last_failure.cmp(&other.last_failure);
	let older_response = self.last_response.cmp(&other.last_response);
	// # Security
	//
	// Compare local times before untrusted gossiped times and services.
	// This gives malicious peers less influence over our peer connection
	// order.
	// If all local times are None, try peers that other peers have seen more recently
	let newer_untrusted_last_seen = self
	.untrusted_last_seen
	.cmp(&other.untrusted_last_seen)
	.reverse();
	// Finally, prefer numerically larger service bit patterns
	//
	// As of June 2021, Zebra only recognises the NODE_NETWORK bit.
	// When making outbound connections, Zebra skips non-nodes.
	// So this comparison will have no impact until Zebra implements
	// more service features.
	//
	// None is less than Some(T), so peers with missing services are chosen last.
	//
	// TODO: order services by usefulness, not bit pattern values (#2324)
	// Security: split gossiped and direct services
	let larger_services = self.services.cmp(&other.services);
	// The remaining comparisons are meaningless for peer connection priority.
	// But they are required so that we have a total order on `MetaAddr` values:
	// self and other must compare as Equal iff they are equal.
	// As a tie-breaker, compare ip and port numerically
	//
	// Since SocketAddrs are unique in the address book, these comparisons
	// guarantee a total, unique order.
	let ip_tie_breaker = match (self.addr.ip(), other.addr.ip()) {
	(V4(a), V4(b)) => a.octets().cmp(&b.octets()),
	(V6(a), V6(b)) => a.octets().cmp(&b.octets()),
	(V4(_), V6(_)) => Less,
	(V6(_), V4(_)) => Greater,
	};
	let port_tie_breaker = self.addr.port().cmp(&other.addr.port());
	more_reliable_state
	.then(more_likely_valid)
	.then(older_attempt)
	.then(older_failure)
	.then(older_response)
	.then(newer_untrusted_last_seen)
	.then(larger_services)
	.then(ip_tie_breaker)
	.then(port_tie_breaker)

And PeerAddrState:

zebra/zebra-network/src/meta_addr.rs

Lines 132 to 145 in 193fd42

	fn cmp(&self, other: &Self) -> Ordering {
	use Ordering::*;
	match (self, other) {
	_ if self == other => Equal,
	// We reconnect to `Responded` peers that have stopped sending messages,
	// then `NeverAttempted` peers, then `Failed` peers
	(Responded, _) => Less,
	(_, Responded) => Greater,
	(NeverAttemptedGossiped, _) => Less,
	(_, NeverAttemptedGossiped) => Greater,
	(NeverAttemptedAlternate, _) => Less,
	(_, NeverAttemptedAlternate) => Greater,
	(Failed, _) => Less,
	(_, Failed) => Greater,

[teor2345]

Candidates for replacement with a random order

     .then(newer_untrusted_last_seen) 
     .then(larger_services) 
     .then(ip_tie_breaker) 
     .then(port_tie_breaker) 

[teor2345]

Replace the tie-breakers with a random_reconnection_order field that is randomised once when the peer is created. This field is only effectively used for the first connection attempt, after that the order is based on the response/failure times.

We should document that if a peer gets a higher random value, then it might never be attempted, because new peers are added with different values.

random_reconnection_order should be a u16 so that almost all peers have a unique value. We won't need any tie-breakers, untrusted fields, and we don't depend on HashMap order.

[teor2345]

This seems like a useful fix, it might also be required, I'm not sure.

[arya2]

Potential Fix: If we're not getting any new blocks in the progress task, send a ChangePeerConnections request to the network to disconnect some peers

[teor2345]

What are the benefits and drawbacks of this fix?

[teor2345]

Benefit: we will probably need something like this anyway for:

• Start disconnecting from outdated peers a few days before network upgrade activation #4545

(But with a setting, or a change to the peer set / handshaker to drop new peers which are on outdated versions.)

[teor2345]

Do we do this when we're not getting anything, or also when we're failing validation?
Do we want a separate ticket for the mempool using this mechanism?

[teor2345]

Do we do this when we're not getting anything, or also when we're failing validation?

Doing this when we haven't had any new blocks for more than validation timeout + restart timeout + time for the first block to verify will address both cases.

Do we want a separate ticket for the mempool using this mechanism?

Yes, it isn't part of this fix, and neither is #4545.

[teor2345]

This seems like it is required.

[arya2]

Potential Fix: Increase GET_ADDR_FANOUT to 2 or 3, to increase the diversity of the address book

[arya2]

I think this helps by making it more likely we'll dial an address from a random peer rather than from the address book which favours addresses from some peers.

I don't see it fixing the core issue.

[teor2345]

I think it is required to fix this issue, so that a single peer is less likely to be able to supply all the best and latest addresses for connection attempts:
#7987 (reply in thread)

[arya2]

Potential Fix: Reduce the TimerCrawl interval to increase the diversity of the address book

[arya2]

This will likely make it easier to fill every connection slot quickly if it continues to dial on every iteration where crawl has sent new addresses to the address book.

We should make sure to skip dialing if there was a dial within 2-3 prior crawl iterations if we do this instead or increasing GET_ADDR_FANOUT.

If we do that, it should have about the same impact as the increasing the GET_ADDR_FANOUT.

[teor2345]

This will likely make it easier to fill every connection slot quickly if it continues to dial on every iteration where crawl has sent new addresses to the address book.

This is a drawback, but a benefit is that a single peer (or small group of peers) are less likely to supply all the best and latest addresses for connection attempts over the span of a minute or two, particularly when Zebra is idle.

That's related to this issue:
#7987 (reply in thread)

[teor2345]

We should make sure to skip dialing if there was a dial within 2-3 prior crawl iterations if we do this instead or increasing GET_ADDR_FANOUT.

If we do that, it should have about the same impact as the increasing the GET_ADDR_FANOUT.

This is an alternative fix.

What are the benefits and drawbacks?

Here is one drawback: if we skip dials, a single peer (or small group of peers) is much more likely to supply all the best and latest addresses for successful connection over the span of a minute or two, particularly when Zebra is idle. This seems like a major impact.

[teor2345]

I think that it is better to increase the fanout, because then each address book update has a more diverse set of peers.

I don't think this fix is guaranteed to be useful, and it seems to have some drawbacks.