OLD aises_7_4

<style type="text/css">
    table.tableLayout{
        margin: auto;
        border: 1px solid;
        border-collapse: collapse;
        border-spacing: 1px;
        caption-side: bottom;
    }



    table.tableLayout > caption.title{
        white-space: unset;
        max-width: unset;
    }

    table.tableLayout tr{
        border: 1px solid;
        border-collapse: collapse;
        padding: 5px;
    }

    table.tableLayout th{
        border: 1px solid;
        border-collapse: collapse;
        padding: 3px;
    }

    table.tableLayout td{
        border: 1px solid;
        padding: 5px;
    }
</style>

<style>
    .visionbox{
    border-radius: 15px;
    border: 2px solid #3585d4;
    background-color: #ebf3fb;
    text-align: left;
    padding: 10px;
    }
</style>

<style>
    .visionboxlegend{
    border-bottom-style: solid;
    border-bottom-color: #3585d4;
    border-bottom-width: 0px;
    margin-left: -12px;
    margin-right: -12px; margin-top: -13px;
    padding: 0.01em 1em; color: #ffffff;
    background-color: #3585d4;
    border-radius: 15px 15px 0px 0px}
</style>

<h1 id="conflict-and-cooperation">8.4 Conflict and Cooperation</h1>
<h2 id="overview">8.4.1 Overview</h2>
<p>In this chapter, we have been exploring the risks that arise from
interactions between multiple agents. So far, we have used game theory
and evolutionary theory to understand how collective behavior can
produce undesirable outcomes. In simple terms, securing morally good
outcomes without cooperation can be extremely difficult, even for
intelligent rational agents. Consequently, the potential for conflict
and the importance of cooperation has emerged as a strong theme in this
chapter. In this third section of this chapter, we examine conflict and
cooperation in more detail.</p>
<h3 id="conflict-overview">Conflict Overview</h3>
<p>We begin this section by exploring the drivers of conflict. Here, we
use the term “conflict” loosely, to describe the decision to defect
rather than cooperate in a competitive situation. This may lead to
violence in some cases, but not necessarily all. Our goal is to uncover
how, despite being costly, conflict can sometimes be a rational choice
nevertheless.<p>
Microorganisms, humans, states, and nations all cooperate and conflict
in different situations. In nature, we can observe cooperation in the
form of social insect behavior, pack hunting, symbiotic relationships,
and much more. In humans, we encounter cooperation in several areas
including coordinated disaster responses, international peace
negotiations, and community service, among many others. By contrast,
conflict in both human and non-human organisms can occur as a
consequence of resource competition, territorial disputes, mating
access, the maintenance of social dominance hierarchies, including
several other factors. Importantly, the mechanisms and factors that
motivate cooperation and conflict are prevalent in various environments.
Thus, we have good reason to suppose that AI agents will be similarly
influenced by these various mechanisms and factors as they decide to
cooperate or conflict with humans and other AIs.<p>
We will begin our discussion of conflict with concepts in bargaining
theory. We then examine some specific features of competitive situations
that make it harder to reach negotiated agreements or avoid
confrontation. We begin with five factors from bargaining theory that
can influence the potential for conflict. These can be divided into the
following two groups: <strong>Commitment problems.</strong> According to
bargaining theory, one reason bargains may fail is that some of the
agents making an agreement may have the ability and incentive to break
it. We explore three examples of commitment problems.</p>
<ul>
<li><p><em>Power shifts</em>: when there are imbalances between agents’
capabilities such that one agent becomes stronger than the other,
conflict is more likely to emerge between them.</p></li>
<li><p><em>First-strike advantages</em>: when one agent possesses the
element of surprise, the ability to choose where conflict takes place,
or the ability to quickly defeat their opponent, the probability of
conflict increases.</p></li>
<li><p><em>Issue indivisibility</em>: agents cannot always divide a good
however they please – some goods are “all or nothing” and this increases
the probability of conflict between agents.</p></li>
</ul>
<p><strong>Information problems.</strong> According to bargaining
theory, the other principal cause of a bargaining failure is that some
of the agents may lack good information. Uncertainty regarding a rival’s
capabilities and intentions can increase the probability of conflict. We
explore two information problems.</p>
<ul>
<li><p><em>Misinformation</em>: in the real world, agents frequently
have incorrect information, which can cause them to miscalculate
suitable bargaining ranges.</p></li>
<li><p><em>Disinformation</em>: agents may sometimes have incentives to
misrepresent the truth intentionally. Even the expectation of
disinformation can make it more difficult to reach a negotiated
settlement.</p></li>
</ul>
<p><strong>Factors outside of bargaining theory.</strong> Bargaining
frameworks do not encompass all possible reasons why agents may decide
to conflict with one another. We end by exploring one example:</p>
<ul>
<li><p><em>Inequality</em>: under conditions of inequality, agents may
fight for access to a larger share of available resources or a desired
social standing.</p></li>
</ul>
<h3 id="cooperation-overview">Cooperation overview</h3>
<p>Next, we turn to cooperation. We observe many forms of cooperation in
biological systems: social insect colonies, pack hunting, symbiotic
relationships, and much more. Humans perform community services,
negotiate international peace agreements, and coordinate aid for
disaster responses. Our very societies are built around
cooperation.<p>
Cooperation between AI stakeholders may be vital for counteracting the
competitive and evolutionary pressures of AI races we have explored in
this chapter. For example, the “merge-and-assist” clause of OpenAI’s
charter outlines their commitment to cease competition with—and provide
assistance to—any “value-aligned, safety-conscious” AI developer who
appears close to producing AGI, in order to reduce the risk of eroding
safety precautions. Cooperation between AI agents is also necessary for
reducing some of the multi-agent risks we have looked at: we want AIs to
cooperate, rather than defect, in Prisoner’s Dilemma scenarios.<p>
However, ensuring that AIs behave cooperatively may not be a total
solution to the collective action problems we have examined in this
chapter. By more closely examining how cooperative relationships can
come about, it is possible to see how they may backfire with disastrous
consequences for AI safety. Instead, we need a more nuanced view of the
potential benefits and risks of promoting cooperation between AI
systems. To do this, we study seven different mechanisms by which
cooperation may arise in multi-agent systems, considering the
ramifications of each for cooperation between and within human agencies
and AI agents:</p>
<ul>
<li><p><em>Direct reciprocity</em>: when individuals are likely to
encounter others in the future, they are more likely to cooperate with
them.</p></li>
<li><p><em>Indirect reciprocity</em>: when it benefits an individual’s
reputation to cooperate with others, they are more likely to do
so.</p></li>
<li><p><em>Group selection</em>: when there is competition between
groups, cooperative groups may outcompete non-cooperative
groups.</p></li>
<li><p><em>Kin selection</em>: when an individual is closely related to
others, they are more likely to cooperate with them.</p></li>
<li><p><em>Individual stakes to common stakes</em>: when individual
interests become aligned with the collective good of the group,
individuals are more likely to behave in ways that benefit the
whole.</p></li>
<li><p><em>Simon’s selection mechanism</em>: when available information
is limited, individuals may be impelled to rely on social channels that
require cooperation.</p></li>
<li><p><em>Institutional mechanisms</em>: when there are externally
imposed incentives (such as laws) that subsidize cooperation and punish
defection, individuals and groups are more likely to cooperate.</p></li>
</ul>
<h2 id="conflict">8.4.2 Conflict</h2>
<h3 id="introduction-to-conflict">Introduction to Conflict</h3>
<p>In this section, we use the term “conflict” to describe the decision
to defect rather than cooperate in competitive situations. This often,
though not always, involves some form of violence, and destroys some
amount of value. Conflict is common in nature. Organisms engage in
conflict to maintain social dominance hierarchies, to hunt, and to
defend territory. People also engage in conflict. Throughout human
history, wars are common, often occurring as a consequence of
power-seeking behavior, which inspired conflict over attempts at
aggressive territorial expansion or resource acquisition.<p>
We begin this section by discussing bargaining theory, which lays the
groundwork for understanding why it may be rational for agents to engage
in conflict. Next, we turn to the specific factors that may motivate
agents to engage in conflict with one another, even when compromise
might be the better option. We explore why AIs may be similarly affected
by these factors, such that they may view conflict as an instrumentally
rational choice in certain contexts.</p>
<p><strong>Conflict can be rational.</strong> Though humans know
conflict can be enormously costly, we often still pursue or instigate
it, even when compromise might be the better option.<p>
Consider the following example: a customer trips in a store and sues the
owner for negligence. There is a 60% probability the lawsuit is
successful. If they win, the owner has to pay them $40,000, and going to
court will cost each of them $10,000 in legal fees. There are three
options: (1) they or the owner concede, (2) they both let the matter go
to court, (3) they both reach an out-of-court settlement.<p>
</p>
<ol>
<li><p>If the owner concedes, the owner loses $40,000, and if the
customer concedes, they gain nothing.</p></li>
<li><p>If both go to court, the owner’s expected payoff is the product
of the payment to the customer and the probability that the lawsuit is
successful minus legal fees. In this case, the owner’s expected payoff
would be <span class="math inline">(−40,000×0.6) − 10, 000</span> while
the customer’s expected payoff would be <span
class="math inline">(40,000×0.6) − 10, 000</span>. As a result, the
owner loses $34,000 dollars and the customer gains $14,000
dollars.</p></li>
<li><p>An out-of-court settlement x where <span
class="math inline">14, 000 &lt; <em>x</em> &lt; 34, 000</span> would
enable the customer to get a higher payoff and the owner to pay lower
costs. Therefore, a mutual settlement is the best option for both if
<span class="math inline"><em>x</em></span> is in this range.</p></li>
</ol>
<p>Hence, if the proposed out-of-court settlement would be greater than
$34,000, it would make sense for the owner to opt for conflict rather
than bargaining. Similarly, if the proposed settlement were less than
$14,000, it would be rational for the customer to opt for conflict.</p>
<h3 id="bargaining-theory">Bargaining Theory</h3>
<p>Here, we begin with a general overview of bargaining theory, to
illustrate how pressures to outcompete rivals or preserve power and
resources may make conflict an instrumentally rational choice. Next, we
turn to the unitary actor assumption, highlighting that when agents view
their rivals as unitary actors, they assume that they will act more
coherently, taking whatever steps necessary to maximize their welfare.
Following this, we discuss the notion of commitment problems, which
occur when agents cannot reliably commit to an agreement or have
incentives to break it. Commitment problems increase the probability of
conflict, and are motivated by specific factors, such as power shifts,
first strike advantages, and issue indivisibility. We then explore how
information problems and inequality can also increase the probability of
conflict.</p>
<p><strong>Bargaining theory.</strong> When agents compete for something
they both value, they may either negotiate to reach an agreement
peacefully, or resort to more forceful alternatives such as violence. We
call the latter outcome “conflict,” and can view this as the decision to
defect rather than cooperate. Unlike peaceful bargaining, conflict is
fundamentally costly for winners and losers alike. However, it may
sometimes be the rational choice. <em>Bargaining theory</em> describes
why rational agents may be unable to reach a peaceful agreement, and
instead end up engaging in violent conflict. Due to pressures to
outcompete rivals or preserve their power and resources, agents
sometimes prefer conflict, especially when they cannot reliably predict
the outcomes of conflict scenarios. When rational agents assume that
potential rivals have the same mindset, the probability of conflict
increases.</p>
<p><strong>The unitary actor assumption.</strong> We tend to assume that
a group is a single entity, and that its leader is only interested in
maximizing the overall welfare of the entity. We call this the
<em>unitary actor assumption</em>, which is another name for the “unity
of purpose” assumption discussed previously in this chapter. A nation in
disarray without coherent leadership is not necessarily a unitary actor.
When we view groups and individuals as unitary actors, we can assume
they will act more coherently, so they can be more easily modeled as
taking steps necessary to maximize their welfare. When parties make this
assumption, they may be less likely to cooperate with others since what
is good for one party’s welfare may not necessarily be good for
another’s.</p>
<p><strong>The bargaining range.</strong> Whether or not agents are
likely to reach a peaceful agreement through negotiation will be
influenced by whether their bargaining ranges overlap. The bargaining
range represents the set of possible outcomes that both agents involved
in a competition find acceptable through negotiation. Recall the lawsuit
example: a bargaining settlement “<span
class="math inline"><em>x</em></span>” is only acceptable if it falls
between $14,000 and $34,000. Any settlement “<span
class="math inline"><em>x</em></span>” below $14,000 will be rejected by
the customer while any settlement “<span
class="math inline"><em>x</em></span>” above $34,000 will be rejected by
the store owner. Thus, the bargaining range is often depicted as a
spectrum with the lowest acceptable outcome for one party at one end and
the highest acceptable outcome for the other party at the opposite end.
Within this range, there is room for negotiation and potential
agreements.</p>
<figure id="fig:overview-barg">
<img src="https://raw.githubusercontent.com/WilliamHodgkins/AISES/main/images/bargain_range.png" class="tb-img-full"  style="width: 70%"/>
<p class="tb-caption">Figure 8.11: The bargaining range is defined by the potential gains from cooperation.</p>
<!--<figcaption>Overview of Bargaining ranges</figcaption>-->
</figure>
<p><strong>Conflict in AIs.</strong> Let us assume that AI agents will
act rationally in the pursuit of their goals (so, at the least, we model
them as unitary actors or as having unity of purpose). In the process of
pursuing and fulfilling their goals, AI agents may encounter potential
conflict scenarios, just as humans do. In certain scenarios, AIs may be
motivated to pursue violent conflict over a peaceful resolution, for the
reasons we now explore.</p>
<h3 id="sec:commitment problems">Commitment problems</h3>
<p>Many conflicts occur over resources, which are key to an agent’s
power. Consider a bargaining failure in which two agents bargain over
resources in an effort to avoid war. If agents were to acquire these
resources, they could invest them into military power. As a result,
neither can credibly commit to use them only for peaceful purposes. This
is one instance of a <em>commitment problem</em> <span class="citation"
data-cites="fearon1995rationalist">[1]</span>, which is when agents
cannot reliably commit to an agreement, or when they may even have
incentives to break an agreement. Commitment problems are usually
motivated by specific factors, such as power shifts, first-strike
advantages, and issue indivisibility, which may make conflict a rational
choice. It is important to note that our discussion of these commitment
problems assumes anarchy: we take for granted that contracts are not
enforceable in the absence of a higher governing authority.</p>
<p><strong>Power shifts overview.</strong> When there are imbalances
between parties’ capabilities such that one party becomes stronger than
the other, <em>power shifts</em> can occur. Such imbalances can arise as
a consequence of several factors including technological and economic
advancements, increases in military capabilities, as well as changes in
governance, political ideology, and demographics. In the context of AIs,
power could shift if AIs become more intelligent or have a change in
resources. Individual parties may initially be able to avoid violent
conflict by arriving at a peaceful and mutually beneficial settlement
with their rivals. However, if they or their rival’s power increases
after this settlement has been made, the stronger party may end up
benefiting from it more than the weaker party. Thus, we encounter the
following commitment problem: the rising power cannot commit not to
exploit their advantage in the future, incentivizing the declining power
to opt for conflict in the present.</p>
<p><strong>Example: The US vs China.</strong> China has been investing
heavily in its military. This has included the acquisition or expansion
of its capabilities in technologies such as nuclear and supersonic
missiles, as well as drones. The future is uncertain, but if this trend
continues, it increase the risk of conflict. If China were to gain a
military advantage over the US, this would shift the balance of power.
This possibility undermines the stability of bargains struck today
between the US and China, because China’s expected outcome from conflict
may increase in the future if they become more powerful. The US may
expect that agreements made with China about cooperating on AI
regulation could lose enforceability later if there is a significant
power shift.<p>
This situation can be modeled using the concept of “Thucydides’ Trap.”
The ancient Greek historian Thucydides suggested that the contemporary
conflict between Sparta and Athens might have been the result of Athens’
increasing military strength, and Sparta’s fear of the looming power
shift. Though this analysis of the Peloponnesian War is now
much-contested, this concept can nevertheless serve to understand how a
rising power threatening the position of an existing superpower in the
global order can increase the potential for conflict rather than
peaceful bargaining.</p>
<p><strong>Effect on the bargaining range.</strong> Consider two agents,
A and B. A is always weaker than B, but relative to the time period, A
is weaker in the future than it is in the present. A will always have a
lower bargaining range, so B will be unlikely to accept any settlements,
especially as B’s power increases. It makes sense for A to prefer
conflict, because if it waits, B’s bargaining range will shift further
and further away, eliminating any overlap between the two. Therefore, A
prefers to gamble on conflict even if the probability that A wins is
lower than B; the costs of war do not outweigh the benefits of a
peaceful but unreasonable settlement. Consider the 1956 Suez Crisis.
Egypt was seen as a rising power in the Middle East, having secured
control over the Suez canal. This threatened the interests of the
British and French governments in the region, who responded by
instigating war. To safeguard their diminishing influence, the British
and French launched a swift and initially successful military
intervention.</p>
<p><strong>Power shifts and AI.</strong> AIs could shift power as they
gain greater intelligence and more access to resources. Recall the
chapter on , where we saw that an agent’s power is highly related to the
efficiency with which they can exploit resources for their benefit,
which often depends on their level of intelligence. The power of future
AI systems is largely unpredictable; we do not know how intelligent or
useful they will be. This could give rise to substantial uncertainty
regarding how powerful potential adversaries using AI might become. If
this is the case, there might be reason to engage in conflict to prevent
the possibility of adversaries further increasing their power.</p>
<p><strong>First strike advantage overview.</strong> If an agent has a
<em>first-strike advantage</em>, they will do better to launch an attack
than respond to one. This gives rise to the following commitment
problem: an offensive advantage may be short-lived, so it is best to act
on it before the enemy does instead. Some ways in which an agent may
have a first strike advantage include:</p>
<ol>
<li><p>As explored above, anticipating a future power shift may motivate
an attack on the rising power to prevent it from gaining the upper
hand.</p></li>
<li><p>The costs of conflict might be lower for the attacker than they
are for the defender, so the attacker is better off securing on
offensive advantage while the defender is still in a position of
relative weakness.</p></li>
<li><p>The odds of victory may be higher for whichever agent attacks
first. The attacker might possess the element of surprise, the ability
to choose where conflict takes place, or the potential to quickly defeat
their opponent. For instance, a pre-emptive nuclear strike could be used
to target an enemy’s nuclear arsenal, thus diminishing their ability to
retaliate.</p></li>
</ol>
<p><strong>Examples: IPOs, patent Infringement, and Pearl
Harbor.</strong> When a company goes public, it can release an IPO,
allowing members of the general public to purchase company shares.
However, company insiders, such as executives and early investors, often
have access to valuable information not available to the general public;
this gives insiders a first-strike advantage. Insiders may buy or sell
shares based on this privileged information, leading to potential
regulatory conflicts or disputes with other investors who do not have
access to the same information. Alternatively, when a company develops a
new technology and files a patent application, they gain a first-strike
advantage by ensuring that their product will not be copied or
reproduced by other companies. If a rival company does create a similar
technology and later files a patent application, conflict can emerge
when the original company claims patent infringement.<p>
On the international level, we note similar dynamics, such as in the
case of Pearl Harbor. Though Japan and the US were not at war in 1941,
their peacetime was destabilized by a commitment problem: if one nation
were to attack the other, they would have an advantage in the ensuing
conflict. The US’ Pacific fleet posed a threat to Japan’s military plans
in Southeast Asia. Japan had the ability to launch a surprise long-range
strategic attack. Thus, neither the US nor Japan could credibly commit
not to attack the other. In the end, Japan struck first, bombing the US
battleships at the naval base at Pearl Harbor. The attack was successful
in securing a first-strike advantage for Japan, but it also ensured the
US’s entry into WWII.<p>
</p>
<br>
<table class="tableLayout">
<caption style="overflow: hidden; white-space: nowrap;">Table 8.6: A pay-off matrix for competitors choosing whether to defend or preemptively attack.</caption>
<thead>
<tr class="header">
<th style="text-align: left;"></th>
<th style="text-align: center;">defend</th>
<th style="text-align: center;">preempt</th>
</tr>
</thead>
<tbody>
<tr class="odd">
<td style="text-align: left;">defend</td>
<td style="text-align: center;">2,2</td>
<td style="text-align: center;">0,3</td>
</tr>
<tr class="even">
<td style="text-align: left;">preempt</td>
<td style="text-align: center;">3,0</td>
<td style="text-align: center;">1,1</td>
</tr>
</tbody>
</table>
<br>
<p><strong>Effect on the bargaining range.</strong> When the advantages
of striking first outweigh the costs of conflict, it can shrink or
destroy the bargaining range entirely. For any two parties to reach a
mutual settlement through bargaining, each must be willing to freely
communicate information with the other. However, in doing so, each party
might have to reveal offensive advantages, which would increase their
vulnerability to attack. The incentive to preserve and therefore conceal
an offensive advantage from opponents’ pressures agents to defect from
bargaining.</p>
<p><strong>First Strike Advantage and AIs.</strong> One scenario in
which an AI may be motivated to secure a first strike advantage is
cyberwarfare. An AI might hack servers for a variety of reasons to
secure an offensive advantage. AIs may want to disrupt and degrade an
adversary’s capabilities by attacking and destroying critical
infrastructure. Alternatively, an AI might gather sensitive information
regarding a rival’s capabilities, vulnerabilities, and strategic plans
to leverage potential offensive advantages.<p>
AIs may provide first strike advantages in other ways, too. Sudden and
dramatic progress in AI capabilities could motivate one party to take
offensive action. For example, if a nation very rapidly develops a much
more powerful AI system than its military enemies, this could present a
powerful first strike advantage: by attacking immediately, they may hope
to prevent their rivals from catching up with them, which would lose
them their advantage. Similar incentives were likely at work when the US
was considering a nuclear strike on the USSR to prevent them from
developing nuclear weapons themselves.<p>
Reducing the possibility of first-strike advantages is challenging,
especially with AI. However, we can lower the probability that they
arise by ensuring that there is a balance between the offensive and
defensive capabilities of potential rivals. In other words, defense
dominance can facilitate peace because attempted attacks between rivals
are likely to be unsuccessful or result in mutually assured destruction.
Therefore, we might reduce the probability that AIs are motivated to
pursue a first-strike advantage by ensuring that humans maintain defense
dominance, for instance, by requiring that advanced AIs have a built-in
incorruptible fail-safe mechanism, such as a manual “off-switch.”</p>
<p><strong>Issue indivisibility overview.</strong> Settlements that fall
within bargaining range will always be preferable to conflict, but this
assumes that whatever issues agents bargain over are divisible. For
instance, two agents can divide a territory in an infinite amount of
ways insofar as the settlement they arrive at falls within the
bargaining range, satisfying both their interests and outweighing the
individual benefits of engaging in conflict. Unfortunately, however,
some goods are indivisible, which inspires the following commitment
problem: parties cannot always divide a good however they please—-some
goods are “all or nothing.” When parties encounter <em>issue
indivisibility</em> <span class="citation"
data-cites="fearon1995rationalist">[1]</span>, the probability of
conflict increases. Indivisible issues include monarchies, small
territories like islands or holy sites, national religion or pride, and
sovereign entities such as states or human beings, among several
others.</p>
<p><strong>Examples: shopping, organ donation, and
co-parenting.</strong> Imagine two friends that go out for a day of
shopping. For lunch, they stop at their favorite deli and find that it
only has one sandwich left: they decide to share this sandwich between
themselves. After lunch, they go to a clothing store, and both come
across a jacket they love, but of which there is only one left. They
begin arguing over who should get the jacket. Simply put, sandwiches can
be shared and jackets can’t. Issue indivisibility can give rise to
conflict, often leaving all parties involved worse off.<p>
The same can be true in more extreme cases, such as organ donation.
Typically, the available organ supply does not meet the transplant needs
of all patients. Decisions as to who gets priority for transplantation
may favor certain groups or individuals and allocation systems may be
unfair, giving rise to conflict between doctors, patients, and
healthcare administrations. Finally, we can also observe issue
indivisibility in co-parenting contexts. Divorced parents sometimes
fight for full custody rights over their children. This can result in
lengthy and costly legal battles that are detrimental to the family as a
whole.</p>
<p><strong>Effect on the bargaining range.</strong> When agents
encounter issue indivisibilities, they cannot arrive at a reasonable
settlement through bargaining. Sometimes, however, issue indivisibility
can be resolved through side payments. One case in which side payments
were effective was during the Spanish-American War of 1898, fought
between Spain and the United States over the territory of the
Philippines. The conflict was resolved when the United States offered to
buy the Philippines from Spain for 20 million dollars. Conversely, the
Munich Agreement at the dawn of WWII represents a major case where side
payments were ineffective. In an attempt to appease Hitler and avoid
war, the British and French governments reached an agreement with
Germany, allowing them to annex certain parts of Czechoslovakia. This
agreement involved side payments in the form of territorial concessions
to Germany, but it ultimately failed, as Hitler’s aggressive
expansionist ambitions were not satisfied, leading to the outbreak of
World War II. Side payments can only resolve issue indivisibility when
the value of the side payments outweighs the value of the good.<p>
</p>
<figure id="fig:first-strike">
<img src="https://raw.githubusercontent.com/WilliamHodgkins/AISES/main/images/bargain_range_2.png" class="tb-img-full" style="width: 70%"/>
<p class="tb-caption">Figure 8.12: A first strike can create an advantage by changing the bargaining range.</p>
<!--<figcaption>First strike advantage</figcaption>-->
</figure>
<p><strong>Issue indivisibility and AIs.</strong> Imagine there is a
very powerful AI training system, and that whoever has access to this
system will eventually be able to dominate the world. In order to reduce
the chance of being dominated, individual parties may compete with one
another to secure access to this system. If parties were to split the
AI’s compute up between themselves, it would no longer be as powerful as
it was previously, perhaps not more powerful than their existing
training systems. Since such an AI cannot be divided up among many
stakeholders easily, it may be rational for parties to conflict over
access to it, since doing so ensures global domination.</p>
<h3 id="information-problems">Information problems</h3>
<p>Misinformation and disinformation both involve the spread of false
information, but they differ in terms of intention. Misinformation is
the dissemination of false information, without the intention to
deceive, due to a lack of knowledge or understanding. Disinformation, on
the other hand, is the deliberate spreading of false or misleading
information with the intent to deceive or manipulate others. Both of
these types of information problem can cause bargains to fail,
generating conflict.</p>
<table class="tableLayout">
<thead>
<tr class="header">
<th style="text-align: center;"></th>
<th style="text-align: center;">Distinguish</th>
<th style="text-align: center;">Defect</th>
</tr>
</thead>
<tbody>
<tr class="odd">
<td style="text-align: center;">Distinguish</td>
<td style="text-align: center;"><span
class="math inline"><em>b</em> − <em>c</em></span></td>
<td style="text-align: center;"><span
class="math inline"> − <em>c</em>(1−<em>a</em>)</span></td>
</tr>
<tr class="even">
<td style="text-align: center;">Defect</td>
<td style="text-align: center;"><span
class="math inline"><em>b</em>(1−<em>a</em>)</span></td>
<td style="text-align: center;">0</td>
</tr>
</tbody>
</table>
<br>
<p>The term <span class="math inline"><em>a</em></span> is the
probability of a player knowing the strategy of its partner. Relevant
for AI since it might reduce uncertainty (though still chaos and
incentives to conceal or misrepresent information or compete).</p>
<p><strong>Misinformation overview.</strong> Uncertainty regarding a
rival’s power or intentions can increase the probability of
conflict<span class="citation"
data-cites="fearon1995rationalist">[1]</span>. Bargaining often requires
placing trust in another not to break an agreement. This harder to
achieve when one agent believes something false about the other’s
preferences, resources, or commitments. This lack of shared, accurate
information can lead to mistrust and a breakdown in negotiations.</p>
<p><strong>Example: Russian invasion of Ukraine.</strong> Incomplete
information may lead overly optimistic parties to make too large
demands, whereas rivals that are tougher than expected reject those
demands and instigate conflict. Examples of misinformation problems
generating conflict may include Russia’s 2022 invasion of Ukraine.
Russian President Putin reportedly miscalculated Ukraine’s willingness
to resist invasion and fight back. With more accurate information
regarding Ukraine’s abilities and determination, Putin may have been
less likely to instigate conflict.</p>
<p><strong>Effect on the bargaining range.</strong> Misinformation can
prevent agents from finding a mutually-agreeable bargaining range, as
shown in Figure 9.14. For example, if each agent believes themself to be
the more powerful party, each may therefore want more than half the
value they are competing for. Thus, each may reject any bargain offer
the other makes, since they expect a better if they opt for conflict
instead.</p>
<figure id="fig:information-problems">
<img src="https://raw.githubusercontent.com/WilliamHodgkins/AISES/main/images/bargaining_range_3.png" class="tb-img-full" style="width: 70%"/>
<p class="tb-caption">Figure 8.13: Misinformation can shift players’ perceptions of the bargaining range. If players’
estimates of Green winning are too high, then they will agree on deals more favorable to Green.</p>
<!--<figcaption>Information problems</figcaption>-->
</figure>
<p><strong>Misinformation and AI.</strong> AI technologies may produce
misinformation directly. Examples of this include large language models
hallucinating false facts. Less directly, AI development could also
exacerbate misinformation problems by increasing uncertainty. For
example, military AI applications may make wars more uncertain, and this
may increase the probability of conflict. AI weaponry innovation
presents an opportunity for states to gain power. However, AI
capabilities advances are often highly uncertain—it may be unclear how
powerful a model trained on an order of magnitude of compute would be,
or how far behind adversaries are in their effort to create powerful
models. As automated warfare technologies become more widespread and
sophisticated, nations may struggle to predict their probability of
victory in any given conflict accurately. This increased potential for
miscalculation may make warfare more likely.<p>
There are other ways reducing information problems can reduce AI risk.
If there are substantial existential risks from AIs but this is not
widely agreed on, improving understanding of these risks could help make
different actors (such as the US and China) get better estimates of the
payoff matrix. With better understanding of AI risk, they may recognize
that it is in their self-interest to cooperate (slow down AI development
and militarization) instead of defecting (engaging in an AI race).
Similarly, creating information channels such as summits can increase
understanding and coordination; even if countries do not agree on shared
commitments, the discussions on the sidelines can reduce
misunderstandings and the risk of conflict.</p>
<p><strong>Disinformation overview.</strong> Unlike misinformation,
where false information is propagated without deceptive intention,
disinformation is the <em>deliberate</em> spreading of false
information: the intent is to mislead, deceive or manipulate. Here, we
explore why competitive situations may motivate agents to try to mislead
others or misrepresent the truth, and how this can increase the
probability of conflict.<p>
<strong>Examples: employment and the real estate industry.</strong>
Throughout labor markets, employers and job seekers often encounter
disinformation problems. Employers may intentionally withhold
information about the salary range or offer lower wages than what the
market standard suggests in order to secure lower employment costs. On
the other hand, job seekers might exaggerate their qualifications or
professional experience to increase their chances of getting hired. Such
discrepancies can lead to legal conflicts and high turnover rates.
Alternatively, in the real estate market, disinformation problems can
emerge between sellers and buyers. Sellers sometimes withhold critical
information about the property’s condition to increase the probability
that the property gets purchased. Buyers, on the other hand, may be
incentivized to misrepresent their budget or willingness to pay to
pressure sellers to lower their price. Oftentimes, this can result in
legal battles or disputes as well as the breakdown of property
transactions.</p>
<p><strong>Effect on the bargaining range.</strong> Consider two agents:
A, which is stronger, and B, which is weaker. B demands “X” amount for a
bargaining settlement, but A, as the stronger agent, will not offer this
to avoid being exploited by B. In other words, A thinks B is just trying
to get more for themself to “bait” A or “bluff” by implying that the
bargaining range is lower. But B might not be bluffing and A might not
be as strong as they think they are. Consider the Sino-Indian war in
this respect. At the time, India had perceived military superiority
relative to China. But in 1962, the Chinese launched an attack on the
Himalayan border with India, which demonstrated China’s superior
military capabilities, and triggered the Sino-Indian war. Thus, stronger
parties may prefer conflict if they believe rivals are bluffing.
Whereas, weaker parties may prefer conflict if they believe rivals are
not as powerful as they believe themselves to be.<p>
<strong>Disinformation and AI.</strong> AIs themselves may have
incentives to misrepresent the facts. For example, the agent “Cicero,”
developed by Meta, is capable of very high performance in the board
wargame “Diplomacy.” Its success requires it to misrepresent certain
information to the other players in strategic fashion. We have seen many
other examples of AIs producing disinformation for a variety of reasons,
such as large language models successfully persuading users that they
are conversing with a human. The ability for AIs to misrepresent
information successfully is only likely to increase in future. This
could exacerbate disinformation problems, and thus contribute to greater
risk of conflict by eroding the potential for peaceful negotiation.</p>
<h3 id="factors-outside-of-bargaining-theory">Factors outside of
bargaining theory</h3>
<p><strong>Inequality is another factor that is highly predictive of
conflict.</strong> Crime is a form of conflict. Income and educational
inequality are robust predictors of violent crime <span class="citation"
data-cites="kelly2000inequality">[2]</span>, with an elasticity in
excess of 0.5 (elasticity measures how sensitive one variable is to
changes in another variable) even when controlling for variables such as
race and family composition. Similarly, individuals and families with a
yearly income below $15,000 are three times more likely to be the
victims of violent crime than are individuals and families with a yearly
income over $75,000 <span class="citation"
data-cites="victimrates2011">[3]</span>. Moreover, economists from the
World Bank have also highlighted that the effects of inequality on both
violent and property crime are robust between countries, finding that
when economic growth improves in a country, violent crime rates decrease
substantially <span class="citation"
data-cites="fajnzylber2002inequality">[4]</span>. This is consistent
with evidence at the national level; in the US, for example, the Bureau
of Justice reports that households below the federal poverty level have
a rate of violent victimization that is more than twice as high as the
rate for households above the federal poverty level. Moreover, these
effects were largely consistent between both rural and urban areas where
poverty was prevalent, further emphasizing the robust relationship
between inequality and conflict.</p>
<p><strong>Inequality and relative deprivation.</strong> Relative
deprivation is the perception or experience of being deprived or
disadvantaged in comparison to others. It is a subjective measure of
social comparison, not an objective measure of deprivation based on
absolute standards. People may feel relatively deprived when they
perceive that others possess more resources, opportunities, or social
status than they do. This can lead to feelings of resentment. For
example, “Strain theory,” proposed by sociologist Robert K. Merton,
suggests that individuals experience strain or pressure when they are
unable to achieve socially approved goals through legitimate means.
Relative deprivation is a form of strain, which may lead individuals to
resort to various coping mechanisms, one of which is criminal behavior.
For example, communities with a high prevalence of relative deprivation
can evolve a subculture of violence <span class="citation"
data-cites="horne2009effect">[5]</span>. Consider the emergence of
gangs, in which violence becomes a way to establish dominance, protect
territory, and retaliate against rival groups, providing an alternative
path for achieving a desired social standing.</p>
<p><strong>AIs and relative deprivation.</strong> Advanced future AIs
and widespread automation may propel humanity into an age of abundance,
where many forms of scarcity have been largely eliminated on the
national, and perhaps even global scale. Under these circumstances, some
might argue that conflict will no longer be an issue; people would have
all of their needs met, and the incentives to resort to aggression would
be greatly diminished. However, as previously discussed, relative
deprivation is a subjective measure of social comparison, and therefore,
it could persist even under conditions of abundance.<p>
Consider the notion of a “hedonic treadmill,” which notes that
regardless of what good or bad things happen to people, they
consistently return to their baseline level of happiness. For instance,
reuniting with a loved one or winning an important competition might
cultivate feelings of joy and excitement. However, as time passes, these
feelings dissipate, and individuals tend to return to the habitual
course of their lives. Even if individuals were to have access to
everything they could possibly need, the satisfaction they gain from
having their needs fulfilled is only temporary.<p>
Abundance becomes scarcity reliably. Dissatisfied individuals can be
favored by natural selection over highly content and comfortable
individuals. In many circumstances, natural selection could disfavor
individuals who stop caring about acquiring more resources and expanding
their influence; natural selection favors selfish behavior (for more
detail, see the section "Levels of Selection and Selfish Behaviour"
of <em>Evolutionary Pressures</em>). Even under conditions of abundance,
individuals may still compete for resources and influence because they
perceive the situation as a zero-sum game, where resources and power
must be divided among competitors. Individuals that acquire more power
and resources could incur a long-term fitness advantage over those that
are “satisfied” with what they already have. Consequently, even with
many resources, conflict over resources could persist in the evolving
population.<p>
Relatedly, in economics, the law of markets, also known as “Say’s Law,”
proposes that production of goods and services generates demand for
goods and services. In other words, supply creates its own demand.
However, if supply creates demand, the amount of resources required to
sustain supply to meet demand must also increase accordingly. Therefore,
steady increases in demand, even under resource-abundant conditions will
reliably result in resource scarcity.</p>
<p><strong>Conflict over social standing and relative power may
continue.</strong> There will always be scarcity of social status and
relative power, which people will continue to compete over. Social envy
is a fundamental part of life; it may persist because it tracks
differential fitness. Motivated by social envy, humans establish and
identify advantageous traits, such as the ability to network or climb
the social ladder. Scarcity of social status motivates individuals to
compete for social standing when doing so enables access to larger
shares of available resources. Although AIs may produce many forms of
abundance, there would still be dimensions on which to compete.
Moreover, AI development could itself exacerbate various forms of
inequality to extreme levels. We discuss this possibility in Chapter 9
Governance; Section 3 - Distribution.</p>
<h3 id="summary">Summary</h3>
<p>Throughout this section, we have discussed some of the major factors
that drive conflict. When any one of these factors is present, agents’
incentives to bargain for a peaceful settlement may shift such that
conflict becomes an instrumentally rational choice. These factors
include power shifts, first strike advantages, issue indivisibility,
information problems and incentives to misrepresent, as well as
inequality.<p>
In our discussion of these factors, we have laid the groundwork for
understanding the conditions under which decisions to instigate conflict
may be considered instrumentally rational. This knowledge base allows us
to better predict the risks and probability of AI-driven conflict
scenarios.<p>
First, we covered how power shifts can incentivize AI agents to pursue
conflict, to maintain strategic advantages or deter potential attacks
from stronger rivals, especially in the context of military AI
use.<p>
Second, we explored how the short-lived nature of offensive advantages
may incentivize AIs to pursue first-strike advantages, to degrade or
identify vulnerabilities in adversaries’ capabilities, as may be the
case in cyberwarfare.<p>
Third, we discussed issue indivisibility, imagining a future scenario in
which individual parties must compete for access to a world-dominating
AI. We reasoned that since dividing this AI between many stakeholders
would reduce its power, parties may find it instrumentally rational to
conflict for access to it.<p>
Fourth, we discussed how AIs may make wars more uncertain, increasing
the probability of conflict. We expect that AI weaponry innovation will
present an opportunity for superpowers to consolidate their dominance,
whereas weaker states may be able to quickly increase their power by
taking advantage of these technologies early on. This dynamic may create
a future in which power shifts are uncertain, which may lead states to
incorrectly expect that there is something to gain from going to
war.<p>
Finally, we explored the relationship between inequality and conflict.
We considered how even under conditions of abundance facilitated by
widespread automation and advanced AI implementation, relative
deprivation, and therefore conflict may persist. We also explored the
possibility that AIs may be motivated by social envy to compete with
other humans or AIs for desired social standing. This may result in a
global landscape in which the majority of humanity’s resources are
controlled by selfish, power-seeking AIs.<p>
Though AIs could evolve cooperative tendencies similarly to humans and
other animals, the possibility that they pursue interests and goals that
promote conflict, no matter how small, could pose catastrophic risks to
humans. It is thus important that we understand the drivers of conflict,
especially in the context of advanced future AIs.<p>
</p>
<h2 id="cooperation">8.4.3 Cooperation</h2>
<h3 id="direct-reciprocity">Direct Reciprocity</h3>
<p><strong>Direct reciprocity overview.</strong> One way agents may
cooperate is through <em>direct reciprocity</em>: when one agent
performs a favor for another because they expect the recipient to return
this favor in the future <span class="citation"
data-cites="trivers1971evolution">[6]</span>. We capture this core idea
in idioms like “quid pro quo,” or “you scratch my back, I’ll scratch
yours.” Direct reciprocity requires repeated interaction between the
agents: the more likely they are to meet again in the future, the
greater the incentive for them to cooperate in the present. We have
already encountered this in the iterated Prisoner’s Dilemma: how an
agent behaves in a present interaction can influence the behavior of
others in future interactions . Game theorists sometimes refer to this
phenomenon as the “shadow of the future.” When individuals know that
future cooperation is valuable, they have increased incentives to behave
in ways that benefit both themselves and others, fostering trust,
reciprocity, and cooperation over time. Cooperation can only evolve as a
consequence of direct reciprocity when the probability, <span
class="math inline"><em>w</em></span>, of subsequent encounters between
the same two individuals is greater than the cost-benefit ratio of the
helpful act. In other words, if agent A decided to help agent B at some
cost to themselves, they will only do so when the expected benefit of
agent B returning the favor outweighs the cost of Agent A’s initially
providing it. Thus, we have the rule <span
class="math inline"><em>w</em> &gt; <em>c</em>/<em>b</em></span>; see
Table 8.7 below.<p>
</p>
<br>
<div id="tab:reciprocity">
<table class="tableLayout">
<caption>Table 8.7: Payoff matrix for direct reciprocity games.</caption>
<thead>
<tr class="header">
<th style="text-align: center;"></th>
<th style="text-align: center;">Cooperate</th>
<th style="text-align: center;">Defect</th>
</tr>
</thead>
<tbody>
<tr class="odd">
<td style="text-align: center;">Cooperate</td>
<td style="text-align: center;"><span
class="math inline"><em>b</em> − <em>c</em>/(1−<em>w</em>)</span></td>
<td style="text-align: center;"><span
class="math inline"> − <em>c</em></span></td>
</tr>
<tr class="even">
<td style="text-align: center;">Defect</td>
<td style="text-align: center;"><span
class="math inline"><em>b</em></span></td>
<td style="text-align: center;"><span class="math inline">0</span></td>
</tr>
</tbody>
</table>
</div>
<br>
<p><strong>Natural examples of direct reciprocity.</strong> Trees and
fungi have evolved symbiotic relationships where they exchange sugars
and nutrients for mutual benefit. Dolphins use cooperative hunting
strategies where one dolphin herds schools of fish while the others form
barriers to encircle them. The dynamics of the role reversal are decided
by an expectation that other dolphins in the group will reciprocate this
behavior during subsequent hunts. Similarly, chimpanzees engage in
reciprocal grooming, where they exchange grooming services with one
another with the expectation that they will be returned during a later
session <span class="citation"
data-cites="schino2007grooming">[7]</span>.<p>
Direct reciprocity in human society. Among humans, one prominent example
of direct reciprocity is commerce. Commerce is a form of direct
reciprocity "which offers positive-sum benefits for both parties and
gives each a selfish stake in the well-being of the other" <span
class="citation" data-cites="pinker2012better">[8]</span>; commerce can
be a win-win scenario for all parties involved. For instance, if Alice
produces wine and Bob produces cheese, but neither Alice nor Bob has the
resources to produce what the other can, both may realize they are
better off trading. Different parties might both need the good the other
has when they can’t produce it themselves, so it is mutually beneficial
for them to trade, especially when they know they will encounter each
other again in the future. If Alice and Bob both rely on each other for
wine and cheese respectively, then they will naturally seek to prevent
harm to one another because it is in their rational best interest. To
this point, commerce can foster <em>complex interdependencies</em>
between economies, which enhances the benefits gained through mutual
exchange while decreasing the probability of conflict or war.</p>
<p><strong>Direct reciprocity and AIs.</strong> The future may contain
multiple AI agents, many of which might interact with one another to
achieve different functions in human society. Such AI agents may
automate parts of our economy and infrastructures, take over mundane and
time-consuming tasks, or provide humans and other AIs with daily
assistance. In a multi-AI agent system, where the probability that
individual AIs would meet again is high, AIs might evolve cooperative
behaviors through direct reciprocity. If one AI in this system has
access to important resources that other AIs need to meet their
objectives, it may decide to share these resources accordingly. However,
since providing this favor would be costly to the given AI, it will do
so only when the probability of meeting the recipient AIs (those that
received the favor) outweighs the cost-benefit ratio of the favor
itself.</p>
<p><strong>Direct reciprocity can backfire: AIs may disfavor cooperation
with humans.</strong> AIs may favor cooperation with other AIs over
humans. As AIs become substantially more capable and efficient than
humans, the benefit of interacting with humans may decrease. It may take
a human several hours to reciprocate a favor provided by an AI, whereas
it may take an AI only seconds to do so. It may therefore become
extremely difficult to formulate exchanges between AIs and humans that
benefit AIs more than exchanges with other AIs would. In other words,
from an AIs perspective, the cost-benefit ratio for cooperation with
humans is not worth it.</p>
<p><strong>Direct reciprocity may backfire: offers of AI cooperation may
undermine human alliances.</strong> The potential for direct reciprocity
can undermine the stability of other, less straightforward cooperative
arrangements within a larger group, thereby posing a collective action
problem. One example of this involves “bandwagoning.” In the section of
the chapter, we discussed the idea of “balancing” in international
relations: state action to counteract the influence of a threatening
power, such as by forming alliances with other states against their
common adversary <span class="citation"
data-cites="mearsheimer2007structural">[9]</span>. However, some
scholars argue that states do not always respond to threatening powers
by trying to thwart them. Rather than trying to prevent them from
becoming too strong, states may instead “bandwagon”: joining up with and
supporting the rising power to gain some personal benefit.<p>
For instance, consider military coups. Sometimes, those attempting a
takeover will offer their various enemies incentives to join forces with
them, promising rewards to whoever allies with them first. If one of
those being made this offer believes that the usurpers are ultimately
likely to win, they may consider it to be in their own best interests to
switch sides early enough to be on the “right side of history.” When
others observe their allies switching sides, they may see their chances
of victory declining and so in turn decide to defect. In this way,
bandwagoning can escalate via positive feedback.<p>
Bandwagoning may therefore present the following collective action
problem: people may be motivated to cooperate with powerful and
threatening AI systems via direct reciprocity, even though it would be
in everyone’s collective best interest if none were to do so. Imagine
that a future AI system, acting autonomously, takes actions that cause a
large-scale catastrophe. In the wake of this event, the international
community might agree that it would be in humanity’s best interest to
constrain or roll back all autonomous AIs. Powerful AI systems might
then offer some states rewards if they ally with them (direct
reciprocity). This could mean protecting the AIs by simply allowing them
to intermingle with the people, making it harder for outside forces to
target the AIs without human casualties. Or the state could provide the
AIs with access to valuable resources. Instead of balancing (cooperating
with the international community to counteract this threatening power),
these states may choose to bandwagon, defecting to form alliances with
AIs. Even though the global community would all be better off if all
states were to cooperate and act together to constrain AIs, individual
states may benefit from defecting. As before, each defection would shift
the balance of power, motivating others to defect in turn.</p>
<h3 id="indirect-reciprocity">Indirect Reciprocity</h3>
<p><strong>Indirect reciprocity overview.</strong> When someone judges
whether to provide a favor to someone else, they may consider the
recipient’s reputation. If the recipient is known to be generous, this
would encourage the donor (the one that provides the favor) to offer
their assistance. On the other hand, if the recipient has a stingy or
selfish reputation, this could discourage the donor from offering a
favor. In considering whether to provide a favor, donors may also
consider the favor’s effect on their own reputation. If a donor gains a
“helpful and trustworthy” reputation by providing a favor, this may
motivate others to cooperate with them more often. We call this
reputation-based mechanism of cooperation <em>indirect reciprocity</em>
<span class="citation" data-cites="nowak1998evolution">[10]</span>.
Agents may cooperate to develop and maintain good reputations since
doing so is likely to benefit them in the long-term. Indirect
reciprocity is particularly useful in larger groups, where the
probability that the same two agents will encounter one another again is
lower. It provides a mechanism for leveraging collective knowledge to
promote cooperation. Where personal interactions are limited,
reputation-based evaluations provide a way to assess the cooperative
tendencies of others. Importantly, cooperation can only emerge within a
population as a consequence of indirect reciprocity when the
probability, <span class="math inline"><em>q</em></span>, that any agent
can discern another agent’s reputation (whether they are cooperative or
not), outweighs the cost-benefit ratio of the helpful behavior to the
donor. Thus, we have the rule <span
class="math inline"><em>q</em> &gt; <em>c</em>/<em>b</em></span>; see
Table 8.2 below.<p>
</p>
<br>
<p><span id="tab:indirect-repr" label="tab:indirect-repr"></span></p>
<div id="tab:indirect-repr">
<table class="tableLayout">
<caption>Table 8.8: Payoff matrix for indirect reciprocity games.</caption>
<thead>
<tr class="header">
<th style="text-align: center;"></th>
<th style="text-align: center;">Discern</th>
<th style="text-align: center;">Defect</th>
</tr>
</thead>
<tbody>
<tr class="odd">
<td style="text-align: center;">Discern</td>
<td style="text-align: center;"><span
class="math inline"><em>b</em> − <em>c</em></span></td>
<td style="text-align: center;">-c(1-q)</td>
</tr>
<tr class="even">
<td style="text-align: center;">Defect</td>
<td style="text-align: center;"><span
class="math inline"><em>b</em>(1−<em>q</em>)</span></td>
<td style="text-align: center;">0</td>
</tr>
</tbody>
</table>
</div>
<br>
<p><strong>Natural examples of indirect reciprocity.</strong> Cleaner
fish (fish that feed on parasites or mucus on the bodies of other fish)
can either cooperate with client fish (fish that receive the “services”
of cleaner fish) by feeding on parasites that live on their bodies, or
cheat, by feeding on the mucus that client fish excrete <span
class="citation" data-cites="bshary2006image">[11]</span>. Client fish
tend to cooperate more frequently with cleaner fish that have a “good
reputation,” which are those that feed on parasites rather than mucus.
Similarly, while vampire bats are known to share food with their kin,
they also share food with unrelated members of their group. Vampire bats
more readily share food with unrelated bats when they know the
recipients of food sharing also have a reputation for being consistent
and reliable food donors <span class="citation"
data-cites="carter2013food">[12]</span>.</p>
<p><strong>Indirect reciprocity in human society.</strong> Language
provides a way to obtain information about others without ever having
interacted with them, allowing humans to adjust reputations accordingly
and facilitate conditional cooperation. Consider sites like Yelp and
TripAdvisor, which allow internet users to gauge the reputations of
businesses through reviews provided by other consumers. Similarly,
gossip is a complex universal human trait that plays an important role
in indirect reciprocity. Through gossip, individuals reveal the nature
of their past interactions with others as well as exchanges they observe
between others but are not a part of. Gossip allows us to track each
others’ reputations and enforce cooperative social norms, reducing the
probability that cooperative efforts are exploited by others with
reputations for dishonesty <span class="citation"
data-cites="balliet2020indirect">[13]</span>.</p>
<p><strong>Indirect reciprocity in AIs.</strong> AIs could develop a
reputation system where they observe and evaluate each others’
behaviors, with each accumulating a reputation score based on their
cooperative actions. AIs with higher reputation scores may be more
likely to receive assistance and cooperation from others, thereby
developing a reputation for reliability. Moreover, sharing insights and
knowledge with <em>reliable</em> partners may establish a network of
cooperative AIs, promoting future reciprocation.</p>
<p><strong>Indirect reciprocity can backfire: extortionists can threaten
reputational damage.</strong> The pressure to maintain a good reputation
can make agents vulnerable to extortion. Other agents may be able to
leverage the fear of reputational harm to extract benefits or force
compliance. For example, political smear campaigns manipulate public
opinion by spreading false information or damaging rumors about
opponents. Similarly, blackmail often involves leveraging damaging
information about others to extort benefits. AIs may manipulate or
extort humans in order to better pursue their objectives. For instance,
an AI might threaten to expose the sensitive, personal information it
has accessed about a human target unless specific demands are met.</p>
<p><strong>Indirect reciprocity can backfire: ruthless reputations may
also work.</strong> Indirect reciprocity can also favor the emergence of
“ruthless” reputations. A reputation for ruthlessness can sometimes be
extremely successful in motivating compliance through fear. For
instance, in military contexts, projecting a reputation for ruthlessness
may deter potential adversaries or enemies. If others perceive an
individual or group as willing to employ extreme measures without
hesitation, they may be less likely to challenge or provoke them. Some
AIs might similarly evolve ruthless reputations, perhaps as a defensive
strategy to discourage potential attempts at exploitation, or control by
others.</p>
<h3 id="group-selection">Group Selection</h3>
<p><strong>Group selection overview.</strong> When there is competition
between groups, groups with more cooperators may outcompete those with
fewer cooperators. Under such conditions, selection at the group level
influences selection at the individual level (traits that benefit the
group may not necessarily benefit the individual), and we refer to this
mechanism as <em>group selection</em> <span class="citation"
data-cites="west2007social">[14]</span>. Importantly, between groups,
groups with a higher proportion of cooperators have an advantage.
Cooperative groups are better able to coordinate their allocation of
resources, establish channels for reciprocal exchange, and maintain
steady communication, making them less likely to go extinct. Moreover,
cooperative groups more frequently split in two: as cooperative groups
grow in size, social tensions may emerge and threaten the cohesion of
the group, leading members to break off into their own cooperative
groups. It so happens that, if <span
class="math inline"><em>m</em></span> is large and is the number of
groups and n is the maximum group size, group selection can only promote
cooperation when <span
class="math inline"><em>b</em>/<em>c</em> &gt; 1 + <em>n</em>/<em>m</em></span>;
see Table 8.9 below.<p>
</p>
<br>
<div id="tab:group">
<table class="tableLayout">
<caption>Table 8.9: Payoff matrix for group selection games.</caption>
<thead>
<tr class="header">
<th style="text-align: center;"></th>
<th style="text-align: center;">Cooperate</th>
<th style="text-align: center;">Defect</th>
</tr>
</thead>
<tbody>
<tr class="odd">
<td style="text-align: center;">Cooperate</td>
<td style="text-align: center;"><span
class="math inline">(<em>n</em>+<em>m</em>)(<em>b</em>+<em>c</em>)</span></td>
<td style="text-align: center;"><span
class="math inline"><em>n</em>(−<em>c</em>) + <em>m</em>(<em>b</em>−<em>c</em>)</span></td>
</tr>
<tr class="even">
<td style="text-align: center;">Defect</td>
<td style="text-align: center;"><span
class="math inline"><em>n</em><em>b</em></span></td>
<td style="text-align: center;"><span class="math inline">0</span></td>
</tr>
</tbody>
</table>
</div>
<br>
<p><strong>Natural examples of group selection.</strong> Group selection
is widely viewed as a mechanism involved in the process of natural
selection, however, direct causal attribution is difficult for many
species. Nonetheless, we may view chimpanzees that engage in lethal
intergroup conflict as a likely example of group selection. Chimpanzees
can be remarkably violent toward outgroups, such as by killing the
offspring of rival males or engaging in brutal fights over territory.
Such behaviors can help groups of chimpanzees secure competitive
advantages over other groups of chimpanzees, by either reducing their
abilities to mate successfully through infanticide, or by securing
larger portions of available territory.</p>
<p><strong>Group selection in human society.</strong> Among humans,
warfare is a salient example of group selection. Imagine two armies: A
and B. The majority of soldiers in army A are brave, while the majority
of soldiers in army B are cowardly. For soldiers in army A, bravery may
be individually costly, since brave soldiers are more willing to risk
losing their lives on the battlefield. For soldiers in army B, cowardice
may be individually beneficial, since cowardly soldiers will take fewer
life-threatening risks on the battlefield. However, at the group-level,
when these two armies conflict, army A will defeat army B: brave
soldiers will be more willing to fight alongside each other for victory,
while cowardly soldiers will not.</p>
<p><strong>Group selection in AIs.</strong> Consider a future in which
the majority of human labor has been fully automated by AIs, such that
AIs are now running most companies. Under these circumstances, AIs may
form corporations with other AIs, creating an economic landscape in
which multiple AI corporations must compete with each other to produce
economic value. AI corporations in which individual AIs work well
together may outcompete those in which individual AIs do not work as
well together. The more cooperative individual AIs within AI
corporations are, the more economic value their corporations will be
able to produce; AI corporations with less cooperative AIs may
eventually run out of resources and lose the ability to sustain
themselves.</p>
<p><strong>Group selection can backfire: in-group favoritism can promote
out-group hostility.</strong> Group selection can inspire in-group
favoritism, which might lead to cruelty toward out-groups. Chimpanzees
will readily cooperate with members of their own groups. However, when
interacting with chimpanzees from other groups, they are often vicious
and merciless. Moreover, when groups gain a competitive advantage, they
may attempt to preserve it by mistreating, exploiting, or marginalizing
outgroups such as people with different political or ideological
beliefs. AIs may be more likely to see other AIs as part of their group,
and this could promote antagonism between AIs and humans.</p>
<h3 id="kin-selection">Kin Selection</h3>
<p><strong>Kin selection overview.</strong> When driven by <em>kin
selection</em>, agents are more likely to cooperate with others with
whom they share a higher degree of genetic relatedness <span
class="citation" data-cites="hamilton1964genetical">[15]</span>. The
more closely related agents are, the more inclined to cooperate they
will be. Thus, kin selection favors cooperation under the following
conditions: an agent will help their relative only when the benefit to
their relative “<span class="math inline"><em>b</em></span>,” multiplied
by the relatedness between the two “<span
class="math inline"><em>r</em></span>,” outweighs the cost to the agent
“<span class="math inline"><em>c</em></span>.” This is known as
Hamilton’s rule: <span
class="math inline"><em>r</em><em>b</em> &gt; <em>c</em></span>, or
equivalently <span
class="math inline"><em>r</em> &gt; <em>c</em>/<em>b</em></span> <span
class="citation" data-cites="hamilton1964genetical">[15]</span>; see
Table 8.10 below.</p>
<br>
<div id="tab:kin">
<table class="tableLayout">
<caption>Table 8.10: Payoff matrix for kin selection games.</caption>
<thead>
<tr class="header">
<th style="text-align: center;"></th>
<th style="text-align: center;">Cooperate</th>
<th style="text-align: center;">Defect</th>
</tr>
</thead>
<tbody>
<tr class="odd">
<td style="text-align: center;">Cooperate</td>
<td style="text-align: center;"><span
class="math inline">(<em>b</em>−<em>c</em>)(1+<em>r</em>)</span></td>
<td style="text-align: center;"><span
class="math inline">(−<em>c</em>+<em>b</em><em>r</em>)</span></td>
</tr>
<tr class="even">
<td style="text-align: center;">Defect</td>
<td style="text-align: center;"><span
class="math inline"><em>b</em> − <em>r</em><em>c</em></span></td>
<td style="text-align: center;">0</td>
</tr>
</tbody>
</table>
</div>
<br>
<p><strong>Natural examples of kin selection.</strong> In social insect
colonies, such as bees and ants, colony members are closely related.
Such insects often assist their kin in raising and producing offspring
while “workers” relinquish their reproductive potential, devoting their
lives to foraging and other means required to sustain the colony as a
whole. Similarly, naked mole rats live in colonies with a single
reproductive queen and non-reproductive workers. The workers are sterile
but still assist in tasks such as foraging, nest building, and
protecting the colony. This behavior benefits the queen’s offspring,
which are their siblings, and enhances the colony’s overall survival
capabilities. Alternatively, some bird species engage in cooperative
breeding practices where older offspring delay breeding to help parents
raise their siblings.</p>
<p><strong>Kin selection in human society.</strong> Some evolutionary
psychologists claim that we can see evidence of kin selection in many
commonplace traditions and activities. For example, in humans, we might
identify the mechanism of kin selection in the way that we treat our
immediate relatives. For instance, people often leave wealth, property,
and other resources to direct relatives upon their deaths. Leaving
behind an inheritance offers no direct benefit to individuals, but it
does help ensure the survival and success of their lineage in subsequent
generations. Similarly, grandparents often care for their grandchildren,
which increases the probability that their lineages will persist.</p>
<p><strong>Kin selection in AIs.</strong> AIs that are similar could
exhibit cooperative tendencies towards each other, similar to genetic
relatedness in biological systems. For instance, AIs may create back ups
or variants of themselves. They may then favor cooperation with these
versions of themselves over other AIs or humans. Variant AIs may
prioritize resource allocation and sharing among themselves, developing
preferential mechanisms for sharing computational resources with other
versions of themselves.</p>
<p><strong>Kin selection can backfire: nepotism.</strong> Kin selection
can lead to nepotism: prioritizing the interests of relatives above
others. For instance, some bird species exhibit differential feeding and
provisioning. When chicks hatch asynchronously, parents may allocate
more resources to those that are older, and therefore more likely to be
their genetic offspring, since smaller chicks are more likely to be the
result of brood parasitism (when birds lay their eggs in other birds’
nests). In humans, too, we often encounter nepotism. Company executives
may hire their sons or daughters, even though they lack the experience
required for the role, which can harm companies and their employees in
the long-run. Similarly, parents often protect their children from the
law, especially when they have committed serious criminal acts that can
result in extended jail time. Such tendencies could apply to AIs as
well: AIs might favor cooperation only with other similar AIs. This
could be especially troubling for humans –— as the differences between
humans and AIs increase, AIs may be increasingly less inclined to
cooperate with humans.</p>
<h3 id="individual-stakes-to-common-stakes">Individual stakes to common
stakes</h3>
<p><strong>Individual stakes to common stakes overview.</strong> The
Nobel laureate economist John Harsanyi suggested a concept like the veil
of ignorance to explore how we should structure our societies. Behind
the veil of ignorance, agents are unaware of their personal
characteristics and roles in society. This forces them to act as
impartial observers when envisioning how the group or society should
interact with each other. (For a more detailed discussion of the veil of
ignorance, see <em>Section social contract of the Ethics chapter</em>: A Brief Introduction).
Because agents do not know what place they will hold (e.g., whether they
will get an average position in society or a different position) in the
society they envision, they often act more impartially and cooperatively
when constructing a group or society. When rational agents are ignorant
of their future position in society—-when they are forced to make
decisions behind the veil of ignorance—-they are more likely to make
decisions that maximize collective wellbeing, enabling cooperation
through the mechanism of <em>individual stakes to common stakes</em>
<span class="citation"
data-cites="nowak2011supercooperators">[16]</span>.</p>
<p><strong>Natural examples of individual stakes to common
stakes.</strong> Intragenomic conflict arises when an individual
organism’s genes have differing interests with respect to their
transmission to the next generation. For example, some genes may have
evolved mechanisms that increase their own replication at the expense of
the organism’s wellbeing. The process of meiosis, however, can resolve
the problem of intragenomic conflict through randomization or the
creation of a “Darwinian veil of ignorance.” Meiosis is a process of
cell division that occurs during the formation of reproductive cells in
sexually reproducing organisms. Meiosis typically results in a 50-50
distribution of genetic material from both parents to offspring,
fostering genetic diversity. While it would be “better” for any
individual gene to make more copies of itself in offspring, it cannot
“know” whether doing so will actually increase the fitness of the
organism, allowing it to pass on its genes to subsequent generations.
However, by fostering genetic diversity, the process of meiosis opens
the door to randomized success, allowing the forces of natural selection
to impartially dictate which genes and their respective traits propagate
<span class="citation" data-cites="okasha2012social">[17]</span>.</p>
<p><strong>Individual stakes to common stakes in human society.</strong>
The “Hutterites” are a religious group known for their cooperative and
communal lifestyle <span class="citation"
data-cites="Harsanyi1955CardinalWI Dennett1995DarwinsDI">[18],
[19]</span>. They represent a prime example of how the mechanism of
individual stakes to common stakes can be used to promote cooperation.
Within their communities, the Hutterites distribute all resources among
themselves, and relinquish any personal wealth or possessions. This
allows the Hutterites to live under a collective identity, where focus
on the common good often overrides individual interests. As another
example, when their group gets too large, Hutterites set up another
place for people to live, and once it is finished, they randomly select
members from their community to go live there. Alternatively, consider a
scenario in which a group of people must row to shore. The ship does not
have enough food for everyone, so the group will have to throw some
rowers overboard at the halfway point to ensure the rest of the group’s
survival. However, no one in the group knows that they may be thrown
overboard, but they do know that to make it to shore, they must all row
now. If people knew that they would be thrown overboard at some point,
they would be far less likely to row.</p>
<p><strong>Individual stakes to common stakes and AIs.</strong> People
might perceive advanced future AIs as an outgroup threat, and this may
motivate humans to cooperate against AIs. As AI’s become more widespread
and powerful, they may begin to threaten human values at the global
scale, motivating nations—-for instance, the US and China–—to set aside
their differences and come together to address a common threat. If AI is
someday viewed by humanity as an “invasive species,” it may promote
global solidarity.</p>
<h3 id="simons-selection-mechanism">Simon’s Selection Mechanism</h3>
<p><strong>Simon’s selection mechanism overview.</strong> Humans and
other animals typically do not have access to all available information,
time, and cognitive abilities required to make the best possible
decision. Individuals are restricted by their “bounded rationality.” To
overcome their bounded rationality and arrive at better solutions,
people and organisms can benefit, on average, from relying on and
receiving information through cooperative social channels. Individuals
that take advantage of cooperative social channels have an increased
ability to acquire socially transmitted skills and conform to socially
established norms, thereby incurring a fitness advantage over those that
do not participate in such channels. Moreover, when individuals choose
to contribute to cooperative social channels, the information they
provide can be freely utilized by anyone who participates in the system.
If this information is widely used and benefits the entire social
structure, social norms may emerge that compel others participating in
the cooperative social channel to conform their behavior appropriately.
In other words, when individuals participate in cooperative social
channels, they may benefit from the collective intelligence they
provide, but they may also face a cost from the group which can
influence their behaviors through norms or rules. This idea was
pioneered by the political scientist and Nobel laureate, Herbert Simon,
and we refer to it as <strong>Simon’s selection mechanism</strong> <span
class="citation" data-cites="simon1990mechanism">[20]</span>.</p>
<p><strong>Simon’s selection mechanism in human society.</strong> Humans
have a tendency to believe in facts and propositions that they have not
had the opportunity to independently verify. Such knowledge is typically
disseminated through established cooperative social channels, such as
the internet or culture. For instance, many people believe that
consuming too much of certain kinds of cholesterol is bad for one’s
health, or that touching a hot stove should always be avoided. People
generally agree upon these beliefs not because they are skilled medical
literature reviewers or seasoned chefs, but because such beliefs have
become socially transmitted common knowledge. In general, people are
better off being a part of society, rather than isolating themselves
from it. Societal isolation may prevent individuals from accessing
critical information that is disseminated through social channels. For
instance, a hermit living in the woods with no internet access might be
unaware that there is steadily expanding forest fire in the
vicinity.</p>
<p><strong>Simon’s selection mechanism and free riding.</strong> One
drawback of Simon’s selection mechanism is that it may enable free
riding. Individuals may benefit from the information contained in
cooperative social channels without themselves contributing to it.
Consider the open-source knowledge base, Wikipedia, in this respect.
Though anyone can, in theory, contribute to Wikipedia, few people
actually do, because doing so can require extensive time and effort.
This can result in a knowledge base that has much outdated content.</p>
<p><strong>Simon’s selection mechanism and AIs.</strong> In a
multi-agent setting, AIs may be interacting with one another directly.
They may create new communication channels and protocols among
themselves, and from their interactions, norms and information channels
may emerge. Such dynamics can give rise to very complex social systems
from which AIs are more able to benefit than humans are. In other words,
AIs may incur fitness advantages over humans by developing and
participating in their own cooperative social channels, which themselves
may preclude human understanding. Moreover, such cooperative social
channels may evolve into AI collective intelligences, just as the
internet now represents a form of human collective intelligence.
However, within these collective intelligences, AIs could maintain large
numbers of complex relationships with other AIs simultaneously, arriving
at potentially new forms of self-organization that increase AIs ability
to achieve their goals. AI collective intelligences would be vastly
superior to human collective intelligences, and as a consequence, humans
may be prevented from participating in and understanding the decisions
AIs make within such systems.</p>
<br>
<div class="visionbox">
<legend class="visionboxlegend">
    <p><span><b>A Note on Morality as Cooperation</b></span></p>
</legend>
The theory of
“Morality as Cooperation” (MAC) proposes that human morality was
generated by evolutionary pressures to solve our most salient
cooperation problems <span class="citation"
data-cites="curry2016morality">[21]</span>. Natural selection has
discovered several mechanisms by which rational and self-interested
agents may cooperate with one another, and MAC theory suggests that some
of these mechanisms have driven the formation of our moral intuitions
and customs. Here, we examine four cooperation problems, the mechanisms
humans have evolved to solve them, and how these mechanisms may have
generated our ideas of morality. These are overviewed in Table 8.11.</p>
<div id="tab:cooperation">
<table class="tableLayout" style="">
<caption>Table 8.11: Mapping cooperation mechanisms to components of morality. <span class="citation"
data-cites="curry2016morality">[21]</span></caption>
<thead>
<tr class="header" style="background-color: #efefef">
<th style="text-align: left;"><strong>Cooperation Problem</strong></th>
<th style="text-align: left;"><strong>Solutions/Mechanism</strong></th>
<th style="text-align: left;"><strong>Component of
Morality</strong></th>
</tr>
</thead>
<tbody>
<tr class="odd" style="background-color: #d9ead3">
    <td rowspan="2" style="text-align: left;"><em> <b>Kinship</b> <br> Agents can benefit by treating genetic  relatives preferentially</em></td>
<td style="text-align: left;">Kin selection</td>
<td style="text-align: left;">Parental duties, family values</td>
</tr>
<tr class="even" style="background-color: #d9ead3">
<td style="text-align: left;">Avoiding inbreeding</td>
<td style="text-align: left;">Incest aversion</td>
</tr>
<tr class="even" style="background-color: #d0e0e3">
    <td rowspan="2" style="text-align: left;"><em> <b> Mutualism </b> <br> Agents must coordinate their behavior to profit from mutually-benefical situations</em></td>
    <td style="text-align: left;">Forming alliances and collaborating</td>
    <td style="text-align: left;">Friendship, loyalty, commitment, team
players</td>
</tr>
<tr class="even" style="background-color: #d0e0e3">
<td style="text-align: left;">Developing theory-of-mind</td>
<td style="text-align: left;">Understanding intention, not merely
action</td>
</tr>
<tr class="odd" style="background-color: #d9d2e9">
<td rowspan="2" style="text-align: left"><em> <b> Exchange </b> <br> Agents need each other to reciprocate
and contribute despite incentives to free ride</em></td>
<td style="text-align: left;">Direct reciprocity (e.g. tit-for-tat)</td>
<td style="text-align: left;">Trust, gratitude, revenge, punishment, forgiveness</td>
</tr>
<tr class="odd" style="background-color: #d9d2e9">
<td style="text-align: left;">Indirect reciprocity (e.g. forming
reputations)</td>
<td style="text-align: left;">Patience, guilt, gratitude</td>
</tr>
<tr class="even" style="background-color: #ead1dc">
<td rowspan="2" style="text-align: left;"> <b> Conflict Resolution</b> <br> <em>Agents can benefit from avoiding
conflict, which is mutually costly</em></td>
<td style="text-align: left;">Division</td>
<td style="text-align: left;">Fairness, negotiation, compromise</td>
</tr>
<tr class="even" style="background-color: #ead1dc">
<td style="text-align: left;">Deference to prior ownership</td>
<td style="text-align: left;">Respecting others’ property, punishing
theft</td>
</tr>
</tbody>
</table>
</div>
    <br>
<p><strong>Kinship.</strong> Natural selection can favor agents who
cooperate with their genetic relatives. This is because there may be
copies of these agents’ genes in their relatives’ genomes, and so
helping them may further propagate their own genes. We call this
mechanism “kin selection” <span class="citation"
data-cites="hamilton1964genetical">[15]</span>: an agent can gain a
fitness advantage by treating their genetic relatives preferentially, so
long as the cost-benefit ratio of helping is less than the relatedness
between the agent and their kin. Similarly, repeated inbreeding can
reduce an agent’s fitness by increasing the probability of producing
offspring with both copies of any recessive, deleterious alleles in the
parents’ genomes <span class="citation"
data-cites="charlesworth2009genetics">[22]</span>.<p>
MAC theory proposes that the solutions to this cooperation problem
(preferentially helping genetic relatives), such as kin selection and
inbreeding avoidance, underpin several major moral ideas and customs.
Evidence for this includes the fact that human societies are usually
built around family units <span class="citation"
data-cites="chagnon1979kin">[23]</span>, in which “family values” are
generally considered highly moral. Loyalty to one’s close relatives and
duties to one’s offspring are ubiquitous moral values across human
cultures <span class="citation"
data-cites="westermarck2022origin">[24]</span>. Our laws regarding
inheritance <span class="citation"
data-cites="smith1987inheritance">[25]</span> and our naming traditions
<span class="citation" data-cites="oates2002nominal">[26]</span>
similarly reflect these moral intuitions, as do our rules and social
taboos against incest <span class="citation"
data-cites="lieberman2003does thornhill1991evolutionary">[27],
[28]</span>.</p>
<p><strong>Mutualism.</strong> In game theory, some games are “positive
sum” and “win-win”: the agents involved can increase the total available
value by interacting with one another in particular ways, and all the
agents can then benefit from this additional value. Sometimes, securing
these mutual benefits requires that the agents coordinate their behavior
with each other. To solve this cooperation problem, agents may form
alliances and coalitions <span class="citation"
data-cites="connor1995benefits">[29]</span>. This may require the
capacity for basic communication, rule-following <span class="citation"
data-cites="van2008leadership">[30]</span>, and perhaps theory-of-mind
<span class="citation"
data-cites="carruthers1996theories">[31]</span>.<p>
MAC theory proposes that these cooperative mechanisms comprise important
components of human morality. Examples include the formation of—and
loyalty to—friendships, commitments to collaborative activities, and a
certain degree of in-group favoritism and conformation to local
conventions. Similarly, we often consider the agent’s intentions when
judging the morality of their actions, which requires a certain degree
of theory-of-mind.</p>
<p><strong>Exchange.</strong> Sometimes, benefiting from “win-win”
situations requires more than mere coordination. If the payoffs are
structured so as to incentivize “free riding” behaviors, the cooperation
problem becomes how to ensure that others will reciprocate help and
contribute to group efforts. To solve this problem, agents can enforce
cooperation via systems of reward, punishment, policing, and reciprocity
<span class="citation" data-cites="west2007evolutionary">[32]</span>.
Direct reciprocity concerns doing someone a favor out of the expectation
that they will reciprocate at a later date <span class="citation"
data-cites="trivers1971evolution">[6]</span>. Indirect reciprocity
concerns doing someone a favor to boost your reputation in the group,
out of the expectation that this will increase the probability of a
third party helping you in the future <span class="citation"
data-cites="nowak1998evolution">[10]</span>.<p>
Once again, MAC theory proposes that these mechanisms are found in our
moral systems. Moral ideas such as trust, gratitude, patience, guilt,
and forgiveness can all help to assure against free riding behaviors.
Likewise, punishment and revenge, both ideas with strong moral
dimensions, can serve to enforce cooperation more assertively. Idioms
such as “an eye for an eye”, or the “Golden Rule” of treating others as
we would like to be treated ourselves, reflect the solutions we evolved
to this cooperation problem.</p>
<p><strong>Conflict resolution.</strong> Conflict is very often
“negative sum”: the interaction of the agents themselves can destroy
some amount of the total value available. Examples span from the wounds
of rutting deer to the casualties of human wars. If the agents instead
manage to cooperate with each other, they may both be able to benefit—a
“win-win” outcome. One way to resolve conflict situations is division
<span class="citation"
data-cites="nash1950bargaining"><strong>nash1950bargaining?</strong></span>:
dividing up the value between the agents, such as through striking a
bargain. Another solution is to respect prior ownership, deferring to
the original “owner” of the valuable item <span class="citation"
data-cites="gintis2007evolution">[33]</span>.<p>
According to MAC theory, we can see both of these solutions in our ideas
of morality. The cross-culturally ubiquitous notions of fairness,
equality, and compromise help us resolve conflict by promoting the
division of value between competitors <span class="citation"
data-cites="henrich2005economic">[34]</span>. We see this in ideas such
as “taking turns” and “I cut, you choose” <span class="citation"
data-cites="brams1996fair">[35]</span>: mechanisms for turning a
negative sum situation (conflict) into a zero sum one (negotiation), to
mutual benefit. Likewise, condemnation of theft and respect for others’
property are extremely important and common moral values <span
class="citation"
data-cites="herskovits1952economic westermarck2022origin">[24],
[36]</span>. This set of moral rules may stem from the conflict
resolution mechanism of deferring to prior ownership.</p>
<p><strong>Conclusion.</strong> MAC theory argues that morality is
composed of biological and cultural solutions humans evolved to the most
salient cooperation problems of our ancestral social environment. Here,
we explored four examples of cooperation problems, and how the solutions
to them discovered by natural selection may have produced our moral
values.</p>
</div>
<br>
<h3 id="institutions">Institutions</h3>
<p><strong>Institutions overview.</strong> Agents are more likely to be
cooperative when there are laws or externally imposed incentives that
reward cooperation and punish defection. We define an
<strong>institution</strong> as an intentionally designed large-scale
structure that is publicly accepted and recognized, has a centralized
logic, and serves to mediate human interaction. Some examples of
institutions include governments, the UN, IAEA, and so on; in this
section, by “institutions,” we do not mean widespread or standardized
social customs such as the “institution” of marriage. Institutions
typically aim to establish collective goals which require collaboration
and engagement from large or diverse groups. Therefore, a possible way
of representing many institutions, such as governments, is with the
concept of a “Leviathan”: a powerful entity that can exert control or
influence over other actors in a system.</p>
<p><strong>The Pacifist’s dilemma and social control.</strong> When
one’s opponent is potentially aggressive, pacifism can be irrational. In
his book, “The Better Angels of Our Nature,” Steven Pinker refers to
this as the “Pacifist’s dilemma” <span class="citation"
data-cites="pinker2012better">[8]</span>. In potential conflict
scenarios, agents have little to gain and a lot to lose when they
respond to aggression with pacifism; see Table 8.12 below.
This dynamic often inspires rational agents to choose conflict over
peace.<p>
</p>
<br>
<div id="tab:pacifist">
<table class="tableLayout">
<caption>Table 8.12: Payoff matrix for the Pacifist’s dilemma without a Leviathan. <span class="citation"
data-cites="pinker2012better">[8]</span></caption>
<thead>
<tr class="header">
<th style="text-align: left;"></th>
<th style="text-align: center;">Pacifist</th>
<th style="text-align: center;">Aggressor</th>
</tr>
</thead>
<tbody>

    <tr class="odd">
        <td style="text-align: left;">Pacifist</td>
        <td style="text-align: center;"> Peace + Profit
            <span class="math inline">(100+5) = 105</span> <br>
            Peace + Profit
            <span class="math inline">(100+5) = 105</span></td>
        <td style="text-align: center;">
            Victory(10)<br>
            Defeat(-100)<br>
        </td>
    </tr>
<tr class="even">
    <td style="text-align: left;">Aggressor</td>
    <td style="text-align: center;">
        Defeat(-100)<br>
        Victory(10)
    </td>
    <td style="text-align: center;">
        War(-50) <br>
        War(-50)
    </td>
</tr>

</tbody>
</table>
</div>
<br>

<p>However, we can shift the interests of agents in this context in
favor of peace by introducing a Leviathan, in the form of a third-party
peacekeeping or balancing mission, which establishes an authoritative
presence that maintains order and prevents conflict escalation.
Peacekeeping missions can take several forms, but they often involve the
deployment of peacekeeping forces such as military, police, and civilian
personnel. These forces work to deter potential aggressors, enhance
security, and set the stage for peaceful resolutions and negotiations as
impartial mediators, usually by penalizing aggression and rewarding
pacifism; see Table 8.13 below.<p>
</p>
<br>
<div id="tab:leviathan">
<table class="tableLayout">
<caption>Table 8.13: Payoff matrix for the Pacifist’s dilemma with a Leviathan. <span class="citation"
data-cites="pinker2012better">[8]</span></caption>
<thead>
<tr class="header">
<th style="text-align: left;"></th>
<th style="text-align: center;">Pacifist</th>
<th style="text-align: center;">Aggressor</th>
</tr>
</thead>
<tbody>

    <tr class="odd">
        <td style="text-align: left;">Pacifist</td>
        <td style="text-align: center;"> Peace(5) <br> Peace(5)</td>
        <td style="text-align: center;">
            Victory - Penalty (10 − 15 = −5)<br>
            Defeat(-100)<br>
        </td>
    </tr>
<tr class="even">
    <td style="text-align: left;">Aggressor</td>
    <td style="text-align: center;">
        Defeat(-100)<br>
        Victory - Penalty (10 − 15 − 5)
    </td>
    <td style="text-align: center;">
        War - Penalty (−50 − 200 = −250) <br>
        War - Penalty (−50 − 200 = −250)
    </td>
</tr>
</tbody>
</table>
</div>
<br>
<p><strong>Institutions in human society.</strong> Institutions play a
central role in promoting cooperation in international relations.
Institutions, such as the UN, can broker agreements or treaties between
nations and across cultures through balancing and peacekeeping
operations. The goal of such operations is to hold nations accountable
on the international scale; when nations break treaties, other nations
may punish them by refusing to cooperate, such as by cutting off trade
routes or imposing sanctions and tariffs. On the other hand, when
nations readily adhere to treaties, other nations may reward them, such
as by fostering trade or providing foreign aid. Similarly, institutions
can incentivize cooperation at the national scale by creating laws and
regulations that reward cooperative behaviors and punish non-cooperative
ones. For example, many nations attempt to prevent criminal behavior by
leveraging the threat of extended jail-time as a legal deterrent to
crime. On the other hand, some nations incentivize cooperative behaviors
through tax breaks, such as those afforded to citizens that make
philanthropic donations or use renewable energy resources like solar
power.</p>
<p><strong>Institutions and AI.</strong> Institutions are crucial in the
context of international AI development. By establishing laws and
regulations concerning AI development, institutions may be able to
reduce AI races, lowering competitive pressures and the probability that
countries cut corners on safety. Moreover, we may want concrete
international agreements on AI development that serve to hold nations
accountable; institutions could play a central role in helping us broker
these kinds of agreements. Ultimately, institutions could improve
coordination mechanisms and international standards for AI development,
which would correspondingly improve AI safety.<p>
Recall that institutions can punish defectors. In the future, harmful
AIs or harmful malicious actors could be punished with or by AIs. In the
future, humanity alone may not possess the power required to combat
advanced dominance-seeking AIs, and existing laws and regulations may be
insufficient if there is no way to enforce them. Such an AI Leviathan
could help regulate other AIs and affect their evolution, in which
selfish AIs are counteracted or domesticated.</p>
<p><strong>How institutions can backfire: corruption, free riding,
inefficiency.</strong> Institutions sometimes fail to achieve the goals
they set for themselves, even if they are well-intended. Failure to
achieve such goals is often the result of corruption, free riding, and
inefficiency at the institutional scale. Some examples of corruption
include bribery, misappropriation of public funds for private interests,
voter fraud and manipulation, and price fixing, among many others.
Examples of free-riding include scenarios like welfare fraud, where
individuals fraudulently receive benefits they may not be entitled to,
reducing the available supply of resources for those genuinely in need.
Institutions can also struggle with inefficiency, which may stem from
factors such as the satisfaction of bureaucratic requirements, the
emergence of natural monopolies, or the development of diseconomies of
scale, which may cause organizations to pay a higher average cost to
produce more goods and services. Institutions can be undermined,
corrupted, and poorly designed or outdated: they do not guarantee that
we will be able to fix cooperation problems.<p>
Like humans, AIs may be motivated to corrupt existing institutions.
Advanced AIs might learn to leverage the institutions we have in place
for their benefit, and might do so in ways that are virtually
undetectable to us. Moreover, as we discussed previously, AIs might form
an AI Leviathan. However, if humanity’s relationship with this Leviathan
is not symbiotic and transparent, humans risk losing control of AIs. For
instance, if groups of AIs within the Leviathan collude behind the
scenes to further their own interests, or power and resources become
concentrated with a few AIs at the “top,” humanity’s collective
wellbeing could be threatened.</p>
<h3 id="summary-1">Summary</h3>
<p>Throughout this section, we discussed a variety of mechanisms that
promote cooperation. These mechanisms included direct reciprocity,
indirect reciprocity, group selection, kin selection, individual stakes
to common stakes, Simon’s selection mechanism, and institutions.<p>
First, we discussed how direct reciprocity may motivate AI agents in a
multi-agent setting to cooperate with each other, if the probability
that the same two AIs meet again is sufficiently high. However, we also
considered the possibility that AIs may disfavor cooperation with humans
as they become progressively more advanced: the cost-benefit ratio for
cooperation with humans may simply be bad from an AI’s
perspective.<p>
Second, we explored how indirect reciprocity may promote cooperation in
AIs that develop a reputation system where they observe and score each
others’ behaviors. AIs with higher reputation scores may be more likely
to receive assistance and cooperation from others. Still, this does not
guarantee that AIs will be cooperative: AIs might leverage the fear of
reputational harm to extort benefits from others, or themselves develop
ruthless reputations to inspire cooperation through fear.<p>
Third, we considered how group selection –– in a future where labor has
been automated such that AIs now run the majority of companies –– could
promote cooperation on a multi-agent scale. AIs may form corporate
coalitions with other AIs to protect their interests; AI groups with a
cooperative AI minority may be outcompeted by AI groups with a
cooperative AI majority. Under such conditions, however, AIs may learn
to favor in-group members and antagonize out-group members, in order to
maintain group solidarity. AIs may be more likely to see other AIs as
part of their group, and this could lead to conflict between AIs and
humans.<p>
Fourth, we explored the possibility that AIs may create variants of
themselves, and the forces of kin selection may drive these related
variants to cooperate with each other. However, this could also give
rise to nepotism, where AIs prioritize the interests of their variants
over other AIs and humans. As the differences between humans and AIs
increase, AIs may be increasingly less inclined to cooperate with
humans.<p>
Fifth, we discussed the mechanism of individual stakes to common stakes.
As an example, we explored the possibility that AIs may someday be
viewed as an out-group, an existential threat by humanity. This may
inspire humanity to set aside its differences, promoting global
solidarity as a means to address catastrophic threats posed by AI.<p>
Sixth, we broke down Simon’s selection mechanism, highlighting that to
overcome their bounded rationality, agents may be better off
establishing and relying on cooperative social channels. We extended
this idea to AI agents as well, supposing that in a multi-agent setting,
AIs may develop their own communication channels and protocols, which
may give them a fitness advantage over humans. Moreover, we also
considered the possibility that AIs might use their cooperative social
channels to build collective intelligences, which, being vastly superior
to human collective intelligences, would preclude humans from taking
part in and understanding the decisions AIs make within such
systems.<p>
Finally, we concluded with a discussion of how institutions can
incentivize cooperation through externally imposed incentives that
enforce cooperation and punish defection <span class="citation"
data-cites="buterin2022institution">[37]</span>. We related this concept
to the idea of an AI Leviathan used to counteract selfish, powerful AIs.
However, we also stressed that humans should take care to ensure their
relationship with the AI Leviathan is symbiotic and transparent,
otherwise humans risk losing control of AIs.<p>
In our discussion of these mechanisms, we not only illustrated their
prevalence in our world, but also showed how they might influence
cooperation with AI agents. In several cases, the mechanisms we discuss
could promote cooperation in AI. However, no single mechanism provides a
foolproof method for ensuring cooperation. In the following section, we
discuss the nature of conflict, namely the various factors that may give
rise to it. In doing so, we enhance our understanding of what might
motivate conflict in AI, and subsequently, our abilities to predict and
address AI-driven conflict scenarios.</p>

<br>
<br>
<h3>References</h3>
<div id="refs" class="references csl-bib-body" data-entry-spacing="0"
role="list">
<div id="ref-fearon1995rationalist" class="csl-entry" role="listitem">
<div class="csl-left-margin">[1] J.
D. Fearon, <span>“Rationalist explanations for war,”</span>
<em>International Organization</em>, vol. 49, no. 3, pp. 379–414, 1995,
Accessed: Oct. 14, 2023. [Online]. Available: <a
href="http://www.jstor.org/stable/2706903">http://www.jstor.org/stable/2706903</a></div>
</div>
<div id="ref-kelly2000inequality" class="csl-entry" role="listitem">
<div class="csl-left-margin">[2] M.
Kelly, <span>“Inequality and crime,”</span> <em>The Review of Economics
and Statistics</em>, vol. 82, pp. 530–539, Feb. 2000, doi: <a
href="https://doi.org/10.1162/003465300559028">10.1162/003465300559028</a>.</div>
</div>
<div id="ref-victimrates2011" class="csl-entry" role="listitem">
<div class="csl-left-margin">[3] </div><div
class="csl-right-inline"><span>“Victimization rates for persons age 12
or older, by type of crime and annual family income of victims.”</span>
US Bureau of Justice, 2011. Available: <a
href="https://bjs.ojp.gov/sites/g/files/xyckuh236/files/media/document/cv0814.pdf ">https://bjs.ojp.gov/sites/g/files/xyckuh236/files/media/document/cv0814.pdf
</a></div>
</div>
<div id="ref-fajnzylber2002inequality" class="csl-entry"
role="listitem">
<div class="csl-left-margin">[4] P.
Fajnzylber, D. Lederman, and N. Loayza, <span>“Inequality and violent
crime,”</span> <em>The Journal of Law and Economics</em>, vol. 45, no.
1, pp. 1–39, 2002, doi: <a
href="https://doi.org/10.1086/338347">10.1086/338347</a>.</div>
</div>
<div id="ref-horne2009effect" class="csl-entry" role="listitem">
<div class="csl-left-margin">[5] A.
Horne, <span>“The effect of relative deprivation on delinquency: An
assessment of juveniles.”</span> University of Central Florida,
2009.</div>
</div>
<div id="ref-trivers1971evolution" class="csl-entry" role="listitem">
<div class="csl-left-margin">[6] R.
Trivers, <span>“The evolution of reciprocal altruism,”</span>
<em>Quarterly Review of Biology</em>, vol. 46, pp. 35–57., Mar. 1971,
doi: <a href="https://doi.org/10.1086/406755">10.1086/406755</a>.</div>
</div>
<div id="ref-schino2007grooming" class="csl-entry" role="listitem">
<div class="csl-left-margin">[7] G.
Schino and F. Aureli, <span>“Grooming reciprocation among female
primates: A meta-analysis,”</span> <em>Biology letters</em>, vol. 4, pp.
9–11, Nov. 2007, doi: <a
href="https://doi.org/10.1098/rsbl.2007.0506">10.1098/rsbl.2007.0506</a>.</div>
</div>
<div id="ref-pinker2012better" class="csl-entry" role="listitem">
<div class="csl-left-margin">[8] S.
Pinker, <em>The better angels of our nature: Why violence has
declined</em>. Penguin Books, 2012.</div>
</div>
<div id="ref-mearsheimer2007structural" class="csl-entry"
role="listitem">
<div class="csl-left-margin">[9] J.
J. Mearsheimer, <em>Structural realism</em>. 2007, pp. 77–94.</div>
</div>
<div id="ref-nowak1998evolution" class="csl-entry" role="listitem">
<div class="csl-left-margin">[10] M.
A. Nowak and K. Sigmund, <span>“Evolution of indirect reciprocity by
image scoring,”</span> <em>Nature</em>, vol. 393, no. 6685, pp. 573–577,
1998.</div>
</div>
<div id="ref-bshary2006image" class="csl-entry" role="listitem">
<div class="csl-left-margin">[11] R.
Bshary and A. S. Grutter, <span>“Image scoring and cooperation in a
cleaner fish mutualism,”</span> <em>Nature</em>, vol. 441, pp. 975–978,
2006, Available: <a
href="https://api.semanticscholar.org/CorpusID:4422277">https://api.semanticscholar.org/CorpusID:4422277</a></div>
</div>
<div id="ref-carter2013food" class="csl-entry" role="listitem">
<div class="csl-left-margin">[12] G.
Carter and G. Wilkinson, <span>“Food sharing in vampire bats: Reciprocal
help predicts donations more than relatedness or harassment,”</span>
<em>Proceedings. Biological sciences / The Royal Society</em>, vol. 280,
p. 20122573, Feb. 2013, doi: <a
href="https://doi.org/10.1098/rspb.2012.2573">10.1098/rspb.2012.2573</a>.</div>
</div>
<div id="ref-balliet2020indirect" class="csl-entry" role="listitem">
<div class="csl-left-margin">[13] D.
Balliet, J. Wu, and P. Lange, <span>“Indirect reciprocity, gossip, and
reputation-based cooperation,”</span> 2020, pp. 265–287.</div>
</div>
<div id="ref-west2007social" class="csl-entry" role="listitem">
<div class="csl-left-margin">[14] S.
A. WEST, A. S. GRIFFIN, and A. GARDNER, <span>“Social semantics:
Altruism, cooperation, mutualism, strong reciprocity and group
selection,”</span> <em>Journal of Evolutionary Biology</em>, vol. 20,
no. 2, pp. 415–432, 2007, doi: <a
href="https://doi.org/10.1111/j.1420-9101.2006.01258.x">https://doi.org/10.1111/j.1420-9101.2006.01258.x</a>.</div>
</div>
<div id="ref-hamilton1964genetical" class="csl-entry" role="listitem">
<div class="csl-left-margin">[15] W.
D. Hamilton, <span>“The genetical evolution of social behaviour.
i,”</span> <em>Journal of theoretical biology</em>, 1964.</div>
</div>
<div id="ref-nowak2011supercooperators" class="csl-entry"
role="listitem">
<div class="csl-left-margin">[16] M.
A. Nowak, R. Highfield, <em>et al.</em>, <em>Supercooperators</em>.
Canongate Edinburgh, 2011.</div>
</div>
<div id="ref-okasha2012social" class="csl-entry" role="listitem">
<div class="csl-left-margin">[17] S.
Okasha, <span>“Social justice, genomic justice and the veil of
ignorance: Harsanyi meets mendel,”</span> <em>Economics &amp;amp;
Philosophy</em>, vol. 28, no. 1, pp. 43–71, 2012, doi: <a
href="https://doi.org/10.1017/S0266267112000119">10.1017/S0266267112000119</a>.</div>
</div>
<div id="ref-Harsanyi1955CardinalWI" class="csl-entry" role="listitem">
<div class="csl-left-margin">[18] J.
C. Harsanyi, <span>“Cardinal welfare, individualistic ethics, and
interpersonal comparisons of utility,”</span> <em>Journal of Political
Economy</em>, 1955.</div>
</div>
<div id="ref-Dennett1995DarwinsDI" class="csl-entry" role="listitem">
<div class="csl-left-margin">[19] D.
C. Dennett, <span>“Darwin’s dangerous idea: Evolution and the meanings
of life,”</span> 1995.</div>
</div>
<div id="ref-simon1990mechanism" class="csl-entry" role="listitem">
<div class="csl-left-margin">[20] H.
A. Simon, <span>“A mechanism for social selection and successful
altruism,”</span> <em>Science</em>, vol. 250, no. 4988, pp. 1665–1668,
1990, doi: <a
href="https://doi.org/10.1126/science.2270480">10.1126/science.2270480</a>.</div>
</div>
<div id="ref-curry2016morality" class="csl-entry" role="listitem">
<div class="csl-left-margin">[21] O.
S. Curry, <span>“Morality as cooperation: A problem-centred
approach,”</span> <em>The evolution of morality</em>, pp. 27–51,
2016.</div>
</div>
<div id="ref-charlesworth2009genetics" class="csl-entry"
role="listitem">
<div class="csl-left-margin">[22] D.
Charlesworth and J. H. Willis, <span>“The genetics of inbreeding
depression,”</span> <em>Nature reviews genetics</em>, vol. 10, no. 11,
pp. 783–796, 2009.</div>
</div>
<div id="ref-chagnon1979kin" class="csl-entry" role="listitem">
<div class="csl-left-margin">[23] N.
A. Chagnon, <span>“Kin selection and conflict: An analysis of a
yanomam<span>ö</span> ax fight,”</span> <em>Evolutionary biology and
human social behavior: An anthropological perspective</em>, pp. 213–238,
1979.</div>
</div>
<div id="ref-westermarck2022origin" class="csl-entry" role="listitem">
<div class="csl-left-margin">[24] E.
Westermarck, <em>The origin and development of the moral ideas</em>.
DigiCat, 2022.</div>
</div>
<div id="ref-smith1987inheritance" class="csl-entry" role="listitem">
<div class="csl-left-margin">[25] M.
S. Smith, B. J. Kish, and C. B. Crawford, <span>“Inheritance of wealth
as human kin investment,”</span> <em>Ethology and Sociobiology</em>,
vol. 8, no. 3, pp. 171–182, 1987.</div>
</div>
<div id="ref-oates2002nominal" class="csl-entry" role="listitem">
<div class="csl-left-margin">[26] K.
Oates and M. Wilson, <span>“Nominal kinship cues facilitate
altruism,”</span> <em>Proceedings of the Royal Society of London. Series
B: Biological Sciences</em>, vol. 269, no. 1487, pp. 105–109,
2002.</div>
</div>
<div id="ref-lieberman2003does" class="csl-entry" role="listitem">
<div class="csl-left-margin">[27] D.
Lieberman, J. Tooby, and L. Cosmides, <span>“Does morality have a
biological basis? An empirical test of the factors governing moral
sentiments relating to incest,”</span> <em>Proceedings of the Royal
Society of London. Series B: Biological Sciences</em>, vol. 270, no.
1517, pp. 819–826, 2003.</div>
</div>
<div id="ref-thornhill1991evolutionary" class="csl-entry"
role="listitem">
<div class="csl-left-margin">[28] N.
W. Thornhill, <span>“An evolutionary analysis of rules regulating human
inbreeding and marriage,”</span> <em>Behavioral and Brain Sciences</em>,
vol. 14, no. 2, pp. 247–261, 1991.</div>
</div>
<div id="ref-connor1995benefits" class="csl-entry" role="listitem">
<div class="csl-left-margin">[29] R.
C. Connor, <span>“The benefits of mutualism: A conceptual
framework,”</span> <em>Biological Reviews</em>, vol. 70, no. 3, pp.
427–457, 1995.</div>
</div>
<div id="ref-van2008leadership" class="csl-entry" role="listitem">
<div class="csl-left-margin">[30] M.
Van Vugt, R. Hogan, and R. B. Kaiser, <span>“Leadership, followership,
and evolution: Some lessons from the past.”</span> <em>American
psychologist</em>, vol. 63, no. 3, p. 182, 2008.</div>
</div>
<div id="ref-carruthers1996theories" class="csl-entry" role="listitem">
<div class="csl-left-margin">[31] P.
Carruthers and P. K. Smith, <em>Theories of theories of mind</em>.
Cambridge university press, 1996.</div>
</div>
<div id="ref-west2007evolutionary" class="csl-entry" role="listitem">
<div class="csl-left-margin">[32] S.
A. West, A. S. Griffin, and A. Gardner, <span>“Evolutionary explanations
for cooperation,”</span> <em>Current biology</em>, vol. 17, no. 16, pp.
R661–R672, 2007.</div>
</div>
<div id="ref-gintis2007evolution" class="csl-entry" role="listitem">
<div class="csl-left-margin">[33] H.
Gintis, <span>“The evolution of private property,”</span> <em>Journal of
Economic Behavior &amp; Organization</em>, vol. 64, no. 1, pp. 1–16,
2007.</div>
</div>
<div id="ref-henrich2005economic" class="csl-entry" role="listitem">
<div class="csl-left-margin">[34] J.
Henrich <em>et al.</em>, <span>“<span>‘Economic man’</span> in
cross-cultural perspective: Behavioral experiments in 15 small-scale
societies,”</span> <em>Behavioral and brain sciences</em>, vol. 28, no.
6, pp. 795–815, 2005.</div>
</div>
<div id="ref-brams1996fair" class="csl-entry" role="listitem">
<div class="csl-left-margin">[35] S.
J. Brams and A. D. Taylor, <em>Fair division: From cake-cutting to
dispute resolution</em>. Cambridge University Press, 1996.</div>
</div>
<div id="ref-herskovits1952economic" class="csl-entry" role="listitem">
<div class="csl-left-margin">[36] M.
J. Herskovits, <em>Economic anthropology: A study in comparative
economics</em>. Knopf New York, 1952.</div>
</div>
<div id="ref-buterin2022institution" class="csl-entry" role="listitem">
<div class="csl-left-margin">[37] V.
Buterin, <span>“What even is an institution?”</span> [Online].
Available: <a
href="https://vitalik.ca/general/2022/12/30/institutions.html">https://vitalik.ca/general/2022/12/30/institutions.html</a></div>
</div>
</div>