Interaction Evolved-Learned Cooperation

A Two-Timescale Theory of Cooperation

Cooperation, like adaptive behavior generally, can emerge through the interaction of two different adaptive processes — the same nature/nurture distinction long used in biology and psychology, applied here in a specific, simulatable form:

• Nurture — within-lifetime learning (behavioral plasticity, reinforcement learning) is the specific mechanism this framework operationalizes and simulates. "Nurture" itself is broader than learning — it covers any non-genetic developmental influence (nutrition, epigenetics, culture, parental care) — but learning is the one instance modeled directly here.
• Nature — selection across generations (evolutionary dynamics).

These processes operate on entirely different timescales:

Learning timescale <<< Evolutionary timescale

In natural systems they interact. The two-timescale simulation family in this section provides a controlled framework where both processes can be analyzed together.

Confirmed: nature and nurture combined beat either alone. A from-scratch replication of Ackley & Littman (1991) — the original computational demonstration that evolution and learning combined can outperform either alone — reproduced that result with real statistical power: combined nature-and-nurture significantly beat evolution alone, learning alone, no adaptation, and pure random behavior, p < 0.00001 against all four (Mann-Whitney U, n = 100 seeds per condition).

That result concerns general fitness-enhancing survival behavior in a predator-prey world, not cooperation specifically — it doesn't itself demonstrate cooperation emerging — but it is direct evidence for the core two-timescale mechanism this whole framework depends on. Full detail in the Darwin/Baldwin Trial Log.

Nature and Nurture Combined Beat Either Alone

Per-seed survival time across all 500 runs of the comparative study — ERL (evolution + learning combined), L (learning alone), E (evolution alone), F (neither — no learning, no evolutionary variation), B (luck alone, random behavior); n=100 seeds each.

Display 1: Every one of the 500 runs behind that result, not just its medians — nature and nurture combined (ERL) piling up at the step ceiling, versus either alone (L, E), neither (F), or pure luck (B) spread across a much shorter, longer-tailed distribution.

A second, independent replication: correlated traits relax the rules. A from-scratch replication of two models from Prosser (2022), a University of Southampton PhD thesis, tested what happens when learning acts on genetically correlated groups of traits instead of single traits in isolation. The classical conditions long thought necessary for genetic assimilation — a cost of learning, a stable learnt target — turn out not to be required once learning and evolution share correlated structure.

In a multi-peaked fitness landscape, evolution with this kind of learning reached the exact global optimum and held it; evolution alone stayed well below optimum and never stabilized over the same run. The thesis itself establishes this reliably across 30 seeds — the replication here confirms the effect reproduces, not that reliability claim specifically. Code, equations, and validation runs are open in the Prosser2022 repository.

---

Fast and Slow Dynamics

Fast timescale — learning

Agents update their policy during their lifetime to increase expected reward.

$$\pi_{t+1} \leftarrow \pi_t + \alpha \nabla_\pi \mathbb{E}[R]$$

Where:

• $s$ = social state (partner history, local interaction context),
• $a$ = action (cooperate or defect),
• $R$ = interaction reward/payoff,
• $\alpha$ = learning rate.

In this simulation family, reward is defined by donation-game payoffs and partner-specific interaction outcomes.

---

Slow timescale — evolution

Population composition changes across generations:

frequency_next = (fitness / mean_fitness) * frequency

Fitness is the payoff accumulated over a lifetime of interactions under the learned policy:

$$\text{fitness}(\pi^*) = \sum_{t=1}^{T} R_t\!\left(s_t,\, \pi^*(s_t)\right)$$

Where $\pi^*$ is the policy the agent has learned by the end of its lifetime and $T$ is the number of interactions per generation. Agents that learned to cooperate with reliable partners and defect against exploiters accumulate higher payoffs and therefore reproduce more.

Evolution therefore selects based on learning outcomes.

---

Baldwinian and Lamarckian Interaction

The key distinction is what crosses the generational boundary.

In a Baldwinian interaction, agents learn useful behavior during life, but the learned behavior itself is not inherited. Selection instead favors inherited traits that make the useful behavior easier, faster, or cheaper to learn. Learning affects fitness; fitness changes which learning-supporting traits persist. See The Baldwin Effect for the full history, mechanism, and computational demonstrations behind this idea.

In a Lamarckian interaction, acquired characteristics are inherited directly. In biological evolution this means the learned or acquired trait itself is passed to offspring. In artificial learning systems, a Lamarckian analogue appears when a trained policy, learned neural-network weights, or acquired hyperparameter state is copied into descendants or replacement populations.

Display 1: Lamarckian inheritance passes acquired change directly, while the Baldwin effect lets learned behavior guide selection without direct inheritance of that learned behavior. Image by Ian Alexander, CC BY-SA 4.0.

For the simulation family on this page, the intended mechanism is mainly Baldwinian: within-lifetime learning affects reproductive success, but descendants inherit learning-relevant settings rather than the final learned policy.

In artificial systems, the same distinction depends on whether descendants receive only inherited settings that affect learnability, or whether they directly receive the acquired policy state produced by learning.

Mechanism	What is learned during life	What is inherited	Project interpretation
Baldwinian	Behavior or policy improves through experience.	Parameters that support learning, not the acquired behavior itself.	Main framing for this two-timescale cooperation page.
Lamarckian	Behavior, policy weights, or acquired parameter state changes through experience.	The acquired state itself is copied into descendants or replacement populations.	Artificial analogue when trained policies, learned weights, or checkpoint state are copied forward.
Darwinian selection without learning	No within-lifetime policy learning is required.	Heritable traits vary and are filtered by reproductive success.	Useful baseline, but too limited for learned cooperation.

Display 2: The distinction depends on what is transmitted across generations.

---

The Baldwin Effect

The Baldwin effect describes how learning changes evolutionary trajectories without requiring inheritance of learned behavior: agents that can learn a beneficial behavior survive long enough to reproduce even before their genes encode that behavior directly, and over generations genetic variants that make the behavior easier to learn accumulate. The mechanism is Darwinian throughout — no acquired traits are inherited. See The Baldwin Effect for the full history, the step-by-step mechanism, the Hinton and Nowlan (1987) computational demonstration and its extensions, and common misconceptions.

Applied to cooperation specifically:

• Without learning, a genotype predisposed to cooperate has low fitness unless partners are also cooperative, which is rare in a defector-dominated population — cooperative genotypes are eliminated before they can spread. This is the needle-in-a-haystack landscape Hinton and Nowlan describe.
• With learning, an agent with even a weak cooperative predisposition can learn to discriminate — cooperating with cooperators, withholding from defectors — and accumulate net positive payoff even in a mixed population. Learning rescues cooperative genotypes that selection alone would eliminate, turning the fitness cliff into a slope evolution can climb.

This is distinct from Waddington's genetic assimilation, where a trait becomes fully encoded and developmentally canalized — the Baldwin effect produces facilitation of learning rather than necessarily replacing it.

Learning creates new selection pressures

A learned cooperative strategy, once widespread in the population, creates selection pressure for genetic variants that achieve the same behavior at lower cost:

• lower learning rates suffice when the behavior is already partially encoded
• initial cooperation biases can be set more aggressively when the social environment has become reliably cooperative
• discrimination thresholds can loosen as defectors become rarer

This feedback loop — learning expands what is reachable; evolution consolidates what learning discovered — is the core dynamic this simulation family is designed to capture.

---

Interaction Regimes

Learning and evolution can interact in different ways:

1. Learning accelerates evolution
   Plasticity enables rapid discovery of cooperation that selection stabilizes.

2. Learning masks selection
   If all agents learn equally well, fitness differences shrink.

3. Learning opposes evolution
   Short-term learned defection may increase individual reward but reduce population fitness.

4. Coevolution of learning ability
   Selection may favor faster or more robust learners.

---

Manifestation in the Simulation Suite

In the integrated two-timescale cooperation simulations:

• interactions are local by default (ring structure)
• agents update behavior within generation (trust or Q-values)
• agents reproduce between generations based on accumulated payoff

This creates a Baldwin-style pathway:

1. Agents learn partner-contingent cooperation during life
2. Learners with better long-run payoff leave more offspring
3. Offspring inherit parameter settings that make successful learning more likely, not the completed learned policy

Cooperation shifts from:

context-dependent learning alone -> learning supported by evolved predispositions

This differs from a Lamarckian artificial-learning setup, where a trained policy or acquired policy weights are copied directly into later agents or replacement populations.

---

What Can Evolve

Selection can act on:

• trust predispositions (trust_prior)
• social responsiveness to experience
• reinforcement-learning parameters (alpha, epsilon, gamma, bias)
• social-cognitive parameters (reputation weighting, rejection threshold, forgiveness)

This leads to cooperation-friendly learning phenotypes rather than fixed cooperative strategies.

---

Testable Predictions

The two-timescale framework generates testable predictions:

• Populations with repeated interaction should evolve higher cooperation than one-shot regimes.
• Selection should favor parameter combinations that improve partner discrimination.
• In stranger-rich environments, reputation-mediated mechanisms should outperform pure partner-memory mechanisms.
• Different learning rules (trust update vs Q-learning) should produce different cooperation-payoff trade-offs.

All of these are testable within the integrated model family.

---

Relation to Classical Theories

Classical evolutionary models:

• fixed strategies
• cooperation via selection only

Pure reinforcement learning models:

• cooperation within lifetimes
• no generational dynamics

This framework unifies both:

Cooperation = f(learning dynamics, evolutionary dynamics)

---

Related Work (Closest by Axis)

No single landmark paper fully matches this integrated setup across all dimensions (reciprocal cooperation, local interaction structure, reinforcement learning, and between-generation selection over learning parameters). The most relevant work falls into three overlapping groups: reciprocal altruism theory, network reciprocity theory, and modern multi-agent learning studies of social dilemmas.

Within this project, Model 1 maps most directly to direct reciprocity and network reciprocity theory, Model 2 maps most directly to learned reciprocity in multi-agent reinforcement learning, and Model 3 extends that line toward reputation, partner choice, and socially mediated cooperation with strangers.

Work	Closest axis to this simulation family	Main gap vs this simulation family
Trivers (1971), The Evolution of Reciprocal Altruism	Foundational logic of repeated reciprocal cooperation	Verbal evolutionary theory rather than an explicit learning-plus-selection simulation
Axelrod and Hamilton (1981), The Evolution of Cooperation	Repeated-interaction conditions under which reciprocity can stabilize cooperation	Strategy tournament framework rather than agents that learn within life and evolve between generations
Nowak (2006), Five Rules for the Evolution of Cooperation	Direct reciprocity, indirect reciprocity, and network reciprocity as a unifying framework	Analytic synthesis rather than a concrete parameter-evolution simulation
Ohtsuki et al. (2006), A Simple Rule for the Evolution of Cooperation on Graphs and Social Networks	Why local interaction structure can protect cooperation	Graph-theoretic selection result rather than within-lifetime partner learning
Claus and Boutilier (1998), The Dynamics of Reinforcement Learning in Cooperative Multiagent Systems	Foundational emergence of cooperation in multi-agent reinforcement learning	Small, abstract cooperative games with no explicit generational inheritance of learning traits
Eccles et al. (2019), Learning Reciprocity in Complex Sequential Social Dilemmas	Reciprocity under temporal and social complexity in learned agents	No explicit reproduction-selection loop over inherited social-learning parameters

Display 3: Related work positioned by proximity to this simulation family along key conceptual axes.

Taken together, these works capture the core logic behind the present simulation family: reciprocal altruism, repeated interaction, local network structure, and partner-contingent learning. The distinctive contribution here is that these ingredients are combined in a single two-timescale setup where learning unfolds within life and learning parameters themselves evolve across generations.

Adjacent computational environments

Some broader MARL and artificial-society papers remain relevant as neighboring context, but they are no longer the primary fit for this page's argument:

• Leibo et al. (2017), Multi-agent Reinforcement Learning in Sequential Social Dilemmas for sequential social-dilemma structure
• Leibo et al. (2018), Malthusian Reinforcement Learning for ecology-linked population dynamics
• Zheng et al. (2018), MAgent and Suarez et al. (2019), Neural MMO for large-scale multi-agent environments
• Leibo et al. (2021), Melting Pot for broad evaluation of social behaviors

These works are best understood here as adjacent environment or benchmark context, not as the closest direct precedents for the trust-learning, Q-learning, and extended reciprocity models documented in this section.

---

Simulation Companion

The concrete two-timescale experiments documented for this site are available in two companion tracks, one abstract and one ecological:

• Two-Timescale Simulations — abstract ring-network models (trust learning, Q-learning, extended social mechanisms), the network-diversity experiment, and focused appendices. This track confirms the theory above cleanly: see its own theory-simulation correspondence table.
• Darwin/Baldwin Trial Log — the same two-timescale question asked inside a full predator-prey ecology rather than an abstract donation game. The first ten trials ran on this project's own PredPreyGrass ecology and came back null; the trial that finally produced a confirmed positive result traded that ecology for a from-scratch rebuild of Ackley & Littman's own World AL, alongside switching from a shared-policy to a per-agent architecture — read there for why both changes were needed.

---

Summary

The interaction between learning and selection:

• couples fast behavioral adaptation with slow population change
• enables the Baldwin effect
• allows plasticity to guide evolution
• explains how cooperation emerges, stabilizes, or collapses

This interaction forms the core mechanism linking nurture and nature in the integrated two-timescale cooperation simulations.

---

References

• Hinton, G.E. & Nowlan, S.J. (1987). How Learning Can Guide Evolution. Complex Systems, 1(3), 495–502.
• Ackley, D.H. & Littman, M.L. (1991). Interactions Between Learning and Evolution. In Artificial Life II, 487–509. Addison-Wesley.
• Prosser, D. (2022). The Interaction Between Lifetime Learning and Evolution. PhD thesis, University of Southampton.
• Waddington, C.H. (1953). Genetic Assimilation of an Acquired Character. Evolution, 7(2), 118–126.
• Alexander, I. (2018). Lamarck Compared to Darwin, Baldwin, Waddington.svg. Wikimedia Commons. CC BY-SA 4.0.

For the primary sources on Baldwin's original proposal, its history, and later computational demonstrations, see the references on The Baldwin Effect.