# Network Coordination

[← Back to Building Amendments: Lifecycle and Core Protocol Impact](/core-dev-bootcamp/module10.md)

***

### Introduction

Activating an amendment on XRPL is not an isolated event occurring on a single node. It is a complex coordination process involving hundreds of geographically distributed nodes, dozens of independent validators, and numerous operators who must synchronize their actions.

This coordination is essential to ensure the network remains consistent and functional during the transition. Failed coordination could lead to network partition, incompatible ledgers, or temporary loss of consensus.

In this section, we explore the coordination mechanisms, communication strategies, deployment plans, and contingency protocols that allow XRPL to activate amendments safely and in a coordinated manner.

***

## Communication via the Overlay Network

### Validation Propagation

The primary coordination mechanism for amendments is the propagation of **validations** across the overlay network.

**Network structure**:

```
┌─────────┐         ┌─────────┐         ┌─────────┐
│Validator│◄───────►│  Node   │◄───────►│Validator│
│    A    │         │  Relay  │         │    B    │
└─────────┘         └─────────┘         └─────────┘
     │                   │                    │
     │                   │                    │
     ▼                   ▼                    ▼
┌─────────┐         ┌─────────┐         ┌─────────┐
│  Node   │         │Validator│         │  Node   │
│   API   │         │    C    │         │  Full   │
└─────────┘         └─────────┘         └─────────┘
```

**Propagation flow**:

1. A validator publishes a validation containing its list of supported amendments
2. The validation is broadcast to all connected peers
3. Each peer propagates the validation to its own peers (flood protocol)
4. Within seconds, all nodes in the network have received the validation
5. Each node extracts the amendment list and updates its counters

### Validation Broadcast Protocol

Validations are broadcast via XRPL's Peer Protocol:

**TMValidation Message**:

```protobuf
message TMValidation {
    required bytes validation = 1;    // Serialized and signed validation
    optional bytes cookie = 2;        // Tracking cookie
}
```

**Processing by receiving node**:

```cpp
void PeerImp::onValidation(
    std::shared_ptr<protocol::TMValidation> const& m)
{
    // Deserialize the validation
    auto validation = std::make_shared<STValidation>(
        SerialIter{m->validation().data(), m->validation().size()},
        [](PublicKey const& pk) { return calcNodeID(pk); });

    // Verify signature
    if (!validation->isValid())
        return;

    // Extract amendments
    auto const amendments = validation->getAmendments();

    // Pass to validation handler
    app_.getValidations().addValidation(validation);

    // Relay to other peers (except sender)
    overlay_.relay(m, validation->getNodeID());
}
```

### Squelching and Optimization

To avoid message flooding, the overlay network uses a **squelching** mechanism:

**Principle**: A node only propagates a message if it hasn't seen it yet

```cpp
bool Overlay::relay(
    std::shared_ptr<Message> const& m,
    NodeID const& source)
{
    // Calculate message hash
    auto const hash = sha512Half(m->SerializeAsString());

    std::lock_guard lock(squelchMutex_);

    // Check if already seen
    if (squelchedMessages_.count(hash))
        return false;  // Don't propagate

    // Mark as seen
    squelchedMessages_.insert(hash);

    // Propagate to all peers except source
    for (auto const& peer : peers_) {
        if (peer->getNodeID() != source)
            peer->send(m);
    }

    return true;
}
```

**Entry expiration**: Squelching entries expire after a few minutes to free memory.

***

## Deployment Strategies

### Phase 1: Pre-Release Communication

**6-8 weeks before release**: Public announcement of the proposed amendment

* Publication of an XLS (XRPL Standard) detailing the amendment
* Public code review on GitHub
* Discussions on community forums
* Technical webinars and presentations

**Communication channels**:

* XRPL.org blog
* GitHub (XLS repository)
* XRPL Dev Forum
* Discord XRPL Developers
* Official Twitter/X

### Phase 2: Code Release

**Objective**: Distribute code before voting begins

**Typical timeline**:

* T+0: Release of rippled with the new amendment (e.g., 1.12.0)
* T+1 week: First non-validator nodes update
* T+2 weeks: \~30% of nodes updated
* T+4 weeks: \~60% of nodes updated

**Progressive deployment strategy**:

```
┌──────────────────────────────────────────────────────────┐
│ Week 1: Testnet and internal nodes                       │
│ ✓ Validation on Testnet/Devnet                          │
│ ✓ Internal test nodes                                   │
└──────────────────────────────────────────────────────────┘
                        ▼
┌──────────────────────────────────────────────────────────┐
│ Week 2-3: API and non-validator nodes                   │
│ ✓ Fullhistory nodes                                     │
│ ✓ Public API nodes                                      │
│ ✓ Exchanges and wallets (infrastructure)                │
└──────────────────────────────────────────────────────────┘
                        ▼
┌──────────────────────────────────────────────────────────┐
│ Week 4-6: Validators                                     │
│ ✓ Validators begin voting                               │
│ ✓ Intensive monitoring                                  │
└──────────────────────────────────────────────────────────┘
```

### Phase 3: Vote Activation

**Communication to validators**:

Validators receive notifications to activate their vote:

```bash
# Email/notification to validator operators
Subject: Ready to vote for Subscriptions amendment

The Subscriptions amendment (XLS-0078) is ready for voting.
Release: rippled 1.12.0
Amendment hash: 7B73B9E8D8E6E8E8A8B8C8D8E8F8A8B8C8D8E8F8A8B8C8D8E8F8A8B8C8D8E8F8

To activate your vote:
rippled feature 7B73B9E8D8E6E8E8A8B8C8D8E8F8A8B8C8D8E8F8A8B8C8D8E8F8A8B8C8D8E8F8 accept

Or add to rippled.cfg:
[amendments]
7B73B9E8D8E6E8E8A8B8C8D8E8F8A8B8C8D8E8F8A8B8C8D8E8F8A8B8C8D8E8F8
```

**Informal coordination**: Validators communicate via:

* Private validator mailing list
* Private Slack/Discord channels
* Regular coordination meetings

### Phase 4: Majority Period

**2 weeks of final preparation**:

Once the 80% threshold is exceeded, node operators receive alerts:

```
[WARNING] Amendment 7B73B9E8... (Subscriptions) has reached majority.
It will activate in approximately 14 days (on 2025-05-04 14:32:15 UTC).

If your node does not support this amendment, please upgrade to rippled 1.12.0 or later immediately.

Current version: 1.11.0
Amendment support: NO
Upgrade urgency: CRITICAL
```

**Required actions**:

* All non-updated nodes must upgrade
* Exchanges and wallets must finalize their integrations
* Final tests on staging environments

### Phase 5: Activation

**Moment of truth**: The activation day

* 24/7 monitoring by technical teams
* Emergency communication channels open
* Ready to coordinate response if issue detected

***

## Managing Non-Updated Nodes

### Early Detection

Nodes can detect that they will soon be blocked:

```cpp
void NetworkOPsImp::checkAmendmentStatus() {
    auto const expected = app_.getAmendmentTable().firstUnsupportedExpected();

    if (expected) {
        auto const now = app_.timeKeeper().closeTime();
        auto const timeUntil = *expected - now;

        if (timeUntil < 24h) {
            // CRITICAL: Less than 24h
            JLOG(j_.fatal())
                << "CRITICAL: Unsupported amendment will activate in "
                << timeUntil.count() << " seconds. UPGRADE NOW!";

            // Send alert to operator
            sendOperatorAlert("CRITICAL_AMENDMENT_WARNING");
        }
        else if (timeUntil < 7days) {
            // ERROR: Less than a week
            JLOG(j_.error())
                << "ERROR: Unsupported amendment will activate in "
                << std::chrono::duration_cast<std::chrono::days>(timeUntil).count()
                << " days. Please upgrade soon.";
        }
        else if (timeUntil < 14days) {
            // WARNING: In majority window
            JLOG(j_.warn())
                << "WARNING: Unsupported amendment will activate in "
                << std::chrono::duration_cast<std::chrono::days>(timeUntil).count()
                << " days.";
        }
    }
}
```

### Emergency Migration Strategies

**For validators**:

If a validator doesn't have time to update before activation:

```cpp
// Option 1: Withdraw vote to delay activation
rippled feature 7B73B9E8... reject

// Option 2: Temporarily withdraw from UNL
// (requires coordination with other validators)
```

**For API nodes/Exchanges**:

* **Plan A**: Quick upgrade (few hours of maintenance)
* **Plan B**: Route traffic to already-updated backup nodes
* **Plan C**: Degraded mode (read-only via third-party nodes)

### Crisis Communication

In case of major problem detected during majority period:

**Channel #xrpl-validators (private)**:

```
[URGENT] Issue detected with Subscriptions amendment.
Potential impact: [description]
Recommendation: All validators please REJECT the amendment immediately.

Command:
rippled feature 7B73B9E8... reject
```

**Quick coordination**: Validators can withdraw their votes to drop support below 80% and avoid automatic activation.

***

## Contingency Plans

### Scenario 1: Critical Bug Discovered After Activation

**Cannot disable an amendment**: There is no rollback mechanism.

**Response strategies**:

1. **Emergency patch**: Develop and deploy a fix quickly

```cpp
// In doApply(), add temporary guard
TER RecurringPaymentClaim::doApply() {
    // EMERGENCY FIX: Disable temporarily until proper fix
    if (view().info().seq < EMERGENCY_DISABLE_LEDGER) {
        return tecDISABLED;
    }

    // Normal logic...
}
```

2. **Corrective amendment**: Develop an amendment that fixes the bug

```cpp
XRPL_FIX(fixSubscriptionsBug, Supported::yes, VoteBehavior::DefaultYes)
```

3. **Out-of-band coordination**: If the bug is catastrophic, manual coordination for an emergency hard fork (extreme scenario never used on XRPL)

### Scenario 2: Network Partition During Activation

**Symptom**: Two parts of the network activate the amendment at different times or with different results.

**Detection**:

```cpp
// Monitoring for forks
void LedgerMaster::checkForFork() {
    auto const ourLedger = getValidatedLedger();
    auto const peerLedgers = getPeerLedgerHashes();

    int matchingCount = 0;
    for (auto const& peerHash : peerLedgers) {
        if (peerHash == ourLedger->getHash())
            matchingCount++;
    }

    if (matchingCount < peerLedgers.size() * 0.5) {
        JLOG(j_.fatal())
            << "Potential network fork detected! "
            << "Our ledger does not match majority of peers.";
    }
}
```

**Resolution**:

* Normal consensus will resolve the partition
* The minority chain will abandon its progress and adopt the majority chain
* Unconfirmed transactions will need to be resubmitted

### Scenario 3: Validators Divided on Vote

**Symptom**: Support oscillates around 80%, amendment gains and loses majority multiple times.

**Impact**:

* Multiple `tfGotMajority` and `tfLostMajority` pseudo-transactions
* Confusion for node operators
* Delay in final activation

**Prevention**:

* Clear validator communication before activating votes
* Informal consensus reached before formal vote
* Avoid "surprise" votes

***

## Monitoring and Observability

### Key Metrics

Operators monitor several metrics during the activation process:

**1. Vote Count**: Number of validators voting for the amendment

```bash
# Regular query
watch -n 60 'rippled feature 3B95AC15... | jq ".result.count"'
```

**2. Network Consensus**: Consensus percentage achieved

```bash
rippled server_info | jq ".info.server_state"
# States: proposing, validating, full, connected
```

**3. Peer Count**: Number of connected peers

```bash
rippled peers | jq ".result.peers | length"
```

**4. Ledger Progress**: Verify ledgers are progressing normally

```bash
watch -n 10 'rippled server_info | jq ".info.validated_ledger.seq"'
```

### Monitoring Dashboards

**XRPL Metrics Dashboard** (conceptual example):

```
┌────────────────────────────────────────────────────────────┐
│ Amendment Activation Status: Subscriptions                 │
├────────────────────────────────────────────────────────────┤
│ Vote Count:    29/35  (82.9%)                              │
│ Threshold:     28                                          │
│ Status:        MAJORITY                                    │
│ Majority Since: 2025-04-20 14:32:15 UTC                   │
│ Activation ETA: 2025-05-04 14:32:15 UTC (13d 23h 45m)     │
├────────────────────────────────────────────────────────────┤
│ Network Health                                             │
│ Consensus:     ✓ 99.8%                                     │
│ Validators:    ✓ 34/35 online                             │
│ Avg Ledger Time: 3.8s                                      │
│ Fork Risk:     ✓ LOW                                       │
└────────────────────────────────────────────────────────────┘
```

### Alerting

Automatic alert configuration:

```yaml
# alerts.yml
alerts:
  - name: amendment_majority_reached
    condition: amendment.status == "MAJORITY"
    action: notify_operators
    channels: [email, slack, pagerduty]

  - name: amendment_activation_imminent
    condition: amendment.activation_eta < "24h"
    action: critical_alert
    channels: [pagerduty, sms]

  - name: unsupported_amendment_warning
    condition: node.unsupported_amendment_active == true
    action: block_node
    channels: [email, slack, pagerduty]
```

***

## Ecosystem Coordination

### Exchanges and Services

Exchanges receive notifications and must coordinate:

**Typical timeline for an exchange**:

```
T-6 weeks: Amendment announcement
  ├─ Review API impact
  └─ Development planning

T-4 weeks: Development begins
  ├─ Update libraries
  ├─ Adapt UI
  └─ Tests on Testnet

T-2 weeks: Majority reached
  ├─ Final tests on staging
  ├─ Validate integrations
  └─ Maintenance planning

T-0: Activation
  ├─ 24/7 monitoring
  ├─ Enhanced customer support
  └─ Rollback plan ready
```

### Application Developers

Client libraries must be updated before activation:

**xrpl.js**:

```javascript
// Version 2.12.0 (with Subscriptions)
export const TRANSACTION_TYPES = {
  // ... existing types
  SubscriptionSet: 'SubscriptionSet',
  SubscriptionClaim: 'SubscriptionClaim',
  SubscriptionCancel: 'SubscriptionCancel',
};
```

**xrpl-py**:

```python
# Version 2.8.0
class TransactionType(str, Enum):
    # ... existing types
    SUBSCRIPTION_SET = "SubscriptionSet"
    SUBSCRIPTION_CLAIM = "SubscriptionClaim"
    SUBSCRIPTION_CANCEL = "SubscriptionCancel"
```

### Documentation and Communication

**Documentation updates**:

* XRPL.org: New pages for RecurringPayment transactions
* GitHub: Updated README and guides
* API docs: New endpoints and fields documented

**Public announcements**:

```markdown
# Subscriptions Amendment Activated

Date: 2025-05-04 14:32:15 UTC
Ledger: 86652345

The Subscriptions amendment (XLS-0078) has been successfully activated on the XRPL Mainnet.

## What's New:
- New transaction types: SubscriptionSet, SubscriptionClaim, SubscriptionCancel
- New ledger object: ltSUBSCRIPTION (0x0055)
- Programmable recurring payment subscriptions with XRP, IOUs, and MPTs

## For Developers:
- Update to rippled 1.12.0+
- Update xrpl.js to 2.12.0+
- Update xrpl-py to 2.8.0+

## Documentation:
- [Subscriptions Standard (XLS-0078)](https://github.com/XRPLF/XRPL-Standards/tree/master/XLS-0078-subscriptions)
- [API Reference](https://xrpl.org/api-reference.html)
```

***

## Lessons Learned: Historical Cases

### fixAmendmentMajorityCalc (2021)

**Problem**: Asymmetry in threshold calculation (rounding up instead of down).

**Impact**: In certain cases, the effective threshold was slightly higher than 80%.

**Resolution**: New amendment to correct the calculation.

**Lesson**: Even simple mathematical calculations must be rigorously tested.

### MultiSign (2016)

**Success**: First major amendment adding significant functionality.

**Timeline**:

* Development: 6 months
* Testing on Testnet: 3 months
* Deployment: 2 months
* Activation: Without incident

**Lesson**: Extensive preparation and rigorous testing are essential for complex amendments.


---

# Agent Instructions: Querying This Documentation

If you need additional information that is not directly available in this page, you can query the documentation dynamically by asking a question.

Perform an HTTP GET request on the current page URL with the `ask` query parameter:

```
GET https://docs.xrpl-commons.org/core-dev-bootcamp/module10/network-coordination.md?ask=<question>
```

The question should be specific, self-contained, and written in natural language.
The response will contain a direct answer to the question and relevant excerpts and sources from the documentation.

Use this mechanism when the answer is not explicitly present in the current page, you need clarification or additional context, or you want to retrieve related documentation sections.