# Welcome
This is the home of the XRPL Commons technical trainings. This space aims to showcase all content related to training on the XRPL.
## Available resources
The following links are the key tools in your journey on the XRP ledger.
Ecosystem map: [https://map.xrpl-commons.org](https://map.xrpl-commons.org/)
Block explorer: [https://testnet.xrpl.org](https://testnet.xrpl.org/)
Faucets:
Transactions references:
Training live sync: (password is training-april-2024)
## Further reading
Main XRPL Documentation: [https://xrpl.org](https://xrpl.org/)
Learning Portal: [https://learn.xrpl.org](https://learn.xrpl.org/)
Check out our events and opportunities at XRPL Commons: [https://xrpl-commons.org](https://xrpl-commons.org/)
# XRPL Basics
In this section we will cover some of the primary functionality of the XRPL ledger.
# Getting Started
Start here :)
First, you will want to set up your coding environment.
Any JavaScript editor will work, and feel free to use your own Node environment to follow along with JavaScript. Most of the code we will use can be run client side as well.
For a hassle-free setup, use Replit (). Accounts are free. Create a new Replit with the TypeScript defaults, or use the Node defaults if you prefer to avoid TypeScript.
If you use another language, you can follow along with your own script, but you will need the appropriate SDK, which may differ from xrpl.js.
You will often need to refer to the main API documentation for transactions here:
## **Faucets**
While faucets are accessible programmatically, you can also create a test account:
## **Wallets**
This tutorial uses the Xaman wallet, which you can download to your mobile phone a from [https://xumm.app](https://xumm.app/). Once installed you can import a wallet via "family seed" (the address secret) for quicker setup.
However, most examples in this tutorial sign transactions programmatically.
## **Replit**
To get started on replit create an account. You can create a free acount.
Go to [developer frameworks](https://replit.com/developer-frameworks), hit create a search for Typescript.
Open the right hand panel to browse files, edit index.ts, and run from the top left play button.
You can close the AI panel to the left for more screen real estate.
# Payments
In this section we will create your first payment transaction.
## Creating the main structure of your script
Create a new file or edit `index.ts`
```javascript
import xrpl from "xrpl"
const client = new xrpl.Client("wss://s.altnet.rippletest.net:51233")
const main = async () => {
console.log("lets get started...");
await client.connect();
// do something interesting here
await client.disconnect();
console.log("all done!");
};
main();
```
Running this to make sure everything works should console log a few things. If you are running this from your own node environment, don't forget to run`npm i xrpl` to add it to your project.
## Creating wallets
Let's add some wallet creation logic, we might as well create 2 wallets to have some fun.
```javascript
console.log('lets fund 2 accounts...')
const { wallet: wallet1, balance: balance1 } = await client.fundWallet()
const { wallet: wallet2, balance: balance2 } = await client.fundWallet()
console.log('wallet1', wallet1)
console.log({
balance1,
address1: wallet1.address, //wallet1.seed
balance2,
address2: wallet2.address
})
```
At this point, we should have 2 wallets with balances of 100 XRP.
## Transferring XRP from one wallet to the other
Of course, we will want to transfer some XRP from one account to another, because that's why we're here.
```typescript
const tx:xrpl.Payment = {
TransactionType: "Payment",
Account: wallet1.classicAddress,
Destination: wallet2.classicAddress,
Amount: xrpl.xrpToDrops("13")
};
```
Let's log it to ensure everything is correct, and then submit it to the ledger:
```javascript
console.log('submitting the payment transaction... ', tx)
const result = await client.submitAndWait(tx, {
autofill: true,
wallet: wallet1,
});
console.log(result)
```
When you run this part you should get the following log output:
```javascript
{
id: 28,
result: {
Account: 'rGGJ71dbSY5yF9BJUDSHsDPSKDhVGGWzpY',
Amount: '13000000',
DeliverMax: '13000000',
Destination: 'rBAbErfjkwFWFerLfAHmbi3qgbRfuFWxEN',
Fee: '12',
Flags: 0,
LastLedgerSequence: 232639,
Sequence: 232617,
SigningPubKey: 'ED13EBC7F89545435E82DC19B2C38AF5ECF39CE099C8FA647280C71CD6FA5BEF3B',
TransactionType: 'Payment',
TxnSignature: '66D9338B3C93D212F89BB4F6731E7F38A01052F8ED94452D7D2A9BB0B0C6130A5708390E9B6233A98A3B28A1922E57E37317609727409B3D289C456BB3250E08',
ctid: 'C0038CAD00000001',
date: 767648991,
hash: 'CC8241E3C4B57ED9183D6031F4E370AC13B6CE9E2332BD7AF77C25BD6ADFA4F6',
inLedger: 232621,
ledger_index: 232621,
meta: {
AffectedNodes: [Array],
TransactionIndex: 0,
TransactionResult: 'tesSUCCESS',
delivered_amount: '13000000'
},
validated: true
},
type: 'response'
}
```
The most important being `TransactionResult: 'tesSUCCESS'`
You can verify the transaction on the ledger, as it is readable by anyone at this point. Go to and paste the hash value (in my example case it would be `CC8241E3C4B57ED9183D6031F4E370AC13B6CE9E2332BD7AF77C25BD6ADFA4F6`. You should see it. Alternatively, check the wallet addresses as well (remember you are using the public key).
I prefer to check the balances programmatically to ensure everything has gone according to plan. Adding this code will do that:
```javascript
console.log({
'balance 1': await client.getBalances(wallet1.classicAddress),
'balance 2': await client.getBalances(wallet2.classicAddress)
})
```
## Putting it all together
Here is the entire index.ts file at this point:
{% code title="index.ts" %}
```typescript
import xrpl from "xrpl"
const client = new xrpl.Client("wss://s.altnet.rippletest.net:51233")
const main = async () => {
console.log("lets get started...");
await client.connect();
// do something interesting here
console.log('lets fund 2 accounts...')
const { wallet: wallet1, balance: balance1 } = await client.fundWallet();
const { wallet: wallet2, balance: balance2 } = await client.fundWallet();
console.log('wallet1', wallet1)
console.log({
balance1,
address1: wallet1.address, //wallet1.seed
balance2,
address2: wallet2.address
});
const tx:xrpl.Payment = {
TransactionType: "Payment",
Account: wallet1.classicAddress,
Destination: wallet2.classicAddress,
Amount: xrpl.xrpToDrops("13")
};
console.log('submitting the payment transaction... ', tx)
const result = await client.submitAndWait(tx, {
autofill: true,
wallet: wallet1,
});
console.log(result)
console.log({
'balance 1': await client.getBalances(wallet1.classicAddress),
'balance 2': await client.getBalances(wallet2.classicAddress)
})
await client.disconnect();
console.log("all done!");
};
main();
```
{% endcode %}
## Extra credit
Don't you find it limiting that the faucet only gives out 100 XRP? Why not create a function to print money to an address?
You have all the necessary building blocks:
1. You will need to create and fund new wallets with `await client.fundWallet()`
2. Remember, you can only transfer XRP up to the reserve, so a maximum of 90 XRP at a time for brand new accounts, unless you are brave enough to use a new transaction type.? (). However, be warned that you will have to wait a block or two before the account can be deleted.
3. The final function signature should look something like this \`await printMoney({ destinationWallet, client })
I can't wait to see who writes this faster than Claude Code !
Here is the popular printMoney function
# Reading and subscribing to Transactions
In this session, we'll demonstrate how to coordinate using the XRPL.
The XRPL is a public ledger, meaning that when you write on-chain, anyone can read it. In this session, we will demonstrate how to subscribe to events as they happen on-chain.
First, we will create a pair of accounts:
{% code title="index.ts" %}
```javascript
import xrpl from "xrpl";
const serverURL = "wss://s.altnet.rippletest.net:51233"; // For testnet
const main = async () => {
const client = new xrpl.Client(serverURL)
await client.connect()
// do something useful
console.log("lets fund 2 accounts...")
const { wallet: wallet1, balance: balance1 } = await client.fundWallet()
const { wallet: wallet2, balance: balance2 } = await client.fundWallet()
console.log({ wallet1, balance1 })
console.log({ wallet2, balance2 })
// end
client.disconnect()
};
main()
```
{% endcode %}
Check the logs to find the seed for both accounts. We will use these later to reuse the accounts deterministically.
### Creating a payment transaction
Here is how to generate a payment transaction between the two accounts.
{% hint style="success" %}
Insert the seeds from the previous section in the top section.
{% endhint %}
{% code title="index.ts" %}
```javascript
import xrpl from "xrpl";
const serverURL = "wss://s.altnet.rippletest.net:51233"; // For testnet
const wallet1 = xrpl.Wallet.fromSeed("s...");
const wallet2 = xrpl.Wallet.fromSeed("s...");
const main = async () => {
const client = new xrpl.Client(serverURL);
await client.connect();
const tx = {
TransactionType: "Payment",
Account: wallet1.classicAddress,
Destination: wallet2.classicAddress,
Amount: "1234" // drops
}
const result = await client.submitAndWait(tx, {
autofill: true,
wallet: wallet1,
});
console.log(result)
// end
client.disconnect()
};
main()
```
{% endcode %}
### Reading transactions
One way to retrieve transactions for a given account is to create a request for transactions. Here is an example
```javascript
// ... define wallet1 before this point
const request = {
command: 'account_tx',
account: wallet1.classicAddress,
ledger_index_min: -1, // To get transactions from all ledgers
ledger_index_max: -1, // To get transactions up to the most recent ledger
limit: 10, // Limit the number of transactions (optional)
}
const response = await client.request(request)
console.log('Account Transactions:', response.result.transactions)
```
### Listening for transactions
Finally, we will demonstrate how to listen for transactions affecting a given account to catch all transactions.
{% hint style="warning" %}
**You could use this to build a notification bot that alerts you when any activity occurs in your account.**
{% endhint %}
You can subscribe to an account, to trigger on each transaction for instance. This is how you would listen to a specific account.
{% code title="index.ts" %}
```javascript
import xrpl from 'xrpl';
const serverURL = 'wss://s.altnet.rippletest.net:51233'; // For testnet
const walletAddress = 'r...' // address to watch
const main = async () => {
const client = new xrpl.Client(serverURL)
await client.connect()
// do something useful
const subscriptionRequest = {
command: 'subscribe',
accounts: [walletAddress]
};
await client.request(subscriptionRequest)
console.log(`Subscribed to transactions for account: ${walletAddress}`)
// Event listener for incoming transactions
client.on('transaction', (transaction) => {
console.log('Transaction:', transaction);
})
// Event listener for errors
client.on('error', (error) => {
console.error('Error:', error);
})
// end
// keep open
console.log('all done')
}
main()
setInterval(() => {
console.log('One second has passed!');
}, 1000)
```
{% endcode %}
###
# Writing and reading memos
Memos are available in every transaction and can be quite useful to label your transactions or pass metadata useful to your business logic.
### Writing a memo
This is how you would include a json object in the memo field of a transaction.
```javascript
// Create the JSON object you want to include in the memo
const memoData = {
invoice_id: '12345',
description: 'Coffee and Milk',
date: '2024-07-02'
};
// Convert JSON object to a string and then to hex
const memoJson = JSON.stringify(memoData);
const memoHex = Buffer.from(memoJson, 'utf8').toString('hex');
// Create the Memo field
const memo = {
Memo: {
MemoData: memoHex
}
};
const tx = {
TransactionType: "Payment",
Account: wallet1.classicAddress,
Destination: wallet2.classicAddress,
Amount: "1234",
Memos: [memo]
};
const result = await client.submitAndWait(tx, {
autofill: true,
wallet: wallet1,
});
console.log(result)
```
### Decoding memos
To decode memos from a transaction, we retrieve the memo components and decode the hexadecimal data to a string, and then parse them.
Create this file in your project and import it as needed, when you are processing the transaction data.
{% code title="parseMemo.ts" %}
```javascript
export const parseMemo = (memos) => {
let report = {};
if (memos && memos.length > 0) {
for (const memo of memos) {
if (memo.Memo.MemoData) {
// Decode the hexadecimal memo data to a string
const memoDataHex = memo.Memo.MemoData;
const memoDataJson = Buffer.from(memoDataHex, "hex").toString("utf8");
// Parse the JSON string into a JavaScript object
const memoDataObject = JSON.parse(memoDataJson);
// console.log('Decoded memo data:', memoDataObject)
report = { ...report, ...memoDataObject };
} else {
console.log("No MemoData found.");
}
}
return report;
} else {
console.log("No memos found in the transaction.");
}
};
export default parseMemo
```
{% endcode %}
### Putting it all together
Here is an example of a fully fledged transaction listener that can parse the memos from the transaction we created above.
{% hint style="info" %}
replace the wallet address with the destination of the original transaction
{% endhint %}
{% code title="index.ts" %}
```javascript
import xrpl from "xrpl";
import { parseMemo } from "./parseMemo";
const serverURL = "wss://s.altnet.rippletest.net:51233"; // For testnet
const walletAddress = "r...";
const main = async () => {
const client = new xrpl.Client(serverURL);
await client.connect();
// do something useful
const subscriptionRequest = {
command: "subscribe",
accounts: [walletAddress],
};
await client.request(subscriptionRequest);
console.log(`Subscribed to transactions for account: ${walletAddress}`);
// Event listener for incoming transactions
client.on("transaction", (transaction) => {
console.log("Transaction:", transaction);
const parsedMemos = parseMemo(transaction.transaction.Memos);
console.log("Parsed memo:", parsedMemos);
});
// Event listener for errors
client.on("error", (error) => {
console.error("Error:", error);
});
// end
// keep open
console.log("listening...");
};
main()
```
{% endcode %}
# Non Fungible Tokens
In this session, we will learn how to create digital art using Non-Fungible Tokens (NFTs) with their metadata and specific attributes.
### Start coding
Create a new file or edit `index.ts`
```javascript
import { Client, Wallet } from "xrpl"
const client = new Client("wss://s.altnet.rippletest.net:51233")
const main = async () => {
console.log("lets get started...");
await client.connect();
// do something interesting here
await client.disconnect();
console.log("all done!");
};
main();
```
Running this to ensure everything works should log a few things to the console. If you are running this from your node environment, don't forget to run`npm i xrpl` to add it to your project.
#### Creating NFT
First, to create digital art, you need to create an NFT using `NFTokenMint`. You need one available wallet for that. For an NFT to host metadata and have specific attributes, we usually provide a JSON file for the metadata. You can gain a better understanding of the OpenSea metadata standards (). When creating the NFT, you will use a URI field to include the URI to the metadata of the NFT you want to create.
```typescript
import { convertStringToHex, NFTokenMint, NFTokenMintFlags, NFTokenCreateOffer } from 'xrpl';
const { wallet: wallet1, balance: balance1 } = await client.fundWallet()
console.log('wallet1', wallet1)
const uri = "My super NFT URI"
const nftMintTx: NFTokenMint = {
TransactionType: "NFTokenMint",
Account: wallet.address,
URI: convertStringToHex(uri),
Flags: NFTokenMintFlags.tfBurnable + NFTokenMintFlags.tfTransferable, // Burnable in case no one is buying it
NFTokenTaxon: 0, // Unique identifier for the NFT type
}
const prepared = await client.autofill(nftMintTx)
const signed = wallet.sign(prepared)
const result = await client.submitAndWait(signed.tx_blob)
const nftId = nftId: (result.result.meta as NFTokenMintMetadata)?.nftoken_id as string
console.log("NFT ID " + nftId + " created")
```
At this point, you should have one account with an NFT created.
#### Putting your NFT on sale
Once you have your NFT, you may want to put it on sale and allow people to bid on it. Thanks to XRPL built-in functions, the NFT marketplace is already provided within the ledger. We only need to use `NFTokenCreateOffer` to put our NFT up for sale.
```javascript
import { NFTokenCreateOffer } from 'xrpl';
const nftCreateOfferTx: NFTokenCreateOffer = {
TransactionType: "NFTokenCreateOffer",
Account: wallet.address,
Destination: destination, // Optional if precised it would precised the offer for a specific address
NFTokenID: nftId,
Amount: "0", // 0 would represent a gift to someone
Flags: 1 // Sell offer
};
const prepared = await client.autofill(nftCreateOfferTx)
const signed = wallet.sign(prepared)
const result = await client.submitAndWait(signed.tx_blob)
const offerId = (result.result.meta as NFTokenCreateOfferMetadata)?.offer_id as string
console.log("Offer ID " + offerId + " created")
```
At this point, your NFT should be on sale, and it will be available for someone specific or anyone to buy.
### Extra credits
**Canceling your offer**
You can cancel your NFT offer, you will need to use the NFTokenCancelOffer function to do so ()
**Burning your NFT**
You can burn your NFT, you will need to use the NFTokenBurn function to do so ()
[ ](https://xrpl-commons-1.gitbook.io/trainings/training-and-formation/subscriptions)
# Multi-Purpose Tokens
Learn how to create and manage Multi-Purpose Tokens (MPT) on the XRP Ledger.
Multi-Purpose Tokens (MPT) are a recent amendment to the XRP Ledger, enhancing the token system by allowing tokens to serve multiple functions beyond just being a medium of exchange. This amendment introduces new capabilities for token issuers and holders.
## Use Cases
* **Stablecoins**: Create stablecoins with features like freezing and clawback.
* **Utility Tokens**: Design tokens with specific utility functions.
* **Security Tokens**: Implement security tokens with transfer restrictions.
* **Community Credit**: Track debts and credits between known parties.
## MPToken Flags
Each flag is a power of 2, allowing them to be combined using bitwise operations. The total value is the sum of all enabled flags.
| Flag | Value | Description |
| ----------- | ----- | ------------------------------------------- |
| canLock | 2 | Lock tokens globally or per account |
| requireAuth | 4 | Require issuer approval for new holders |
| canEscrow | 8 | Enable time-locked escrow (Not implemented) |
| canTrade | 16 | Enable DEX trading (Not implemented) |
| canTransfer | 32 | Enable basic token transfers |
| canClawback | 64 | Allow issuer to recover tokens |
### Common Flag Combinations:
* **Basic**: canTransfer (32)
* **Secure**: canLock + canClawback + canTransfer (98)
* **Regulated**: canLock + canClawback + canTransfer + requireAuth (102)
## Getting Started: Creating the Main Structure of Your Script
Create a new file or edit `index.ts`:
```typescript
import { Client } from 'xrpl';
const client = new Client("wss://s.devnet.rippletest.net:51233");
const main = async () => {
try {
console.log("Let's create a Multi-Purpose Token...");
await client.connect();
// Create issuer and holder wallets
const { wallet: issuerWallet } = await client.fundWallet();
const { wallet: holderWallet } = await client.fundWallet();
console.log('Issuer Wallet:', issuerWallet.address);
console.log('Holder Wallet:', holderWallet.address);
// Define token metadata
const tokenMetadata = {
name: "MyMPToken",
symbol: "MPT",
description: "A sample Multi-Purpose Token",
};
// Convert metadata to hex string
const metadataHex = Buffer.from(JSON.stringify(tokenMetadata)).toString('hex');
// Set flags for a regulated token
const totalFlagsValue = 102; // canLock + canClawback + canTransfer + requireAuth
// Create MPToken issuance transaction
const transactionBlob = {
TransactionType: "MPTokenIssuanceCreate",
Account: issuerWallet.address,
Flags: totalFlagsValue,
MPTokenMetadata: metadataHex
// MaximumAmount: "1000000000000000000", (Optional) The maximum asset amount of this token that can ever be issued
// TransferFee: 5000, (Optional) between 0 and 50000 for 50.000% fees charged by the issuer for secondary sales of Token
// AssetScale: 2, (Optional) 10^(-scale) of a corresponding fractional unit. For example, a US Dollar Stablecoin will likely have an asset scale of 2, representing 2 decimal places.
};
// Submit token issuance
const mptokenCreationResult = await client.submitAndWait(transactionBlob, {
autofill: true,
wallet: issuerWallet
});
console.log('MPToken Creation Result:', mptokenCreationResult.result.meta?.TransactionResult);
console.log('MPToken Issuance ID:', mptokenCreationResult.result.meta?.mpt_issuance_id);
await client.disconnect();
console.log("All done!");
} catch (error) {
console.error("Error creating MPToken:", error);
}
};
main();
```
### Understanding MPTokenIssuanceID
The `mpt_issuance_id` is a unique identifier generated when an MPToken is created via `MPTokenIssuanceCreate`. This ID is:
* **Automatically generated** by the XRPL network upon successful token creation
* **Globally unique** across all MPTokens on the ledger
* **Required** for all subsequent operations (payments, clawbacks, authorization)
**To obtain the MPTokenIssuanceID:**
1. Submit an `MPTokenIssuanceCreate` transaction
2. Retrieve the ID from the transaction result: `result.meta?.mpt_issuance_id`
3. Store this ID for future operations with your token
**Alternative ways to find the MPTokenIssuanceID:**
* **From Explorer**: You can find the ID on the XRPL explorer () by searching with the transaction hash or the creator's address
* **Query by Issuer**: Use the `account_lines` API method with the issuer's address to find all tokens issued by that account
```typescript
// Query the ledger for all MPTokens issued by a specific account
const response = await client.request({
command: "account_objects",
account: issuerAddress,
ledger_index: "validated",
type: "mpt_issuance"
});
// Find the specific token by currency code in the metadata
const mpTokens = response.result.account_objects;
const targetToken = mpTokens.find(token => {
// Parse metadata from hex to JSON
const metadata = JSON.parse(Buffer.from(token.MPTokenMetadata, 'hex').toString());
return metadata.symbol === "TECH"; // Replace with your token symbol
});
const mpTokenId = targetToken?.mpt_issuance_id;
console.log("mpTokenId: ", mpTokenId);
```
The relationship between issuer, token properties, and ID:
* **Issuer**: The account that created the token (unchangeable)
* **Token Properties**: Metadata, flags, and rules defined at creation
* **MPTokenIssuanceID**: The unique identifier linking all operations to this specific token issuance
## Technical Specifications
* Uses decimal (base-10) math with 15 digits of precision
* Can express values from 1.0 × 10^-81 to 9999999999999999 × 10^80
* Supports transfer fees that are automatically deducted
* Allows issuers to define tick sizes for exchange rates
## Destroying MPTokens
The `MPTokenIssuanceDestroy` transaction allows an issuer to permanently destroy an MPToken issuance. This is useful for:
* Retiring deprecated or unused tokens
* Removing tokens that were created in error
* Regulatory compliance and token lifecycle management
### Requirements
* Only the original issuer can destroy the MPToken issuance
* All tokens must be owned by the issuer (transferred back) before destruction
* The MPTokenIssuanceID must be valid and reference an existing issuance
### Example Transaction
```typescript
const transactionBlob = {
TransactionType: "MPTokenIssuanceDestroy",
Account: wallet.address,
MPTokenIssuanceID: mpTokenId
};
```
## Freezing MPToken Issuance
The `MPTokenIssuanceSet` transaction allows an issuer to modify the state of an MPToken issuance, including freezing/unfreezing transfers. This is useful for:
* Temporarily halting transfers during maintenance or upgrades
* Responding to security incidents
* Complying with regulatory requirements
* Managing token lifecycle events
### Requirements
* Only the original issuer can modify the issuance settings
* The MPTokenIssuanceID must be valid and reference an existing issuance
* Appropriate flags must be set during issuance to enable freezing
### Example Transaction
```typescript
const transactionBlob = {
TransactionType: "MPTokenIssuanceSet",
Account: wallet.address,
MPTokenIssuanceID: mpTokenId,
Flags: 1, // 1: Lock, 2: Unlock
// Holder: r3d4... // Specify holder to freeze; if not specified, it will freeze globally
};
```
## Authorizing MPToken Holders
The `MPTokenAuthorize` transaction enables fine-grained control over who can hold and transfer Multi-Purpose Tokens (MPTs) on the XRPL. This mechanism supports compliance, prevents unsolicited token spam, and allows issuers to manage token distribution effectively.
### Features
* **Authorize a Holder**: Permit a specific account to receive and hold a designated MPT
* **Revoke Authorization**: Remove a holder's permission, effectively locking their MPT balance
* **Self-Unauthorize**: Allow a holder to voluntarily relinquish their MPT holdings (requires zero balance)
* **Global Locking**: Restrict all transfers of a particular MPT issuance
### Requirements
* Only the original issuer can authorize/revoke holders
* The MPTokenIssuanceID must be valid and reference an existing issuance
* Authorization flags must be enabled during issuance
* Holder accounts must be valid XRPL addresses
### Example Transaction
```typescript
const transactionBlob = {
TransactionType: "MPTokenAuthorize",
Account: wallet.address,
MPTokenIssuanceID: mptokenID
Flags: 0 // If set to 1, and transaction is submitted by a holder, it indicates that the holder no longer wants to hold the MPToken
// Holder: r3d4... (Optional) Specify address to authorize; if not specified, it will authorize globally
};
```
## Transaction Type modifications
The introduction of Multi-Purpose Tokens (MPT) on the XRP Ledger has brought significant changes to existing transaction types, especially Payment and Clawback. These transactions now support MPTokens, allowing you to transfer or recover tokens issued under the new amendment.
### Changes introduced:
* **Payment**:\
The Payment transaction can now transfer MPTokens by using the `Amount` field as an object containing the issuance identifier (`mpt_issuance_id`) and the amount to transfer.\
Example:
```typescript
const transactionBlob = {
TransactionType: "Payment",
Account: wallet.address,
Amount: {
"mpt_issuance_id": mpTokenId,
"value": "10000"
},
Destination: destinationAddress.value,
};
```
* **Clawback**:\
The Clawback transaction allows the issuer to recover MPTokens from a specific account, also using the adapted `Amount` field for MPTokens.\
Example:
```typescript
const transactionBlob = {
TransactionType: "Clawback",
Account: wallet.address,
Amount: {
"mpt_issuance_id": mpTokenId,
"value": "10000"
},
Holder: holderAddress.value,
};
```
## Putting It All Together
Let's create a complete example that demonstrates all the MPToken features by simulating a real-world scenario: issuing company shares to investors. In this example, we'll:
1. Create a company share token with regulatory controls
2. Authorize investors to receive shares
3. Distribute shares to investors
4. Demonstrate compliance controls (locking and clawback)
* You can check every transaction hash on :
```javascript
import { Client } from 'xrpl';
const client = new Client("wss://s.devnet.rippletest.net:51233");
const main = async () => {
try {
console.log("Creating company shares using Multi-Purpose Tokens...");
await client.connect();
console.log("Setting up company and investor wallets...");
// Create company (issuer) and investor wallets
const { wallet: companyWallet } = await client.fundWallet();
const { wallet: investor1Wallet } = await client.fundWallet();
const { wallet: investor2Wallet } = await client.fundWallet();
console.log('Company Wallet:', companyWallet.address);
console.log('Investor 1 Wallet:', investor1Wallet.address);
console.log('Investor 2 Wallet:', investor2Wallet.address);
console.log("");
// Define company share token metadata
const tokenMetadata = {
name: "TechCorp Shares",
symbol: "TECH",
description: "Equity shares in TechCorp with regulatory compliance features",
};
// Convert metadata to hex string
const metadataHex = Buffer.from(JSON.stringify(tokenMetadata)).toString('hex');
// Set flags for a regulated security token
const totalFlagsValue = 102; // canLock + canClawback + canTransfer + requireAuth
// Create company share token issuance
let transactionBlob = {
TransactionType: "MPTokenIssuanceCreate",
Account: companyWallet.address,
Flags: totalFlagsValue,
MPTokenMetadata: metadataHex
};
console.log("Issuing company share tokens...");
// Submit token issuance
const createTx = await client.submitAndWait(transactionBlob, { wallet: companyWallet });
// Get the MPTokenID for our company shares
const MPTokenID = createTx.result.meta?.mpt_issuance_id;
console.log('Share token creation transaction hash:', createTx.result.hash);
console.log('Company Share Token ID:', MPTokenID);
console.log("");
// First, investors need to self-authorize to receive the tokens
// Investor 1 self-authorization
transactionBlob = {
TransactionType: "MPTokenAuthorize",
Account: investor1Wallet.address,
MPTokenIssuanceID: MPTokenID,
};
console.log("Investor 1 authorizing to receive shares...");
const investor1SelfAuthTx = await client.submitAndWait(transactionBlob, {wallet: investor1Wallet });
// Investor 2 self-authorization
transactionBlob = {
TransactionType: "MPTokenAuthorize",
Account: investor2Wallet.address,
MPTokenIssuanceID: MPTokenID,
};
console.log("Investor 2 authorizing to receive shares...");
const investor2SelfAuthTx = await client.submitAndWait(transactionBlob, {wallet: investor2Wallet });
console.log("Investor 1 self-authorization transaction hash:", investor1SelfAuthTx.result.hash);
console.log("Investor 2 self-authorization transaction hash:", investor2SelfAuthTx.result.hash);
console.log("");
// With requireAuth flag, the company (issuer) must authorize investors
// Authorize investor 1
transactionBlob = {
TransactionType: "MPTokenAuthorize",
Account: companyWallet.address,
MPTokenIssuanceID: MPTokenID,
Holder: investor1Wallet.address
};
console.log("Company authorizing investor 1 to receive shares...");
const investor1AuthTx = await client.submitAndWait(transactionBlob, {wallet: companyWallet });
// Authorize investor 2
transactionBlob = {
TransactionType: "MPTokenAuthorize",
Account: companyWallet.address,
MPTokenIssuanceID: MPTokenID,
Holder: investor2Wallet.address
};
console.log("Company authorizing investor 2 to receive shares...");
const investor2AuthTx = await client.submitAndWait(transactionBlob, {wallet: companyWallet });
console.log("Investor 1 issuer authorization transaction hash:", investor1AuthTx.result.hash);
console.log("Investor 2 issuer authorization transaction hash:", investor2AuthTx.result.hash);
console.log("");
// Distribute shares to investor 1 (10,000 shares)
transactionBlob = {
TransactionType: "Payment",
Account: companyWallet.address,
Amount: {
"mpt_issuance_id": MPTokenID, // Company share token ID
"value": "10000" // 10,000 shares
},
Destination: investor1Wallet.address,
};
console.log("Distributing 10,000 shares to investor 1...");
const paymentTx = await client.submitAndWait(transactionBlob, {wallet: companyWallet });
console.log("Share distribution transaction hash: ", paymentTx.result.hash);
console.log("");
// Demonstrate compliance: Lock investor 1's shares (e.g., during regulatory investigation)
transactionBlob = {
TransactionType: "MPTokenIssuanceSet",
Account: companyWallet.address,
MPTokenIssuanceID: MPTokenID,
Holder: investor1Wallet.address,
Flags: 1, // Lock the shares
};
console.log("Locking investor 1's shares for compliance review...");
const lockTx = await client.submitAndWait(transactionBlob, {wallet: companyWallet });
console.log("Lock transaction hash: ", lockTx.result.hash);
console.log("TransactionResult: ", lockTx.result.meta.TransactionResult);
console.log("Investor 1 can no longer transfer their shares");
console.log("");
// Attempt transfer while locked (this will fail)
transactionBlob = {
TransactionType: "Payment",
Account: investor1Wallet.address,
Amount: {
"mpt_issuance_id": MPTokenID,
"value": "5000"
},
Destination: investor2Wallet.address,
};
console.log("Attempting to transfer locked shares to investor 2 (this will fail)...");
const transferTx = await client.submitAndWait(transactionBlob, {wallet: investor1Wallet });
console.log("Transfer transaction hash: ", transferTx.result.hash);
console.log("TransactionResult: ", transferTx.result.meta.TransactionResult);
console.log("Transfer failed as expected - shares are locked");
console.log("");
// Company exercises clawback rights (e.g., for regulatory compliance)
transactionBlob = {
TransactionType: "Clawback",
Account: companyWallet.address,
Amount: {
"mpt_issuance_id": MPTokenID,
"value": "10000"
},
Holder: investor1Wallet.address,
};
console.log("Company exercising clawback rights on investor 1's shares...");
const clawbackTx = await client.submitAndWait(transactionBlob, {wallet: companyWallet });
console.log("Clawback transaction hash: ", clawbackTx.result.hash);
console.log("All 10,000 shares have been returned to the company");
console.log("");
await client.disconnect();
console.log("Company share token demonstration complete!");
} catch (error) {
console.error("Error in share token operations:", error);
}
};
main();
```
## Resources
* [XRP Ledger Documentation](https://xrpl.org/docs.html)
* [MPTokenIssuanceCreate](https://xrpl.org/docs/references/protocol/transactions/types/mptokenissuancecreate)
# PathFinding
Pathfinding on the XRP Ledger (XRPL) enables efficient payment routing by discovering the optimal path for transferring value between accounts. This feature facilitates cross-currency transactions, ensuring payments can be made even when the sender and receiver use different currencies.
***
### Pathfinding Explained
#### What Is Pathfinding?
Pathfinding is a feature on the XRPL that identifies the most cost-effective route for payments involving multiple currencies. It leverages the decentralized exchange (DEX) on the XRPL, which supports cross-currency transactions by converting currencies seamlessly using order books.
#### Why Use Pathfinding?
Pathfinding ensures that payments are completed with minimal fees and optimal exchange rates, even if the sender and receiver operate in entirely different currencies. This makes XRPL a robust platform for international remittances and currency exchanges.
#### How Pathfinding Works
When initiating a cross-currency transaction, XRPL scans its order books and liquidity pools to identify the best combination of offers to complete the payment. This process includes:
1. **Finding Direct Offers**: Direct currency pairs between the sender and receiver.
2. **Building Complex Paths**: Identifying intermediate steps involving multiple currencies if no direct offer is available.
***
### Advantages of Pathfinding
1. **Efficient Cross-Currency Payments**\
Pathfinding automatically converts currencies using the best available rates, eliminating the need for intermediaries.
2. **Optimized Transaction Costs**\
By leveraging competitive offers on the XRPL’s DEX, pathfinding minimizes fees associated with currency conversion.
3. **Global Reach**\
Enables seamless payments across borders, even in currencies that are not natively linked.
4. **Automated Exchange**\
Eliminates manual intervention by automating the selection of the best path for payments.
***
## Pathfinding in Action
### Prerequisites
To use the pathfinding feature, ensure the following:
* Both the sender and receiver accounts are active on XRPL.
* Sufficient XRP balance is maintained to cover transaction fees.
* The sender has trustlines established for any non-XRP currencies involved in the payment.
* The currencies involved in the transaction must be active on the DEX with some trading activity. If you are using a testnet, you may need to create an active market using bots to simulate trading activity.
#### Initialize the wallets
Begin by installing `npm i xrpl`
```javascript
import { Client, Wallet } from "xrpl";
// To test pathfinding the best way is to use MainNet as we need active trading on the currencies. You can do this on testnet, this would require to make an active market between Currency 1 <> XRP <> Currency 2
const client = new Client("wss://xrplcluster.com");
await client.connect();
// The source and destination address can be the same, you will now effectively swap between a currency pair
const sourceWallet = Wallet.fromSeed("s...");
const destinationWallet = Wallet.fromSeed("s...");
```
### Example: Cross-Currency Payment
#### Retrieve Payment Paths
To initiate a cross-currency payment, start by retrieving available paths with the [pathfinding method](https://xrpl.org/docs/references/http-websocket-apis/public-api-methods/path-and-order-book-methods/path_find).
```javascript
client.on('path_find', (path_find) => {
console.log('New path: ', path_find);
});
const pathfindRequest = {
command: "path_find",
subcommand: "create",
source_account: sourceWallet.classicAddress, // Sender's account
destination_account: destinationWallet.classicAddress, // Receiver's account
send_max: "1000000", // Maximum amount the sender is willing to spend (optional)
destination_amount: {
currency: "USD",
value: -1, // Using -1 to indicate unspecified destination amount
issuer: "rIssuerAddress" // Issuer of the destination currency
}
};
const paths = await client.request(pathfindRequest);
console.log('Available Paths:', paths.result.alternatives);
```
**Alternative Example 2: Specified destination Amount**
This way you will replace the other request and you will see what you need to spend in order for the destination address to receive a specified amount. This requires a list of source currencies.
```javascript
const pathfindRequest = {
command: "path_find",
subcommand: "create",
source_account: sourceWallet.classicAddress, // Sender's account
destination_account: destinationWallet.classicAddress, // Receiver's account
destination_amount: {
currency: "USD",
value: "100", // Specify the exact amount the destination should receive
issuer: "rIssuerAddress" // Issuer of the destination currency
},
source_currencies: [{
currency: "524C555344000000000000000000000000000000", // RLUSD
issuer: "rMxCKbEDwqr76QuheSUMdEGf4B9xJ8m5De"
}]
};
```
#### Send the Payment
Once a suitable path is identified, construct the [payment transaction](https://xrpl.org/docs/references/protocol/transactions/types/payment).
```javascript
const paymentTransaction = {
TransactionType: "Payment",
Account: "rSenderAddress",
Destination: "rReceiverAddress",
Amount: {
currency: "USD",
value: "100",
issuer: "rIssuerAddress"
},
SendMax: {
currency: "EUR",
value: "90",
issuer: "rIssuerAddress" // Specify the maximum amount in source currency
},
Paths: paths.result.alternatives[0].paths_computed // Include computed path
};
try {
const response = await client.submitAndWait(paymentTransaction, { autofill: true, wallet: sourceWallet });
console.log('Transaction Result:', response);
} catch (error) {
console.error('Failed to submit payment:', error);
}
// Don't forget to close it!
await client.request({
command: 'path_find',
subcommand: 'close'
});
```
***
## Key Considerations
#### Trustlines
For non-XRP currencies, both the sender and receiver must establish trustlines with the respective issuers.
#### Liquidity
Pathfinding depends on the liquidity of the currency pairs in the XRPL’s order books. Ensure sufficient market depth for transactions.
#### Fees
Each path incurs minimal fees, determined by the order book offers and XRPL’s transaction costs.
#### Currency Conversion
The conversion rate depends on the offers available on the DEX. Using higher-liquidity pairs generally results in better rates.
***
## Advantages in Multi-Currency Ecosystems
Pathfinding is a cornerstone of XRPL’s utility in financial ecosystems that require:
1. **International Remittances**\
Facilitates seamless currency exchanges for cross-border payments.
2. **Decentralized Finance (DeFi)**\
Powers complex financial workflows by automating currency conversions.
3. **Multi-Currency Wallets**\
Enhances user experience by enabling payments in any supported currency.
Pathfinding on XRPL underscores the ledger’s commitment to efficiency and accessibility in global financial transactions. By leveraging its DEX and automated routing, XRPL ensures that payments are seamless, cost-effective, and scalable across diverse currencies.
# Escrow
In this session, we will create an escrow.
An escrow on the XRP Ledger allows users to lock XRP and release it under specific conditions. This feature enables conditional payments, ensuring funds are only transferred when criteria are met. If conditions aren't fulfilled, the escrow can be canceled, returning the funds to the sender.
## Start coding
Create a new file or edit `index.ts`
```typescript
import dayjs from 'dayjs';
import { Client, isoTimeToRippleTime, xrpToDrops } from 'xrpl';
import { generateConditionAndFulfillment, escrowTransaction } from './helpers';
const main = async () => {
console.log('lets get started...');
// Connect the client to the network
const client = new Client('wss://s.altnet.rippletest.net:51233');
await client.connect();
const { wallet: walletOne } = await client.fundWallet();
const { wallet: walletTwo } = await client.fundWallet();
console.log({ walletOne, walletTwo });
};
main();
```
## Create the escrow
In order to create an escrow on the XRP Ledger, you need to specify the amount of XRP to lock, the destination account, and the conditions for release, such as a time-based or cryptographic condition. Additionally, you must ensure the transaction is properly signed and submitted to the network, and verify its success to confirm the escrow is active.
```typescript
// Time after which the destination user can claim the funds
const WAITING_TIME = 10; // seconds
// Define the time from when the Destination wallet can claim the money in the escrow. So here it would be 10 seconds after the escrow creation.
const finishAfter = dayjs().add(WAITING_TIME, 'seconds').toISOString();
// Generate the condition and fulfillment
const { condition, fulfillment } = generateConditionAndFulfillment();
const escrowCreateResponse = await escrowTransaction({
txn: {
Account: walletOne.address,
TransactionType: 'EscrowCreate',
Amount: xrpToDrops('1'),
Destination: walletTwo.address,
FinishAfter: isoTimeToRippleTime(finishAfter),
Condition: condition,
},
client,
wallet: walletOne,
});
// We need the sequence to finish an escrow, if it is not there, stop the function
if (!escrowCreateResponse.result.Sequence) {
await client.disconnect();
return;
}
```
Create a `helpers.ts` file and add the generateConditionAndFulfillment and escrowTransaction function:
```typescript
import crypto from 'crypto';
import {
Client,
EscrowCreate,
EscrowFinish,
EscrowCancel,
Wallet,
Transaction,
} from 'xrpl';
// @ts-expect-error no types available
import cc from 'five-bells-condition';
export const generateConditionAndFulfillment = () => {
console.log(
"******* LET'S GENERATE A CRYPTO CONDITION AND FULFILLMENT *******"
);
console.log();
// use cryptographically secure random bytes generation
const preimage = crypto.randomBytes(32);
const fulfillment = new cc.PreimageSha256();
fulfillment.setPreimage(preimage);
const condition = fulfillment
.getConditionBinary()
.toString('hex')
.toUpperCase();
console.log('Condition:', condition);
// Keep secret until you want to finish the escrow
const fulfillment_hex = fulfillment
.serializeBinary()
.toString('hex')
.toUpperCase();
console.log(
'Fulfillment (keep secret until you want to finish the escrow):',
fulfillment_hex
);
console.log();
return {
condition,
fulfillment: fulfillment_hex,
};
};
export type TransactionProps = {
txn: T;
client: Client;
wallet: Wallet;
};
export const escrowTransaction = async ({
txn,
client,
wallet,
}: TransactionProps) => {
const escrowResponse = await client.submitAndWait(txn, {
autofill: true,
wallet,
});
console.log(JSON.stringify(escrowResponse, null, 2));
return escrowResponse;
};
```
## Finishing the escrow
To finish an escrow on the XRP Ledger, you must wait until the specified conditions are met, such as a time-based or cryptographic condition. Then, submit an "EscrowFinish" transaction, providing the necessary details like the condition, fulfillment, and sequence number, to release the locked funds to the designated recipient.
```typescript
// Wait "WAITING_TIME" seconds before finishing the escrow
console.log(`Waiting ${WAITING_TIME} seconds`);
const sleep = (ms: number) => {
return new Promise((resolve) => setTimeout(resolve, ms));
};
await sleep(WAITING_TIME * 1000);
await escrowTransaction({
txn: {
Account: walletTwo.address,
TransactionType: 'EscrowFinish',
Condition: condition,
Fulfillment: fulfillment,
OfferSequence: escrowCreateResponse.result.Sequence,
Owner: walletOne.address,
},
client,
wallet: walletTwo, // Make sure this is the wallet which was in the "Destination" field during the escrow creation
});
console.log('Escrow transaction sent successfully');
await client.disconnect();
```
After this step, the escrow transaction has been successfully submitted to the XRP Ledger, signaling the completion of the escrow process. The funds should now be released to the designated recipient, provided all conditions have been met. The client is then disconnected from the network, indicating the end of the transaction session.
## Extra credits
[The training in video](https://youtu.be/Uw5eedmXVbE?si=G36gvwRNUr7aLlWV)
### Canceling your escrow
If the conditions aren't met, you can cancel the escrow with the EscrowCancel transaction type:
```typescript
await escrowTransaction({
txn: {
Account: walletOne.address, // The account submitting the cancel request
TransactionType: 'EscrowCancel',
Owner: walletOne.address, // The account that created the escrow
OfferSequence: escrowCreateResponse.result.Sequence, // The sequence number of the EscrowCreate transaction
},
client,
wallet: walletOne, // The wallet of the account that created the escrow
});
```
[Learn more about escrow cancel](https://xrpl.org/docs/references/protocol/transactions/types/escrowcancel)
### Use the escrow as a smart contract
You can use XRP Ledger escrows as smart contracts that release XRP after a certain time has passed or after a cryptographic condition has been fulfilled.
[Learn about escrow as a smart contract](https://xrpl.org/docs/tutorials/how-tos/use-specialized-payment-types/use-escrows/use-an-escrow-as-a-smart-contract)
# Price Oracles
[Price Oracles](https://xrpl.org/docs/concepts/decentralized-storage/price-oracles) on the XRP Ledger bring real-world pricing data on-chain—bridging external market data with decentralized applications (dApps). This feature enables dApps, decentralized exchanges, and DeFi protocols to function with up-to-date, verifiable price feeds.
***
### Price Oracles Explained
#### What Are Price Oracles?
Price Oracles are on-chain ledger objects that store external price information (e.g., XRP/USD). They are created and updated via the **OracleSet** transaction, and can be removed using the **OracleDelete** transaction. dApps can access this data to trigger smart contract logic, perform asset conversions, or maintain stablecoins.
#### Why Use Price Oracles?
* **Reliable Data Feeds**: Provide verified, real-time pricing data from trusted providers.
* **Decentralized Trust**: Price data is recorded on the immutable XRPL ledger.
* **Data Aggregation**: Multiple oracle instances can be aggregated to calculate mean, median, or trimmed mean values, reducing the impact of outliers.
#### How Price Oracles Work
The PriceOracle object includes fields such as `Owner`, `Provider`, `PriceDataSeries`, and `LastUpdateTime`. You use an **OracleSet** transaction to create or update an oracle instance, while the **OracleDelete** transaction removes it from the ledger. Additionally, the `get_aggregate_price` API allows for combining data from multiple Price Oracle objects to generate reliable price statistics.
***
### Advantages of Price Oracles
1. **Real-World Data Integration**\
Seamlessly integrate external pricing data into on-chain applications.
2. **Enhanced Transparency and Security**\
Data stored on XRPL is verifiable and tamper-resistant.
3. **Robust Data Aggregation**\
Aggregating data from several oracles minimizes anomalies and improves reliability.
4. **Dynamic Data Management**\
Easily update or remove price feeds with dedicated transactions.
***
## Price Oracles in Action
### Prerequisites
Before working with Price Oracles, ensure you have:
* An active **Owner** account on the XRPL that meets the XRP reserve requirements.
* Sufficient XRP to cover transaction fees.
* Trust in your external data provider (e.g., a reputable API or on-chain oracle service).
#### Initialize the Client and Wallet
Begin by installing the XRPL library:
```bash
npm install xrpl
```
Then, initialize your XRPL client and wallet:
```javascript
import { Client, Wallet } from "xrpl";
const client = new Client("wss://xrplcluster.com");
await client.connect();
// Initialize your owner wallet (replace with your actual secret)
const ownerWallet = Wallet.fromSeed("sXXXXXXXXXXXXXXXXXXXXXXXXXXXX");
```
### Example: Creating/Updating a Price Oracle
Submit an **OracleSet** transaction to create or update a Price Oracle instance:
```javascript
const oracleSetTx = {
TransactionType: "OracleSet",
Account: ownerWallet.classicAddress,
OracleDocumentID: 34, // Unique identifier for this Price Oracle instance
Provider: "70726F7669646572", // Hex-encoded provider identifier (e.g., "provider")
AssetClass: "63757272656E6379", // Hex-encoded asset class (e.g., "currency")
LastUpdateTime: Math.floor(Date.now() / 1000), // Current Unix time
PriceDataSeries: [
{
PriceData: {
BaseAsset: "XRP",
QuoteAsset: "USD",
AssetPrice: 740, // Example: represents 7.40 with a Scale of 2
Scale: 2
}
}
]
};
try {
const response = await client.submitAndWait(oracleSetTx, { autofill: true, wallet: ownerWallet });
console.log("OracleSet Transaction Result:", response);
} catch (error) {
console.error("Failed to submit OracleSet transaction:", error);
}
```
### Retrieving a Price Oracle
Use the `ledger_entry` API to fetch the on-chain Price Oracle object:
```javascript
const ledgerEntryRequest = {
method: "ledger_entry",
oracle: {
account: ownerWallet.classicAddress,
oracle_document_id: 34
},
ledger_index: "validated"
};
const ledgerEntryResponse = await client.request(ledgerEntryRequest);
console.log("Retrieved Price Oracle:", ledgerEntryResponse.result.node);
```
#### Convert AssetPrice to Integer
To get a decimal representation of the AssetPrice convert it from HEX to a Number:
```javascript
const AssetPriceHex = '0x' + '2e4'
const price = Number(AssetPriceHex)
```
### Aggregating Price Data
Aggregate data from multiple oracles with the `get_aggregate_price` API:
```javascript
const aggregatePriceRequest = {
method: "get_aggregate_price",
ledger_index: "current",
base_asset: "XRP",
quote_asset: "USD",
trim: 20, // Trim 20% of outlier data
oracles: [
{
account: ownerWallet.classicAddress,
oracle_document_id: 34
}
// Include additional oracle objects as needed
]
};
const aggregatePriceResponse = await client.request(aggregatePriceRequest);
console.log("Aggregated Price Data:", aggregatePriceResponse.result);
```
### Additionally Delete the Oracle Object
To delete the Oracle Object from your Account perform an `OracleDelete` Transaction.
```javascript
const oracleDeleteTx = {
TransactionType: "OracleDelete",
Account: ownerWallet.classicAddress,
OracleDocumentID: 34
};
try {
const response = await client.submitAndWait(oracleDeleteTx, { autofill: true, wallet: ownerWallet });
console.log("OracleDelete Transaction Result:", response);
} catch (error) {
console.error("Failed to submit OracleDelete transaction:", error);
}
```
***
## Key Considerations
#### XRP Reserve Requirements
* Ensure your account meets the reserve requirement, especially when including more than five token pairs in `PriceDataSeries`.
#### Data Accuracy and Validity
* Validate that the `LastUpdateTime` reflects current pricing data.
* Maintain consistency for the `Provider` and `AssetClass` fields across updates.
#### Transaction Signing
* All OracleSet and OracleDelete transactions must be signed by the owner account or its authorized multi-signers.
***
## Advantages in Financial Applications
Price Oracles empower the XRPL ecosystem by enabling:
1. **Decentralized Finance (DeFi)**\
Utilize real-world price data to support lending, derivatives, and other DeFi products.
2. **Stablecoin Management**\
Maintain stablecoin peg values with aggregated price feeds.
3. **Efficient Trading and DEX Integration**\
Provide reliable market data for order matching and automated market making.
By integrating Price Oracles, developers can build robust, data-driven applications that benefit from transparent and up-to-date market insights.
***
Remember to disconnect your client after operations:
```javascript
await client.disconnect();
```
Happy building with Price Oracles on XRPL!
# Tickets
Ticket Feature on XRPL
## Introduction
Tickets on the XRPL allow you to reserve sequence numbers for transactions, enabling asynchronous transaction submission. This feature is particularly useful in multi-signature setups, where sequence management can become challenging due to independent signers.
***
## Tickets Explained
### **What Are Tickets?**
Tickets are a feature on the XRP Ledger that allow you to preallocate sequence numbers for future transactions. This helps avoid sequence number conflicts, especially in scenarios where multiple parties (like signers in a MultiSignature setup) need to sign a transaction.
### **Why Use Tickets?**
In standard transactions, the Sequence field determines the order of execution. When signers asynchronously sign transactions, the sequence number may become outdated. Tickets provide a solution by reserving sequence numbers in advance, ensuring transactions can proceed without conflict.
### **How Tickets Work**
Tickets are created using the TicketCreate transaction. Once created, these tickets can be used in place of the Sequence field in any subsequent transaction.
***
## Advantages of Tickets
1. Avoiding Sequence Conflicts
Tickets prevent sequence number mismatches in multi-signature transactions, where signers may not act in a synchronized manner.
2. Streamlining MultiSignature
By reserving sequence numbers, Tickets simplify the signing and submission process in MultiSignature environments.
3. Flexibility for Transaction Management
Tickets can be used to prioritize certain transactions or reserve future transaction slots, offering better control over transaction flow.
## Ticket Setup
### Prerequisites
To create a Ticket on the XRPL, your account must hold sufficient XRP to meet the reserve requirements for the associated account object.
### Create Wallet
```javascript
import { Client, Wallet } from "xrpl";
const client = new Client("wss://s.altnet.rippletest.net:51233");
await client.connect();
const wallet = Wallet.fromSeed("s...");
```
### Retrieve Tickets
You can query your account to retrieve active tickets.
```javascript
const accountObjects = await client.request({
command: "account_objects",
account: wallet.classicAddress,
type: "ticket"
});
console.log('Active Tickets:', accountObjects.result.account_objects);
// tickets [
// {
// Account: 'rHH6GByFtaKXXAEeue4myzFuq4ftAZW9un',
// TicketSequence: 2856829
// }
// ]
```
### Using Tickets in Transactions
Once you have created tickets, you can use them in subsequent transactions by specifying the TicketSequence field instead of Sequence.
```javascript
const ticketSequence = accountObjects.result.account_objects[0].TicketSequence;
const transaction = {
TransactionType: "Payment",
Account: wallet.classicAddress,
Destination: "rrrrrrrrrrrrrrrrrNAMEtxvNvQ",
Amount: '1000000', // 1 XRP in drops
Sequence: 0,
TicketSequence: ticketSequence // Use the ticket's sequence number
};
try {
const result = await client.submit(transaction, { autofill: true, failHard: true, wallet });
console.log('Transaction Submit Result:', result);
} catch (error) {
console.error('Failed to submit transaction:', error);
}
```
### Optional Remove a Ticket
Deleting a Ticket is sometimes required. To free up the associated Account Object. The way to do this is generate a no-op transaction
```javascript
const ticketToDelete = accountObjects.result.account_objects[0].TicketSequence;
const noopTransaction = {
TransactionType: "AccountSet",
Account: wallet.classicAddress,
TicketSequence: ticketToDelete
};
try {
const result = await client.submit(noopTransaction, { autofill: true, failHard: true, wallet });
console.log('No-Op Transaction Result:', result);
} catch (error) {
console.error('Failed to delete ticket:', error);
}
```
### Explanation of the Script
1. Ticket Creation
* The TicketCreate transaction reserves sequence numbers for future transactions.
* Specify the number of tickets you want to create using the TicketCount field.
2. Querying Tickets
* Use the account\_objects command to fetch active tickets linked to your account.
3. Using Tickets
* Include the TicketSequence field in your transaction to use a ticket instead of a regular sequence number.
### Key Considerations
#### **Cost of Tickets**
Each ticket costs 2 XRP to reserve. Ensure you have sufficient XRP balance in your account before creating tickets.
#### **Ticket Expiry**
Tickets remain valid until used or until the account’s Sequence surpasses the ticket’s TicketSequence.
#### **Managing Tickets**
Efficiently manage tickets to avoid unnecessary fees and ensure smooth transaction workflows.
#### **Combining Tickets with MultiSign**
Tickets are particularly effective in MultiSignature setups, preventing sequence mismatches and allowing independent signers to collaborate seamlessly.
With the Ticket feature, XRPL provides a powerful mechanism to manage transaction order and avoid sequence conflicts, especially in asynchronous environments like MultiSignature. By leveraging Tickets, you can ensure smooth and conflict-free transaction workflows.
# Multi-Signature
In this session, we will work with multiple signatures
## Introduction
MultiSignature (or MultiSig) on the XRPL allows multiple users to collectively authorize transactions from a shared account. This ensures enhanced security and reduces risks associated with a single point of control. Think of it like a group-controlled vault where multiple stakeholders must approve access.
***
## MultiSign Explained
### 1. Configuration and Setup
MultiSignature starts with a shared account on the XRPL. This account is configured with specific rules, defining how many and which signers are required for transaction approval. Each signer is assigned a "weight," reflecting their influence in the decision-making process.
### 2. Establishing a Quorum
The quorum is the minimum number of signers required to authorize a transaction. For example, a quorum of three means at least three pre-approved parties must sign off for the transaction to proceed.
### 3. Transaction Authorization
When a transaction is initiated, it must receive the necessary number of approvals as specified in the MultiSignature configuration. For instance, if the quorum requires two out of three signers, the transaction will execute only after two parties provide their signatures.
### 4. Enhanced Security Features
MultiSignature ensures that even if one signer's private key is compromised, unauthorized transactions cannot occur without meeting the quorum. This reduces risks associated with fraud or hacking.
### 5. Customizable Flexibility
MultiSignature is highly adaptable, allowing configurations to be updated over time. For example, fewer signers can be required for everyday transactions, but unanimous approval might be enforced for high-value transfers.
***
## Why MultiSignature Stands Out
### 1. Distributed Control and Governance
MultiSignature decentralizes decision-making power by requiring consensus from multiple signers. This ensures no single individual has complete control, making it ideal for businesses, joint ventures, and collaborative setups.
### 2. Enhanced Security Against Threats
By requiring multiple approvals, MultiSignature significantly reduces the risk of unauthorized transactions, even if one key is compromised. This layered security is a robust defense against hacking and fraud.
### 3. Fostering Trust in Shared Transactions
MultiSignature enhances trust in shared financial operations by ensuring transparency and consensus among all parties involved.
### 4. Risk Mitigation for Organizations
MultiSignature mitigates risks such as insider threats and key compromises by spreading responsibility across multiple signers.
### 5. Customizable for Your Needs
Configurations can be tailored to operational requirements, such as setting different approval thresholds for routine versus high-value transactions.
***
### What are we going to do
* **Wallet Initialization**\
Create wallets for the main account and the signers using their seeds.
* **Setting Up the Signer List**\
Configure the signer list with their respective weights and set the quorum.
* **Submitting the SignerListSet Transaction**\
Send the signer list configuration to the XRPL.
* **Creating the Payment Transaction**\
Define the payment transaction that will be multi-signed.
* **Autofilling Transaction Details**\
Use autofill to complete transaction fields like `Sequence` and `Fee`.
* **Signing the Transaction by Each Signer**\
Each signer independently signs the transaction.
* **Combining the Signatures**\
Use `multisign` to combine the individual signatures into one transaction.
* **Submitting the MultiSigned Transaction**\
Submit the combined transaction to the XRPL.
### Setup
Accounts
For this example you need a total of four accounts:
1. One main account
2. Three signers
The signers do not require funds but must enable the master key.
* **SignerQuorum**: The minimum weight sum required for valid transactions.
* **SignerEntries**: A list with at least one member and no more than 32 members.
***
#### Create Wallets
```javascript
import { Client, Wallet, multisign } from "xrpl";
const client = new Client("wss://s.altnet.rippletest.net:51233");
await client.connect();
const mainSeed = "s...";
const signerSeed_1 = "s...";
const signerSeed_2 = "s...";
const signerSeed_3 = "s...";
const mainWallet = Wallet.fromSeed(mainSeed);
const signer_1 = Wallet.fromSeed(signerSeed_1);
const signer_2 = Wallet.fromSeed(signerSeed_2);
const signer_3 = Wallet.fromSeed(signerSeed_3);
// Fund the main wallet only (testnet)
await client.fundWallet(mainWallet);
```
**Generate Wallets instead**
Alternatively you can create Wallets dynamically:
```javascript
const mainWallet = Wallet.generate();
```
#### Create and Set the Signer List
With the accounts we just created we want to configure our signerlist and submit this to the ledger, we do this by creating a [SignerListSet transaction](https://xrpl.org/docs/references/protocol/transactions/types/signerlistset).
```javascript
const SignerListSetTransaction = {
TransactionType: "SignerListSet",
Account: mainWallet.classicAddress,
SignerQuorum: 2,
SignerEntries: [
{
SignerEntry: {
Account: signer_1.classicAddress,
SignerWeight: 2
}
},
{
SignerEntry: {
Account: signer_2.classicAddress,
SignerWeight: 1
}
},
{
SignerEntry: {
Account: signer_3.classicAddress,
SignerWeight: 1
}
}
]
};
try {
const result = await client.submitAndWait(SignerListSetTransaction, { autofill: true, failHard: true, wallet: mainWallet });
console.log('SignerListSet Transaction Result:', result);
} catch (error) {
console.error('Failed to submit transaction:', error);
}
```
### Creating a Transaction
For creating transactions in a multi-sign environment, you have two options: either use [**Sequence**](https://xrpl.org/docs/references/protocol/data-types/basic-data-types#account-sequence) or a [**Ticket**](https://xrpl.org/docs/concepts/accounts/tickets). Tickets are used to reserve sequence numbers. Since MultiSign transactions are asynchronous, the sequence number the transactions have been based on might be outdated. In this example, we do not use Tickets.
#### Create Transaction
In this example, we prepare a simple payment transaction to be later signed using MultiSignature. The transaction sends 1 XRP (in drops) from the main account to a specified destination address.
```javascript
const transaction = {
TransactionType: "Payment",
Account: mainWallet.classicAddress,
Destination: "rrrrrrrrrrrrrrrrrNAMEtxvNvQ",
Amount: '1000000' // 1 XRP expressed in drops
};
```
#### Prepare the transaction
The fee and Sequence needs to be set in advance. The calculation for it goes as follows: (Base Fee + (Incremental Fee × Number of Signers)).\
We can use [`.autofill()`](https://js.xrpl.org/classes/Client.html#autofill) for this, 3 is the number of signers.
```javascript
const autofilledTransaction = await client.autofill(transaction, 3);
console.log('Transaction to sign:', autofilledTransaction);
```
#### Signing the Transaction
Each signer will [sign](https://js.xrpl.org/classes/Wallet.html#sign) the same Transaction
```javascript
const signedWallet_1 = signer_1.sign(autofilledTransaction, true);
const signedWallet_2 = signer_2.sign(autofilledTransaction, true);
const signedWallet_3 = signer_3.sign(autofilledTransaction, true);
console.log('Signed by Signer 1:', signedWallet_1);
// Sample Output:
// {
// tx_blob: '1200002200000000...F1',
// hash: '5C4AC6A8FF7E3B...'
// }
```
#### Combine the signatures
[Combine](https://js.xrpl.org/functions/multisign.html) all signatures for one final Transaction
```javascript
const combinedTransaction = multisign([signedWallet_1.tx_blob, signedWallet_2.tx_blob, signedWallet_3.tx_blob]);
console.log(combinedTransaction);
// '12000022000......97A6AEE1F1'
```
#### Submit
[Submit](https://js.xrpl.org/classes/Client.html#submit) the multi-signed transaction
```javascript
try {
const result = await client.submitAndWait(combinedTransaction);
console.log('Transaction Submit Result', result);
// Sample Output:
// {
// result: {
// accepted: true,
// engine_result: 'tesSUCCESS',
// tx_json: { ... },
// validated_ledger_index: ...
// }
// }
} catch (error) {
console.error('Failed to submit transaction:', error);
}
```
***
### Key Points to Remember
1. **Signer Weights and Quorum**
* The sum of the signer weights must meet or exceed the `SignerQuorum` for a transaction to be valid.
* In the example, Signer 1 has a weight of 2, while Signers 2 and 3 have a weight of 1 each.
2. **Fee Calculation**
* Multi-signed transactions require higher fees due to their increased complexity.
* Ensure the fee is sufficient to cover the multi-signature. In this example we used `autofill(transaction, 3)` specifying 3 signers for the Fee calculation.
3. **Transaction Sequence or Tickets**
* Multi-signing is asynchronous, so using Tickets can help manage sequence numbers. More information here:
* In this example, we use the Sequence number provided by `autofill`.
4. **Asynchronous Signing**
* Each signer can sign the transaction independently and at different times.
5. **Security Considerations**
* MultiSignature enhances security by requiring multiple keys to authorize a transaction.
* Even if one key is compromised, the attacker cannot perform unauthorized transactions without additional signatures, if the signer list is correctly set.
***
With MultiSignature, you can ensure robust security and reliable transaction governance on the XRPL.
# Token Issuance and Liquidity
This session will cover token issuance and providing liquidity for your tokens using the automatic market maker built in to the XRPL
# Creating Accounts and Setup
Before we get into the heart of the matter, lets setup a few wallets.
## Creating wallets
Let's add some wallet creation logic. We might as well create two wallets for some fun.
```javascript
console.log('lets fund 2 accounts...')
const { wallet: wallet1, balance: balance1 } = await client.fundWallet()
const { wallet: wallet2, balance: balance2 } = await client.fundWallet()
console.log('wallet1', wallet1)
console.log('wallet2', wallet2)
console.log({
balance1,
address1: wallet1.address, //wallet1.seed
balance2,
address2: wallet2.address
})
```
At this point, we should have two wallets with balances of 100 XRP.
We will save these in a more convenient way to reuse them as we progress through this tutorial.
Collect the seed values from the logs for both accounts, and let's create wallets from those seeds from now on. We'll need an issuer and a receiver so here we go:
First, we set the seed in the code
```javascript
const issuerSeed = "s...";
const receiverSeed = "s...";
```
Then we create issuer and receiver wallets from the seeds in the main function as follows:
```javascript
const issuer = Wallet.fromSeed(issuerSeed);
const receiver = Wallet.fromSeed(receiverSeed);
```
Putting it all together we have the following
{% code title="index.ts" %}
```ts
import { Client, Wallet } from "xrpl"
const client = new Client("wss://s.altnet.rippletest.net:51233")
const issuerSeed = "s...";
const receiverSeed = "s...";
const issuer = Wallet.fromSeed(issuerSeed);
const receiver = Wallet.fromSeed(receiverSeed);
const main = async () => {
console.log("lets get started...");
await client.connect();
// do something interesting here
await client.disconnect();
console.log("all done!");
};
main();
```
{% endcode %}
# Issuing Tokens
In this section we will create fungible tokens or IOUs and transfert them between accounts.
## Creating tokens
### **Dealing with token codes**
Token codes can be three letters or a padded hex string. Here is a convenience function to enable codes longer than three letters. This needs to be done to work with our training app, so you will want to include this in your code. This could easily be done via an import for instance, or placed in the index.ts file directly.
Here is the function code:
```javascript
const convertStringToHexPadded = (str: string): string => {
// Convert string to hexadecimal
let hex: string = "";
for (let i = 0; i < str.length; i++) {
const hexChar: string = str.charCodeAt(i).toString(16);
hex += hexChar;
}
// Pad with zeros to ensure it's 40 characters long
const paddedHex: string = hex.padEnd(40, "0");
return paddedHex.toUpperCase(); // Typically, hex is handled in uppercase
}
```
### **Enable rippling**
For AMMs and the whole process to work, we need to enable rippling from the issuer account. To enable rippling we use the AccountSet transaction with the appropriate flag. Here is the function.
```javascript
const enableRippling = async ({ wallet, client }: any) => {
const accountSet: AccountSet = {
TransactionType: "AccountSet",
Account: wallet.address,
SetFlag: AccountSetAsfFlags.asfDefaultRipple,
};
const prepared = await client.autofill(accountSet);
const signed = wallet.sign(prepared);
const result = await client.submitAndWait(signed.tx_blob);
console.log(result);
console.log("Enable rippling tx: ", result.result.hash);
return;
}
```
We will add these functions to a utility file.
{% code title="utils.ts" %}
```typescript
import { AccountSet, AccountSetAsfFlags } from "xrpl"
export const convertStringToHexPadded = (str: string): string => {
// Convert string to hexadecimal
let hex: string = "";
for (let i = 0; i < str.length; i++) {
const hexChar: string = str.charCodeAt(i).toString(16);
hex += hexChar;
}
// Pad with zeros to ensure it's 40 characters long
const paddedHex: string = hex.padEnd(40, "0");
return paddedHex.toUpperCase(); // Typically, hex is handled in uppercase
}
export const enableRippling = async ({ wallet, client }: any) => {
const accountSet: AccountSet = {
TransactionType: "AccountSet",
Account: wallet.address,
SetFlag: AccountSetAsfFlags.asfDefaultRipple,
};
const prepared = await client.autofill(accountSet);
const signed = wallet.sign(prepared);
const result = await client.submitAndWait(signed.tx_blob);
console.log(result);
console.log("Enable rippling tx: ", result.result.hash);
return;
}
```
{% endcode %}
And in the main function of `index.ts` we can now add the following code, to trigger this functionality.
{% code title="index.ts" %}
```javascript
// enable ripling
await enableRippling({ wallet: issuer, client });
```
{% endcode %}
### **Creating our issued token**
To create an issued token, the receiver first needs to add a trust line to the issuer. It's straightforward, we create a TrustSet transaction and sign it with the receiver account.
Once the trustline is set, we send an amount from the issuer to the receiver that is less than the trust line maximum (500M tokens in this case).
Here is the whole file for `createToken.ts`
{% code title="createToken.ts" %}
```typescript
import { TrustSet, TrustSetFlags } from "xrpl";
import { Payment } from "xrpl/src/models";
export const createToken = async ({ issuer, receiver, client, tokenCode }: any) => {
// Create the trust line to send the token
const trustSet: TrustSet = {
TransactionType: "TrustSet",
Account: receiver.address,
LimitAmount: {
currency: tokenCode,
issuer: issuer.address,
value: "500000000", // 500M tokens
},
Flags: TrustSetFlags.tfClearNoRipple,
};
console.log(trustSet);
// Receiver opening trust lines
const preparedTrust = await client.autofill(trustSet);
const signedTrust = receiver.sign(preparedTrust);
const resultTrust = await client.submitAndWait(signedTrust.tx_blob);
console.log(resultTrust);
console.log("Trust line issuance tx result: ", resultTrust.result.hash);
// Send the token to the receiver
const sendPayment: Payment = {
TransactionType: "Payment",
Account: issuer.address,
Destination: receiver.address,
Amount: {
currency: tokenCode,
issuer: issuer.address,
value: "200000000", // 200M tokens
},
};
console.log(sendPayment);
const preparedPayment = await client.autofill(sendPayment);
const signedPayment = issuer.sign(preparedPayment);
const resultPayment = await client.submitAndWait(signedPayment.tx_blob);
console.log(resultPayment);
console.log("Transfer issuance tx result: ", resultPayment.result.hash);
return;
}
```
{% endcode %}
We can now call this function from the main function inside`index.ts`, remembering to wrap our token currency code with the `convertStringToHexPadded` function.
We can check in the explorer that the issuer and receiver have balances in the new token at this point.
# Creating an AMM Pool
Now we will create an AMM Pool to provide some liquidity for our new token.
## Create an AMM Pool
Now that the receiver has tokens, we can use the receiver's account to create an AMM. Note that this is the usual architecture where the issuer account solely issues the token, and other proprietary accounts hold the token and can create liquidity.
For this example, we will use pools that have XRP as one side.
Here is the `createAMM.ts` file:
{% code title="createAMM.ts" %}
```javascript
import { AMMCreate } from "xrpl";
export const createAMM = async ({
issuer,
receiver,
client,
tokenCode,
}: any) => {
console.log("create AMM", { issuer, receiver, tokenCode });
let createAmm: AMMCreate = {
TransactionType: "AMMCreate",
Account: receiver.address,
TradingFee: 600,
Amount: {
currency: tokenCode,
issuer: issuer.classicAddress,
value: "2000000", // 2M tokens
},
Amount2: "50000000", // 50 XRP in drops
};
console.log(createAmm);
const prepared = await client.autofill(createAmm);
const signed = receiver.sign(prepared);
const result = await client.submitAndWait(signed.tx_blob);
console.log(result);
console.log("Create amm tx: ", result.result.hash);
return;
};
```
{% endcode %}
Putting it all together, the final main function and `index.ts` should now look like this:
{% code title="index.ts" %}
```javascript
import { Client, Wallet } from "xrpl"
import { createToken } from "./createToken"
import { createAMM } from "./createAMM"
import { convertStringToHexPadded, enableRippling } from "./utils"
const client = new Client("wss://s.altnet.rippletest.net:51233")
// get these from https://xrpl.org/resources/dev-tools/xrp-faucets
const issuerSeed = "s...";
const receiverSeed = "s...;
const issuer = Wallet.fromSeed(issuerSeed);
const receiver = Wallet.fromSeed(receiverSeed);
// edit this to match your vibe
const token = "COOL"
const main = async () => {
console.log("lets get started...");
await client.connect();
// enable ripling
await enableRippling({ wallet: issuer, client });
// create Token
await createToken({
issuer,
receiver,
client,
tokenCode: convertStringToHexPadded(token),
});
// create AMM
await createAMM({
issuer,
receiver,
client,
tokenCode: convertStringToHexPadded(token),
});
await client.disconnect();
console.log("all done!");
};
main();
```
{% endcode %}
Don't forget to import `enableRippling`, `createToken`, `createAMM` and `convertStringToHexPadded` if needed.
You can now go to the training website at [https://amm.xrpl.at/](https://trainings.xrpl.at/) and interact with other pools.
You will need to set up Xaman with the **receiver account** you created above. You can use the family seed import route. If you are new to Xaman don't forget to enable developer mode in the advanced settings and then switch to testnet from the home page of the Xaman app.
#### Interact with your pool (swap)
Using the training app, connect and add your public address to the list. When you click a user in the list you can see that account's token holdings.
You can create Trustlines for tokens you have not interacted with yet. You can swap XRP for other tokens where you have trustlines using the pool view's swap feature.
You may need to mint more XRP for your receiver account. That's where the print money function might come in handy.
## Extra credits
**Become a liquidity provider for other pools**
You can become a Liquidity Provider for other pools by using the AMMDeposit function: ()
**Withdraw from pools**
You can also withdraw from pools using your LP tokens by utilizing the AMMWithdraw transaction. ()
# Cyphered Chat on XRPL
Send encrypted messages over XRPL
This session will explore how to use the memo field present on every transaction on the XRP Ledger to send encrypted messages, effectively creating a chat engine.
While nothing is encrypted forever, we propose this activity to explore the basic principles of cryptography and using the memo field.
# Set up
Description of key libraries
To build our encrypted messaging system, we'll be using these key JavaScript libraries:
* `xrpl`: The official XRPL client library to interact with the XRP Ledger
* `tweetnacl`: For encryption/decryption operations using the NaCl cryptographic library
* `@noble/curves`: For handling elliptic curve operations and key conversions
* `buffer`: To handle binary data manipulation
If you are running your own environment, install everything by running:
```
npm install xrpl tweetnacl @noble/curves buffer
```
# Set up Keys
First, we need to get the public key of the person we want to chat with.\
\
To do so, we can go on the block explorer and look at the transactions the address has sent. From there, by looking at the raw transaction data, we can find their public key.\
\
Once we have their public key, we can use it to encrypt messages that only they can decrypt.
Then we need to set up our wallet to cypher the message and send the tx
```typescript
import {Wallet} from "xrpl"
const receiverPubKey = "the public key from the explorer here"
const myWallet = Wallet.fromSeed("your seed here")
```
# Cypher the message
Now that we have their public key, we can encrypt our secret message.\
Remember our "outside-in" principle: we use THEIR public (outside) key to encrypt the message, which ensures that only THEIR private (inside) key can decrypt it.\
\
The message gets transformed into a scrambled format that is completely unreadable to anyone who might intercept it - only the owner of the matching private key can convert it back to the original text. Each encrypted message is unique, even if you encrypt the same text multiple times, adding an extra layer of security.
```typescript
import tweetnacl from 'tweetnacl';
import { Buffer } from "buffer";
import { Wallet } from "xrpl";
import {
edwardsToMontgomeryPub,
edwardsToMontgomeryPriv,
} from "@noble/curves/ed25519";
const { box, randomBytes } = tweetnacl;
/**
* Encrypts a message using X25519 (Montgomery curve) for Diffie-Hellman key exchange
*
* @param message - Plain text message to encrypt
* @param recipientPublicKey - Recipient's Ed25519 public key (hex encoded)
* @param senderSecretKey - Sender's Ed25519 private key (hex encoded)
* @returns JSON string containing base64 encoded encrypted message and nonce
*
* Steps:
* 1. Convert hex keys to byte arrays
* 2. Generate random nonce for uniqueness
* 3. Convert message to bytes
* 4. Convert Ed25519 keys to X25519 (Montgomery) format for encryption
* 5. Encrypt using NaCl box with converted keys
* 6. Return encrypted message and nonce as base64 JSON
*/
export function encryptMessage(
message: string,
recipientPublicKey: string,
senderSecretKey: string,
): string {
const pubKeyBytes = Buffer.from(recipientPublicKey.slice(2), "hex");
const secretKeyBytes = Buffer.from(senderSecretKey.slice(2), "hex");
const nonce = randomBytes(box.nonceLength);
const messageUint8 = Buffer.from(message);
const pubKeyCurve = edwardsToMontgomeryPub(pubKeyBytes);
const privKeyCurve = edwardsToMontgomeryPriv(secretKeyBytes);
const encryptedMessage = box(messageUint8, nonce, pubKeyCurve, privKeyCurve);
return JSON.stringify({
encrypted: Buffer.from(encryptedMessage).toString("base64"),
nonce: Buffer.from(nonce).toString("base64"),
});
}
function main() {
const recipientPublicKey ="Recipient pubkey goes here";
const myWallet = Wallet.fromSeed("your seed goes here");
const cypheredMessage = encryptMessage(
"Hello World",
recipientPublicKey,
myWallet.privateKey,
);
console.log(cypheredMessage);
}
main();
```
# Set up the memo & send the tx
Now that we have our encrypted message, we can send it through the XRPL network.\
\
We'll create a transaction with a minimal amount (1 drop) and include our encrypted message in the transaction's memo field.\
The memo field is perfect for this as it can store arbitrary data, making it an ideal place for our encrypted message.\
\
Once the transaction is validated, our secret message will be securely stored on the blockchain - visible to everyone but readable only by the intended recipient who holds the matching private key.
```typescript
import tweetnacl from 'tweetnacl';
import { Buffer } from "buffer";
import { Client, Wallet, deriveAddress, Payment } from "xrpl";
import {
edwardsToMontgomeryPub,
edwardsToMontgomeryPriv,
} from "@noble/curves/ed25519";
const { box, randomBytes } = tweetnacl;
/**
* Encrypts a message using X25519 (Montgomery curve) for Diffie-Hellman key exchange
*
* @param message - Plain text message to encrypt
* @param recipientPublicKey - Recipient's Ed25519 public key (hex encoded)
* @param senderSecretKey - Sender's Ed25519 private key (hex encoded)
* @returns JSON string containing base64 encoded encrypted message and nonce
*
* Steps:
* 1. Convert hex keys to byte arrays
* 2. Generate random nonce for uniqueness
* 3. Convert message to bytes
* 4. Convert Ed25519 keys to X25519 (Montgomery) format for encryption
* 5. Encrypt using NaCl box with converted keys
* 6. Return encrypted message and nonce as base64 JSON
*/
export function encryptMessage(
message: string,
recipientPublicKey: string,
senderSecretKey: string,
): string {
const pubKeyBytes = Buffer.from(recipientPublicKey.slice(2), "hex");
const secretKeyBytes = Buffer.from(senderSecretKey.slice(2), "hex");
const nonce = randomBytes(box.nonceLength);
const messageUint8 = Buffer.from(message);
const pubKeyCurve = edwardsToMontgomeryPub(pubKeyBytes);
const privKeyCurve = edwardsToMontgomeryPriv(secretKeyBytes);
const encryptedMessage = box(messageUint8, nonce, pubKeyCurve, privKeyCurve);
return JSON.stringify({
encrypted: Buffer.from(encryptedMessage).toString("base64"),
nonce: Buffer.from(nonce).toString("base64"),
});
}
/**
* Sends an XRPL transaction containing an encrypted message in its memo field
*
* @param cypherMessage - The encrypted message to send
* @param myWallet - Sender's XRPL wallet for signing the transaction
* @param receiverPubKey - Recipient's public key to derive their XRPL address
* @returns Transaction result or undefined if error occurs
*
* Flow:
* 1. Connect to XRPL testnet
* 2. Create Payment transaction:
* - Minimal amount (1 drop)
* - Include encrypted message in memo field
* - Derive recipient's address from their public key
* 3. Prepare, sign and submit transaction
* 4. Wait for validation and return result
* 5. Always disconnect client when done
*
* Note: MemoType is hex encoded "http://example.com/memo/generic"
*/
async function sendTX(
cypherMessage: string,
myWallet: Wallet,
recipientPublicKey: string,
) {
const client = new Client("wss://clio.altnet.rippletest.net:51233/");
try {
await client.connect();
const receiverAddress = deriveAddress(recipientPublicKey);
const tx: Payment = {
TransactionType: "Payment",
Account: myWallet.classicAddress,
Destination: receiverAddress,
Amount: "10000000",
Memos: [
{
Memo: {
MemoData: Buffer.from(cypherMessage).toString("hex"),
},
},
],
};
const prepared = await client.autofill(tx);
const signed = myWallet.sign(prepared);
const result = await client.submitAndWait(signed.tx_blob);
return result;
} catch (error) {
console.log(error);
} finally {
await client.disconnect();
}
}
async function main(): Promise {
const recipientPublicKey = "Recipient public key goes here";
const myWallet = Wallet.fromSeed("Your seed goes here");
const cypherMessage = encryptMessage(
"Hello World",
recipientPublicKey,
myWallet.privateKey,
);
console.log("==============CYPHERED MESSAGE==============");
console.log(cypherMessage);
const tx = await sendTX(cypherMessage, myWallet, recipientPublicKey);
console.log("==============TRANSACTION==============");
console.log(tx);
console.log("all done");
}
main();
```
# Get the message and decypher it
When you receive a 1-drop transaction, it might contain a hidden message for you.\
\
By checking the raw transaction data in your block explorer and looking at the memo field, you can find an encrypted message. You'll see a scrambled string of characters - this is your encrypted message. Since this message was encrypted using your public (outside) key, only your private (inside) key can decrypt it.\
\
By copying this encrypted text and using your private key, you can transform this unreadable data back into the original message that was meant for your eyes only.
```typescript
import tweetnacl from 'tweetnacl';
import { Wallet } from "xrpl";
import {
edwardsToMontgomeryPub,
edwardsToMontgomeryPriv,
} from "@noble/curves/ed25519";
const { box } = tweetnacl;
export function decryptMessage(
messageWithNonce: string,
recipientPublicKey: string,
senderSecretKey: string,
): string {
const pubKeyBytes = Buffer.from(recipientPublicKey.slice(2), "hex");
const secretKeyBytes = Buffer.from(senderSecretKey.slice(2), "hex");
const pubKeyCurve = edwardsToMontgomeryPub(pubKeyBytes);
const privKeyCurve = edwardsToMontgomeryPriv(secretKeyBytes);
console.log(messageWithNonce)
const { encrypted, nonce } = JSON.parse(messageWithNonce);
const messageBytes = Buffer.from(encrypted,"base64");
const nonceBytes = Buffer.from(nonce,"base64");
const decryptedMessage = box.open(
messageBytes,
nonceBytes,
pubKeyCurve,
privKeyCurve,
);
if (!decryptedMessage) {
throw new Error("Failed to decrypt message");
}
return new TextDecoder().decode(decryptedMessage);
}
function main() {
const cypherMessage =
"Your cypher message goes here";
const senderPubKey =
"The sender public key goes here";
const mySeed = "your seed goes here";
const myPrivateKey = Wallet.fromSeed(mySeed).privateKey;
const clearMessage = decryptMessage(
Buffer.from(cypherMessage, "hex").toString(),
senderPubKey,
myPrivateKey,
);
console.log("==============CLEAR MESSAGE==============");
console.log(clearMessage);
console.log("all done");
}
main();
```
# EVM Sidechain
This workshop will walk you through deploying a smart contract and building a full app on the XRPL EVM sidechain
## EVM workshop Jan 2024
### Documentation:
| Resource | Link |
| --------------------------------------- | ----------------------------------------------------------------------------- |
| Getting started with the EVM sidechain: | |
| EVM Docs | |
| Peersyst Docs | |
| Faucet & Bridge | |
| Explorer | |
### Connect Metamask
#### EVM Network for Metamask
Field
Value
Network Name
XRPL EVM Sidechain
New RPC URL
https://rpc-evm-sidechain.xrpl.org
Chain ID
1440002
Currency Symbol
XRP
Block Explorer
https://evm-sidechain.xrpl.org
### Bridge Funds
You will need to use the bridge tool to fund the EVM sidechain account
### Mint Smart Contract
There are many ways, here are a few\...
#### -> Using Forge
#### -> Using Remix
-> then link into remix and publish with Metamask (
#### -> Using Hardhat
### Using the contract
#### -> Using React and hardhat
#### -> Using Web3
This workshop is based on Floren Uzio's Coding on the XRPL Ledger series.
{% embed url="" %}
XRPL EVM Sidechain
{% endembed %}
# Connecting Metamask
### Connect Metamask
#### EVM Network for Metamask
Field
Value
Network Name
XRPL EVM Sidechain
New RPC URL
https://rpc-evm-sidechain.xrpl.org
Chain ID
1440002
Currency Symbol
XRP
Block Explorer
https://evm-sidechain.xrpl.org
###
# Set up a Foundry project
In this workshop we will learn how to create an evm project with hardhat
This guide will walk you through setting up an EVM project on the XRPL sidechain using Foundry, a powerful toolkit for Ethereum application development.
## Part 1: Installing Foundry
Foundry is a blazing fast, portable and modular toolkit for Ethereum application development written in Rust. It consists of four main tools:
* **Forge**: Ethereum testing framework
* **Cast**: Swiss army knife for interacting with EVM smart contracts
* **Anvil**: Local Ethereum node for development
* **Chisel**: Fast, utilitarian, and verbose solidity REPL
### Prerequisites
Before installing Foundry, ensure you have:
* A terminal application (Git BASH or WSL for Windows users)
* Internet connection for downloading the installer
### Installation Methods
#### Option 1: Using Foundryup (Recommended)
Foundryup is the official installer for the Foundry toolchain and the easiest way to get started.
1. **Install Foundryup**
Open your terminal and run:
```bash
curl -L https://foundry.paradigm.xyz | bash
```
2. **Follow the on-screen instructions** to complete the installation
3. **Install Foundry tools**
Run the following command to install the latest stable version:
```bash
foundryup
```
For the latest nightly build (with newest features), use:
```bash
foundryup --version nightly
```
#### Option 2: Building from Source
If you prefer to build from source or need a custom configuration:
1. **Install Rust**
First, install Rust using rustup:
```bash
curl --proto '=https' --tlsv1.2 -sSf https://sh.rustup.rs | sh
```
2. **Update Rust** (if already installed)
```bash
rustup update stable
```
3. **Install via Cargo**
```bash
cargo install --git https://github.com/foundry-rs/foundry --profile release --locked forge cast chisel anvil
```
#### Option 3: Using Docker
For containerized development:
```bash
docker pull ghcr.io/foundry-rs/foundry:latest
```
### Verification
After installation, verify that Foundry is properly installed by checking the version:
```bash
forge --version
```
You should see version information for Forge, confirming the installation was successful.
### Platform-Specific Notes
* **Windows**: Use Git BASH or WSL as your terminal. PowerShell and Command Prompt are not currently supported by foundryup
* **macOS**: Standard terminal works perfectly
* **Linux**: Any terminal emulator will work
### Troubleshooting
If you encounter issues during installation:
1. **Check your internet connection**
2. **Ensure you have the latest version of your terminal**
3. **For Windows users**: Make sure you're using Git BASH or WSL
4. **Refer to the** [**Foundry FAQ**](https://book.getfoundry.sh/faq) for additional help
### Security Note
Foundry binaries are verified using GitHub artifact attestations to ensure integrity and authenticity. The installer automatically verifies these attestations during installation.
***
✅ **Checkpoint**: You now have Foundry installed and ready to use for XRPL sidechain development!
## Part 2: Initializing a Foundry Project
Now that Foundry is installed, let's create a new project and understand the project structure.
### Creating a New Project
Initialize a new Foundry project using the `forge init` command:
```bash
forge init my-xrpl-project
cd my-xrpl-project
```
This creates a new directory with a complete Foundry project structure.
### Understanding the Project Structure
After initialization, your project will have the following structure:
```
my-xrpl-project/
├── foundry.toml # Foundry configuration file
├── src/ # Smart contracts directory
│ └── Counter.sol # Example contract
├── script/ # Deployment and interaction scripts
│ └── Counter.s.sol # Example deployment script
├── test/ # Test files
│ └── Counter.t.sol # Example test file
└── lib/ # Dependencies (managed by Soldeer)
```
#### Key Directories Explained
* **`/src`**: Contains your smart contracts written in Solidity
* **`/test`**: Contains test files for your contracts (typically with `.t.sol` extension)
* **`/script`**: Contains deployment scripts and other automation scripts
* **`/lib`**: Contains external dependencies managed by Soldeer package manager
### Setting Up Soldeer Package Manager
Soldeer is Foundry's native package manager for handling smart contract dependencies. Let's configure it for your project:
1. **Initialize Soldeer**
```bash
forge soldeer init
```
2. **Verify Soldeer Configuration**
Check that `foundry.toml` now includes Soldeer configuration:
```bash
cat foundry.toml
```
You should see something like:
```toml
[profile.default]
src = "src"
out = "out"
libs = ["lib"]
[dependencies]
# Dependencies will be listed here
```
### Building Your Project
Compile the smart contracts to ensure everything is set up correctly:
```bash
forge build
```
You should see output indicating successful compilation of the example Counter contract.
### Running Tests
Run the included tests to verify the setup:
```bash
forge test
```
This will execute all tests in the `/test` directory and show the results.
### Understanding the Foundry Tools
Your project now has access to all Foundry tools:
| Tool | Purpose |
| ---------- | ------------------------------------------------------------------- |
| **forge** | Build, test, debug, deploy and verify smart contracts |
| **anvil** | Run a local Ethereum development node with forking capabilities |
| **cast** | Interact with contracts, send transactions, and retrieve chain data |
| **chisel** | Fast Solidity REPL for rapid prototyping and debugging |
### Adding Dependencies with Soldeer
To add external smart contract libraries (like OpenZeppelin), use Soldeer:
```bash
# Example: Add OpenZeppelin contracts
forge soldeer install @openzeppelin/contracts@5.0.0
```
Dependencies will be automatically added to your `foundry.toml` and downloaded to the `/lib` directory.
### Customizing Your Project
You can customize various aspects of your project by editing `foundry.toml`:
```toml
[profile.default]
src = "src"
out = "out"
libs = ["lib"]
optimizer = true
optimizer_runs = 200
via_ir = true
[rpc_endpoints]
xrpl_devnet = "https://rpc-evm-sidechain.xrpl.org"
```
***
✅ **Checkpoint**: You now have a fully initialized Foundry project with proper structure and dependency management!
## Part 3: Setting Up Your Environment
Before deploying contracts to the XRPL sidechain, you need to set up a wallet and obtain test tokens for deployment.
### Installing MetaMask
MetaMask is a popular Ethereum wallet that works with XRPL EVM sidechain. Follow these steps to install and set it up:
1. **Install MetaMask**
* Visit [metamask.io](https://metamask.io/)
* Click "Download" and choose your browser
* Install the browser extension
* Create a new wallet or import an existing one
2. **Create a New Wallet** (if you don't have one)
* Click "Create a wallet"
* Set a strong password
* **IMPORTANT**: Write down your seed phrase securely and never share it
* Confirm your seed phrase
### Adding XRPL EVM Sidechain to MetaMask
Configure MetaMask to connect to the XRPL EVM sidechain:
1. **Open MetaMask** and click the network dropdown (usually shows "Ethereum Mainnet")
2. **Add Custom Network** with these details:
```
Network Name: XRPL EVM Sidechain
New RPC URL: https://rpc-evm-sidechain.xrpl.org
Chain ID: 1440002
Currency Symbol: XRP
Block Explorer URL: https://evm-sidechain.xrpl.org
```
3. **Save** the network configuration
4. **Switch** to the XRPL EVM Sidechain network
### Getting Your Private Key
**⚠️ Security Warning**: Never share your private key with anyone. Only use it for development and testing purposes.
1. **Open MetaMask**
2. **Click the three dots** next to your account name
3. **Select "Account Details"**
4. **Click "Show private key"**
5. **Enter your password**
6. **Copy the private key** (you'll need this for deployment)
### Getting Test Tokens from the Faucet
You need test XRP tokens to deploy contracts on the XRPL sidechain:
1. **Visit the XRPL EVM Faucet**:
2. **Connect Your Wallet**
* Click "Connect Wallet"
* Select MetaMask
* Approve the connection
3. **Request Test Tokens**
* Make sure you're on the XRPL EVM Sidechain network
* Your wallet address should be displayed
* Click "Send me XRP" or the equivalent button
* Wait for the transaction to complete
4. **Verify Token Receipt**
* Check your MetaMask balance
* You should see test XRP tokens in your wallet
### Setting Up Environment Variables
Create a `.env` file in your project root to store your private key securely:
1. **Create the file**:
```bash
touch .env
```
2. **Add your private key**:
```env
PRIVATE_KEY=your_private_key_here
RPC_URL=https://rpc.testnet.xrplevm.org
```
3. **Add `.env` to `.gitignore`** (if not already present):
```bash
echo ".env" >> .gitignore
```
### Configuring Foundry for XRPL
Update your `foundry.toml` to include XRPL-specific settings:
```toml
[profile.default]
src = "src"
out = "out"
libs = ["lib"]
optimizer = true
optimizer_runs = 200
[rpc_endpoints]
xrpl = "https://rpc.testnet.xrplevm.org"
[etherscan]
xrpl = { key = "your_api_key_if_available", url = "https://evm-sidechain.xrpl.org/api" }
```
### Testing Your Setup
Verify your environment is ready:
1. **Check your balance**:
```bash
cast balance --rpc-url https://rpc.testnet.xrplevm.org YOUR_WALLET_ADDRESS
```
2. **Load environment variables** (if using .env):
```bash
source .env
```
3. **Test connection**:
```bash
cast block-number --rpc-url $RPC_URL
```
### Security Best Practices
* **Never commit your private key** to version control
* **Use environment variables** for sensitive data
* **Use separate wallets** for development and production
* **Regularly backup** your seed phrase securely
* **Consider using hardware wallets** for production deployments
***
✅ **Checkpoint**: Your environment is now configured with MetaMask, test tokens, and proper security setup!
## Part 4: Deploying Your Smart Contract
Now that your environment is set up, let's deploy the example Counter contract to the XRPL EVM sidechain using the `forge create` command.
### Understanding the Counter Contract
First, let's look at the Counter contract in `src/Counter.sol`:
```solidity
// SPDX-License-Identifier: UNLICENSED
pragma solidity ^0.8.13;
contract Counter {
uint256 public number;
function setNumber(uint256 newNumber) public {
number = newNumber;
}
function increment() public {
number++;
}
}
```
This simple contract stores a number and provides functions to set and increment it.
### Deploying with forge create
The `forge create` command allows you to deploy a single contract directly without deployment scripts.
#### Basic Deployment Command
```bash
forge create src/Counter.sol:Counter \
--rpc-url https://rpc.testnet.xrplevm.org \
--private-key $PRIVATE_KEY \
--broadcast
```
#### Using Environment Variables
For better security, load your environment variables first:
```bash
source .env
forge create src/Counter.sol:Counter \
--rpc-url $RPC_URL \
--private-key $PRIVATE_KEY \
--broadcast
```
#### Using Foundry Configuration
If you configured the RPC endpoint in `foundry.toml`, you can use:
```bash
forge create src/Counter.sol:Counter \
--rpc-url xrpl \
--private-key $PRIVATE_KEY \
--broadcast
```
### Understanding Deployment Output
After successful deployment, you'll see output like:
```
[⠢] Compiling...
[⠆] Compiling 1 files with 0.8.19
[⠰] Solc 0.8.19 finished in 1.23s
Compiler run successful!
Deployer: 0xa735b3c25f...
Deployed to: 0x4054415432...
Transaction hash: 0x6b4e0ff93a...
```
**Key Information**:
* **Deployer**: Your wallet address
* **Deployed to**: Your contract's address on the blockchain
* **Transaction hash**: Reference for the deployment transaction
### Deploying Contracts with Constructor Arguments
If your contract has constructor parameters, use the `--constructor-args` flag:
```solidity
// Example contract with constructor
contract MyToken {
constructor(string memory name, string memory symbol, uint256 supply) {
// constructor logic
}
}
```
Deploy with arguments:
```bash
forge create src/MyToken.sol:MyToken \
--rpc-url $RPC_URL \
--private-key $PRIVATE_KEY \
--broadcast \
--constructor-args "MyToken" "MTK" 1000000
```
### Dry Run Deployment
Test your deployment without broadcasting to the network:
```bash
forge create src/Counter.sol:Counter \
--rpc-url $RPC_URL \
--private-key $PRIVATE_KEY
# Note: No --broadcast flag
```
This simulates the deployment and shows gas estimates without spending real tokens.
### Verifying Your Deployment
1. **Check deployment on block explorer**:
* Visit
* Search for your contract address
* View the contract details
2. **Interact with your contract using Cast**:
```bash
# Set your contract address
CONTRACT_ADDRESS="your_deployed_contract_address"
# Get the current counter value (should be 0)
cast call $CONTRACT_ADDRESS "number()" --rpc-url $RPC_URL
# Increment the counter
cast send $CONTRACT_ADDRESS "increment()" \
--rpc-url $RPC_URL \
--private-key $PRIVATE_KEY
# Check the new value (should be 1)
cast call $CONTRACT_ADDRESS "number()" --rpc-url $RPC_URL
# Set a specific number
cast send $CONTRACT_ADDRESS "setNumber(uint256)" 42 \
--rpc-url $RPC_URL \
--private-key $PRIVATE_KEY
```
### Contract Verification (Optional)
If the XRPL EVM sidechain supports contract verification, you can verify your contract:
```bash
forge verify-contract \
--chain-id 1440002 \
--watch \
--verifier blockscout \
--verifier-url https://evm-sidechain.xrpl.org/api \
$CONTRACT_ADDRESS \
src/Counter.sol:Counter
```
### Deployment Best Practices
1. **Always test locally first**:
```bash
# Start local Anvil node
anvil
# Deploy to local node (in another terminal)
forge create src/Counter.sol:Counter \
--rpc-url http://127.0.0.1:8545 \
--private-key 0xac0974bec39a17e36ba4a6b4d238ff944bacb478cbed5efcae784d7bf4f2ff80 \
--broadcast
```
2. **Check gas costs before deployment**:
```bash
# Dry run to see gas estimates
forge create src/Counter.sol:Counter --rpc-url $RPC_URL
```
3. **Keep track of deployed contracts**:
* Save contract addresses in a deployment log
* Document which version was deployed when
* Keep constructor arguments for reference
### Troubleshooting Common Issues
1. **Insufficient Funds**:
* Check your balance: `cast balance YOUR_ADDRESS --rpc-url $RPC_URL`
* Get more test tokens from the faucet if needed
2. **Private Key Format**:
* Ensure your private key starts with `0x`
* Check for any extra spaces or characters
3. **Network Issues**:
* Verify the RPC URL is correct
* Check if the XRPL EVM sidechain is operational
4. **Compilation Errors**:
* Run `forge build` first to check for compilation issues
* Ensure all dependencies are properly installed
### Using Legacy Transactions
If you encounter EIP-1559 issues, use the `--legacy` flag:
```bash
forge create src/Counter.sol:Counter \
--rpc-url $RPC_URL \
--private-key $PRIVATE_KEY \
--broadcast \
--legacy
```
***
🎉 **Congratulations!** You have successfully deployed your first smart contract to the XRPL EVM sidechain using Foundry! Your contract is now live and ready for interaction.
# Bridging Assets
How to get XRP on the EVM sidechain
### Bridge Funds
You will need to use the bridge tool to fund the EVM sidechain account
On the lefthand side select the Faucet.
On the righthand side connect the wallet you created with Metamask.
Use this document to share wallet info:
That's it!
# Remix
How to compile, deploy and interact with a solidity smart contract on chain
{% hint style="info" %}
Remix is an open-source, web-based integrated development environment (IDE) for writing, compiling, deploying, and debugging Solidity smart contracts on the Ethereum blockchain. It provides a user-friendly interface with tools for testing, debugging, and interacting with smart contracts directly from the browser.
{% endhint %}
Open Remix at [https://remix.etherium.org](https://remix.ethereum.org/)
## Deploying your first smart contract
By default you can find the HelloWorld.sol contract in the file explorer under contracts.
Compile it using the third tab.
You can deploy it using the fourth tab. You will need to select the `injected provider` environment and make sure your metamask is connected to the right network and account.
Deploy the contract, it should open a window for signature (notice the fees in XRP).
Once deployed successfully you can interact with it by scrolling down tab 4.
## Creating other contracts
Create the following contract in remix, give it the name `counter.sol`
{% code title="counter.sol" %}
```solidity
// SPDX-License-Identifier: MIT
pragma solidity ^0.8.0;
contract Counter {
uint256 private counter;
constructor() {
counter = 0; // Initialize counter to 0
}
function incrementAndGet() public returns (uint256) {
counter += 1; // Increment the counter
return counter; // Return the updated counter value
}
function getCounter() public view returns (uint256) {
return counter; // Return the current counter value
}
}
```
{% endcode %}
Compile the new contract using the third tab.
Deploy the contract using the fourth tab.
You can now interact with the counter smart contract:
* send a transaction to increment it, this will trigger a transaction to sign in Metamask
* read the current value (notice how this is free)
To go further you can explore the [key concepts](/evm-sidechain/banking-contract-key-concepts) behind the simple banking contract we deploy in the banking app.
# Banking App
In this workshop we will learn how to create a Fullstack app using hardhat
## Clone repo
First off we will clone the repo or import it into replit depending on your configuration.
From the root directory, run `npm i`to install common dependencies.
## Deploy the banking smart contract
Navigate to the `lending-contract` folder and follow the instructions in the readme to deploy the contract:
* install dependencies with `npm i`
* add your private key to the .env file
* fix the hardhat config
* review the contract ([in depth review of the contract here](/evm-sidechain/banking-contract-key-concepts))
* compile the contract using `npm run compile`
* deploy the contract using `npm run deploy`
You can verify your contract has been deployed using the XRPL EVM Explorer
## Run the Frontend
Navigate to the `lending-frontend` folder and follow the instructions in the readme to run the front end:
* install dependencies with `npm i`
* run the frontend with `npm run dev` and notice the Connect Wallet button does not work
We need to set up the web 3 context, to do this navigate to the `.shared` folder and fix the `web3-context.ts` file, your web app should now look like this.
* connect your account (the same EVM account you deployed the contract to)
* try to deposit
* fix the deposit function
* try to withdraw
* fix the withdraw function
* try to lend
* fix the lend function
* try to repay
* fix the repay function
Congratulations, you have just completed a full stack app using the XRPL EVM Sidechain.
# Banking Contract Key Concepts
Understand the banking contract in depth
#### Summary:
* **Owner Management**: Teaches how to restrict access to certain functions for contract owners.
* **Banking Functions**: Demonstrates how users can deposit and withdraw Ether.
* **Balance and Loan Tracking**: Illustrates keeping track of user balances and loan amounts.
* **Events**: Shows how to log contract actions for transparency using events.
* **Modifiers**: Demonstrates using function modifiers for access control and logical checks.
These contracts can be used to teach the basic functionality associated with each concept in the original `SimpleBank`contract.
#### 1. **Owner Management**
A contract where only the owner can execute certain functions.
```solidity
solidityCopy code// SPDX-License-Identifier: MIT
pragma solidity ^0.8.0;
contract OwnerContract {
address public owner;
constructor() {
owner = msg.sender; // Set the owner as the account that deploys the contract
}
modifier onlyOwner() {
require(msg.sender == owner, "Only the owner can call this function");
_;
}
function changeOwner(address newOwner) public onlyOwner {
owner = newOwner;
}
function ownerOnlyFunction() public onlyOwner {
// Only owner can execute this function
}
}
```
#### 2. **Banking Functions (Deposit and Withdrawal)**
A simple contract where users can deposit Ether, and only the owner can withdraw it.
```solidity
solidityCopy code// SPDX-License-Identifier: MIT
pragma solidity ^0.8.0;
contract DepositWithdraw {
address public owner;
mapping(address => uint256) public balances;
constructor() {
owner = msg.sender;
}
function deposit() public payable {
balances[msg.sender] += msg.value;
}
function withdraw() public {
require(balances[msg.sender] > 0, "No balance to withdraw");
uint256 amount = balances[msg.sender];
balances[msg.sender] = 0;
payable(msg.sender).transfer(amount);
}
function ownerWithdraw() public {
require(msg.sender == owner, "Only the owner can withdraw all funds");
payable(owner).transfer(address(this).balance);
}
}
```
#### 3. **Balance and Loan Tracking**
A contract that tracks balances and loans for users.
```solidity
solidityCopy code// SPDX-License-Identifier: MIT
pragma solidity ^0.8.0;
contract LoanTracker {
mapping(address => uint256) public balances;
mapping(address => uint256) public loans;
function deposit() public payable {
balances[msg.sender] += msg.value;
}
function takeLoan(uint256 amount) public {
require(amount > 0, "Loan must be greater than 0");
loans[msg.sender] += amount;
}
function repayLoan() public payable {
require(loans[msg.sender] > 0, "No outstanding loan");
require(msg.value == loans[msg.sender], "Must repay exact loan amount");
loans[msg.sender] = 0;
}
function getLoanAmount() public view returns (uint256) {
return loans[msg.sender];
}
}
```
#### 4. **Events**
A contract that demonstrates emitting events to log actions such as deposits and withdrawals.
```solidity
solidityCopy code// SPDX-License-Identifier: MIT
pragma solidity ^0.8.0;
contract EventLogger {
event DepositMade(address indexed user, uint256 amount);
event WithdrawalMade(address indexed user, uint256 amount);
mapping(address => uint256) public balances;
function deposit() public payable {
balances[msg.sender] += msg.value;
emit DepositMade(msg.sender, msg.value);
}
function withdraw(uint256 amount) public {
require(balances[msg.sender] >= amount, "Insufficient balance");
balances[msg.sender] -= amount;
payable(msg.sender).transfer(amount);
emit WithdrawalMade(msg.sender, amount);
}
}
```
#### 5. **Modifiers**
A contract that demonstrates the use of function modifiers for access control.
```solidity
solidityCopy code// SPDX-License-Identifier: MIT
pragma solidity ^0.8.0;
contract ModifierExample {
address public owner;
constructor() {
owner = msg.sender;
}
modifier onlyOwner() {
require(msg.sender == owner, "You are not the owner");
_;
}
function restrictedFunction() public onlyOwner {
// Only the owner can call this function
}
function openFunction() public {
// Anyone can call this function
}
}
```
# Core Dev Bootcamp
The XRPL Core Dev Bootcamp – Online Edition is a program designed for intermediate to advanced C++ developers looking to deepen their skills in the core of the XRP Ledger. This digital edition lets
# Rippled I – Compilation, Configuration, and Launch
### Module Overview
The first module of the XRPL Core Dev Bootcamp is dedicated to setting up the development environment and compiling **Rippled**, the software at the core of the XRP Ledger.
This module is your hands-on guide to transforming the open-source code into a functional, running server. You will gain a practical understanding of the client's internal structure by mastering the compilation, configuration, and launch of Rippled in **standalone mode**.
***
### What You Will Learn
* **Setup:** Configure a C++ development environment (macOS/Ubuntu).
* **Tools:** Master Conan and CMake for the build process.
* **Build:** Successfully compile the *rippled* binary from source.
* **Launch:** Run and configure the server in standalone mode.
* **Test:** Interact with your local ledger via API and test tools.
***
#### 💻 Environment & Build Toolchain Setup
**Preparing Your Development Machine and Mastering the C++ Toolchain**
Set up all the essential prerequisites for your development environment from installing compilers and dependencies to mastering modern C++ build tools.
You’ll configure **Clang/G++**, **Homebrew** or **apt**, and install **Python** and **Node.js**, before diving into **Conan** for dependency management and **CMake** for build configuration, ensuring full compliance with the **C++20** standard.
[Explore Environment & Toolchain Setup →](/core-dev-bootcamp/module01/environment-build-toolchain)
***
#### 🔨 Compilation Process
**Generating the Rippled Binary**
Step-by-step guide to the compilation: dependency installation, CMake configuration, and running the final build command to create the executable.
[Explore Compilation Process →](/core-dev-bootcamp/module01/compilation-process-generating-binary)
***
#### 🧩 Local Development & Testing
**Running Rippled in Standalone Mode and Interacting Locally**
Run Rippled in **standalone mode** for local testing without connecting to the XRPL network.\
Configure your environment, launch the server, and interact with it via the **API**, **WebSocket**, or tools like **XRPL Explorer** and **Playground** to test transactions and ledger operations in isolation.
**Key Topics**: Standalone mode, configuration, API (RPC/WebSocket), local testing, XRPL Explorer, Playground
[Explore Local Development & Testing →](/core-dev-bootcamp/module01/local-development-testing)
***
## Homework
We often learn best through hands-on practice, which is why we offer an exercise that involves configuring and launching Rippled on the XRPL mainnet, then analyzing the logs to understand the node's behavior during initial synchronization.
[Homework: Rippled Node Setup and Mainnet Synchronization →](/core-dev-bootcamp/module01/homeworks/homework1)
## Questions
If you have any questions about the homework or would like us to review your work, feel free to contact us.
[Submit Feedback →](https://docs.google.com/forms/d/e/1FAIpQLSe7qIbqepUKGVJxmWFHv0rs1zsBAzk8G6y5-bfz0Yd5rqW_5A/viewform)
## Additional Resources
* [Official Rippled Documentation – Build](https://github.com/XRPLF/rippled/blob/develop/BUILD.md)
* [Build Environment Guide](https://github.com/XRPLF/rippled/blob/develop/docs/build/environment.md#macos)
* [Rippled GitHub Repository](https://github.com/XRPLF/rippled)
***
➡️ **Next Module**: Rippled II - [Deep Dive into the Protocol and Codebase](/core-dev-bootcamp/module02)
# Environment & Build Toolchain
[← Back to Rippled I Overview](/core-dev-bootcamp/module01)
***
## Introduction
Before diving into Rippled’s architecture or contributing to the codebase, it’s essential to prepare a clean and consistent development environment. Rippled is a high-performance C++ application with multiple system dependencies from compiler toolchains to build systems and scripting utilities that must be properly configured to ensure smooth compilation and runtime behavior.
This section provides a **step-by-step setup guide for macOS**, focusing on the tools and configurations required to compile Rippled from source. You’ll install Node.js for build tooling, configure your compiler (Clang) and Xcode environment, and prepare Python dependencies used in the build process.
By the end of this section, your environment will be fully ready to build and run Rippled locally, following the same structure used by production and continuous integration setups.
***
## macOS
Install Node.js via nvm for easy version management:
```bash
# Install nvm (if not already installed)
curl -o- https://raw.githubusercontent.com/nvm-sh/nvm/v0.39.0/install.sh | bash
# Install and use the latest LTS version of Node.js
nvm install --lts
nvm use --lts
```
### Prerequisites for Compiling Rippled
* macOS with administrator rights
* Apple account (to download certain versions of Xcode)
* Stable internet connection for dependencies
### Checking Clang Version
```bash
clang --version
```
### Installing Xcode (if needed)
1. Download Xcode from Apple Developer Downloads
2. Extract the .xip file and rename it (e.g., Xcode\_16.2.app)
3. Move it to /Applications and set it as the default toolchain:
```bash
sudo mv Xcode_16.2.app /Applications/
sudo xcode-select -s /Applications/Xcode_16.2.app/Contents/Developer
export DEVELOPER_DIR=/Applications/Xcode_16.2.app/Contents/Developer
```
4. Verify the installation:
```bash
clang --version
git --version
```
> You should see something like this for clang: `Apple clang version 16.0.0 (clang-1600.0.26.3)`
### Installing Build Tools
```bash
# Homebrew
/bin/bash -c "$(curl -fsSL https://raw.githubusercontent.com/Homebrew/install/HEAD/install.sh)"
# Essential dependencies
brew update
brew install xz pyenv
```
### Python Configuration
```bash
# Install Python 3.13 via pyenv
pyenv install 3.11.13
pyenv global 3.11.13
eval "$(pyenv init -)"
# Install Conan and CMake
pip install 'conan>2.16'
pip install 'cmake>3.21'
```
***
## Ubuntu
### Prerequisites
* Ubuntu with administrator rights
* Stable internet connection
### Installing Build Tools
```bash
apt update
apt install --yes curl git libssl-dev pipx python3.11 python3-pip make g++-11 libprotobuf-dev protobuf-compiler
# Install CMake
curl -LO "https://github.com/Kitware/CMake/releases/download/v3.25.1/cmake-3.25.1.tar.gz"
tar -xzf cmake-3.25.1.tar.gz
cd cmake-3.25.1
./bootstrap --parallel=$(nproc)
make --jobs $(nproc)
make install
cd ..
# Install Conan
pipx install 'conan>2.16'
pipx ensurepath
```
# Compilation Process & Generating Binary
### Building Rippled from Source
[← Back to Rippled I Overview](/core-dev-bootcamp/module01)
***
#### Introduction
Once your development environment is properly configured, the next step is to **obtain and build the Rippled source code**.\
Rippled uses a modern C++ toolchain and relies on **Conan** for dependency management and **CMake** for project configuration. These tools ensure consistent builds across different systems while maintaining compatibility with the **C++20** standard.
This section will guide you through:
* Cloning the official Rippled repository from the XRPL Foundation’s GitHub.
* Setting up Conan profiles for macOS and Ubuntu.
* Configuring the CMake build system.
* Compiling the Rippled executable in **Debug** mode.
By following these steps, you’ll have a local, fully compiled version of Rippled — ready for testing, development, and contributing to the XRPL Core.
***
### Cloning the Rippled Repository
```
mkdir -p ~/projects
cd ~/projects
git clone https://github.com/XRPLF/rippled.git
cd rippled
git checkout develop
```
### Conan Configuration
Importing default Conan profil.
```bash
conan config install conan/profiles/ -tf $(conan config home)/profiles/
```
* You can check your Conan profile by running
```bash
conan profile show
```
If the default profile does not work for you and you do not yet have a Conan profile, you can create one by running:
```bash
conan profile detect
```
The recipes in Conan Center occasionally need to be patched for compatibility with the latest version of rippled.
To ensure our patched recipes are used, you must add our Conan remote at a higher index than the default Conan Center remote, so it is consulted first. You can do this by running:
```
conan remote add --index 0 xrplf https://conan.ripplex.io
```
### Build Directory Structure
```
rippled/
├── bin/ # Compiled executables
├── build/ # Build artifacts and temporary files
├── src/ # Source code
├── CMakeLists.txt # CMake configuration
└── README.md
```
### Compiling Rippled
```bash
# Create the build directory
mkdir -p build && cd build
# Install dependencies via Conan
conan install .. --output-folder . --build missing --settings build_type=Debug
# Pass the CMake variable CMAKE_BUILD_TYPE and make sure it matches the one of the build_type settings you chose in the previous step
cmake -DCMAKE_TOOLCHAIN_FILE:FILEPATH=build/generators/conan_toolchain.cmake -DCMAKE_BUILD_TYPE=Debug -Dxrpld=ON -Dtests=ON ..
# Build Rippled
cmake --build . --parallel 10
```
> ⚠️ Compilation may take 30 to 60 minutes depending on your machine.
### Verifying the Build
Once compilation completes successfully, confirm that the Rippled binary has been created:
```bash
ls build/rippled
```
Then, check the version to ensure the binary runs correctly:
```
./build/rippled --version
```
# Local Development & Testing
### Running Rippled in Standalone Mode
[← Back to Rippled I Overview](/core-dev-bootcamp/module01)
***
#### Introduction
Once built, Rippled can run in **standalone mode** — a fully local setup for **testing transactions and ledger behavior** without connecting to the XRPL network.\
It offers full API access and manual ledger control, perfect for isolated development.
You’ll learn to launch Rippled locally and explore it with **XRPL Explorer** and **Playground** to simulate end-to-end XRPL workflows.
***
> In stand-alone mode, the server operates without connecting to the network and participating in the consensus process. Without the consensus process, you have to manually advance the ledger and no distinction is made between "closed" and "validated" ledgers. However, the server still provides API access and processes transactions the same.
[Learn more about stand-alone mode](https://xrpl.org/docs/concepts/networks-and-servers/rippled-server-modes#stand-alone-mode).
```bash
cd ~/projects/rippled/build
cp ../config -r ./config
./rippled -a --conf ./config/rippled.cfg
```
* Option with genesis ledger:
```bash
./rippled -a --conf ./config/rippled.cfg --ledgerfile ./config/genesis.json
```
> [The following options determine which ledger to load first when starting up.](https://xrpl.org/docs/infrastructure/commandline-usage#initial-ledger-options)
* Check the logs to confirm that the server is running correctly.
### Interacting with Rippled via XRPL Explorer
1. Install dependencies:
```bash
cd ~/core-dev-bootcamp-2025/explorer
npm install
```
2. Launch the explorer:
```bash
npm run serve
```
3. Navigate to to view transactions and the ledger.
4. Test commands such as `server_info`, `ledger_current`, `account_info` from the interface or via cURL/WebSocket.
### Playground – Connection and Tests
```bash
git clone https://github.com/XRPL-Commons/core-dev-bootcamp-2025/tree/main/playground
cd core-dev-bootcamp-2025/playground
yarn install
ts-node src/connect.ts
ts-node src/fund.ts
```
* Create and fund test accounts to interact with the local ledger.
* Verify transactions in the explorer and via the interface/command.
# Homeworks
# Homework 1: Rippled Node Setup and Mainnet Synchronization
[← Back to Rippled I Overview](/core-dev-bootcamp/module01)
***
### Objective
* Configure and launch Rippled on the XRPL mainnet
* Analyze the logs to understand the node's behavior during initial synchronization
***
### Tasks
1. Create a `config` directory and copy the example configuration files.
2. Configure `rippled.cfg` to connect to the mainnet.
3. Start synchronization:
```bash
./rippled --conf ./config/rippled.cfg
```
4. Identify your node on the network using `server_info` and check its visibility on the XRPL explorer.
5. Analyze the logs to observe: startup, peer connections, ledger acquisition, and participation in consensus.
6. Document your experience and any errors encountered.
#### Bonus
* Compare your node with other nodes on the network
* Monitor connectivity and latency relative to the global network
# Rippled II: Deep Dive into the Protocol and Codebase
### Module Overview
The second module of the XRPL Core Dev Bootcamp Online Edition builds upon your foundational knowledge to explore the architecture of Rippled. Now that you can compile and run the software, it's time to understand how it works internally, from transaction processing to consensus mechanisms and peer-to-peer networking.
This module transforms you from someone who can operate Rippled into a developer who understands its inner workings. You'll learn to navigate the complex C++ codebase, trace transaction flows through the system, and comprehend the architectural decisions that make the XRP Ledger fast, secure, and decentralized.
***
### Explore the Topics
This deep dive is organized into focused topics, each exploring a critical component of the Rippled architecture. Click on any topic below to dive deeper into the concepts, codebase structure, and practical implementations.
***
### What You Will Learn
By completing this module, you will be able to:
* Navigate the Rippled codebase and locate key components efficiently
* Understand the architectural layers: Application, Transaction Processing, Consensus, and Networking
* Trace transaction flows from submission to ledger inclusion
* Comprehend the Transactor framework and how different transaction types are implemented
* Explore peer-to-peer networking, message protocols, and peer discovery mechanisms
* Use debugging tools, logging systems, and standalone mode for development
* Read and interpret Rippled configuration files and runtime parameters
* Understand protocol messages, RPC interfaces, and WebSocket subscriptions
These skills are fundamental for contributing to the XRPL core development, implementing protocol amendments, and building sophisticated blockchain applications.
***
#### 🌐 Protocols
**Communication and Interoperability in Distributed Systems**
Learn how distributed nodes communicate, synchronize, and maintain consensus through peer-to-peer networking and protocol messages. Understand the overlay network, message types, and how Rippled nodes discover and interact with each other.
**Key Topics**: Peer-to-peer networking, Protocol Buffers, message propagation, connection lifecycle
**Codebase**: `src/xrpld/overlay/`
[Explore Protocols →](/core-dev-bootcamp/module02/protocols-communication-and-interoperability)
***
#### ⚙️ Transactors
**Transaction Processing Framework**
Understand the transaction processing framework and the three-phase validation process that ensures ledger integrity. Learn how different transaction types (Payment, Offer, Escrow) are implemented and how to create custom transactors.
**Key Topics**: Preflight, Preclaim, DoApply phases, transaction types, custom transactor creation
**Codebase**: `src/xrpld/app/tx/detail/`
[Explore Transactors →](/core-dev-bootcamp/module02/transactors-transaction-processing-framework)
***
#### 🏗️ Application Layer
**Central Orchestration and Coordination**
Explore how the Application class orchestrates all subsystems and manages the server lifecycle. Understand initialization sequences, job queue management, and how components interact.
**Key Topics**: Application class, subsystem coordination, job queues, initialization
**Codebase**: `src/xrpld/app/main/`
[Explore Application Layer →](/core-dev-bootcamp/module02/application-layer-central-orchestration-and-coordination)
***
#### 🤝 Consensus Engine
**XRP Ledger Consensus Protocol**
Discover how validators reach agreement on transaction sets and ledger state without proof-of-work. Learn about the consensus rounds, proposals, validations, and dispute resolution mechanisms.
**Key Topics**: Consensus algorithm, validator coordination, UNL management, ledger close
**Codebase**: `src/xrpld/consensus/`
[Explore Consensus Engine →](/core-dev-bootcamp/module02/consensus-engine-xrp-ledger-consensus-protocol)
***
#### 🔗 Overlay Network
**Peer-to-Peer Networking Layer**
Master peer discovery, connection management, and message propagation in the decentralized network. Understand network topology, peer quality assessment, and network resilience.
**Key Topics**: Network topology, peer discovery, connection management, message broadcasting
**Codebase**: `src/xrpld/overlay/detail/`
[Explore Overlay Network →](/core-dev-bootcamp/module02/overlay-network-peer-to-peer-networking-layer)
***
#### 🔄 Transaction Lifecycle
**Complete Transaction Journey**
Trace the complete journey of a transaction from submission to ledger inclusion. Understand each phase: submission, validation, consensus, application, and finalization.
**Key Topics**: Submission methods, validation phases, consensus inclusion, canonical application
**Codebase**: Multiple locations across `src/xrpld/`
[Explore Transaction Lifecycle →](/core-dev-bootcamp/module02/transaction-lifecycle-complete-transaction-journey)
***
#### 🗺️ Codebase Navigation
**Efficiently Navigating the Rippled Source**
Learn to efficiently navigate the Rippled source code and locate key components. Understand directory structure, naming conventions, code patterns, and how to find specific functionality.
**Key Topics**: Directory structure, naming conventions, code patterns (Keylets, Views), IDE usage
**Codebase**: `src/xrpld/` - all directories
[Explore Codebase Navigation →](/core-dev-bootcamp/module02/codebase-navigation-efficiently-navigating-the-rippled-source)
***
#### 🐛 Debugging Tools
**Development and Debugging Techniques**
Master logging systems, standalone mode, and debugging techniques for Rippled development. Learn to interpret logs, use debuggers, and troubleshoot common issues.
**Key Topics**: Logging system, standalone mode, GDB usage, log interpretation, testing
**Codebase**: Development tools and techniques
[Explore Debugging Tools →](/core-dev-bootcamp/module02/debugging-tools-development-and-debugging-techniques)
## Homework
Trace a Payment transaction from submission through ledger closure while exploring the Payment transactor code, including `preflight`, `preclaim`, and `doApply`.
[Homework: Transaction Flow Analysis →](/core-dev-bootcamp/module02/homeworks/homework1)
[Homework: Transactor Code Exploration →](/core-dev-bootcamp/module02/homeworks/homework2)
Analyze detailed logs to understand key messages and system behavior, and create a diagram illustrating how Rippled components interact during transaction processing.
[Homework: Logging and Debugging →](/core-dev-bootcamp/module02/homeworks/homework3)
[Homework: Architecture Diagram →](/core-dev-bootcamp/module02/homeworks/homework4)
### Helpful Resources
* **Rippled Repository**: [github.com/XRPLF/rippled](https://github.com/XRPLF/rippled)
* **Module Materials**: [docs.xrpl-commons.org/core-dev-bootcamp](https://docs.xrpl-commons.org/core-dev-bootcamp/)
* **Code Navigation Tips**: Look in `src/ripple/app/tx/impl/` for transaction implementations
### Tips for Success
1. **Use Grep**: Search the codebase with `grep -r "preflight" src/ripple/app/tx/`
2. **Take Notes**: Document your findings as you explore
3. **Ask Questions**: Use the feedback form if you're stuck
4. **Be Specific**: Provide file paths, line numbers, and concrete examples
## Knowledge Check
**Review and Reinforce Your Understanding**
Before moving on, take a few minutes to review key concepts from this module.\
From consensus mechanisms to transaction flows, this short quiz will help you confirm your understanding of Rippled’s architecture.
[Ready? Let’s Quiz →](https://xrpl.at/cdbc-quiz02)
## Questions
If you have any questions about the homework or would like us to review your work, feel free to contact us.
[Submit Feedback →](https://docs.google.com/forms/d/e/1FAIpQLSe7qIbqepUKGVJxmWFHv0rs1zsBAzk8G6y5-bfz0Yd5rqW_5A/viewform)
***
➡️ **Next Module**: Data Architecture - [SHAMap and NodeStore](/core-dev-bootcamp/module03)
# Protocols: Communication and Interoperability
[← Back to Rippled II Overview](/core-dev-bootcamp/module02)
***
### Introduction
Understanding protocols is fundamental to grasping how the XRP Ledger operates as a distributed system. This deep dive explores how Rippled nodes discover each other, communicate, and synchronize ledger state across the decentralized network without any central authority.
The protocol layer is the foundation that enables the XRP Ledger to function as a truly decentralized system, where nodes around the world coordinate to maintain a consistent, validated ledger without relying on a single trusted entity.
{% embed url="" %}
***
### Peer-to-Peer Networking Fundamentals
#### The Overlay Network Architecture
The XRP Ledger operates as a **decentralized network** of Rippled servers (nodes) that communicate through peer-to-peer connections. Each node maintains connections with multiple peers, forming an **overlay network** on top of the internet infrastructure. This architecture ensures no single point of failure and enables the network to remain operational even if individual nodes go offline.
Unlike traditional client-server architectures where clients connect to centralized servers, the XRP Ledger uses a mesh topology where every node can communicate with multiple other nodes. This design provides:
* **Resilience**: No single point of failure
* **Scalability**: Network can grow organically as new nodes join
* **Censorship Resistance**: No central authority can block transactions
* **Redundancy**: Multiple paths exist for information propagation
#### Peer Discovery Mechanisms
Nodes discover peers through several complementary mechanisms, ensuring robust network connectivity:
**1. Configured Peer Lists**
Administrators can specify fixed peers in the `rippled.cfg` configuration file:
```ini
[ips_fixed]
r.ripple.com 51235
s1.ripple.com 51235
s2.ripple.com 51235
```
These peers are trusted connections that the node will always attempt to maintain. Fixed peers are particularly important for validators and high-availability servers.
**2. DNS Seeds**
Rippled uses DNS to discover bootstrap nodes:
* DNS seeds provide a list of currently active nodes
* Useful for initial network entry
* Regularly updated to reflect network state
* Multiple DNS providers ensure availability
**3. Peer Gossip**
Once connected, nodes share information about other available peers:
* Nodes exchange lists of known peers
* Information about peer quality and reliability is shared
* Network topology naturally adapts to node availability
* Helps discover new nodes joining the network
This multi-layered approach ensures that even if some discovery mechanisms fail, nodes can still find and connect to peers, maintaining network connectivity.
#### Connection Management
Each Rippled node actively manages its peer connections:
**Connection Limits**: Nodes maintain a configured number of active connections (typically 10-20 peers) to balance network visibility with resource usage.
**Peer Quality Assessment**: Nodes continuously evaluate peer behavior:
* Response times
* Message accuracy
* Protocol compliance
* Uptime and reliability
**Connection Pruning**: Poor-quality peers are disconnected and replaced with better alternatives.
**Connection Diversity**: Nodes prefer geographically and administratively diverse peers to improve network resilience.
***
### Protocol Messages
#### Protocol Buffers (Protobuf)
Rippled uses **Protocol Buffers** (protobuf) for efficient binary serialization of messages exchanged between nodes. This choice provides several advantages:
**Compact Message Sizes**: Binary encoding is more efficient than text-based formats like JSON, reducing bandwidth usage.
**Fast Serialization**: Protobuf libraries provide high-performance encoding and decoding, critical for high-throughput systems.
**Forward/Backward Compatibility**: Protocol Buffers support schema evolution, allowing the protocol to evolve without breaking existing nodes.
**Strong Typing**: Protocol definitions provide clear contracts for message formats, reducing errors.
**Cross-Language Support**: Protobuf supports multiple languages, facilitating development of diverse XRPL tools.
#### Message Definition
Protocol messages are defined in `.proto` files located in the Rippled codebase. These definitions are compiled into C++ classes used throughout the application.
**Example Structure**:
```protobuf
message TMTransaction {
required bytes raw_transaction = 1;
required uint32 status = 2;
optional bytes signature = 3;
}
```
The compiled classes provide methods for:
* Creating messages
* Serializing to binary
* Deserializing from binary
* Accessing fields with type safety
***
### Critical Message Types
Understanding message types is essential for debugging network issues and implementing protocol improvements. Each message type serves a specific purpose in maintaining network consensus and ledger synchronization.
#### tmMANIFESTS
**Purpose**: Validator key rotation and identity verification
Validators use manifests to announce their identity and key rotation information. This allows validators to change their signing keys without losing their identity, improving security by enabling regular key rotation.
**Key Information**:
* Validator's master public key (long-term identity)
* Current ephemeral signing key (used for validations)
* Key rotation sequence number
* Signature proving authorization
**When Sent**:
* When a validator starts up
* When a validator rotates keys
* Periodically to ensure all peers have current information
#### tmTRANSACTION
**Purpose**: Transaction propagation across the network
When a transaction is submitted to any node, it needs to reach all validators to be considered for inclusion in the next ledger. The `tmTRANSACTION` message broadcasts transactions throughout the network.
**Key Information**:
* Serialized transaction data
* Transaction signature
* Account information
* Fee and sequence number
**Routing Logic**:
* Nodes verify transaction validity before relaying
* Already-seen transactions are not re-broadcast (deduplication)
* Invalid transactions are dropped without propagation
#### tmPROPOSE\_LEDGER
**Purpose**: Consensus proposals from validators
During the consensus process, validators broadcast their proposed transaction sets. These proposals inform other validators about which transactions should be included in the next ledger.
**Key Information**:
* Validator's public key
* Proposed ledger close time
* Transaction set hash
* Previous ledger hash
* Validation signature
**Consensus Flow**:
1. Validators collect transactions from the open ledger
2. Each validator creates a proposal of transactions to include
3. Proposals are broadcast to other validators
4. Validators adjust their proposals based on what others propose
5. Consensus converges on a common transaction set
#### tmVALIDATION
**Purpose**: Ledger validations signaling agreement
After a ledger closes, validators broadcast validations confirming their agreement on the ledger state. A ledger becomes fully validated when it receives validations from a supermajority of trusted validators.
**Key Information**:
* Ledger hash being validated
* Ledger sequence number
* Ledger close time
* Validator's signature
* Flag indicating full/partial validation
**Validation Process**:
1. Validator applies consensus transaction set
2. Computes resulting ledger hash
3. Broadcasts validation message
4. Other nodes collect validations
5. Ledger is considered validated when quorum is reached
#### tmGET\_LEDGER / tmLEDGER\_DATA
**Purpose**: Request and response for ledger synchronization
When a node is behind or missing ledger data, it requests information from peers. These messages enable ledger history synchronization.
**tmGET\_LEDGER Fields**:
* Ledger hash or sequence number
* Type of data requested (transactions, state tree, etc.)
* Specific nodes or ranges requested
**tmLEDGER\_DATA Fields**:
* Requested ledger data
* Merkle proofs for verification
* Metadata about the ledger
**Use Cases**:
* Node startup (catching up to current ledger)
* Network partition recovery
* Historical ledger retrieval
* Filling gaps in ledger history
#### tmSTATUS\_CHANGE
**Purpose**: Notifications about ledger close events
Nodes broadcast status changes to inform peers about important events, particularly ledger closures. This helps the network stay synchronized on the current ledger state.
**Key Information**:
* New ledger sequence number
* Ledger close time
* Flags indicating network status
* Previous ledger hash
***
### Message Propagation and Routing
#### Intelligent Message Routing
When a node receives a message, it must decide whether to relay it to other peers. Rippled implements intelligent routing to prevent message flooding while ensuring necessary information reaches all relevant nodes.
#### Routing Techniques
**1. Squelch (Echo Prevention)**
**Problem**: Without squelch, messages would bounce back to their sender, creating infinite loops.
**Solution**: When node A sends a message to node B, node B remembers that A already has this message and won't send it back to A.
**Implementation**: Each message carries an originator identifier, and nodes track which peers already have which messages.
**2. Deduplication**
**Problem**: The same message might arrive from multiple peers, wasting processing resources.
**Solution**: Nodes track recently seen messages using a hash-based cache. If a message hash is already in the cache, it's discarded without reprocessing.
**Cache Management**:
* Time-based expiration (messages older than X seconds are removed)
* Size limits (oldest entries removed when cache is full)
* Hash collisions handled safely
**3. Selective Relay**
**Problem**: Not all peers need all messages. Broadcasting everything wastes bandwidth.
**Solution**: Messages are only relayed to peers that are likely to need them based on:
* Message type (validators need proposals, regular nodes may not)
* Peer capabilities (what protocol versions and features they support)
* Network topology (avoid sending to peers who likely already have it)
**4. Priority Queuing**
**Problem**: Under high load, important messages might be delayed behind less critical ones.
**Solution**: Messages are categorized by importance and processed in priority order:
**High Priority**:
* Validations (needed for ledger finalization)
* Consensus proposals (time-sensitive for consensus)
* Status changes (network coordination)
**Medium Priority**:
* Transactions (should be processed promptly)
* Ledger data responses (nodes are waiting for this)
**Low Priority**:
* Peer discovery announcements
* Historical data requests
#### Message Flow Example
Let's trace how a transaction propagates through the network:
```
1. User submits transaction to Node A
↓
2. Node A validates transaction
↓
3. Node A adds transaction to open ledger
↓
4. Node A broadcasts tmTRANSACTION to peers (Nodes B, C, D)
↓
5. Nodes B, C, D validate transaction
↓
6. Nodes B, C, D relay to their peers (excluding Node A - squelch)
↓
7. Transaction reaches all network nodes
↓
8. Validators include transaction in proposals
↓
9. Consensus reached, transaction included in ledger
```
Total time: Typically 3-5 seconds from submission to ledger inclusion.
***
### Connection Lifecycle
#### Establishing a Peer Connection
The process of two Rippled nodes establishing a connection involves multiple steps, each verifying compatibility and authenticity:
#### Step 1: DNS Resolution
```bash
# Node A wants to connect to peer.example.com
1. Query DNS for peer.example.com
2. Receive IP address (e.g., 203.0.113.50)
3. Prepare to connect to 203.0.113.50:51235
```
#### Step 2: TCP Connection
```bash
# Establish TCP socket connection
1. Send SYN packet to peer
2. Receive SYN-ACK response
3. Send ACK to complete three-way handshake
4. TCP connection established
```
#### Step 3: TLS Handshake (Optional)
If TLS is configured (recommended for security):
```bash
1. Client sends ClientHello (supported cipher suites, TLS versions)
2. Server sends ServerHello (chosen cipher suite)
3. Server sends certificate
4. Key exchange and authentication
5. Both sides send Finished messages
6. Encrypted channel established
```
#### Step 4: Protocol Handshake
Rippled-specific handshake to exchange capabilities:
```
Node A → Node B: Hello Message
{
"protocolVersion": 2,
"publicKey": "n9KorY8...",
"ledgerIndex": 75623421,
"ledgerHash": "8B3F...",
"features": ["MultiSign", "FlowCross"]
}
Node B → Node A: Hello Response
{
"protocolVersion": 2,
"publicKey": "n9LkzF...",
"ledgerIndex": 75623421,
"ledgerHash": "8B3F...",
"features": ["MultiSign", "FlowCross", "DepositAuth"]
}
```
#### Step 5: Peer Verification
Both nodes verify compatibility:
**Protocol Version Check**:
* Ensure both nodes speak compatible protocol versions
* Reject if versions are too far apart
**Public Key Validation**:
* Verify cryptographic signatures
* Check against any configured trust lists
* Ensure key is properly formatted
**Network Compatibility**:
* Verify both nodes are on the same network (mainnet, testnet, devnet)
* Check ledger hashes to confirm network agreement
**Feature Negotiation**:
* Identify common supported features
* Use lowest common denominator for communication
#### Step 6: Connection Establishment or Rejection
**If Compatible**: Connection is accepted and added to peer list
* Begin exchanging regular protocol messages
* Start participating in consensus (if validator)
* Share transactions and ledger data
**If Incompatible**: Connection is rejected
* Send rejection message with reason
* Close TCP connection
* Optionally log rejection reason for diagnostics
#### Simplified Connection Flow
```
DNS Resolution → TCP Connect → TLS Handshake → Protocol Handshake
↓ ↓ ↓ ↓
IP Address Socket Open Encrypted Version Exchange
↓
Verification
↙ ↘
Accept Reject
↓ ↓
Active Peer Close Connection
```
***
### Codebase Deep Dive
#### Key Files and Directories
The protocol implementation is primarily located in `src/ripple/overlay/`. Understanding this directory structure is essential for working with networking code.
**Primary Files**
**`src/ripple/overlay/Overlay.h`**
* Main interface for the overlay network
* Defines public API for network operations
* Used by other subsystems to access network functionality
**`src/ripple/overlay/impl/OverlayImpl.h` and `.cpp`**
* Core implementation of overlay network
* Manages peer connections
* Handles message routing
* Implements connection lifecycle
**`src/ripple/overlay/Peer.h`**
* Interface for individual peer connections
* Defines peer state and capabilities
* Methods for sending messages to specific peers
**`src/ripple/overlay/impl/PeerImp.h` and `.cpp`**
* Implementation of peer connections
* State machine for connection lifecycle
* Message parsing and serialization
**`src/ripple/overlay/Message.h`**
* Protocol message definitions
* Message type enumeration
* Message handling interfaces
**`src/ripple/protocol/messages.proto`**
* Protocol Buffer definitions
* Defines structure of all network messages
* Compiled into C++ classes
#### Code Navigation Tips
**Finding Message Handlers**:
```cpp
// Look in PeerImp.cpp for message processing
void PeerImp::onMessage(std::shared_ptr const& m)
{
switch(m->getType())
{
case protocol::mtTRANSACTION:
onTransaction(m);
break;
case protocol::mtVALIDATION:
onValidation(m);
break;
// ... other message types
}
}
```
**Understanding Peer Connection State**:
```cpp
// Peer states (simplified)
enum class State
{
connecting, // TCP connection in progress
connected, // TCP connected, handshake in progress
active, // Fully connected and operational
closing, // Graceful shutdown in progress
closed // Connection terminated
};
```
**Message Creation Example**:
```cpp
// Creating and sending a transaction message
protocol::TMTransaction tx;
tx.set_rawtransaction(serializedTx);
tx.set_status(protocol::tsCURRENT);
send(std::make_shared(tx, protocol::mtTRANSACTION));
```
***
### Hands-On Exercise
#### Exercise: Observe Protocol Messages
**Objective**: Monitor and analyze protocol messages exchanged between Rippled nodes.
**Setup**
**Step 1**: Enable detailed overlay logging
```bash
# In rippled.cfg or via RPC
rippled log_level Overlay trace
```
This will produce very detailed logs showing every message sent and received.
**Step 2**: Set up two Rippled instances in standalone mode
```bash
# Terminal 1 - Node A
rippled --conf=/path/to/rippled-node1.cfg --standalone
# Terminal 2 - Node B
rippled --conf=/path/to/rippled-node2.cfg --standalone
```
**Step 3**: Configure nodes to peer with each other
In `rippled-node1.cfg`:
```ini
[ips_fixed]
127.0.0.1 51236
```
In `rippled-node2.cfg`:
```ini
[port_peer]
port = 51236
ip = 127.0.0.1
[ips_fixed]
127.0.0.1 51235
```
**Observation Tasks**
**Task 1**: Observe connection establishment
Watch the logs as the nodes connect. You should see:
* TCP connection establishment
* Hello message exchange
* Peer verification
* Connection acceptance
**Task 2**: Submit a transaction
```bash
# Submit to Node A
rippled submit
```
Watch the logs to see:
* `tmTRANSACTION` message created on Node A
* Message sent to Node B
* Node B receives and processes transaction
* Node B adds to its open ledger
**Task 3**: Trigger a ledger close
```bash
# In standalone mode
rippled ledger_accept
```
Observe:
* Status change messages
* Ledger close coordination
* State synchronization
**Analysis Questions**
Answer these questions based on your observations:
1. **What messages are exchanged during peer connection establishment?**
* List the message types in order
* Note any authentication or verification steps
2. **How is a transaction propagated from one node to another?**
* Trace the message flow
* Identify any validation or filtering
3. **What happens when a ledger closes?**
* What messages are exchanged?
* How do nodes coordinate?
4. **How does deduplication work?**
* Try submitting the same transaction twice
* Observe how the second submission is handled
5. **What is the average message propagation time?**
* Timestamp when transaction is submitted to Node A
* Timestamp when Node B logs receiving it
* Calculate the latency
#### Expected Outcomes
You should gain practical understanding of:
* How protocol messages flow through the network
* The purpose of different message types
* Network latency and performance characteristics
* How nodes maintain synchronized state
***
### Key Takeaways
#### Core Concepts
✅ **Decentralized Architecture**: The XRP Ledger uses peer-to-peer networking to eliminate single points of failure and enable censorship resistance.
✅ **Multiple Discovery Mechanisms**: Peers are discovered through configured lists, DNS seeds, and gossip protocols, ensuring robust connectivity.
✅ **Efficient Serialization**: Protocol Buffers provide compact, fast, and version-compatible message encoding.
✅ **Intelligent Routing**: Message propagation uses squelch, deduplication, selective relay, and priority queuing to optimize network performance.
✅ **Secure Connections**: Multi-step handshake process ensures only compatible, authenticated peers connect.
✅ **Message Types Matter**: Each message type (tmTRANSACTION, tmVALIDATION, tmPROPOSE\_LEDGER, etc.) serves a specific purpose in network coordination.
#### Development Skills
✅ **Codebase Location**: Protocol implementation is in `src/ripple/overlay/`
✅ **Debugging**: Use log levels and standalone mode to observe network behavior
✅ **Protocol Evolution**: Understanding message formats is essential for implementing protocol improvements
✅ **Network Analysis**: Monitoring protocol messages helps diagnose network issues and optimize performance
***
### Additional Resources
#### Official Documentation
* **XRP Ledger Dev Portal**: [xrpl.org/docs](https://xrpl.org/docs)
* **Rippled Overlay Network**: [xrpl.org/peer-protocol](https://xrpl.org/docs/concepts/networks-and-servers/peer-protocol)
* **Rippled Repository**: [github.com/XRPLF/rippled](https://github.com/XRPLF/rippled)
#### Codebase References
* `src/ripple/overlay/` - Overlay network implementation
* `src/ripple/protocol/messages.proto` - Protocol Buffer message definitions
* `src/ripple/overlay/impl/OverlayImpl.cpp` - Core networking logic
* `src/ripple/overlay/impl/PeerImp.cpp` - Peer connection handling
#### Related Topics
* [Transactors](/core-dev-bootcamp/module02/transactors-transaction-processing-framework) - How transactions are processed after propagation
* [Consensus Engine](/core-dev-bootcamp/module02/consensus-engine-xrp-ledger-consensus-protocol) - How validators use protocol messages to reach consensus
* [Overlay Network](/core-dev-bootcamp/module02/overlay-network-peer-to-peer-networking-layer) - Deeper dive into P2P networking architecture
***
# Transactors: Transaction Processing Framework
[← Back to Rippled II Overview](/core-dev-bootcamp/module02)
***
### Introduction
Transactors are the heart of transaction processing in the XRP Ledger. Every transaction type—from simple XRP payments to complex escrow operations—is implemented as a **Transactor**, a C++ class that defines how that transaction is validated and executed. Understanding the transactor framework is essential for anyone who wants to contribute to the core protocol, implement custom transaction types, or debug transaction-related issues.
The transactor architecture ensures that all transactions undergo rigorous validation before modifying ledger state, maintaining the integrity and security that makes the XRP Ledger reliable for financial applications.
{% embed url="" %}
***
### The Transactor Base Class
#### Architecture Overview
Every transaction type in Rippled inherits from the `Transactor` base class, which provides the fundamental framework for transaction processing. This inheritance model ensures consistent behavior across all transaction types while allowing each type to implement its specific business logic.
The base `Transactor` class is defined in `src/ripple/app/tx/impl/Transactor.h` and provides:
* **Common validation logic** - Signature verification, fee checks, sequence number validation
* **Helper methods** - Account balance queries, ledger state access, fee calculation
* **Virtual methods** - Hooks for transaction-specific logic (preflight, preclaim, doApply)
* **Transaction context** - Access to the ledger, transaction data, and application state
#### Base Class Structure
```cpp
class Transactor
{
public:
// Main entry point for transaction application
static std::pair
apply(Application& app, OpenView& view, STTx const& tx, ApplyFlags flags);
// Virtual methods for transaction-specific logic
static NotTEC preflight(PreflightContext const& ctx);
static TER preclaim(PreclaimContext const& ctx);
virtual TER doApply() = 0;
protected:
// Constructor - available to derived classes
Transactor(ApplyContext& ctx);
// Helper methods
TER payFee();
TER checkSeq();
TER checkSign(PreclaimContext const& ctx);
// Member variables
ApplyContext& ctx_;
beast::Journal j_;
AccountID account_;
XRPAmount mPriorBalance;
XRPAmount mSourceBalance;
};
```
#### Key Concepts
**ApplyContext**: Provides access to the transaction being processed, the ledger view, and application services. This context object is passed through all stages of transaction processing.
**Transaction Engine Result (TER)**: Every validation step returns a TER code indicating success (`tesSUCCESS`), temporary failure (`ter` codes), or permanent failure (`tem` or `tef` codes). These codes determine whether a transaction can be retried or should be permanently rejected.
**Ledger Views**: Transactors work with "views" of the ledger state, allowing tentative modifications that can be committed or rolled back. This ensures atomic transaction processing.
***
### Three-Phase Validation Process
The transactor framework implements a rigorous three-phase validation process. Each phase has a specific purpose and access to different levels of information, creating a defense-in-depth approach to transaction validation.
#### Phase 1: Preflight
**Purpose**: Static validation that doesn't require ledger state
**Access**: Only the raw transaction data and protocol rules
**When It Runs**: Before any ledger state is accessed, can run in parallel
**What It Checks**:
* Transaction format is valid
* Required fields are present
* Field values are within valid ranges
* Amounts are positive and properly formatted
* No malformed or contradictory data
**Key Characteristic**: Preflight checks are **deterministic and stateless**—they depend only on the transaction itself, not on current ledger state.
**Preflight Example: Payment Transaction**
```cpp
NotTEC Payment::preflight(PreflightContext const& ctx)
{
// Check if the Payment transaction type is enabled
if (!ctx.rules.enabled(featurePayment))
return temDISABLED;
// Call base class preflight checks
auto const ret = preflight1(ctx);
if (!isTesSuccess(ret))
return ret;
// Verify destination account is specified
if (!ctx.tx.isFieldPresent(sfDestination))
return temDST_NEEDED;
// Verify amount is specified and valid
auto const amount = ctx.tx[sfAmount];
if (!amount)
return temBAD_AMOUNT;
// Amount must be positive
if (amount <= zero)
return temBAD_AMOUNT;
// Check for valid currency code if not XRP
if (!isXRP(amount))
{
if (!amount.issue().currency)
return temBAD_CURRENCY;
}
// Additional format validations...
return preflight2(ctx);
}
```
**Why Preflight Matters**: By catching format errors early, preflight prevents wasting resources on obviously invalid transactions. It also provides fast feedback to clients about transaction formatting issues.
#### Phase 2: Preclaim
**Purpose**: Validation requiring read-only access to ledger state
**Access**: Current ledger state (read-only), transaction data, protocol rules
**When It Runs**: After preflight passes, but before any state modifications
**What It Checks**:
* Source account exists and has sufficient balance
* Destination account exists (or can be created)
* Required authorizations are in place
* Trust lines exist for non-XRP currencies
* Account flags and settings permit the transaction
* Sequence numbers are correct
**Key Characteristic**: Preclaim can **read** ledger state but cannot **modify** it. This allows for safe concurrent execution and caching of preclaim results.
**Preclaim Example: Payment Transaction**
```cpp
TER Payment::preclaim(PreclaimContext const& ctx)
{
// Get source and destination account IDs
AccountID const src = ctx.tx[sfAccount];
AccountID const dst = ctx.tx[sfDestination];
// Source and destination cannot be the same
if (src == dst)
return temREDUNDANT;
// Check if destination account exists
auto const dstID = ctx.tx[sfDestination];
auto const sleDst = ctx.view.read(keylet::account(dstID));
// If destination doesn't exist, check if we can create it
if (!sleDst)
{
auto const amount = ctx.tx[sfAmount];
// Only XRP can create accounts
if (!isXRP(amount))
return tecNO_DST;
// Amount must meet reserve requirement
if (amount < ctx.view.fees().accountReserve(0))
return tecNO_DST_INSUF_XRP;
}
else
{
// Destination exists - check if it requires dest tag
auto const flags = sleDst->getFlags();
if (flags & lsfRequireDestTag)
{
// Destination requires a tag but none provided
if (!ctx.tx.isFieldPresent(sfDestinationTag))
return tecDST_TAG_NEEDED;
}
// Check if destination has disallowed XRP
if (flags & lsfDisallowXRP && isXRP(ctx.tx[sfAmount]))
return tecNO_TARGET;
}
// Check source account balance
auto const sleSrc = ctx.view.read(keylet::account(src));
if (!sleSrc)
return terNO_ACCOUNT;
auto const balance = (*sleSrc)[sfBalance];
auto const amount = ctx.tx[sfAmount];
// Ensure sufficient balance (including fee)
if (balance < amount + ctx.tx[sfFee])
return tecUNFUNDED_PAYMENT;
return tesSUCCESS;
}
```
**Why Preclaim Matters**: Preclaim catches state-dependent errors before attempting state modifications. This prevents partially-applied transactions and provides clear error messages about why a transaction cannot succeed.
#### Phase 3: DoApply
**Purpose**: Actual ledger state modification
**Access**: Full read/write access to ledger state
**When It Runs**: After both preflight and preclaim succeed
**What It Does**:
* Debits source account
* Credits destination account
* Creates or modifies ledger objects
* Applies transaction-specific business logic
* Records transaction metadata
* Consumes transaction fee
**Key Characteristic**: DoApply **modifies ledger state**. All changes are atomic—either the entire transaction succeeds and all changes are applied, or it fails and no changes are made.
**DoApply Example: Payment Transaction**
```cpp
TER Payment::doApply()
{
// Pay the transaction fee (happens for all transactions)
auto const result = payFee();
if (result != tesSUCCESS)
return result;
// Get amount to send
auto const amount = ctx_.tx[sfAmount];
auto const dst = ctx_.tx[sfDestination];
// Perform the actual transfer
auto const transferResult = accountSend(
view(), // Ledger view to modify
account_, // Source account
dst, // Destination account
amount, // Amount to transfer
j_ // Journal for logging
);
if (transferResult != tesSUCCESS)
return transferResult;
// Handle partial payments and path finding if applicable
if (ctx_.tx.isFlag(tfPartialPayment))
{
// Partial payment logic...
}
// Record transaction metadata
ctx_.deliver(amount);
return tesSUCCESS;
}
```
**Why DoApply Matters**: This is where the actual ledger state changes happen. DoApply ensures that only transactions that have passed all validation steps can modify the ledger, maintaining data integrity.
***
### Transaction Types in Detail
The XRP Ledger supports numerous transaction types, each implemented as a specific transactor. Understanding the most common types helps you navigate the codebase and understand protocol capabilities.
#### Payment
**File**: `src/ripple/app/tx/impl/Payment.cpp`
**Purpose**: Transfer XRP or issued currencies between accounts
**Key Features**:
* Direct XRP transfers
* Issued currency transfers via trust lines
* Path-based payments (automatic currency conversion)
* Partial payments (deliver less than requested if full amount unavailable)
**Common Fields**:
* `Account` - Source account
* `Destination` - Recipient account
* `Amount` - Amount to deliver
* `SendMax` (optional) - Maximum amount to send
* `Paths` (optional) - Payment paths for currency conversion
* `DestinationTag` (optional) - Identifier for the recipient
**Use Cases**:
* Simple XRP transfers
* Issued currency payments
* Cross-currency payments
* Payment channel settlements
#### OfferCreate
**File**: `src/ripple/app/tx/impl/CreateOffer.cpp`
**Purpose**: Place an offer on the decentralized exchange (DEX)
**Key Features**:
* Buy or sell any currency pair
* Immediate-or-cancel orders
* Fill-or-kill orders
* Passive offers (don't consume existing offers)
* Auto-bridging via XRP
**Common Fields**:
* `TakerPays` - Asset the taker (matcher) pays
* `TakerGets` - Asset the taker receives
* `Expiration` (optional) - When offer expires
* `OfferSequence` (optional) - Sequence of offer to replace
**Use Cases**:
* Currency exchange
* Market making
* Arbitrage
* Limit orders
#### OfferCancel
**File**: `src/ripple/app/tx/impl/CancelOffer.cpp`
**Purpose**: Remove an offer from the order book
**Key Features**:
* Cancel by offer sequence number
* Only offer owner can cancel
**Common Fields**:
* `OfferSequence` - Sequence number of offer to cancel
#### TrustSet
**File**: `src/ripple/app/tx/impl/SetTrust.cpp`
**Purpose**: Create or modify a trust line for issued currencies
**Key Features**:
* Set trust limit for a currency
* Authorize/deauthorize trust lines
* Configure trust line flags
**Common Fields**:
* `LimitAmount` - Trust line limit and currency
* `QualityIn` (optional) - Exchange rate for incoming transfers
* `QualityOut` (optional) - Exchange rate for outgoing transfers
**Use Cases**:
* Accept issued currencies
* Set credit limits
* Freeze trust lines
#### EscrowCreate
**File**: `src/ripple/app/tx/impl/Escrow.cpp`
**Purpose**: Lock XRP until conditions are met
**Key Features**:
* Time-based release (CryptoConditions)
* Conditional release (Interledger Protocol conditions)
* Guaranteed delivery or return
**Common Fields**:
* `Destination` - Who can claim the escrow
* `Amount` - Amount of XRP to escrow
* `FinishAfter` (optional) - Earliest finish time
* `CancelAfter` (optional) - When escrow can be cancelled
* `Condition` (optional) - Cryptographic condition for release
#### EscrowFinish
**File**: `src/ripple/app/tx/impl/Escrow.cpp`
**Purpose**: Complete an escrow and deliver XRP
**Key Features**:
* Must meet time and/or condition requirements
* Can be executed by anyone (typically destination)
**Common Fields**:
* `Owner` - Account that created the escrow
* `OfferSequence` - Sequence of EscrowCreate transaction
* `Fulfillment` (optional) - Fulfillment of cryptographic condition
#### EscrowCancel
**File**: `src/ripple/app/tx/impl/Escrow.cpp`
**Purpose**: Return escrowed XRP to owner
**Key Features**:
* Only after CancelAfter time passes
* Can be executed by anyone
#### AccountSet
**File**: `src/ripple/app/tx/impl/SetAccount.cpp`
**Purpose**: Modify account settings and flags
**Key Features**:
* Set account flags
* Configure transfer rate
* Set domain and message key
* Configure email hash
**Common Fields**:
* `SetFlag` / `ClearFlag` - Flags to modify
* `TransferRate` (optional) - Fee for transferring issued currencies
* `Domain` (optional) - Domain associated with account
* `MessageKey` (optional) - Public key for encrypted messaging
**Important Flags**:
* `asfRequireDest` - Require destination tag
* `asfRequireAuth` - Require authorization for trust lines
* `asfDisallowXRP` - Disallow XRP payments
* `asfDefaultRipple` - Enable rippling by default
#### SignerListSet
**File**: `src/ripple/app/tx/impl/SetSignerList.cpp`
**Purpose**: Create or modify multi-signature configuration
**Key Features**:
* Define list of authorized signers
* Set signing quorum
* Enable complex authorization schemes
**Common Fields**:
* `SignerQuorum` - Required signature weight
* `SignerEntries` - List of authorized signers with weights
#### PaymentChannelCreate
**File**: `src/ripple/app/tx/impl/PayChan.cpp`
**Purpose**: Open a unidirectional payment channel
**Key Features**:
* Lock XRP for fast, off-ledger payments
* Asynchronous payments with cryptographic claims
* Efficient micropayments
***
### Creating Custom Transactors
When implementing new features through amendments, you'll often need to create custom transactors. Here's the complete process:
#### Step 1: Define Transaction Format
Add your transaction type to `src/ripple/protocol/TxFormats.cpp`:
```cpp
add(jss::MyCustomTx,
ttMY_CUSTOM_TX,
{
// Required fields
{sfAccount, soeREQUIRED},
{sfDestination, soeREQUIRED},
{sfCustomField, soeREQUIRED},
// Optional fields
{sfOptionalField, soeOPTIONAL},
},
commonFields);
```
#### Step 2: Create Transactor Class
Create `src/ripple/app/tx/impl/MyCustomTx.h`:
```cpp
#ifndef RIPPLE_TX_MYCUSTOMTX_H_INCLUDED
#define RIPPLE_TX_MYCUSTOMTX_H_INCLUDED
#include
namespace ripple {
class MyCustomTx : public Transactor
{
public:
static constexpr ConsequencesFactoryType ConsequencesFactory{Normal};
explicit MyCustomTx(ApplyContext& ctx) : Transactor(ctx) {}
static NotTEC preflight(PreflightContext const& ctx);
static TER preclaim(PreclaimContext const& ctx);
TER doApply() override;
};
} // namespace ripple
#endif
```
#### Step 3: Implement Preflight
Create `src/ripple/app/tx/impl/MyCustomTx.cpp`:
```cpp
#include
#include
#include
namespace ripple {
NotTEC MyCustomTx::preflight(PreflightContext const& ctx)
{
// Check if amendment is enabled
if (!ctx.rules.enabled(featureMyCustomTx))
return temDISABLED;
// Perform base class preflight checks
auto const ret = preflight1(ctx);
if (!isTesSuccess(ret))
return ret;
// Validate custom field format
if (!ctx.tx.isFieldPresent(sfCustomField))
return temMALFORMED;
auto const customValue = ctx.tx[sfCustomField];
if (customValue < 0 || customValue > 1000000)
return temBAD_AMOUNT;
// Additional validation...
return preflight2(ctx);
}
```
#### Step 4: Implement Preclaim
```cpp
TER MyCustomTx::preclaim(PreclaimContext const& ctx)
{
// Get account IDs
AccountID const src = ctx.tx[sfAccount];
AccountID const dst = ctx.tx[sfDestination];
// Verify destination account exists
auto const sleDst = ctx.view.read(keylet::account(dst));
if (!sleDst)
return tecNO_DST;
// Check source account has sufficient balance
auto const sleSrc = ctx.view.read(keylet::account(src));
if (!sleSrc)
return terNO_ACCOUNT;
auto const balance = (*sleSrc)[sfBalance];
auto const fee = ctx.tx[sfFee];
if (balance < fee)
return tecUNFUNDED;
// Additional state-based validation...
return tesSUCCESS;
}
```
#### Step 5: Implement DoApply
```cpp
TER MyCustomTx::doApply()
{
// Pay transaction fee
auto const result = payFee();
if (result != tesSUCCESS)
return result;
// Get transaction fields
auto const dst = ctx_.tx[sfDestination];
auto const customValue = ctx_.tx[sfCustomField];
// Perform custom logic
// Example: Create a new ledger object
auto const sleNew = std::make_shared(
keylet::custom(account_, ctx_.tx.getSeqProxy().value()));
sleNew->setAccountID(sfAccount, account_);
sleNew->setAccountID(sfDestination, dst);
sleNew->setFieldU32(sfCustomField, customValue);
// Insert into ledger
view().insert(sleNew);
// Log the operation
JLOG(j_.trace()) << "MyCustomTx applied successfully";
return tesSUCCESS;
}
} // namespace ripple
```
#### Step 6: Register the Transactor
Add to `src/ripple/app/tx/applySteps.cpp`:
```cpp
#include
// In the invoke function, add:
case ttMY_CUSTOM_TX:
return MyCustomTx::makeTxConsequences(ctx);
```
#### Step 7: Write Tests
Create `src/test/app/MyCustomTx_test.cpp`:
```cpp
#include
#include
#include
namespace ripple {
namespace test {
class MyCustomTx_test : public beast::unit_test::suite
{
public:
void testBasicOperation()
{
using namespace jtx;
Env env(*this, supported_amendments() | featureMyCustomTx);
// Create test accounts
Account const alice{"alice"};
Account const bob{"bob"};
env.fund(XRP(10000), alice, bob);
// Submit custom transaction
Json::Value jv;
jv[jss::Account] = alice.human();
jv[jss::Destination] = bob.human();
jv[jss::TransactionType] = jss::MyCustomTx;
jv[jss::CustomField] = 12345;
jv[jss::Fee] = "10";
env(jv);
env.close();
// Verify results
// Add assertions...
}
void run() override
{
testBasicOperation();
// More tests...
}
};
BEAST_DEFINE_TESTSUITE(MyCustomTx, app, ripple);
} // namespace test
} // namespace ripple
```
***
### Transaction Lifecycle Within Framework
Understanding how a transaction flows through the transactor framework helps debug issues and optimize performance.
#### Complete Flow Diagram
```
Transaction Submission
↓
Preflight (Static Validation)
↓
✓ Pass → Continue
✗ Fail → Reject (return tem code)
↓
Preclaim (State Validation)
↓
✓ Pass → Continue
✗ Fail → Reject (return tec/ter code)
↓
Enter Consensus
↓
Reach Agreement
↓
DoApply (State Modification)
↓
✓ Success → Commit changes
✗ Fail → Rollback (still consumes fee)
↓
Transaction Finalized in Ledger
```
#### Error Code Categories
**tem (Malformed)**: Transaction is permanently invalid due to format issues
* Example: `temMALFORMED`, `temBAD_AMOUNT`, `temDISABLED`
* Action: Reject immediately, never retry
**tef (Failure)**: Transaction failed during local checks
* Example: `tefFAILURE`, `tefPAST_SEQ`
* Action: Reject, may indicate client error
**ter (Retry)**: Transaction failed but might succeed later
* Example: `terQUEUED`, `terPRE_SEQ`
* Action: Can be retried after conditions change
**tec (Claimed Fee)**: Transaction failed but consumed fee
* Example: `tecUNFUNDED`, `tecNO_DST`, `tecNO_PERMISSION`
* Action: Failed permanently, fee charged
**tes (Success)**: Transaction succeeded
* Example: `tesSUCCESS`
* Action: Changes committed to ledger
***
### Hands-On Exercise
#### Exercise: Trace and Modify a Payment Transaction
**Objective**: Understand the payment transactor implementation through debugging and modification.
**Part 1: Code Exploration**
**Step 1**: Navigate to the Payment transactor
```bash
cd rippled/src/ripple/app/tx/impl/
open Payment.cpp # or use your IDE
```
**Step 2**: Identify the three phases
Find and read:
* `Payment::preflight()` - Lines implementing static checks
* `Payment::preclaim()` - Lines checking ledger state
* `Payment::doApply()` - Lines modifying state
**Step 3**: Trace a specific check
Follow how the Payment transactor checks if a destination requires a destination tag:
```cpp
// In preclaim:
if (sleDst->getFlags() & lsfRequireDestTag)
{
if (!ctx.tx.isFieldPresent(sfDestinationTag))
return tecDST_TAG_NEEDED;
}
```
Questions:
* Where is `lsfRequireDestTag` defined?
* How is this flag set on an account?
* What transaction type sets this flag?
**Part 2: Debug a Payment**
**Step 1**: Set up standalone mode with logging
```bash
rippled --conf=rippled.cfg --standalone
```
Enable transaction logging:
```bash
rippled log_level Transaction trace
```
**Step 2**: Create test accounts
```bash
# Create and fund two accounts
rippled account_info
rippled account_info
```
**Step 3**: Set destination tag requirement
```bash
# Set requireDestTag flag on destination account
rippled submit '{
"TransactionType": "AccountSet",
"Account": "",
"SetFlag": 1,
"Fee": "12"
}'
```
**Step 4**: Try payment without destination tag
```bash
# This should fail with tecDST_TAG_NEEDED
rippled submit '{
"TransactionType": "Payment",
"Account": "",
"Destination": "",
"Amount": "1000000",
"Fee": "12"
}'
```
**Step 5**: Try payment with destination tag
```bash
# This should succeed
rippled submit '{
"TransactionType": "Payment",
"Account": "",
"Destination": "",
"Amount": "1000000",
"DestinationTag": 12345,
"Fee": "12"
}'
```
**Part 3: Modify the Transactor (Advanced)**
**Step 1**: Add custom logging
Edit `Payment.cpp` and add logging to `doApply()`:
```cpp
TER Payment::doApply()
{
JLOG(j_.info()) << "Payment doApply started";
JLOG(j_.info()) << "Source: " << account_;
JLOG(j_.info()) << "Destination: " << ctx_.tx[sfDestination];
JLOG(j_.info()) << "Amount: " << ctx_.tx[sfAmount];
// ... existing code ...
}
```
**Step 2**: Recompile rippled
```bash
cd rippled/build
cmake --build . --target rippled
```
**Step 3**: Run with your modified code
```bash
./rippled --conf=rippled.cfg --standalone
```
**Step 4**: Submit a payment and observe your logs
**Analysis Questions**
Answer these based on your exploration:
1. **What happens in each validation phase?**
* List the checks performed in preflight
* List the checks performed in preclaim
* What state modifications occur in doApply?
2. **How are transaction fees handled?**
* Where is `payFee()` called?
* What happens if an account can't pay the fee?
3. **How does the code handle XRP vs issued currencies?**
* Find the code that distinguishes between them
* How do payment paths work for issued currencies?
4. **What's the role of the `accountSend()` helper?**
* Where is it implemented?
* What does it do internally?
***
### Key Takeaways
#### Core Concepts
✅ **Three-Phase Validation**: Preflight (static), Preclaim (read state), DoApply (modify state) ensures robust transaction processing
✅ **Inheritance Architecture**: All transaction types inherit from Transactor base class, ensuring consistent behavior
✅ **Error Code System**: tem/tef/ter/tec/tes codes provide clear feedback about transaction status
✅ **Atomic Execution**: Transactions either fully succeed or fully fail (except fee consumption)
✅ **State Views**: Ledger modifications happen in views that can be committed or rolled back
#### Development Skills
✅ **Codebase Location**: Transaction implementations in `src/ripple/app/tx/impl/`
✅ **Creating Custom Transactions**: Follow the pattern of defining format, implementing phases, registering transactor
✅ **Debugging**: Use standalone mode and logging to trace transaction execution
✅ **Testing**: Write comprehensive unit tests for all transaction scenarios
✅ **Amendment Integration**: New transaction types typically require amendments for activation
***
### Common Patterns and Best Practices
#### Pattern 1: Check-Then-Act
Always validate before modifying state:
```cpp
// Bad - might partially modify state before failing
auto const result1 = modifyState1();
auto const result2 = modifyState2(); // If this fails, state1 is modified
if (result2 != tesSUCCESS)
return result2;
// Good - validate first, then modify
if (!canModifyState1())
return tecFAILURE;
if (!canModifyState2())
return tecFAILURE;
modifyState1();
modifyState2();
```
#### Pattern 2: Use Helper Functions
The Transactor base class provides many helpers:
```cpp
// Check sequence number
auto const result = checkSeq();
if (result != tesSUCCESS)
return result;
// Pay fee
auto const feeResult = payFee();
if (feeResult != tesSUCCESS)
return feeResult;
// Check authorization
if (!hasAuthority())
return tecNO_PERMISSION;
```
#### Pattern 3: Ledger Object Patterns
Creating, modifying, and deleting ledger objects:
```cpp
// Read existing object
auto const sle = view().read(keylet::account(accountID));
if (!sle)
return tecNO_TARGET;
// Modify object
auto sleMutable = view().peek(keylet::account(accountID));
(*sleMutable)[sfBalance] = newBalance;
view().update(sleMutable);
// Create new object
auto const sleNew = std::make_shared(keylet);
sleNew->setFieldU32(sfFlags, 0);
view().insert(sleNew);
// Delete object
view().erase(sle);
```
***
### Additional Resources
#### Official Documentation
* **XRP Ledger Dev Portal**: [xrpl.org/docs](https://xrpl.org/docs)
* **Transaction Types**: [xrpl.org/transaction-types](https://xrpl.org/docs/references/protocol/transactions/types/)
* **Create Custom Transactors**: [xrpl.org/create-custom-transactors](https://xrpl.org/resources/contribute-code/create-custom-transactors)
#### Codebase References
* `src/ripple/app/tx/impl/` - All transactor implementations
* `src/ripple/app/tx/impl/Transactor.h` - Base transactor class
* `src/ripple/protocol/TxFormats.cpp` - Transaction format definitions
* `src/ripple/protocol/TER.h` - Transaction result codes
#### Related Topics
* [Protocols](/core-dev-bootcamp/module02/protocols-communication-and-interoperability) - How transactions are propagated across the network
* [Transaction Lifecycle](/core-dev-bootcamp/module02/transaction-lifecycle-complete-transaction-journey) - Complete journey from submission to ledger
* [Application Layer](/core-dev-bootcamp/module02/application-layer-central-orchestration-and-coordination) - How transactors integrate with the overall system
***
# Application Layer: Central Orchestration and Coordination
[← Back to Rippled II Overview](/core-dev-bootcamp/module02)
***
### Introduction
The Application layer is the heart of Rippled's architecture—the central orchestrator that initializes, coordinates, and manages all subsystems. Understanding the Application layer is essential for grasping how Rippled operates as a cohesive system, where consensus, networking, transaction processing, and ledger management all work together seamlessly.
At its core, the Application class acts as a **dependency injection container** and **service locator**, providing every component access to the resources it needs while maintaining clean separation of concerns. Whether you're debugging a startup issue, optimizing system performance, or implementing a new feature, you'll inevitably interact with the Application layer.
***
### The Application Class Architecture
#### Design Philosophy
The Application class follows several key design principles that make Rippled maintainable and extensible:
**Single Point of Coordination**: Instead of components directly creating and managing their dependencies, everything flows through the Application. This centralization makes it easy to understand system initialization and component relationships.
**Dependency Injection**: Components receive their dependencies through constructor parameters rather than creating them internally. This makes testing easier and dependencies explicit.
**Interface-Based Design**: The Application class implements the `Application` interface, allowing for different implementations (production, test, mock) without changing dependent code.
**Lifetime Management**: The Application controls the creation, initialization, and destruction of all major subsystems, ensuring proper startup/shutdown sequences.
#### Application Interface
The `Application` interface is defined in `src/ripple/app/main/Application.h`:
```cpp
class Application : public beast::PropertyStream::Source
{
public:
// Core services
virtual Logs& logs() = 0;
virtual Config const& config() const = 0;
// Networking
virtual Overlay& overlay() = 0;
virtual JobQueue& getJobQueue() = 0;
// Ledger management
virtual LedgerMaster& getLedgerMaster() = 0;
virtual OpenLedger& openLedger() = 0;
// Transaction processing
virtual NetworkOPs& getOPs() = 0;
virtual TxQ& getTxQ() = 0;
// Consensus
virtual Validations& getValidations() = 0;
// Storage
virtual NodeStore::Database& getNodeStore() = 0;
virtual RelationalDatabase& getRelationalDatabase() = 0;
// RPC and subscriptions
virtual RPCHandler& getRPCHandler() = 0;
// Lifecycle
virtual void setup() = 0;
virtual void run() = 0;
virtual void signalStop() = 0;
// Utility
virtual bool isShutdown() = 0;
virtual std::chrono::seconds getMaxDisallowedLedger() = 0;
protected:
Application() = default;
};
```
#### ApplicationImp Implementation
The concrete implementation `ApplicationImp` is in `src/ripple/app/main/ApplicationImp.cpp`. This class:
* Implements all interface methods
* Owns all major subsystem objects
* Manages initialization order
* Coordinates shutdown
* Provides cross-cutting services
**Key Member Variables**:
```cpp
class ApplicationImp : public Application
{
private:
// Configuration and logging
std::unique_ptr logs_;
Config config_;
// Core services
std::unique_ptr jobQueue_;
std::unique_ptr nodeStore_;
std::unique_ptr relationalDB_;
// Networking
std::unique_ptr overlay_;
// Ledger management
std::unique_ptr ledgerMaster_;
std::unique_ptr openLedger_;
// Transaction processing
std::unique_ptr networkOPs_;
std::unique_ptr txQ_;
// Consensus
std::unique_ptr validations_;
// RPC
std::unique_ptr rpcHandler_;
// State
std::atomic isShutdown_{false};
std::condition_variable cv_;
std::mutex mutex_;
};
```
***
### Initialization and Lifecycle
#### Startup Sequence
Understanding the startup sequence is crucial for debugging initialization issues and understanding component dependencies.
**Phase 1: Configuration Loading**
```cpp
// In main.cpp
auto config = std::make_unique();
if (!config->setup(configFile, quiet))
{
// Configuration failed
return -1;
}
```
**What Happens**:
* Parse `rippled.cfg` configuration file
* Load validator list configuration
* Set up logging configuration
* Validate configuration parameters
* Apply defaults for unspecified options
**Configuration Sections**:
* `[server]` - Server ports and interfaces
* `[node_db]` - NodeStore database configuration
* `[node_size]` - Performance tuning parameters
* `[validation_seed]` - Validator key configuration
* `[ips_fixed]` - Fixed peer connections
* `[features]` - Amendment votes
**Phase 2: Application Construction**
```cpp
// Create the application instance
auto app = make_Application(
std::move(config),
std::move(logs),
std::move(timeKeeper));
```
**Constructor Sequence** (`ApplicationImp::ApplicationImp()`):
```cpp
ApplicationImp::ApplicationImp(
std::unique_ptr config,
std::unique_ptr logs,
std::unique_ptr timeKeeper)
: config_(std::move(config))
, logs_(std::move(logs))
, timeKeeper_(std::move(timeKeeper))
{
// 1. Create basic services
jobQueue_ = std::make_unique(
*logs_,
config_->WORKERS);
// 2. Initialize databases
nodeStore_ = NodeStore::Manager::make(
"NodeStore.main",
scheduler,
*logs_,
config_->section("node_db"));
relationalDB_ = makeRelationalDatabase(
*config_,
*logs_);
// 3. Create ledger management
ledgerMaster_ = std::make_unique(
*this,
stopwatch(),
*logs_);
// 4. Create networking
overlay_ = std::make_unique(
*this,
config_->section("overlay"),
*logs_);
// 5. Create transaction processing
networkOPs_ = std::make_unique(
*this,
*logs_);
txQ_ = std::make_unique(
*config_,
*logs_);
// 6. Create consensus components
validations_ = std::make_unique(
*this);
// 7. Create RPC handler
rpcHandler_ = std::make_unique(
*this,
*logs_);
// Note: Order matters! Components may depend on earlier ones
}
```
**Phase 3: Setup**
```cpp
app->setup();
```
**What Happens** (`ApplicationImp::setup()`):
```cpp
void ApplicationImp::setup()
{
// 1. Load existing ledger state
auto initLedger = getLastFullLedger();
// 2. Initialize ledger master
ledgerMaster_->setLastFullLedger(initLedger);
// 3. Start open ledger
openLedger_->accept(
initLedger,
orderTx,
consensusParms,
{}); // Empty transaction set for new ledger
// 4. Initialize overlay network
overlay_->start();
// 5. Start RPC servers
rpcHandler_->setup();
// 6. Additional subsystem initialization
// ...
JLOG(j_.info()) << "Application setup complete";
}
```
**Phase 4: Run**
```cpp
app->run();
```
**Main Event Loop** (`ApplicationImp::run()`):
```cpp
void ApplicationImp::run()
{
JLOG(j_.info()) << "Application starting";
// Start processing jobs
jobQueue_->start();
// Enter main loop
{
std::unique_lock lock(mutex_);
// Wait until shutdown signal
while (!isShutdown_)
{
cv_.wait(lock);
}
}
JLOG(j_.info()) << "Application stopping";
}
```
**What Runs**:
* Job queue processes queued work
* Overlay network handles peer connections
* Consensus engine processes rounds
* NetworkOPs coordinates operations
* RPC handlers process client requests
All work happens in background threads managed by various subsystems. The main thread simply waits for a shutdown signal.
**Phase 5: Shutdown**
```cpp
app->signalStop();
```
**Graceful Shutdown** (`ApplicationImp::signalStop()`):
```cpp
void ApplicationImp::signalStop()
{
JLOG(j_.info()) << "Shutdown requested";
// 1. Set shutdown flag
isShutdown_ = true;
// 2. Stop accepting new work
overlay_->stop();
rpcHandler_->stop();
// 3. Complete in-flight operations
jobQueue_->finish();
// 4. Stop subsystems (reverse order of creation)
networkOPs_->stop();
ledgerMaster_->stop();
// 5. Close databases
nodeStore_->close();
relationalDB_->close();
// 6. Wake up main thread
cv_.notify_all();
JLOG(j_.info()) << "Shutdown complete";
}
```
**Shutdown Order**: Components are stopped in reverse order of their creation to ensure dependencies are still available when each component shuts down.
#### Complete Lifecycle Diagram
```
Program Start
↓
Load Configuration (rippled.cfg)
↓
Create Application Instance
↓
Construct Subsystems
• Databases
• Networking
• Ledger Management
• Transaction Processing
• Consensus
↓
Setup Phase
• Load Last Ledger
• Initialize Components
• Start Network
↓
Run Phase (Main Loop)
• Process Jobs
• Handle Consensus
• Process Transactions
• Serve RPC Requests
↓
Shutdown Signal Received
↓
Graceful Shutdown
• Stop Accepting Work
• Complete In-Flight Operations
• Stop Subsystems
• Close Databases
↓
Program Exit
```
***
### Subsystem Coordination
#### The Service Locator Pattern
The Application acts as a **service locator**, allowing any component to access any other component through the app reference:
```cpp
class SomeComponent
{
public:
SomeComponent(Application& app)
: app_(app)
{
// Components store app reference
}
void doWork()
{
// Access other components through app
auto& ledgerMaster = app_.getLedgerMaster();
auto& overlay = app_.overlay();
auto& jobs = app_.getJobQueue();
// Use the components...
}
private:
Application& app_;
};
```
#### Major Subsystems
**LedgerMaster**
**Purpose**: Manages the chain of validated ledgers and coordinates ledger progression.
**Key Responsibilities**:
* Track current validated ledger
* Build candidate ledgers for consensus
* Synchronize ledger history
* Maintain ledger cache
* Coordinate with consensus engine
**Access**: `app.getLedgerMaster()`
**Important Methods**:
```cpp
// Get current validated ledger
std::shared_ptr getValidatedLedger();
// Get closed ledger (not yet validated)
std::shared_ptr getClosedLedger();
// Advance to new ledger
void advanceLedger();
// Fetch missing ledgers
void fetchLedger(LedgerHash const& hash);
```
**NetworkOPs**
**Purpose**: Coordinates network operations and transaction processing.
**Key Responsibilities**:
* Process submitted transactions
* Manage transaction queue
* Coordinate consensus participation
* Track network state
* Publish ledger close events
**Access**: `app.getOPs()`
**Important Methods**:
```cpp
// Submit transaction
void submitTransaction(std::shared_ptr const& tx);
// Process transaction
void processTransaction(
std::shared_ptr& transaction,
bool trusted,
bool local);
// Get network state
OperatingMode getOperatingMode();
```
**Overlay**
**Purpose**: Manages peer-to-peer networking layer.
**Key Responsibilities**:
* Peer discovery and connection
* Message routing
* Network topology maintenance
* Bandwidth management
**Access**: `app.overlay()`
**Important Methods**:
```cpp
// Send message to all peers
void broadcast(std::shared_ptr const& message);
// Get active peer count
std::size_t size() const;
// Connect to specific peer
void connect(std::string const& ip);
```
**TxQ (Transaction Queue)**
**Purpose**: Manages transaction queuing when network is busy.
**Key Responsibilities**:
* Queue transactions during high load
* Fee-based prioritization
* Account-based queuing limits
* Transaction expiration
**Access**: `app.getTxQ()`
**Important Methods**:
```cpp
// Check if transaction can be added
std::pair
apply(Application& app, OpenView& view, STTx const& tx);
// Get queue status
Json::Value getJson();
```
**NodeStore**
**Purpose**: Persistent storage for ledger data.
**Key Responsibilities**:
* Store ledger state nodes
* Provide efficient retrieval
* Cache frequently accessed data
* Support different backend databases (RocksDB, NuDB)
**Access**: `app.getNodeStore()`
**Important Methods**:
```cpp
// Store ledger node
void store(
NodeObjectType type,
Blob const& data,
uint256 const& hash);
// Fetch ledger node
std::shared_ptr
fetch(uint256 const& hash);
```
**RelationalDatabase**
**Purpose**: SQL database for indexed data and historical queries.
**Key Responsibilities**:
* Store transaction metadata
* Maintain account transaction history
* Support RPC queries (account\_tx, tx)
* Ledger header storage
**Access**: `app.getRelationalDatabase()`
**Database Types**:
* SQLite (default, embedded)
* PostgreSQL (production deployments)
**Validations**
**Purpose**: Manages validator signatures on ledger closes.
**Key Responsibilities**:
* Collect validations from validators
* Track validator key rotations (manifests)
* Determine ledger validation quorum
* Publish validation stream
**Access**: `app.getValidations()`
**Important Methods**:
```cpp
// Add validation
void addValidation(STValidation const& val);
// Get validation for ledger
std::vector>
getValidations(LedgerHash const& hash);
// Check if ledger is validated
bool hasQuorum(LedgerHash const& hash);
```
***
### Job Queue System
#### Purpose and Design
The job queue is Rippled's work scheduling system. Instead of each subsystem creating its own threads, work is submitted as **jobs** to a centralized queue processed by a thread pool. This provides:
* **Centralized thread management**: Easier to control thread count and CPU usage
* **Priority-based scheduling**: Critical jobs processed before low-priority ones
* **Visibility**: Easy to monitor what work is queued
* **Deadlock prevention**: Structured concurrency patterns
#### Job Types
Jobs are categorized by type, which determines priority:
```cpp
enum JobType
{
// Special job types
jtINVALID = -1,
jtPACK, // Job queue work pack
// High priority - consensus critical
jtPUBOLDLEDGER, // Publish old ledger
jtVALIDATION_ut, // Process validation (untrusted)
jtPROPOSAL_ut, // Process consensus proposal
jtLEDGER_DATA, // Process ledger data
// Medium priority
jtTRANSACTION, // Process transaction
jtADVANCE, // Advance ledger
jtPUBLEDGER, // Publish ledger
jtTXN_DATA, // Transaction data retrieval
// Low priority
jtUPDATE_PF, // Update path finding
jtCLIENT, // Handle client request
jtRPC, // Process RPC
jtTRANSACTION_l, // Process transaction (low priority)
// Lowest priority
jtPEER, // Peer message
jtDISK, // Disk operations
jtADMIN, // Administrative operations
};
```
#### Submitting Jobs
Components submit work to the job queue:
```cpp
// Get job queue reference
JobQueue& jobs = app.getJobQueue();
// Submit a job
jobs.addJob(
jtTRANSACTION, // Job type
"processTx", // Job name (for logging)
[this, tx](Job&) // Job function
{
// Do work here
processTransaction(tx);
});
```
#### Job Priority and Scheduling
**Priority Levels**:
* **Critical**: Consensus, validations (must not be delayed)
* **High**: Transaction processing, ledger advancement
* **Medium**: RPC requests, client operations
* **Low**: Maintenance, administrative tasks
**Scheduling Algorithm**:
1. Jobs sorted by priority and submission time
2. Worker threads pick highest priority job
3. Long-running jobs can be split into chunks
4. System monitors queue depth and adjusts behavior
#### Job Queue Configuration
In `rippled.cfg`:
```ini
[node_size]
# Affects worker thread count
tiny # 1 thread
small # 2 threads
medium # 4 threads (default)
large # 8 threads
huge # 16 threads
```
Thread count is also influenced by CPU core count:
```cpp
// Typically: max(2, std::thread::hardware_concurrency() - 1)
```
***
### Configuration Management
#### Configuration File Structure
The `rippled.cfg` file controls all aspects of server behavior. The Application loads and provides access to this configuration.
**Example Configuration**
```ini
[server]
port_rpc_admin_local
port_peer
port_ws_admin_local
[port_rpc_admin_local]
port = 5005
ip = 127.0.0.1
admin = 127.0.0.1
protocol = http
[port_peer]
port = 51235
ip = 0.0.0.0
protocol = peer
[port_ws_admin_local]
port = 6006
ip = 127.0.0.1
admin = 127.0.0.1
protocol = ws
[node_size]
medium
[node_db]
type=RocksDB
path=/var/lib/rippled/db/rocksdb
open_files=512
cache_mb=256
filter_bits=12
compression=1
[database_path]
/var/lib/rippled/db
[debug_logfile]
/var/log/rippled/debug.log
[sntp_servers]
time.windows.com
time.apple.com
time.nist.gov
pool.ntp.org
[ips_fixed]
r.ripple.com 51235
[validators_file]
validators.txt
[rpc_startup]
{ "command": "log_level", "severity": "warning" }
[features]
# Vote for or against amendments
# AmendmentName
```
#### Accessing Configuration
Components access configuration through the Application:
```cpp
void SomeComponent::configure()
{
// Get config reference
Config const& config = app_.config();
// Access specific sections
auto const& nodeDB = config.section("node_db");
auto const type = get(nodeDB, "type");
auto const path = get(nodeDB, "path");
// Access node size
auto nodeSize = config.NODE_SIZE;
// Access ports
for (auto const& port : config.ports)
{
// Configure port...
}
}
```
#### Runtime Configuration
Some settings can be adjusted at runtime via RPC:
```bash
# Change log verbosity
rippled log_level partition severity
# Connect to peer
rippled connect ip:port
# Get server info
rippled server_info
```
***
### Component Interaction Patterns
#### Pattern 1: Direct Method Calls
Most common pattern—components call each other's methods:
```cpp
void NetworkOPs::submitTransaction(STTx const& tx)
{
// Validate transaction
auto result = Transactor::preflight(tx);
if (!isTesSuccess(result))
return;
// Apply to open ledger
auto& openLedger = app_.openLedger();
openLedger.modify([&](OpenView& view)
{
Transactor::apply(app_, view, tx);
});
// Broadcast to network
auto& overlay = app_.overlay();
overlay.broadcast(makeTransactionMessage(tx));
}
```
#### Pattern 2: Job Queue for Asynchronous Work
For work that should not block the caller:
```cpp
void LedgerMaster::fetchLedger(LedgerHash const& hash)
{
// Submit fetch job
app_.getJobQueue().addJob(
jtLEDGER_DATA,
"fetchLedger",
[this, hash](Job&)
{
// Request from peers
app_.overlay().sendRequest(hash);
// Wait for response
// Process received data
// ...
});
}
```
#### Pattern 3: Event Publication
Components publish events that others subscribe to:
```cpp
// Publisher (LedgerMaster)
void LedgerMaster::newLedgerValidated()
{
// Notify subscribers
for (auto& subscriber : subscribers_)
{
subscriber->onLedgerValidated(currentLedger_);
}
}
// Subscriber (NetworkOPs)
void NetworkOPs::onLedgerValidated(
std::shared_ptr const& ledger)
{
// React to new ledger
updateSubscribers(ledger);
processQueuedTransactions();
}
```
#### Pattern 4: Callback Registration
Components register callbacks for specific events:
```cpp
// Register callback
app_.getLedgerMaster().onConsensusReached(
[this](std::shared_ptr const& ledger)
{
handleConsensusLedger(ledger);
});
```
***
### Codebase Deep Dive
#### Key Files and Directories
**Application Core**:
* `src/ripple/app/main/Application.h` - Application interface
* `src/ripple/app/main/ApplicationImp.h` - Implementation header
* `src/ripple/app/main/ApplicationImp.cpp` - Implementation
* `src/ripple/app/main/main.cpp` - Entry point, creates Application
**Job Queue**:
* `src/ripple/core/JobQueue.h` - Job queue interface
* `src/ripple/core/impl/JobQueue.cpp` - Implementation
* `src/ripple/core/Job.h` - Job definition
**Configuration**:
* `src/ripple/core/Config.h` - Config class
* `src/ripple/core/ConfigSections.h` - Section definitions
**Subsystem Implementations**:
* `src/ripple/app/ledger/LedgerMaster.h`
* `src/ripple/app/misc/NetworkOPs.h`
* `src/ripple/overlay/Overlay.h`
* `src/ripple/app/tx/TxQ.h`
#### Code Navigation Tips
**Finding Application Creation**
Start in `main.cpp`:
```cpp
int main(int argc, char** argv)
{
// Parse command line
// Load configuration
// Create logs
// Create application
auto app = make_Application(
std::move(config),
std::move(logs),
std::move(timeKeeper));
// Setup and run
app->setup();
app->run();
return 0;
}
```
**Tracing Component Access**
Follow how components access each other:
```cpp
// In any component
void MyComponent::work()
{
// Access through app_
auto& ledgerMaster = app_.getLedgerMaster(); // → ApplicationImp::getLedgerMaster()
// → return *ledgerMaster_;
}
```
**Understanding Job Submission**
Find job submissions:
```bash
# Search for addJob calls
grep -r "addJob" src/ripple/app/
```
Example:
```cpp
app_.getJobQueue().addJob(jtTRANSACTION, "processTx", [&](Job&) {
// Job code
});
```
***
### Hands-On Exercise
#### Exercise: Trace Application Startup and Analyze Job Queue
**Objective**: Understand the application initialization sequence and monitor job queue activity.
**Part 1: Code Exploration**
**Step 1**: Navigate to application source
```bash
cd rippled/src/ripple/app/main/
```
**Step 2**: Read the main entry point
Open `main.cpp` and trace:
1. Command-line argument parsing
2. Configuration loading
3. Application creation
4. Setup call
5. Run call
**Step 3**: Follow ApplicationImp construction
Open `ApplicationImp.cpp` and identify:
1. The order subsystems are created (constructor)
2. Dependencies between components
3. What happens in `setup()`
4. What happens in `run()`
**Questions to Answer**:
* Why is NodeStore created before LedgerMaster?
* What does LedgerMaster need from Application?
* Which components are created first and why?
**Part 2: Monitor Job Queue Activity**
**Step 1**: Enable detailed job queue logging
Edit `rippled.cfg`:
```ini
[rpc_startup]
{ "command": "log_level", "partition": "JobQueue", "severity": "trace" }
```
**Step 2**: Start rippled in standalone mode
```bash
rippled --conf=rippled.cfg --standalone
```
**Step 3**: Watch the startup logs
Observe jobs during startup:
* What job types execute first?
* How many worker threads are created?
* What's the initial job queue depth?
**Step 4**: Submit transactions and observe
```bash
# Submit a payment
rippled submit '{
"TransactionType": "Payment",
"Account": "...",
"Destination": "...",
"Amount": "1000000"
}'
```
Watch the logs for:
* `jtTRANSACTION` jobs being queued
* Job processing time
* Queue depth changes
**Step 5**: Manually close a ledger
```bash
rippled ledger_accept
```
Observe jobs related to ledger close:
* `jtADVANCE` - Advance to next ledger
* `jtPUBLEDGER` - Publish ledger
* `jtUPDATE_PF` - Update path finding
**Part 3: Add Custom Logging**
**Step 1**: Modify ApplicationImp.cpp
Add logging to track component initialization:
```cpp
ApplicationImp::ApplicationImp(/* ... */)
{
JLOG(j_.info()) << "Creating JobQueue...";
jobQueue_ = std::make_unique(/* ... */);
JLOG(j_.info()) << "JobQueue created";
JLOG(j_.info()) << "Creating NodeStore...";
nodeStore_ = NodeStore::Manager::make(/* ... */);
JLOG(j_.info()) << "NodeStore created";
// Add similar logs for other components
}
```
**Step 2**: Recompile
```bash
cd rippled/build
cmake --build . --target rippled
```
**Step 3**: Run and observe
```bash
./rippled --conf=rippled.cfg --standalone
```
You should see your custom log messages showing component creation order.
**Analysis Questions**
Answer these based on your exploration:
1. **What's the first subsystem created?**
* Why does it need to be first?
2. **How does the job queue decide which job to process next?**
* What factors influence priority?
3. **What happens if a job throws an exception?**
* Find the exception handling code
4. **How many jobs are queued during a typical ledger close?**
* Count from your logs
5. **What's the relationship between Application and ApplicationImp?**
* Why use an interface?
6. **How would you add a new subsystem?**
* What's the process?
* Where would you add it?
***
### Key Takeaways
#### Core Concepts
✅ **Central Orchestration**: Application class coordinates all subsystems and manages their lifecycle
✅ **Dependency Injection**: Components receive dependencies through Application reference, not by creating them
✅ **Service Locator**: Application provides access to all major services (getLedgerMaster(), overlay(), etc.)
✅ **Initialization Order**: Subsystems are created in dependency order during construction
✅ **Job Queue**: Centralized work scheduling with priority-based execution
✅ **Configuration**: All server behavior controlled through rippled.cfg
#### Development Skills
✅ **Codebase Location**: Application implementation in `src/ripple/app/main/`
✅ **Adding Components**: Create in constructor, expose through interface method
✅ **Job Submission**: Use `app.getJobQueue().addJob()` for asynchronous work
✅ **Debugging Startup**: Add logging in ApplicationImp constructor to trace initialization
✅ **Configuration Access**: Use `app.config()` to read configuration values
***
### Common Patterns and Best Practices
#### Pattern 1: Accessing Subsystems
Always access subsystems through the Application:
```cpp
// Good - through Application
void doWork(Application& app)
{
auto& ledgerMaster = app.getLedgerMaster();
ledgerMaster.getCurrentLedger();
}
// Bad - storing subsystem reference
class BadComponent
{
LedgerMaster& ledgerMaster_; // Don't do this
BadComponent(LedgerMaster& lm)
: ledgerMaster_(lm) {} // Tight coupling
};
// Good - storing Application reference
class GoodComponent
{
Application& app_;
GoodComponent(Application& app)
: app_(app) {} // Loose coupling
void work()
{
// Access when needed
auto& lm = app_.getLedgerMaster();
}
};
```
#### Pattern 2: Asynchronous Work
Use job queue for work that shouldn't block:
```cpp
// Don't block the caller
void expensiveOperation(Application& app)
{
app.getJobQueue().addJob(
jtCLIENT,
"expensiveWork",
[&app](Job&)
{
// Long-running work here
performExpensiveCalculation();
// Access other subsystems as needed
app.getLedgerMaster().doSomething();
});
}
```
#### Pattern 3: Lifecycle Management
Let Application manage component lifetime:
```cpp
// In ApplicationImp constructor
myComponent_ = std::make_unique(*this);
// In ApplicationImp::setup()
myComponent_->initialize();
// In ApplicationImp::signalStop()
myComponent_->shutdown();
// Destructor automatically cleans up
// (unique_ptr handles deletion)
```
***
### Additional Resources
#### Official Documentation
* **XRP Ledger Dev Portal**: [xrpl.org/docs](https://xrpl.org/docs)
* **Rippled Setup**: [xrpl.org/install-rippled](https://xrpl.org/docs/infrastructure/installation)
* **Configuration Reference**: [xrpl.org/rippled-server-configuration](https://xrpl.org/docs/infrastructure/configuration)
#### Codebase References
* `src/ripple/app/main/` - Application layer implementation
* `src/ripple/core/JobQueue.h` - Job queue system
* `src/ripple/core/Config.h` - Configuration management
* `src/ripple/app/main/main.cpp` - Program entry point
#### Related Topics
* [Transactors](/core-dev-bootcamp/module02/transactors-transaction-processing-framework) - How transactions are processed
* [Consensus Engine](/core-dev-bootcamp/module02/consensus-engine-xrp-ledger-consensus-protocol) - How consensus integrates with Application
* [Codebase Navigation](/core-dev-bootcamp/module02/codebase-navigation-efficiently-navigating-the-rippled-source) - Finding your way around the code
***
# Consensus Engine: XRP Ledger Consensus Protocol
[← Back to Rippled II Overview](/core-dev-bootcamp/module02)
***
### Introduction
The Consensus Engine is the heart of the XRP Ledger's ability to reach agreement on transaction sets and ledger state without requiring proof-of-work mining or proof-of-stake validation. Understanding the consensus mechanism is crucial for anyone working on the core protocol, as it's what makes the XRP Ledger both fast (3-5 second confirmation times) and secure (Byzantine fault tolerant).
Unlike blockchain systems that require computational work to achieve consensus, the XRP Ledger uses a **consensus protocol** where a network of independent validators exchanges proposals and votes to agree on which transactions to include in the next ledger. This approach enables the network to process transactions quickly while maintaining strong security guarantees.
{% embed url="" %}
***
### Consensus Protocol Overview
#### What is Consensus?
**Consensus** is the process by which a distributed network of nodes agrees on a single shared state. In the context of the XRP Ledger, this means agreeing on:
1. **Which transactions to include** in the next ledger
2. **The order of those transactions** (deterministic ordering)
3. **The resulting ledger state** after applying all transactions
Without consensus, different nodes might have different views of the ledger, leading to double-spending and inconsistencies.
#### Why Not Proof-of-Work?
Traditional blockchains like Bitcoin use proof-of-work (PoW):
* Miners compete to solve cryptographic puzzles
* Winner proposes the next block
* Requires massive computational resources
* Block confirmation takes \~10 minutes (Bitcoin) or \~15 seconds (Ethereum)
**XRP Ledger's approach is different:**
* No mining or computational puzzles
* Validators vote on transaction sets
* Consensus achieved in rounds lasting 3-5 seconds
* Energy efficient (no wasted computation)
* Fast finality (transactions confirmed quickly)
#### Byzantine Fault Tolerance
The XRP Ledger consensus protocol is **Byzantine Fault Tolerant (BFT)**, meaning it can tolerate some validators being:
* Offline or unreachable
* Malicious (trying to disrupt consensus)
* Byzantine (behaving arbitrarily or incorrectly)
**Key Property**: As long as **>80%** of trusted validators are honest and online, consensus will be reached correctly.
**Security Model**:
```
Network can tolerate up to 20% faulty validators
Examples:
- 100 validators → Can handle 20 failures
- 50 validators → Can handle 10 failures
- 35 validators (XRP Ledger mainnet) → Can handle 7 failures
```
***
### Validators and UNL
Validators are nodes that participate in the XRP Ledger consensus process, validating transactions and agreeing on the state of the ledger. Each validator proposes and votes on ledger updates during consensus rounds.
A Unique Node List (UNL) is a trusted set of validators chosen by a participant. By relying on their UNL, a node can efficiently reach consensus while protecting against faulty or malicious validators. Proper UNL selection is crucial for network security, decentralization, and ledger reliability.
{% embed url="" %}
#### What is a Validator?
A **validator** is a rippled server configured to participate in consensus by:
* Proposing transaction sets
* Voting on other validators' proposals
* Signing validated ledgers
**Not all rippled servers are validators.** Most servers are:
* **Tracking servers**: Follow the network and process transactions but don't participate in consensus
* **Stock servers**: Serve API requests but don't store full history
**To become a validator**, a server needs:
1. A validator key pair (generated with `validator-keys` tool)
2. Configuration in `rippled.cfg` to enable validation
3. To be trusted by other validators (added to their UNLs)
#### Unique Node List (UNL)
Each validator maintains a **Unique Node List (UNL)**—a list of validators it trusts to be honest and not collude.
**Key Concepts**:
**Personal Choice**: Each validator operator chooses their own UNL based on their trust relationships.
**Overlap Required**: For the network to reach consensus, there must be sufficient overlap between validators' UNLs. The protocol requires >90% overlap to ensure agreement.
**Default UNL**: Most operators use the default UNL provided by the XRP Ledger Foundation, which is regularly updated and reviewed.
**Dynamic Updates**: UNLs can be updated over time as validators join or leave the network.
#### UNL Configuration
In `validators.txt`:
```ini
# Validator List (maintained by XRP Ledger Foundation)
# Format: validator_public_key [optional_comment]
nHUon2tpyJEHHYGmxqeGu37cvPYHzrMtUNQFVdCgGNvYkr4k
nHBidG3pZK11zQD6kpNDoAhDxH6WLGui6ZxSbUx7LSqLHsgzMPe
nHUcNC5ni7XjVYfCMe38Rm3KQaq27jw7wJpcUYdo4miWwpNePRTw
nHU95JxeaHJoSdpE7R49Mxp4611Yk5yL9SGEc12UDJLr4oEUN
# ... more validators
# Optional: Add custom validators
# nH... My Custom Validator
```
#### Validator List Management
The validator list can be automatically fetched from trusted sources:
```ini
[validators_file]
validators.txt
[validator_list_sites]
https://vl.ripple.com
https://vl.xrplf.org
[validator_list_keys]
ED2677ABFFD1B33AC6FBC3062B71F1E8397C1505E1C42C64D11AD1B28FF73F4734
```
This allows dynamic updates without manual configuration changes.
***
### Consensus Rounds
#### Round Structure
Consensus operates in discrete **rounds**, each typically lasting 3-5 seconds. Each round attempts to agree on the next ledger.
**Round Phases**:
```
Open Phase (20-50s) → Establish Phase (2-4s) → Accepted Phase (instant) → Validated Phase (1-2s)
↓ ↓ ↓ ↓
Collect TXs Exchange Proposals Reach Agreement Confirm Ledger
```
#### Phase 1: Open Phase
**Duration**: Variable (typically 20-50 seconds, but can be shorter)
**Purpose**: Collect transactions for the next ledger
**What Happens**:
* Transactions arrive from clients and peers
* Transactions are validated and added to the open ledger
* Each validator builds its own transaction set
* Open ledger is tentatively applied (provides immediate feedback)
**Key Point**: The open ledger is **not final**—it shows what *might* be in the next ledger, but consensus hasn't been reached yet.
```cpp
// Transactions entering open ledger
void NetworkOPs::processTransaction(
std::shared_ptr const& transaction)
{
// Apply to open ledger
auto const result = app_.openLedger().modify(
[&](OpenView& view)
{
return transaction->apply(app_, view);
});
if (result.second) // Transaction applied successfully
{
// Relay to network
app_.overlay().relay(transaction);
}
}
```
#### Phase 2: Establish Phase (Consensus Rounds)
**Duration**: 2-4 seconds (multiple sub-rounds with 50% increase each time)
**Purpose**: Validators exchange proposals and converge on a common transaction set
**Process**:
**Initial Proposal**
Each validator creates a proposal containing:
* Hash of their proposed transaction set
* Their validator signature
* Previous ledger hash
* Proposed close time
```cpp
// Simplified proposal structure
struct ConsensusProposal
{
uint256 previousLedger; // Hash of previous ledger
uint256 position; // Hash of proposed transaction set
NetClock::time_point closeTime; // Proposed close time
PublicKey publicKey; // Validator's public key
Signature signature; // Proposal signature
};
```
**Proposal Exchange**
Validators broadcast proposals to the network using `tmPROPOSE_LEDGER` messages.
**Agreement Threshold**
Validators track which transactions appear in proposals from their UNL:
* **80% agreement**: Transaction is considered "likely to be included"
* **50% agreement**: Transaction is "disputed"
* **<50% agreement**: Transaction is "unlikely to be included"
**Iterative Refinement**
Multiple rounds of proposals:
**Round 1 (Initial)**: Each validator proposes their transaction set
**Round 2 (50% threshold)**: Validators update proposals, including only transactions with >50% support
**Round 3+ (Increasing threshold)**: Threshold increases each round, converging toward agreement
```
Round 1: 50% threshold, 2 second timer
Round 2: 65% threshold, 3 second timer (50% increase)
Round 3: 80% threshold, 4.5 second timer
Round 4: 95% threshold, 6.75 second timer
...
```
**Avalanche Effect**
Once enough validators converge on the same set, others quickly follow (avalanche effect), achieving rapid consensus.
#### Phase 3: Accepted Phase
**Duration**: Instant (threshold is reached)
**Purpose**: Consensus is reached, transaction set is accepted
**Trigger**: When a validator sees >80% of its UNL agreeing on the same transaction set
**What Happens**:
* The agreed-upon transaction set becomes the "accepted" set
* Ledger is closed with this transaction set
* Validators compute the resulting ledger hash
* Consensus round ends
```cpp
// Simplified consensus acceptance check
bool hasConsensus(ConsensusMode mode, int validations)
{
if (mode == ConsensusMode::proposing)
{
// Need 80% of UNL to agree
return validations >= (unlSize_ * 4 / 5);
}
return false;
}
```
#### Phase 4: Validated Phase
**Duration**: 1-2 seconds
**Purpose**: Validators sign and broadcast validations
**What Happens**:
* Each validator applies the agreed transaction set
* Computes the resulting ledger hash
* Creates a validation message
* Signs and broadcasts the validation
**Validation Message** (`tmVALIDATION`):
```cpp
struct STValidation
{
uint256 ledgerHash; // Hash of validated ledger
uint32 ledgerSequence; // Ledger sequence number
NetClock::time_point signTime; // When validation was signed
PublicKey publicKey; // Validator's public key
Signature signature; // Validation signature
bool full; // Full validation vs partial
};
```
**Validation Collection**:
* Nodes collect validations from validators
* When >80% of trusted validators validate the same ledger hash, it's considered **fully validated**
* The ledger becomes immutable and part of the permanent ledger history
#### Consensus Round Timeline
```
Time 0s: Open Phase begins
↓
... transactions accumulate ...
↓
Time 20-50s: Consensus triggered (sufficient transactions or timer)
↓
Time 0s (consensus start): Initial proposals broadcast
↓
Time 2s: Round 1 complete, update proposals
↓
Time 5s: Round 2 complete, update proposals
↓
Time 9.5s: Round 3 complete, 80% threshold reached
↓
Time 9.5s: Accepted phase - consensus reached
↓
Time 9.5-11.5s: Validators apply transactions and create validations
↓
Time 11.5s: Validation phase - ledger fully validated
↓
Time 11.5s: Next open phase begins
```
**Total time from consensus start to validation: \~10 seconds** **Total time from transaction submission to confirmation: \~15-60 seconds** (depending on when submitted during open phase)
***
### Transaction Ordering and Determinism
#### Why Ordering Matters
For all validators to reach the same ledger state, they must apply transactions in **exactly the same order**. Different orders can produce different results:
**Example**:
```
Account A has 100 XRP
Transaction 1: Send 60 XRP to B
Transaction 2: Send 60 XRP to C
Order 1 (TX1 then TX2):
- TX1 succeeds: A has 40 XRP, B has 60 XRP
- TX2 fails: insufficient balance
Order 2 (TX2 then TX1):
- TX2 succeeds: A has 40 XRP, C has 60 XRP
- TX1 fails: insufficient balance
Different results!
```
#### Canonical Ordering
The XRP Ledger uses **canonical ordering** to ensure determinism:
**Primary Sort**: By account (lexicographic order of account IDs)
**Secondary Sort**: By transaction sequence number (nonce)
```cpp
// Canonical transaction ordering
bool txOrderCompare(STTx const& tx1, STTx const& tx2)
{
// First, sort by account
if (tx1.getAccountID(sfAccount) < tx2.getAccountID(sfAccount))
return true;
if (tx1.getAccountID(sfAccount) > tx2.getAccountID(sfAccount))
return false;
// Same account, sort by sequence number
return tx1.getSequence() < tx2.getSequence();
}
```
This ensures:
* All transactions from the same account are processed in sequence order
* Transactions from different accounts are processed in a deterministic order
* All validators apply transactions identically
#### Transaction Set Hash
The transaction set is represented by a hash:
```cpp
// Calculate transaction set hash
uint256 calculateTxSetHash(std::vector const& transactions)
{
// Sort transactions canonically
auto sortedTxs = transactions;
std::sort(sortedTxs.begin(), sortedTxs.end(), txOrderCompare);
// Hash all transactions together
Serializer s;
for (auto const& tx : sortedTxs)
{
s.addBitString(tx.getHash());
}
return s.getSHA512Half();
}
```
This hash is what validators include in their proposals—a compact representation of the entire transaction set.
***
### Dispute Resolution
#### What is a Dispute?
A **dispute** occurs when validators initially disagree about which transactions should be included in the next ledger. This is normal and expected—validators may have different views due to:
* Network latency (different arrival times)
* Transaction validity questions
* Different open ledger states
#### Resolution Process
Disputes are resolved through the iterative consensus rounds:
**Round 1: Initial Disagreement**
```
Validator A proposes: {TX1, TX2, TX3, TX4, TX5}
Validator B proposes: {TX1, TX2, TX3, TX6, TX7}
Validator C proposes: {TX1, TX2, TX4, TX5, TX8}
Agreement:
- TX1: 100% (all three)
- TX2: 100% (all three)
- TX3: 67% (A, B)
- TX4: 67% (A, C)
- TX5: 67% (A, C)
- TX6: 33% (B only)
- TX7: 33% (B only)
- TX8: 33% (C only)
```
**Round 2: Converge on High-Agreement TXs**
Validators drop transactions with <50% support:
```
Validator A proposes: {TX1, TX2, TX3, TX4, TX5} (drops nothing, all >50%)
Validator B proposes: {TX1, TX2, TX3} (drops TX6, TX7)
Validator C proposes: {TX1, TX2, TX4, TX5} (drops TX8)
Agreement:
- TX1: 100%
- TX2: 100%
- TX3: 67% (A, B)
- TX4: 67% (A, C)
- TX5: 67% (A, C)
```
**Round 3: Further Convergence**
As threshold increases to 80%, validators must drop disputed transactions:
```
All validators propose: {TX1, TX2}
Agreement:
- TX1: 100% ✓ (exceeds 80%)
- TX2: 100% ✓ (exceeds 80%)
Consensus reached on {TX1, TX2}
TX3, TX4, TX5 deferred to next ledger
```
#### Deferred Transactions
Transactions that don't reach consensus are not lost:
* They remain in the open ledger
* They'll be included in the next consensus round
* They're only dropped if they become invalid
#### Byzantine Validators
If a validator behaves maliciously:
* Their proposals are signed, so misbehavior is detectable
* Other validators ignore proposals that don't follow protocol rules
* Byzantine validators cannot force consensus on invalid states (requires >80% support)
* Operators can remove misbehaving validators from their UNL
***
### Ledger Close Process
#### Close Triggers
A ledger close is triggered when:
**Timer-based**: Minimum time has elapsed (typically 2-10 seconds)
**Transaction threshold**: Sufficient transactions have accumulated
**Consensus readiness**: Validators are ready to reach agreement
```cpp
// Simplified close trigger logic
bool shouldCloseLedger()
{
auto const elapsed = std::chrono::steady_clock::now() - lastClose_;
// Minimum close interval elapsed?
if (elapsed < minCloseInterval_)
return false;
// Sufficient transactions?
if (openLedger_.size() >= closeThreshold_)
return true;
// Maximum close interval elapsed?
if (elapsed >= maxCloseInterval_)
return true;
return false;
}
```
#### Close Time Agreement
Validators must also agree on the **close time** of the ledger:
**Why it matters**: Some transactions are time-dependent (escrows, offers with expiration)
**Process**:
* Each validator proposes a close time
* Consensus includes the close time in proposals
* Final close time is the median of proposed times (Byzantine fault tolerant)
**Close Time Resolution**:
* Rounded to nearest 10 seconds for efficiency
* Prevents clock skew from causing issues
#### Post-Consensus Application
After consensus is reached:
**Step 1**: Apply agreed transaction set
```cpp
// Apply transactions in canonical order
for (auto const& tx : canonicalOrder(agreedTxSet))
{
auto const result = applyTransaction(tx, view);
// Record metadata
}
```
**Step 2**: Compute ledger hash
```cpp
// Hash includes:
// - Parent ledger hash
// - Transaction set hash
// - Account state hash
// - Close time
// - Ledger sequence
auto ledgerHash = computeLedgerHash(
parentHash,
txSetHash,
stateHash,
closeTime,
ledgerSeq);
```
**Step 3**: Create and broadcast validation
```cpp
STValidation validation;
validation.setLedgerHash(ledgerHash);
validation.setLedgerSequence(ledgerSeq);
validation.setSignTime(now);
validation.sign(validatorKey);
overlay().broadcast(validation);
```
**Step 4**: Collect validations
```cpp
// Wait for validations from UNL
while (validationCount < unlSize_ * 4 / 5)
{
// Process incoming validations
auto val = receiveValidation();
if (val.getLedgerHash() == ledgerHash)
validationCount++;
}
// Ledger is now fully validated
ledgerMaster_.setFullyValidated(ledger);
```
***
### Codebase Deep Dive
#### Key Files and Directories
**Consensus Core**:
* `src/ripple/consensus/Consensus.h` - Main consensus engine interface
* `src/ripple/consensus/ConsensusProposal.h` - Proposal structure
* `src/ripple/consensus/Validations.h` - Validation tracking
**Consensus Implementation**:
* `src/ripple/app/consensus/RCLConsensus.h` - XRP Ledger-specific consensus
* `src/ripple/app/consensus/RCLValidations.cpp` - Validation handling
**Network Messages**:
* `src/ripple/overlay/impl/ProtocolMessage.h` - tmPROPOSE\_LEDGER, tmVALIDATION
**Configuration**:
* `validators.txt` - UNL configuration
* `rippled.cfg` - Validator key configuration
#### Key Classes
**Consensus Class**
```cpp
template
class Consensus
{
public:
// Start new consensus round
void startRound(
LedgerHash const& prevLedgerHash,
Ledger const& prevLedger,
NetClock::time_point closeTime);
// Process peer proposal
void peerProposal(
NetClock::time_point now,
ConsensusProposal const& proposal);
// Simulate new round
void timerEntry(NetClock::time_point now);
// Check if consensus reached
bool haveConsensus() const;
private:
// Current round state
ConsensusPhase phase_;
std::map peerProposals_;
std::set disputes_;
TxSet ourPosition_;
};
```
**RCLConsensus (Ripple Consensus Ledger)**
XRP Ledger-specific consensus implementation:
```cpp
class RCLConsensus
{
public:
// Handle consensus result
void onAccept(
Result const& result,
RCLCxLedger const& prevLedger,
NetClock::duration closeResolution,
CloseTimes const& rawCloseTimes,
ConsensusMode mode);
// Create initial position
RCLTxSet getInitialPosition(
RCLCxLedger const& prevLedger);
// Check if we should close ledger
void checkClose(NetClock::time_point now);
};
```
#### Code Navigation Tips
**Finding Consensus Start**
Search for ledger close triggers:
```cpp
// In NetworkOPs or LedgerMaster
void beginConsensus(LedgerHash const& prevHash)
{
// Build initial transaction set
auto initialSet = buildTxSet();
// Start consensus round
consensus_.startRound(
prevHash,
prevLedger,
suggestCloseTime());
}
```
**Tracing Proposal Handling**
Follow proposal processing:
```cpp
// Overlay receives tmPROPOSE_LEDGER message
void onProposal(std::shared_ptr const& proposal)
{
// Validate proposal signature
if (!verifyProposal(proposal))
return;
// Pass to consensus engine
consensus_.peerProposal(
now(),
parseProposal(proposal));
}
```
**Understanding Validation**
Follow validation creation and verification:
```cpp
// Create validation
auto validation = std::make_shared(
ledgerHash,
signTime,
publicKey,
nodeID,
[&](STValidation& v)
{
v.sign(secretKey);
});
// Broadcast to network
overlay().send(validation);
```
***
### Hands-On Exercise
#### Exercise: Observe and Analyze a Consensus Round
**Objective**: Watch a complete consensus round and understand the proposal exchange process.
**Part 1: Setup Multi-Validator Environment**
This is advanced—requires multiple validators. For learning, we'll use logs from a single validator.
**Step 1**: Enable detailed consensus logging
Edit `rippled.cfg`:
```ini
[rpc_startup]
{ "command": "log_level", "partition": "Consensus", "severity": "trace" }
{ "command": "log_level", "partition": "LedgerConsensus", "severity": "trace" }
```
**Step 2**: Start rippled
```bash
rippled --conf=rippled.cfg
```
**Step 3**: Watch the logs
```bash
tail -f /var/log/rippled/debug.log | grep -E "Consensus|Proposal|Validation"
```
**Part 2: Identify Consensus Phases**
From the logs, identify:
**Open Phase Start**:
```
"Consensus":"open ledger started, seq=12345"
```
**Consensus Round Start**:
```
"Consensus":"Starting consensus round, prevLedger=ABCD..."
```
**Proposals Received**:
```
"Consensus":"Received proposal from nHU...., position=XYZ..."
```
**Agreement Tracking**:
```
"Consensus":"Transaction TX1 has 85% agreement"
"Consensus":"Transaction TX2 has 45% agreement (disputed)"
```
**Consensus Reached**:
```
"Consensus":"Consensus reached on transaction set, hash=..."
```
**Validation Created**:
```
"Consensus":"Created validation for ledger 12345, hash=..."
```
**Validation Received**:
```
"Validations":"Received validation from nHU...., ledger=12345"
```
**Ledger Fully Validated**:
```
"LedgerConsensus":"Ledger 12345 fully validated with 28/35 validations"
```
**Part 3: Timing Analysis**
Measure the duration of each phase:
1. **Open Phase Duration**: Time between "open ledger started" and "Starting consensus round"
2. **Consensus Duration**: Time from "Starting consensus round" to "Consensus reached"
3. **Validation Duration**: Time from "Consensus reached" to "fully validated"
**Create a timeline**:
```
T+0s: Open phase begins
T+25s: Consensus triggered
T+27s: Consensus reached (2s consensus time)
T+29s: Ledger fully validated (2s validation time)
T+29s: Next open phase begins
Total cycle: 29 seconds
```
**Part 4: Analysis Questions**
Answer these based on your observations:
1. **How many consensus rounds occurred?**
* Count from logs
2. **What was the average consensus time?**
* Measure multiple rounds
3. **How many transactions were included in each ledger?**
* Look for transaction count in logs
4. **Were there any disputed transactions?**
* Look for agreement percentages <100%
5. **How many validations did each ledger receive?**
* Count validation messages
6. **What percentage of UNL validated each ledger?**
* Compare validations received vs UNL size
**Part 5: Compare to Whitepaper**
Read the XRP Ledger Consensus Protocol whitepaper and compare:
* Does the observed behavior match the description?
* Are the timing estimates accurate?
* How does the network handle disputes?
***
### Key Takeaways
#### Core Concepts
✅ **Byzantine Fault Tolerance**: Network can tolerate up to 20% faulty validators while maintaining security
✅ **UNL-Based Trust**: Each validator chooses which other validators to trust, creating a trust graph
✅ **Iterative Consensus**: Multiple rounds of proposals converge on an agreed transaction set
✅ **Fast Finality**: 3-5 second consensus rounds enable quick transaction confirmation
✅ **No Mining**: Consensus achieved through voting, not computational work
✅ **Deterministic Ordering**: Canonical transaction ordering ensures all validators reach identical state
✅ **Dispute Resolution**: Disagreements resolved by dropping disputed transactions to next round
#### Security Properties
✅ **Safety**: No conflicting ledgers validated (no forks in normal operation)
✅ **Liveness**: Network makes progress as long as >80% of UNL is honest and responsive
✅ **Censorship Resistance**: No single entity can block valid transactions
✅ **Sybil Resistance**: Trust relationships (UNL) prevent fake validator attacks
#### Development Skills
✅ **Codebase Location**: Consensus implementation in `src/ripple/consensus/` and `src/ripple/app/consensus/`
✅ **Proposal Format**: Understand ConsensusProposal structure and tmPROPOSE\_LEDGER messages
✅ **Validation Format**: Understand STValidation structure and tmVALIDATION messages
✅ **Debugging**: Use consensus logs to trace round progression and identify issues
***
### Common Misconceptions
#### Misconception 1: "Validators mine like Bitcoin"
**False**: Validators don't perform computational work. They simply vote on which transactions to include.
#### Misconception 2: "Ripple controls consensus"
**False**: Any organization can run validators, and each validator operator independently chooses their UNL. While many operators use the recommended UNL from the XRP Ledger Foundation, they're free to customize it.
#### Misconception 3: "All servers participate in consensus"
**False**: Most rippled servers are tracking servers that follow consensus but don't vote. Only configured validators participate.
#### Misconception 4: "Consensus can be blocked by one entity"
**False**: As long as >80% of a validator's UNL is operational and honest, consensus proceeds normally.
#### Misconception 5: "XRP Ledger has forked"
**False**: The XRP Ledger has never had a fork (competing chains). The consensus protocol prevents this by design.
***
### Additional Resources
#### Official Documentation
* **XRP Ledger Dev Portal**: [xrpl.org/docs](https://xrpl.org/docs)
* **Consensus Protocol**: [xrpl.org/consensus](https://xrpl.org/docs/concepts/consensus-protocol/)
* **Run a Validator**: [xrpl.org/run-a-rippled-validator](https://xrpl.org/docs/infrastructure/configuration/server-modes/run-rippled-as-a-validator)
#### Academic Papers
* **Original Consensus Whitepaper**: David Schwartz, Noah Youngs, Arthur Britto
* **Analysis of the XRP Ledger Consensus Protocol**: Brad Chase, Ethan MacBrough
* **Cobalt: BFT Governance in Open Networks**: Ethan MacBrough
#### Codebase References
* `src/ripple/consensus/` - Generic consensus framework
* `src/ripple/app/consensus/` - XRP Ledger-specific implementation
* `src/ripple/app/ledger/ConsensusTransSetSF.cpp` - Transaction set management
#### Related Topics
* [Protocols](/core-dev-bootcamp/module02/protocols-communication-and-interoperability) - How consensus messages are propagated
* [Application Layer](/core-dev-bootcamp/module02/application-layer-central-orchestration-and-coordination) - How consensus integrates with other components
* [Transaction Lifecycle](/core-dev-bootcamp/module02/transaction-lifecycle-complete-transaction-journey) - How transactions flow through consensus
***
# Overlay Network: Peer-to-Peer Networking Layer
[← Back to Rippled II Overview](/core-dev-bootcamp/module02)
***
### Introduction
The Overlay Network is Rippled's peer-to-peer networking layer that enables distributed nodes to discover each other, establish connections, and communicate efficiently. Without the overlay network, the XRP Ledger would be a collection of isolated servers—the overlay network is what transforms individual nodes into a cohesive, decentralized system.
Understanding the overlay network is essential for debugging connectivity issues, optimizing network performance, and ensuring your node participates effectively in the XRP Ledger network. Whether you're running a validator, a stock server, or developing network enhancements, deep knowledge of the overlay network is crucial.
{% embed url="" %}
***
### Network Topology and Architecture
#### Mesh Network Design
The XRP Ledger uses a **mesh topology** where nodes maintain direct connections with multiple peers. This differs from:
* **Star topology**: Central hub (single point of failure)
* **Ring topology**: Sequential connections (vulnerable to breaks)
* **Tree topology**: Hierarchical structure (root node critical)
**Mesh Advantages**:
* **No single point of failure**: Network remains operational if individual nodes fail
* **Multiple communication paths**: Messages can route around failed nodes
* **Scalability**: Network can grow organically as nodes join
* **Resilience**: Network topology self-heals as nodes enter and exit
#### Network Layers
```
┌─────────────────────────────────────────────┐
│ Application Layer │
│ (Consensus, Transactions, Ledger) │
├─────────────────────────────────────────────┤
│ Overlay Network Layer │
│ (Peer Discovery, Connection Mgmt, │
│ Message Routing) │
├─────────────────────────────────────────────┤
│ Transport Layer (TCP/TLS) │
├─────────────────────────────────────────────┤
│ Internet Layer (IP) │
└─────────────────────────────────────────────┘
```
The overlay network sits between the application logic and the transport layer, abstracting away the complexities of peer-to-peer communication.
#### Connection Types
Rippled maintains three types of peer connections:
**1. Outbound Connections**
**Definition**: Connections initiated by your node to other peers
**Characteristics**:
* Your node acts as client
* You choose which peers to connect to
* Configurable connection limits
* Active connection management
**Configuration**:
```ini
[ips]
# DNS or IP addresses to connect to
r.ripple.com 51235
s1.ripple.com 51235
s2.ripple.com 51235
```
**2. Inbound Connections**
**Definition**: Connections initiated by other nodes to your server
**Characteristics**:
* Your node acts as server
* Must listen on public interface
* Accept connections from unknown peers
* Subject to connection limits
**Configuration**:
```ini
[port_peer]
port = 51235
ip = 0.0.0.0 # Listen on all interfaces
protocol = peer
```
**3. Fixed Connections**
**Definition**: Persistent connections to trusted peers
**Characteristics**:
* High priority, always maintained
* Automatically reconnect if disconnected
* Bypass some connection limits
* Ideal for validators and cluster peers
**Configuration**:
```ini
[ips_fixed]
# Always maintain connections to these peers
validator1.example.com 51235
validator2.example.com 51235
cluster-peer.example.com 51235
```
#### Target Connection Count
Rippled aims to maintain a target number of active peer connections:
**Default Targets** (based on `node_size`):
```
tiny: 10 peers
small: 15 peers
medium: 20 peers (default)
large: 30 peers
huge: 40 peers
```
**Connection Distribution**:
* Approximately 50% outbound connections
* Approximately 50% inbound connections
* Fixed connections count toward total
* System adjusts dynamically to maintain target
***
### Peer Discovery Mechanisms
#### 1. Configured Peer Lists
The most basic discovery method—manually configured peers:
**`[ips]` Section**: Peers to connect to automatically
```ini
[ips]
r.ripple.com 51235
s1.ripple.com 51235
validator.example.com 51235
```
**`[ips_fixed]` Section**: High-priority persistent connections
```ini
[ips_fixed]
critical-peer.example.com 51235
```
**Advantages**:
* Reliable, known peers
* Administrative control
* Suitable for private networks
**Disadvantages**:
* Manual maintenance required
* Limited to configured peers
* Doesn't scale automatically
#### 2. DNS Seeds
DNS-based peer discovery for bootstrap:
**How It Works**:
1. Node queries DNS for peer addresses
2. DNS returns A records (IP addresses)
3. Node connects to returned addresses
4. Learns about additional peers through gossip
**Configuration**:
```ini
[ips]
# These resolve via DNS
r.ripple.com 51235
s1.ripple.com 51235
```
**DNS Resolution Example**:
```bash
$ dig +short r.ripple.com
54.186.73.52
54.184.149.41
52.24.169.78
```
**Advantages**:
* Easy bootstrap for new nodes
* Dynamic peer lists
* Load balancing via DNS
**Disadvantages**:
* Requires DNS infrastructure
* Vulnerable to DNS attacks
* Single point of failure for initial connection
#### 3. Peer Gossip Protocol
Peers share information about other peers they know:
**Message Type**: Endpoint announcements (part of peer protocol)
**Process**:
1. Peer A connects to Peer B
2. Peer B shares list of other known peers
3. Peer A considers these peers for connection
4. Peer A may connect to some of the suggested peers
**Gossip Information Includes**:
* Peer IP addresses
* Peer public keys
* Last seen time
* Connection quality hints
**Advantages**:
* Network self-organizes
* No central directory needed
* Discovers new peers automatically
* Network grows organically
**Disadvantages**:
* Potential for malicious peer injection
* Network topology influenced by gossip patterns
* Initial bootstrapping still needed
#### 4. Peer Crawler
Some nodes run peer crawlers to discover and monitor network topology:
**What Crawlers Do**:
* Connect to known peers
* Request peer lists
* Recursively discover more peers
* Map network topology
* Provide public peer directories
**Public Peer Lists**:
* Various community-maintained lists
* Used by new nodes to bootstrap
* Updated regularly
***
### Connection Establishment and Handshake
#### Connection Lifecycle
```
┌──────────────┐
│ Disconnected│
└──────┬───────┘
│ initiate()
↓
┌──────────────┐
│ Connecting │ ← TCP handshake, TLS negotiation
└──────┬───────┘
│ connected()
↓
┌──────────────┐
│ Connected │ ← Protocol handshake in progress
└──────┬───────┘
│ handshake complete
↓
┌──────────────┐
│ Active │ ← Fully operational, exchanging messages
└──────┬───────┘
│ close() or error
↓
┌──────────────┐
│ Closing │ ← Graceful shutdown
└──────┬───────┘
│
↓
┌──────────────┐
│ Closed │
└──────────────┘
```
#### Detailed Handshake Process
**Step 1: TCP Connection**
Standard TCP three-way handshake:
```
Client Server
│ │
│──────── SYN ──────────────────────>│
│ │
│<─────── SYN-ACK ──────────────────│
│ │
│──────── ACK ──────────────────────>│
│ │
│ TCP Connection Established │
```
**Configuration**:
```ini
[port_peer]
port = 51235
ip = 0.0.0.0
protocol = peer
```
**Step 2: TLS Handshake (Optional but Recommended)**
If TLS is configured, encrypted channel is established:
```
Client Server
│ │
│──────── ClientHello ──────────────>│
│ │
│<─────── ServerHello ──────────────│
│<─────── Certificate ──────────────│
│<─────── ServerHelloDone ──────────│
│ │
│──────── ClientKeyExchange ────────>│
│──────── ChangeCipherSpec ─────────>│
│──────── Finished ─────────────────>│
│ │
│<─────── ChangeCipherSpec ─────────│
│<─────── Finished ─────────────────│
│ │
│ Encrypted Channel Established │
```
**Benefits of TLS**:
* Encrypted communication (privacy)
* Peer authentication (security)
* Protection against eavesdropping
* Man-in-the-middle prevention
**Step 3: Protocol Handshake**
Rippled-specific handshake exchanges capabilities:
**Hello Message** (from initiator):
```protobuf
message TMHello {
required uint32 protoVersion = 1; // Protocol version
required uint32 protoVersionMin = 2; // Minimum supported version
required bytes publicKey = 3; // Node's public key
optional bytes nodePrivate = 4; // Proof of key ownership
required uint32 ledgerIndex = 5; // Current ledger index
optional bytes ledgerClosed = 6; // Closed ledger hash
optional bytes ledgerPrevious = 7; // Previous ledger hash
optional uint32 closedTime = 8; // Ledger close time
}
```
**Response** (from receiver):
```protobuf
// Same TMHello structure with receiver's information
```
**Handshake Validation**:
```cpp
bool validateHandshake(TMHello const& hello)
{
// Check protocol version compatibility
if (hello.protoVersion < minSupportedVersion)
return false;
if (hello.protoVersionMin > currentVersion)
return false;
// Verify public key
if (!isValidPublicKey(hello.publicKey()))
return false;
// Verify key ownership proof
if (!verifySignature(hello.nodePrivate(), hello.publicKey()))
return false;
// Check we're on same network (same genesis ledger)
if (!isSameNetwork(hello.ledgerClosed()))
return false;
return true;
}
```
**Compatibility Check**:
```
Node A: version 1.7.0, min 1.5.0
Node B: version 1.6.0, min 1.4.0
Check: max(1.5.0, 1.4.0) ≤ min(1.7.0, 1.6.0)
1.5.0 ≤ 1.6.0 ✓ Compatible
Use protocol version: 1.6.0 (minimum of max versions)
```
**Step 4: Connection Acceptance/Rejection**
After handshake validation:
**If Compatible**:
* Connection moves to Active state
* Add to peer list
* Begin normal message exchange
* Log successful connection
**If Incompatible**:
* Send rejection message with reason
* Close connection gracefully
* Log rejection reason
* May add to temporary ban list
**Rejection Reasons**:
```cpp
enum DisconnectReason
{
drBadData, // Malformed handshake
drProtocol, // Protocol incompatibility
drSaturated, // Too many connections
drDuplicate, // Already connected to this peer
drNetworkID, // Different network (testnet vs mainnet)
drBanned, // Peer is banned
drSelf, // Trying to connect to self
};
```
***
### Connection Management
#### Connection Limits
Rippled enforces various connection limits:
**Per-IP Limits**
```cpp
// Maximum connections from single IP
constexpr size_t maxPeersPerIP = 2;
// Prevents single entity from dominating connections
bool acceptConnection(IPAddress const& ip)
{
auto count = countConnectionsFromIP(ip);
return count < maxPeersPerIP;
}
```
**Total Connection Limits**
Based on `node_size` configuration:
```
tiny: max 10 connections
small: max 21 connections
medium: max 40 connections
large: max 62 connections
huge: max 88 connections
```
Formula: `target + (target / 2)`
**Fixed Peer Priority**
Fixed peers bypass some limits:
```cpp
bool shouldAcceptConnection(Peer const& peer)
{
// Always accept fixed peers
if (isFixed(peer))
return true;
// Check against limits for regular peers
if (activeConnections() >= maxConnections())
return false;
return true;
}
```
#### Connection Quality Assessment
Rippled continuously monitors peer quality:
**Metrics Tracked**
**Latency**: Response time to ping messages
```cpp
// Ping-pong protocol
void sendPing()
{
auto ping = std::make_shared();
ping->set_type(protocol::TMPing::ptPING);
ping->set_seq(nextPingSeq_++);
ping->set_timestamp(now());
send(ping);
}
void onPong(protocol::TMPing const& pong)
{
auto latency = now() - pong.timestamp();
updateLatencyMetrics(latency);
}
```
**Message Rate**: Messages per second
```cpp
void trackMessageRate()
{
messagesReceived_++;
auto elapsed = now() - windowStart_;
if (elapsed >= 1s)
{
messageRate_ = messagesReceived_ / elapsed.count();
messagesReceived_ = 0;
windowStart_ = now();
}
}
```
**Error Rate**: Protocol errors, malformed messages
```cpp
void onProtocolError()
{
errorCount_++;
if (errorCount_ > maxErrorThreshold)
{
// Disconnect problematic peer
disconnect(drBadData);
}
}
```
**Uptime**: Connection duration
```cpp
auto uptime = now() - connectionTime_;
```
**Quality Scoring**
Peers are scored based on metrics:
```cpp
int calculatePeerScore(Peer const& peer)
{
int score = 100; // Start with perfect score
// Penalize high latency
if (peer.latency() > 500ms)
score -= 20;
else if (peer.latency() > 200ms)
score -= 10;
// Penalize low message rate (inactive peer)
if (peer.messageRate() < 0.1)
score -= 15;
// Penalize errors
score -= peer.errorCount() * 5;
// Reward long uptime
if (peer.uptime() > 24h)
score += 10;
return std::max(0, std::min(100, score));
}
```
**Score Usage**:
* Low-scoring peers may be disconnected
* High-scoring peers prioritized for reconnection
* Informs peer selection decisions
#### Connection Pruning
When connection limits are reached, low-quality peers are pruned:
```cpp
void pruneConnections()
{
if (activeConnections() <= targetConnections())
return;
// Sort peers by score (lowest first)
auto peers = getAllPeers();
std::sort(peers.begin(), peers.end(),
[](auto const& a, auto const& b)
{
return a->score() < b->score();
});
// Disconnect lowest-scoring non-fixed peers
for (auto& peer : peers)
{
if (isFixed(peer))
continue; // Never disconnect fixed peers
peer->disconnect(drSaturated);
if (activeConnections() <= targetConnections())
break;
}
}
```
#### Reconnection Logic
After disconnection, Rippled may attempt to reconnect:
**Exponential Backoff**:
```cpp
Duration calculateReconnectDelay(int attempts)
{
// Exponential backoff with jitter
auto delay = minDelay * std::pow(2, attempts);
delay = std::min(delay, maxDelay);
// Add random jitter (±25%)
auto jitter = delay * (0.75 + random() * 0.5);
return jitter;
}
// Example progression:
// Attempt 1: ~5 seconds
// Attempt 2: ~10 seconds
// Attempt 3: ~20 seconds
// Attempt 4: ~40 seconds
// Attempt 5+: ~60 seconds (capped)
```
**Fixed Peer Priority**:
```cpp
void scheduleReconnect(Peer const& peer)
{
Duration delay;
if (isFixed(peer))
{
// Aggressive reconnection for fixed peers
delay = 5s;
}
else
{
// Exponential backoff for regular peers
delay = calculateReconnectDelay(peer.reconnectAttempts());
}
scheduleJob(delay, [this, peer]()
{
attemptConnection(peer.address());
});
}
```
***
### Message Routing and Broadcasting
#### Message Types
Different message types require different routing strategies:
**Critical Messages (Broadcast to All)**
**Validations** (`tmVALIDATION`):
* Must reach all validators
* Broadcast to all peers immediately
* Critical for consensus
**Consensus Proposals** (`tmPROPOSE_LEDGER`):
* Must reach all validators
* Time-sensitive
* Broadcast widely
**Broadcast Pattern**:
```cpp
void broadcastCritical(std::shared_ptr const& msg)
{
for (auto& peer : getAllPeers())
{
// Send to everyone
peer->send(msg);
}
}
```
**Transactions (Selective Relay)**
**Transaction Messages** (`tmTRANSACTION`):
* Should reach all nodes eventually
* Don't need immediate broadcast to all
* Use intelligent relay
**Relay Logic**:
```cpp
void relayTransaction(
std::shared_ptr const& msg,
Peer* source)
{
for (auto& peer : getAllPeers())
{
// Don't echo back to source
if (peer.get() == source)
continue;
// Check if peer likely already has it
if (peerLikelyHas(peer, msg))
continue;
// Send to peer
peer->send(msg);
}
}
```
**Request/Response (Unicast)**
**Ledger Data Requests** (`tmGET_LEDGER`):
* Directed to specific peer
* Response goes back to requester
* No broadcasting needed
**Unicast Pattern**:
```cpp
void requestLedgerData(
LedgerHash const& hash,
Peer* peer)
{
auto request = makeGetLedgerMessage(hash);
peer->send(request); // Send only to this peer
}
```
#### Squelch Algorithm
**Squelch** prevents message echo loops:
**Problem**:
```
Node A → sends to B
Node B → receives from A
Node B → broadcasts to all (including A)
Node A → receives echo from B
Node A → broadcasts again...
(infinite loop)
```
**Solution**:
```cpp
void onMessageReceived(
std::shared_ptr const& msg,
Peer* source)
{
// Track message hash
auto hash = msg->getHash();
// Have we seen this before?
if (recentMessages_.contains(hash))
return; // Ignore duplicate
// Record that we've seen it
recentMessages_.insert(hash);
// Process message
processMessage(msg);
// Relay to others (excluding source)
relayToOthers(msg, source);
}
```
**Recent Message Cache**:
* Time-based expiration (e.g., 30 seconds)
* Size-based limits (e.g., 10,000 entries)
* LRU eviction policy
#### Message Priority Queues
Outbound messages are queued with priority:
```cpp
enum MessagePriority
{
priVeryHigh, // Validations, critical consensus
priHigh, // Proposals, status changes
priMedium, // Transactions
priLow, // Historical data, maintenance
};
class PeerMessageQueue
{
private:
std::map> queues_;
public:
void enqueue(Message msg, MessagePriority priority)
{
queues_[priority].push(msg);
}
Message dequeue()
{
// Dequeue from highest priority non-empty queue
for (auto& [priority, queue] : queues_)
{
if (!queue.empty())
{
auto msg = queue.front();
queue.pop();
return msg;
}
}
throw std::runtime_error("No messages");
}
};
```
**Benefits**:
* Critical messages sent first
* Prevents head-of-line blocking
* Better network utilization
***
### Network Health and Monitoring
#### Health Metrics
**Connectivity Metrics**
**Active Peers**: Current peer count
```cpp
size_t activePeers = overlay.size();
```
**Target vs Actual**: Comparison to target
```cpp
bool isHealthy = activePeers >= (targetPeers * 0.75);
```
**Connection Distribution**:
```cpp
size_t outbound = countOutboundPeers();
size_t inbound = countInboundPeers();
float ratio = float(outbound) / inbound;
// Healthy: ratio between 0.5 and 2.0
bool balancedConnections = (ratio > 0.5 && ratio < 2.0);
```
**Network Quality Metrics**
**Average Latency**:
```cpp
auto avgLatency = calculateAverageLatency(getAllPeers());
// Healthy: < 200ms average
bool lowLatency = avgLatency < 200ms;
```
**Message Rate**:
```cpp
auto totalRate = sumMessageRates(getAllPeers());
// Messages per second across all peers
```
**Validator Connectivity**:
```cpp
auto validatorPeers = countValidatorPeers();
auto unlSize = getUNLSize();
// Should be connected to most of UNL
bool goodValidatorConnectivity =
validatorPeers >= (unlSize * 0.8);
```
#### RPC Monitoring Commands
**peers Command**
Get current peer list:
```bash
rippled peers
```
Response:
```json
{
"result": {
"peers": [
{
"address": "54.186.73.52:51235",
"latency": 45,
"uptime": 3600,
"version": "rippled-1.9.0",
"public_key": "n9KorY8QtTdRx...",
"complete_ledgers": "32570-75234891"
}
// ... more peers
]
}
}
```
**peer\_reservations Command**
View reserved peer slots:
```bash
rippled peer_reservations_add
rippled peer_reservations_list
```
**connect Command**
Manually connect to peer:
```bash
rippled connect 192.168.1.100:51235
```
#### Logging and Diagnostics
Enable detailed overlay logging:
```ini
[rpc_startup]
{ "command": "log_level", "partition": "Overlay", "severity": "trace" }
```
**Log Messages to Monitor**:
```
"Overlay": "Connected to peer 54.186.73.52:51235"
"Overlay": "Disconnected from peer 54.186.73.52:51235, reason: saturated"
"Overlay": "Handshake failed with peer: protocol version mismatch"
"Overlay": "Received invalid message from peer, closing connection"
"Overlay": "Active peers: 18/20 (target)"
```
***
### Codebase Deep Dive
#### Key Files and Directories
**Overlay Core**:
* `src/ripple/overlay/Overlay.h` - Main overlay interface
* `src/ripple/overlay/impl/OverlayImpl.h` - Implementation header
* `src/ripple/overlay/impl/OverlayImpl.cpp` - Core implementation
**Peer Management**:
* `src/ripple/overlay/Peer.h` - Peer interface
* `src/ripple/overlay/impl/PeerImp.h` - Peer implementation
* `src/ripple/overlay/impl/PeerImp.cpp` - Peer logic
**Connection Handling**:
* `src/ripple/overlay/impl/ConnectAttempt.h` - Outbound connections
* `src/ripple/overlay/impl/InboundHandoff.h` - Inbound connections
**Message Processing**:
* `src/ripple/overlay/impl/ProtocolMessage.h` - Message definitions
* `src/ripple/overlay/impl/Message.cpp` - Message handling
#### Key Classes
**Overlay Class**
```cpp
class Overlay
{
public:
// Start/stop overlay network
virtual void start() = 0;
virtual void stop() = 0;
// Peer management
virtual void connect(std::string const& ip) = 0;
virtual std::size_t size() const = 0;
// Message broadcasting
virtual void broadcast(std::shared_ptr const&) = 0;
virtual void relay(
std::shared_ptr const&,
Peer* source = nullptr) = 0;
// Peer information
virtual Json::Value json() = 0;
virtual std::vector getActivePeers() = 0;
};
```
**PeerImp Class**
```cpp
class PeerImp : public Peer
{
public:
// Send message to this peer
void send(std::shared_ptr const& m) override;
// Process received message
void onMessage(std::shared_ptr const& m);
// Connection state
bool isConnected() const;
void disconnect(DisconnectReason reason);
// Quality metrics
std::chrono::milliseconds latency() const;
int score() const;
private:
// Connection management
boost::asio::ip::tcp::socket socket_;
boost::asio::ssl::stream stream_;
// Message queues
std::queue> sendQueue_;
// Metrics
std::chrono::steady_clock::time_point connected_;
std::chrono::milliseconds latency_;
int score_;
};
```
#### Code Navigation Tips
**Finding Connection Logic**
Search for connection establishment:
```cpp
// In OverlayImpl.cpp
void OverlayImpl::connect(std::string const& ip)
{
// Parse IP and port
auto endpoint = parseEndpoint(ip);
// Create connection attempt
auto attempt = std::make_shared(
app_,
io_service_,
endpoint,
peerFinder_.config());
// Begin async connection
attempt->run();
}
```
**Tracing Message Flow**
Follow message from receipt to processing:
```cpp
// PeerImp::onMessage (entry point)
void PeerImp::onMessage(std::shared_ptr const& msg)
{
// Check for duplicates (squelch)
if (app_.overlay().hasSeen(msg->getHash()))
return;
// Mark as seen
app_.overlay().markSeen(msg->getHash());
// Process based on type
switch (msg->getType())
{
case protocol::mtTRANSACTION:
onTransaction(msg);
break;
case protocol::mtVALIDATION:
onValidation(msg);
break;
// ... other types
}
// Relay to other peers
app_.overlay().relay(msg, this);
}
```
***
### Hands-On Exercise
#### Exercise: Monitor and Analyze Network Topology
**Objective**: Understand your node's position in the network and analyze peer connections.
**Part 1: Initial Network State**
**Step 1**: Get current peer list
```bash
rippled peers > peers_initial.json
```
**Step 2**: Analyze the output
Count:
* Total peers
* Outbound vs inbound connections
* Peer versions
* Geographic distribution (if known)
**Questions**:
* Do you have the target number of peers?
* Is the outbound/inbound ratio balanced?
* Are you connected to validators in your UNL?
**Part 2: Connection Quality Analysis**
**Step 1**: Enable overlay logging
```bash
rippled log_level Overlay debug
```
**Step 2**: Monitor for 5 minutes
```bash
tail -f /var/log/rippled/debug.log | grep -E "latency|score|disconnect"
```
**Step 3**: Identify patterns
Look for:
* Average peer latency
* Connection failures
* Disconnection reasons
* Reconnection attempts
**Part 3: Connectivity Test**
**Step 1**: Manually connect to a peer
```bash
# Connect to XRP Ledger Foundation validator
rippled connect r.ripple.com:51235
```
**Step 2**: Verify connection
```bash
rippled peers | grep "r.ripple.com"
```
**Step 3**: Observe handshake in logs
```
"Overlay": "Connected to r.ripple.com:51235"
"Overlay": "Handshake complete with peer n9KorY8..."
"Overlay": "Added peer n9KorY8... to active peers"
```
**Part 4: Network Health Check**
**Step 1**: Check peer count over time
```bash
# Run every minute for 10 minutes
for i in {1..10}; do
echo "$(date): $(rippled peers | grep -c address) peers"
sleep 60
done
```
**Step 2**: Monitor connection churn
```bash
# Count new connections and disconnections
grep -c "Connected to peer" /var/log/rippled/debug.log
grep -c "Disconnected from peer" /var/log/rippled/debug.log
```
**Step 3**: Assess stability
Calculate:
* Connection churn rate (disconnections per hour)
* Average peer lifetime
* Reconnection frequency
**Part 5: Peer Quality Distribution**
**Step 1**: Extract peer metrics
From `peers` output, record for each peer:
* Latency
* Uptime
* Complete ledgers range
**Step 2**: Create distribution charts
Latency distribution:
```
0-50ms: |||||| (6 peers)
51-100ms: |||||||||| (10 peers)
101-200ms: ||| (3 peers)
201+ms: | (1 peer)
```
**Step 3**: Identify issues
* Are any peers consistently high-latency?
* Do any peers have incomplete ledger history?
* Are there peers with low uptime?
**Analysis Questions**
Answer these based on your observations:
1. **What's your average peer latency?**
* Is it acceptable (<200ms)?
2. **How stable are your connections?**
* High churn may indicate network issues
3. **Are you well-connected to validators?**
* Check against your UNL
4. **What's your network position?**
* Are you mostly receiving or mostly sending connections?
5. **Do you see any problematic peers?**
* High latency, frequent disconnections?
6. **How does your node handle connection limits?**
* Does it maintain target peer count?
***
### Key Takeaways
#### Core Concepts
✅ **Mesh Topology**: Decentralized network with no single point of failure
✅ **Three Connection Types**: Outbound, inbound, and fixed connections serve different purposes
✅ **Multi-Mechanism Discovery**: DNS seeds, configured peers, and gossip protocol enable robust peer discovery
✅ **Connection Quality**: Continuous monitoring and scoring of peer quality
✅ **Intelligent Routing**: Message-specific routing strategies optimize network efficiency
✅ **Squelch Algorithm**: Prevents message loops and duplicate processing
✅ **Priority Queuing**: Ensures critical messages are transmitted first
#### Network Health
✅ **Target Peer Count**: Based on node\_size configuration
✅ **Balanced Connections**: \~50% outbound, \~50% inbound
✅ **Quality Metrics**: Latency, message rate, error rate, uptime
✅ **Connection Pruning**: Low-quality peers replaced with better alternatives
✅ **Fixed Peer Priority**: Critical connections maintained aggressively
#### Development Skills
✅ **Codebase Location**: Overlay implementation in `src/ripple/overlay/`
✅ **Configuration**: Understanding `[ips]`, `[ips_fixed]`, `[port_peer]` sections
✅ **Monitoring**: Using RPC commands and logs to assess network health
✅ **Debugging**: Tracing connection issues and message flow
***
### Common Issues and Solutions
#### Issue 1: Low Peer Count
**Symptoms**: Active peers consistently below target
**Possible Causes**:
* Firewall blocking inbound connections
* ISP blocking port
* Poor peer quality (all disconnect quickly)
**Solutions**:
```bash
# Check firewall
sudo iptables -L | grep 51235
# Verify port is accessible
telnet your-ip 51235
# Check if node is reachable
rippled server_info | grep pubkey_node
```
#### Issue 2: High Latency Peers
**Symptoms**: Average latency >200ms
**Possible Causes**:
* Geographic distance to peers
* Network congestion
* Poor quality peers
**Solutions**:
```bash
# Manually connect to closer peers
rippled connect low-latency-peer.example.com:51235
# Add fixed peers in same region
[ips_fixed]
local-peer-1.example.com 51235
local-peer-2.example.com 51235
```
#### Issue 3: Frequent Disconnections
**Symptoms**: High connection churn rate
**Possible Causes**:
* Network instability
* Protocol incompatibility
* Being saturated by other peers
**Solutions**:
```bash
# Check logs for disconnect reasons
grep "Disconnected" /var/log/rippled/debug.log
# Look for patterns
grep "Disconnected.*reason" /var/log/rippled/debug.log | \
cut -d: -f4 | sort | uniq -c
```
#### Issue 4: No Validator Connections
**Symptoms**: Not connected to any UNL validators
**Possible Causes**:
* Validators are unreachable
* Validators' connection slots full
* Network configuration issues
**Solutions**:
```bash
# Manually connect to validators
rippled connect validator.example.com:51235
# Use fixed connections for validators
[ips_fixed]
validator1.example.com 51235
validator2.example.com 51235
```
***
### Additional Resources
#### Official Documentation
* **XRP Ledger Dev Portal**: [xrpl.org/docs](https://xrpl.org/docs)
* **Peer Protocol**: [xrpl.org/peer-protocol](https://xrpl.org/docs/concepts/networks-and-servers/peer-protocol)
* **Server Configuration**: [xrpl.org/rippled-server-configuration](https://xrpl.org/docs/infrastructure/configuration)
#### Codebase References
* `src/ripple/overlay/` - Overlay network implementation
* `src/ripple/overlay/impl/PeerImp.cpp` - Peer connection handling
* `src/ripple/overlay/impl/OverlayImpl.cpp` - Core overlay logic
#### Related Topics
* [Protocols](/core-dev-bootcamp/module02/protocols-communication-and-interoperability) - Protocol message formats and communication
* [Consensus Engine](/core-dev-bootcamp/module02/consensus-engine-xrp-ledger-consensus-protocol) - How consensus uses overlay network
* [Application Layer](/core-dev-bootcamp/module02/application-layer-central-orchestration-and-coordination) - How overlay integrates with application
***
# Transaction Lifecycle: Complete Transaction Journey
[← Back to Rippled II Overview](/core-dev-bootcamp/module02)
***
### Introduction
Understanding the complete lifecycle of a transaction—from the moment it's created to its final inclusion in a validated ledger—is crucial for developing applications on the XRP Ledger, debugging transaction issues, and optimizing transaction processing. Every transaction follows a well-defined path through multiple validation stages, consensus rounds, and finalization steps.
This deep dive traces the entire journey of a transaction, explaining each phase, the checks performed, the state transitions, and how to monitor and query transaction status at every step. Whether you're building a wallet, an exchange integration, or contributing to the core protocol, mastering the transaction lifecycle is essential.
{% embed url="" %}
***
### Transaction Lifecycle Overview
#### Complete Journey Diagram
```
┌─────────────────────────────────────────────────────────────┐
│ 1. Transaction Creation │
│ (Client creates and signs transaction) │
└──────────────────────────┬──────────────────────────────────┘
│
↓
┌─────────────────────────────────────────────────────────────┐
│ 2. Transaction Submission │
│ (Submit via RPC, WebSocket, or peer network) │
└──────────────────────────┬──────────────────────────────────┘
│
↓
┌─────────────────────────────────────────────────────────────┐
│ 3. Initial Validation │
│ (Signature check, format validation, preflight) │
└──────────────────────────┬──────────────────────────────────┘
│
┌────────┴────────┐
│ │
↓ ↓
✓ Valid ✗ Invalid
│ │
│ └──→ Rejected (temMALFORMED, etc.)
↓
┌─────────────────────────────────────────────────────────────┐
│ 4. Preclaim Validation │
│ (Check ledger state, balance, account existence) │
└──────────────────────────┬──────────────────────────────────┘
│
┌────────┴────────┐
│ │
↓ ↓
✓ Valid ✗ Invalid
│ │
│ └──→ Rejected (tecUNFUNDED, etc.)
↓
┌─────────────────────────────────────────────────────────────┐
│ 5. Open Ledger Application │
│ (Tentatively apply to open ledger for preview) │
└──────────────────────────┬──────────────────────────────────┘
│
↓
┌─────────────────────────────────────────────────────────────┐
│ 6. Network Propagation │
│ (Broadcast to peers via tmTRANSACTION) │
└──────────────────────────┬──────────────────────────────────┘
│
↓
┌─────────────────────────────────────────────────────────────┐
│ 7. Consensus Round │
│ (Validators propose and agree on transaction set) │
└──────────────────────────┬──────────────────────────────────┘
│
┌────────┴────────┐
│ │
↓ ↓
Included in Set Not Included
│ │
│ └──→ Deferred to next round
↓
┌─────────────────────────────────────────────────────────────┐
│ 8. Canonical Application (DoApply) │
│ (Apply to ledger in canonical order, final) │
└──────────────────────────┬──────────────────────────────────┘
│
┌────────┴────────┐
│ │
↓ ↓
tesSUCCESS tecFAILURE
│ │
│ └──→ Failed but fee charged
↓
┌─────────────────────────────────────────────────────────────┐
│ 9. Ledger Closure │
│ (Ledger closed with transaction included) │
└──────────────────────────┬──────────────────────────────────┘
│
↓
┌─────────────────────────────────────────────────────────────┐
│ 10. Validation Phase │
│ (Validators sign and broadcast validations) │
└──────────────────────────┬──────────────────────────────────┘
│
↓
┌─────────────────────────────────────────────────────────────┐
│ 11. Fully Validated │
│ (Transaction immutable, part of permanent history) │
└─────────────────────────────────────────────────────────────┘
```
#### Typical Timeline
**Fast Path** (ideal conditions):
```
T+0s: Transaction submitted
T+0.1s: Initial validation complete
T+0.2s: Open ledger application
T+0.3s: Network propagation
T+5s: Consensus round completes
T+5.1s: Canonical application
T+5.2s: Ledger closed
T+7s: Validations collected
T+7s: Transaction fully validated
Total: ~7 seconds from submission to finality
```
**Slow Path** (transaction arrives late in open phase):
```
T+0s: Transaction submitted
T+0.1s: Validation complete
T+25s: Consensus round starts (waiting for ledger close)
T+30s: Consensus completes
T+30.1s: Canonical application
T+30.2s: Ledger closed
T+32s: Transaction fully validated
Total: ~32 seconds (depends on when submitted)
```
***
### Phase 1: Transaction Creation
#### Transaction Structure
Before submission, a transaction must be properly constructed:
```json
{
"TransactionType": "Payment",
"Account": "rN7n7otQDd6FczFgLdlqtyMVrn3HMtthca",
"Destination": "rLNaPoKeeBjZe2qs6x52yVPZpZ8td4dc6w",
"Amount": "1000000",
"Fee": "12",
"Sequence": 42,
"LastLedgerSequence": 75234567,
"SigningPubKey": "03AB40A0490F9B7ED8DF29D246BF2D6269820A0EE7742ACDD457BEA7C7D0931EDB",
"TxnSignature": "30450221008..."
}
```
#### Required Fields
**Universal Fields** (all transaction types):
* `TransactionType` - Type of transaction (Payment, OfferCreate, etc.)
* `Account` - Source account (sender)
* `Fee` - Transaction fee in drops (1 XRP = 1,000,000 drops)
* `Sequence` - Account sequence number (nonce)
* `SigningPubKey` - Public key used for signing
* `TxnSignature` - Cryptographic signature (or multi-signatures)
**Optional but Recommended**:
* `LastLedgerSequence` - Expiration ledger (transaction invalid after this)
* `SourceTag` / `DestinationTag` - Integer tags for routing/identification
* `Memos` - Arbitrary data attached to transaction
#### Transaction Signing
**Single Signature**:
```javascript
// Using xrpl.js
const xrpl = require('xrpl');
// Create transaction
const tx = {
TransactionType: 'Payment',
Account: wallet.address,
Destination: 'rLNaPoKeeBjZe2qs6x52yVPZpZ8td4dc6w',
Amount: '1000000',
Fee: '12'
};
// Auto-fill (adds Sequence, LastLedgerSequence, etc.)
const prepared = await client.autofill(tx);
// Sign transaction
const signed = wallet.sign(prepared);
console.log(signed.tx_blob); // Signed transaction blob
console.log(signed.hash); // Transaction hash
```
**Multi-Signature**:
```javascript
// For accounts with SignerList
const tx = {
TransactionType: 'Payment',
Account: multiSigAccount,
Destination: 'rLNaPoKeeBjZe2qs6x52yVPZpZ8td4dc6w',
Amount: '1000000',
Fee: '12',
SigningPubKey: '', // Empty for multi-sig
Sequence: 42
};
// Each signer signs independently
const signer1Sig = wallet1.sign(tx, true);
const signer2Sig = wallet2.sign(tx, true);
// Combine signatures
const multisigned = xrpl.multisign([signer1Sig, signer2Sig]);
```
#### Transaction Hash
The transaction hash (ID) is calculated from the signed transaction:
```cpp
// Simplified hash calculation
uint256 calculateTransactionID(STTx const& tx)
{
// Serialize the entire signed transaction
Serializer s;
tx.add(s);
// Hash with SHA-512 and take first 256 bits
return s.getSHA512Half();
}
```
**Important**: The hash is deterministic—the same signed transaction always produces the same hash.
***
### Phase 2: Transaction Submission
#### Submission Methods
**Method 1: RPC Submit**
Submit via JSON-RPC:
```bash
curl -X POST https://s1.ripple.com:51234/ \
-H "Content-Type: application/json" \
-d '{
"method": "submit",
"params": [{
"tx_blob": "120000228000000024..."
}]
}'
```
Response:
```json
{
"result": {
"engine_result": "tesSUCCESS",
"engine_result_code": 0,
"engine_result_message": "The transaction was applied.",
"tx_blob": "120000228000000024...",
"tx_json": {
"Account": "rN7n7otQDd6FczFgLdlqtyMVrn3HMtthca",
"Amount": "1000000",
"Destination": "rLNaPoKeeBjZe2qs6x52yVPZpZ8td4dc6w",
"Fee": "12",
"TransactionType": "Payment",
"hash": "E08D6E9754025BA2534A78707605E0601F03ACE063687A0CA1BDDACFCD1698C7"
}
}
}
```
**Method 2: WebSocket Submit**
Real-time submission with streaming updates:
```javascript
const ws = new WebSocket('wss://s1.ripple.com');
ws.on('open', () => {
ws.send(JSON.stringify({
command: 'submit',
tx_blob: '120000228000000024...'
}));
});
ws.on('message', (data) => {
const response = JSON.parse(data);
console.log(response.result.engine_result);
});
```
**Method 3: Peer Network Submission**
Transactions submitted to one node propagate to all nodes:
```
Client → Node A → Overlay Network → All Nodes
```
Even if submitted to a non-validator, the transaction reaches validators through peer-to-peer propagation.
#### Submission Response
Immediate response indicates initial validation result:
**Success Codes**:
* `tesSUCCESS` - Transaction applied to open ledger
* `terQUEUED` - Transaction queued (network busy)
**Temporary Failure** (can retry):
* `terPRE_SEQ` - Sequence too high, earlier tx needed
* `tefPAST_SEQ` - Sequence too low (already used)
**Permanent Failure** (don't retry):
* `temMALFORMED` - Malformed transaction
* `temBAD_FEE` - Invalid fee
* `temBAD_SIGNATURE` - Invalid signature
***
### Phase 3: Initial Validation
#### Preflight Checks
Before accessing ledger state, static validation occurs:
```cpp
NotTEC Transactor::preflight(PreflightContext const& ctx)
{
// 1. Verify signature
if (!checkSignature(ctx.tx))
return temBAD_SIGNATURE;
// 2. Check transaction format
if (!ctx.tx.isFieldPresent(sfAccount))
return temMALFORMED;
// 3. Validate fee
auto const fee = ctx.tx.getFieldAmount(sfFee);
if (fee < ctx.baseFee)
return telINSUF_FEE_P;
// 4. Check sequence
if (ctx.tx.getSequence() == 0)
return temBAD_SEQUENCE;
// 5. Verify amounts are valid
auto const amount = ctx.tx.getFieldAmount(sfAmount);
if (amount <= zero)
return temBAD_AMOUNT;
return tesSUCCESS;
}
```
**Checks Performed**:
* ✓ Cryptographic signature valid
* ✓ Transaction format correct
* ✓ Required fields present
* ✓ Fee sufficient
* ✓ Amounts positive and properly formatted
* ✓ No contradictory fields
**Why Preflight Matters**: Catches obvious errors before expensive ledger state access.
***
### Phase 4: Preclaim Validation
#### Ledger State Checks
Read-only validation against current ledger state:
```cpp
TER Transactor::preclaim(PreclaimContext const& ctx)
{
// 1. Source account must exist
auto const sleAccount = ctx.view.read(
keylet::account(ctx.tx[sfAccount]));
if (!sleAccount)
return terNO_ACCOUNT;
// 2. Check sequence number
auto const txSeq = ctx.tx.getSequence();
auto const acctSeq = (*sleAccount)[sfSequence];
if (txSeq != acctSeq)
return tefPAST_SEQ; // Already used or too high
// 3. Verify sufficient balance
auto const balance = (*sleAccount)[sfBalance];
auto const fee = ctx.tx[sfFee];
auto const amount = ctx.tx[sfAmount];
if (balance < fee + amount)
return tecUNFUNDED_PAYMENT;
// 4. Check destination exists (for Payment)
if (ctx.tx.getTransactionType() == ttPAYMENT)
{
auto const dest = ctx.tx[sfDestination];
auto const sleDest = ctx.view.read(keylet::account(dest));
if (!sleDest)
{
// Can create account if amount >= reserve
if (amount < ctx.view.fees().accountReserve(0))
return tecNO_DST_INSUF_XRP;
}
}
return tesSUCCESS;
}
```
**Checks Performed**:
* ✓ Source account exists
* ✓ Sequence number correct
* ✓ Sufficient balance (including fee)
* ✓ Destination account requirements met
* ✓ Trust lines exist (for issued currencies)
* ✓ Account flags permit operation
***
### Phase 5: Open Ledger Application
#### Tentative Application
Transaction is tentatively applied to provide immediate feedback:
```cpp
std::pair
NetworkOPs::processTransaction(
std::shared_ptr const& transaction)
{
// Apply to open ledger
auto result = app_.openLedger().modify(
[&](OpenView& view, beast::Journal j)
{
return transaction->apply(app_, view, ApplyFlags::tapNONE);
});
if (result.second) // Transaction applied successfully
{
JLOG(j_.trace())
<< "Transaction " << transaction->getID()
<< " applied to open ledger with result: "
<< transToken(result.first);
// Relay to network
app_.overlay().relay(transaction);
return {result.first, true};
}
return {result.first, false};
}
```
**Open Ledger Characteristics**:
* **Not Final**: Open ledger is tentative, changes frequently
* **No Consensus**: Local view only, other nodes may differ
* **Immediate Feedback**: Clients get instant response
* **Can Change**: Transaction may be removed or re-ordered
**Why It Matters**:
* Users get immediate confirmation
* Wallets can show pending transactions
* Applications can provide real-time updates
***
### Phase 6: Network Propagation
#### Transaction Broadcasting
Once applied to open ledger, transaction broadcasts to peers:
```cpp
void Overlay::relay(std::shared_ptr const& tx)
{
// Create protocol message
protocol::TMTransaction msg;
msg.set_rawtransaction(tx->getSerialized());
msg.set_status(protocol::tsNEW);
msg.set_receivetimestamp(
std::chrono::system_clock::now().time_since_epoch().count());
// Wrap in Message object
auto m = std::make_shared(msg, protocol::mtTRANSACTION);
// Broadcast to all peers (except source)
for (auto& peer : getActivePeers())
{
if (peer.get() != source)
peer->send(m);
}
JLOG(j_.trace())
<< "Relayed transaction " << tx->getID()
<< " to " << getActivePeers().size() << " peers";
}
```
**Propagation Speed**:
* Local network: < 100ms
* Global network: 200-500ms
* All nodes receive transaction within 1 second
**Deduplication**:
* Nodes track recently seen transactions
* Duplicate transactions not re-processed
* Prevents network flooding
***
### Phase 7: Consensus Round
#### Transaction Set Building
As ledger close approaches, validators build transaction sets:
```cpp
std::set buildInitialPosition(OpenView const& openLedger)
{
std::set position;
// Include transactions from open ledger
for (auto const& tx : openLedger.transactions())
{
// Only include transactions that:
// 1. Are still valid
// 2. Have sufficient fee
// 3. Haven't expired (LastLedgerSequence)
if (isValidForConsensus(tx))
position.insert(tx.getTransactionID());
}
return position;
}
```
#### Consensus Process
Validators exchange proposals and converge:
**Round 1**: Initial proposals
```
Validator A proposes: {TX1, TX2, TX3, TX4}
Validator B proposes: {TX1, TX2, TX3, TX5}
Validator C proposes: {TX1, TX2, TX4, TX5}
Agreement:
TX1: 100% ✓
TX2: 100% ✓
TX3: 67%
TX4: 67%
TX5: 67%
```
**Round 2**: Converge on high-agreement transactions
```
All validators propose: {TX1, TX2}
Agreement:
TX1: 100% ✓ (included)
TX2: 100% ✓ (included)
TX3, TX4, TX5 deferred to next ledger
```
**Transaction Inclusion Criteria**:
* > 80% of UNL must agree to include
* Transaction must still be valid
* Must not have expired (LastLedgerSequence)
***
### Phase 8: Canonical Application
#### Deterministic Execution
After consensus, transactions are applied in canonical order:
```cpp
void applyTransactionsCanonically(
Ledger& ledger,
std::set const& txSet)
{
// 1. Retrieve transactions from set
std::vector> transactions;
for (auto const& id : txSet)
{
auto tx = fetchTransaction(id);
transactions.push_back(tx);
}
// 2. Sort in canonical order
std::sort(transactions.begin(), transactions.end(),
[](auto const& a, auto const& b)
{
// Sort by account, then sequence
if (a->getAccountID(sfAccount) != b->getAccountID(sfAccount))
return a->getAccountID(sfAccount) < b->getAccountID(sfAccount);
return a->getSequence() < b->getSequence();
});
// 3. Apply each transaction
for (auto const& tx : transactions)
{
auto const result = applyTransaction(ledger, tx);
// Record in metadata
ledger.recordTransaction(tx, result);
JLOG(j_.debug())
<< "Applied " << tx->getTransactionID()
<< " with result " << transToken(result);
}
}
```
**DoApply Execution**:
```cpp
TER Payment::doApply()
{
// 1. Debit source account
auto const result = accountSend(
view(),
account_,
ctx_.tx[sfDestination],
ctx_.tx[sfAmount],
j_);
if (result != tesSUCCESS)
return result;
// 2. Update sequence number
auto const sleAccount = view().peek(keylet::account(account_));
(*sleAccount)[sfSequence] = ctx_.tx.getSequence() + 1;
view().update(sleAccount);
// 3. Record metadata
ctx_.deliver(ctx_.tx[sfAmount]);
return tesSUCCESS;
}
```
**Result Codes**:
* `tesSUCCESS` - Transaction succeeded
* `tecUNFUNDED` - Failed but fee charged
* `tecNO_TARGET` - Failed but fee charged
**Important**: Even failed transactions (`tec` codes) consume the fee and advance the sequence number.
***
### Phase 9: Ledger Closure
#### Closing the Ledger
After all transactions are applied:
```cpp
void closeLedger(
std::shared_ptr& ledger,
NetClock::time_point closeTime)
{
// 1. Set close time
ledger->setCloseTime(closeTime);
// 2. Calculate state hash
auto const stateHash = ledger->stateMap().getHash();
ledger->setAccountHash(stateHash);
// 3. Calculate transaction tree hash
auto const txHash = ledger->txMap().getHash();
ledger->setTxHash(txHash);
// 4. Calculate ledger hash
auto const ledgerHash = ledger->getHash();
JLOG(j_.info())
<< "Closed ledger " << ledger->seq()
<< " hash: " << ledgerHash
<< " txns: " << ledger->txCount();
// 5. Mark as closed
ledger->setClosed();
}
```
**Ledger Hash Calculation**:
```cpp
uint256 Ledger::getHash() const
{
Serializer s;
s.add32(seq_); // Ledger sequence
s.add64(closeTime_.count()); // Close time
s.addBitString(parentHash_); // Parent ledger hash
s.addBitString(txHash_); // Transaction tree hash
s.addBitString(accountHash_); // Account state hash
s.add64(totalCoins_); // Total XRP
s.add64(closeTimeResolution_); // Close time resolution
s.add8(closeFlags_); // Flags
return s.getSHA512Half();
}
```
***
### Phase 10: Validation Phase
#### Creating Validations
Validators sign the closed ledger:
```cpp
std::shared_ptr createValidation(
Ledger const& ledger,
SecretKey const& secretKey)
{
auto validation = std::make_shared(
ledger.getHash(),
ledger.seq(),
NetClock::now(),
publicKeyFromSecretKey(secretKey),
calculateNodeID(publicKeyFromSecretKey(secretKey)),
[&](STValidation& v)
{
v.setFlag(vfFullValidation);
v.sign(secretKey);
});
JLOG(j_.info())
<< "Created validation for ledger " << ledger.seq()
<< " hash: " << ledger.getHash();
return validation;
}
```
#### Broadcasting Validations
```cpp
void broadcastValidation(std::shared_ptr const& val)
{
// Create protocol message
protocol::TMValidation msg;
Serializer s;
val->add(s);
msg.set_validation(s.data(), s.size());
// Broadcast to all peers
auto m = std::make_shared(msg, protocol::mtVALIDATION);
overlay().broadcast(m);
JLOG(j_.trace())
<< "Broadcast validation for ledger " << val->getLedgerSeq();
}
```
#### Collecting Validations
```cpp
bool hasValidationQuorum(
LedgerHash const& hash,
std::set const& validators)
{
auto const& unl = getUNL();
size_t validationCount = 0;
for (auto const& validator : validators)
{
if (unl.contains(validator))
validationCount++;
}
// Need >80% of UNL
return validationCount >= (unl.size() * 4 / 5);
}
```
***
### Phase 11: Fully Validated
#### Finalization
When quorum is reached, ledger becomes fully validated:
```cpp
void markLedgerValidated(std::shared_ptr const& ledger)
{
// 1. Mark as validated
ledger->setValidated();
// 2. Add to validated chain
ledgerMaster_.addValidatedLedger(ledger);
// 3. Update current validated ledger
ledgerMaster_.setValidatedLedger(ledger);
// 4. Publish to subscribers
publishValidatedLedger(ledger);
// 5. Start next open ledger
openLedger_.accept(
ledger,
orderTx,
consensusParms,
{}); // Empty initial transaction set
JLOG(j_.info())
<< "Ledger " << ledger->seq()
<< " validated with " << validationCount(ledger)
<< " validations";
}
```
**Characteristics of Validated Ledger**:
* **Immutable**: Cannot be changed
* **Permanent**: Part of ledger history forever
* **Canonical**: All nodes have identical copy
* **Final**: Transactions cannot be reversed
***
### Transaction Status Querying
#### Methods to Check Transaction Status
**Method 1: tx RPC**
Query by transaction hash:
```bash
rippled tx E08D6E9754025BA2534A78707605E0601F03ACE063687A0CA1BDDACFCD1698C7
```
Response:
```json
{
"result": {
"Account": "rN7n7otQDd6FczFgLdlqtyMVrn3HMtthca",
"Amount": "1000000",
"Destination": "rLNaPoKeeBjZe2qs6x52yVPZpZ8td4dc6w",
"Fee": "12",
"Sequence": 42,
"TransactionType": "Payment",
"hash": "E08D6E9754025BA2534A78707605E0601F03ACE063687A0CA1BDDACFCD1698C7",
"ledger_index": 75234567,
"validated": true,
"meta": {
"TransactionIndex": 5,
"TransactionResult": "tesSUCCESS",
"AffectedNodes": [
// Nodes modified by transaction
]
}
}
}
```
**Key Fields**:
* `validated`: true = in validated ledger, false = pending
* `meta.TransactionResult`: Final result code
* `ledger_index`: Which ledger contains transaction
**Method 2: account\_tx RPC**
Query all transactions for an account:
```bash
rippled account_tx rN7n7otQDd6FczFgLdlqtyMVrn3HMtthca
```
Lists transactions in reverse chronological order.
**Method 3: WebSocket Subscriptions**
Real-time transaction monitoring:
```javascript
ws.send(JSON.stringify({
command: 'subscribe',
accounts: ['rN7n7otQDd6FczFgLdlqtyMVrn3HMtthca']
}));
ws.on('message', (data) => {
const msg = JSON.parse(data);
if (msg.type === 'transaction') {
console.log('Transaction:', msg.transaction);
console.log('Status:', msg.validated ? 'Validated' : 'Pending');
}
});
```
**Subscription Types**:
* `accounts` - Transactions affecting specific accounts
* `transactions` - All transactions network-wide
* `ledger` - Ledger close events
***
### Transaction Metadata
#### Metadata Structure
Metadata records the effects of a transaction:
```json
{
"meta": {
"TransactionIndex": 5,
"TransactionResult": "tesSUCCESS",
"AffectedNodes": [
{
"ModifiedNode": {
"LedgerEntryType": "AccountRoot",
"LedgerIndex": "13F1A95D7AAB7108D5CE7EEAF504B2894B8C674E6D68499076441C4837282BF8",
"PreviousFields": {
"Balance": "10000000",
"Sequence": 42
},
"FinalFields": {
"Balance": "8999988",
"Sequence": 43,
"Account": "rN7n7otQDd6FczFgLdlqtyMVrn3HMtthca"
}
}
},
{
"ModifiedNode": {
"LedgerEntryType": "AccountRoot",
"LedgerIndex": "4F83A2CF7E70F77F79A307E6A472BFC2585B806A70833CCD1C26105BAE0D6E05",
"PreviousFields": {
"Balance": "5000000"
},
"FinalFields": {
"Balance": "6000000",
"Account": "rLNaPoKeeBjZe2qs6x52yVPZpZ8td4dc6w"
}
}
}
],
"delivered_amount": "1000000"
}
}
```
**AffectedNodes Types**:
* `CreatedNode` - New ledger object created
* `ModifiedNode` - Existing object modified
* `DeletedNode` - Object deleted
**Key Metadata Fields**:
* `TransactionIndex` - Position in ledger
* `TransactionResult` - Final result code
* `delivered_amount` - Actual amount delivered (for partial payments)
***
### Transaction Expiration
#### LastLedgerSequence
Transactions can specify an expiration:
```json
{
"TransactionType": "Payment",
"Account": "rN7n7otQDd6FczFgLdlqtyMVrn3HMtthca",
"Destination": "rLNaPoKeeBjZe2qs6x52yVPZpZ8td4dc6w",
"Amount": "1000000",
"LastLedgerSequence": 75234567
}
```
**Behavior**:
* If not included by ledger 75234567, transaction becomes invalid
* Prevents transactions from being stuck indefinitely
* Recommended: Set to current ledger + 4
**Checking Expiration**:
```cpp
bool isExpired(STTx const& tx, LedgerIndex currentLedger)
{
if (!tx.isFieldPresent(sfLastLedgerSequence))
return false; // No expiration set
return tx[sfLastLedgerSequence] < currentLedger;
}
```
***
### Hands-On Exercise
#### Exercise: Track a Transaction Through Its Complete Lifecycle
**Objective**: Submit a transaction and observe it at each phase of the lifecycle.
**Part 1: Prepare and Submit**
**Step 1**: Create and fund test accounts
```bash
# Using testnet
rippled account_info
```
**Step 2**: Prepare transaction with monitoring
```javascript
const xrpl = require('xrpl');
const client = new xrpl.Client('wss://s.altnet.rippletest.net:51233');
await client.connect();
const wallet = xrpl.Wallet.fromSeed('sXXXXXXXXXXXXXXXX');
// Create transaction
const tx = {
TransactionType: 'Payment',
Account: wallet.address,
Destination: 'rLNaPoKeeBjZe2qs6x52yVPZpZ8td4dc6w',
Amount: '1000000',
Fee: '12'
};
// Prepare with LastLedgerSequence
const prepared = await client.autofill(tx);
console.log('Prepared TX:', prepared);
console.log('LastLedgerSequence:', prepared.LastLedgerSequence);
// Sign
const signed = wallet.sign(prepared);
console.log('Transaction Hash:', signed.hash);
```
**Step 3**: Submit and record time
```javascript
const startTime = Date.now();
const result = await client.submit(signed.tx_blob);
console.log('Submission Time:', Date.now() - startTime, 'ms');
console.log('Initial Result:', result.result.engine_result);
console.log('Applied to Open Ledger:', result.result.engine_result === 'tesSUCCESS');
```
**Part 2: Monitor Progress**
**Step 4**: Subscribe to transaction
```javascript
await client.request({
command: 'subscribe',
accounts: [wallet.address]
});
client.on('transaction', (tx) => {
console.log('Transaction Event:');
console.log(' Hash:', tx.transaction.hash);
console.log(' Validated:', tx.validated);
console.log(' Ledger Index:', tx.ledger_index);
console.log(' Time:', Date.now() - startTime, 'ms');
if (tx.validated) {
console.log('✓ Transaction fully validated!');
console.log(' Result:', tx.meta.TransactionResult);
}
});
```
**Step 5**: Poll for status
```javascript
async function checkStatus(hash) {
try {
const tx = await client.request({
command: 'tx',
transaction: hash
});
console.log('Status Check:');
console.log(' Found:', true);
console.log(' Validated:', tx.result.validated);
console.log(' Ledger:', tx.result.ledger_index);
console.log(' Result:', tx.result.meta.TransactionResult);
return tx.result.validated;
} catch (e) {
console.log('Status Check: Not yet in validated ledger');
return false;
}
}
// Poll every second
const pollInterval = setInterval(async () => {
const validated = await checkStatus(signed.hash);
if (validated) {
clearInterval(pollInterval);
console.log('Total time to validation:', Date.now() - startTime, 'ms');
}
}, 1000);
```
**Part 3: Analyze Results**
**Step 6**: Examine metadata
```javascript
const finalTx = await client.request({
command: 'tx',
transaction: signed.hash
});
console.log('\n=== Final Transaction Analysis ===');
console.log('Transaction Hash:', finalTx.result.hash);
console.log('Ledger Index:', finalTx.result.ledger_index);
console.log('Result Code:', finalTx.result.meta.TransactionResult);
console.log('Transaction Index:', finalTx.result.meta.TransactionIndex);
// Analyze affected nodes
console.log('\nAffected Nodes:');
for (const node of finalTx.result.meta.AffectedNodes) {
if (node.ModifiedNode) {
console.log(' Modified:', node.ModifiedNode.LedgerEntryType);
console.log(' Account:', node.ModifiedNode.FinalFields.Account);
if (node.ModifiedNode.PreviousFields.Balance) {
console.log(' Balance Change:',
node.ModifiedNode.FinalFields.Balance -
node.ModifiedNode.PreviousFields.Balance);
}
}
}
// Calculate total time
console.log('\nTiming:');
console.log(' Total time:', Date.now() - startTime, 'ms');
```
**Analysis Questions**
Answer these based on your observations:
1. **How long did each phase take?**
* Submission to initial result: \_\_\_ ms
* Initial result to validated: \_\_\_ ms
* Total time: \_\_\_ ms
2. **What was the initial result?**
* Did it apply to open ledger?
3. **Which ledger included the transaction?**
* Ledger index?
* How many ledgers closed between submission and inclusion?
4. **What was the metadata?**
* Which nodes were affected?
* What balances changed?
5. **Did the transaction expire?**
* Was LastLedgerSequence set?
* How close to expiration was it?
***
### Key Takeaways
#### Core Concepts
✅ **11-Phase Journey**: Transactions go through creation, submission, validation, consensus, application, and finalization
✅ **Multiple Validation Stages**: Preflight (static), Preclaim (state-based), DoApply (execution)
✅ **Open Ledger Preview**: Tentative application provides immediate feedback before consensus
✅ **Consensus Inclusion**: Validators must agree (>80%) to include transaction
✅ **Canonical Order**: Deterministic ordering ensures all nodes reach identical state
✅ **Immutable Finality**: Once validated, transactions cannot be reversed
✅ **Metadata Records Effects**: Complete record of all ledger modifications
#### Timing Expectations
✅ **Fast Path**: \~7 seconds submission to validation
✅ **Slow Path**: \~30 seconds if submitted late in open phase
✅ **Network Propagation**: <1 second to reach all nodes
✅ **Consensus Round**: 3-5 seconds
#### Development Skills
✅ **Transaction Construction**: Proper signing and field selection
✅ **Status Monitoring**: Using tx, account\_tx, and subscriptions
✅ **Error Handling**: Understanding result codes (tem/tef/ter/tec/tes)
✅ **Expiration Management**: Setting LastLedgerSequence appropriately
✅ **Metadata Analysis**: Understanding transaction effects
***
### Common Issues and Solutions
#### Issue 1: Transaction Stuck Pending
**Symptoms**: Transaction not validating after 30+ seconds
**Possible Causes**:
* Insufficient fee (transaction queued)
* Network congestion
* Sequence gap (earlier transaction missing)
**Solutions**:
```bash
# Check transaction status
rippled tx
# Check for sequence gaps
rippled account_info
# Increase fee and resubmit if needed
```
#### Issue 2: tefPAST\_SEQ Error
**Symptoms**: Sequence number already used
**Cause**: Sequence out of sync or transaction already processed
**Solution**:
```javascript
// Always fetch current sequence
const accountInfo = await client.request({
command: 'account_info',
account: wallet.address
});
const currentSeq = accountInfo.result.account_data.Sequence;
```
#### Issue 3: Transaction Not Found
**Symptoms**: tx command returns "txnNotFound"
**Possible Causes**:
* Transaction not yet in validated ledger
* Transaction expired (LastLedgerSequence)
* Transaction rejected during validation
**Solution**:
```javascript
// Wait for validation or check expiration
const ledger = await client.request({command: 'ledger_current'});
if (ledger.result.ledger_current_index > tx.LastLedgerSequence) {
console.log('Transaction expired');
}
```
#### Issue 4: tecUNFUNDED\_PAYMENT
**Symptoms**: Transaction failed with fee charged
**Cause**: Insufficient balance between submission and execution
**Prevention**:
```javascript
// Always check balance including reserves
const reserve = (2 + ownerCount) * baseReserve;
const available = balance - reserve;
if (amount + fee > available) {
throw new Error('Insufficient funds');
}
```
***
### Additional Resources
#### Official Documentation
* **XRP Ledger Dev Portal**: [xrpl.org/docs](https://xrpl.org/docs)
* **Transaction Types**: [xrpl.org/transaction-types](https://xrpl.org/docs/references/protocol/transactions/types/)
* **Transaction Results**: [xrpl.org/transaction-results](https://xrpl.org/docs/references/protocol/transactions/transaction-results)
#### Codebase References
* `src/ripple/app/tx/impl/Transactor.cpp` - Transaction processing
* `src/ripple/app/misc/NetworkOPs.cpp` - Network operations and transaction handling
* `src/ripple/app/ledger/OpenLedger.cpp` - Open ledger management
#### Related Topics
* [Transactors](/core-dev-bootcamp/module02/transactors-transaction-processing-framework) - How transactions are validated and executed
* [Consensus Engine](/core-dev-bootcamp/module02/consensus-engine-xrp-ledger-consensus-protocol) - How transactions are included in consensus
* [Protocols](/core-dev-bootcamp/module02/protocols-communication-and-interoperability) - How transactions are propagated across the network
***
# Codebase Navigation: Efficiently Navigating the Rippled Source
[← Back to Rippled II Overview](/core-dev-bootcamp/module02)
***
### Introduction
The Rippled codebase is large, complex, and can be intimidating for new developers. With hundreds of files, thousands of classes, and millions of lines of code, knowing how to navigate efficiently is crucial for productivity. This guide teaches you the skills to quickly locate functionality, understand code organization, and become proficient in exploring the Rippled source code.
Whether you're tracking down a bug, implementing a new feature, or simply trying to understand how something works, mastering codebase navigation will dramatically accelerate your development workflow and deepen your understanding of the XRP Ledger protocol.
***
### Directory Structure Overview
#### Top-Level Organization
```
rippled/
├── src/ # Source code
│ ├── ripple/ # Main Rippled code
│ └── test/ # Unit and integration tests
├── Builds/ # Build configurations
├── bin/ # Compiled binaries
├── cfg/ # Configuration examples
├── docs/ # Documentation
├── external/ # Third-party dependencies
└── CMakeLists.txt # CMake build configuration
```
#### The src/ripple/ Directory
This is where all the core Rippled code lives:
```
src/ripple/
├── app/ # Application layer (80% of code)
│ ├── consensus/ # Consensus implementation
│ ├── ledger/ # Ledger management
│ ├── main/ # Application initialization
│ ├── misc/ # Network operations, utilities
│ ├── paths/ # Payment path finding
│ ├── rpc/ # RPC command handlers
│ └── tx/ # Transaction implementations
│ └── impl/ # Transactor implementations
├── basics/ # Fundamental utilities
│ ├── base58/ # Base58 encoding
│ ├── contract/ # Assertions and contracts
│ └── log/ # Logging infrastructure
├── beast/ # Boost.Beast networking (vendored)
├── conditions/ # Crypto-conditions (escrow)
├── consensus/ # Generic consensus framework
├── core/ # Core services (Config, JobQueue)
├── crypto/ # Cryptographic functions
├── json/ # JSON handling
├── ledger/ # Ledger data structures
├── net/ # Network utilities
├── nodestore/ # Persistent storage layer
├── overlay/ # Peer-to-peer networking
├── protocol/ # Protocol definitions
│ ├── messages.proto # Protobuf message definitions
│ ├── TxFormats.cpp # Transaction format definitions
│ └── SField.cpp # Serialized field definitions
├── resource/ # Resource management
├── rpc/ # RPC infrastructure
└── shamap/ # SHAMap (Merkle tree) implementation
```
#### Key Directories by Function
**Transaction Processing**:
* `src/ripple/app/tx/impl/` - All transactor implementations
* `src/ripple/protocol/TxFormats.cpp` - Transaction type definitions
**Consensus**:
* `src/ripple/consensus/` - Generic consensus framework
* `src/ripple/app/consensus/` - XRPL-specific consensus
**Networking**:
* `src/ripple/overlay/` - P2P overlay network
* `src/ripple/beast/` - Low-level networking (HTTP, WebSocket)
**Ledger Management**:
* `src/ripple/app/ledger/` - Ledger operations
* `src/ripple/ledger/` - Ledger data structures
* `src/ripple/shamap/` - Merkle tree implementation
**Storage**:
* `src/ripple/nodestore/` - Key-value storage backend
**RPC**:
* `src/ripple/app/rpc/handlers/` - RPC command implementations
* `src/ripple/rpc/` - RPC infrastructure
**Application Core**:
* `src/ripple/app/main/` - Application initialization
* `src/ripple/core/` - Core services (Config, JobQueue)
***
### Naming Conventions
#### Common Prefixes and Abbreviations
Understanding naming conventions is essential for quickly identifying what a class or type represents:
**ST\* Classes (Serialized Types)**
Classes representing serializable protocol objects:
**STTx** - Serialized Transaction
```cpp
// Represents a transaction
class STTx : public STObject
{
TransactionType getTransactionType() const;
AccountID getAccountID(SField const& field) const;
STAmount getFieldAmount(SField const& field) const;
};
```
**STObject** - Serialized Object (base class)
```cpp
// Base for all serialized objects
class STObject
{
void add(Serializer& s) const;
Json::Value getJson(JsonOptions options) const;
};
```
**STAmount** - Serialized Amount
```cpp
// Represents XRP or issued currency amount
class STAmount
{
bool isXRP() const;
Issue const& issue() const;
std::int64_t mantissa() const;
};
```
**STValidation** - Serialized Validation
```cpp
// Validator signature on a ledger
class STValidation
{
uint256 getLedgerHash() const;
std::uint32_t getLedgerSeq() const;
};
```
**STArray** - Serialized Array
```cpp
// Array of STObjects
class STArray : public STBase
{
std::size_t size() const;
STObject const& operator[](std::size_t i) const;
};
```
**SLE - Serialized Ledger Entry**
Represents an object stored in the ledger:
```cpp
// An entry in the ledger state
class SLE
{
LedgerEntryType getType() const;
Keylet const& key() const;
// Field accessors
STAmount const& getFieldAmount(SField const& field) const;
AccountID getAccountID(SField const& field) const;
};
```
**Common SLE Types**:
* Account (AccountRoot)
* Offer
* RippleState (Trust Line)
* SignerList
* PayChannel
* Escrow
* NFToken
**TER - Transaction Engine Result**
Result codes from transaction processing:
```cpp
// Result code enumeration
enum TER : int
{
// Success
tesSUCCESS = 0,
// Malformed (tem)
temMALFORMED = -299,
temBAD_FEE = -298,
temBAD_SIGNATURE = -297,
// Failure (tef)
tefFAILURE = -199,
tefPAST_SEQ = -198,
// Retry (ter)
terRETRY = -99,
terQUEUED = -89,
// Claimed fee (tec)
tecCLAIMED = -100,
tecUNFUNDED_PAYMENT = -101,
tecNO_TARGET = -102,
};
```
**Categories**:
* `tes*` - Success
* `tem*` - Malformed (permanent failure)
* `tef*` - Failure (local, temporary)
* `ter*` - Retry (waiting for condition)
* `tec*` - Claimed fee (failed but fee charged)
**SF\* - Serialized Field**
Field identifiers for serialized data:
```cpp
// Field definitions
extern SField const sfAccount;
extern SField const sfDestination;
extern SField const sfAmount;
extern SField const sfFee;
extern SField const sfSequence;
extern SField const sfSigningPubKey;
extern SField const sfTxnSignature;
```
**Naming Pattern**: `sf` + CamelCase field name
**Other Common Prefixes**
**LedgerEntryType** - Types of ledger objects
```cpp
enum LedgerEntryType
{
ltACCOUNT_ROOT = 'a',
ltOFFER = 'o',
ltRIPPLE_STATE = 'r',
ltESCROW = 'u',
ltPAYCHAN = 'x',
};
```
**Keylet** - Keys for accessing ledger objects
```cpp
// Factory functions for creating keylets
Keylet account(AccountID const& id);
Keylet offer(AccountID const& id, std::uint32_t seq);
Keylet escrow(AccountID const& src, std::uint32_t seq);
```
**RPC**\* - RPC-related classes
```cpp
class RPCHandler;
class RPCContext;
```
***
### Code Patterns and Idioms
#### Pattern 1: Keylet Access
Keylets are the standard way to access ledger objects:
```cpp
// Create keylet for account
AccountID const accountID = ...;
Keylet const k = keylet::account(accountID);
// Read from ledger (immutable)
auto const sle = view.read(k);
if (!sle)
return tecNO_ACCOUNT;
// Access fields
auto const balance = (*sle)[sfBalance];
auto const sequence = (*sle)[sfSequence];
```
**Common Keylet Functions**:
```cpp
// In src/ripple/protocol/Indexes.h
namespace keylet {
Keylet account(AccountID const& id);
Keylet offer(AccountID const& id, std::uint32_t seq);
Keylet line(AccountID const& id1, AccountID const& id2, Currency const& currency);
Keylet escrow(AccountID const& src, std::uint32_t seq);
Keylet payChan(AccountID const& src, AccountID const& dst, std::uint32_t seq);
}
```
#### Pattern 2: View Abstraction
Views provide read/write access to ledger state:
**Read-Only View**:
```cpp
void analyzeAccount(ReadView const& view, AccountID const& id)
{
// Can only read, cannot modify
auto const sle = view.read(keylet::account(id));
// Safe for concurrent access
auto balance = (*sle)[sfBalance];
}
```
**Modifiable View**:
```cpp
TER modifyAccount(ApplyView& view, AccountID const& id)
{
// Can read and modify
auto sle = view.peek(keylet::account(id));
if (!sle)
return tecNO_ACCOUNT;
// Modify
(*sle)[sfBalance] = newBalance;
(*sle)[sfSequence] = (*sle)[sfSequence] + 1;
// Commit changes
view.update(sle);
return tesSUCCESS;
}
```
**View Types**:
* `ReadView` - Read-only access
* `ApplyView` - Read/write for transaction application
* `OpenView` - Open ledger view
* `PaymentSandbox` - Sandboxed view for payments
#### Pattern 3: Field Access with Optional
Many fields are optional, use `~` operator:
```cpp
// Required field (asserts if missing)
auto const account = tx[sfAccount];
// Optional field (returns std::optional)
auto const destTag = tx[~sfDestinationTag];
if (destTag)
useDestinationTag(*destTag);
// Optional with default
auto const flags = tx[~sfFlags].value_or(0);
```
#### Pattern 4: RAII and Smart Pointers
Extensive use of RAII and smart pointers:
```cpp
// Unique ownership
std::unique_ptr ledgerMaster_;
// Shared ownership
std::shared_ptr ledger = getLedger();
// Weak references
std::weak_ptr weakPeer_;
```
#### Pattern 5: Application Reference Pattern
Most components hold an Application reference:
```cpp
class SomeComponent
{
public:
SomeComponent(Application& app)
: app_(app)
, j_(app.journal("SomeComponent"))
{
}
void doWork()
{
// Access other components via app_
auto& ledgerMaster = app_.getLedgerMaster();
auto& overlay = app_.overlay();
}
private:
Application& app_;
beast::Journal j_;
};
```
***
### Finding Functionality
#### Strategy 1: Grep and Search
Command-line searching is often the fastest way:
**Find where a function is defined**:
```bash
# Find definition of a function
grep -r "void processTransaction" src/ripple/
# Find class definition
grep -r "class NetworkOPs" src/ripple/
```
**Find where a variable is used**:
```bash
# Find all uses of a variable
grep -r "ledgerMaster_" src/ripple/app/
# Case-insensitive search
grep -ri "transaction" src/ripple/app/tx/
```
**Find specific transaction type**:
```bash
# Find Payment transactor
grep -r "class Payment" src/ripple/app/tx/impl/
# Find all transactor implementations
ls src/ripple/app/tx/impl/*.cpp
```
**Find RPC command handler**:
```bash
# Find account_info handler
grep -r "doAccountInfo" src/ripple/app/rpc/handlers/
```
#### Strategy 2: Follow the Types
Use type information to navigate:
**Example: Finding where STTx is used**
```bash
# Find STTx usage
grep -r "STTx" src/ripple/ | grep -v ".h:" | head -20
# Find function taking STTx parameter
grep -r "STTx const&" src/ripple/
```
**Example: Finding transaction submission**
```bash
# Find where transactions are submitted
grep -r "submitTransaction" src/ripple/
# Follow to NetworkOPs
cat src/ripple/app/misc/NetworkOPs.h | grep submitTransaction
```
#### Strategy 3: Start from Entry Points
**Entry Points**:
1. **main()** - `src/ripple/app/main/main.cpp`
2. **RPC handlers** - `src/ripple/app/rpc/handlers/*.cpp`
3. **Transaction types** - `src/ripple/app/tx/impl/*.cpp`
4. **Protocol messages** - `src/ripple/overlay/impl/ProtocolMessage.h`
**Example: Tracing RPC Call**
```
1. Client calls "account_info" RPC
2. Find handler: src/ripple/app/rpc/handlers/AccountInfo.cpp
3. Handler function: doAccountInfo()
4. Calls: view.read(keylet::account(accountID))
5. View implementation: src/ripple/ledger/ReadView.h
```
#### Strategy 4: Use IDE Features
Modern IDEs provide powerful navigation:
**Visual Studio Code**:
* `Ctrl/Cmd + Click` - Go to definition
* `F12` - Go to definition
* `Shift + F12` - Find all references
* `Ctrl/Cmd + T` - Go to symbol
* `Ctrl/Cmd + P` - Quick file open
**CLion**:
* `Ctrl + B` - Go to declaration
* `Ctrl + Alt + B` - Go to implementation
* `Alt + F7` - Find usages
* `Ctrl + N` - Go to class
* `Ctrl + Shift + N` - Go to file
**XCode**:
* `Cmd + Click` - Go to definition
* `Ctrl + 1` - Show related items
* `Cmd + Shift + O` - Open quickly
* `Cmd + Shift + F` - Find in project
***
### Understanding File Organization
#### Transaction Files
**Format**: `src/ripple/app/tx/impl/.cpp`
```
Payment.cpp - Payment transactions
CreateOffer.cpp - Offer creation
CancelOffer.cpp - Offer cancellation
SetTrust.cpp - Trust line creation/modification
SetAccount.cpp - Account settings
Escrow.cpp - Escrow operations
PayChan.cpp - Payment channels
SetSignerList.cpp - Multi-signature configuration
```
**Finding Transaction Implementation**:
```bash
# If you know the transaction type
ls src/ripple/app/tx/impl/ | grep -i payment
# List all transaction implementations
ls src/ripple/app/tx/impl/*.cpp
```
#### RPC Handler Files
**Format**: `src/ripple/app/rpc/handlers/.cpp`
```
AccountInfo.cpp - account_info command
AccountLines.cpp - account_lines command
AccountTx.cpp - account_tx command
Tx.cpp - tx command
Submit.cpp - submit command
LedgerCurrent.cpp - ledger_current command
ServerInfo.cpp - server_info command
```
**Finding RPC Handler**:
```bash
# Find specific handler
ls src/ripple/app/rpc/handlers/ | grep -i account
# Find handler function
grep -r "doAccountInfo" src/ripple/app/rpc/handlers/
```
#### Header vs Implementation
**Header Files (.h)**:
* Class declarations
* Function prototypes
* Template definitions
* Inline functions
**Implementation Files (.cpp)**:
* Function implementations
* Static variables
* Template specializations
**Finding Pattern**:
```bash
# Find header
find src/ripple/ -name "NetworkOPs.h"
# Find implementation
find src/ripple/ -name "NetworkOPs.cpp"
```
***
### Reading and Understanding Code
#### Step 1: Start with the Interface
Always read the header file first:
```cpp
// In LedgerMaster.h
class LedgerMaster
{
public:
// Public interface - what can be called
std::shared_ptr getValidatedLedger();
std::shared_ptr getClosedLedger();
void addValidatedLedger(std::shared_ptr const& ledger);
// ...
private:
// Implementation details - how it works
std::shared_ptr mCurrentLedger;
std::shared_ptr mClosedLedger;
// ...
};
```
**What to Look For**:
1. Public methods (API)
2. Constructor parameters (dependencies)
3. Member variables (state)
4. Comments and documentation
#### Step 2: Trace Data Flow
Follow how data flows through functions:
```cpp
// Example: Following a payment
void NetworkOPs::submitTransaction(STTx const& tx)
{
// 1. Initial validation
auto const result = checkTransaction(tx);
if (!isTesSuccess(result))
return;
// 2. Apply to open ledger
app_.openLedger().modify([&](OpenView& view)
{
return Transactor::apply(app_, view, tx); // → Go here
});
// 3. Broadcast
app_.overlay().relay(tx); // → And here
}
```
#### Step 3: Understand Control Flow
Identify key decision points:
```cpp
TER Payment::doApply()
{
// Key decision: XRP or issued currency?
if (isXRP(amount_))
{
// XRP path
return payXRP();
}
else
{
// Issued currency path
return payIssued();
}
}
```
#### Step 4: Read Tests
Tests show how code is meant to be used:
```cpp
// In Payment_test.cpp
void testPayment()
{
// Setup
Env env(*this);
Account alice{"alice"};
Account bob{"bob"};
env.fund(XRP(10000), alice, bob);
// Execute
env(pay(alice, bob, XRP(100)));
// Verify
BEAST_EXPECT(env.balance(alice) == XRP(9900));
BEAST_EXPECT(env.balance(bob) == XRP(10100));
}
```
***
### IDE Setup and Configuration
#### Visual Studio Code Setup
**Extensions**:
* C/C++ (Microsoft)
* C/C++ Extension Pack
* CMake Tools
* GitLens
**Configuration** (`.vscode/settings.json`):
```json
{
"C_Cpp.default.configurationProvider": "ms-vscode.cmake-tools",
"C_Cpp.default.compileCommands": "${workspaceFolder}/build/compile_commands.json",
"files.associations": {
"*.h": "cpp",
"*.cpp": "cpp"
},
"search.exclude": {
"**/build": true,
"**/external": true
}
}
```
#### CLion Setup
**CMake Configuration**:
1. Open rippled directory
2. CLion auto-detects CMakeLists.txt
3. Configure build profiles (Debug, Release)
4. Let CLion index the project
**Tips**:
* Use "Find in Path" (Ctrl+Shift+F) for project-wide search
* Use "Go to Symbol" (Ctrl+Alt+Shift+N) to find classes/functions
* Enable "Compact Middle Packages" in Project view
#### Compile Commands Database
Generate for better IDE support:
```bash
cd rippled
mkdir build && cd build
cmake -DCMAKE_EXPORT_COMPILE_COMMANDS=ON ..
```
This creates `compile_commands.json` that IDEs use for accurate code intelligence.
***
### Documentation and Comments
#### Code Documentation
Rippled uses various documentation styles:
**Doxygen-Style Comments**:
```cpp
/**
* @brief Apply a transaction to a view
*
* @param app Application instance
* @param view Ledger view to apply to
* @param tx Transaction to apply
* @return Pair of result code and success flag
*/
std::pair
applyTransaction(
Application& app,
OpenView& view,
STTx const& tx);
```
**Inline Comments**:
```cpp
// Check if destination requires a tag
if (sleDest->getFlags() & lsfRequireDestTag)
{
if (!ctx.tx.isFieldPresent(sfDestinationTag))
return tecDST_TAG_NEEDED;
}
```
#### In-Source Documentation
**README Files**:
```
src/ripple/README.md
src/ripple/app/README.md
src/ripple/consensus/README.md
```
**Design Documents**:
```
docs/consensus.md
docs/build-unix.md
```
#### External Documentation
**Dev Null Productions Source Code Guide**:
* Comprehensive walkthrough of rippled codebase
* Available online
* Covers architecture and key components
**XRP Ledger Dev Portal**:
*
* Protocol documentation
* API reference
***
### Hands-On Exercise
#### Exercise: Navigate to Find Specific Functionality
**Objective**: Practice navigation skills by locating and understanding specific code.
**Task 1: Find Payment Transactor**
**Goal**: Locate and read the Payment transactor implementation
**Steps**:
1. Navigate to transaction implementations:
```bash
cd rippled/src/ripple/app/tx/impl/
ls *.cpp
```
2. Open Payment.cpp
3. Find the three key methods:
* `Payment::preflight()`
* `Payment::preclaim()`
* `Payment::doApply()`
4. Answer:
* What checks are performed in preflight?
* What ledger state does preclaim verify?
* What does doApply() do?
**Task 2: Trace account\_info RPC**
**Goal**: Understand how account\_info RPC works
**Steps**:
1. Find the handler:
```bash
grep -r "doAccountInfo" src/ripple/app/rpc/handlers/
```
2. Open `AccountInfo.cpp`
3. Trace the execution:
* How is the account parameter extracted?
* How is account data retrieved?
* What information is returned?
4. Follow the keylet usage:
```cpp
auto const sleAccept = ledger->read(keylet::account(accountID));
```
5. Find keylet::account definition:
```bash
grep -r "Keylet account" src/ripple/protocol/
```
**Task 3: Find Consensus Round Start**
**Goal**: Locate where consensus rounds begin
**Steps**:
1. Search for consensus entry point:
```bash
grep -r "startRound" src/ripple/consensus/
```
2. Find RCLConsensus usage:
```bash
grep -r "RCLConsensus" src/ripple/app/consensus/
```
3. Trace from NetworkOPs:
* Find where consensus is triggered
* Locate initial transaction set building
* Find proposal creation
**Task 4: Understand Transaction Propagation**
**Goal**: Follow how transactions propagate through the network
**Steps**:
1. Start at submission:
```bash
grep -r "submitTransaction" src/ripple/app/misc/
```
2. Find open ledger application:
```bash
grep -r "openLedger().modify" src/ripple/
```
3. Find relay function:
```bash
grep -r "void.*relay.*Transaction" src/ripple/overlay/
```
4. Trace message creation:
* How is tmTRANSACTION message created?
* How is it sent to peers?
**Task 5: Explore Ledger Closure**
**Goal**: Understand ledger close process
**Steps**:
1. Find LedgerMaster:
```bash
find src/ripple/ -name "LedgerMaster.h"
```
2. Look for close-related methods:
```bash
grep "close" src/ripple/app/ledger/LedgerMaster.h
```
3. Find doApply function in consensus:
```bash
grep -r "consensus.*doApply" src/ripple/
```
4. Trace the complete sequence:
* Consensus reached
* Transactions applied
* Ledger hash calculated
* Validations created
***
### Quick Reference Cheat Sheet
#### Common File Locations
```
Transaction Types: src/ripple/app/tx/impl/
RPC Handlers: src/ripple/app/rpc/handlers/
Consensus: src/ripple/consensus/ and src/ripple/app/consensus/
Overlay Network: src/ripple/overlay/
Ledger Management: src/ripple/app/ledger/
Application Core: src/ripple/app/main/
Protocol Definitions: src/ripple/protocol/
Tests: src/test/
```
#### Common Naming Patterns
```
ST* classes: Serialized types (STTx, STAmount, STObject)
SLE: Serialized Ledger Entry
TER: Transaction Engine Result codes
SF*: Serialized Field identifiers (sfAccount, sfAmount)
Keylet: Keys for ledger object access
*View: Ledger view abstractions
*Imp: Implementation classes (PeerImp, OverlayImpl)
```
#### Essential Grep Commands
```bash
# Find class definition
grep -r "class ClassName" src/ripple/
# Find function implementation
grep -r "ReturnType functionName(" src/ripple/
# Find where something is used
grep -r "variableName" src/ripple/
# Case-insensitive search
grep -ri "searchterm" src/ripple/
# Search in specific file types
grep -r --include="*.cpp" "searchterm" src/ripple/
# Exclude directories
grep -r --exclude-dir="test" "searchterm" src/ripple/
```
#### IDE Shortcuts
**VS Code**:
```
Go to Definition: F12 or Ctrl+Click
Find References: Shift+F12
Go to Symbol: Ctrl+T
Search in Files: Ctrl+Shift+F
```
**CLion**:
```
Go to Declaration: Ctrl+B
Go to Implementation: Ctrl+Alt+B
Find Usages: Alt+F7
Search Everywhere: Double Shift
```
***
### Key Takeaways
#### Navigation Strategies
✅ **Directory Structure**: Understand the organization (app/, protocol/, consensus/, overlay/)
✅ **Naming Conventions**: Learn prefixes (ST\*, SLE, TER, SF\*)
✅ **Code Patterns**: Master Keylets, Views, Application reference pattern
✅ **Search Tools**: Use grep, IDE features, and type following
✅ **Entry Points**: Start from main(), RPC handlers, or transaction types
#### Efficiency Tips
✅ **Read Headers First**: Understand the interface before implementation
✅ **Follow Data Flow**: Trace how data moves through functions
✅ **Use Tests**: Tests show intended usage
✅ **IDE Setup**: Configure properly for code intelligence
✅ **Documentation**: Read in-source docs and external guides
#### Development Skills
✅ **Quick Location**: Find any function or class in seconds
✅ **Code Understanding**: Comprehend complex code quickly
✅ **Debugging**: Trace execution paths efficiently
✅ **Contributing**: Navigate confidently when making changes
***
### Additional Resources
#### Official Documentation
* **XRP Ledger Dev Portal**: [xrpl.org/docs](https://xrpl.org/docs)
* **Rippled Repository**: [github.com/XRPLF/rippled](https://github.com/XRPLF/rippled)
* **Build Instructions**: [github.com/XRPLF/rippled/BUILD.md](https://github.com/XRPLF/rippled/blob/develop/BUILD.md)
#### Codebase Guides
* **Dev Null Productions Source Code Guide**: Comprehensive rippled walkthrough
* **In-Source Documentation**: `src/ripple/README.md` and `docs/` directory
* **Code Comments**: Doxygen-style documentation throughout
#### Tools
* **Visual Studio Code**: Free, excellent C++ support
* **CLion**: Powerful C++ IDE (commercial)
* **grep/ag/ripgrep**: Command-line search tools
* **ctags/cscope**: Code indexing tools
#### Related Topics
* [Application Layer](/core-dev-bootcamp/module02/application-layer-central-orchestration-and-coordination) - Understanding the overall architecture
* [Transactors](/core-dev-bootcamp/module02/transactors-transaction-processing-framework) - How to read transaction implementations
* [Debugging Tools](/core-dev-bootcamp/module02/debugging-tools-development-and-debugging-techniques) - Tools for exploring code at runtime
***
# Debugging Tools: Development and Debugging Techniques
[← Back to Rippled II Overview](/core-dev-bootcamp/module02)
***
### Introduction
Debugging is an essential skill for any Rippled developer. Whether you're tracking down a subtle transaction processing bug, investigating network connectivity issues, or optimizing performance, having a solid understanding of debugging tools and techniques is critical. The complexity of distributed systems like the XRP Ledger means that effective debugging often requires multiple approaches—from analyzing logs to using debuggers to running controlled experiments.
This deep dive covers the comprehensive toolkit available for debugging Rippled, from basic logging to advanced profiling techniques. You'll learn how to diagnose issues quickly, understand system behavior, and develop confidence in working with the Rippled codebase.
***
### Logging System
#### Overview
Rippled includes a sophisticated logging system that provides detailed visibility into system behavior. Understanding how to configure and use logging effectively is the foundation of debugging Rippled.
#### Log Structure
**Partitions**: Logs are organized by subsystem (partition) **Severity Levels**: Each log entry has a severity level **Timestamps**: All logs include precise timestamps **Context**: Logs include relevant context (account IDs, ledger numbers, etc.)
#### Severity Levels
From most to least verbose:
```
trace - Extremely detailed, every function call
debug - Detailed debugging information
info - General informational messages
warning - Warning conditions
error - Error conditions
fatal - Fatal errors that cause termination
```
**Usage Guidelines**:
* **Production**: Use `warning` or `error` to minimize disk I/O
* **Development**: Use `debug` or `trace` for active debugging
* **Investigation**: Temporarily enable `trace` for specific partitions
#### Log Partitions
Major subsystems have their own partitions:
```
Ledger - Ledger operations
LedgerMaster - Ledger master coordination
Transaction - Transaction processing
Consensus - Consensus rounds
Overlay - P2P networking
Peer - Individual peer connections
Protocol - Protocol message handling
RPC - RPC request handling
JobQueue - Job queue operations
NodeObject - NodeStore operations
Application - Application lifecycle
OrderBookDB - Order book database
PathRequest - Path finding
ValidatorList - Validator list management
Amendments - Amendment processing
```
#### Configuring Logging
**In Configuration File**
Edit `rippled.cfg`:
```ini
[rpc_startup]
{ "command": "log_level", "severity": "warning" }
{ "command": "log_level", "partition": "Transaction", "severity": "trace" }
{ "command": "log_level", "partition": "Consensus", "severity": "debug" }
```
**Via RPC Command**
Dynamic adjustment without restart:
```bash
# Set all partitions to warning
rippled log_level warning
# Set specific partition to trace
rippled log_level Transaction trace
# Set multiple partitions
rippled log_level Consensus debug
rippled log_level Overlay debug
rippled log_level Peer trace
```
**Programmatically**
In code:
```cpp
// Get logger for this partition
beast::Journal j = app_.journal("MyComponent");
// Log at different levels
JLOG(j.trace()) << "Entering function with param: " << param;
JLOG(j.debug()) << "Processing transaction: " << tx.getTransactionID();
JLOG(j.info()) << "Ledger closed: " << ledger.seq();
JLOG(j.warning()) << "Unusual condition detected";
JLOG(j.error()) << "Failed to process: " << error;
JLOG(j.fatal()) << "Critical error, shutting down";
```
#### Log File Location
**Default Locations**:
* **Linux**: `/var/log/rippled/debug.log`
* **macOS**: `~/Library/Application Support/rippled/debug.log`
* **Custom**: Set in `rippled.cfg`:
```ini
[debug_logfile]
/path/to/custom/debug.log
```
#### Log Rotation
Configure log rotation to prevent disk space issues:
```ini
[debug_logfile]
/var/log/rippled/debug.log
# Rotate when file reaches 100MB
# Keep 10 old log files
```
Using system tools (Linux):
```bash
# /etc/logrotate.d/rippled
/var/log/rippled/debug.log {
daily
rotate 7
compress
delaycompress
missingok
notifempty
copytruncate
}
```
#### Reading Log Files
**Tail Live Logs**:
```bash
tail -f /var/log/rippled/debug.log
```
**Filter by Partition**:
```bash
grep "Transaction:" /var/log/rippled/debug.log
```
**Filter by Severity**:
```bash
grep "ERR" /var/log/rippled/debug.log
```
**Timestamp Range**:
```bash
# Logs between specific times
awk '/2025-01-15 10:00/,/2025-01-15 11:00/' /var/log/rippled/debug.log
```
**Common Patterns**:
```bash
# Find transaction processing
grep "Transaction.*tesSUCCESS" /var/log/rippled/debug.log
# Find consensus rounds
grep "Consensus.*Starting round" /var/log/rippled/debug.log
# Find peer connections
grep "Overlay.*Connected to peer" /var/log/rippled/debug.log
# Find errors
grep -E "ERROR|ERR|Fatal" /var/log/rippled/debug.log
```
***
### Standalone Mode
#### What is Standalone Mode?
Standalone mode runs Rippled as a single-node network where you have complete control:
* **No peers**: Runs without connecting to other nodes
* **Manual ledger close**: You trigger ledger closes
* **Deterministic**: No network randomness
* **Fast**: No consensus delays
* **Isolated**: Perfect for testing
#### Starting Standalone Mode
```bash
rippled --standalone --conf=/path/to/rippled.cfg
```
**Configuration for Standalone**:
```ini
[server]
port_rpc_admin_local
port_ws_admin_local
[port_rpc_admin_local]
port = 5005
ip = 127.0.0.1
admin = 127.0.0.1
protocol = http
[port_ws_admin_local]
port = 6006
ip = 127.0.0.1
admin = 127.0.0.1
protocol = ws
# No peer port needed in standalone
[node_db]
type=NuDB
path=/var/lib/rippled/standalone/db
[database_path]
/var/lib/rippled/standalone
```
#### Using Standalone Mode
**Check Status**:
```bash
rippled server_info
```
Look for:
```json
{
"result": {
"info": {
"build_version": "1.9.0",
"complete_ledgers": "1-5",
"peers": 0,
"server_state": "proposing",
"standalone": true
}
}
}
```
**Submit Transaction**:
```bash
rippled submit '{
"TransactionType": "Payment",
"Account": "rN7n7otQDd6FczFgLdlqtyMVrn3HMtthca",
"Destination": "rLNaPoKeeBjZe2qs6x52yVPZpZ8td4dc6w",
"Amount": "1000000",
"Fee": "12",
"Sequence": 1
}'
```
**Manually Close Ledger**:
```bash
rippled ledger_accept
```
This immediately closes the current ledger and advances to the next one.
**Check Transaction**:
```bash
rippled tx
```
#### Standalone Mode Workflow
```bash
# 1. Start standalone
rippled --standalone --conf=standalone.cfg
# 2. Fund accounts (in another terminal)
rippled wallet_propose
# 3. Submit transactions
rippled submit
# 4. Close ledger to include transaction
rippled ledger_accept
# 5. Verify transaction
rippled tx
# 6. Repeat steps 3-5 as needed
```
#### Advantages for Debugging
**Deterministic Behavior**:
* No network randomness
* Repeatable tests
* Predictable timing
**Complete Control**:
* Manual ledger progression
* No unexpected transactions
* Isolated environment
**Fast Iteration**:
* Instant ledger closes
* No waiting for consensus
* Quick test cycles
**Safe Experimentation**:
* Can't affect mainnet
* Easy to reset (delete database)
* Test dangerous operations safely
***
### GDB Debugging
#### Setting Up GDB
**Install GDB**:
```bash
# Linux
sudo apt-get install gdb
# macOS
brew install gdb
```
**Compile with Debug Symbols**:
```bash
cd rippled/build
cmake -DCMAKE_BUILD_TYPE=Debug ..
cmake --build . --target rippled
```
**Launch with GDB**:
```bash
gdb --args ./rippled --conf=/path/to/rippled.cfg --standalone
```
#### Basic GDB Commands
**Starting**:
```gdb
(gdb) run # Start program
(gdb) start # Start and break at main()
```
**Breakpoints**:
```gdb
(gdb) break Payment.cpp:123 # Break at file:line
(gdb) break Payment::doApply # Break at function
(gdb) break Transactor.cpp:apply # Break at method
(gdb) info breakpoints # List all breakpoints
(gdb) delete 1 # Delete breakpoint #1
(gdb) disable 2 # Disable breakpoint #2
```
**Execution Control**:
```gdb
(gdb) continue # Continue execution
(gdb) next # Step over (one line)
(gdb) step # Step into (enter function)
(gdb) finish # Run until current function returns
```
**Inspection**:
```gdb
(gdb) print variable # Print variable value
(gdb) print *pointer # Dereference pointer
(gdb) print object.method() # Call method
(gdb) backtrace # Show call stack
(gdb) frame 3 # Switch to frame #3
(gdb) info locals # Show local variables
```
**Advanced**:
```gdb
(gdb) watch variable # Break when variable changes
(gdb) condition 1 i == 5 # Conditional breakpoint
(gdb) commands 1 # Execute commands at breakpoint
```
#### Debugging Transaction Processing
**Example Session**:
```gdb
# Start GDB with rippled
gdb --args ./rippled --standalone --conf=standalone.cfg
# Set breakpoints
(gdb) break Payment::doApply
(gdb) break Transactor::apply
(gdb) break NetworkOPs::processTransaction
# Run
(gdb) run
# In another terminal, submit transaction
$ rippled submit
# GDB will break at processTransaction
(gdb) backtrace
#0 NetworkOPs::processTransaction
#1 RPCHandler::doCommand
#2 ...
# Step through
(gdb) next
(gdb) next
# Examine transaction
(gdb) print transaction->getTransactionID()
(gdb) print transaction->getFieldAmount(sfAmount)
# Continue to Payment::doApply
(gdb) continue
# Examine state
(gdb) print account_
(gdb) print ctx_.tx[sfDestination]
(gdb) print view().read(keylet::account(account_))
# Step through payment logic
(gdb) step
(gdb) next
# Check result
(gdb) print result
(gdb) continue
```
#### Debugging Consensus
```gdb
# Set breakpoints in consensus
(gdb) break RCLConsensus::startRound
(gdb) break Consensus::propose
(gdb) break Consensus::peerProposal
# Run
(gdb) run
# When consensus starts
(gdb) print prevLedgerHash
(gdb) print transactions.size()
(gdb) backtrace
# Step through proposal creation
(gdb) step
(gdb) print position_
# Continue to peer proposal handling
(gdb) continue
(gdb) print proposal.position()
(gdb) print peerID
```
#### Debugging Crashes
**Core Dumps**:
Enable core dumps:
```bash
ulimit -c unlimited
```
Run program until crash:
```bash
./rippled --standalone --conf=standalone.cfg
# ... crash occurs
```
Analyze core dump:
```bash
gdb ./rippled core
(gdb) backtrace
(gdb) frame 0
(gdb) info locals
```
**Common Crash Patterns**:
```gdb
# Null pointer dereference
(gdb) print pointer
$1 = 0x0
(gdb) backtrace
# Look for where pointer should have been set
# Segmentation fault
(gdb) print array[index]
# Check if index is out of bounds
# Assert failure
(gdb) backtrace
# Look at assertion condition and surrounding code
```
***
### Log Analysis and Interpretation
#### Transaction Logs
**Successful Payment**:
```
2025-01-15 10:23:45.123 Transaction:DBG Transaction E08D6E9754... submitted
2025-01-15 10:23:45.125 Transaction:TRC Preflight check passed
2025-01-15 10:23:45.126 Transaction:TRC Preclaim check passed
2025-01-15 10:23:45.127 Transaction:DBG Applied to open ledger: tesSUCCESS
2025-01-15 10:23:45.128 Overlay:TRC Relaying transaction to 18 peers
2025-01-15 10:23:50.234 Consensus:DBG Transaction included in consensus set
2025-01-15 10:23:50.456 Transaction:INF Applied to ledger 75234567: tesSUCCESS
```
**Failed Payment**:
```
2025-01-15 10:23:45.123 Transaction:DBG Transaction E08D6E9754... submitted
2025-01-15 10:23:45.125 Transaction:TRC Preflight check passed
2025-01-15 10:23:45.126 Transaction:WRN Preclaim check failed: tecUNFUNDED
2025-01-15 10:23:45.127 Transaction:DBG Rejected: insufficient funds
```
#### Consensus Logs
**Normal Consensus Round**:
```
2025-01-15 10:23:50.000 Consensus:INF Starting consensus round
2025-01-15 10:23:50.001 Consensus:DBG Building initial position: 147 transactions
2025-01-15 10:23:50.010 Consensus:TRC Proposal sent: hash=ABC123...
2025-01-15 10:23:52.123 Consensus:TRC Received proposal from nHU...: 145 txns
2025-01-15 10:23:52.125 Consensus:TRC Received proposal from nHB...: 146 txns
2025-01-15 10:23:52.500 Consensus:DBG Agreement: 145/147 transactions (98%)
2025-01-15 10:23:52.501 Consensus:INF Consensus reached on transaction set
2025-01-15 10:23:52.600 LedgerMaster:INF Ledger 75234567 closed
2025-01-15 10:23:54.000 LedgerMaster:INF Ledger 75234567 validated with 28/35 validations
```
**Disputed Transaction**:
```
2025-01-15 10:23:50.000 Consensus:INF Starting consensus round
2025-01-15 10:23:52.123 Consensus:DBG Transaction TX123 agreement: 65%
2025-01-15 10:23:54.456 Consensus:DBG Transaction TX123 agreement: 75%
2025-01-15 10:23:56.789 Consensus:WRN Transaction TX123 not included: only 75% agreement
2025-01-15 10:23:56.790 Consensus:INF Consensus reached on transaction set (TX123 excluded)
```
#### Network Logs
**Peer Connection**:
```
2025-01-15 10:23:45.123 Overlay:INF Connecting to r.ripple.com:51235
2025-01-15 10:23:45.234 Overlay:DBG TCP connection established
2025-01-15 10:23:45.345 Overlay:TRC TLS handshake complete
2025-01-15 10:23:45.456 Overlay:TRC Protocol handshake: version 2, node nHU...
2025-01-15 10:23:45.567 Overlay:INF Connected to peer nHU... (validator)
2025-01-15 10:23:45.568 Peer:DBG Added to active peers (18/20)
```
**Connection Failure**:
```
2025-01-15 10:23:45.123 Overlay:INF Connecting to bad-peer.example.com:51235
2025-01-15 10:23:50.123 Overlay:WRN Connection timeout
2025-01-15 10:23:50.124 Overlay:DBG Scheduling reconnect in 10 seconds
```
#### Performance Logs
**Slow Ledger Close**:
```
2025-01-15 10:23:50.000 LedgerMaster:INF Closing ledger 75234567
2025-01-15 10:23:55.000 LedgerMaster:WRN Ledger close took 5000ms (expected <2000ms)
2025-01-15 10:23:55.001 Transaction:WRN Applied 500 transactions in 4800ms
2025-01-15 10:23:55.002 OrderBookDB:WRN Order book update took 1200ms
```
***
### Performance Profiling
#### CPU Profiling
**Using perf (Linux)**:
```bash
# Record profile while running
perf record -g ./rippled --standalone --conf=standalone.cfg
# Generate report
perf report
# Generate flamegraph
perf script | stackcollapse-perf.pl | flamegraph.pl > rippled.svg
```
**Using Instruments (macOS)**:
```bash
# Launch with Instruments
instruments -t "Time Profiler" ./rippled --standalone --conf=standalone.cfg
```
**Interpreting Results**:
Look for:
* Hot functions (high CPU usage)
* Unexpected call patterns
* Inefficient algorithms
* Lock contention
#### Memory Profiling
**Using Valgrind**:
```bash
# Memory leak detection
valgrind --leak-check=full ./rippled --standalone --conf=standalone.cfg
# Memory profiler
valgrind --tool=massif ./rippled --standalone --conf=standalone.cfg
ms_print massif.out.12345
```
**Using AddressSanitizer**:
```bash
# Compile with sanitizer
cmake -DCMAKE_BUILD_TYPE=Debug \
-DCMAKE_CXX_FLAGS="-fsanitize=address" ..
cmake --build . --target rippled
# Run (crashes on memory errors)
./rippled --standalone --conf=standalone.cfg
```
#### Network Profiling
**Wireshark**:
```bash
# Capture traffic
sudo tcpdump -i any port 51235 -w rippled.pcap
# Analyze with Wireshark
wireshark rippled.pcap
```
Filter by:
* Protocol messages
* Connection handshakes
* Bandwidth usage
**Measuring Latency**:
```bash
# Ping peer
rippled ping
# Check peer latency
rippled peers | grep latency
```
***
### Testing Strategies
#### Unit Testing
**Run All Tests**:
```bash
./rippled --unittest
```
**Run Specific Test Suite**:
```bash
./rippled --unittest=Payment
./rippled --unittest=Consensus
```
**Run with Verbose Output**:
```bash
./rippled --unittest --unittest-log
```
**Writing Tests**:
```cpp
#include
namespace ripple {
namespace test {
class MyTest_test : public beast::unit_test::suite
{
public:
void testBasicOperation()
{
using namespace jtx;
// Create test environment
Env env(*this);
// Create accounts
Account alice{"alice"};
Account bob{"bob"};
env.fund(XRP(10000), alice, bob);
// Test operation
env(pay(alice, bob, XRP(100)));
env.close();
// Verify result
BEAST_EXPECT(env.balance(bob) == XRP(10100));
}
void run() override
{
testBasicOperation();
}
};
BEAST_DEFINE_TESTSUITE(MyTest, app, ripple);
} // namespace test
} // namespace ripple
```
#### Integration Testing
**Test Network Setup**:
```bash
# Start multiple rippled instances
./rippled --conf=node1.cfg &
./rippled --conf=node2.cfg &
./rippled --conf=node3.cfg &
# Configure them to peer
rippled --conf=node1.cfg connect localhost:51236
rippled --conf=node2.cfg connect localhost:51237
```
**Test Scenarios**:
* Multi-node consensus
* Network partitions
* Peer discovery
* Transaction propagation
***
### Common Debugging Scenarios
#### Scenario 1: Transaction Not Validating
**Symptoms**: Transaction stuck in pending state
**Debug Steps**:
1. **Check transaction status**:
```bash
rippled tx
```
2. **Check for sequence gaps**:
```bash
rippled account_info
# Compare Sequence with expected
```
3. **Check logs for rejection**:
```bash
grep "" /var/log/rippled/debug.log
```
4. **Verify fee is sufficient**:
```bash
rippled server_info | grep "load_factor"
# Fee should be baseFee * loadFactor
```
5. **Check LastLedgerSequence**:
```bash
rippled ledger_current
# Compare with transaction's LastLedgerSequence
```
#### Scenario 2: Consensus Not Progressing
**Symptoms**: Ledger not closing, network stalled
**Debug Steps**:
1. **Check validator connectivity**:
```bash
rippled validators
```
2. **Examine consensus logs**:
```bash
rippled log_level Consensus trace
tail -f /var/log/rippled/debug.log | grep Consensus
```
3. **Check network connectivity**:
```bash
rippled peers
# Verify connected to enough peers
```
4. **Look for disputes**:
```bash
grep "dispute" /var/log/rippled/debug.log
```
#### Scenario 3: High Memory Usage
**Symptoms**: Rippled consuming excessive memory
**Debug Steps**:
1. **Check ledger history**:
```bash
rippled server_info | grep "complete_ledgers"
```
2. **Review configuration**:
```ini
[node_db]
cache_mb=256 # Reduce if too high
```
3. **Profile memory usage**:
```bash
valgrind --tool=massif ./rippled --standalone
ms_print massif.out.12345
```
4. **Check for leaks**:
```bash
valgrind --leak-check=full ./rippled --standalone
```
#### Scenario 4: Slow Ledger Closes
**Symptoms**: Ledgers taking >5 seconds to close
**Debug Steps**:
1. **Enable performance logging**:
```bash
rippled log_level LedgerMaster debug
rippled log_level Transaction debug
```
2. **Check transaction count**:
```bash
grep "Applied.*transactions" /var/log/rippled/debug.log
```
3. **Profile CPU usage**:
```bash
perf record -g ./rippled
perf report
```
4. **Check database performance**:
```bash
# Check NodeStore backend performance
grep "NodeStore" /var/log/rippled/debug.log
```
***
### Hands-On Exercise
#### Exercise: Debug a Transaction Failure
**Objective**: Use debugging tools to diagnose and fix a transaction issue.
**Part 1: Setup**
**Step 1**: Start standalone mode with detailed logging
```bash
# Configure logging
cat > standalone.cfg << EOF
[server]
port_rpc_admin_local
[port_rpc_admin_local]
port = 5005
ip = 127.0.0.1
admin = 127.0.0.1
protocol = http
[node_db]
type=NuDB
path=/tmp/rippled_debug
[database_path]
/tmp/rippled_debug
[rpc_startup]
{ "command": "log_level", "severity": "debug" }
{ "command": "log_level", "partition": "Transaction", "severity": "trace" }
EOF
# Start
rippled --standalone --conf=standalone.cfg
```
**Step 2**: Create test scenario with intentional issue
```javascript
// Create underfunded transaction
const tx = {
TransactionType: 'Payment',
Account: 'rN7n7otQDd6FczFgLdlqtyMVrn3HMtthca',
Destination: 'rLNaPoKeeBjZe2qs6x52yVPZpZ8td4dc6w',
Amount: '999999999999', // More than account has
Fee: '12',
Sequence: 1
};
```
**Part 2: Debugging Process**
**Step 3**: Submit and observe failure
```bash
rippled submit
```
**Step 4**: Examine logs
```bash
tail -100 /var/log/rippled/debug.log | grep Transaction
```
Look for:
```
Transaction:TRC Preflight check: passed
Transaction:TRC Preclaim check: failed - tecUNFUNDED_PAYMENT
Transaction:DBG Rejected transaction: insufficient funds
```
**Step 5**: Verify balance
```bash
rippled account_info rN7n7otQDd6FczFgLdlqtyMVrn3HMtthca
```
**Step 6**: Calculate required amount
```javascript
const balance = accountInfo.account_data.Balance;
const reserve = 20000000; // Base reserve
const available = balance - reserve;
console.log(`Available: ${available} drops`);
console.log(`Requested: 999999999999 drops`);
console.log(`Shortfall: ${999999999999 - available} drops`);
```
**Step 7**: Fix and resubmit
```javascript
// Corrected transaction
const fixedTx = {
...tx,
Amount: String(available - 12) // Account for fee
};
```
**Part 3: Advanced Debugging**
**Step 8**: Debug with GDB
```bash
# Recompile with debug symbols
cd rippled/build
cmake -DCMAKE_BUILD_TYPE=Debug ..
make
# Start with GDB
gdb --args ./rippled --standalone --conf=standalone.cfg
```
**Step 9**: Set breakpoints
```gdb
(gdb) break Payment::preclaim
(gdb) run
```
**Step 10**: Submit transaction (in another terminal)
```bash
rippled submit
```
**Step 11**: Examine state in GDB
```gdb
# Breakpoint hit
(gdb) print ctx.tx[sfAmount]
(gdb) print (*sleAccount)[sfBalance]
(gdb) print fee
(gdb) print balance < (amount + fee)
# Should be true, causing tecUNFUNDED
(gdb) continue
```
**Analysis Questions**
1. **What error code was returned?**
* tecUNFUNDED\_PAYMENT
2. **At which validation phase did it fail?**
* Preclaim (ledger state check)
3. **What was the root cause?**
* Insufficient balance for payment + fee + reserve
4. **How would you prevent this in client code?**
* Check balance before submitting
* Account for reserve requirements
* Include fee in calculation
5. **What logs helped identify the issue?**
* Transaction partition trace logs
* Preclaim failure message
***
### Key Takeaways
#### Core Debugging Skills
✅ **Logging System**: Understand partitions, severity levels, and configuration
✅ **Standalone Mode**: Essential for controlled testing and debugging
✅ **GDB**: Set breakpoints, inspect variables, trace execution
✅ **Log Analysis**: Read and interpret logs to diagnose issues
✅ **Performance Profiling**: Identify bottlenecks and optimize
#### Best Practices
✅ **Start Simple**: Use logs before reaching for debugger
✅ **Reproduce Reliably**: Use standalone mode for consistent reproduction
✅ **Isolate Issues**: Narrow down to specific component
✅ **Read the Code**: Logs point you to code, understand the implementation
✅ **Test Thoroughly**: Write unit tests for bugs you fix
#### Tool Selection
✅ **Logs**: First line of defense, always available
✅ **Standalone Mode**: Controlled environment, fast iteration
✅ **GDB**: Deep inspection, understanding execution flow
✅ **Profiling**: Performance issues, optimization
✅ **Tests**: Regression prevention, continuous validation
***
### Additional Resources
#### Official Documentation
* **XRP Ledger Dev Portal**: [xrpl.org/docs](https://xrpl.org/docs)
* **Rippled Repository**: [github.com/XRPLF/rippled](https://github.com/XRPLF/rippled)
* **Build Instructions**: [github.com/XRPLF/rippled/blob/develop/BUILD.md](https://github.com/XRPLF/rippled/blob/develop/BUILD.md)
#### Debugging Resources
* **GDB Documentation**: [sourceware.org/gdb/documentation](https://sourceware.org/gdb/current/onlinedocs/gdb/)
* **Valgrind Manual**: [valgrind.org/docs/manual](https://valgrind.org/docs/manual/)
* **Linux Perf Wiki**: [perf.wiki.kernel.org](https://perf.wiki.kernel.org/)
#### Codebase References
* `src/ripple/core/impl/JobQueue.cpp` - Job queue debugging
* `src/ripple/app/main/Application.cpp` - Application startup debugging
* `src/test/` - Unit test examples
#### Related Topics
* [Application Layer](/core-dev-bootcamp/module02/application-layer-central-orchestration-and-coordination) - Understanding system architecture
* [Transaction Lifecycle](/core-dev-bootcamp/module02/transaction-lifecycle-complete-transaction-journey) - Understanding transaction flow
* [Codebase Navigation](/core-dev-bootcamp/module02/codebase-navigation-efficiently-navigating-the-rippled-source) - Finding code to debug
***
# Homeworks
# Homework 1: Transaction Flow Analysis
[← Back to Rippled II Overview](/core-dev-bootcamp/module02)
***
### Objective
This homework reinforces your understanding of the Rippled architecture, transaction processing, and codebase navigation. You will perform hands-on exploration of the codebase and analyze transaction
**Format**: Written report (PDF or Markdown) with screenshots/code snippets
***
### Task
Run Rippled in standalone mode and trace a Payment transaction from submission to ledger closure.
#### Requirements
1. **Setup**
* Start Rippled in standalone mode
* Submit a Payment transaction using RPC
* Manually close the ledger
2. **Documentation**
* Document the complete transaction lifecycle with timestamps
* Capture and explain the transaction JSON
* Take a screenshot of the transaction result
* Identify which files in the codebase handle each phase:
* Submission
* Validation (Preflight/Preclaim)
* Application (DoApply)
* Ledger closure
***
### Deliverable
A written report with:
* Commands used
* Transaction details (hash, account, destination, amount)
* Screenshots of the transaction submission and result
* List of relevant source files with brief explanations of their roles
# Homework 2: Transactor Code Exploration
[← Back to Rippled II Overview](/core-dev-bootcamp/module02)
***
### Objective
Explore the Payment transactor implementation in the Rippled codebase.
**Format**: Written report (PDF or Markdown) with screenshots/code snippets
#### Requirements
1. **Locate the Code**
* Find `Payment.cpp` in the codebase
* Identify the class declaration
2. **Analyze the Three Phases**
* Document what checks occur in `preflight()`
* Document what checks occur in `preclaim()`
* Document what state changes occur in `doApply()`
* For each phase, provide:
* Line numbers where the function is defined
* 2-3 specific validation checks performed
* Code snippet (5-10 lines) showing a key validation
#### Deliverable
A technical document with:
* File path and line numbers
* Description of each phase
* Code snippets with explanations
* At least one example of an error condition that would cause the transaction to fail
# Homework 3: Logging and Debugging
[← Back to Rippled II Overview](/core-dev-bootcamp/module02)
***
### Objective
Configure logging and analyze Rippled's internal behavior.
**Format**: Written report (PDF or Markdown) with screenshots/code snippets
#### Requirements
1. **Enable Detailed Logging**
* Enable trace-level logging for the Transaction partition
* Submit a transaction
* Capture the relevant log output
2. **Log Analysis**
* Identify and explain 5 key log messages related to transaction processing
* For each log message, explain:
* What subsystem generated it
* What phase of processing it represents
* What information it provides to developers
#### Deliverable
A report containing:
* Commands used to enable logging
* Log excerpts (formatted properly)
* Analysis of 5 log messages with explanations
# Homework 4: Architecture Diagram
[← Back to Rippled II Overview](/core-dev-bootcamp/module02)
***
## Objective
Deepen your understanding of Rippled by exploring additional transaction types, customizing log filtering for targeted analysis, and visualizing the Transactor framework with a detailed flowchart.
**Format**: Written report (PDF or Markdown) with screenshots/code snippets
#### Deliverable
* Explore an additional transaction type (e.g., OfferCreate, TrustSet)
* Implement a custom log filter for specific transaction types
* Create a detailed flowchart of the Transactor framework
# Anatomy of Rippled: SHAMap and NodeStore
### Module Overview
This Module focuses on the backbone of XRPL's state management: the **SHAMap**, which maintains cryptographically-verified ledger state in memory, and the **NodeStore**, which persists this state to disk.
Every transaction on the XRP Ledger generates a new snapshot of the ledger state. Without proper storage and retrieval, this state would be lost on crash. Even worse, without efficient algorithms for comparison and synchronization, the network could stall while sharing state across thousands of nodes.
SHAMap and NodeStore solve these problems by providing:
* Cryptographically committed state changes
* Persistent historical ledger states
* Fast network synchronization for new nodes
* Microsecond access latency for hot data
***
### Explore the Topics
This module covers the full architecture of SHAMap and NodeStore, from conceptual foundations to production-ready implementations. You will explore each component in detail:
* **SHAMap** – Understand the Merkle-Patricia trie structure, node hierarchy, hashing, traversal, and synchronization. Learn how state changes propagate and how nodes efficiently compare ledger states.
* **NodeStore** – Learn how persistent storage is abstracted, including backend choices, caching strategies, and database lifecycle management.
* **Integration** – See how SHAMap and NodeStore work together to maintain ledger integrity and enable fast node synchronization.
* **Advanced Topics** – Explore cryptographic proofs, state reconstruction guarantees, resource management, and real-world performance optimization.
* **Appendices** – Gain guidance on navigating the codebase, debugging, and configuring NodeStore for production workloads.
Through these topics, you will gain both conceptual understanding and hands-on knowledge of the systems that ensure XRPL’s ledger state remains consistent, verifiable, and performant across thousands of nodes.
***
### What You Will Learn
By completing this module, you will be able to:
* Navigate SHAMap and NodeStore code confidently
* Trace the lifecycle of ledger state from creation to persistent storage
* Explain why Merkle-Patricia tries are optimal for blockchain state
* Understand caching strategies and backend choices for NodeStore
* Optimize synchronization performance and troubleshoot issues
* Implement efficient state queries and cryptographic proofs
* Appreciate the engineering elegance enabling distributed consistency across thousands of nodes
* Apply concepts in real code exploration and practical exercises
***
#### 🌱 Blockchain State Management Challenges
**Understanding the Challenges**
Explore why simple approaches fail for blockchain state management and why XRPL’s design choices are necessary to maintain consistency, performance, and synchronization across thousands of nodes.
**Key Topics**: State snapshots, crash recovery, network synchronization challenges\
**Codebase**: Conceptual overview
[Explore Blockchain State Management Challenges →](/core-dev-bootcamp/module03/state-management-challenges)
***
#### 🌳 Trees, Hashing, and Cryptographic Commitments
**Foundations for State Integrity**
Learn the mathematical and cryptographic foundations behind SHAMap and NodeStore, including trees, hashing, and cryptographic commitments.
**Key Topics**: Merkle trees, cryptographic hashes, commitment schemes\
**Codebase**: Conceptual overview
[Explore Trees, Hashing, and Cryptographic Commitments →](/core-dev-bootcamp/module03/tree-fundamentals)
***
#### 🌲 SHAMap Architecture and Node Hierarchy
**Inner and Leaf Nodes Structure**
Dive into the SHAMap architecture, understanding how inner and leaf nodes are organized, and how the tree structure supports efficient state storage.
**Key Topics**: Node hierarchy, tree structure, leaf vs inner nodes\
**Codebase**: `src/xrpld/app/ledger/`
[Explore SHAMap Architecture and Node Hierarchy →](/core-dev-bootcamp/module03/shamap-architecture)
***
#### 🌲 Navigation, Hashing, and Merkle Properties
**Traversing and Hashing the Tree**
Learn how to traverse SHAMap, compute hashes for each node, and leverage Merkle properties for cryptographic verification.
**Key Topics**: Tree traversal, node hashing, Merkle proofs\
**Codebase**: `src/xrpld/app/ledger/`
[Explore Navigation, Hashing, and Merkle Properties →](/core-dev-bootcamp/module03/shamap-navigation-hashing)
***
#### 🌲 Traversal, Iteration, and Synchronization
**Efficient State Comparison Across Nodes**
Understand how nodes synchronize efficiently using hash comparisons, avoiding large data transfers and enabling fast state alignment.
**Key Topics**: Iteration, traversal, synchronization algorithms\
**Codebase**: `src/xrpld/app/ledger/`
[Explore Traversal, Iteration, and Synchronization →](/core-dev-bootcamp/module03/shamap-synchronization)
***
#### 💾 NodeStore Architecture and Design Principles
**Persistent Storage Layer**
Discover how NodeStore abstracts database complexity while providing persistence and performance. Learn why caching is critical, how the rotating database allows online deletion, and backend differences (RocksDB vs NuDB).
**Key Topics**: Storage abstraction, caching strategies, backend independence\
**Codebase**: `src/xrpld/core/`
[Explore NodeStore Architecture and Design Principles →](/core-dev-bootcamp/module03/nodestore-architecture)
***
#### 💾 Storage Abstraction and Backend Implementations
**Database Flexibility**
Learn how NodeStore separates the logical interface from backend implementation, supporting multiple storage engines while maintaining performance.
**Key Topics**: Backend abstraction, RocksDB, NuDB, pluggable storage\
**Codebase**: `src/xrpld/core/`
[Explore Storage Abstraction and Backend Implementations →](/core-dev-bootcamp/module03/storage-backends)
***
#### 💾 Cache Layer and Performance Optimization
**Critical Role of Caching**
Understand multi-tier caching strategies, why hot data access is crucial, and how to optimize NodeStore performance.
**Key Topics**: Cache layers, cache eviction policies, performance tuning\
**Codebase**: `src/xrpld/core/`
[Explore Cache Layer and Performance Optimization →](/core-dev-bootcamp/module03/cache-layer)
***
#### 💾 Database Operations and Lifecycle Management
**Managing Persistent Storage**
Explore NodeStore lifecycle operations, including initialization, rotation, online deletion, and shutdown, ensuring data integrity.
**Key Topics**: Database lifecycle, initialization, maintenance, shutdown procedures\
**Codebase**: `src/xrpld/core/`
[Explore Database Operations and Lifecycle Management →](/core-dev-bootcamp/module03/database-operations)
***
#### 🔐 Cryptographic Proofs and State Reconstruction
**Ensuring Ledger Integrity**
Learn how cryptographic proofs guarantee unique ledger history, enable state reconstruction, and verify correctness without transferring entire datasets.
**Key Topics**: Merkle proofs, state reconstruction, historical verification\
**Codebase**: `src/xrpld/app/ledger/`
[Explore Cryptographic Proofs and State Reconstruction →](/core-dev-bootcamp/module03/proofs-state-reconstruction)
***
#### ⚡ Resource Management and Performance Characteristics
**Optimizing for Production**
Understand NodeStore and SHAMap resource usage, latency characteristics, and practical optimization strategies in real-world XRPL deployments.
**Key Topics**: Memory management, disk I/O, throughput, latency\
**Codebase**: `src/xrpld/core/`
[Explore Resource Management and Performance Characteristics →](/core-dev-bootcamp/module03/resource-management)
***
### 📚 Appendices
[Explore Codebase Navigation Guide →](/core-dev-bootcamp/module03/appendices/navigation)
[Explore Configuration Reference →](/core-dev-bootcamp/module03/appendices/configuration)
[Explore Debugging and Development Tools →](/core-dev-bootcamp/module03/appendices/debugging)
### Helpful Resources
* Rippled Repository: [github.com/XRPLF/rippled](https://github.com/XRPLF/rippled)
* Core Dev Bootcamp Docs: [docs.xrpl-commons.org/core-dev-bootcamp](https://docs.xrpl-commons.org/core-dev-bootcamp/)
* SHAMap Code: `src/ripple/app/ledger/`
* NodeStore Code: `src/ripple/core/`
## Knowledge Check
**Review and Reinforce Your Understanding**
Before moving on, take a few minutes to review key concepts from this module.\
From SHAMap architecture and node hierarchies to NodeStore caching and cryptographic proofs, this short quiz will help you confirm your understanding of XRPL’s ledger state management.
[Ready? Let’s Quiz →](https://xrpl.at/cdbc-quiz03)
## Questions
If you have any questions about the homework or would like us to review your work, feel free to contact us.
[Submit Feedback →](https://docs.google.com/forms/d/e/1FAIpQLSe7qIbqepUKGVJxmWFHv0rs1zsBAzk8G6y5-bfz0Yd5rqW_5A/viewform)
***
➡️ **Next Module**: Transactors: [Understanding the Lifecycle of a Transaction →](https://github.com/XRPL-Commons/xrpl-trainings/blob/main/core-dev-bootcamp/module04bis/README.md)
# Foundational Concepts
[← Back to SHAMap and NodeStore: Data Persistence and State Management](/core-dev-bootcamp/module03)
***
### Introduction
Imagine you're building a bank's ledger. Every day, thousands of transactions modify account balances. You need to:
1. Store the current state (who owns what)
2. Retrieve state quickly (show me Alice's balance)
3. Detect changes efficiently (what changed since yesterday?)
4. Synchronize across branches (all offices must agree on state)
5. Recover from crashes (no data loss)
A traditional database handles all this. But blockchain adds a critical constraint: **no trusted central authority**. Every node must independently verify that state is correct, and thousands of nodes must reach consensus on a single shared state.
This chapter explores why naive approaches fail and what makes XRPL's SHAMap and NodeStore necessary.
### The Naive Approach: What Fails
Let's consider what happens if you store blockchain state like a simple key-value database:
**State Storage:**
```
accounts = {
"rN7n7otQDd6FczFgLdlqtyMVrn3LNU8B4C": { balance: 100 XRP, ... },
"rLHzPsX6oXkzU2qL12kHCH8G8cnZv1rBJh": { balance: 50 XRP, ... },
"r3kmLJN5D28dHuH8vZvVrDjiV5sNSiUQXD": { balance: 75 XRP, ... },
...
}
```
**Transaction Processing:**
1. Validator receives transaction: send 10 XRP from Alice to Bob
2. Validator checks Alice's balance (100 XRP available)
3. Validator updates Alice's balance to 90 XRP
4. Validator updates Bob's balance to 60 XRP
5. State is now modified
**The Problem: No Verification**
Without a cryptographic commitment to the state, any node can claim any state is correct:
```
Alice broadcasts: "My balance is 1,000,000 XRP"
Bob broadcasts: "My balance is 1,000,000 XRP"
Charlie broadcasts: "Everyone's balance is 0 XRP"
Which is correct? Without a central authority, there's no way to know.
```
**The Problem: Expensive Synchronization**
A new node joining the network needs to learn the current state. With naive storage:
1. New node requests: "Send me all account state"
2. Network sends millions of accounts, gigabytes of data
3. New node has no way to verify this data is correct
4. Process takes hours or days
### The Requirement: Cryptographic Commitment
Blockchain state needs a **cryptographic commitment**: a single value that guarantees:
1. **Authenticity**: The value commits to the actual state (not a forgery)
2. **Completeness**: All accounts are included (not cherry-picked)
3. **Uniqueness**: Only one commitment can represent a given state
This is what cryptographic hashes provide:
```cpp
uint256 stateHash = hashFunction(serializeAllState());
```
Now a validator can broadcast: "The current state root is 0xAB12EF..."
Other nodes can verify this is correct by computing the same hash. If someone tries to cheat with different state, the hash will be different.
**The Trade-off Problem:**
But hashing all state from scratch has a terrible cost:
* Every account lookup requires computing the hash of all million accounts
* Synchronizing with a peer requires hashing millions of accounts multiple times
* A single account change requires rehashing everything
* Performance becomes prohibitive
### The SHAMap Solution: Merkle Trees
XRPL solves the cryptographic commitment problem with a **Merkle tree**:
Instead of hashing all accounts together, organize them in a tree structure:
```
Root (hash of all state)
/ \
Hash(Accounts 0-500k) Hash(Accounts 500k-1M)
/ \ / \
Hash(0-250k) Hash(250-500k) ...
/ \ / \
Hash(0-125k) ...
/ \
Hash(Account0) Hash(Account1)
| |
Account0 Account1
```
**Key Insight:** Each node's hash depends only on its descendants, not the entire tree:
* Change Account0 → rehash 4 nodes (path from leaf to root)
* Not millions of nodes
**Synchronization Benefit:**
When syncing with a peer:
1. Compare root hashes
2. If they match: entire state is identical (no need to compare anything else)
3. If they differ: identify which subtree diverges
4. Only synchronize the different parts
5. Recursive process: compare child hashes, descend into differences
A tree of 1 million accounts becomes synchronizable in a few thousand comparisons instead of millions.
### The Patricia Trie Optimization
A simple binary tree has a problem: **unbalanced growth**. If accounts are added sequentially, the tree becomes a linked list, losing logarithmic properties.
Patricia tries (Radix N tries) solve this:
* Use the account identifier (a 256-bit hash) as a navigation guide
* Each level of the tree represents 4 bits (one hex digit) of the account hash
* This produces a balanced, predictable tree structure
* Tree depth is always \~64 levels (256 bits / 4 bits per level)
**XRPL's Choice:** Patricia trie with **radix 16** (hex digits):
```
Level 0 (root): Evaluate first hex digit of account hash (0-F)
→ Child 0, 1, 2, ... or F
Level 1: Evaluate second hex digit
→ One of 16 children
... and so on
```
This gives a balanced tree with:
* Depth: 64 levels (one per hex digit of 256-bit key)
* Branching: Up to 16 children per node
* Perfect for accounts identified by 160-bit hashes
### NodeStore: From Memory to Disk
Now we have SHAMap: an elegant in-memory data structure for cryptographically-committed state.
**But there's a problem:** When the validator crashes, all in-memory state vanishes.
The next startup:
1. Reads the ledger blockchain from disk
2. Replays every transaction from genesis
3. Reconstructs the current state
4. For mainnet: **this takes weeks**
NodeStore solves this by making SHAMap persistent:
* Every node in the SHAMap is serialized to storage
* Identified by its cryptographic hash
* Retrievable by any peer that needs it
* On startup, state is reconstructed from disk in minutes, not weeks
**The Storage Challenge:**
But persistence introduces new challenges:
1. **Database Size**: A mature XRPL ledger creates millions of nodes. Storage can be terabytes.
2. **Lookup Performance**: Database queries are 1000x slower than memory access
3. **Write Efficiency**: Persisting every state change is I/O intensive
4. **Backend Flexibility**: Different operators need different storage engines (RocksDB, NuDB, SQLite)
NodeStore addresses each:
* **Caching**: Keep hot data in memory, query disk only when needed
* **Abstraction**: Support multiple database backends with identical logic
* **Batch Operations**: Write multiple nodes atomically
* **Online Deletion**: Rotate databases to manage disk space without downtime
### The Complete Picture
**SHAMap's Role:**
* Maintains blockchain state in a Merkle tree structure
* Provides cryptographic commitment through root hash
* Enables efficient synchronization through hash comparison
* Supports proof generation for trustless verification
**NodeStore's Role:**
* Persists SHAMap nodes to durable storage
* Provides on-demand node retrieval
* Implements intelligent caching to minimize I/O
* Abstracts database implementation details
**Together:**
They solve the complete blockchain state management problem:
```
Application → SHAMap (in-memory, verified)
↓
NodeStore (persisted, indexed by hash)
↓
Database (RocksDB, NuDB, etc.)
↓
Disk
```
A validator can:
* Process transactions at microsecond latencies (SHAMap is in-memory)
* Know state is persisted safely (NodeStore writes atomically)
* Sync new peers in minutes (hash-based comparison finds differences)
* Recover from crashes without replaying genesis (state reconstructed from disk)
* Switch database backends without changing application logic
### Why This Matters
Understanding SHAMap and NodeStore is essential because:
1. **Consensus Correctness**: The root hash is what validators vote on. You cannot understand consensus without understanding how that hash is computed.
2. **Synchronization Performance**: Why can a new node catch up to the network in minutes? Because hash-based tree comparison eliminates redundant data transfer.
3. **API Performance**: Why do account lookups return in milliseconds? Because careful caching keeps hot nodes in memory.
4. **Operational Reliability**: Why can validators safely delete old data? Because the rotating database enables online deletion without service interruption.
5. **Scalability Limits**: Why does XRPL have practical limits on transaction volume? Because synchronizing and storing the ever-growing tree hits physical limits of disk I/O and memory.
These aren't just implementation details—they're fundamental to what XRPL is and how it works.
### Looking Ahead
In the next chapter, we'll explore the mathematical foundations: Merkle trees, Patricia tries, and cryptographic hashing. Then we'll dive deep into the SHAMap implementation, followed by the NodeStore persistence layer.
By the end of this module, you'll understand not just *what* SHAMap and NodeStore do, but *why* they're architected the way they are, and how to reason about their correctness, performance, and limitations.
# Trees, Hashing, and Cryptographic Commitments
[← Back to SHAMap and NodeStore: Data Persistence and State Management](/core-dev-bootcamp/module03)
***
### Introduction
Before diving into the implementation of SHAMap and NodeStore, we need to understand the mathematical foundations they're built on. This chapter covers:
* Cryptographic hashing and why it creates mathematical guarantees
* Tree structures and why they matter for efficiency
* How Merkle trees combine both to create verifiable commitments to data
* Patricia tries and why they're optimal for key-value storage
Don't skip this chapter thinking it's just theory. Every design decision in SHAMap flows directly from the properties of these data structures. Understanding the foundations will make the implementation clear.
### Cryptographic Hash Functions
A cryptographic hash function `H` has three critical properties:
**1. Determinism**
```
H(x) always returns the same result
H("hello") = 0x2cf24dba5fb0a30e26e83b2ac5b9e29e1b161e5c1fa7425e73043362938b9824
H("hello") = 0x2cf24dba5fb0a30e26e83b2ac5b9e29e1b161e5c1fa7425e73043362938b9824
```
Same input, always same output. This enables verification: others can compute the same hash and confirm correctness.
**2. Collision Resistance**
Finding two different inputs with the same hash output is computationally infeasible:
```
H(x) = H(y) where x ≠ y → "collision"
For SHA-256: Finding a collision requires ~2^128 operations
Current computers: ~10^18 operations/second
Time required: 10^20 seconds ≈ 3 billion years
Cryptographic security: "practical impossibility"
```
**3. Avalanche Effect**
Tiny changes produce completely different outputs:
```
H("hello") = 0x2cf24dba5fb0a30e26e83b2ac5b9e29e1b161e5c1fa7425e73043362938b9824
H("hallo") = 0xd3751713b7ac3d1ab39b1f7b1859d6e7baeac08b8cdb1c8fefd1f96a3f3c17f8
Change one character → hash completely changes
```
**Implication for XRPL:**
Hash output acts as a "fingerprint" of data:
* Two accounts with identical state have identical hashes
* Even one byte difference produces completely different hashes
* Anyone can verify a claimed hash by recomputing it
* Cannot forge a hash without recomputing it (collision-resistant)
XRPL uses **SHA-512Half**: First 256 bits of SHA-512. This gives:
* 256-bit output (32 bytes, fits in a uint256)
* Security level of \~128 bits (same as AES-128)
* Good performance for cryptographic verification
### Tree Structures
Trees are recursive data structures: a root node with zero or more children, each of which is a tree.
```
Root
/ | \
/ | \
A B C
/ \ | / \
D E F G H
```
**Key Tree Property: Logarithmic Depth**
For a balanced tree with branching factor B and N items:
* Depth = log\_B(N)
* Binary tree (B=2): 1M items → depth 20
* Radix-16 tree (B=16): 1M items → depth 5
* Radix-256 tree (B=256): 1M items → depth 3
**Why Depth Matters:**
* Access time: proportional to depth
* Update time: change leaf → rehash path from leaf to root (depth operations)
* Synchronization: identify differences by comparing hashes (depth comparisons)
Higher branching factor = shallower tree = faster operations.
**The Balance Problem:**
But trees can become unbalanced:
```
Sequential insertion:
Balanced (good): Unbalanced (bad - linked list):
4 1
/ \ \
2 6 2
/| |\ \
1 3 5 7 3
\
4
```
For blockchain state, accounts are inserted in unpredictable order. Without careful tree structure, you get unbalanced trees and logarithmic operations degrade.
### Patricia Tries (Radix Tries)
Patricia tries solve the balance problem by using the **key itself as navigation**.
**Basic Idea:**
For a 256-bit key (like an account address), use the bits to guide traversal:
* Radix-2 (binary): Each bit (0 or 1) determines left or right child
* Radix-4: Each 2-bit pair determines one of 4 children
* Radix-16 (hex): Each 4-bit nibble determines one of 16 children
* Radix-256: Each 8-bit byte determines one of 256 children
**Example: Radix-16 Patricia Trie**
```
Account1 hash: 0x3A7F2E1B...
Account2 hash: 0x7B4C9D3A...
Account1: Navigate using hex digits
Level 0: Digit 0 (3) → go to child 3
Level 1: Digit 1 (A) → go to child A
Level 2: Digit 2 (7) → go to child 7
...
Eventually reaches leaf containing Account1
Account2: Navigate using hex digits
Level 0: Digit 0 (7) → go to child 7
Level 1: Digit 1 (B) → go to child B
...
Eventually reaches leaf containing Account2
```
**Balance Guarantee:**
Since navigation is determined by the key:
* All paths from root to any leaf have the same structure
* Tree depth depends only on key length (256 bits / 4 bits = 64 levels)
* Perfect balance for any set of keys
**Space Trade-off:**
Inner nodes have up to 16 pointers (for radix-16), but most sparse:
* Many branches are empty (no accounts in that range)
* Compressed representation avoids wasting space
* Net result: similar space as binary tree despite higher branching
**XRPL's Choice: Radix-16**
```cpp
// From rippled source: 16 possible children per level
static const int NUM_BRANCHES = 16;
// Each level represents 4 bits (one hex digit) of the key
for (int i = 0; i < keyLengthInBits; i += 4) {
int branch = (key >> (keyLengthInBits - i - 4)) & 0x0F;
// Navigate to child at position 'branch'
}
```
Why radix-16?
* Shallow trees: 256-bit keys → 64 levels (log\_16 of ledger size)
* Manageable fanout: 16 children per node (balanced tree complexity)
* Natural alignment: hex notation matches code
* Proven in Bitcoin and Ethereum: both use similar approaches
### Merkle Trees
A Merkle tree combines tree structure with cryptographic hashing:
**Definition:**
* **Leaf nodes**: Contain data (accounts, transactions)
* **Inner nodes**: Contain hashes of their children
* **Root hash**: Represents the entire tree
**Key Property: Hash Propagation**
When a leaf changes, its hash changes. This affects its parent:
```
Step 1: Account changes
Old hash: H("Alice: 100 XRP")
New hash: H("Alice: 90 XRP")
Step 2: Parent recomputes its hash
Old hash: H(H(Account) || H(Other_Accounts))
New hash: H(H(NewAccount) || H(Other_Accounts))
Step 3: Grandparent recomputes
... and so on up the tree
Root hash changes
```
**Benefit: O(1) Subtree Comparison**
Compare two trees:
```
Tree A: Tree B:
Root: 0xABCD Root: 0xXYZW
0xABCD == 0xXYZW?
YES: Entire trees are identical
NO: At least one leaf differs
```
Without Merkle trees:
```
Compare all N leaves directly: O(N) time
```
With Merkle trees:
```
Compare root hashes: O(1) time
If different, compare child hashes to find divergence: O(log N) comparisons
```
**Benefit: Logarithmic Proofs**
Prove that a specific item is in the tree:
```
Merkle Proof for "Alice: 90 XRP":
Show: H("Alice: 90 XRP") = 0x1234...
H(Sibling1) = 0x5678...
H(Sibling2) = 0x9ABC...
Verifier computes:
Combine with siblings working up to root
Verify final hash matches claimed root
Only need to show O(log N) nodes, not entire tree
```
### XRPL's Merkle-Patricia Trie
XRPL's SHAMap combines both approaches:
**Patricia Trie Structure:**
* Navigation determined by account key (256-bit hash)
* Radix-16 branching (hex digit at each level)
* Perfect balance regardless of account insertion order
**Merkle Properties:**
* Each node contains hash of its content
* Inner node hash computed from children's hashes
* Root hash represents entire ledger state
* Changes propagate up to root
**The Result:**
```
Root Hash (e.g., 0xABCD...)
/ | \
Hash(0x0...) Hash(0x1...) Hash(0x2...)
/ | \
Hash(0x00) Hash(0x01) Hash(0x20)
/ \ / | / \
Data Data Data Data Data Data
(acct) (acct)
```
**Properties:**
* Leaf depth: exactly log\_16(number of accounts) ≈ 5 levels for 1M accounts
* Hash changes: only O(log N) nodes affected by any modification
* Verification: O(1) root hash comparison, O(log N) proof verification
* Synchronization: O(log N) hash comparisons to find differences
### Hashing Algorithms in XRPL
XRPL uses SHA-512Half for all hashing:
```cpp
// From rippled source
uint256 hashFunc(Blob const& data) {
using hasher = SHA512Half; // SHA-512, keep first 256 bits
return hasher()(data);
}
```
**Why SHA-512Half instead of SHA-256?**
* SHA-512 has better performance on 64-bit CPUs
* Taking first 256 bits gives same security as SHA-256
* But faster on modern hardware
* 256-bit output fits uint256 perfectly
**Hash Computation in SHAMap:**
Inner nodes hash their children's hashes:
```cpp
uint256 computeHash(SHAMapInnerNode& node) {
Blob data;
for (int i = 0; i < 16; ++i) {
if (node.hasChild(i)) {
uint256 childHash = node.getChildHash(i);
data.append(childHash); // Concatenate child hashes
}
}
return SHA512Half(data);
}
```
Leaf nodes hash their content with a type prefix:
```cpp
uint256 computeHash(SHAMapLeafNode& leaf, Type type) {
Blob data;
data.push_back(type); // Type prefix prevents hash collision
data.append(leaf.getKey());
data.append(leaf.getData());
return SHA512Half(data);
}
```
**Why Type Prefixes?**
Prevent collisions between different types of data:
```
Account with data: 0xAA || 0xBBBB
Leaf type 1, data 0xAA, item 0xBBBB
Transaction with data: 0xAABB || 0xBB
Leaf type 2, data 0xAABB, item 0xBB
These have same content but different meaning!
With type prefix:
H(1 || 0xAA || 0xBBBB) ≠ H(2 || 0xAABB || 0xBB)
```
### Immutability and Copy-on-Write
Once a Merkle tree root hash is committed to (published in a ledger), that entire tree must be immutable:
* Changing any leaf would change the root hash
* Root hash commitment would become invalid
* Trust in the ledger is broken
**Solution: Snapshots**
For each ledger version, create a snapshot:
```
Current ledger state (in-memory, mutable):
SHAMap with live nodes, modified as transactions arrive
Ledger closes:
Create snapshot of current SHAMap
This becomes immutable historical ledger
All nodes locked (cannot be modified)
Next ledger:
Start new mutable SHAMap (sharing unchanged nodes)
Apply next block of transactions
Close creates new immutable snapshot
```
**Copy-on-Write:**
When a mutable SHAMap modifies a shared node:
```
Historical Ledger 1: Historical Ledger 2:
Node A → Hash(old) Node A → Hash(old)
Transaction modifies Node A:
Can't modify Node A in place (Ledger 1 depends on it)
Create copy: Node A' → Hash(new)
Ledger 2 points to Node A'
Ledger 1 still points to Node A
```
Both ledgers are verified by their root hashes, but they share most of the tree (unchanged nodes).
### Summary
**Key Concepts:**
1. **Cryptographic hashing**: Creates fingerprints of data that are collision-resistant and deterministic
2. **Trees**: Enable hierarchical organization with logarithmic depth and operations
3. **Patricia tries**: Use key structure to guarantee balance regardless of insertion order
4. **Merkle trees**: Combine hashing with tree structure to create verifiable commitments
5. **XRPL's design**: Radix-16 Patricia trie with Merkle hashing creates:
* Perfect balance for any set of accounts
* O(log N) operations for updates and proofs
* O(1) tree comparison through root hash
* Safe snapshots through copy-on-write
**Why This Matters:**
Every algorithm in SHAMap—navigation, hashing, synchronization—follows directly from these properties. Understanding the math explains why the code is structured the way it is.
In the next chapter, we'll see how these principles are implemented in rippled's actual C++ code.
# SHAMap Architecture and Node Hierarchy
[← Back to SHAMap and NodeStore: Data Persistence and State Management](/core-dev-bootcamp/module03)
***
### Introduction
Now that you understand the mathematical foundations of Merkle-Patricia tries, let's explore how XRPL actually implements them in the SHAMap data structure.
The SHAMap is responsible for:
* Maintaining account and transaction state in a cryptographically-verified Merkle tree
* Computing root hashes that represent entire ledger state
* Enabling efficient navigation through key-based lookups
* Supporting snapshots and immutability for historical ledgers
* Providing proof generation for trustless state verification
This chapter covers the architectural overview and node types. Later chapters dive into specific algorithms for navigation, hashing, and synchronization.
{% embed url="" %}
### Core Design Principles
The SHAMap architecture achieves three critical properties:
**1. Cryptographic Integrity**
Every change to data propagates through hashes up to the root:
```
Modify Account0 data:
Account0 hash changes
Parent hash changes (incorporates new Account0 hash)
Grandparent hash changes
... up to root
New root hash is deterministically different from old root hash
```
This ensures that no data can be modified undetected. Changing even one bit in an account's balance changes the root hash.
**2. Efficient Navigation**
The Patricia trie structure uses account identifiers as navigation guides:
```
Account hash: 0x3A7F2E1B4C9D...
Tree navigation:
Level 0: Extract digit 3 → go to child 3
Level 1: Extract digit A → go to child A
Level 2: Extract digit 7 → go to child 7
... follow 4-bit chunks down the tree
```
No binary search needed. The path is directly encoded in the key.
**3. Optimized Synchronization**
Hash-based comparison eliminates unnecessary data transfer:
```
Peer A claims: "Root is 0xABCD"
Peer B claims: "Root is 0xXYZW"
If 0xABCD == 0xXYZW:
Both have identical ledger state
No synchronization needed
If different:
Compare children's hashes
Identify exactly which subtrees differ
Request only those subtrees
```
This allows new nodes to synchronize full ledgers from peers in minutes.
### The SHAMap Instance
A SHAMap instance represents a complete tree of ledger state:
**Core Data:**
```cpp
class SHAMap {
// The root node (always an inner node)
std::shared_ptr mRoot;
// Tree state (Immutable, Mutable, or Synching)
State mState;
// For mutable trees: unique identifier
std::uint32_t mCowID; // Copy-on-write identifier
// For navigating to nodes
std::shared_ptr mFamily;
};
```
**Key Properties:**
* **Root Node**: Always an inner node, never a leaf (even if only one account, still has inner structure)
* **Depth**: Exactly 64 levels (256-bit keys ÷ 4 bits per level)
* **Navigation Determinism**: Any key uniquely determines its path from root to leaf
### Node Hierarchy
The SHAMap consists of three conceptual layers:
**Layer 1: Root**
```
[Inner Node - Root]
(hashes all state)
```
* Always present
* Always an inner node
* Contains the root hash representing entire tree
* Can have up to 16 children (one for each hex digit 0-F)
**Layer 2: Internal Structure**
```
[Inner Node - Root]
/ | \ ... \
[Inner] [Inner] [Inner] ... [Inner]
/ | \ / | \
[Inner][Inner][Inner]
```
* Inner nodes serve as branch points
* Each can have 0-16 children
* Store only hashes of children, not actual data
* No data items in inner nodes
**Layer 3: Leaf Nodes**
```
[Inner Node]
/ | \ ... \
[Leaf][Leaf][Leaf]
| | |
Account Account Account
State State State
```
* Terminal nodes containing actual data items
* Types: Account state, Transaction, Transaction+Metadata
* All leaves in a SHAMap tree are homogeneous (same type)
### Inner Nodes
Inner nodes form the branching structure:
**Structure:**
```cpp
class SHAMapInnerNode {
// Up to 16 child slots (indexed 0-F)
std::array mBranches;
// Cryptographic hash of this node
uint256 mHash;
// Bitset: which children exist
std::uint16_t mChildBits;
// For copy-on-write: which SHAMap owns this node
std::uint32_t mCowID;
// Synchronization optimization: generation marker
std::uint32_t mFullBelow;
};
// Each branch slot contains:
struct Branch {
std::shared_ptr mNode; // nullptr if empty
uint256 mHash; // Hash of child (or empty)
};
```
**Key Characteristics:**
1. **Do not store data items directly** - Only hashing information
2. **Maintain cryptographic commitments** through child hashes
3. **Variable occupancy** - Not all 16 children present
4. **Support both serialization formats** - Compressed and full
**Serialization Formats:**
Inner nodes support two wire formats:
**Compressed Format** (used when most slots are empty):
```
Header: "Inner (compressed)"
Bitmap: Which branches exist
For each existing branch:
Branch index
Child hash
```
Saves space by omitting empty branches.
**Full Format** (used when most slots are occupied):
```
Header: "Inner (full)"
For each of 16 branches:
Child hash (or empty marker)
```
Simpler structure despite larger size.
**Format Selection Algorithm:**
XRPL automatically chooses format based on branch count:
```
Branch count:
0-8: Use compressed (saves ~40 bytes)
9+: Use full (simpler, fewer bytes overall)
```
### Leaf Nodes
Leaf nodes store the actual blockchain data:
**Base Properties:**
```cpp
class SHAMapLeafNode {
// The data contained in this leaf
std::shared_ptr mItem;
// Cryptographic hash
uint256 mHash;
// For copy-on-write
std::uint32_t mCowID;
};
```
**SHAMapItem Structure:**
```cpp
class SHAMapItem {
// 256-bit unique identifier (determines tree position)
uint256 mTag;
// Variable-length data (transaction, account state, etc.)
Blob mData;
// Memory management: intrusive reference counting
};
```
**Leaf Node Specializations:**
Three distinct leaf node types exist, each with unique hashing:
**1. Account State Leaves** (hotACCOUNT\_NODE)
* Store account information
* Include balances, settings, owned objects, trust lines
* Type prefix during hashing prevents collision with other types
* Updated when transactions affect accounts
**2. Transaction Leaves** (hotTRANSACTION\_NODE)
* Store transaction data
* Do not include execution metadata
* Immutable once added to ledger
* Enable verification of transaction history
**3. Transaction+Metadata Leaves** (hotTRANSACTION\_LEDGER\_ENTRY)
* Store transactions with execution metadata
* Include results (success/failure, ledger entries modified)
* Complete information for replaying or auditing
* Support full transaction reconstruction
**Why Multiple Types?**
```cpp
// Type prefix prevents collisions
uint256 hash_account_state = SHA512Half(
PREFIX_ACCOUNT || key || data);
uint256 hash_transaction = SHA512Half(
PREFIX_TRANSACTION || key || data);
// Even with identical (key, data), hashes differ
hash_account_state != hash_transaction
```
Ensures that moving data between leaves would be immediately detected as invalid.
### SHAMapNodeID: Node Identification
Every node in the tree is uniquely identified by its position:
**Components:**
```cpp
class SHAMapNodeID {
// Distance from root (0 = root)
std::uint32_t mDepth;
// Path from root to this node
// Packed as 4-bit nibbles in a uint256
uint256 mNodeID;
};
```
**Path Encoding:**
The path is encoded as 4-bit chunks (nibbles) in a uint256:
```
Node at depth 3, path [3, A, 7]:
mNodeID = 0x3A7000...000
^^^ (significant nibbles)
^^^^^^ (zero padding for remaining levels)
Depth 0 (root): mNodeID = 0x0000...000
Depth 1: mNodeID = 0x3000...000
Depth 2: mNodeID = 0x3A00...000
Depth 3: mNodeID = 0x3A70...000
...
Depth 64: mNodeID = complete (all 64 nibbles filled)
```
**Key Operations:**
**getChildNodeID(branch)** - Compute child position:
```cpp
SHAMapNodeID parent(depth=2, nodeID=0x3A00...);
SHAMapNodeID child = parent.getChildNodeID(7);
// Result: depth=3, nodeID=0x3A70...
```
**selectBranch(nodeID, key)** - Determine which branch to follow:
```cpp
uint256 key = 0x3A7F2E1B4C9D...;
int branch = key.nthNibble(depth); // Extract nth 4-bit chunk
// For depth=2: branch = 7 (extract bits 8-11)
```
This deterministic navigation ensures every key has exactly one path through the tree.
### State Management
SHAMaps exist in different states reflecting their role in the system:
**Immutable State**
```cpp
mState == State::Immutable;
mCowID == 0;
```
* Represents a finalized, historical ledger
* Nodes cannot be modified
* Multiple readers can access simultaneously
* Critical limitation: **cannot be trimmed** (nodes loaded stay in memory)
* Used for consensus history
**Mutable State**
```cpp
mState == State::Modifying;
mCowID != 0; // Unique identifier for this tree
```
* Represents work-in-progress ledger state
* Nodes can be modified through copy-on-write
* Safe mutations without affecting other SHAMap instances
* Single writer (typically), multiple readers possible
* Used for constructing new ledger
**Synching State**
```cpp
mState == State::Synching;
```
* Transitional state during network synchronization
* Allows incremental tree construction
* Transitions to Immutable or Modifying when complete
* Used when receiving missing nodes from peers
**State Transitions:**
```
New Tree
|
v
[Synching] <-- Receiving nodes from peers
|
+--> [Immutable] <-- Finalized ledger
|
+--> [Modifying] <-- New transactions incoming
```
### Copy-on-Write Mechanism
The copy-on-write system enables safe node sharing and snapshots:
**Principle:**
Nodes are shared between SHAMaps using shared pointers. When a mutable SHAMap needs to modify a shared node, it creates a copy:
```cpp
// Check if node is owned by this SHAMap
if (node->getCowID() != mCowID) {
// Node is shared or immutable
// Create a copy
auto newNode = node->clone();
newNode->setCowID(mCowID); // Mark as owned by this tree
return newNode;
} else {
// Node already owned by this tree
// Safe to modify in place
return node;
}
```
**Node Sharing Rules:**
```
cowID == 0: Immutable (shared by all)
cowID == tree.mCowID: Owned by this tree (safe to modify)
cowID != tree.mCowID: Owned by another tree (must copy before modifying)
```
**Benefits:**
1. **Efficient Snapshots**: New SHAMap instances share unmodified subtrees
```
Create snapshot before ledger close:
New immutable SHAMap shares root and all unchanged nodes
No deep copy required
```
2. **Safe Concurrent Access**: Modifications don't affect other instances
```
SHAMap A modifies Account1:
Clones nodes along path to Account1
Leaves rest of tree shared
SHAMap B (different ledger) unaffected
```
3. **Memory Efficiency**: Identical subtrees stored once
```
100 ledgers share 99% of tree structure
Only 1% of data duplicated for different accounts
```
### Integration with NodeStore
SHAMap is designed for in-memory operation, but nodes can be persisted:
**Persistent Nodes:**
Some nodes are marked as "backed" - they exist in NodeStore:
```cpp
// Node came from persistent storage
mNode->setBacked();
// When synchronizing, try to retrieve from storage
std::shared_ptr node = nodestore.fetch(hash);
if (node) {
// Add to tree
canonicalizeNode(node); // Ensure uniqueness
}
```
**Canonicalization:**
Ensures nodes are unique in memory:
```cpp
// Check cache for existing node with same hash
std::shared_ptr cached = cache.get(hash);
if (cached) {
return cached; // Use existing node
} else {
cache.insert(hash, newNode); // Cache new node
return newNode;
}
```
This enables:
* Safe node sharing across multiple SHAMap instances
* Fast equality checking (compare pointers, not content)
* Memory efficiency (identical nodes stored once)
### Summary
**Key Architectural Elements:**
1. **SHAMap Instance**: Complete tree representing one ledger version
2. **Inner Nodes**: Branching structure with 0-16 children, storing hashes
3. **Leaf Nodes**: Terminal nodes containing account or transaction data
4. **SHAMapNodeID**: Identifies node position through depth and path
5. **State Management**: Immutable (historical), Mutable (work-in-progress), Synching (incomplete)
6. **Copy-on-Write**: Enables snapshots and safe sharing
7. **NodeStore Integration**: Persistent layer for large datasets
**Design Properties:**
* Perfect balance: All leaves at approximately same depth (log\_16 of account count)
* Deterministic navigation: Key uniquely determines path
* Immutable persistence: Historical ledgers safe from modification
* Efficient sharing: Snapshots require minimal additional memory
* Scalable: Handles millions of accounts with logarithmic operations
In the next chapter, we'll explore how nodes are navigated, hashes are computed, and the tree structure is maintained.
# Navigation, Hashing, and Merkle Properties
[← Back to SHAMap and NodeStore: Data Persistence and State Management](/core-dev-bootcamp/module03)
***
### Introduction
Now that you understand SHAMap's architecture and node types, let's explore how navigation actually works and how hashes propagate through the tree.
These operations are at the heart of what makes SHAMap efficient:
* **Navigation**: Finding the correct path to any account
* **Hashing**: Computing cryptographic commitments at each level
* **Verification**: Ensuring data integrity through hash chains
### Navigation Algorithm
Finding a leaf in a SHAMap is straightforward because the account's key determines the exact path:
**Algorithm: findLeaf**
```cpp
std::shared_ptr findLeaf(uint256 key) {
std::shared_ptr node = mRoot;
for (int depth = 0; depth < 64; ++depth) {
// Is this a leaf?
if (auto leaf = std::dynamic_pointer_cast(node)) {
return leaf;
}
// Must be inner node
auto inner = std::dynamic_pointer_cast(node);
// Extract 4-bit chunk (nibble) at position 'depth'
int branch = key.nthNibble(depth);
// Get child node at that branch
node = inner->getChild(branch);
if (!node) {
// Child doesn't exist - key not in tree
return nullptr;
}
}
return nullptr; // Key not found
}
```
**Step-by-Step Example:**
```
Key: 0x3A7F2E1B4C9D... (account hash)
Depth 0: Root
Extract nibble 0 (first 4 bits): 3
Navigate to child 3
Depth 1:
Extract nibble 1 (next 4 bits): A
Navigate to child A in the child-3 subtree
Depth 2:
Extract nibble 2: 7
Navigate to child 7
... continue until reaching leaf
```
**Time Complexity:**
* Worst case: O(64) inner node traversals (fixed depth)
* Each traversal: O(1) array access to branch pointer
* Total: O(1) expected time (with high probability leaf found before depth 64)
**Space Requirement:**
* Path from root to leaf: \~64 pointers maximum
* Working memory: O(1) per operation
### Hash Computation
Hashing is fundamental to SHAMap's integrity guarantees:
**Leaf Node Hashing**
Leaves compute their hash from their data with a type prefix:
```cpp
uint256 SHAMapLeafNode::computeHash() {
Blob data;
// Type prefix (1 byte) - prevents collisions
data.push_back(mLeafType); // ACCOUNT, TX, or TX_WITH_META
// Account key (32 bytes)
data.append(mItem->getTag());
// Account data (variable)
data.append(mItem->getData());
// Hash the complete structure
return SHA512Half(data);
}
```
**Example:**
```
Account0 data: mTag=0x123ABC..., mData=<100 XRP, flags>
Type prefix: ACCOUNT_LEAF (1 byte)
Hash input: [0x01][123ABC...][100 XRP, flags]
Hash output: 0x47FA... (256 bits)
Changed: mData to "99 XRP"
Hash input: [0x01][123ABC...][99 XRP, flags]
Hash output: 0xB8EF... (completely different)
```
**Inner Node Hashing**
Inner nodes compute their hash from their children's hashes:
```cpp
uint256 SHAMapInnerNode::computeHash() {
Blob data;
// For each of 16 possible children
for (int i = 0; i < 16; ++i) {
if (hasChild(i)) {
// Get child's hash (whether child exists in memory or disk)
uint256 childHash = getChildHash(i);
data.append(childHash); // Append 32 bytes
}
}
// Hash all non-empty child hashes
return SHA512Half(data);
}
```
**Example:**
```
Inner node has children 0, 3, 7, 15:
mBranches[0] → child hash: 0xAA11...
mBranches[3] → child hash: 0xBB22...
mBranches[7] → child hash: 0xCC33...
mBranches[15] → child hash: 0xDD44...
Compute hash:
data = [0xAA11...][0xBB22...][0xCC33...][0xDD44...]
hash = SHA512Half(data)
result: 0x5678...
```
**Hash Update Process**
When tree is modified, hashes must be recomputed bottom-up:
```
Scenario: Account0 balance changes
Step 1: Leaf changes
Old: Account0 leaf hash = 0xOLD1
New: Account0 leaf hash = 0xNEW1
Step 2: Parent recomputes
Parent stored hash references to all 16 children
One changed: [0xAAA][0xNEW1][0xBBB]...
Old parent hash: 0xOLD2
New parent hash: 0xNEW2
Step 3: Grandparent recomputes
Parent hash changed: 0xOLD2 → 0xNEW2
Grandparent's child references change: [...][0xNEW2][...]
Old grandparent hash: 0xOLD3
New grandparent hash: 0xNEW3
... propagates up to root
```
**Critical Property: All Hashes Change**
```
Change 1 account → 1 leaf hash changes
→ parent hash changes
→ grandparent hash changes
→ ... all ancestors up to root change
Root hash changes with certainty
```
This is why root hash is used as the ledger's cryptographic commitment. Change anything in the ledger → root hash changes.
### Merkle Tree Properties
**Property 1: O(1) Subtree Comparison**
Two complete SHAMaps can be compared by comparing single 256-bit values:
```cpp
bool sameLedgerState = (treeA.getRootHash() == treeB.getRootHash());
if (sameLedgerState) {
// Entire trees identical
// Verified cryptographically
} else {
// At least one difference
// Must investigate child hashes to find it
}
```
**Implication:**
Peers can quickly determine if their ledgers match:
```
Peer A: "My ledger root is 0xABCD..."
Peer B: "My ledger root is 0xABCD..."
Comparison: 1 operation
Result: "Ledgers identical" (proven cryptographically)
No need to compare millions of accounts
```
**Property 2: Efficient Difference Detection**
If root hashes differ, child hashes pinpoint the differences:
```
Root hash differs: 0xABCD != 0xXYZW
Compare children:
Child 0: 0xAA == 0xAA (same)
Child 1: 0xBB == 0xBB (same)
Child 2: 0xCC != 0xXX (different!)
Child 3: 0xDD == 0xDD (same)
...
Recursively descend into Child 2 and its siblings
Eventually reach specific accounts that differ
```
**Search Complexity:**
```
Complete recount: O(N) comparisons (N = number of accounts)
Merkle tree method: O(log N) comparisons (log_16 of N)
Example: 1M accounts = 64 bits / 4 bits per level
Binary search depth = 20
Radix-16 Merkle depth = 5
Merkle method: 5 hash comparisons to find differences
vs. 1,000,000 account comparisons
```
**Property 3: Cryptographic Proofs**
Prove that a specific item is in the tree:
```cpp
// Prove Account A exists in tree with root hash R
MerkleProof proof; // Vector of serialized nodes
std::optional> getProofPath(uint256 key) {
std::vector path;
std::shared_ptr node = mRoot;
for (int depth = 0; depth < 64; ++depth) {
// Serialize current node
Blob serialized = node->serialize();
path.push_back(serialized);
if (auto leaf = std::dynamic_pointer_cast(node)) {
// Reached target leaf
return path;
}
// Descend to next node following key
auto inner = std::dynamic_pointer_cast(node);
int branch = key.nthNibble(depth);
node = inner->getChild(branch);
if (!node) {
return std::nullopt; // Key not found
}
}
return std::nullopt;
}
```
**Proof Verification:**
```cpp
bool verifyProofPath(
uint256 key,
uint256 expectedRootHash,
std::vector proof)
{
// Start from leaf end of proof
auto leafNode = deserialize(proof.back());
// Verify leaf contains the key
if (leafNode->getKey() != key) {
return false;
}
uint256 computedHash = leafNode->computeHash();
// Move from leaf toward root
for (int i = proof.size() - 2; i >= 0; --i) {
auto innerNode = deserialize(proof[i]);
// Verify the branch we came from
int depth = i; // Depth in tree
int branch = key.nthNibble(depth);
// Child hash must match computed hash
if (innerNode->getChildHash(branch) != computedHash) {
return false;
}
// Compute this node's hash
computedHash = innerNode->computeHash();
}
// Verify final hash matches expected root
return (computedHash == expectedRootHash);
}
```
**Proof Size:**
```
Tree with 1M accounts:
Depth: log_16(1M) ≈ 5
Proof includes:
1 leaf node: ~50-100 bytes
5 inner nodes: ~100 bytes each
Total: ~600 bytes
Compare to:
Sending all accounts: millions of bytes
Merkle proof: <1 KB
Verification requires hashing ~5 nodes
vs. hashing millions of accounts
```
**Use Cases:**
1. **Light Clients**: Verify account state without full ledger
2. **Cross-Chain Bridges**: Prove XRPL state to other chains
3. **Auditing**: Prove specific transactions were executed
### Mutable Tree Operations
When constructing a new ledger, trees must support modifications:
**Adding a New Leaf**
```cpp
void addLeaf(uint256 key, SHAMapItem item) {
auto leaf = std::make_shared(key, item);
leaf->setCowID(mCowID); // Mark as owned by this tree
std::shared_ptr node = mRoot;
// Navigate to position
for (int depth = 0; depth < 64; ++depth) {
if (auto inner = std::dynamic_pointer_cast(node)) {
int branch = key.nthNibble(depth);
auto child = inner->getChild(branch);
if (!child) {
// Empty slot - insert leaf here
inner = unshareNode(inner); // Copy-on-write
inner->setChild(branch, leaf);
updateHashes(inner); // Recompute hashes up to root
return;
} else if (auto childLeaf =
std::dynamic_pointer_cast(child)) {
// Slot occupied by another leaf
// Need to split into inner node
// ... (complex branch splitting logic)
}
node = child;
}
}
}
```
**Hash Updates**
After modification, hashes propagate up:
```cpp
void updateHashes(std::shared_ptr node) {
uint256 oldHash = node->getHash();
uint256 newHash = node->computeHash();
if (oldHash == newHash) {
return; // Nothing changed
}
// Find parent
auto parent = node->getParent();
if (parent) {
// Update parent's reference to this node
int branch = node->getNodeID().getNibbleAtDepth(
node->getDepth() - 1);
parent->setChildHash(branch, newHash);
// Recursively update parent
updateHashes(parent);
} else {
// This is root - update root hash
mRootHash = newHash;
}
}
```
### Summary
**Navigation:**
* Keys determine paths through tree
* O(1) expected lookup time
* Fixed depth ensures bounded operations
**Hashing:**
* Leaves hash their data with type prefixes
* Inner nodes hash their children's hashes
* Changes propagate bottom-up to root
**Merkle Properties:**
* O(1) tree comparison through root hash
* O(log N) difference detection
* O(log N) proof generation and verification
**Performance:**
* Single account lookup: microseconds (tree traversal)
* Root hash computation: milliseconds (hash all changed nodes)
* Proof verification: thousands of hashes (proof depth)
These properties make SHAMap practical for blockchain state management: efficient updates, verifiable state, and fast synchronization.
# Traversal, Iteration, and Synchronization
[← Back to SHAMap and NodeStore: Data Persistence and State Management](/core-dev-bootcamp/module03)
***
### Introduction
With navigation and hashing understood, we now explore how SHAMap enables efficient tree traversal and network synchronization. These operations are critical for:
* New nodes catching up to the network
* Peers verifying they have identical state
* Extracting state information for queries
* Detecting and resolving ledger divergences
### Traversal Algorithms
SHAMap provides multiple traversal strategies depending on the use case:
**Depth-First Traversal: visitNodes**
The `visitNodes` method provides complete tree traversal:
```cpp
void SHAMap::visitNodes(
std::function callback)
{
std::stack> toVisit;
toVisit.push(mRoot);
while (!toVisit.empty()) {
auto node = toVisit.top();
toVisit.pop();
callback(node);
// Process inner node's children
if (auto inner = std::dynamic_pointer_cast(node)) {
for (int i = 15; i >= 0; --i) { // Reverse order for stack
if (auto child = inner->getChild(i)) {
toVisit.push(child);
}
}
}
}
}
```
**Use Cases:**
* Tree validation (verify all hashes)
* Bulk operations on all nodes
* Custom tree analysis
**Leaf-Only Traversal: visitLeaves**
```cpp
void SHAMap::visitLeaves(
std::function callback)
{
visitNodes([this, &callback](SHAMapTreeNode* node) {
if (auto leaf = dynamic_cast(node)) {
callback(leaf->getItem());
}
});
}
```
**Iterator-Based Traversal**
```cpp
for (auto it = shamap.begin(); it != shamap.end(); ++it) {
// Access SHAMapItem via *it
// Iteration order matches key ordering
}
```
**Parallel Traversal: walkMapParallel**
For performance-critical operations:
```cpp
void SHAMap::walkMapParallel(
std::function callback,
int numThreads)
{
// Divide tree into subtrees
// Process subtrees concurrently
// Aggregate results
}
```
**Use Cases:**
* Missing node detection at scale
* High-throughput synchronization
### Missing Node Detection
The core synchronization primitive identifies nodes needed for complete tree reconstruction:
**Algorithm: getMissingNodes**
```cpp
std::vector>
SHAMap::getMissingNodes(
std::function nodeAvailable)
{
std::vector> missing;
std::stack toVisit;
toVisit.push(SHAMapNodeID(0)); // Root node
while (!toVisit.empty() && missing.size() < MAX_RESULTS) {
SHAMapNodeID nodeID = toVisit.top();
toVisit.pop();
// Check if we have this node
auto node = getNode(nodeID);
if (!node) {
missing.push_back({nodeID, getExpectedHash(nodeID)});
continue;
}
// For inner nodes, check children
if (auto inner = dynamic_cast(node.get())) {
for (int branch = 0; branch < 16; ++branch) {
uint256 childHash = inner->getChildHash(branch);
if (childHash.isValid()) {
// Child should exist
if (!nodeAvailable(childHash)) {
// Child is missing
SHAMapNodeID childID = nodeID.getChildNodeID(branch);
toVisit.push(childID);
}
}
}
}
}
return missing;
}
```
**Output:**
Returns vector of (NodeID, Hash) pairs representing missing nodes, prioritized for network retrieval.
**Full Below Optimization**
An optimization preventing redundant traversal:
```cpp
class SHAMapInnerNode {
// Generation counter: when this subtree was marked "complete"
std::uint32_t mFullBelow = 0;
};
if (node->mFullBelow == currentGeneration) {
// Entire subtree known complete
// Skip traversal
continue;
}
```
When a subtree is verified complete (all descendants present), skip traversing it again until a new sync starts.
### Node Addition and Canonicalization
**Adding the Root Node: addRootNode**
Initializes or verifies the root node:
```cpp
SHAMapAddNode SHAMap::addRootNode(
uint256 const& hash,
Blob const& nodeData,
SHANodeFilter* filter = nullptr)
{
// Check if root already exists with matching hash
if (mRoot && mRoot->getHash() == hash) {
return SHAMapAddNode::duplicate();
}
// Validate and deserialize
auto node = deserializeNode(nodeData, SHAMapNodeID(0));
if (!node) {
return SHAMapAddNode::invalid();
}
// Canonicalize: ensure uniqueness in cache
canonicalizeNode(node);
// Set as root
mRoot = std::dynamic_pointer_cast(node);
if (filter) {
filter->foundNode(hash);
}
return SHAMapAddNode::useful();
}
```
**Adding Known Nodes: addKnownNode**
Adds interior or leaf nodes during synchronization:
```cpp
SHAMapAddNode SHAMap::addKnownNode(
SHAMapNodeID const& nodeID,
Blob const& nodeData,
SHANodeFilter* filter = nullptr)
{
// Deserialize node
auto newNode = deserializeNode(nodeData, nodeID);
if (!newNode) {
return SHAMapAddNode::invalid();
}
// Canonicalize (prevent duplicate nodes in memory)
canonicalizeNode(newNode);
// Navigate from root to parent
auto parent = getNode(nodeID.getParentNodeID());
if (!parent || !parent->isInner()) {
return SHAMapAddNode::invalid();
}
// Verify hash matches before insertion
int branch = nodeID.getBranch();
if (parent->getChildHash(branch) != newNode->getHash()) {
return SHAMapAddNode::invalid();
}
// Insert into tree
auto parentInner = std::dynamic_pointer_cast(parent);
parentInner->setChild(branch, newNode);
if (filter) {
filter->foundNode(newNode->getHash());
}
return SHAMapAddNode::useful();
}
```
### Canonicalization
**Purpose:**
Ensure nodes are unique in memory (one NodeObject per hash):
```cpp
std::shared_ptr
SHAMap::canonicalizeNode(std::shared_ptr node)
{
uint256 hash = node->getHash();
// Check cache for existing node
auto cached = mNodeCache->get(hash);
if (cached) {
return cached; // Use cached instance
}
// New node - insert into cache
mNodeCache->insert(hash, node);
return node;
}
```
**Benefits:**
1. **Memory Efficiency**: Identical nodes stored once
2. **Thread Safety**: Cache handles concurrent insertion atomically
3. **Fast Equality**: Compare pointers instead of content
4. **Shared Trees**: Multiple SHAMaps can share nodes
### Synchronization Scenario
**The Complete Flow:**
```
1. Syncing node receives ledger header with:
- Ledger sequence number
- Account state tree root hash
- Transaction tree root hash
2. Request missing nodes from peers:
- Start with root hashes
- Call getMissingNodes()
- Get list of missing (NodeID, Hash) pairs
3. Fetch missing nodes from network:
- Request nodes in parallel
- Asynchronously add them with addKnownNode()
4. Repeat until complete:
- Call getMissingNodes() again
- As new nodes are added, fewer appear missing
- Eventually: getMissingNodes() returns empty list
5. Verification:
- Recompute root hashes from all nodes
- Verify they match received values
- Ledger is now complete and verified
6. Persist:
- Nodes are stored in NodeStore
- SHAMap is marked immutable
- Ledger is now available locally
```
**Performance Metrics:**
```
Small ledger (100k accounts):
Nodes in tree: ~20,000
Network requests: ~20,000 (batch fetch reduces count)
Time to sync: seconds to minutes
Large ledger (10M accounts):
Nodes in tree: ~2,000,000
Network requests: ~100,000 (batching and parallel)
Time to sync: hours to days
```
### State Reconstruction Guarantee
The transaction tree, persisted in NodeStore, ensures unique history:
**Problem:**
Without transaction history, many sequences could produce same state:
```
State A --tx1--> State B
\--tx2--/
Multiple paths, same end state
Which actually happened?
```
**Solution:**
The transaction tree proves the exact sequence:
```
Ledger header contains:
- Account state tree root (current balances)
- Transaction tree root (complete history)
Given state tree root + transaction tree root:
Can verify exact sequence that produced this state
No ambiguity about what happened
```
**NodeStore's Role:**
Both tree nodes are persisted:
* Query state tree to find current values
* Query transaction tree to find history
* Together: complete, verifiable record
### Summary
**Key Algorithms:**
1. **Traversal**: visitNodes, visitLeaves, iterators
2. **Missing Detection**: getMissingNodes with Full Below cache
3. **Node Addition**: addRootNode, addKnownNode with canonicalization
4. **Synchronization**: Complete flow from root to leaves
5. **Verification**: Hash chain validation
**Performance Properties:**
```
Tree navigation: O(log N) worst case, O(1) typical
Missing detection: O(log N) comparisons
Complete sync: O(N) for N nodes, ~O(log N) * bandwidth
Parallel optimization: N-way parallelism with thread pools
```
**Critical Insight:**
Synchronization works because of hash-based tree structure:
* Compare root hashes: O(1) comparison
* If differ: compare children: O(16) comparisons max
* Recursive descent: O(log N) total comparisons
* vs. comparing all N leaves directly
This logarithmic advantage is what makes blockchain synchronization practical.
# The NodeStore
[← Back to SHAMap and NodeStore: Data Persistence and State Management](/core-dev-bootcamp/module03)
***
### Introduction
Now that you understand SHAMap's in-memory structure and efficient algorithms, we turn to the critical question: **how do you persist this state?**
Without persistence, every validator restart requires replaying transactions from genesis—weeks of computation. With persistence, recovery takes minutes. But persistence introduces challenges:
* **Scale**: Millions of nodes, gigabytes of data
* **Performance**: Database queries are 1000x slower than memory access
* **Flexibility**: Different operators need different storage engines
* **Reliability**: Data must survive crashes without corruption
The NodeStore solves all these problems through elegant abstraction and careful design.
{% embed url="" %}
### NodeStore's Role in XRPL Architecture
NodeStore sits at a critical junction in XRPL's architecture:
```
Application Layer
|
v
SHAMap (In-Memory State)
|
v
NodeStore Interface (Abstraction)
|
v
Cache Layer (TaggedCache) -- Hot Data in Memory
|
v
Backend Abstraction (Interface) -- Multiple implementations
|
v
Database Implementation (RocksDB, NuDB, etc.)
|
v
Physical Storage (Disk)
```
**SHAMap's Dependency:**
SHAMap needs to retrieve historical nodes:
```cpp
// During synchronization or historical queries:
std::shared_ptr node = nodestore.fetch(nodeHash);
```
But SHAMap doesn't know or care about:
* How data is stored
* Which database backend is used
* Where the data is physically located
* How caching is implemented
All that complexity is hidden behind NodeStore's interface.
### Core Purpose
NodeStore provides four critical services:
**1. Persistence**
```
SHAMap state exists in memory
↓
Serialize nodes to disk
↓
Survive application crash
↓
Reconstruct state on startup
```
**2. Consistent Interface**
```
// Application code doesn't change regardless of backend
nodestore.store(node); // Works with RocksDB, NuDB, SQLite...
auto node = nodestore.fetch(hash);
```
**3. Performance Optimization**
```
Database queries: 1-10 milliseconds
Memory access: 1-10 microseconds
1000x difference!
NodeStore uses caching to keep hot data in memory
Typical hit rate: 90-95%
Result: Average latency near memory speed
```
**4. Lifecycle Management**
```
Startup: Locate and open database
Runtime: Store and retrieve nodes as needed
Shutdown: Cleanly close database
Rotation: Enable online deletion and archival
```
### NodeObject: The Fundamental Storage Unit
The atomic unit of storage in XRPL is the NodeObject:
**Structure:**
```cpp
class NodeObject {
// Type of object (LEDGER_HEADER, ACCOUNT_NODE, TRANSACTION_NODE)
NodeObjectType mType;
// 256-bit unique identifier
uint256 mHash;
// Serialized content (variable length)
Blob mData;
public:
// Factory: create NodeObject from components
static std::shared_ptr createObject(
NodeObjectType type,
Blob const& data,
uint256 const& hash);
// Access methods
NodeObjectType getType() const { return mType; }
uint256 const& getHash() const { return mHash; }
Blob const& getData() const { return mData; }
};
```
**Key Characteristics:**
1. **Immutable Once Created**: Cannot modify data after creation
2. **Hash as Key**: Hash uniquely identifies the object
3. **Type Distinguishing**: Type prevents hash collisions between different data types
4. **Serialized Format**: Data is already in wire format
**NodeObject Types:**
| Type | Purpose | Numeric Value |
| -------------------- | -------------------------------- | ------------- |
| hotLEDGER | Ledger headers and metadata | 1 |
| hotACCOUNT\_NODE | Account state tree nodes | 3 |
| hotTRANSACTION\_NODE | Transaction tree nodes | 4 |
| hotUNKNOWN | Unknown/unrecognized types | 0 |
| hotDUMMY | Cache marker for missing entries | 512 |
**Type Prefix in Hashing:**
Type fields prevent collisions:
```cpp
// Two different types of data, might have same structure
// Type prefix ensures different hashes
uint256 hash_account = SHA512Half(
ACCOUNT_TYPE_BYTE || accountData);
uint256 hash_transaction = SHA512Half(
TRANSACTION_TYPE_BYTE || accountData);
// hash_account != hash_transaction
```
### NodeObject Lifecycle
**Creation**
```
During transaction processing:
1. Transaction validated and applied to SHAMap
2. SHAMap nodes modified
3. Each modified node serialized
4. NodeObject created with type, hash, serialized data
5. Stored in NodeStore
```
**Storage**
```
For each NodeObject:
1. Encode to persistent format
2. Compute key (same as hash)
3. Write to database
4. Backend handles actual I/O
```
**Caching**
```
After storage:
1. Keep in memory for fast reaccess
2. Move to cache tier
3. Evict when cache capacity exceeded
4. Dummy objects mark "known missing" (avoid repeated lookups)
```
**Retrieval**
```
When SHAMap needs a node:
1. Check cache (microseconds)
2. If miss, query database (milliseconds)
3. Deserialize and validate
4. Add to cache
5. Return to SHAMap
```
**Archival**
```
After ledger is validated and no longer current:
1. May be retained for history
2. Or moved to archive during rotation
3. Or deleted based on retention policy
```
### Backend Abstraction
The Backend class defines the minimal interface for any storage system:
**Core Operations:**
```cpp
class Backend {
// Store single object
virtual Status store(NodeObject const& object) = 0;
// Retrieve single object by hash
virtual Status fetch(uint256 const& hash,
std::shared_ptr& object) = 0;
// Persist multiple objects atomically
virtual Status storeBatch(std::vector const& batch) = 0;
// Retrieve multiple objects efficiently
virtual Status fetchBatch(std::vector const& hashes,
std::vector& objects) = 0;
// Lifecycle
virtual Status open(std::string const& path) = 0;
virtual Status close() = 0;
virtual int fdRequired() const = 0; // File descriptors needed
};
```
**Status Codes:**
```cpp
enum class Status {
ok, // Operation succeeded
notFound, // Key doesn't exist
dataCorrupt, // Data integrity check failed (fatal)
backendError // Backend error
};
```
**Backend Independence:**
NodeStore sits above backends, application logic unchanged:
```cpp
// Same code works with any backend
struct DatabaseConfig {
std::string type; // "rocksdb", "nudb", "sqlite", etc.
std::string path;
// ... backend-specific options
};
auto backend = createBackend(config);
NodeStore store(backend);
// Application uses NodeStore
store.fetch(hash); // Works regardless of backend
store.store(node);
```
### Supported Backends
**RocksDB (Recommended for Most Cases)**
```cpp
Backend* createRocksDBBackend(std::string const& path) {
return new RocksDBBackend(path);
}
```
* Modern key-value store developed by Facebook
* LSM tree (Log-Structured Merge tree) design
* Excellent performance for XRPL workloads
* Built-in compression support
* Active maintenance
**Characteristics:**
* Write throughput: \~10,000-50,000 objects/second
* Read throughput: \~100,000+ objects/second
* Compression: Reduces disk space by 50-70%
**NuDB (High-Throughput Alternative)**
```cpp
Backend* createNuDBBackend(std::string const& path) {
return new NuDBBackend(path);
}
```
* Purpose-built for XRPL by Ripple
* Append-only design optimized for SSD
* Higher write throughput than RocksDB
* Efficient space utilization
**Characteristics:**
* Write throughput: \~50,000-200,000 objects/second
* Read throughput: \~100,000+ objects/second
* Better for high-volume systems
**Testing Backends**
```cpp
Backend* createMemoryBackend() {
return new MemoryBackend(); // In-memory, non-persistent
}
Backend* createNullBackend() {
return new NullBackend(); // No-op backend
}
```
### Data Encoding Format
To enable backend independence, NodeStore uses a standardized encoding:
**Encoded Blob Structure:**
```
Byte Offset | Field | Description
0-7 | Reserved | Set to zero, reserved for future use
8 | Type | NodeObjectType enumeration value
9+ | Data | Serialized object payload (variable length)
```
**Encoding Process:**
```cpp
void encodeNodeObject(NodeObject const& obj, Blob& blob) {
// Add 8 reserved bytes
blob.resize(8, 0);
// Add type byte
blob.push_back(obj.getType());
// Add data payload
blob.append(obj.getData());
}
```
**Decoding Process:**
```cpp
std::shared_ptr decodeNodeObject(
uint256 const& hash,
Blob const& blob)
{
if (blob.size() < 9) {
return nullptr; // Corrupted
}
NodeObjectType type = static_cast(blob[8]);
Blob data(blob.begin() + 9, blob.end());
return NodeObject::createObject(type, data, hash);
}
```
**Benefits:**
1. **Backend Agnostic**: Any backend can store/retrieve encoded blobs
2. **Self-Describing**: Type embedded, forward-compatible with unknown types
3. **Efficient**: Minimal overhead (8 bytes) per object
4. **Validated**: Type byte catches most corruption
### Database Key as Hash
The database key is the object's hash (not a sequential ID):
```cpp
Status Backend::store(NodeObject const& obj) {
uint256 key = obj.getHash(); // 256-bit hash as key
Blob value = encode(obj); // Encoded blob as value
return database.put(key, value); // Key-value store
}
```
**Implications:**
1. **Direct Retrieval**: Any node retrievable by hash
2. **Deduplication**: Identical content produces identical hash → same key
3. **Immutability**: Hash never changes for given data
4. **Verification**: Can verify data by recomputing hash
### Integration Architecture
NodeStore integrates with SHAMap through the Family pattern:
```cpp
// Family provides NodeStore access to SHAMap
class Family {
virtual std::shared_ptr getNodeStore() = 0;
virtual std::shared_ptr getTreeNodeCache() = 0;
virtual std::shared_ptr getFullBelowCache() = 0;
};
class NodeFamily : public Family {
std::shared_ptr mNodeStore;
std::shared_ptr mTreeCache;
std::shared_ptr mFullBelow;
// ... implement Family interface
};
// SHAMap uses Family for storage access
class SHAMap {
std::shared_ptr mFamily;
std::shared_ptr getNode(uint256 const& hash) {
// Try cache first
auto cached = mFamily->getTreeNodeCache()->get(hash);
if (cached) return cached;
// Fetch from NodeStore
auto obj = mFamily->getNodeStore()->fetch(hash);
if (obj) {
auto node = deserializeNode(obj);
// Cache for future access
mFamily->getTreeNodeCache()->insert(hash, node);
return node;
}
return nullptr;
}
};
```
### Summary
**Key Architectural Elements:**
1. **NodeObject**: Atomic storage unit (type, hash, data)
2. **Backend Interface**: Minimal, consistent interface for storage
3. **Abstraction**: Decouples application logic from storage implementation
4. **Encoding Format**: Standardized format enables backend independence
5. **Key as Hash**: Direct retrieval without index lookups
6. **Family Pattern**: Provides access to caching and storage
**Design Properties:**
* **Backend Flexibility**: Switch storage engines without code changes
* **Scale**: Handles millions of objects efficiently
* **Persistence**: Survives crashes and restarts
* **Verification**: Data integrity through hashing
* **Simplicity**: Minimal interface hides complexity
In the next chapter, we'll explore the critical Cache Layer that makes NodeStore practical for high-performance systems.
# Storage Abstraction and Backend Implementations
[← Back to SHAMap and NodeStore: Data Persistence and State Management](/core-dev-bootcamp/module03)
***
### Introduction
This chapter explains the NodeStore storage abstraction and available backends in Rippled. You will learn how to choose between RocksDB, NuDB, and testing backends, configure them for optimal performance, and understand the standardized encoding format that ensures backend interoperability. Practical guidance on tuning and migration is also provided to help maintain reliable and efficient ledger storage.
## Quick Reference
### RocksDB (Recommended)
**When to use:**
* General-purpose validators
* Balanced performance and simplicity
* Standard configurations
**Configuration:**
```ini
[node_db]
type = RocksDB
path = /data/rippled.db
cache_size = 256
```
**Performance:**
* Write throughput: 10,000-50,000 objects/sec
* Compression: 50-70% disk space savings
* Good for: Most deployments
### NuDB (High-Performance)
**When to use:**
* High-transaction-volume networks
* Maximum write throughput needed
* Modern SSD storage
**Configuration:**
```ini
[node_db]
type = NuDB
path = /data/nudb
```
**Performance:**
* Write throughput: 50,000-200,000 objects/sec
* Optimized for: Sequential writes
* Good for: Archive nodes, heavily-used validators
### Testing Backends
* **Memory**: In-memory, non-persistent (testing only)
* **Null**: No-op backend (unit tests)
## Encoding Format Reference
The standardized encoding format enables backend independence:
**Structure (from Chapter 6):**
```
Bytes 0-7: Reserved (set to zero)
Byte 8: Type (NodeObjectType enumeration)
Bytes 9+: Serialized data payload
```
This format is handled transparently by the Database layer, but understanding it is important for:
* Backend implementation
* Data corruption diagnosis
* Migration between backends
## Configuration Tuning
### RocksDB Options
```ini
[node_db]
type = RocksDB
path = /var/lib/rippled/db/rocksdb
cache_size = 256 # Cache size in MB
cache_age = 60 # Age limit in seconds
# Performance tuning
compression = true # Enable compression
block_cache_size = 256 # Block cache in MB
write_buffer_size = 64 # Write buffer in MB
max_open_files = 100
```
### NuDB Options
```ini
[node_db]
type = NuDB
path = /var/lib/rippled/db/nudb
# NuDB specific
key_size = 32 # Key size (always 32 for SHA256)
block_size = 4096 # Block size for writes
```
## Migration Between Backends
To migrate from one backend to another:
```bash
# 1. Stop the server
systemctl stop rippled
# 2. Export from current backend
rippled --export current_db export.json
# 3. Update configuration
# Change [node_db] type in rippled.cfg
# 4. Import to new backend
rippled --import export.json --ledger-db new_db
# 5. Restart server
systemctl start rippled
```
## For Detailed Reference
See the following sections of Chapter 6:
* "Backend Abstraction" - Interface design
* "Supported Backends" - Feature comparison
* "Data Encoding Format" - Serialization details
For implementation details, consult:
* `rippled/src/xrpld/nodestore/Backend.h`
* `rippled/src/xrpld/nodestore/Database.h`
* `rippled/src/xrpld/nodestore/backend/*Factory.cpp`
# Cache Layer and Performance Optimization
[← Back to SHAMap and NodeStore: Data Persistence and State Management](/core-dev-bootcamp/module03)
***
### Introduction
Database queries are fundamentally slow compared to memory access:
```
Memory access: 1-10 microseconds
Cache miss latency: 1-10 milliseconds
Ratio: 1000x slower
```
With millions of nodes and transactions processing at ledger-close speeds (every 3-5 seconds), any system that naively queries the database for every node access will fail.
The NodeStore's Cache Layer solves this through a multi-tier strategy that keeps hot data in memory while safely delegating cold data to disk.
### The Performance Problem
**Scenario: Synchronizing from the Network**
A new node joins XRPL and must catch up to current ledger. This requires:
1. Fetching missing ledgers (blocks of transactions)
2. For each ledger, fetching all state nodes
3. Verifying each node's hash
4. Storing nodes to disk
**Naive Approach (No Cache):**
```cpp
for (each node in ledger) {
backend.fetch(nodeHash); // Database query - 10ms
}
For 10,000 nodes in ledger:
10,000 × 10ms = 100 seconds per ledger
For 100,000 ledgers to catch up:
100 seconds × 100,000 = 1,157 days
```
**Clearly infeasible.**
**With Caching (90% Hit Rate):**
```cpp
for (each node in ledger) {
if (cache.has(nodeHash)) {
cache.get(nodeHash); // 10 microseconds
} else {
backend.fetch(nodeHash); // 10 milliseconds
}
}
For 10,000 nodes:
9,000 × 10µs (cache hits) = 90ms
1,000 × 10ms (cache misses) = 10,000ms
Total: 10,090ms ≈ 10 seconds per ledger
For 100,000 ledgers:
100,000 × 10 seconds = 1,000,000 seconds ≈ 11.5 days
```
**Still slow, but realistic with parallel processing.**
The difference between possible and impossible is caching.
### TaggedCache Architecture
The NodeStore's primary cache is the TaggedCache:
**Purpose:**
```cpp
class TaggedCache {
// Keep frequently accessed NodeObjects in memory
// Minimize expensive database queries
// Provide thread-safe concurrent access
};
```
**Structure:**
```cpp
class TaggedCache {
private:
// Key: NodeObject hash (uint256)
// Value: shared_ptr
std::unordered_map> mCache;
// Protect concurrent access
std::mutex mLock;
// Configuration
size_t mMaxSize; // Maximum objects in cache
std::chrono::seconds mMaxAge; // Maximum age before eviction
};
public:
// Retrieve from cache
std::shared_ptr get(uint256 const& hash);
// Store in cache
void insert(uint256 const& hash,
std::shared_ptr const& obj);
// Remove from cache
void remove(uint256 const& hash);
// Evict old entries
void evictExpired();
};
```
**Cache Tiers:**
NodeStore implements a two-tier caching strategy:
```
Application
↓
L1: TaggedCache (In-Memory)
Fast: 1-10 microseconds
Size: ~100MB - 5GB (configurable)
↓ (on miss)
L2: Backend Database (Persistent)
Slow: 1-10 milliseconds
Size: Unlimited (disk)
↓
Physical Storage (Disk)
```
**Fetch Algorithm:**
```cpp
std::shared_ptr fetch(uint256 const& hash) {
// Step 1: Check cache
{
std::lock_guard lock(mCacheLock);
auto it = mCache.find(hash);
if (it != mCache.end()) {
// Found in cache
recordHit(hash);
return it->second;
}
}
// Step 2: Cache miss - query backend
auto obj = backend->fetch(hash);
if (!obj) {
// Object not found anywhere
// Cache a dummy marker to prevent repeated lookups
cacheDummy(hash);
return nullptr;
}
// Step 3: Update cache with newly fetched object
{
std::lock_guard lock(mCacheLock);
mCache.insert({hash, obj});
}
recordMiss(hash);
return obj;
}
```
### Cache Insertion Strategy
**When Objects Enter Cache:**
```
1. On Database Fetch
fetch() fails or succeeds → always cache result
(Success: cache object, Failure: cache dummy)
2. On Storage
store() called → immediately cache object
Object will be reaccessed soon
3. Predictive Loading
When traversing SHAMap, prefetch likely-needed siblings
```
**Dummy Objects:**
Special marker objects prevent wasted lookups:
```cpp
class NodeObject {
static std::shared_ptr createDummy(uint256 const& hash) {
auto obj = std::make_shared();
obj->mType = hotDUMMY; // Type 512: marker
obj->mHash = hash;
obj->mData.clear(); // Empty data
return obj;
}
bool isDummy() const {
return mType == hotDUMMY;
}
};
// In fetch:
std::shared_ptr backend_result = backend->fetch(hash);
if (!backend_result) {
// Not found - cache dummy to avoid retry
auto dummy = NodeObject::createDummy(hash);
cache.insert(hash, dummy);
return nullptr;
}
```
**Benefit of Dummies:**
```
Scenario: Syncing but peer doesn't have a node
Without dummies:
Request node X from network
Peer: "I don't have it"
Local check: Try backend
Backend: "Not there"
(repeat this sequence multiple times)
With dummies:
Request node X from network
Peer: "I don't have it"
Local check: Cache hit on dummy
Immediately know: "Not available"
Don't retry
```
Prevents thundering herd of repeated failed lookups.
### Cache Eviction
Cache capacity is limited. When full, old objects must be evicted.
**Eviction Triggers:**
```cpp
void insertWithEviction(uint256 const& hash,
std::shared_ptr const& obj) {
{
std::lock_guard lock(mCacheLock);
// Check size limit
if (mCache.size() >= mMaxSize) {
evictLRU(); // Remove least recently used
}
mCache.insert({hash, obj});
}
// Check age limit (periodic)
if (shouldEvictExpired()) {
evictExpired(); // Remove objects older than mMaxAge
}
}
```
**LRU (Least Recently Used) Eviction:**
```cpp
void evictLRU() {
// Find object with oldest access time
auto oldest = findOldestAccess();
// Remove from cache
mCache.erase(oldest->hash);
}
// Track access times
struct CacheEntry {
std::shared_ptr object;
std::chrono::steady_clock::time_point lastAccess;
};
```
**Age-Based Eviction:**
```cpp
void evictExpired() {
auto now = std::chrono::steady_clock::now();
for (auto it = mCache.begin(); it != mCache.end();) {
auto age = now - it->second.lastAccess;
if (age > mMaxAge) {
// Object too old - remove
it = mCache.erase(it);
} else {
++it;
}
}
}
```
**Configuration Parameters:**
```cpp
// From rippled.cfg
[node_db]
cache_size = 256 // MB
cache_age = 60 // seconds
```
**Impact of Configuration:**
```
Small cache:
cache_size = 32 MB
Hit rate: ~60% (more evictions)
Disk queries: 40% of lookups (slower)
Large cache:
cache_size = 1024 MB
Hit rate: ~95% (fewer evictions)
Disk queries: 5% of lookups (faster)
Memory usage: Higher
Operators choose based on available RAM and performance needs
```
### Cache Performance Metrics
NodeStore tracks cache effectiveness:
```cpp
struct CacheMetrics {
uint64_t hits; // Cache hits
uint64_t misses; // Cache misses
uint64_t inserts; // Objects added
uint64_t evictions; // Objects removed
double hitRate() const {
return hits / (double)(hits + misses);
}
};
// Typical production hit rates:
// Well-configured: 90-95%
// Poorly tuned: 60-75%
// Synchronized node: 98% (accessing recent ledgers)
```
**Example Metrics:**
```
From a running XRPL validator:
Period: 1 hour
Cache hits: 86,400 (hit on hot data)
Cache misses: 7,200 (query database)
Hit rate: 92.3% (excellent)
Latency impact:
Average: 0.92 * 1µs + 0.08 * 10ms = 0.81 milliseconds
Without cache: Average 10 milliseconds
Speedup: 12.3x
Throughput impact:
Queries handled: 93,600 per hour
If all disk: 360 per hour (would be ~260x slower)
Caching enables actual performance
```
### Synchronization Optimization
During network synchronization, special techniques optimize caching:
**Prefetching**
```cpp
void synchronizeNode(SHAMapNodeID nodeID,
uint256 const& nodeHash) {
// Fetch this node
auto node = fetch(nodeHash);
if (auto inner = dynamic_cast(node)) {
// This is an inner node
// Likely next accesses are to its children
// Prefetch children to warm cache
for (int i = 0; i < 16; ++i) {
uint256 childHash = inner->getChildHash(i);
if (childHash.isValid()) {
// Asynchronously prefetch
asyncFetch(childHash);
}
}
}
}
```
**Batch Loading**
```cpp
// Fetch multiple nodes in single operation
std::vector hashes = {hash1, hash2, hash3, ...};
auto results = backend->fetchBatch(hashes);
// Reduces backend overhead
// Populates cache efficiently
// Parallelizes I/O operations
```
**Deferred Reads**
```cpp
// During synchronization, identify missing nodes
std::vector missing = getMissingNodes(shamap);
// Request from peers asynchronously
// When they arrive, cache them
// Continue traversal without blocking on network
// This allows pipelining: request more while processing previous results
```
### Working Set Management
Different phases of operation have different access patterns:
**During Normal Operation (Steady State)**
```
Access pattern: Recent ledgers frequently accessed
Root hash: checked at consensus
Recent state nodes: queried for transactions
Old historical data: rarely accessed
Cache configuration:
Keep recent ledgers fully cached
Let old ledgers evict to make room
Result: Excellent hit rate for current operations
```
**During Synchronization**
```
Access pattern: Missing nodes from network
Need to verify hash chain from root to leaf
Often fetching siblings (related nodes)
May access same node multiple times
Cache strategy:
Smaller cache acceptable (still beneficial)
Prefetch siblings when fetching parent
Use dummy markers to avoid retry storms
Result: Synchronization completes in hours vs days
```
**After Sync Completion**
```
Access pattern: Back to steady-state recent ledger access
Cache characteristics:
Most-accessed nodes pinned in cache
Hit rate quickly reaches 90%+
Warm cache from prior work
```
### Thread Safety
Cache must be safe for concurrent access:
```cpp
class TaggedCache {
std::unordered_map mCache;
mutable std::shared_mutex mLock; // Allow multiple readers
public:
std::shared_ptr get(uint256 const& hash) {
std::shared_lock lock(mLock);
auto it = mCache.find(hash);
return (it != mCache.end()) ? it->second.object : nullptr;
}
void insert(uint256 const& hash,
std::shared_ptr const& obj) {
std::unique_lock lock(mLock);
// Check if size exceeded
if (mCache.size() >= mMaxSize) {
evictLRU(); // Exclusive lock held, safe to modify
}
mCache[hash] = {obj, now()};
}
};
```
**Concurrency Properties:**
```
Multiple readers:
Many threads can fetch simultaneously
No contention for cache hits
Scaling: hundreds of concurrent fetches possible
Insert/evict operations:
Exclusive lock for modification
Short-lived (just map operations)
Background eviction: doesn't block fetches
```
### Advanced Caching: Full Below Optimization
During synchronization, special tracking prevents redundant work:
**The Problem:**
```
Synchronizing large subtree:
1. Check root: hash differs (need to sync)
2. Check children: most hashes match, one differs
3. Recurse into different child
4. Check its children: all match
5. Later: receive more nodes
6. Check same grandparent again
7. Redundant check (we already knew it was full)
```
**The Solution: Full Below Generation Counter**
```cpp
class SHAMapInnerNode {
// Generation number marking when this node was verified complete
std::uint32_t mFullBelow;
};
class FullBelowCache {
// Current generation
std::uint32_t mCurrentGeneration = 0;
bool isKnownFull(SHAMapInnerNode* node) {
return node->mFullBelow == mCurrentGeneration;
}
void markFull(SHAMapInnerNode* node) {
node->mFullBelow = mCurrentGeneration;
}
void invalidate() {
++mCurrentGeneration; // Invalidate all prior markings
}
};
```
**Benefit:**
```
When synchronizing:
Fetch subtree, verify all descendants present
Mark as "full below" with generation ID
Later sync process checks generation
If matches current: skip this subtree (known complete)
If differs: need to re-verify (new sync started)
Result: Avoids re-traversing known-complete subtrees
Significant speedup in incremental sync scenarios
```
### Summary
**Key Concepts:**
1. **Multi-Tier Caching**: Memory + disk strategy balances performance and capacity
2. **LRU Eviction**: Keeps frequently-accessed data, evicts cold data
3. **Dummy Markers**: Prevent repeated failed lookups
4. **Metrics Tracking**: Monitor cache effectiveness
5. **Synchronization Optimization**: Prefetch and batch loading
6. **Full Below Cache**: Avoid redundant traversal during sync
7. **Thread Safety**: Shared locks for multiple readers
**Performance Impact:**
```
Without caching: Synchronization would take months
With 90% hit rate: Synchronization takes hours
With 95% hit rate: Synchronization takes minutes
Production systems: Carefully tuned caches running at 92-96% hit rate
```
The Cache Layer transforms NodeStore from theoretical to practical. Database queries are unavoidable, but caching hides their latency, allowing XRPL to operate at microsecond efficiency despite millisecond database performance.
# Database Operations and Lifecycle Management
[← Back to SHAMap and NodeStore: Data Persistence and State Management](/core-dev-bootcamp/module03)
***
### Introduction
Beyond the cache layer, the Database class orchestrates the complete lifecycle of NodeStore operations:
* Initialization on startup
* Runtime fetch/store operations
* Asynchronous background operations
* Graceful shutdown
* Database rotation and archival
This chapter covers these operational aspects that are critical for production XRPL nodes.
### Core Database Interface
The Database class provides higher-level operations above Backend:
**Key Responsibilities:**
```cpp
class Database {
public:
// Synchronous operations
std::shared_ptr fetchNodeObject(
uint256 const& hash,
std::uint32_t ledgerSeq = 0);
void store(std::shared_ptr const& obj);
void storeBatch(std::vector> const& batch);
// Asynchronous operations
void asyncFetch(
uint256 const& hash,
std::function)> callback);
// Management
void open(std::string const& path);
void close();
// Metrics and diagnostics
Json::Value getCountsJson() const;
};
```
### DatabaseNodeImp: Single Backend Implementation
The standard implementation for most XRPL validators:
**Architecture:**
```
Application
↓
DatabaseNodeImp (Coordination)
/ | \
/ | \
Cache Backend Threads
(Hot) (Disk) (Async)
```
**Storage Flow:**
```cpp
void DatabaseNodeImp::store(std::shared_ptr const& obj) {
// Step 1: Update cache immediately (likely reaccess soon)
{
std::lock_guard lock(mCacheLock);
mCache.insert(obj->getHash(), obj);
}
// Step 2: Encode to persistent format
Blob encoded = encodeObject(obj);
// Step 3: Persist to backend
Status status = mBackend->store(obj->getHash(), encoded);
if (status != Status::ok) {
// Log error but don't crash
// Backend error doesn't lose data (already in cache)
logError("Backend store failed", status);
}
// Step 4: Update metrics
mMetrics.bytesWritten += encoded.size();
mMetrics.objectsWritten++;
}
```
**Fetch Flow:**
```cpp
std::shared_ptr DatabaseNodeImp::fetchNodeObject(
uint256 const& hash,
uint32_t ledgerSeq)
{
// Step 1: Check cache
{
std::lock_guard lock(mCacheLock);
auto cached = mCache.get(hash);
if (cached) {
mMetrics.cacheHits++;
return cached;
}
}
// Step 2: Query backend (potentially slow)
Blob encoded;
Status status = mBackend->fetch(hash, encoded);
std::shared_ptr result;
if (status == Status::ok) {
result = decodeObject(hash, encoded);
} else if (status == Status::notFound) {
// Not found - cache dummy to prevent retry
result = nullptr;
} else {
// Backend error
logWarning("Backend fetch error", status);
return nullptr;
}
// Step 3: Update cache
{
std::lock_guard lock(mCacheLock);
if (result) {
mCache.insert(hash, result);
} else {
mCache.insertDummy(hash);
}
}
// Step 4: Update metrics
mMetrics.cacheMisses++;
mMetrics.bytesRead += encoded.size();
return result;
}
```
### Batch Operations
Batch operations improve efficiency:
**Batch Store:**
```cpp
void DatabaseNodeImp::storeBatch(
std::vector> const& batch)
{
// Step 1: Update cache for all objects
{
std::lock_guard lock(mCacheLock);
for (auto const& obj : batch) {
mCache.insert(obj->getHash(), obj);
}
}
// Step 2: Encode all objects
std::vector> encoded;
encoded.reserve(batch.size());
for (auto const& obj : batch) {
encoded.emplace_back(obj->getHash(), encodeObject(obj));
}
// Step 3: Store atomically in backend
Status status = mBackend->storeBatch(encoded);
// Step 4: Update metrics
for (auto const& [hash, blob] : encoded) {
mMetrics.bytesWritten += blob.size();
}
mMetrics.objectsWritten += batch.size();
}
```
**Benefits:**
```
Without batch:
Write 1000 objects → 1000 backend transactions
1000 disk I/O operations
With batch:
Write 1000 objects → 1 backend transaction
1 disk I/O operation (atomic write)
Throughput improvement: 10-50x depending on backend
```
**Batch Size Limits:**
```cpp
static const size_t BATCH_WRITE_PREALLOCATE_SIZE = 256;
static const size_t BATCH_WRITE_LIMIT_SIZE = 65536;
// Prevents:
// 1. Memory exhaustion (unbounded batches)
// 2. Transaction timeout (backend transaction too large)
// 3. Excessive latency (batching too much)
```
### Asynchronous Operations
Background threads handle expensive operations without blocking:
**Asynchronous Fetch:**
```cpp
void DatabaseNodeImp::asyncFetch(
uint256 const& hash,
std::function)> callback)
{
// Step 1: Queue request
mAsyncQueue.enqueue({hash, callback});
// Step 2: Background thread processes
// Wakes up, dequeues batch, fetches, invokes callbacks
// Meanwhile, caller continues without blocking
}
// Thread pool implementation
void asyncWorkerThread() {
while (running) {
// Wait for work or timeout
auto batch = mAsyncQueue.dequeueBatch(timeout);
if (batch.empty()) {
continue;
}
// Fetch all in batch (more efficient)
std::vector hashes;
for (auto const& [hash, callback] : batch) {
hashes.push_back(hash);
}
auto results = mBackend->fetchBatch(hashes);
// Invoke callbacks
for (auto const& [hash, callback] : batch) {
auto result = results[hash];
callback(result);
}
}
}
```
**Use Cases:**
```
1. Synchronization:
Requesting many nodes from network
Can queue hundreds of async fetches
Process results as they arrive
2. API queries:
Historical account queries
Don't block validator thread
Return results via callback
3. Background tasks:
Cache warming
Prefetching likely-needed nodes
Doesn't impact real-time performance
```
### Initialization and Shutdown
**Startup Sequence:**
```cpp
void DatabaseNodeImp::open(DatabaseConfig const& config) {
// Step 1: Parse configuration
std::string backend_type = config.get("type");
std::string database_path = config.get("path");
// Step 2: Create backend instance
mBackend = createBackend(backend_type, database_path);
// Step 3: Open backend (connects to database)
Status status = mBackend->open();
if (status != Status::ok) {
throw std::runtime_error("Failed to open database");
}
// Step 4: Allocate cache
size_t cache_size_mb = config.get("cache_size");
mCache.setMaxSize(cache_size_mb * 1024 * 1024);
// Step 5: Start background threads
int num_threads = config.get("async_threads", 4);
for (int i = 0; i < num_threads; ++i) {
mThreadPool.emplace_back([this] { asyncWorkerThread(); });
}
// Step 6: Optional: import from another database
if (config.has("import_db")) {
importFromDatabase(config.get("import_db"));
}
// Step 7: Ready for operations
mReady = true;
}
```
**Shutdown Sequence:**
```cpp
void DatabaseNodeImp::close() {
// Step 1: Stop accepting new operations
mReady = false;
// Step 2: Wait for in-flight async operations to complete
mAsyncQueue.stop();
for (auto& thread : mThreadPool) {
thread.join();
}
// Step 3: Flush any pending writes
// (Most backends buffer writes)
mBackend->flush();
// Step 4: Clear cache (will be regenerated on restart)
mCache.clear();
// Step 5: Close backend database
Status status = mBackend->close();
if (status != Status::ok) {
logWarning("Backend close not clean", status);
}
}
```
### DatabaseRotatingImp: Advanced Rotation
For production systems needing online deletion:
**Problem Solved:**
Without deletion, database grows unbounded:
```
Each ledger adds new nodes
Over time: thousands of gigabytes
Eventually: disk full
Options:
1. Stop validator (unacceptable)
2. Manual pruning (requires downtime)
3. Rotation (online deletion)
```
**Rotation Architecture:**
```
Application
↓
DatabaseRotatingImp
/ | \
/ | \
Writable Archive Cache
Backend Backend
(New) (Old)
```
**Rotation Process:**
```cpp
void DatabaseRotatingImp::rotate() {
// Step 1: Stop writes to current backend
auto old_writable = mWritableBackend;
auto old_archive = mArchiveBackend;
// Step 2: Create new writable backend
mWritableBackend = createNewBackend();
mWritableBackend->open();
// Step 3: Transition current writable → archive
mArchiveBackend = old_writable;
// Step 4: Delete old archive (in background)
deleteBackendAsync(old_archive);
// Step 5: Copy critical data if needed
// (e.g., ledger headers required for validation)
copyCriticalData(old_archive, mWritableBackend);
// Step 6: Continue operation with no downtime
}
```
**Dual Fetch Logic:**
```cpp
std::shared_ptr DatabaseRotatingImp::fetchNodeObject(
uint256 const& hash,
uint32_t ledgerSeq,
bool duplicate)
{
// Check cache first
auto cached = mCache.get(hash);
if (cached) {
return cached;
}
// Try writable (current ledgers)
auto obj = mWritableBackend->fetch(hash);
if (obj) {
mCache.insert(hash, obj);
return obj;
}
// Try archive (older ledgers)
obj = mArchiveBackend->fetch(hash);
if (obj) {
mCache.insert(hash, obj);
// Optionally duplicate to writable for longevity
if (duplicate) {
mWritableBackend->store(hash, obj);
}
return obj;
}
return nullptr;
}
```
**Benefits:**
```
With rotation:
Ledger 1000000: stored in Writable
Ledger 1000001-1100000: stored in Writable
Ledger 900000-999999: stored in Archive
Ledger 900000: Delete ledger → Keep recent 100k only
Old archive deleted → Disk space reclaimed
No downtime, no backups needed, bounded growth
```
### Metrics and Monitoring
NodeStore exposes comprehensive metrics for monitoring:
```cpp
struct NodeStoreMetrics {
// Storage metrics
uint64_t objectsWritten;
uint64_t bytesWritten;
std::chrono::microseconds writeLatency;
// Retrieval metrics
uint64_t objectsFetched;
uint64_t cacheHits;
uint64_t cacheMisses;
uint64_t bytesFetched;
std::chrono::microseconds fetchLatency;
// Cache metrics
size_t cacheObjects;
double cacheHitRate() const {
return cacheHits / (double)(cacheHits + cacheMisses);
}
// Threading metrics
size_t asyncQueueDepth;
int activeAsyncThreads;
};
```
**Monitoring Typical Values:**
```
Hit rate: 92-96% (well-configured systems)
Write latency: 0.1-1 ms per object
Fetch latency: 0.01-0.1 ms per object (mostly cache hits)
Cache size: 128MB - 2GB
Async queue depth: 0-100 (queue length)
```
**Alert Thresholds:**
```
If hit rate < 80%: Cache too small or thrashing
If write latency > 10ms: Backend I/O struggling
If queue depth > 10000: Not keeping up with load
If fetch latency > 100ms: Serious performance issue
```
### Summary
**Key Operational Components:**
1. **Fetch/Store**: Core operations with caching
2. **Batch Operations**: Efficient bulk operations
3. **Async Operations**: Background threads for non-blocking I/O
4. **Lifecycle**: Startup, shutdown, error handling
5. **Rotation**: Online deletion and archival
6. **Metrics**: Monitoring and diagnostics
**Design Properties:**
* **Reliability**: Graceful error handling, no data loss
* **Performance**: Batch operations, async threading
* **Scalability**: Online rotation enables unbounded operation
* **Observability**: Comprehensive metrics for diagnostics
* **Flexibility**: Different database backends, configurations
The Database layer transforms raw backend capabilities into a reliable, performant storage system that can handle blockchain-scale data volumes while maintaining the responsiveness required for real-time validation.
# Cryptographic Proofs and State Reconstruction
[← Back to SHAMap and NodeStore: Data Persistence and State Management](/core-dev-bootcamp/module03)
***
### Introduction
The combination of SHAMap and NodeStore provides more than just efficient storage—they enable cryptographic proofs that transactions were executed correctly and state was computed honestly.
This chapter explores:
* Merkle proof generation and verification
* State reconstruction from transaction history
* Cross-chain or light-client verification
* The guarantee of verifiable ledger history
### Merkle Proof Generation
A Merkle proof allows someone to verify that data is in a tree without reconstructing the entire tree.
**Algorithm: getProofPath**
```cpp
std::optional>
SHAMap::getProofPath(uint256 const& key)
{
std::vector path;
auto node = std::dynamic_pointer_cast(mRoot);
for (int depth = 0; depth < 64; ++depth) {
// Serialize current node
Blob serialized = node->serialize();
path.push_back(serialized);
// Is this the target leaf?
if (auto leaf = std::dynamic_pointer_cast(node)) {
if (leaf->getKey() == key) {
return path; // Success
} else {
return std::nullopt; // Wrong leaf
}
}
// Navigate to next level
int branch = key.nthNibble(depth);
node = std::dynamic_pointer_cast(
node->getChild(branch));
if (!node) {
return std::nullopt; // Path doesn't exist
}
}
return std::nullopt; // Shouldn't reach here
}
```
**Example Proof:**
For an account with key 0x3A7F2E1B...:
```
Proof path (from root to leaf):
[0]: Inner node (root)
Hash: 0x1234... (all state)
Children: [3 -> 0xABCD..., 5 -> 0x5678..., ...]
[1]: Inner node (depth 1, branch 3)
Hash: 0xABCD... (state with branch 3)
Children: [A -> 0xEF12..., B -> 0x3456..., ...]
[2]: Inner node (depth 2, branch A)
Hash: 0xEF12... (state with branch 3, then A)
Children: [7 -> 0x7890..., ...]
[3]: Inner node (depth 3, branch 7)
Hash: 0x7890... (state with branch 3, A, 7)
Children: [F -> 0x8765..., ...]
[4]: Leaf node
Content: {Account, Data}
Hash: 0x8765... (account data)
Total: 5 nodes (including leaf)
Size: ~500 bytes for 5 serialized nodes
```
**Proof Size Properties:**
```
Tree size: 1 million accounts
Tree depth: log_16(1M) ≈ 5 levels
Proof includes:
5 inner nodes + 1 leaf: ~100 bytes per node
Total: ~600 bytes
Without proof:
Send all 1M accounts: gigabytes
Proof is logarithmic in tree size
Verification requires one hash per level: O(log N)
```
### Proof Verification
Verifying a proof requires only the root hash and the proof:
**Algorithm: verifyProofPath**
```cpp
bool verifyProofPath(
uint256 const& key,
uint256 const& expectedRootHash,
std::vector const& proof)
{
if (proof.empty()) {
return false;
}
// Start from leaf (last in proof)
auto leafNode = deserializeNode(proof.back(), /* leaf */);
if (!leafNode || leafNode->getKey() != key) {
return false;
}
// Compute leaf hash
uint256 computedHash = leafNode->computeHash();
// Walk from leaf toward root
for (int i = (int)proof.size() - 2; i >= 0; --i) {
auto innerNode = deserializeNode(proof[i], /* inner */);
if (!innerNode) {
return false;
}
// Determine which branch we came from
int depth = i;
int branch = key.nthNibble(depth);
// Verify the child hash matches
if (innerNode->getChildHash(branch) != computedHash) {
return false; // Proof is invalid
}
// Compute this node's hash for next iteration
computedHash = innerNode->computeHash();
}
// Verify final hash matches expected root
return (computedHash == expectedRootHash);
}
```
**Verification Process:**
```
Given:
- Key: 0x3A7F2E1B...
- Expected root: 0x1234...
- Proof: [inner_0, inner_1, inner_2, inner_3, leaf]
Step 1: Deserialize leaf, verify it contains key
Compute leaf hash: 0x8765...
Step 2: Deserialize inner_3
Check: child[F] hash == 0x8765... ✓
Compute inner_3 hash: 0x7890...
Step 3: Deserialize inner_2
Check: child[7] hash == 0x7890... ✓
Compute inner_2 hash: 0xEF12...
Step 4: Deserialize inner_1
Check: child[A] hash == 0xEF12... ✓
Compute inner_1 hash: 0xABCD...
Step 5: Deserialize inner_0 (root)
Check: child[3] hash == 0xABCD... ✓
Compute root hash: 0x1234...
Final check: Computed hash == expected? 0x1234 == 0x1234 ✓
Result: Proof is valid! Account is in ledger.
```
### Proof Use Cases
**1. Light Clients**
```
Scenario: Mobile wallet wanting to verify account balance
Traditional: Download entire ledger (~gigabytes)
With proofs: Download only account proof (~600 bytes)
Client verifies:
1. Receives account balance and proof
2. Verifies root hash matches known ledger
3. Uses proof verification algorithm
4. Trusts balance without downloading full ledger
```
**2. Cross-Chain Bridges**
```
Scenario: Ethereum bridge wants to verify XRPL state
Bridge requires: Proof that account exists in XRPL
Verification:
1. Receives XRPL ledger header (small)
2. Receives Merkle proof (small)
3. Verifies proof against root hash
4. Acts on verified state
```
**3. Auditing and Compliance**
```
Scenario: Auditor wants to prove a transaction executed
Proof includes:
- Transaction existence proof (proof in tx tree)
- Resulting state proof (proof in state tree)
- Chain of ledgers connecting them
Verifier can:
- Confirm transaction was processed
- Confirm state effects were applied
- Build complete chain of evidence
```
**4. Rollups and Scaling**
```
Scenario: Layer-2 rollup batches XRPL transactions
Instead of: All transactions + full ledger
Rollup uses: Transaction proofs + state root
Verification:
- Verify each transaction in batch
- Verify resulting state
- All with minimal data
```
### State Reconstruction Guarantee
The transaction tree provides a critical guarantee: **verifiable unique history**.
**The Problem Without Transaction Tree:**
```
State progression:
Initial state: S₀ (all accounts have 0 balance)
Could have arrived via many paths:
Path 1: S₀ --tx1--> S₁ --tx2--> S₂ --tx3--> S₃
Path 2: S₀ --tx4--> S₁' --tx5--> S₂' --tx3--> S₃
Path 3: S₀ --tx6--> S₁'' --tx1--> S₂'' --tx2--> S₃
Same final state S₃, but completely different history!
Which is correct? Without history, no way to know.
```
**The Solution: Transaction Tree**
The XRP Ledger maintains both trees in every ledger:
```cpp
struct LedgerHeader {
uint256 accountTreeHash; // Root of account state
uint256 transactionTreeHash; // Root of transaction history
// Both are cryptographically committed
// Both are signed by validators
// Both must match
};
```
**State Reconstruction:**
```
Given: LedgerHeader with both tree hashes
Verifiable state reconstruction:
1. Fetch account tree → current state
2. Fetch transaction tree → history
3. Replay transactions: S₀ + all_tx → S_computed
4. Verify: S_computed matches account tree root
5. Verify: transaction tree contains all_tx
Result: Proven that state is correct result of transactions
```
**Why This Matters:**
```
Without verification: Anyone could claim different history
With verification: Only one possible history is correct
Example:
Alice claims she sent 100 XRP to Bob
Bob claims he received only 50 XRP
Ledger history is immutable and verified
Proof shows exactly what happened
No ambiguity
```
### Ledger Verifiability
Every aspect of XRPL state is cryptographically verifiable:
**Account Proof**
```
Prove: Account has balance 1000 XRP
Components:
1. Ledger header with account state root
2. Merkle proof from root to account leaf
3. Account data (balance, etc.)
Verification:
Verify proof leads from root to account
Account balance is in proof
Root hash signed by supermajority of validators
```
**Transaction Proof**
```
Prove: Transaction T was executed in ledger L
Components:
1. Ledger L header with transaction tree root
2. Merkle proof from root to transaction leaf
3. Transaction data and execution results
Verification:
Verify proof leads from root to transaction
Matches all known transaction identifiers
Root hash signed by validators
```
**Full Ledger Proof**
```
Prove: Ledger state transitions from L1 to L2
Components:
1. L1 ledger header (initial state)
2. All transactions between L1 and L2
3. L2 ledger header (final state)
Verification:
1. Verify all transaction proofs against L2
2. Verify all account proofs against L2
3. Verify L2 header is valid
Result: Complete proof that L2 is correct result of applying
all transactions to L1
```
### Practical Verification Workflow
**Step 1: Trust the Ledger Header**
```cpp
// Ledger headers are small (~100 bytes)
// Signed by supermajority (>80%) of validators
// Distributed via gossip protocol
LedgerHeader verified_header = getLedgerHeader(ledgerSeq);
// Root hashes are in verified_header
```
**Step 2: Request Proof**
```cpp
// Client requests proof of account existence
MerkleProof proof = peer.requestProof(
accountID,
verified_header.accountStateRoot);
// Proof is small (~500 bytes)
```
**Step 3: Verify Proof**
```cpp
// Client verifies locally (no network needed)
if (verifyProofPath(accountID,
verified_header.accountStateRoot,
proof)) {
// Account exists and is proven
// Can trust all data in proof
auto account = parseAccountFromProof(proof);
}
```
**Step 4: Use Verified Data**
```cpp
// Application can now use the verified account data
// with absolute confidence it's correct
double balance = account.balance; // Proven correct
```
### Summary
**Key Concepts:**
1. **Merkle Proofs**: Logarithmic-sized proofs of inclusion
2. **Proof Verification**: O(log N) hash checks to verify
3. **Light Clients**: Verify without downloading full ledger
4. **Unique History**: Transaction tree guarantees verifiable history
5. **Ledger Integrity**: All state cryptographically provable
**Security Properties:**
```
Proof tampering: Detected by hash mismatch
Missing nodes: Detected by proof verification failure
Fork detection: Different ledger headers = detected forgery
State divergence: Account proof path breaks
All attacks: Detectable by cryptographic verification
```
**Performance:**
```
Traditional verification: All data (gigabytes) + full hashing
Proof-based verification: 600 bytes + 5 hash operations
Speedup: 10,000x less data, 100,000x less compute
```
This is the power of Merkle trees applied to blockchain: trustless verification at scale.
# Resource Management and Performance Characteristics
[← Back to SHAMap and NodeStore: Data Persistence and State Management](/core-dev-bootcamp/module03)
***
### Introduction
Understanding SHAMap and NodeStore theoretically is one thing. Operating them in production is another.
This final chapter covers:
* Real-world resource requirements
* Performance measurement and optimization
* Bottleneck identification
* Tuning for different deployment scenarios
### Lookup Complexity Analysis
**SHAMap Lookup**
```
Operation: Find account by ID
Worst case: O(64) node traversals
Tree depth: 256 bits / 4 bits per level = 64 levels
Typical case: O(1)
Most accounts found before depth 64
Average depth in realistic ledger: ~25 levels
Expected time:
Each traversal: O(1) array access (branch[0..15])
Total: O(1) expected time (linear in actual tree size,
but tree size ~ account count)
Cache hit: 1-10 microseconds
Direct pointer access, no I/O
Cache miss: 1-10 milliseconds
Database query required
```
**Batch Fetch**
```
N objects requested:
Naive (sequential):
N × database_latency = N × 10ms
Example: 100 objects = 1000ms
Batched:
single_batch_latency + deserialize
Example: 100 objects = 10ms + 5ms = 15ms
Speedup: 66x
```
### Write Throughput
**Ledger Close Cycle**
```
Timeline for typical ledger close (3-5 seconds):
1. Receive transactions: 2 seconds
- Validate signatures
- Check preconditions
- Execute in SHAMap
2. Consensus: 1 second
- Reach agreement on state
- Sign ledger
3. Store phase: 0.5-1 second
- Serialize modified nodes
- Write to NodeStore
- Update indexes
Total: 3.5-5 seconds
```
**Node Object Volume**
```
Typical ledger modification:
200-400 transactions per ledger
Average 2-4 modified accounts per transaction
= 500-1000 modified nodes
Plus structural nodes (parent rehashing):
Depth of modified accounts: ~25 levels
= 25 ancestor nodes modified
Total objects created: ~600-1100 per ledger
At 4 ledgers/second:
2400-4400 objects/second
Database requirement:
RocksDB: Handles 10,000-50,000 obj/sec easily
NuDB: Handles 50,000-200,000 obj/sec
```
**Write Latency**
```
Store a NodeObject:
1. Cache update: 1-10 microseconds
2. Encode to blob: 100 microseconds
3. Database write: 100-1000 microseconds (SSD)
4. Batch accumulation: 10-100 milliseconds
Total for batch of 100: 10-100 milliseconds
Per object in batch: 0.1-1 millisecond
```
### Read Performance Characteristics
**Cache Hit Scenario**
```
Hit rate: 95% (well-tuned system)
1000 object requests:
950 cache hits × 5 microseconds = 4.75 milliseconds
50 cache misses × 10 milliseconds = 500 milliseconds
Total: 504.75 milliseconds = 0.5 seconds
Average per request: 0.5 milliseconds
```
**Cache Miss Scenario**
```
Hit rate: 60% (poorly tuned system)
1000 object requests:
600 cache hits × 5 microseconds = 3 milliseconds
400 cache misses × 10 milliseconds = 4000 milliseconds = 4 seconds
Total: 4.003 seconds
Average per request: 4 milliseconds
10x slower due to cache misses!
```
### Memory Requirements
**NodeStore Memory**
```
Cache layer:
Size: Configurable (32MB - 4GB typical)
Per object: ~100-500 bytes
At 256MB cache: ~300,000-500,000 cached objects
Backend buffers:
RocksDB: ~100-300MB for block cache
NuDB: ~50-100MB
Thread pools:
Each async thread: ~1-2MB stack
10 threads: ~20MB
Total NodeStore memory: cache_size + backend_buffers + thread_stacks
Typical: 256MB cache + 200MB backend = 500MB total
Large: 1GB cache + 300MB backend = 1.3GB total
```
**SHAMap Memory**
```
In-memory tree of current + recent ledgers:
Active ledger: ~10-50MB
Depends on account count and modification volume
Recent immutable ledgers (kept for quick access):
2-3 most recent: ~30-150MB
Total SHAMap: 50-200MB typical
Plus cached nodes (shared with NodeStore cache):
Counted above in NodeStore memory
```
**Total Memory Budget**
```
Minimal validator:
SHAMap: 50MB
NodeStore: 200MB
Other rippled: 100MB
Total: 350MB
Standard validator:
SHAMap: 100MB
NodeStore: 500MB
Other rippled: 100MB
Total: 700MB
Large validator:
SHAMap: 200MB
NodeStore: 2000MB
Other rippled: 100MB
Total: 2.3GB
```
### Disk Space Requirements
**Database Growth**
```
Without rotation: Unbounded growth
Per ledger:
~600-1100 new objects per ledger
~200KB per object (with compression)
= 120-220MB per ledger
Per day:
~20 ledgers per day
= 2.4-4.4 GB per day
Per year:
= 876GB - 1.6TB per year
Clearly unsustainable (disk fills in weeks)
```
**With Rotation**
```
Retention policy: Keep last 100,000 ledgers
Ledger creation rate: 1 ledger per ~3 seconds
100,000 ledgers = ~8 days of history
Database size:
100,000 × 0.2MB = 20GB (stable)
With overhead: 30-50GB typical
Bounded growth enables indefinite operation
```
**Actual Sizes on Mainnet**
```
Small validator (RocksDB, compressed):
Database: 30-50GB
With binaries/logs: 60GB total
Archive node (full history):
Database: 500GB-1TB
With redundancy: 1.5TB total
Growth per day (with rotation):
~500MB-1GB per day
(old data deleted as new data added)
```
### File Descriptor Usage
**File Descriptor Requirements**
```
Each backend type requires different FDs:
RocksDB:
- Main database: 1
- WAL (write-ahead log): 1
- SSTable files: 20-100 (per configuration)
- Total: 25-100 FDs
NuDB:
- Main data file: 1
- Index file: 1
- Total: 2-5 FDs
Operating system overhead:
stdin, stdout, stderr: 3
Socket listening: 2-5
Network connections: ~50 typical
Total rippled process:
- Without NodeStore: 50-100 FDs
- With RocksDB: 100-200 FDs
- Comfortable limit: 4096 FDs
Configuration:
ulimit -n 4096 # Set FD limit
```
### Performance Tuning
**Identifying Bottlenecks**
Monitor these metrics:
```cpp
// Cache hit rate - most important
if (metrics.hitRate() < 90%) {
// Increase cache_size
problem = "Cache too small";
}
// Write latency - latency-sensitive
if (metrics.writeLatency > 100ms) {
// Switch to faster backend or increase batch size
problem = "Backend I/O too slow";
}
// Fetch latency
if (metrics.fetchLatency > 50ms) {
// Check cache hit rate
// Check disk health
problem = "Database queries too slow";
}
// Async queue depth
if (metrics.asyncQueueDepth > 10000) {
// Not keeping up with demand
problem = "Async processing overwhelmed";
}
```
**Tuning Parameters**
```ini
[node_db]
type = RocksDB
path = /var/lib/rippled/db
# Cache tuning
cache_size = 256 # Increase if memory available
cache_age = 60 # Longer = better hit rate
# Threading
async_threads = 4 # Increase for I/O-bound systems
# Batch operations
batch_write_size = 256 # Larger batches, fewer transactions
[node_db_rotation]
online_delete = 256 # Keep 256K ledgers (8 days)
```
**Scenario 1: High-Traffic Validator**
```
Problem: Write latency too high (ledgers close slowly)
Solution:
- Increase cache_size to 1GB+
- Switch to NuDB backend (higher throughput)
- Increase async_threads to 8-16
- Ensure SSD (not HDD)
- Increase batch_write_size
Result: Write throughput 50K+ objects/sec
```
**Scenario 2: Memory-Constrained**
```
Problem: Only 512MB RAM available
Solution:
- Set cache_size = 64MB (small)
- Still runs, but slower
- Increase cache_age for working set
- Monitor hit rate (may drop to 80%)
Result: Functional but slower sync and queries
```
**Scenario 3: Archive Node**
```
Problem: Need complete history, very large disk
Solution:
- No rotation (online_delete disabled)
- RocksDB with compression
- Smaller cache_size (less frequently accessed)
- Parallel database with rotated copy
Result: Full history, terabyte+ database
```
### Performance Characteristics Summary
**Lookup Performance:**
```
Single object lookup:
Cache hit: 1-10 microseconds
Cache miss: 1-10 milliseconds
95% hit rate: ~0.5 milliseconds average
Batch operation (100 objects):
Sequential: 1000 milliseconds
Batched: 10 milliseconds
Speedup: 100x
```
**Write Performance:**
```
Per ledger:
1000 objects per ledger
Per-object: 0.1-1 millisecond
Batch overhead: 10-100 milliseconds
Total per ledger: 100-1100 milliseconds
Throughput:
4 ledgers/second × 1000 objects/ledger = 4000 obj/sec
Well within RocksDB/NuDB capacity
```
**Memory Usage:**
```
Minimum: 200-300MB
Typical: 500-700MB
Large: 2-4GB
Depends on cache_size configuration
```
**Disk Space:**
```
With rotation: 30-50GB (8-10 days history)
Unbounded: ~1TB per year (without rotation)
Growth rate: ~500MB-1GB per day
```
**Scalability Limits:**
```
Current network:
2000+ validators
100-400 transactions/ledger
Proven sustainable
Theoretical limits:
Cache hit rate: 80%+ maintainable at any size
Write throughput: 100K obj/sec possible
Read throughput: 1M obj/sec with cache
Practical limits:
Memory: 4-16GB per validator typical
Disk: 100GB-1TB per validator typical
Network: Synchronization limits transaction volume
Consensus: Agreement time limits throughput
```
### Monitoring in Production
**Key Metrics to Track**
```
1. Cache Statistics:
- Hit rate (target: >90%)
- Size (should be close to configured max)
- Eviction rate
2. Database Performance:
- Write latency (target: <100ms per ledger)
- Read latency (target: <50ms per request)
- Queue depth (target: <1000)
3. Resource Usage:
- Memory (should stabilize)
- CPU (typically 20-50% on modern systems)
- Disk I/O (peaks during sync)
4. Application:
- Ledger close time (target: 3-5 seconds)
- Synchronization lag (target: 0 when caught up)
- Block proposal success (target: >95%)
```
**Alerting Thresholds**
```
Warning:
- Hit rate < 80%
- Write latency > 200ms
- Queue depth > 5000
Critical:
- Hit rate < 60%
- Write latency > 500ms
- Ledger close > 10 seconds
- Disk space < 10GB free
```
# Appendices
# Appendix : Codebase Navigation Guide
[← Back to SHAMap and NodeStore: Data Persistence and State Management](/core-dev-bootcamp/module03)
***
### Introduction
This appendix helps you navigate the rippled codebase to find SHAMap and NodeStore implementations.
## Directory Structure
### SHAMap Source Files
```
rippled/src/xrpld/shamap/
├── SHAMap.h # Main SHAMap class
├── SHAMapInnerNode.h # Inner node implementation
├── SHAMapLeafNode.h # Leaf node implementations
├── SHAMapNodeID.h # Node identification
├── SHAMapHash.h # Hash computation
├── SHAMapMissingNode.h # Missing node tracking
├── detail/
│ ├── SHAMap.cpp # Core algorithms
│ ├── SHAMapSync.cpp # Synchronization logic
│ └── SHAMapDelta.cpp # Tree traversal
```
### NodeStore Source Files
```
rippled/src/xrpld/nodestore/
├── Database.h # Database interface
├── Backend.h # Backend interface
├── Manager.h # NodeStore manager
├── Types.h # Type definitions
├── Task.h # Async task definitions
├── Scheduler.h # Scheduling
├── backend/
│ ├── RocksDBFactory.cpp # RocksDB backend
│ ├── NuDBFactory.cpp # NuDB backend
│ ├── MemoryFactory.cpp # In-memory backend
│ └── NullFactory.cpp # No-op backend
├── detail/
│ ├── Database.cpp # Database implementation
│ ├── ManagerImp.cpp # Manager implementation
│ ├── BatchWriter.h/cpp # Batch write handling
│ ├── DatabaseNodeImp.h # Single-backend implementation
│ ├── DatabaseRotatingImp.h # Rotating-database implementation
│ ├── NodeObject.cpp # NodeObject implementation
│ ├── EncodedBlob.h # Encoding/decoding
│ ├── DecodedBlob.h # Blob structure
│ └── varint.h # Variable-length integers
```
### Integration Points
```
rippled/src/xrpld/app/
├── misc/SHAMapStore.h # SHAMap-NodeStore integration
├── misc/SHAMapStoreImp.h/cpp # Implementation
└── main/NodeStoreScheduler.h/cpp # Background scheduling
```
## Key Classes and Their Locations
### SHAMap Classes
| Class | File | Purpose |
| ----------------- | -------------------------- | ---------------------- |
| `SHAMap` | `shamap/SHAMap.h` | Main tree class |
| `SHAMapTreeNode` | `shamap/SHAMapTreeNode.h` | Base node class |
| `SHAMapInnerNode` | `shamap/SHAMapInnerNode.h` | Inner nodes (branches) |
| `SHAMapLeafNode` | `shamap/SHAMapLeafNode.h` | Leaf nodes (data) |
| `SHAMapNodeID` | `shamap/SHAMapNodeID.h` | Node identification |
| `SHAMapItem` | `shamap/SHAMapItem.h` | Data in leaf nodes |
### NodeStore Classes
| Class | File | Purpose |
| --------------------- | ---------------------------------------- | -------------------- |
| `Database` | `nodestore/Database.h` | High-level interface |
| `Backend` | `nodestore/Backend.h` | Low-level interface |
| `NodeObject` | `nodestore/detail/NodeObject.h` | Storage unit |
| `DatabaseNodeImp` | `nodestore/detail/DatabaseNodeImp.h` | Single backend |
| `DatabaseRotatingImp` | `nodestore/detail/DatabaseRotatingImp.h` | Rotating backend |
| `TaggedCache` | `nodestore/detail/TaggedCache.h` | Cache implementation |
## Navigation by Task
### Understanding SHAMap Structure
1. Start: `SHAMap.h` - Overview of the class
2. Read: `SHAMapTreeNode.h` - Base class and type system
3. Explore: `SHAMapInnerNode.h` - Branch structure
4. Explore: `SHAMapLeafNode.h` - Data storage
5. Study: `detail/SHAMap.cpp` - Implementation details
### Understanding Node Synchronization
1. Start: `shamap/SHAMapMissingNode.h` - Missing node representation
2. Study: `detail/SHAMapSync.cpp` - Synchronization algorithm
3. Cross-reference: `nodestore/Database.h` - Fetch operations called during sync
4. Understand: `shamap/SHAMapNodeID.h` - How nodes are identified
### Understanding NodeStore Architecture
1. Start: `nodestore/Database.h` - Public interface
2. Understand: `nodestore/Backend.h` - Storage abstraction
3. Explore: `nodestore/detail/NodeObject.h` - Storage unit
4. Study: `nodestore/detail/DatabaseNodeImp.h` - Standard implementation
5. Study: `nodestore/detail/DatabaseRotatingImp.h` - Rotation implementation
### Understanding Database Rotation
1. Read: `nodestore/detail/DatabaseRotatingImp.h` - Architecture
2. Study: `nodestore/detail/DatabaseRotatingImp.cpp` - Implementation
3. Understand: Synchronization with `app/misc/SHAMapStoreImp.h`
### Understanding Cache Layer
1. Explore: `nodestore/detail/TaggedCache.h` - Cache implementation
2. Study usage in: `nodestore/detail/DatabaseNodeImp.cpp`
3. Understand metrics in: Database metrics methods
## Common Code Patterns
### Accessing a Node in SHAMap
```cpp
// In shamap/SHAMap.h or detail/SHAMap.cpp
std::shared_ptr node = getNodePointer(nodeID);
```
### Storing a Node in NodeStore
```cpp
// In app/misc/SHAMapStoreImp.cpp
auto obj = NodeObject::createObject(type, data, hash);
mNodeStore->store(obj);
```
### Retrieving a Node
```cpp
// In nodestore/detail/DatabaseNodeImp.cpp
auto obj = mDatabase->fetchNodeObject(hash);
```
### Cache Hit Path
```cpp
// In nodestore/detail/DatabaseNodeImp.cpp
auto cached = mCache.get(hash); // L1 cache
if (!cached) {
cached = mBackend->fetch(hash); // L2 backend
if (cached) mCache.insert(hash, cached);
}
```
## Important Files to Read
### Essential (Required Reading)
* `shamap/SHAMap.h` - Core API
* `shamap/SHAMapNodeID.h` - Navigation understanding
* `nodestore/Database.h` - NodeStore API
* `nodestore/Backend.h` - Abstraction principle
### Important (Strongly Recommended)
* `shamap/detail/SHAMap.cpp` - Implementation
* `nodestore/detail/DatabaseNodeImp.h` - Cache logic
* `app/misc/SHAMapStoreImp.h` - Integration
### Reference (For Deep Understanding)
* `shamap/SHAMapInnerNode.h` - Branch structure details
* `shamap/SHAMapLeafNode.h` - Leaf implementations
* `nodestore/detail/SHAMapSync.cpp` - Sync algorithm
* `nodestore/detail/DatabaseRotatingImp.h` - Rotation details
## Building and Exploring
### Compile rippled
```bash
cd rippled
mkdir build && cd build
cmake ..
make -j4
```
### Navigate with IDE
Using VS Code or similar:
1. Open rippled repository
2. Go to Definition (Ctrl+Click) to jump to class definitions
3. Find All References (Shift+Ctrl+F) to see usage patterns
4. Use search to navigate between related classes
### Debug and Trace
```cpp
// Add to your code to trace execution
#include
std::cout << "Node hash: " << node->getHash() << std::endl;
std::cout << "Node type: " << (int)node->getNodeType() << std::endl;
```
### Test Files
Locate tests in:
```
rippled/src/test/*/shamap* and */nodestore*
```
Study how these are tested to understand expected usage patterns.
## Configuration Reference
### Configuration Files
NodeStore configuration in `rippled.cfg`:
```
[node_db]
type = RocksDB # Backend choice
path = /data/node.db # Location
cache_size = 256 # Cache size in MB
cache_age = 60 # Max age in seconds
# For NuDB
# type = NuDB
# path = /data/nudb
# For Rotating
# online_delete = 256 # Keep last N ledgers
```
### Common Configuration Patterns
```
# Small validator (less memory available)
[node_db]
cache_size = 64
cache_age = 30
# Large validator (plenty of resources)
[node_db]
cache_size = 512
cache_age = 120
```
## Quick Reference
| Task | File | Function |
| ----------------- | ---------------------------------------- | ------------------- |
| Create NodeObject | `nodestore/detail/NodeObject.h` | `createObject()` |
| Fetch Node | `nodestore/Database.h` | `fetchNodeObject()` |
| Store Node | `nodestore/Database.h` | `store()` |
| Find in SHAMap | `shamap/SHAMap.h` | `findLeaf()` |
| Add to SHAMap | `shamap/detail/SHAMap.cpp` | `addNode()` |
| Check Cache | `nodestore/detail/TaggedCache.h` | `get()` |
| Rotate Database | `nodestore/detail/DatabaseRotatingImp.h` | `rotate()` |
| Get Metrics | `nodestore/Database.h` | `getCountsJson()` |
## Learning Path
1. **Week 1**: Understand architecture (Chapters 1-4 of this module)
2. **Week 2**: Read `SHAMap.h` and understand node types
3. **Week 3**: Study `shamap/detail/SHAMap.cpp` - implementation
4. **Week 4**: Read `nodestore/Database.h` and understand NodeStore design
5. **Week 5**: Study `nodestore/detail/DatabaseNodeImp.h` - caching
6. **Week 6**: Trace actual code execution (single transaction flow)
7. **Week 7**: Build and run tests
8. **Week 8**: Modify and experiment
This progression takes you from conceptual understanding to implementation mastery.
# Appendix : Configuration Reference
[← Back to SHAMap and NodeStore: Data Persistence and State Management](/core-dev-bootcamp/module03)
***
### Introduction
Complete reference for NodeStore and SHAMap configuration options in `rippled.cfg`.
## NodeStore Configuration
### `[node_db]` Section
Core database configuration:
```ini
[node_db]
type = RocksDB # Type: RocksDB, NuDB, SQLite, Memory
path = /var/lib/rippled/db # Database location
cache_size = 256 # Cache size in MB (32-4096 typical)
cache_age = 60 # Cache entry age limit in seconds
```
### Backend Selection
**RocksDB** (Recommended)
```ini
[node_db]
type = RocksDB
path = /var/lib/rippled/db/rocksdb
# RocksDB specific options
compression = true # Enable compression (reduces disk ~50%)
block_cache_size = 256 # Block cache in MB
write_buffer_size = 64 # Write buffer in MB
max_open_files = 100 # Max concurrent file handles
```
**NuDB** (High-Throughput)
```ini
[node_db]
type = NuDB
path = /var/lib/rippled/db/nudb
# NuDB specific options
key_size = 32 # SHA256 key size (always 32)
block_size = 4096 # Block size for writes
```
**SQLite** (Legacy)
```ini
[node_db]
type = SQLite
path = /var/lib/rippled/db/rippled.db
```
**In-Memory** (Testing)
```ini
[node_db]
type = Memory
# No path needed
```
### Cache Configuration
```ini
[node_db]
cache_size = 256 # Size in MB
# Tuning guide:
# Small (32MB): Minimal memory, slower
# Standard (256MB): Good for most validators
# Large (1GB): Better sync performance
# Very Large (4GB): Archive nodes
```
**Cache Age:**
```ini
cache_age = 60 # Seconds before eviction
# Tuning guide:
# Short (30s): Low memory, frequent eviction
# Standard (60s): Good balance
# Long (300s): More memory, longer lifespan
```
### Threading Configuration
```ini
[node_db]
async_threads = 4 # Background fetch threads
# Tuning guide:
# Few (2-4): Lower CPU, simpler
# Standard (4-8): Balance CPU and throughput
# Many (16-32): High-throughput systems
```
### Batch Operation Configuration
```ini
[node_db]
batch_write_size = 256 # Objects per batch
# Note: Most systems don't need to adjust this
# Default of 256 is well-optimized
```
## Online Deletion (Database Rotation)
Enable automatic deletion of old ledgers:
```ini
[node_db_rotation]
online_delete = 256 # Keep last N thousand ledgers
# Ledger counts:
# 8 (1 day): 3-5 seconds per ledger
# 256 (8 days): Common for validators
# 1000 (30 days): Archive-ish
```
**Without Rotation:**
```ini
# Don't set online_delete section
# Database grows unbounded (~1-2 GB per day)
# Eventually disk fills
# Requires manual pruning
```
## Import/Export Configuration
**Import from another database:**
```ini
[node_db]
type = RocksDB
path = /var/lib/rippled/db/new_rocksdb
import_db = /path/to/old/database # Source database
# During startup, rippled will:
# 1. Open source database
# 2. Read all objects
# 3. Write to destination
# 4. Verify counts match
# 5. Continue with destination as primary
```
## Historical Database Configuration
Keep complete history without deletion:
```ini
[node_db]
type = RocksDB
path = /var/lib/rippled/db/rocksdb
# Don't enable online_delete
# Don't set import_db
# Result: Complete ledger history preserved
# Disk grows ~1GB per day initially
# ~500GB - 1TB for ~2 years mainnet
```
## Tuning by Deployment Type
### Small Validator
```ini
[node_db]
type = RocksDB
path = /var/lib/rippled/db/rocksdb
cache_size = 128 # Limited memory
cache_age = 30 # Short lifespan
async_threads = 2 # Few threads
[node_db_rotation]
online_delete = 256 # Keep 8 days
```
**Expected:**
* Memory: \~300MB
* Disk: \~50GB
* CPU: 20-40%
### Standard Validator
```ini
[node_db]
type = RocksDB
path = /var/lib/rippled/db/rocksdb
cache_size = 256 # Standard size
cache_age = 60 # Standard lifespan
async_threads = 4 # Normal concurrency
compression = true # Enable compression
[node_db_rotation]
online_delete = 256 # Keep 8 days
```
**Expected:**
* Memory: \~500-700MB
* Disk: \~50GB (with rotation)
* CPU: 30-50%
### High-Performance Validator
```ini
[node_db]
type = NuDB # Higher throughput
path = /var/lib/rippled/db/nudb
cache_size = 1024 # Large cache
cache_age = 120 # Longer lifespan
async_threads = 8 # More parallelism
batch_write_size = 512 # Larger batches
[node_db_rotation]
online_delete = 512 # Keep 15 days
```
**Expected:**
* Memory: \~1.5GB
* Disk: \~80GB (with rotation)
* CPU: 40-60%
### Archive Node
```ini
[node_db]
type = RocksDB
path = /var/lib/rippled/db/rocksdb
cache_size = 512 # Medium cache
cache_age = 300 # Long lifespan
async_threads = 16 # Many threads
compression = true # Important for space
# No online_delete section - keep all history
```
**Expected:**
* Memory: \~1GB
* Disk: \~500GB - 1TB
* CPU: 50-70%
## Monitoring Configuration
### Logging Levels
```ini
[logging]
debug # Verbose logging
# or
info # Standard logging
# or
warning # Only warnings and errors
```
### RPC Configuration for Monitoring
```ini
[rpc_startup]
command = log_level
severity = debug
# Get metrics via RPC
# rippled-cli server_info | jq '.result.node_db'
```
## Performance Impact Summary
| Parameter | Lower Value | Higher Value |
| -------------- | --------------------------- | -------------------------------- |
| cache\_size | Less memory, slower | More memory, faster |
| cache\_age | Quick eviction, less memory | Slow eviction, more memory |
| async\_threads | Less CPU, slower I/O | More CPU, faster I/O |
| compression | Faster disk I/O | Slower disk I/O, less disk space |
| online\_delete | More disk space needed | Smaller database, bounded growth |
## Configuration Validation
Check configuration syntax:
```bash
# Validate config file
rippled --validate-cfg
# Expected output:
# Config appears to be valid
```
## Changing Configuration
### Changing Cache Size
```bash
# Edit rippled.cfg
nano rippled.cfg
# Change cache_size value
# Restart rippled
systemctl stop rippled
systemctl start rippled
# Cache takes effect immediately
```
### Changing Backend
```bash
# This requires data migration
# 1. Stop rippled
systemctl stop rippled
# 2. Export current database
rippled --export current_db export.json
# 3. Update config with new backend
nano rippled.cfg # Change type = XXX
# 4. Import to new backend
mkdir -p /var/lib/rippled/db/new_backend
rippled --import export.json --ledger-db new_backend
# 5. Backup old database
mv /var/lib/rippled/db/old_backend \
/var/lib/rippled/db/old_backend.backup
# 6. Restart with new database
systemctl start rippled
# 7. Verify it works
rippled-cli server_info | jq '.result.node_db.type'
```
### Enabling Online Deletion
```bash
# Add to rippled.cfg
[node_db_rotation]
online_delete = 256
# Restart rippled
systemctl restart rippled
# Monitor deletion (may take time)
tail -f /var/log/rippled/rippled.log | grep -i delete
```
## Troubleshooting Configuration Issues
### Issue: Cache hits too low
**Possible causes:**
* `cache_size` too small
* `cache_age` too short
* High variance in access patterns
**Check:**
```bash
rippled-cli server_info | jq '.result.node_db.cache_hit_rate'
```
**Fix:**
* Increase `cache_size` by 50%
* Increase `cache_age` to 120
* Verify available memory
### Issue: Database corrupted
**Possible causes:**
* Unclean shutdown
* Disk failure
* Backend corruption
**Recovery:**
```bash
# Stop rippled
systemctl stop rippled
# Backup corrupted database
mv /var/lib/rippled/db /var/lib/rippled/db.corrupt
# Restart (will resync from network)
systemctl start rippled
# Check progress
rippled-cli server_info | jq '.result.ledger.ledger_index'
```
### Issue: Disk space filling
**Check:**
```bash
df -h /var/lib/rippled/
# If near full:
du -sh /var/lib/rippled/db/*
```
**Solution:**
* If rotation enabled: wait for deletion to complete
* If rotation disabled: enable it with `online_delete = 256`
* Monitor with `watch -n 1 'du -sh /var/lib/rippled/db'`
### Issue: Poor write performance
**Check:**
```bash
iostat -x 1 /dev/sda # Check I/O wait
iotop -o # Check top I/O processes
```
**Solutions:**
* Use SSD (not HDD)
* Enable compression: `compression = true`
* Switch to NuDB: `type = NuDB`
* Increase `async_threads`
## Quick Configuration Checklist
```ini
[node_db]
type = RocksDB # ✓ Choose backend
path = /var/lib/rippled/db # ✓ Choose location
cache_size = 256 # ✓ Tune for hardware
cache_age = 60 # ✓ Default is good
async_threads = 4 # ✓ Default is good
compression = true # ✓ Enable (if RocksDB)
[node_db_rotation]
online_delete = 256 # ✓ Prevent unbounded growth
```
This configuration is production-ready for most validators.
## Performance Monitoring Script
```bash
#!/bin/bash
# Monitor NodeStore health
while true; do
clear
echo "=== NodeStore Health Check ==="
rippled-cli server_info | jq '{
"cache_hit_rate": .result.node_db.cache_hit_rate,
"cache_size_mb": .result.node_db.cache_size,
"write_latency_us": .result.node_db.write_latency_us,
"read_latency_us": .result.node_db.read_latency_us,
"async_queue_depth": .result.node_db.async_queue_depth
}'
echo ""
echo "=== Disk Usage ==="
du -sh /var/lib/rippled/db/*
sleep 5
done
```
***
For more details, see:
* Chapter 6: NodeStore Architecture
* Chapter 8: Cache Layer
* Appendix B: Debugging
# Appendix : Debugging and Development Tools
[← Back to SHAMap and NodeStore: Data Persistence and State Management](/core-dev-bootcamp/module03)
***
### Introduction
This appendix provides techniques and tools for investigating SHAMap and NodeStore behavior.
## Logging and Diagnostics
### Enable Verbose Logging
Edit `rippled.cfg`:
```ini
[rpc_startup]
command = log_level
severity = debug
[logging]
debug
rpc
```
Then restart rippled and check logs:
```bash
tail -f /var/log/rippled/rippled.log | grep -i nodestore
```
### Key Log Messages
```
TRACE Ledger: Ledger opened/closed
DEBUG SHAMap: Tree operations
DEBUG NodeStore: Database operations
INFO Consensus: Validation and agreement
WARN Performance: Slow operations detected
```
## Metrics Inspection
### JSON-RPC Inspection
```bash
# Get storage metrics
rippled-cli server_info | jq '.result.node_db'
# Expected output:
{
"type": "RocksDB",
"path": "/var/lib/rippled/db/rocksdb",
"cache_size": 256,
"cache_hit_rate": 0.923,
"writes": 1000000,
"bytes_written": 1000000000,
"reads": 50000000,
"cache_hits": 46150000,
"read_latency_us": 15
}
```
### File System Inspection
```bash
# Check database size
du -sh /var/lib/rippled/db/*
# Monitor growth
watch -n 1 'du -sh /var/lib/rippled/db/*'
# Check free space
df -h /var/lib/rippled/
# Monitor I/O
iostat -x 1 /dev/sda
```
## Debugging Specific Issues
### Issue: Cache Hit Rate Too Low
**Symptoms:**
* Database queries slow
* Ledger close times increasing
* Hit rate < 80%
**Investigation:**
```bash
# Check cache metrics
rippled-cli server_info | jq '.result.node_db.cache_hit_rate'
# Check cache size configuration
grep cache_size rippled.cfg
# Monitor cache evictions
tail -f /var/log/rippled/rippled.log | \
grep -i "evict\|cache"
```
**Solutions:**
1. Increase `cache_size` if memory available
2. Reduce `cache_age` for faster eviction of cold data
3. Check if system is memory-constrained (use `free`)
### Issue: Write Performance Degradation
**Symptoms:**
* Ledger closes slow (>10 seconds)
* Database write errors in logs
* Validator falling behind network
**Investigation:**
```bash
# Check write latency
rippled-cli server_info | \
jq '.result.node_db.write_latency_us'
# Monitor disk I/O
iotop -o -b -n 1
# Check disk space
df -h
# Monitor async queue
tail /var/log/rippled/rippled.log | \
grep -i "async.*queue"
```
**Solutions:**
1. Ensure SSD (not HDD) for database
2. Check disk I/O isn't saturated
3. Increase `async_threads` if I/O bound
4. Switch to faster backend (NuDB vs RocksDB)
5. Enable compression if disk is bottleneck
### Issue: Synchronization Slow
**Symptoms:**
* New nodes take hours to sync
* Falling behind network
* High database query count
**Investigation:**
```bash
# Monitor sync progress
rippled-cli server_info | jq '.result.ledger.ledger_index'
# Track fetch operations
tail -f /var/log/rippled/rippled.log | \
grep -i "fetch\|sync"
# Monitor thread pool
ps -p $(pidof rippled) -L
# Check queue depths
rippled-cli server_info | jq '.result.node_db.async_queue_depth'
```
**Solutions:**
1. Increase cache size for better hit rate during sync
2. Increase `async_threads` (more parallel fetches)
3. Use faster SSD
4. Check network bandwidth (might be bottleneck)
5. Switch to NuDB for higher throughput
## Code Debugging
### Building with Debug Symbols
```bash
cd rippled
mkdir build
cd build
cmake -DCMAKE_BUILD_TYPE=Debug ..
make -j4
```
### GDB Debugging
```bash
# Run under GDB
gdb --args rippled --conf /path/to/rippled.cfg
# Inside GDB:
(gdb) run
(gdb) break SHAMap::addKnownNode
(gdb) continue
# When breakpoint hit:
(gdb) print node->getHash()
(gdb) print nodeID
(gdb) step
(gdb) quit
```
### Common Breakpoints
```cpp
// Node addition
break SHAMap::addKnownNode
break Database::store
// Cache operations
break TaggedCache::get
break TaggedCache::insert
// Synchronization
break SHAMap::getMissingNodes
break NodeStore::fetchNodeObject
```
### Print Useful Data
```cpp
(gdb) print node->getHash().hex()
(gdb) print nodeID.mDepth
(gdb) print nodeID.mNodeID.hex()
(gdb) print metrics.cacheHits
(gdb) print metrics.cacheMisses
```
## Performance Profiling
### CPU Profiling with Perf
```bash
# Record 60 seconds of system behavior
perf record -F 99 -p $(pidof rippled) -- sleep 60
# Analyze results
perf report
# Show flame graph
perf record -F 99 -p $(pidof rippled) -- sleep 60
perf script | stackcollapse-perf.pl | flamegraph.pl > profile.svg
```
### Memory Profiling with Valgrind
```bash
# Run under memcheck (very slow)
valgrind --leak-check=full rippled --conf rippled.cfg
# Run specific test
valgrind --leak-check=full rippled --unittest test.nodestore
```
### Custom Instrumentation
Add to rippled source:
```cpp
// In Database::fetchNodeObject
auto startTime = std::chrono::steady_clock::now();
auto obj = mBackend->fetch(hash);
auto elapsed = std::chrono::steady_clock::now() - startTime;
auto ms = std::chrono::duration_cast
