Debugging Tools: Development and Debugging Techniques

← Back to Rippled II Overview


Introduction

Debugging is an essential skill for any Rippled developer. Whether you're tracking down a subtle transaction processing bug, investigating network connectivity issues, or optimizing performance, having a solid understanding of debugging tools and techniques is critical. The complexity of distributed systems like the XRP Ledger means that effective debugging often requires multiple approaches—from analyzing logs to using debuggers to running controlled experiments.

This deep dive covers the comprehensive toolkit available for debugging Rippled, from basic logging to advanced profiling techniques. You'll learn how to diagnose issues quickly, understand system behavior, and develop confidence in working with the Rippled codebase.


Logging System

Overview

Rippled includes a sophisticated logging system that provides detailed visibility into system behavior. Understanding how to configure and use logging effectively is the foundation of debugging Rippled.

Log Structure

Partitions: Logs are organized by subsystem (partition) Severity Levels: Each log entry has a severity level Timestamps: All logs include precise timestamps Context: Logs include relevant context (account IDs, ledger numbers, etc.)

Severity Levels

From most to least verbose:

trace   - Extremely detailed, every function call
debug   - Detailed debugging information
info    - General informational messages
warning - Warning conditions
error   - Error conditions
fatal   - Fatal errors that cause termination

Usage Guidelines:

  • Production: Use warning or error to minimize disk I/O

  • Development: Use debug or trace for active debugging

  • Investigation: Temporarily enable trace for specific partitions

Log Partitions

Major subsystems have their own partitions:

Ledger              - Ledger operations
LedgerMaster        - Ledger master coordination
Transaction         - Transaction processing
Consensus           - Consensus rounds
Overlay             - P2P networking
Peer                - Individual peer connections
Protocol            - Protocol message handling
RPC                 - RPC request handling
JobQueue            - Job queue operations
NodeObject          - NodeStore operations
Application         - Application lifecycle
OrderBookDB         - Order book database
PathRequest         - Path finding
ValidatorList       - Validator list management
Amendments          - Amendment processing

Configuring Logging

In Configuration File

Edit rippled.cfg:

[rpc_startup]
{ "command": "log_level", "severity": "warning" }
{ "command": "log_level", "partition": "Transaction", "severity": "trace" }
{ "command": "log_level", "partition": "Consensus", "severity": "debug" }

Via RPC Command

Dynamic adjustment without restart:

# Set all partitions to warning
rippled log_level warning

# Set specific partition to trace
rippled log_level Transaction trace

# Set multiple partitions
rippled log_level Consensus debug
rippled log_level Overlay debug
rippled log_level Peer trace

Programmatically

In code:

// Get logger for this partition
beast::Journal j = app_.journal("MyComponent");

// Log at different levels
JLOG(j.trace()) << "Entering function with param: " << param;
JLOG(j.debug()) << "Processing transaction: " << tx.getTransactionID();
JLOG(j.info()) << "Ledger closed: " << ledger.seq();
JLOG(j.warning()) << "Unusual condition detected";
JLOG(j.error()) << "Failed to process: " << error;
JLOG(j.fatal()) << "Critical error, shutting down";

Log File Location

Default Locations:

  • Linux: /var/log/rippled/debug.log

  • macOS: ~/Library/Application Support/rippled/debug.log

  • Custom: Set in rippled.cfg:

[debug_logfile]
/path/to/custom/debug.log

Log Rotation

Configure log rotation to prevent disk space issues:

[debug_logfile]
/var/log/rippled/debug.log

# Rotate when file reaches 100MB
# Keep 10 old log files

Using system tools (Linux):

# /etc/logrotate.d/rippled
/var/log/rippled/debug.log {
    daily
    rotate 7
    compress
    delaycompress
    missingok
    notifempty
    copytruncate
}

Reading Log Files

Tail Live Logs:

tail -f /var/log/rippled/debug.log

Filter by Partition:

grep "Transaction:" /var/log/rippled/debug.log

Filter by Severity:

grep "ERR" /var/log/rippled/debug.log

Timestamp Range:

# Logs between specific times
awk '/2025-01-15 10:00/,/2025-01-15 11:00/' /var/log/rippled/debug.log

Common Patterns:

# Find transaction processing
grep "Transaction.*tesSUCCESS" /var/log/rippled/debug.log

# Find consensus rounds
grep "Consensus.*Starting round" /var/log/rippled/debug.log

# Find peer connections
grep "Overlay.*Connected to peer" /var/log/rippled/debug.log

# Find errors
grep -E "ERROR|ERR|Fatal" /var/log/rippled/debug.log

Standalone Mode

What is Standalone Mode?

Standalone mode runs Rippled as a single-node network where you have complete control:

  • No peers: Runs without connecting to other nodes

  • Manual ledger close: You trigger ledger closes

  • Deterministic: No network randomness

  • Fast: No consensus delays

  • Isolated: Perfect for testing

Starting Standalone Mode

rippled --standalone --conf=/path/to/rippled.cfg

Configuration for Standalone:

[server]
port_rpc_admin_local
port_ws_admin_local

[port_rpc_admin_local]
port = 5005
ip = 127.0.0.1
admin = 127.0.0.1
protocol = http

[port_ws_admin_local]
port = 6006
ip = 127.0.0.1
admin = 127.0.0.1
protocol = ws

# No peer port needed in standalone

[node_db]
type=NuDB
path=/var/lib/rippled/standalone/db

[database_path]
/var/lib/rippled/standalone

Using Standalone Mode

Check Status:

rippled server_info

Look for:

{
  "result": {
    "info": {
      "build_version": "1.9.0",
      "complete_ledgers": "1-5",
      "peers": 0,
      "server_state": "proposing",
      "standalone": true
    }
  }
}

Submit Transaction:

rippled submit '{
  "TransactionType": "Payment",
  "Account": "rN7n7otQDd6FczFgLdlqtyMVrn3HMtthca",
  "Destination": "rLNaPoKeeBjZe2qs6x52yVPZpZ8td4dc6w",
  "Amount": "1000000",
  "Fee": "12",
  "Sequence": 1
}'

Manually Close Ledger:

rippled ledger_accept

This immediately closes the current ledger and advances to the next one.

Check Transaction:

rippled tx <hash>

Standalone Mode Workflow

# 1. Start standalone
rippled --standalone --conf=standalone.cfg

# 2. Fund accounts (in another terminal)
rippled wallet_propose

# 3. Submit transactions
rippled submit <signed_tx>

# 4. Close ledger to include transaction
rippled ledger_accept

# 5. Verify transaction
rippled tx <hash>

# 6. Repeat steps 3-5 as needed

Advantages for Debugging

Deterministic Behavior:

  • No network randomness

  • Repeatable tests

  • Predictable timing

Complete Control:

  • Manual ledger progression

  • No unexpected transactions

  • Isolated environment

Fast Iteration:

  • Instant ledger closes

  • No waiting for consensus

  • Quick test cycles

Safe Experimentation:

  • Can't affect mainnet

  • Easy to reset (delete database)

  • Test dangerous operations safely


GDB Debugging

Setting Up GDB

Install GDB:

# Linux
sudo apt-get install gdb

# macOS
brew install gdb

Compile with Debug Symbols:

cd rippled/build
cmake -DCMAKE_BUILD_TYPE=Debug ..
cmake --build . --target rippled

Launch with GDB:

gdb --args ./rippled --conf=/path/to/rippled.cfg --standalone

Basic GDB Commands

Starting:

(gdb) run                    # Start program
(gdb) start                  # Start and break at main()

Breakpoints:

(gdb) break Payment.cpp:123  # Break at file:line
(gdb) break Payment::doApply # Break at function
(gdb) break Transactor.cpp:apply # Break at method

(gdb) info breakpoints       # List all breakpoints
(gdb) delete 1               # Delete breakpoint #1
(gdb) disable 2              # Disable breakpoint #2

Execution Control:

(gdb) continue               # Continue execution
(gdb) next                   # Step over (one line)
(gdb) step                   # Step into (enter function)
(gdb) finish                 # Run until current function returns

Inspection:

(gdb) print variable         # Print variable value
(gdb) print *pointer         # Dereference pointer
(gdb) print object.method()  # Call method

(gdb) backtrace              # Show call stack
(gdb) frame 3                # Switch to frame #3
(gdb) info locals            # Show local variables

Advanced:

(gdb) watch variable         # Break when variable changes
(gdb) condition 1 i == 5     # Conditional breakpoint
(gdb) commands 1             # Execute commands at breakpoint

Debugging Transaction Processing

Example Session:

# Start GDB with rippled
gdb --args ./rippled --standalone --conf=standalone.cfg

# Set breakpoints
(gdb) break Payment::doApply
(gdb) break Transactor::apply
(gdb) break NetworkOPs::processTransaction

# Run
(gdb) run

# In another terminal, submit transaction
$ rippled submit <signed_tx>

# GDB will break at processTransaction
(gdb) backtrace
#0  NetworkOPs::processTransaction
#1  RPCHandler::doCommand
#2  ...

# Step through
(gdb) next
(gdb) next

# Examine transaction
(gdb) print transaction->getTransactionID()
(gdb) print transaction->getFieldAmount(sfAmount)

# Continue to Payment::doApply
(gdb) continue

# Examine state
(gdb) print account_
(gdb) print ctx_.tx[sfDestination]
(gdb) print view().read(keylet::account(account_))

# Step through payment logic
(gdb) step
(gdb) next

# Check result
(gdb) print result
(gdb) continue

Debugging Consensus

# Set breakpoints in consensus
(gdb) break RCLConsensus::startRound
(gdb) break Consensus::propose
(gdb) break Consensus::peerProposal

# Run
(gdb) run

# When consensus starts
(gdb) print prevLedgerHash
(gdb) print transactions.size()
(gdb) backtrace

# Step through proposal creation
(gdb) step
(gdb) print position_

# Continue to peer proposal handling
(gdb) continue
(gdb) print proposal.position()
(gdb) print peerID

Debugging Crashes

Core Dumps:

Enable core dumps:

ulimit -c unlimited

Run program until crash:

./rippled --standalone --conf=standalone.cfg
# ... crash occurs

Analyze core dump:

gdb ./rippled core
(gdb) backtrace
(gdb) frame 0
(gdb) info locals

Common Crash Patterns:

# Null pointer dereference
(gdb) print pointer
$1 = 0x0
(gdb) backtrace
# Look for where pointer should have been set

# Segmentation fault
(gdb) print array[index]
# Check if index is out of bounds

# Assert failure
(gdb) backtrace
# Look at assertion condition and surrounding code

Log Analysis and Interpretation

Transaction Logs

Successful Payment:

2025-01-15 10:23:45.123 Transaction:DBG Transaction E08D6E9754... submitted
2025-01-15 10:23:45.125 Transaction:TRC Preflight check passed
2025-01-15 10:23:45.126 Transaction:TRC Preclaim check passed
2025-01-15 10:23:45.127 Transaction:DBG Applied to open ledger: tesSUCCESS
2025-01-15 10:23:45.128 Overlay:TRC Relaying transaction to 18 peers
2025-01-15 10:23:50.234 Consensus:DBG Transaction included in consensus set
2025-01-15 10:23:50.456 Transaction:INF Applied to ledger 75234567: tesSUCCESS

Failed Payment:

2025-01-15 10:23:45.123 Transaction:DBG Transaction E08D6E9754... submitted
2025-01-15 10:23:45.125 Transaction:TRC Preflight check passed
2025-01-15 10:23:45.126 Transaction:WRN Preclaim check failed: tecUNFUNDED
2025-01-15 10:23:45.127 Transaction:DBG Rejected: insufficient funds

Consensus Logs

Normal Consensus Round:

2025-01-15 10:23:50.000 Consensus:INF Starting consensus round
2025-01-15 10:23:50.001 Consensus:DBG Building initial position: 147 transactions
2025-01-15 10:23:50.010 Consensus:TRC Proposal sent: hash=ABC123...
2025-01-15 10:23:52.123 Consensus:TRC Received proposal from nHU...: 145 txns
2025-01-15 10:23:52.125 Consensus:TRC Received proposal from nHB...: 146 txns
2025-01-15 10:23:52.500 Consensus:DBG Agreement: 145/147 transactions (98%)
2025-01-15 10:23:52.501 Consensus:INF Consensus reached on transaction set
2025-01-15 10:23:52.600 LedgerMaster:INF Ledger 75234567 closed
2025-01-15 10:23:54.000 LedgerMaster:INF Ledger 75234567 validated with 28/35 validations

Disputed Transaction:

2025-01-15 10:23:50.000 Consensus:INF Starting consensus round
2025-01-15 10:23:52.123 Consensus:DBG Transaction TX123 agreement: 65%
2025-01-15 10:23:54.456 Consensus:DBG Transaction TX123 agreement: 75%
2025-01-15 10:23:56.789 Consensus:WRN Transaction TX123 not included: only 75% agreement
2025-01-15 10:23:56.790 Consensus:INF Consensus reached on transaction set (TX123 excluded)

Network Logs

Peer Connection:

2025-01-15 10:23:45.123 Overlay:INF Connecting to r.ripple.com:51235
2025-01-15 10:23:45.234 Overlay:DBG TCP connection established
2025-01-15 10:23:45.345 Overlay:TRC TLS handshake complete
2025-01-15 10:23:45.456 Overlay:TRC Protocol handshake: version 2, node nHU...
2025-01-15 10:23:45.567 Overlay:INF Connected to peer nHU... (validator)
2025-01-15 10:23:45.568 Peer:DBG Added to active peers (18/20)

Connection Failure:

2025-01-15 10:23:45.123 Overlay:INF Connecting to bad-peer.example.com:51235
2025-01-15 10:23:50.123 Overlay:WRN Connection timeout
2025-01-15 10:23:50.124 Overlay:DBG Scheduling reconnect in 10 seconds

Performance Logs

Slow Ledger Close:

2025-01-15 10:23:50.000 LedgerMaster:INF Closing ledger 75234567
2025-01-15 10:23:55.000 LedgerMaster:WRN Ledger close took 5000ms (expected <2000ms)
2025-01-15 10:23:55.001 Transaction:WRN Applied 500 transactions in 4800ms
2025-01-15 10:23:55.002 OrderBookDB:WRN Order book update took 1200ms

Performance Profiling

CPU Profiling

Using perf (Linux):

# Record profile while running
perf record -g ./rippled --standalone --conf=standalone.cfg

# Generate report
perf report

# Generate flamegraph
perf script | stackcollapse-perf.pl | flamegraph.pl > rippled.svg

Using Instruments (macOS):

# Launch with Instruments
instruments -t "Time Profiler" ./rippled --standalone --conf=standalone.cfg

Interpreting Results:

Look for:

  • Hot functions (high CPU usage)

  • Unexpected call patterns

  • Inefficient algorithms

  • Lock contention

Memory Profiling

Using Valgrind:

# Memory leak detection
valgrind --leak-check=full ./rippled --standalone --conf=standalone.cfg

# Memory profiler
valgrind --tool=massif ./rippled --standalone --conf=standalone.cfg
ms_print massif.out.12345

Using AddressSanitizer:

# Compile with sanitizer
cmake -DCMAKE_BUILD_TYPE=Debug \
      -DCMAKE_CXX_FLAGS="-fsanitize=address" ..
cmake --build . --target rippled

# Run (crashes on memory errors)
./rippled --standalone --conf=standalone.cfg

Network Profiling

Wireshark:

# Capture traffic
sudo tcpdump -i any port 51235 -w rippled.pcap

# Analyze with Wireshark
wireshark rippled.pcap

Filter by:

  • Protocol messages

  • Connection handshakes

  • Bandwidth usage

Measuring Latency:

# Ping peer
rippled ping <peer_ip>

# Check peer latency
rippled peers | grep latency

Testing Strategies

Unit Testing

Run All Tests:

./rippled --unittest

Run Specific Test Suite:

./rippled --unittest=Payment
./rippled --unittest=Consensus

Run with Verbose Output:

./rippled --unittest --unittest-log

Writing Tests:

#include <test/jtx.h>

namespace ripple {
namespace test {

class MyTest_test : public beast::unit_test::suite
{
public:
    void testBasicOperation()
    {
        using namespace jtx;
        
        // Create test environment
        Env env(*this);
        
        // Create accounts
        Account alice{"alice"};
        Account bob{"bob"};
        env.fund(XRP(10000), alice, bob);
        
        // Test operation
        env(pay(alice, bob, XRP(100)));
        env.close();
        
        // Verify result
        BEAST_EXPECT(env.balance(bob) == XRP(10100));
    }
    
    void run() override
    {
        testBasicOperation();
    }
};

BEAST_DEFINE_TESTSUITE(MyTest, app, ripple);

} // namespace test
} // namespace ripple

Integration Testing

Test Network Setup:

# Start multiple rippled instances
./rippled --conf=node1.cfg &
./rippled --conf=node2.cfg &
./rippled --conf=node3.cfg &

# Configure them to peer
rippled --conf=node1.cfg connect localhost:51236
rippled --conf=node2.cfg connect localhost:51237

Test Scenarios:

  • Multi-node consensus

  • Network partitions

  • Peer discovery

  • Transaction propagation


Common Debugging Scenarios

Scenario 1: Transaction Not Validating

Symptoms: Transaction stuck in pending state

Debug Steps:

  1. Check transaction status:

rippled tx <hash>
  1. Check for sequence gaps:

rippled account_info <account>
# Compare Sequence with expected
  1. Check logs for rejection:

grep "<hash>" /var/log/rippled/debug.log
  1. Verify fee is sufficient:

rippled server_info | grep "load_factor"
# Fee should be baseFee * loadFactor
  1. Check LastLedgerSequence:

rippled ledger_current
# Compare with transaction's LastLedgerSequence

Scenario 2: Consensus Not Progressing

Symptoms: Ledger not closing, network stalled

Debug Steps:

  1. Check validator connectivity:

rippled validators
  1. Examine consensus logs:

rippled log_level Consensus trace
tail -f /var/log/rippled/debug.log | grep Consensus
  1. Check network connectivity:

rippled peers
# Verify connected to enough peers
  1. Look for disputes:

grep "dispute" /var/log/rippled/debug.log

Scenario 3: High Memory Usage

Symptoms: Rippled consuming excessive memory

Debug Steps:

  1. Check ledger history:

rippled server_info | grep "complete_ledgers"
  1. Review configuration:

[node_db]
cache_mb=256  # Reduce if too high
  1. Profile memory usage:

valgrind --tool=massif ./rippled --standalone
ms_print massif.out.12345
  1. Check for leaks:

valgrind --leak-check=full ./rippled --standalone

Scenario 4: Slow Ledger Closes

Symptoms: Ledgers taking >5 seconds to close

Debug Steps:

  1. Enable performance logging:

rippled log_level LedgerMaster debug
rippled log_level Transaction debug
  1. Check transaction count:

grep "Applied.*transactions" /var/log/rippled/debug.log
  1. Profile CPU usage:

perf record -g ./rippled
perf report
  1. Check database performance:

# Check NodeStore backend performance
grep "NodeStore" /var/log/rippled/debug.log

Hands-On Exercise

Exercise: Debug a Transaction Failure

Objective: Use debugging tools to diagnose and fix a transaction issue.

Part 1: Setup

Step 1: Start standalone mode with detailed logging

# Configure logging
cat > standalone.cfg << EOF
[server]
port_rpc_admin_local

[port_rpc_admin_local]
port = 5005
ip = 127.0.0.1
admin = 127.0.0.1
protocol = http

[node_db]
type=NuDB
path=/tmp/rippled_debug

[database_path]
/tmp/rippled_debug

[rpc_startup]
{ "command": "log_level", "severity": "debug" }
{ "command": "log_level", "partition": "Transaction", "severity": "trace" }
EOF

# Start
rippled --standalone --conf=standalone.cfg

Step 2: Create test scenario with intentional issue

// Create underfunded transaction
const tx = {
  TransactionType: 'Payment',
  Account: 'rN7n7otQDd6FczFgLdlqtyMVrn3HMtthca',
  Destination: 'rLNaPoKeeBjZe2qs6x52yVPZpZ8td4dc6w',
  Amount: '999999999999',  // More than account has
  Fee: '12',
  Sequence: 1
};

Part 2: Debugging Process

Step 3: Submit and observe failure

rippled submit <signed_tx>

Step 4: Examine logs

tail -100 /var/log/rippled/debug.log | grep Transaction

Look for:

Transaction:TRC Preflight check: passed
Transaction:TRC Preclaim check: failed - tecUNFUNDED_PAYMENT
Transaction:DBG Rejected transaction: insufficient funds

Step 5: Verify balance

rippled account_info rN7n7otQDd6FczFgLdlqtyMVrn3HMtthca

Step 6: Calculate required amount

const balance = accountInfo.account_data.Balance;
const reserve = 20000000; // Base reserve
const available = balance - reserve;
console.log(`Available: ${available} drops`);
console.log(`Requested: 999999999999 drops`);
console.log(`Shortfall: ${999999999999 - available} drops`);

Step 7: Fix and resubmit

// Corrected transaction
const fixedTx = {
  ...tx,
  Amount: String(available - 12)  // Account for fee
};

Part 3: Advanced Debugging

Step 8: Debug with GDB

# Recompile with debug symbols
cd rippled/build
cmake -DCMAKE_BUILD_TYPE=Debug ..
make

# Start with GDB
gdb --args ./rippled --standalone --conf=standalone.cfg

Step 9: Set breakpoints

(gdb) break Payment::preclaim
(gdb) run

Step 10: Submit transaction (in another terminal)

rippled submit <signed_tx>

Step 11: Examine state in GDB

# Breakpoint hit
(gdb) print ctx.tx[sfAmount]
(gdb) print (*sleAccount)[sfBalance]
(gdb) print fee
(gdb) print balance < (amount + fee)
# Should be true, causing tecUNFUNDED

(gdb) continue

Analysis Questions

  1. What error code was returned?

    • tecUNFUNDED_PAYMENT

  2. At which validation phase did it fail?

    • Preclaim (ledger state check)

  3. What was the root cause?

    • Insufficient balance for payment + fee + reserve

  4. How would you prevent this in client code?

    • Check balance before submitting

    • Account for reserve requirements

    • Include fee in calculation

  5. What logs helped identify the issue?

    • Transaction partition trace logs

    • Preclaim failure message


Key Takeaways

Core Debugging Skills

Logging System: Understand partitions, severity levels, and configuration

Standalone Mode: Essential for controlled testing and debugging

GDB: Set breakpoints, inspect variables, trace execution

Log Analysis: Read and interpret logs to diagnose issues

Performance Profiling: Identify bottlenecks and optimize

Best Practices

Start Simple: Use logs before reaching for debugger

Reproduce Reliably: Use standalone mode for consistent reproduction

Isolate Issues: Narrow down to specific component

Read the Code: Logs point you to code, understand the implementation

Test Thoroughly: Write unit tests for bugs you fix

Tool Selection

Logs: First line of defense, always available

Standalone Mode: Controlled environment, fast iteration

GDB: Deep inspection, understanding execution flow

Profiling: Performance issues, optimization

Tests: Regression prevention, continuous validation


Additional Resources

Official Documentation

Debugging Resources

Codebase References

  • src/ripple/core/impl/JobQueue.cpp - Job queue debugging

  • src/ripple/app/main/Application.cpp - Application startup debugging

  • src/test/ - Unit test examples


Next Steps

Congratulations! You've completed the Rippled II deep dive topics. You now have a solid understanding of:

  • Protocols and networking

  • Transaction processing

  • Application architecture

  • Consensus mechanisms

  • Network topology

  • Transaction lifecycle

  • Debugging techniques

Continue to Module 3: Data Architecture - SHAMap and NodeStore //TODO

⬅️ Back to: Rippled II Overview


Get Started

Access the course: docs.xrpl-commons.org/core-dev-bootcamp

Got questions? Contact us here: Submit Feedback


© 2025 XRPL Commons - Core Dev Bootcamp

Last updated