Debugging is an essential skill for any Rippled developer. Whether you're tracking down a subtle transaction processing bug, investigating network connectivity issues, or optimizing performance, having a solid understanding of debugging tools and techniques is critical. The complexity of distributed systems like the XRP Ledger means that effective debugging often requires multiple approaches—from analyzing logs to using debuggers to running controlled experiments.
This deep dive covers the comprehensive toolkit available for debugging Rippled, from basic logging to advanced profiling techniques. You'll learn how to diagnose issues quickly, understand system behavior, and develop confidence in working with the Rippled codebase.
Logging System
Overview
Rippled includes a sophisticated logging system that provides detailed visibility into system behavior. Understanding how to configure and use logging effectively is the foundation of debugging Rippled.
Log Structure
Partitions: Logs are organized by subsystem (partition) Severity Levels: Each log entry has a severity level Timestamps: All logs include precise timestamps Context: Logs include relevant context (account IDs, ledger numbers, etc.)
Severity Levels
From most to least verbose:
trace - Extremely detailed, every function call
debug - Detailed debugging information
info - General informational messages
warning - Warning conditions
error - Error conditions
fatal - Fatal errors that cause termination
Usage Guidelines:
Production: Use warning or error to minimize disk I/O
Development: Use debug or trace for active debugging
Investigation: Temporarily enable trace for specific partitions
# Set all partitions to warning
rippled log_level warning
# Set specific partition to trace
rippled log_level Transaction trace
# Set multiple partitions
rippled log_level Consensus debug
rippled log_level Overlay debug
rippled log_level Peer trace
// Get logger for this partition
beast::Journal j = app_.journal("MyComponent");
// Log at different levels
JLOG(j.trace()) << "Entering function with param: " << param;
JLOG(j.debug()) << "Processing transaction: " << tx.getTransactionID();
JLOG(j.info()) << "Ledger closed: " << ledger.seq();
JLOG(j.warning()) << "Unusual condition detected";
JLOG(j.error()) << "Failed to process: " << error;
JLOG(j.fatal()) << "Critical error, shutting down";
[debug_logfile]
/path/to/custom/debug.log
[debug_logfile]
/var/log/rippled/debug.log
# Rotate when file reaches 100MB
# Keep 10 old log files
(gdb) run # Start program
(gdb) start # Start and break at main()
(gdb) break Payment.cpp:123 # Break at file:line
(gdb) break Payment::doApply # Break at function
(gdb) break Transactor.cpp:apply # Break at method
(gdb) info breakpoints # List all breakpoints
(gdb) delete 1 # Delete breakpoint #1
(gdb) disable 2 # Disable breakpoint #2
(gdb) continue # Continue execution
(gdb) next # Step over (one line)
(gdb) step # Step into (enter function)
(gdb) finish # Run until current function returns
(gdb) print variable # Print variable value
(gdb) print *pointer # Dereference pointer
(gdb) print object.method() # Call method
(gdb) backtrace # Show call stack
(gdb) frame 3 # Switch to frame #3
(gdb) info locals # Show local variables
(gdb) watch variable # Break when variable changes
(gdb) condition 1 i == 5 # Conditional breakpoint
(gdb) commands 1 # Execute commands at breakpoint
# Start GDB with rippled
gdb --args ./rippled --standalone --conf=standalone.cfg
# Set breakpoints
(gdb) break Payment::doApply
(gdb) break Transactor::apply
(gdb) break NetworkOPs::processTransaction
# Run
(gdb) run
# In another terminal, submit transaction
$ rippled submit <signed_tx>
# GDB will break at processTransaction
(gdb) backtrace
#0 NetworkOPs::processTransaction
#1 RPCHandler::doCommand
#2 ...
# Step through
(gdb) next
(gdb) next
# Examine transaction
(gdb) print transaction->getTransactionID()
(gdb) print transaction->getFieldAmount(sfAmount)
# Continue to Payment::doApply
(gdb) continue
# Examine state
(gdb) print account_
(gdb) print ctx_.tx[sfDestination]
(gdb) print view().read(keylet::account(account_))
# Step through payment logic
(gdb) step
(gdb) next
# Check result
(gdb) print result
(gdb) continue
# Set breakpoints in consensus
(gdb) break RCLConsensus::startRound
(gdb) break Consensus::propose
(gdb) break Consensus::peerProposal
# Run
(gdb) run
# When consensus starts
(gdb) print prevLedgerHash
(gdb) print transactions.size()
(gdb) backtrace
# Step through proposal creation
(gdb) step
(gdb) print position_
# Continue to peer proposal handling
(gdb) continue
(gdb) print proposal.position()
(gdb) print peerID
# Null pointer dereference
(gdb) print pointer
$1 = 0x0
(gdb) backtrace
# Look for where pointer should have been set
# Segmentation fault
(gdb) print array[index]
# Check if index is out of bounds
# Assert failure
(gdb) backtrace
# Look at assertion condition and surrounding code
2025-01-15 10:23:45.123 Transaction:DBG Transaction E08D6E9754... submitted
2025-01-15 10:23:45.125 Transaction:TRC Preflight check passed
2025-01-15 10:23:45.126 Transaction:TRC Preclaim check passed
2025-01-15 10:23:45.127 Transaction:DBG Applied to open ledger: tesSUCCESS
2025-01-15 10:23:45.128 Overlay:TRC Relaying transaction to 18 peers
2025-01-15 10:23:50.234 Consensus:DBG Transaction included in consensus set
2025-01-15 10:23:50.456 Transaction:INF Applied to ledger 75234567: tesSUCCESS
# Recompile with debug symbols
cd rippled/build
cmake -DCMAKE_BUILD_TYPE=Debug ..
make
# Start with GDB
gdb --args ./rippled --standalone --conf=standalone.cfg
(gdb) break Payment::preclaim
(gdb) run
rippled submit <signed_tx>
# Breakpoint hit
(gdb) print ctx.tx[sfAmount]
(gdb) print (*sleAccount)[sfBalance]
(gdb) print fee
(gdb) print balance < (amount + fee)
# Should be true, causing tecUNFUNDED
(gdb) continue