Ctrlk
For the complete documentation index, see llms.txt. This page is also available as Markdown.

Debugging Tools: Development and Debugging Techniques

← Back to Rippled II Overview

Introduction

Debugging is an essential skill for any Rippled developer. Whether you're tracking down a subtle transaction processing bug, investigating network connectivity issues, or optimizing performance, having a solid understanding of debugging tools and techniques is critical. The complexity of distributed systems like the XRP Ledger means that effective debugging often requires multiple approaches—from analyzing logs to using debuggers to running controlled experiments.

This deep dive covers the comprehensive toolkit available for debugging Rippled, from basic logging to advanced profiling techniques. You'll learn how to diagnose issues quickly, understand system behavior, and develop confidence in working with the Rippled codebase.

Logging System

Overview

Rippled includes a sophisticated logging system that provides detailed visibility into system behavior. Understanding how to configure and use logging effectively is the foundation of debugging Rippled.

Log Structure

Partitions: Logs are organized by subsystem (partition) Severity Levels: Each log entry has a severity level Timestamps: All logs include precise timestamps Context: Logs include relevant context (account IDs, ledger numbers, etc.)

Severity Levels

From most to least verbose:

trace   - Extremely detailed, every function call
debug   - Detailed debugging information
info    - General informational messages
warning - Warning conditions
error   - Error conditions
fatal   - Fatal errors that cause termination

Usage Guidelines:

  • Production: Use warning or error to minimize disk I/O

  • Development: Use debug or trace for active debugging

  • Investigation: Temporarily enable trace for specific partitions

Log Partitions

Major subsystems have their own partitions:

Configuring Logging

In Configuration File

Edit rippled.cfg:

Via RPC Command

Dynamic adjustment without restart:

Programmatically

In code:

Log File Location

Default Locations:

  • Linux: /var/log/rippled/debug.log

  • macOS: ~/Library/Application Support/rippled/debug.log

  • Custom: Set in rippled.cfg:

Log Rotation

Configure log rotation to prevent disk space issues:

Using system tools (Linux):

Reading Log Files

Tail Live Logs:

Filter by Partition:

Filter by Severity:

Timestamp Range:

Common Patterns:

Standalone Mode

What is Standalone Mode?

Standalone mode runs Rippled as a single-node network where you have complete control:

  • No peers: Runs without connecting to other nodes

  • Manual ledger close: You trigger ledger closes

  • Deterministic: No network randomness

  • Fast: No consensus delays

  • Isolated: Perfect for testing

Starting Standalone Mode

Configuration for Standalone:

Using Standalone Mode

Check Status:

Look for:

Submit Transaction:

Manually Close Ledger:

This immediately closes the current ledger and advances to the next one.

Check Transaction:

Standalone Mode Workflow

Advantages for Debugging

Deterministic Behavior:

  • No network randomness

  • Repeatable tests

  • Predictable timing

Complete Control:

  • Manual ledger progression

  • No unexpected transactions

  • Isolated environment

Fast Iteration:

  • Instant ledger closes

  • No waiting for consensus

  • Quick test cycles

Safe Experimentation:

  • Can't affect mainnet

  • Easy to reset (delete database)

  • Test dangerous operations safely

GDB Debugging

Setting Up GDB

Install GDB:

Compile with Debug Symbols:

Launch with GDB:

Basic GDB Commands

Starting:

Breakpoints:

Execution Control:

Inspection:

Advanced:

Debugging Transaction Processing

Example Session:

Debugging Consensus

Debugging Crashes

Core Dumps:

Enable core dumps:

Run program until crash:

Analyze core dump:

Common Crash Patterns:

Log Analysis and Interpretation

Transaction Logs

Successful Payment:

Failed Payment:

Consensus Logs

Normal Consensus Round:

Disputed Transaction:

Network Logs

Peer Connection:

Connection Failure:

Performance Logs

Slow Ledger Close:

Performance Profiling

CPU Profiling

Using perf (Linux):

Using Instruments (macOS):

Interpreting Results:

Look for:

  • Hot functions (high CPU usage)

  • Unexpected call patterns

  • Inefficient algorithms

  • Lock contention

Memory Profiling

Using Valgrind:

Using AddressSanitizer:

Network Profiling

Wireshark:

Filter by:

  • Protocol messages

  • Connection handshakes

  • Bandwidth usage

Measuring Latency:

Testing Strategies

Unit Testing

Run All Tests:

Run Specific Test Suite:

Run with Verbose Output:

Writing Tests:

Integration Testing

Test Network Setup:

Test Scenarios:

  • Multi-node consensus

  • Network partitions

  • Peer discovery

  • Transaction propagation

Common Debugging Scenarios

Scenario 1: Transaction Not Validating

Symptoms: Transaction stuck in pending state

Debug Steps:

  1. Check transaction status:

  1. Check for sequence gaps:

  1. Check logs for rejection:

  1. Verify fee is sufficient:

  1. Check LastLedgerSequence:

Scenario 2: Consensus Not Progressing

Symptoms: Ledger not closing, network stalled

Debug Steps:

  1. Check validator connectivity:

  1. Examine consensus logs:

  1. Check network connectivity:

  1. Look for disputes:

Scenario 3: High Memory Usage

Symptoms: Rippled consuming excessive memory

Debug Steps:

  1. Check ledger history:

  1. Review configuration:

  1. Profile memory usage:

  1. Check for leaks:

Scenario 4: Slow Ledger Closes

Symptoms: Ledgers taking >5 seconds to close

Debug Steps:

  1. Enable performance logging:

  1. Check transaction count:

  1. Profile CPU usage:

  1. Check database performance:

Hands-On Exercise

Exercise: Debug a Transaction Failure

Objective: Use debugging tools to diagnose and fix a transaction issue.

Part 1: Setup

Step 1: Start standalone mode with detailed logging

Step 2: Create test scenario with intentional issue

Part 2: Debugging Process

Step 3: Submit and observe failure

Step 4: Examine logs

Look for:

Step 5: Verify balance

Step 6: Calculate required amount

Step 7: Fix and resubmit

Part 3: Advanced Debugging

Step 8: Debug with GDB

Step 9: Set breakpoints

Step 10: Submit transaction (in another terminal)

Step 11: Examine state in GDB

Analysis Questions

  1. What error code was returned?

    • tecUNFUNDED_PAYMENT

  2. At which validation phase did it fail?

    • Preclaim (ledger state check)

  3. What was the root cause?

    • Insufficient balance for payment + fee + reserve

  4. How would you prevent this in client code?

    • Check balance before submitting

    • Account for reserve requirements

    • Include fee in calculation

  5. What logs helped identify the issue?

    • Transaction partition trace logs

    • Preclaim failure message

Key Takeaways

Core Debugging Skills

✅ Logging System: Understand partitions, severity levels, and configuration

✅ Standalone Mode: Essential for controlled testing and debugging

✅ GDB: Set breakpoints, inspect variables, trace execution

✅ Log Analysis: Read and interpret logs to diagnose issues

✅ Performance Profiling: Identify bottlenecks and optimize

Best Practices

✅ Start Simple: Use logs before reaching for debugger

✅ Reproduce Reliably: Use standalone mode for consistent reproduction

✅ Isolate Issues: Narrow down to specific component

✅ Read the Code: Logs point you to code, understand the implementation

✅ Test Thoroughly: Write unit tests for bugs you fix

Tool Selection

✅ Logs: First line of defense, always available

✅ Standalone Mode: Controlled environment, fast iteration

✅ GDB: Deep inspection, understanding execution flow

✅ Profiling: Performance issues, optimization

✅ Tests: Regression prevention, continuous validation

Additional Resources

Official Documentation

Debugging Resources

Codebase References

  • src/ripple/core/impl/JobQueue.cpp - Job queue debugging

  • src/ripple/app/main/Application.cpp - Application startup debugging

  • src/test/ - Unit test examples

Related Topics

PreviousCodebase Navigation: Efficiently Navigating the Rippled Source
NextHomeworks

Last updated