Appendix : Debugging and Development Tools

← Back to SHAMap and NodeStore: Data Persistence and State Management

Introduction

This appendix provides techniques and tools for investigating SHAMap and NodeStore behavior.

Logging and Diagnostics

Enable Verbose Logging

Edit rippled.cfg:

[rpc_startup]
command = log_level
severity = debug

[logging]
debug
rpc

Then restart rippled and check logs:

tail -f /var/log/rippled/rippled.log | grep -i nodestore

Key Log Messages

TRACE Ledger:        Ledger opened/closed
DEBUG SHAMap:        Tree operations
DEBUG NodeStore:     Database operations
INFO  Consensus:     Validation and agreement
WARN  Performance:   Slow operations detected

Metrics Inspection

JSON-RPC Inspection

# Get storage metrics
rippled-cli server_info | jq '.result.node_db'

# Expected output:
{
  "type": "RocksDB",
  "path": "/var/lib/rippled/db/rocksdb",
  "cache_size": 256,
  "cache_hit_rate": 0.923,
  "writes": 1000000,
  "bytes_written": 1000000000,
  "reads": 50000000,
  "cache_hits": 46150000,
  "read_latency_us": 15
}

File System Inspection

# Check database size
du -sh /var/lib/rippled/db/*

# Monitor growth
watch -n 1 'du -sh /var/lib/rippled/db/*'

# Check free space
df -h /var/lib/rippled/

# Monitor I/O
iostat -x 1 /dev/sda

Debugging Specific Issues

Issue: Cache Hit Rate Too Low

Symptoms:

Database queries slow
Ledger close times increasing
Hit rate < 80%

Investigation:

# Check cache metrics
rippled-cli server_info | jq '.result.node_db.cache_hit_rate'

# Check cache size configuration
grep cache_size rippled.cfg

# Monitor cache evictions
tail -f /var/log/rippled/rippled.log | \
  grep -i "evict\|cache"

Solutions:

Increase cache_size if memory available
Reduce cache_age for faster eviction of cold data
Check if system is memory-constrained (use free)

Issue: Write Performance Degradation

Symptoms:

Ledger closes slow (>10 seconds)
Database write errors in logs
Validator falling behind network

Investigation:

# Check write latency
rippled-cli server_info | \
  jq '.result.node_db.write_latency_us'

# Monitor disk I/O
iotop -o -b -n 1

# Check disk space
df -h

# Monitor async queue
tail /var/log/rippled/rippled.log | \
  grep -i "async.*queue"

Solutions:

Ensure SSD (not HDD) for database
Check disk I/O isn't saturated
Increase async_threads if I/O bound
Switch to faster backend (NuDB vs RocksDB)
Enable compression if disk is bottleneck

Issue: Synchronization Slow

Symptoms:

New nodes take hours to sync
Falling behind network
High database query count

Investigation:

# Monitor sync progress
rippled-cli server_info | jq '.result.ledger.ledger_index'

# Track fetch operations
tail -f /var/log/rippled/rippled.log | \
  grep -i "fetch\|sync"

# Monitor thread pool
ps -p $(pidof rippled) -L

# Check queue depths
rippled-cli server_info | jq '.result.node_db.async_queue_depth'

Solutions:

Increase cache size for better hit rate during sync
Increase async_threads (more parallel fetches)
Use faster SSD
Check network bandwidth (might be bottleneck)
Switch to NuDB for higher throughput

Code Debugging

Building with Debug Symbols

cd rippled
mkdir build
cd build
cmake -DCMAKE_BUILD_TYPE=Debug ..
make -j4

GDB Debugging

# Run under GDB
gdb --args rippled --conf /path/to/rippled.cfg

# Inside GDB:
(gdb) run
(gdb) break SHAMap::addKnownNode
(gdb) continue

# When breakpoint hit:
(gdb) print node->getHash()
(gdb) print nodeID
(gdb) step
(gdb) quit

Common Breakpoints

// Node addition
break SHAMap::addKnownNode
break Database::store

// Cache operations
break TaggedCache::get
break TaggedCache::insert

// Synchronization
break SHAMap::getMissingNodes
break NodeStore::fetchNodeObject

Print Useful Data

(gdb) print node->getHash().hex()
(gdb) print nodeID.mDepth
(gdb) print nodeID.mNodeID.hex()
(gdb) print metrics.cacheHits
(gdb) print metrics.cacheMisses

Performance Profiling

CPU Profiling with Perf

# Record 60 seconds of system behavior
perf record -F 99 -p $(pidof rippled) -- sleep 60

# Analyze results
perf report

# Show flame graph
perf record -F 99 -p $(pidof rippled) -- sleep 60
perf script | stackcollapse-perf.pl | flamegraph.pl > profile.svg

Memory Profiling with Valgrind

# Run under memcheck (very slow)
valgrind --leak-check=full rippled --conf rippled.cfg

# Run specific test
valgrind --leak-check=full rippled --unittest test.nodestore

Custom Instrumentation

Add to rippled source:

// In Database::fetchNodeObject
auto startTime = std::chrono::steady_clock::now();

auto obj = mBackend->fetch(hash);

auto elapsed = std::chrono::steady_clock::now() - startTime;
auto ms = std::chrono::duration_cast<std::chrono::milliseconds>(elapsed);

if (ms.count() > 100) {
    JLOG(mLog.warning()) << "Slow fetch: " << hash.hex()
                        << " took " << ms.count() << "ms";
}

Test-Driven Debugging

Running Unit Tests

# Build with tests enabled
cd rippled/build
cmake -DENABLE_TESTING=ON ..

# Run all tests
ctest

# Run specific test
ctest -R "shamap" -V

# Run single test file
./bin/rippled --unittest test.SHAMap

Writing Debug Tests

// In rippled/src/test/shamap_test.cpp
SECTION("Debug specific case") {
    // Create SHAMap
    auto shamap = std::make_shared<SHAMap>(...);

    // Add nodes
    shamap->addRootNode(...);

    // Test operation
    auto node = shamap->getNode(hash);

    // Assert behavior
    REQUIRE(node != nullptr);
    REQUIRE(node->getHash() == expectedHash);
}

Useful Commands

Check Configuration

grep -E "^[a-z]|^\[" rippled.cfg | head -30

Monitor in Real Time

# CPU/Memory
top -p $(pidof rippled)

# Disk I/O
iotop -p $(pidof rippled)

# Network traffic
netstat -an | grep ripple

# File descriptors
lsof -p $(pidof rippled) | wc -l

Database Inspection

For RocksDB:

# Use RocksDB tools
rocksdb_ldb --db=/var/lib/rippled/db/rocksdb scan

# List files
ls -lah /var/lib/rippled/db/rocksdb/

Log Analysis

# Count errors
grep ERROR /var/log/rippled/rippled.log | wc -l

# Find slow operations
grep "took.*ms" /var/log/rippled/rippled.log

# Timeline of events
tail -f /var/log/rippled/rippled.log | \
  awk '{print $1" "$2" "$3" "$4" ..."}'

Performance Regression Testing

Benchmark Before/After

# Get baseline
rippled --unittest test.SHAMap > baseline.txt 2>&1

# Modify code...

# Test after change
rippled --unittest test.SHAMap > modified.txt 2>&1

# Compare
diff baseline.txt modified.txt

Load Testing

# Submit transactions and measure
./load_test.sh --transactions 1000 --duration 60

# Monitor metrics
watch -n 1 'rippled-cli server_info | jq ".result.node_db"'

Common Issues and Solutions

Issue

Investigation

Solution

High cache miss rate

Cache metrics

Increase cache_size

Slow sync

Fetch latency

Increase async_threads

Disk full

df -h

Enable online_delete

Memory leak

Valgrind

Fix code (likely nodes not freed)

Hang on startup

strace

Check database corruption

Consensus failing

Logs for validation errors

Check NodeStore consistency

See Appendix A for codebase navigation to find files mentioned here.

PreviousAppendix : Configuration Reference NextCryptography I: Blockchain Security and Cryptographic Foundations

Last updated 7 days ago