> For the complete documentation index, see [llms.txt](https://docs.xrpl-commons.org/llms.txt). Markdown versions of documentation pages are available by appending `.md` to page URLs; this page is available as [Markdown](https://docs.xrpl-commons.org/core-dev-bootcamp/module03/appendices/debugging.md). # Appendix : Debugging and Development Tools [← Back to SHAMap and NodeStore: Data Persistence and State Management](/core-dev-bootcamp/module03.md) *** ### Introduction This appendix provides techniques and tools for investigating SHAMap and NodeStore behavior. ## Logging and Diagnostics ### Enable Verbose Logging Edit `rippled.cfg`: ```ini [rpc_startup] command = log_level severity = debug [logging] debug rpc ``` Then restart rippled and check logs: ```bash tail -f /var/log/rippled/rippled.log | grep -i nodestore ``` ### Key Log Messages ``` TRACE Ledger: Ledger opened/closed DEBUG SHAMap: Tree operations DEBUG NodeStore: Database operations INFO Consensus: Validation and agreement WARN Performance: Slow operations detected ``` ## Metrics Inspection ### JSON-RPC Inspection ```bash # Get storage metrics rippled-cli server_info | jq '.result.node_db' # Expected output: { "type": "RocksDB", "path": "/var/lib/rippled/db/rocksdb", "cache_size": 256, "cache_hit_rate": 0.923, "writes": 1000000, "bytes_written": 1000000000, "reads": 50000000, "cache_hits": 46150000, "read_latency_us": 15 } ``` ### File System Inspection ```bash # Check database size du -sh /var/lib/rippled/db/* # Monitor growth watch -n 1 'du -sh /var/lib/rippled/db/*' # Check free space df -h /var/lib/rippled/ # Monitor I/O iostat -x 1 /dev/sda ``` ## Debugging Specific Issues ### Issue: Cache Hit Rate Too Low **Symptoms:** * Database queries slow * Ledger close times increasing * Hit rate < 80% **Investigation:** ```bash # Check cache metrics rippled-cli server_info | jq '.result.node_db.cache_hit_rate' # Check cache size configuration grep cache_size rippled.cfg # Monitor cache evictions tail -f /var/log/rippled/rippled.log | \ grep -i "evict\|cache" ``` **Solutions:** 1. Increase `cache_size` if memory available 2. Reduce `cache_age` for faster eviction of cold data 3. Check if system is memory-constrained (use `free`) ### Issue: Write Performance Degradation **Symptoms:** * Ledger closes slow (>10 seconds) * Database write errors in logs * Validator falling behind network **Investigation:** ```bash # Check write latency rippled-cli server_info | \ jq '.result.node_db.write_latency_us' # Monitor disk I/O iotop -o -b -n 1 # Check disk space df -h # Monitor async queue tail /var/log/rippled/rippled.log | \ grep -i "async.*queue" ``` **Solutions:** 1. Ensure SSD (not HDD) for database 2. Check disk I/O isn't saturated 3. Increase `async_threads` if I/O bound 4. Switch to faster backend (NuDB vs RocksDB) 5. Enable compression if disk is bottleneck ### Issue: Synchronization Slow **Symptoms:** * New nodes take hours to sync * Falling behind network * High database query count **Investigation:** ```bash # Monitor sync progress rippled-cli server_info | jq '.result.ledger.ledger_index' # Track fetch operations tail -f /var/log/rippled/rippled.log | \ grep -i "fetch\|sync" # Monitor thread pool ps -p $(pidof rippled) -L # Check queue depths rippled-cli server_info | jq '.result.node_db.async_queue_depth' ``` **Solutions:** 1. Increase cache size for better hit rate during sync 2. Increase `async_threads` (more parallel fetches) 3. Use faster SSD 4. Check network bandwidth (might be bottleneck) 5. Switch to NuDB for higher throughput ## Code Debugging ### Building with Debug Symbols ```bash cd rippled mkdir build cd build cmake -DCMAKE_BUILD_TYPE=Debug .. make -j4 ``` ### GDB Debugging ```bash # Run under GDB gdb --args rippled --conf /path/to/rippled.cfg # Inside GDB: (gdb) run (gdb) break SHAMap::addKnownNode (gdb) continue # When breakpoint hit: (gdb) print node->getHash() (gdb) print nodeID (gdb) step (gdb) quit ``` ### Common Breakpoints ```cpp // Node addition break SHAMap::addKnownNode break Database::store // Cache operations break TaggedCache::get break TaggedCache::insert // Synchronization break SHAMap::getMissingNodes break NodeStore::fetchNodeObject ``` ### Print Useful Data ```cpp (gdb) print node->getHash().hex() (gdb) print nodeID.mDepth (gdb) print nodeID.mNodeID.hex() (gdb) print metrics.cacheHits (gdb) print metrics.cacheMisses ``` ## Performance Profiling ### CPU Profiling with Perf ```bash # Record 60 seconds of system behavior perf record -F 99 -p $(pidof rippled) -- sleep 60 # Analyze results perf report # Show flame graph perf record -F 99 -p $(pidof rippled) -- sleep 60 perf script | stackcollapse-perf.pl | flamegraph.pl > profile.svg ``` ### Memory Profiling with Valgrind ```bash # Run under memcheck (very slow) valgrind --leak-check=full rippled --conf rippled.cfg # Run specific test valgrind --leak-check=full rippled --unittest test.nodestore ``` ### Custom Instrumentation Add to rippled source: ```cpp // In Database::fetchNodeObject auto startTime = std::chrono::steady_clock::now(); auto obj = mBackend->fetch(hash); auto elapsed = std::chrono::steady_clock::now() - startTime; auto ms = std::chrono::duration_cast(elapsed); if (ms.count() > 100) { JLOG(mLog.warning()) << "Slow fetch: " << hash.hex() << " took " << ms.count() << "ms"; } ``` ## Test-Driven Debugging ### Running Unit Tests ```bash # Build with tests enabled cd rippled/build cmake -DENABLE_TESTING=ON .. # Run all tests ctest # Run specific test ctest -R "shamap" -V # Run single test file ./bin/rippled --unittest test.SHAMap ``` ### Writing Debug Tests ```cpp // In rippled/src/test/shamap_test.cpp SECTION("Debug specific case") { // Create SHAMap auto shamap = std::make_shared(...); // Add nodes shamap->addRootNode(...); // Test operation auto node = shamap->getNode(hash); // Assert behavior REQUIRE(node != nullptr); REQUIRE(node->getHash() == expectedHash); } ``` ## Useful Commands ### Check Configuration ```bash grep -E "^[a-z]|^\[" rippled.cfg | head -30 ``` ### Monitor in Real Time ```bash # CPU/Memory top -p $(pidof rippled) # Disk I/O iotop -p $(pidof rippled) # Network traffic netstat -an | grep ripple # File descriptors lsof -p $(pidof rippled) | wc -l ``` ### Database Inspection For RocksDB: ```bash # Use RocksDB tools rocksdb_ldb --db=/var/lib/rippled/db/rocksdb scan # List files ls -lah /var/lib/rippled/db/rocksdb/ ``` ### Log Analysis ```bash # Count errors grep ERROR /var/log/rippled/rippled.log | wc -l # Find slow operations grep "took.*ms" /var/log/rippled/rippled.log # Timeline of events tail -f /var/log/rippled/rippled.log | \ awk '{print $1" "$2" "$3" "$4" ..."}' ``` ## Performance Regression Testing ### Benchmark Before/After ```bash # Get baseline rippled --unittest test.SHAMap > baseline.txt 2>&1 # Modify code... # Test after change rippled --unittest test.SHAMap > modified.txt 2>&1 # Compare diff baseline.txt modified.txt ``` ### Load Testing ```bash # Submit transactions and measure ./load_test.sh --transactions 1000 --duration 60 # Monitor metrics watch -n 1 'rippled-cli server_info | jq ".result.node_db"' ``` ## Common Issues and Solutions | Issue | Investigation | Solution | | -------------------- | -------------------------- | --------------------------------- | | High cache miss rate | Cache metrics | Increase cache\_size | | Slow sync | Fetch latency | Increase async\_threads | | Disk full | df -h | Enable online\_delete | | Memory leak | Valgrind | Fix code (likely nodes not freed) | | Hang on startup | strace | Check database corruption | | Consensus failing | Logs for validation errors | Check NodeStore consistency | *** See Appendix A for codebase navigation to find files mentioned here.