# Appendix : Debugging and Development Tools [← Back to SHAMap and NodeStore: Data Persistence and State Management](/core-dev-bootcamp/module03.md) *** ### Introduction This appendix provides techniques and tools for investigating SHAMap and NodeStore behavior. ## Logging and Diagnostics ### Enable Verbose Logging Edit `rippled.cfg`: ```ini [rpc_startup] command = log_level severity = debug [logging] debug rpc ``` Then restart rippled and check logs: ```bash tail -f /var/log/rippled/rippled.log | grep -i nodestore ``` ### Key Log Messages ``` TRACE Ledger: Ledger opened/closed DEBUG SHAMap: Tree operations DEBUG NodeStore: Database operations INFO Consensus: Validation and agreement WARN Performance: Slow operations detected ``` ## Metrics Inspection ### JSON-RPC Inspection ```bash # Get storage metrics rippled-cli server_info | jq '.result.node_db' # Expected output: { "type": "RocksDB", "path": "/var/lib/rippled/db/rocksdb", "cache_size": 256, "cache_hit_rate": 0.923, "writes": 1000000, "bytes_written": 1000000000, "reads": 50000000, "cache_hits": 46150000, "read_latency_us": 15 } ``` ### File System Inspection ```bash # Check database size du -sh /var/lib/rippled/db/* # Monitor growth watch -n 1 'du -sh /var/lib/rippled/db/*' # Check free space df -h /var/lib/rippled/ # Monitor I/O iostat -x 1 /dev/sda ``` ## Debugging Specific Issues ### Issue: Cache Hit Rate Too Low **Symptoms:** * Database queries slow * Ledger close times increasing * Hit rate < 80% **Investigation:** ```bash # Check cache metrics rippled-cli server_info | jq '.result.node_db.cache_hit_rate' # Check cache size configuration grep cache_size rippled.cfg # Monitor cache evictions tail -f /var/log/rippled/rippled.log | \ grep -i "evict\|cache" ``` **Solutions:** 1. Increase `cache_size` if memory available 2. Reduce `cache_age` for faster eviction of cold data 3. Check if system is memory-constrained (use `free`) ### Issue: Write Performance Degradation **Symptoms:** * Ledger closes slow (>10 seconds) * Database write errors in logs * Validator falling behind network **Investigation:** ```bash # Check write latency rippled-cli server_info | \ jq '.result.node_db.write_latency_us' # Monitor disk I/O iotop -o -b -n 1 # Check disk space df -h # Monitor async queue tail /var/log/rippled/rippled.log | \ grep -i "async.*queue" ``` **Solutions:** 1. Ensure SSD (not HDD) for database 2. Check disk I/O isn't saturated 3. Increase `async_threads` if I/O bound 4. Switch to faster backend (NuDB vs RocksDB) 5. Enable compression if disk is bottleneck ### Issue: Synchronization Slow **Symptoms:** * New nodes take hours to sync * Falling behind network * High database query count **Investigation:** ```bash # Monitor sync progress rippled-cli server_info | jq '.result.ledger.ledger_index' # Track fetch operations tail -f /var/log/rippled/rippled.log | \ grep -i "fetch\|sync" # Monitor thread pool ps -p $(pidof rippled) -L # Check queue depths rippled-cli server_info | jq '.result.node_db.async_queue_depth' ``` **Solutions:** 1. Increase cache size for better hit rate during sync 2. Increase `async_threads` (more parallel fetches) 3. Use faster SSD 4. Check network bandwidth (might be bottleneck) 5. Switch to NuDB for higher throughput ## Code Debugging ### Building with Debug Symbols ```bash cd rippled mkdir build cd build cmake -DCMAKE_BUILD_TYPE=Debug .. make -j4 ``` ### GDB Debugging ```bash # Run under GDB gdb --args rippled --conf /path/to/rippled.cfg # Inside GDB: (gdb) run (gdb) break SHAMap::addKnownNode (gdb) continue # When breakpoint hit: (gdb) print node->getHash() (gdb) print nodeID (gdb) step (gdb) quit ``` ### Common Breakpoints ```cpp // Node addition break SHAMap::addKnownNode break Database::store // Cache operations break TaggedCache::get break TaggedCache::insert // Synchronization break SHAMap::getMissingNodes break NodeStore::fetchNodeObject ``` ### Print Useful Data ```cpp (gdb) print node->getHash().hex() (gdb) print nodeID.mDepth (gdb) print nodeID.mNodeID.hex() (gdb) print metrics.cacheHits (gdb) print metrics.cacheMisses ``` ## Performance Profiling ### CPU Profiling with Perf ```bash # Record 60 seconds of system behavior perf record -F 99 -p $(pidof rippled) -- sleep 60 # Analyze results perf report # Show flame graph perf record -F 99 -p $(pidof rippled) -- sleep 60 perf script | stackcollapse-perf.pl | flamegraph.pl > profile.svg ``` ### Memory Profiling with Valgrind ```bash # Run under memcheck (very slow) valgrind --leak-check=full rippled --conf rippled.cfg # Run specific test valgrind --leak-check=full rippled --unittest test.nodestore ``` ### Custom Instrumentation Add to rippled source: ```cpp // In Database::fetchNodeObject auto startTime = std::chrono::steady_clock::now(); auto obj = mBackend->fetch(hash); auto elapsed = std::chrono::steady_clock::now() - startTime; auto ms = std::chrono::duration_cast(elapsed); if (ms.count() > 100) { JLOG(mLog.warning()) << "Slow fetch: " << hash.hex() << " took " << ms.count() << "ms"; } ``` ## Test-Driven Debugging ### Running Unit Tests ```bash # Build with tests enabled cd rippled/build cmake -DENABLE_TESTING=ON .. # Run all tests ctest # Run specific test ctest -R "shamap" -V # Run single test file ./bin/rippled --unittest test.SHAMap ``` ### Writing Debug Tests ```cpp // In rippled/src/test/shamap_test.cpp SECTION("Debug specific case") { // Create SHAMap auto shamap = std::make_shared(...); // Add nodes shamap->addRootNode(...); // Test operation auto node = shamap->getNode(hash); // Assert behavior REQUIRE(node != nullptr); REQUIRE(node->getHash() == expectedHash); } ``` ## Useful Commands ### Check Configuration ```bash grep -E "^[a-z]|^\[" rippled.cfg | head -30 ``` ### Monitor in Real Time ```bash # CPU/Memory top -p $(pidof rippled) # Disk I/O iotop -p $(pidof rippled) # Network traffic netstat -an | grep ripple # File descriptors lsof -p $(pidof rippled) | wc -l ``` ### Database Inspection For RocksDB: ```bash # Use RocksDB tools rocksdb_ldb --db=/var/lib/rippled/db/rocksdb scan # List files ls -lah /var/lib/rippled/db/rocksdb/ ``` ### Log Analysis ```bash # Count errors grep ERROR /var/log/rippled/rippled.log | wc -l # Find slow operations grep "took.*ms" /var/log/rippled/rippled.log # Timeline of events tail -f /var/log/rippled/rippled.log | \ awk '{print $1" "$2" "$3" "$4" ..."}' ``` ## Performance Regression Testing ### Benchmark Before/After ```bash # Get baseline rippled --unittest test.SHAMap > baseline.txt 2>&1 # Modify code... # Test after change rippled --unittest test.SHAMap > modified.txt 2>&1 # Compare diff baseline.txt modified.txt ``` ### Load Testing ```bash # Submit transactions and measure ./load_test.sh --transactions 1000 --duration 60 # Monitor metrics watch -n 1 'rippled-cli server_info | jq ".result.node_db"' ``` ## Common Issues and Solutions | Issue | Investigation | Solution | | -------------------- | -------------------------- | --------------------------------- | | High cache miss rate | Cache metrics | Increase cache\_size | | Slow sync | Fetch latency | Increase async\_threads | | Disk full | df -h | Enable online\_delete | | Memory leak | Valgrind | Fix code (likely nodes not freed) | | Hang on startup | strace | Check database corruption | | Consensus failing | Logs for validation errors | Check NodeStore consistency | *** See Appendix A for codebase navigation to find files mentioned here. --- # Agent Instructions: Querying This Documentation If you need additional information that is not directly available in this page, you can query the documentation dynamically by asking a question. Perform an HTTP GET request on the current page URL with the `ask` query parameter: ``` GET https://docs.xrpl-commons.org/core-dev-bootcamp/module03/appendices/debugging.md?ask= ``` The question should be specific, self-contained, and written in natural language. The response will contain a direct answer to the question and relevant excerpts and sources from the documentation. Use this mechanism when the answer is not explicitly present in the current page, you need clarification or additional context, or you want to retrieve related documentation sections.