# Resource Management and Performance Characteristics [← Back to SHAMap and NodeStore: Data Persistence and State Management](/core-dev-bootcamp/module03.md) *** ### Introduction Understanding SHAMap and NodeStore theoretically is one thing. Operating them in production is another. This final chapter covers: * Real-world resource requirements * Performance measurement and optimization * Bottleneck identification * Tuning for different deployment scenarios ### Lookup Complexity Analysis **SHAMap Lookup** ``` Operation: Find account by ID Worst case: O(64) node traversals Tree depth: 256 bits / 4 bits per level = 64 levels Typical case: O(1) Most accounts found before depth 64 Average depth in realistic ledger: ~25 levels Expected time: Each traversal: O(1) array access (branch[0..15]) Total: O(1) expected time (linear in actual tree size, but tree size ~ account count) Cache hit: 1-10 microseconds Direct pointer access, no I/O Cache miss: 1-10 milliseconds Database query required ``` **Batch Fetch** ``` N objects requested: Naive (sequential): N × database_latency = N × 10ms Example: 100 objects = 1000ms Batched: single_batch_latency + deserialize Example: 100 objects = 10ms + 5ms = 15ms Speedup: 66x ``` ### Write Throughput **Ledger Close Cycle** ``` Timeline for typical ledger close (3-5 seconds): 1. Receive transactions: 2 seconds - Validate signatures - Check preconditions - Execute in SHAMap 2. Consensus: 1 second - Reach agreement on state - Sign ledger 3. Store phase: 0.5-1 second - Serialize modified nodes - Write to NodeStore - Update indexes Total: 3.5-5 seconds ``` **Node Object Volume** ``` Typical ledger modification: 200-400 transactions per ledger Average 2-4 modified accounts per transaction = 500-1000 modified nodes Plus structural nodes (parent rehashing): Depth of modified accounts: ~25 levels = 25 ancestor nodes modified Total objects created: ~600-1100 per ledger At 4 ledgers/second: 2400-4400 objects/second Database requirement: RocksDB: Handles 10,000-50,000 obj/sec easily NuDB: Handles 50,000-200,000 obj/sec ``` **Write Latency** ``` Store a NodeObject: 1. Cache update: 1-10 microseconds 2. Encode to blob: 100 microseconds 3. Database write: 100-1000 microseconds (SSD) 4. Batch accumulation: 10-100 milliseconds Total for batch of 100: 10-100 milliseconds Per object in batch: 0.1-1 millisecond ``` ### Read Performance Characteristics **Cache Hit Scenario** ``` Hit rate: 95% (well-tuned system) 1000 object requests: 950 cache hits × 5 microseconds = 4.75 milliseconds 50 cache misses × 10 milliseconds = 500 milliseconds Total: 504.75 milliseconds = 0.5 seconds Average per request: 0.5 milliseconds ``` **Cache Miss Scenario** ``` Hit rate: 60% (poorly tuned system) 1000 object requests: 600 cache hits × 5 microseconds = 3 milliseconds 400 cache misses × 10 milliseconds = 4000 milliseconds = 4 seconds Total: 4.003 seconds Average per request: 4 milliseconds 10x slower due to cache misses! ``` ### Memory Requirements **NodeStore Memory** ``` Cache layer: Size: Configurable (32MB - 4GB typical) Per object: ~100-500 bytes At 256MB cache: ~300,000-500,000 cached objects Backend buffers: RocksDB: ~100-300MB for block cache NuDB: ~50-100MB Thread pools: Each async thread: ~1-2MB stack 10 threads: ~20MB Total NodeStore memory: cache_size + backend_buffers + thread_stacks Typical: 256MB cache + 200MB backend = 500MB total Large: 1GB cache + 300MB backend = 1.3GB total ``` **SHAMap Memory** ``` In-memory tree of current + recent ledgers: Active ledger: ~10-50MB Depends on account count and modification volume Recent immutable ledgers (kept for quick access): 2-3 most recent: ~30-150MB Total SHAMap: 50-200MB typical Plus cached nodes (shared with NodeStore cache): Counted above in NodeStore memory ``` **Total Memory Budget** ``` Minimal validator: SHAMap: 50MB NodeStore: 200MB Other rippled: 100MB Total: 350MB Standard validator: SHAMap: 100MB NodeStore: 500MB Other rippled: 100MB Total: 700MB Large validator: SHAMap: 200MB NodeStore: 2000MB Other rippled: 100MB Total: 2.3GB ``` ### Disk Space Requirements **Database Growth** ``` Without rotation: Unbounded growth Per ledger: ~600-1100 new objects per ledger ~200KB per object (with compression) = 120-220MB per ledger Per day: ~20 ledgers per day = 2.4-4.4 GB per day Per year: = 876GB - 1.6TB per year Clearly unsustainable (disk fills in weeks) ``` **With Rotation** ``` Retention policy: Keep last 100,000 ledgers Ledger creation rate: 1 ledger per ~3 seconds 100,000 ledgers = ~8 days of history Database size: 100,000 × 0.2MB = 20GB (stable) With overhead: 30-50GB typical Bounded growth enables indefinite operation ``` **Actual Sizes on Mainnet** ``` Small validator (RocksDB, compressed): Database: 30-50GB With binaries/logs: 60GB total Archive node (full history): Database: 500GB-1TB With redundancy: 1.5TB total Growth per day (with rotation): ~500MB-1GB per day (old data deleted as new data added) ``` ### File Descriptor Usage **File Descriptor Requirements** ``` Each backend type requires different FDs: RocksDB: - Main database: 1 - WAL (write-ahead log): 1 - SSTable files: 20-100 (per configuration) - Total: 25-100 FDs NuDB: - Main data file: 1 - Index file: 1 - Total: 2-5 FDs Operating system overhead: stdin, stdout, stderr: 3 Socket listening: 2-5 Network connections: ~50 typical Total rippled process: - Without NodeStore: 50-100 FDs - With RocksDB: 100-200 FDs - Comfortable limit: 4096 FDs Configuration: ulimit -n 4096 # Set FD limit ``` ### Performance Tuning **Identifying Bottlenecks** Monitor these metrics: ```cpp // Cache hit rate - most important if (metrics.hitRate() < 90%) { // Increase cache_size problem = "Cache too small"; } // Write latency - latency-sensitive if (metrics.writeLatency > 100ms) { // Switch to faster backend or increase batch size problem = "Backend I/O too slow"; } // Fetch latency if (metrics.fetchLatency > 50ms) { // Check cache hit rate // Check disk health problem = "Database queries too slow"; } // Async queue depth if (metrics.asyncQueueDepth > 10000) { // Not keeping up with demand problem = "Async processing overwhelmed"; } ``` **Tuning Parameters** ```ini [node_db] type = RocksDB path = /var/lib/rippled/db # Cache tuning cache_size = 256 # Increase if memory available cache_age = 60 # Longer = better hit rate # Threading async_threads = 4 # Increase for I/O-bound systems # Batch operations batch_write_size = 256 # Larger batches, fewer transactions [node_db_rotation] online_delete = 256 # Keep 256K ledgers (8 days) ``` **Scenario 1: High-Traffic Validator** ``` Problem: Write latency too high (ledgers close slowly) Solution: - Increase cache_size to 1GB+ - Switch to NuDB backend (higher throughput) - Increase async_threads to 8-16 - Ensure SSD (not HDD) - Increase batch_write_size Result: Write throughput 50K+ objects/sec ``` **Scenario 2: Memory-Constrained** ``` Problem: Only 512MB RAM available Solution: - Set cache_size = 64MB (small) - Still runs, but slower - Increase cache_age for working set - Monitor hit rate (may drop to 80%) Result: Functional but slower sync and queries ``` **Scenario 3: Archive Node** ``` Problem: Need complete history, very large disk Solution: - No rotation (online_delete disabled) - RocksDB with compression - Smaller cache_size (less frequently accessed) - Parallel database with rotated copy Result: Full history, terabyte+ database ``` ### Performance Characteristics Summary **Lookup Performance:** ``` Single object lookup: Cache hit: 1-10 microseconds Cache miss: 1-10 milliseconds 95% hit rate: ~0.5 milliseconds average Batch operation (100 objects): Sequential: 1000 milliseconds Batched: 10 milliseconds Speedup: 100x ``` **Write Performance:** ``` Per ledger: 1000 objects per ledger Per-object: 0.1-1 millisecond Batch overhead: 10-100 milliseconds Total per ledger: 100-1100 milliseconds Throughput: 4 ledgers/second × 1000 objects/ledger = 4000 obj/sec Well within RocksDB/NuDB capacity ``` **Memory Usage:** ``` Minimum: 200-300MB Typical: 500-700MB Large: 2-4GB Depends on cache_size configuration ``` **Disk Space:** ``` With rotation: 30-50GB (8-10 days history) Unbounded: ~1TB per year (without rotation) Growth rate: ~500MB-1GB per day ``` **Scalability Limits:** ``` Current network: 2000+ validators 100-400 transactions/ledger Proven sustainable Theoretical limits: Cache hit rate: 80%+ maintainable at any size Write throughput: 100K obj/sec possible Read throughput: 1M obj/sec with cache Practical limits: Memory: 4-16GB per validator typical Disk: 100GB-1TB per validator typical Network: Synchronization limits transaction volume Consensus: Agreement time limits throughput ``` ### Monitoring in Production **Key Metrics to Track** ``` 1. Cache Statistics: - Hit rate (target: >90%) - Size (should be close to configured max) - Eviction rate 2. Database Performance: - Write latency (target: <100ms per ledger) - Read latency (target: <50ms per request) - Queue depth (target: <1000) 3. Resource Usage: - Memory (should stabilize) - CPU (typically 20-50% on modern systems) - Disk I/O (peaks during sync) 4. Application: - Ledger close time (target: 3-5 seconds) - Synchronization lag (target: 0 when caught up) - Block proposal success (target: >95%) ``` **Alerting Thresholds** ``` Warning: - Hit rate < 80% - Write latency > 200ms - Queue depth > 5000 Critical: - Hit rate < 60% - Write latency > 500ms - Ledger close > 10 seconds - Disk space < 10GB free ``` --- # Agent Instructions: Querying This Documentation If you need additional information that is not directly available in this page, you can query the documentation dynamically by asking a question. Perform an HTTP GET request on the current page URL with the `ask` query parameter: ``` GET https://docs.xrpl-commons.org/core-dev-bootcamp/module03/resource-management.md?ask= ``` The question should be specific, self-contained, and written in natural language. The response will contain a direct answer to the question and relevant excerpts and sources from the documentation. Use this mechanism when the answer is not explicitly present in the current page, you need clarification or additional context, or you want to retrieve related documentation sections.