> For the complete documentation index, see [llms.txt](https://docs.xrpl-commons.org/llms.txt). Markdown versions of documentation pages are available by appending `.md` to page URLs; this page is available as [Markdown](https://docs.xrpl-commons.org/core-dev-bootcamp/module03/nodestore-architecture.md). # The NodeStore [← Back to SHAMap and NodeStore: Data Persistence and State Management](/core-dev-bootcamp/module03.md) *** ### Introduction Now that you understand SHAMap's in-memory structure and efficient algorithms, we turn to the critical question: **how do you persist this state?** Without persistence, every validator restart requires replaying transactions from genesis—weeks of computation. With persistence, recovery takes minutes. But persistence introduces challenges: * **Scale**: Millions of nodes, gigabytes of data * **Performance**: Database queries are 1000x slower than memory access * **Flexibility**: Different operators need different storage engines * **Reliability**: Data must survive crashes without corruption The NodeStore solves all these problems through elegant abstraction and careful design. {% embed url="" %} ### NodeStore's Role in XRPL Architecture NodeStore sits at a critical junction in XRPL's architecture: ``` Application Layer | v SHAMap (In-Memory State) | v NodeStore Interface (Abstraction) | v Cache Layer (TaggedCache) -- Hot Data in Memory | v Backend Abstraction (Interface) -- Multiple implementations | v Database Implementation (RocksDB, NuDB, etc.) | v Physical Storage (Disk) ``` **SHAMap's Dependency:** SHAMap needs to retrieve historical nodes: ```cpp // During synchronization or historical queries: std::shared_ptr node = nodestore.fetch(nodeHash); ``` But SHAMap doesn't know or care about: * How data is stored * Which database backend is used * Where the data is physically located * How caching is implemented All that complexity is hidden behind NodeStore's interface. ### Core Purpose NodeStore provides four critical services: **1. Persistence** ``` SHAMap state exists in memory ↓ Serialize nodes to disk ↓ Survive application crash ↓ Reconstruct state on startup ``` **2. Consistent Interface** ``` // Application code doesn't change regardless of backend nodestore.store(node); // Works with RocksDB, NuDB, SQLite... auto node = nodestore.fetch(hash); ``` **3. Performance Optimization** ``` Database queries: 1-10 milliseconds Memory access: 1-10 microseconds 1000x difference! NodeStore uses caching to keep hot data in memory Typical hit rate: 90-95% Result: Average latency near memory speed ``` **4. Lifecycle Management** ``` Startup: Locate and open database Runtime: Store and retrieve nodes as needed Shutdown: Cleanly close database Rotation: Enable online deletion and archival ``` ### NodeObject: The Fundamental Storage Unit The atomic unit of storage in XRPL is the NodeObject: **Structure:** ```cpp class NodeObject { // Type of object (LEDGER_HEADER, ACCOUNT_NODE, TRANSACTION_NODE) NodeObjectType mType; // 256-bit unique identifier uint256 mHash; // Serialized content (variable length) Blob mData; public: // Factory: create NodeObject from components static std::shared_ptr createObject( NodeObjectType type, Blob const& data, uint256 const& hash); // Access methods NodeObjectType getType() const { return mType; } uint256 const& getHash() const { return mHash; } Blob const& getData() const { return mData; } }; ``` **Key Characteristics:** 1. **Immutable Once Created**: Cannot modify data after creation 2. **Hash as Key**: Hash uniquely identifies the object 3. **Type Distinguishing**: Type prevents hash collisions between different data types 4. **Serialized Format**: Data is already in wire format **NodeObject Types:** | Type | Purpose | Numeric Value | | -------------------- | -------------------------------- | ------------- | | hotLEDGER | Ledger headers and metadata | 1 | | hotACCOUNT\_NODE | Account state tree nodes | 3 | | hotTRANSACTION\_NODE | Transaction tree nodes | 4 | | hotUNKNOWN | Unknown/unrecognized types | 0 | | hotDUMMY | Cache marker for missing entries | 512 | **Type Prefix in Hashing:** Type fields prevent collisions: ```cpp // Two different types of data, might have same structure // Type prefix ensures different hashes uint256 hash_account = SHA512Half( ACCOUNT_TYPE_BYTE || accountData); uint256 hash_transaction = SHA512Half( TRANSACTION_TYPE_BYTE || accountData); // hash_account != hash_transaction ``` ### NodeObject Lifecycle **Creation** ``` During transaction processing: 1. Transaction validated and applied to SHAMap 2. SHAMap nodes modified 3. Each modified node serialized 4. NodeObject created with type, hash, serialized data 5. Stored in NodeStore ``` **Storage** ``` For each NodeObject: 1. Encode to persistent format 2. Compute key (same as hash) 3. Write to database 4. Backend handles actual I/O ``` **Caching** ``` After storage: 1. Keep in memory for fast reaccess 2. Move to cache tier 3. Evict when cache capacity exceeded 4. Dummy objects mark "known missing" (avoid repeated lookups) ``` **Retrieval** ``` When SHAMap needs a node: 1. Check cache (microseconds) 2. If miss, query database (milliseconds) 3. Deserialize and validate 4. Add to cache 5. Return to SHAMap ``` **Archival** ``` After ledger is validated and no longer current: 1. May be retained for history 2. Or moved to archive during rotation 3. Or deleted based on retention policy ``` ### Backend Abstraction The Backend class defines the minimal interface for any storage system: **Core Operations:** ```cpp class Backend { // Store single object virtual Status store(NodeObject const& object) = 0; // Retrieve single object by hash virtual Status fetch(uint256 const& hash, std::shared_ptr& object) = 0; // Persist multiple objects atomically virtual Status storeBatch(std::vector const& batch) = 0; // Retrieve multiple objects efficiently virtual Status fetchBatch(std::vector const& hashes, std::vector& objects) = 0; // Lifecycle virtual Status open(std::string const& path) = 0; virtual Status close() = 0; virtual int fdRequired() const = 0; // File descriptors needed }; ``` **Status Codes:** ```cpp enum class Status { ok, // Operation succeeded notFound, // Key doesn't exist dataCorrupt, // Data integrity check failed (fatal) backendError // Backend error }; ``` **Backend Independence:** NodeStore sits above backends, application logic unchanged: ```cpp // Same code works with any backend struct DatabaseConfig { std::string type; // "rocksdb", "nudb", "sqlite", etc. std::string path; // ... backend-specific options }; auto backend = createBackend(config); NodeStore store(backend); // Application uses NodeStore store.fetch(hash); // Works regardless of backend store.store(node); ``` ### Supported Backends **RocksDB (Recommended for Most Cases)** ```cpp Backend* createRocksDBBackend(std::string const& path) { return new RocksDBBackend(path); } ``` * Modern key-value store developed by Facebook * LSM tree (Log-Structured Merge tree) design * Excellent performance for XRPL workloads * Built-in compression support * Active maintenance **Characteristics:** * Write throughput: \~10,000-50,000 objects/second * Read throughput: \~100,000+ objects/second * Compression: Reduces disk space by 50-70% **NuDB (High-Throughput Alternative)** ```cpp Backend* createNuDBBackend(std::string const& path) { return new NuDBBackend(path); } ``` * Purpose-built for XRPL by Ripple * Append-only design optimized for SSD * Higher write throughput than RocksDB * Efficient space utilization **Characteristics:** * Write throughput: \~50,000-200,000 objects/second * Read throughput: \~100,000+ objects/second * Better for high-volume systems **Testing Backends** ```cpp Backend* createMemoryBackend() { return new MemoryBackend(); // In-memory, non-persistent } Backend* createNullBackend() { return new NullBackend(); // No-op backend } ``` ### Data Encoding Format To enable backend independence, NodeStore uses a standardized encoding: **Encoded Blob Structure:** ``` Byte Offset | Field | Description 0-7 | Reserved | Set to zero, reserved for future use 8 | Type | NodeObjectType enumeration value 9+ | Data | Serialized object payload (variable length) ``` **Encoding Process:** ```cpp void encodeNodeObject(NodeObject const& obj, Blob& blob) { // Add 8 reserved bytes blob.resize(8, 0); // Add type byte blob.push_back(obj.getType()); // Add data payload blob.append(obj.getData()); } ``` **Decoding Process:** ```cpp std::shared_ptr decodeNodeObject( uint256 const& hash, Blob const& blob) { if (blob.size() < 9) { return nullptr; // Corrupted } NodeObjectType type = static_cast(blob[8]); Blob data(blob.begin() + 9, blob.end()); return NodeObject::createObject(type, data, hash); } ``` **Benefits:** 1. **Backend Agnostic**: Any backend can store/retrieve encoded blobs 2. **Self-Describing**: Type embedded, forward-compatible with unknown types 3. **Efficient**: Minimal overhead (8 bytes) per object 4. **Validated**: Type byte catches most corruption ### Database Key as Hash The database key is the object's hash (not a sequential ID): ```cpp Status Backend::store(NodeObject const& obj) { uint256 key = obj.getHash(); // 256-bit hash as key Blob value = encode(obj); // Encoded blob as value return database.put(key, value); // Key-value store } ``` **Implications:** 1. **Direct Retrieval**: Any node retrievable by hash 2. **Deduplication**: Identical content produces identical hash → same key 3. **Immutability**: Hash never changes for given data 4. **Verification**: Can verify data by recomputing hash ### Integration Architecture NodeStore integrates with SHAMap through the Family pattern: ```cpp // Family provides NodeStore access to SHAMap class Family { virtual std::shared_ptr getNodeStore() = 0; virtual std::shared_ptr getTreeNodeCache() = 0; virtual std::shared_ptr getFullBelowCache() = 0; }; class NodeFamily : public Family { std::shared_ptr mNodeStore; std::shared_ptr mTreeCache; std::shared_ptr mFullBelow; // ... implement Family interface }; // SHAMap uses Family for storage access class SHAMap { std::shared_ptr mFamily; std::shared_ptr getNode(uint256 const& hash) { // Try cache first auto cached = mFamily->getTreeNodeCache()->get(hash); if (cached) return cached; // Fetch from NodeStore auto obj = mFamily->getNodeStore()->fetch(hash); if (obj) { auto node = deserializeNode(obj); // Cache for future access mFamily->getTreeNodeCache()->insert(hash, node); return node; } return nullptr; } }; ``` ### Summary **Key Architectural Elements:** 1. **NodeObject**: Atomic storage unit (type, hash, data) 2. **Backend Interface**: Minimal, consistent interface for storage 3. **Abstraction**: Decouples application logic from storage implementation 4. **Encoding Format**: Standardized format enables backend independence 5. **Key as Hash**: Direct retrieval without index lookups 6. **Family Pattern**: Provides access to caching and storage **Design Properties:** * **Backend Flexibility**: Switch storage engines without code changes * **Scale**: Handles millions of objects efficiently * **Persistence**: Survives crashes and restarts * **Verification**: Data integrity through hashing * **Simplicity**: Minimal interface hides complexity In the next chapter, we'll explore the critical Cache Layer that makes NodeStore practical for high-performance systems.