Package com.linkedin.davinci.store.rocksdb

Class ReplicationMetadataRocksDBStoragePartition

java.lang.Object

com.linkedin.davinci.store.AbstractStoragePartition

com.linkedin.davinci.store.rocksdb.RocksDBStoragePartition

com.linkedin.davinci.store.rocksdb.ReplicationMetadataRocksDBStoragePartition

public class ReplicationMetadataRocksDBStoragePartition
extends RocksDBStoragePartition

Extends RocksDBStoragePartition to store per-key replication metadata (RMD) alongside values using separate RocksDB column families. Used in active/active replication mode for deterministic conflict resolution (DCR). The RMD tracks record-level and field-level timestamps for conflict resolution.

                      Single RocksDB Instance

   CF: "default"                CF: "timestamp_metadata"
   +-------+-------------+      +-------+---------------------+
   | Key   | Value       |      | Key   | Replication MD      |
   +-------+-------------+      +-------+---------------------+
   | key_1 | value_1     |      | key_1 | rmd for key_1       |
   | key_2 | value_2     |      | key_2 | rmd for key_2       |
   | key_3 | (deleted)   |      | key_3 | rmd for key_3       |
   +-------+-------------+      +-------+---------------------+
 

Nested Class Summary

Nested classes/interfaces inherited from class com.linkedin.davinci.store.rocksdb.RocksDBStoragePartition

RocksDBStoragePartition.RocksDBOperation<T>, RocksDBStoragePartition.RocksDBVoidOperation

Field Summary

Fields inherited from class com.linkedin.davinci.store.rocksdb.RocksDBStoragePartition

blobDbEnabled, blobTransferInProgress, columnFamilyDescriptors, columnFamilyHandleList, deferredWrite, iteratorReadOptions, partitionId, READ_OPTIONS_DEFAULT, readCloseRWLock, readOnly, readWriteLeaderForDefaultCF, readWriteLeaderForRMDCF, replicaId, ROCKSDB_ERROR_MESSAGE_FOR_RUNNING_OUT_OF_DISK_QUOTA, storeName, storeNameAndVersion, storeVersion, writeOnly, writeOptions

Constructor Summary

Constructors

Constructor	Description
ReplicationMetadataRocksDBStoragePartition(StoragePartitionConfig storagePartitionConfig, RocksDBStorageEngineFactory factory, String dbDir, RocksDBMemoryStats rocksDBMemoryStats, RocksDBThrottler rocksDbThrottler, RocksDBServerConfig rocksDBServerConfig)

Method Summary

Modifier and Type	Method	Description
void	beginBatchWrite(Map<String,String> checkpointedInfo, Optional<Supplier<byte[]>> expectedChecksumSupplier)
boolean	checkDatabaseIntegrity(Map<String,String> checkpointedInfo)	checks whether the current state of the database is valid during the start of ingestion.
void	close()	Close the specific partition
void	deleteWithReplicationMetadata(byte[] key, byte[] replicationMetadata)	Deletes a key's value but updates its replication metadata.
void	drop()	Drop when it is not required anymore.
void	endBatchWrite()
String	getFullPathForTempSSTFileDir()
byte[]	getReplicationMetadata(ByteBuffer key)	This API retrieves replication metadata from replicationMetadataColumnFamily.
long	getRmdByteUsage()
RocksDBSstFileWriter	getRocksDBSstFileWriter()
String	getValueFullPathForTempSSTFileDir()
RocksDBSstFileWriter	getValueRocksDBSstFileWriter()
void	putReplicationMetadata(byte[] key, byte[] metadata)	Stores replication metadata for a key.
void	putWithReplicationMetadata(byte[] key, byte[] value, byte[] metadata)	Stores a key-value pair along with replication metadata.
void	putWithReplicationMetadata(byte[] key, ByteBuffer value, byte[] metadata)	This API takes in value and metadata as ByteBuffer format and put it into RocksDB.
Map<String,String>	sync()	Flushes the memtable to disk.
boolean	validateBatchIngestion()

Methods inherited from class com.linkedin.davinci.store.rocksdb.RocksDBStoragePartition

checkAndThrowDiskLimitException, cleanupSnapshot, cleanupSnapshot, createSnapshot, createSnapshot, delete, deleteFilesInDirectory, get, get, get, get, getApproximateMemoryUsageByType, getByKeyPrefix, getColumnFamilyHandleList, getEnvOptions, getIterator, getKeyCountEstimate, getOptions, getPartitionSizeInBytes, getRocksDBStatValue, getStoreOptions, isRocksDBPartitionBlobTransferInProgress, makeSureRocksDBIsStillOpen, multiGet, multiGet, put, put, put, reopen, verifyConfig, withOpenDatabase, withOpenDatabaseVoid, withSynchronizedDatabase, withSynchronizedDatabaseVoid

Methods inherited from class com.linkedin.davinci.store.AbstractStoragePartition

getPartitionId

Methods inherited from class java.lang.Object

clone, equals, finalize, getClass, hashCode, notify, notifyAll, toString, wait, wait, wait

Constructor Details

ReplicationMetadataRocksDBStoragePartition

public ReplicationMetadataRocksDBStoragePartition(StoragePartitionConfig storagePartitionConfig,
 RocksDBStorageEngineFactory factory,
 String dbDir,
 RocksDBMemoryStats rocksDBMemoryStats,
 RocksDBThrottler rocksDbThrottler,
 RocksDBServerConfig rocksDBServerConfig)

Method Details

putWithReplicationMetadata

public void putWithReplicationMetadata(byte[] key,
 byte[] value,
 byte[] metadata)

Stores a key-value pair along with replication metadata. In deferred-write mode, the value is written via the parent SST writer and metadata via this partition's SST writer. Otherwise, both are written atomically via a WriteBatch.

Overrides:

putWithReplicationMetadata in class AbstractStoragePartition

Throws:

VeniceException - if the database is closed, read-only, or the write fails

putReplicationMetadata

public void putReplicationMetadata(byte[] key,
 byte[] metadata)

Stores replication metadata for a key. In deferred-write mode, writes via the SST writer; otherwise writes directly to the replication metadata column family.

Overrides:

putReplicationMetadata in class AbstractStoragePartition

Throws:

VeniceException - if the database is closed, read-only, or the write fails

getRmdByteUsage

public long getRmdByteUsage()

Overrides:

getRmdByteUsage in class RocksDBStoragePartition

putWithReplicationMetadata

public void putWithReplicationMetadata(byte[] key,
 ByteBuffer value,
 byte[] metadata)

This API takes in value and metadata as ByteBuffer format and put it into RocksDB. Note that it is not an efficient implementation as it copies the content to perform the ByteBuffer -> byte[] conversion. TODO: Rewrite this implementation after we adopt the thread-local direct bytebuffer approach.

Overrides:

putWithReplicationMetadata in class AbstractStoragePartition

getReplicationMetadata

public byte[] getReplicationMetadata(ByteBuffer key)

Description copied from class: AbstractStoragePartition

This API retrieves replication metadata from replicationMetadataColumnFamily. Only ReplicationMetadataRocksDBStoragePartition will execute this method, other storage partition implementation will VeniceUnsupportedOperationException.

Overrides:

getReplicationMetadata in class AbstractStoragePartition

deleteWithReplicationMetadata

public void deleteWithReplicationMetadata(byte[] key,
 byte[] replicationMetadata)

Deletes a key's value but updates its replication metadata. In deferred-write mode (repush), only the metadata is written via the SST writer. Otherwise, the delete and metadata update are applied atomically via a WriteBatch.

Overrides:

deleteWithReplicationMetadata in class AbstractStoragePartition

Throws:

VeniceException - if the database is closed, read-only, or the write fails

checkDatabaseIntegrity

public boolean checkDatabaseIntegrity(Map<String,String> checkpointedInfo)

Description copied from class: AbstractStoragePartition

checks whether the current state of the database is valid during the start of ingestion.

Overrides:

checkDatabaseIntegrity in class RocksDBStoragePartition

beginBatchWrite

public void beginBatchWrite(Map<String,String> checkpointedInfo,
 Optional<Supplier<byte[]>> expectedChecksumSupplier)

Overrides:

beginBatchWrite in class RocksDBStoragePartition

endBatchWrite

public void endBatchWrite()

Overrides:

endBatchWrite in class RocksDBStoragePartition

sync

public Map<String,String> sync()

Description copied from class: RocksDBStoragePartition

Flushes the memtable to disk. In deferred-write mode, syncs the SST file writer instead.

Overrides:

sync in class RocksDBStoragePartition

Returns:

Database related info, which is required to be checkpointed.

close

public void close()

Description copied from class: AbstractStoragePartition

Close the specific partition

Overrides:

close in class RocksDBStoragePartition

validateBatchIngestion

public boolean validateBatchIngestion()

Overrides:

validateBatchIngestion in class RocksDBStoragePartition

drop

public void drop()

Description copied from class: AbstractStoragePartition

Drop when it is not required anymore.

Overrides:

drop in class RocksDBStoragePartition

getFullPathForTempSSTFileDir

public String getFullPathForTempSSTFileDir()

Overrides:

getFullPathForTempSSTFileDir in class RocksDBStoragePartition

getRocksDBSstFileWriter

public RocksDBSstFileWriter getRocksDBSstFileWriter()

Overrides:

getRocksDBSstFileWriter in class RocksDBStoragePartition

getValueFullPathForTempSSTFileDir

public String getValueFullPathForTempSSTFileDir()

getValueRocksDBSstFileWriter

public RocksDBSstFileWriter getValueRocksDBSstFileWriter()