Package com.linkedin.davinci.store.rocksdb

Class ReplicationMetadataRocksDBStoragePartition

java.lang.Object

com.linkedin.davinci.store.AbstractStoragePartition

com.linkedin.davinci.store.rocksdb.RocksDBStoragePartition

com.linkedin.davinci.store.rocksdb.ReplicationMetadataRocksDBStoragePartition

public class ReplicationMetadataRocksDBStoragePartition
extends RocksDBStoragePartition

This ReplicationMetadataRocksDBStoragePartition is built to store key value pair along with the timestamp metadata. It is designed for active/active replication mode, which uses putWithReplicationMetadata and getReplicationMetadata to insert and retrieve replication metadata associated with a key. The implementation relies on different column family in RocksDB to isolate the value and replication metadata of a key.

Field Summary

Fields inherited from class com.linkedin.davinci.store.rocksdb.RocksDBStoragePartition

blobTransferInProgress, columnFamilyDescriptors, columnFamilyHandleList, deferredWrite, iteratorReadOptions, partitionId, READ_OPTIONS_DEFAULT, readCloseRWLock, readOnly, readWriteLeaderForDefaultCF, readWriteLeaderForRMDCF, replicaId, rocksDB, ROCKSDB_ERROR_MESSAGE_FOR_RUNNING_OUT_OF_DISK_QUOTA, storeName, storeNameAndVersion, storeVersion, writeOnly, writeOptions

Constructor Summary

Constructors

Constructor	Description
ReplicationMetadataRocksDBStoragePartition(StoragePartitionConfig storagePartitionConfig, RocksDBStorageEngineFactory factory, String dbDir, RocksDBMemoryStats rocksDBMemoryStats, RocksDBThrottler rocksDbThrottler, RocksDBServerConfig rocksDBServerConfig)

Method Summary

Modifier and Type	Method	Description
void	beginBatchWrite(Map<String,String> checkpointedInfo, Optional<Supplier<byte[]>> expectedChecksumSupplier)
boolean	checkDatabaseIntegrity(Map<String,String> checkpointedInfo)	checks whether the current state of the database is valid during the start of ingestion.
void	close()	Close the specific partition
void	deleteWithReplicationMetadata(byte[] key, byte[] replicationMetadata)	This API deletes a record from RocksDB but updates the metadata in ByteBuffer format and puts it into RocksDB.
void	drop()	Drop when it is not required anymore.
void	endBatchWrite()
String	getFullPathForTempSSTFileDir()
byte[]	getReplicationMetadata(ByteBuffer key)	This API retrieves replication metadata from replicationMetadataColumnFamily.
long	getRmdByteUsage()
RocksDBSstFileWriter	getRocksDBSstFileWriter()
String	getValueFullPathForTempSSTFileDir()
RocksDBSstFileWriter	getValueRocksDBSstFileWriter()
void	putReplicationMetadata(byte[] key, byte[] metadata)
void	putWithReplicationMetadata(byte[] key, byte[] value, byte[] metadata)	This API takes in value and metadata as byte array format and put it into RocksDB.
void	putWithReplicationMetadata(byte[] key, ByteBuffer value, byte[] metadata)	This API takes in value and metadata as ByteBuffer format and put it into RocksDB.
Map<String,String>	sync()	Sync current database.
boolean	validateBatchIngestion()

Methods inherited from class com.linkedin.davinci.store.rocksdb.RocksDBStoragePartition

checkAndThrowDiskLimitException, cleanupSnapshot, cleanupSnapshot, createSnapshot, createSnapshot, delete, deleteFilesInDirectory, get, get, get, get, getApproximateMemoryUsageByType, getByKeyPrefix, getColumnFamilyHandleList, getEnvOptions, getIterator, getKeyCountEstimate, getOptions, getPartitionSizeInBytes, getRocksDBStatValue, getStoreOptions, isRocksDBPartitionBlobTransferInProgress, makeSureRocksDBIsStillOpen, multiGet, multiGet, put, put, put, reopen, verifyConfig

Methods inherited from class com.linkedin.davinci.store.AbstractStoragePartition

getPartitionId

Methods inherited from class java.lang.Object

clone, equals, finalize, getClass, hashCode, notify, notifyAll, toString, wait, wait, wait

Constructor Details

ReplicationMetadataRocksDBStoragePartition

public ReplicationMetadataRocksDBStoragePartition(StoragePartitionConfig storagePartitionConfig,
 RocksDBStorageEngineFactory factory,
 String dbDir,
 RocksDBMemoryStats rocksDBMemoryStats,
 RocksDBThrottler rocksDbThrottler,
 RocksDBServerConfig rocksDBServerConfig)

Method Details

putWithReplicationMetadata

public void putWithReplicationMetadata(byte[] key,
 byte[] value,
 byte[] metadata)

Description copied from class: AbstractStoragePartition

This API takes in value and metadata as byte array format and put it into RocksDB. Only ReplicationMetadataRocksDBStoragePartition will execute this method, other storage partition implementation will VeniceUnsupportedOperationException.

Overrides:

putWithReplicationMetadata in class AbstractStoragePartition

putReplicationMetadata

public void putReplicationMetadata(byte[] key,
 byte[] metadata)

Overrides:

putReplicationMetadata in class AbstractStoragePartition

getRmdByteUsage

public long getRmdByteUsage()

Overrides:

getRmdByteUsage in class RocksDBStoragePartition

putWithReplicationMetadata

public void putWithReplicationMetadata(byte[] key,
 ByteBuffer value,
 byte[] metadata)

This API takes in value and metadata as ByteBuffer format and put it into RocksDB. Note that it is not an efficient implementation as it copies the content to perform the ByteBuffer -> byte[] conversion. TODO: Rewrite this implementation after we adopt the thread-local direct bytebuffer approach.

Overrides:

putWithReplicationMetadata in class AbstractStoragePartition

getReplicationMetadata

public byte[] getReplicationMetadata(ByteBuffer key)

Description copied from class: AbstractStoragePartition

This API retrieves replication metadata from replicationMetadataColumnFamily. Only ReplicationMetadataRocksDBStoragePartition will execute this method, other storage partition implementation will VeniceUnsupportedOperationException.

Overrides:

getReplicationMetadata in class AbstractStoragePartition

deleteWithReplicationMetadata

public void deleteWithReplicationMetadata(byte[] key,
 byte[] replicationMetadata)

This API deletes a record from RocksDB but updates the metadata in ByteBuffer format and puts it into RocksDB.

Overrides:

deleteWithReplicationMetadata in class AbstractStoragePartition

checkDatabaseIntegrity

public boolean checkDatabaseIntegrity(Map<String,String> checkpointedInfo)

Description copied from class: AbstractStoragePartition

checks whether the current state of the database is valid during the start of ingestion.

Overrides:

checkDatabaseIntegrity in class RocksDBStoragePartition

beginBatchWrite

public void beginBatchWrite(Map<String,String> checkpointedInfo,
 Optional<Supplier<byte[]>> expectedChecksumSupplier)

Overrides:

beginBatchWrite in class RocksDBStoragePartition

endBatchWrite

public void endBatchWrite()

Overrides:

endBatchWrite in class RocksDBStoragePartition

sync

public Map<String,String> sync()

Description copied from class: AbstractStoragePartition

Sync current database.

Overrides:

sync in class RocksDBStoragePartition

Returns:

Database related info, which is required to be checkpointed.

close

public void close()

Description copied from class: AbstractStoragePartition

Close the specific partition

Overrides:

close in class RocksDBStoragePartition

validateBatchIngestion

public boolean validateBatchIngestion()

Overrides:

validateBatchIngestion in class RocksDBStoragePartition

drop

public void drop()

Description copied from class: AbstractStoragePartition

Drop when it is not required anymore.

Overrides:

drop in class RocksDBStoragePartition

getFullPathForTempSSTFileDir

public String getFullPathForTempSSTFileDir()

Overrides:

getFullPathForTempSSTFileDir in class RocksDBStoragePartition

getRocksDBSstFileWriter

public RocksDBSstFileWriter getRocksDBSstFileWriter()

Overrides:

getRocksDBSstFileWriter in class RocksDBStoragePartition

getValueFullPathForTempSSTFileDir

public String getValueFullPathForTempSSTFileDir()

getValueRocksDBSstFileWriter

public RocksDBSstFileWriter getValueRocksDBSstFileWriter()