Package com.linkedin.davinci.store.rocksdb

Class RocksDBSstFileWriter

java.lang.Object

com.linkedin.davinci.store.rocksdb.RocksDBSstFileWriter

public class RocksDBSstFileWriter
extends Object

Field Summary

Fields

Modifier and Type	Field	Description
protected static final int	DEFAULT_COLUMN_FAMILY_INDEX
protected static final int	REPLICATION_METADATA_COLUMN_FAMILY_INDEX
protected static final String	ROCKSDB_LAST_FINISHED_RMD_SST_FILE_NO
protected static final String	ROCKSDB_LAST_FINISHED_SST_FILE_NO	This field is being stored during offset checkpointing in StoreIngestionTask.

Constructor Summary

Constructors

Constructor	Description
RocksDBSstFileWriter(String storeName, int partitionId, String dbDir, org.rocksdb.EnvOptions envOptions, org.rocksdb.Options options, String fullPathForTempSSTFileDir, boolean isRMD, RocksDBServerConfig rocksDBServerConfig)

Method Summary

Modifier and Type	Method	Description
void	close()
protected org.rocksdb.Checkpoint	createCheckpoint(org.rocksdb.RocksDB rocksDB)
String	getLastCheckPointedSSTFileNum()
long	getRecordNumInAllSSTFiles()
void	ingestSSTFiles(org.rocksdb.RocksDB rocksDB, List<org.rocksdb.ColumnFamilyHandle> columnFamilyHandleList)
void	open(Map<String,String> checkpointedInfo, Optional<Supplier<byte[]>> expectedChecksumSupplier)
void	put(byte[] key, ByteBuffer valueBuffer)
Map<String,String>	sync()	Closes currentSSTFileWriter, update lastCheckPointedSSTFileNum with the current SST file number, validates checksum on this SST file and return updated checkpointingInfo with this lastCheckPointedSSTFileNum.
boolean	validateBatchIngestion()

Methods inherited from class java.lang.Object

clone, equals, finalize, getClass, hashCode, notify, notifyAll, toString, wait, wait, wait

Field Details

ROCKSDB_LAST_FINISHED_SST_FILE_NO

protected static final String ROCKSDB_LAST_FINISHED_SST_FILE_NO

This field is being stored during offset checkpointing in StoreIngestionTask. With the field, RocksDB could recover properly during restart. Essentially, during recovery, this class will remove all the un-committed files after ROCKSDB_LAST_FINISHED_SST_FILE_NO, and start a new file with no: ROCKSDB_LAST_FINISHED_SST_FILE_NO + 1. With this way, we could achieve exact-once ingestion, which is required by SstFileWriter.

See Also:

• Constant Field Values

ROCKSDB_LAST_FINISHED_RMD_SST_FILE_NO

protected static final String ROCKSDB_LAST_FINISHED_RMD_SST_FILE_NO

See Also:

• Constant Field Values

DEFAULT_COLUMN_FAMILY_INDEX

protected static final int DEFAULT_COLUMN_FAMILY_INDEX

See Also:

• Constant Field Values

REPLICATION_METADATA_COLUMN_FAMILY_INDEX

protected static final int REPLICATION_METADATA_COLUMN_FAMILY_INDEX

See Also:

• Constant Field Values

Constructor Details

RocksDBSstFileWriter

public RocksDBSstFileWriter(String storeName,
 int partitionId,
 String dbDir,
 org.rocksdb.EnvOptions envOptions,
 org.rocksdb.Options options,
 String fullPathForTempSSTFileDir,
 boolean isRMD,
 RocksDBServerConfig rocksDBServerConfig)

Method Details

createCheckpoint

protected org.rocksdb.Checkpoint createCheckpoint(org.rocksdb.RocksDB rocksDB)

getLastCheckPointedSSTFileNum

public String getLastCheckPointedSSTFileNum()

put

public void put(byte[] key,
 ByteBuffer valueBuffer)
         throws org.rocksdb.RocksDBException

Throws:

org.rocksdb.RocksDBException

open

public void open(Map<String,String> checkpointedInfo,
 Optional<Supplier<byte[]>> expectedChecksumSupplier)

close

public void close()

sync

public Map<String,String> sync()

Closes currentSSTFileWriter, update lastCheckPointedSSTFileNum with the current SST file number, validates checksum on this SST file and return updated checkpointingInfo with this lastCheckPointedSSTFileNum.

validateBatchIngestion

public boolean validateBatchIngestion()

ingestSSTFiles

public void ingestSSTFiles(org.rocksdb.RocksDB rocksDB,
 List<org.rocksdb.ColumnFamilyHandle> columnFamilyHandleList)

getRecordNumInAllSSTFiles

public long getRecordNumInAllSSTFiles()