Package com.linkedin.davinci.store.rocksdb

Class RocksDBSstFileWriter

  • public class RocksDBSstFileWriter
extends java.lang.Object

    • Constructor Summary

      Constructors 
      Constructor Description
      RocksDBSstFileWriter​(java.lang.String storeName, int partitionId, java.lang.String dbDir, org.rocksdb.EnvOptions envOptions, org.rocksdb.Options options, java.lang.String fullPathForTempSSTFileDir, boolean isRMD, RocksDBServerConfig rocksDBServerConfig, boolean blobTransferEnabled)  

    • Method Summary

      All Methods Instance Methods Concrete Methods 
      Modifier and Type Method Description
      void close()  
      protected org.rocksdb.Checkpoint createCheckpoint​(org.rocksdb.RocksDB rocksDB)  
      void createSnapshot​(org.rocksdb.RocksDB rocksDB)  
      java.lang.String getLastCheckPointedSSTFileNum()  
      long getRecordNumInAllSSTFiles()  
      void ingestSSTFiles​(org.rocksdb.RocksDB rocksDB, java.util.List<org.rocksdb.ColumnFamilyHandle> columnFamilyHandleList)  
      void open​(java.util.Map<java.lang.String,​java.lang.String> checkpointedInfo, java.util.Optional<java.util.function.Supplier<byte[]>> expectedChecksumSupplier)  
      void put​(byte[] key, java.nio.ByteBuffer valueBuffer)  
      java.util.Map<java.lang.String,​java.lang.String> sync()
      Closes currentSSTFileWriter, update lastCheckPointedSSTFileNum with the current SST file number, validates checksum on this SST file and return updated checkpointingInfo with this lastCheckPointedSSTFileNum.
      boolean validateBatchIngestion()  

      • Methods inherited from class java.lang.Object

        clone, equals, finalize, getClass, hashCode, notify, notifyAll, toString, wait, wait, wait

    • Field Detail

      • ROCKSDB_LAST_FINISHED_SST_FILE_NO

        protected static final java.lang.String ROCKSDB_LAST_FINISHED_SST_FILE_NO
        This field is being stored during offset checkpointing in StoreIngestionTask. With the field, RocksDB could recover properly during restart. Essentially, during recovery, this class will remove all the un-committed files after ROCKSDB_LAST_FINISHED_SST_FILE_NO, and start a new file with no: ROCKSDB_LAST_FINISHED_SST_FILE_NO + 1. With this way, we could achieve exact-once ingestion, which is required by SstFileWriter.
        See Also:
        Constant Field Values

      • ROCKSDB_LAST_FINISHED_RMD_SST_FILE_NO

        protected static final java.lang.String ROCKSDB_LAST_FINISHED_RMD_SST_FILE_NO
        See Also:
        Constant Field Values

      • DEFAULT_COLUMN_FAMILY_INDEX

        protected static final int DEFAULT_COLUMN_FAMILY_INDEX
        See Also:
        Constant Field Values

      • REPLICATION_METADATA_COLUMN_FAMILY_INDEX

        protected static final int REPLICATION_METADATA_COLUMN_FAMILY_INDEX
        See Also:
        Constant Field Values

    • Constructor Detail

      • RocksDBSstFileWriter

        public RocksDBSstFileWriter​(java.lang.String storeName,
                            int partitionId,
                            java.lang.String dbDir,
                            org.rocksdb.EnvOptions envOptions,
                            org.rocksdb.Options options,
                            java.lang.String fullPathForTempSSTFileDir,
                            boolean isRMD,
                            RocksDBServerConfig rocksDBServerConfig,
                            boolean blobTransferEnabled)

    • Method Detail

      • createCheckpoint

        protected org.rocksdb.Checkpoint createCheckpoint​(org.rocksdb.RocksDB rocksDB)

      • getLastCheckPointedSSTFileNum

        public java.lang.String getLastCheckPointedSSTFileNum()

      • put

        public void put​(byte[] key,
                java.nio.ByteBuffer valueBuffer)
         throws org.rocksdb.RocksDBException
        Throws:
        org.rocksdb.RocksDBException

      • open

        public void open​(java.util.Map<java.lang.String,​java.lang.String> checkpointedInfo,
                 java.util.Optional<java.util.function.Supplier<byte[]>> expectedChecksumSupplier)

      • close

        public void close()

      • sync

        public java.util.Map<java.lang.String,​java.lang.String> sync()
        Closes currentSSTFileWriter, update lastCheckPointedSSTFileNum with the current SST file number, validates checksum on this SST file and return updated checkpointingInfo with this lastCheckPointedSSTFileNum.

      • validateBatchIngestion

        public boolean validateBatchIngestion()

      • ingestSSTFiles

        public void ingestSSTFiles​(org.rocksdb.RocksDB rocksDB,
                           java.util.List<org.rocksdb.ColumnFamilyHandle> columnFamilyHandleList)

      • createSnapshot

        public void createSnapshot​(org.rocksdb.RocksDB rocksDB)

      • getRecordNumInAllSSTFiles

        public long getRecordNumInAllSSTFiles()