Package com.linkedin.davinci.store.rocksdb

Class RocksDBServerConfig

java.lang.Object
com.linkedin.davinci.store.rocksdb.RocksDBServerConfig
public class RocksDBServerConfig extends Object

  • Field Details

    • ROCKSDB_OPTIONS_USE_DIRECT_READS

      public static final String ROCKSDB_OPTIONS_USE_DIRECT_READS
      Ability to use direct IO for disk reads, might yield better performance on Azure disks. Also makes caching behavior more consistent, by limiting the caching to only RocksDB. This also reduces number of mem copies which might yield improved performance.
      See Also:

    • ROCKSDB_ENV_FLUSH_POOL_SIZE

      public static final String ROCKSDB_ENV_FLUSH_POOL_SIZE
      Thread pool being used by all the RocksDB databases. https://github.com/facebook/rocksdb/wiki/RocksDB-Tuning-Guide The maximum number of concurrent flush operations. It is usually good enough to set this to 1.
      See Also:

    • ROCKSDB_ENV_COMPACTION_POOL_SIZE

      public static final String ROCKSDB_ENV_COMPACTION_POOL_SIZE
      Thread pool being used by all the RocksDB databases. https://github.com/facebook/rocksdb/wiki/RocksDB-Tuning-Guide The maximum number of concurrent background compactions. The default is 1, but to fully utilize your CPU and storage you might want to increase this to the minimum of (the number of cores in the system, the disk throughput divided by the average throughput of one compaction thread).
      See Also:

    • ROCKSDB_OPTIONS_COMPRESSION_TYPE

      public static final String ROCKSDB_OPTIONS_COMPRESSION_TYPE
      Compression type, and please check this enum class to find out all the available options: CompressionType. For now, the default option is to disable compression, and use Venice native compression support if necessary.
      See Also:

    • ROCKSDB_OPTIONS_COMPACTION_STYLE

      public static final String ROCKSDB_OPTIONS_COMPACTION_STYLE
      Please check CompactionStyle to find all the available options.
      See Also:

    • ROCKSDB_BLOCK_CACHE_SIZE_IN_BYTES

      public static final String ROCKSDB_BLOCK_CACHE_SIZE_IN_BYTES
      Shared block cache across all the RocksDB databases.
      See Also:

    • ROCKSDB_RMD_BLOCK_CACHE_SIZE_IN_BYTES

      public static final String ROCKSDB_RMD_BLOCK_CACHE_SIZE_IN_BYTES
      Shared block cache used by RMD column family across all the RocksDB databases.
      See Also:

    • ROCKSDB_BLOCK_CACHE_COMPRESSED_SIZE_IN_BYTES

      public static final String ROCKSDB_BLOCK_CACHE_COMPRESSED_SIZE_IN_BYTES
      Shared block cache for compressed data.
      See Also:

    • ROCKSDB_BLOCK_CACHE_SHARD_BITS

      public static final String ROCKSDB_BLOCK_CACHE_SHARD_BITS
      number of bits to count cache shards, total shard count would be 2 to the power of this number.
      See Also:

    • ROCKSDB_BLOCK_CACHE_IMPLEMENTATION

      public static final String ROCKSDB_BLOCK_CACHE_IMPLEMENTATION
      the implementation of block cache that the venice server should use. For supported implementations @see RocksDBBlockCacheImplementations Defaults to LRU
      See Also:

    • ROCKSDB_BLOCK_CACHE_STRICT_CAPACITY_LIMIT

      public static final String ROCKSDB_BLOCK_CACHE_STRICT_CAPACITY_LIMIT
      if set to True, Cache size will strictly stay within set bounds, by allocating space for indexes and metadata within cache size. This needs to be set to true to make OHC behavior and memory sizing predictable.
      See Also:

    • ROCKSDB_SET_CACHE_INDEX_AND_FILTER_BLOCKS

      public static final String ROCKSDB_SET_CACHE_INDEX_AND_FILTER_BLOCKS
      If set to true, we will put index/filter blocks to the block cache. Otherwise, each "table reader" object will pre-load index/filter block during table initialization.
      See Also:

    • ROCKSDB_SST_FILE_BLOCK_SIZE_IN_BYTES

      public static final String ROCKSDB_SST_FILE_BLOCK_SIZE_IN_BYTES
      File block size, and this config has impact to the index size and read performance.
      See Also:

    • ROCKSDB_MEMTABLE_SIZE_IN_BYTES

      public static final String ROCKSDB_MEMTABLE_SIZE_IN_BYTES
      Max memtable size per database;
      See Also:

    • ROCKSDB_MAX_MEMTABLE_COUNT

      public static final String ROCKSDB_MAX_MEMTABLE_COUNT
      Max memtable count per database;
      See Also:

    • ROCKSDB_MIN_WRITE_BUFFER_NUMBER_TO_MERGE

      public static final String ROCKSDB_MIN_WRITE_BUFFER_NUMBER_TO_MERGE
      See Also:

    • ROCKSDB_MAX_TOTAL_WAL_SIZE_IN_BYTES

      public static final String ROCKSDB_MAX_TOTAL_WAL_SIZE_IN_BYTES
      Max total WAL log size per database;
      See Also:

    • ROCKSDB_MAX_BYTES_FOR_LEVEL_BASE

      public static final String ROCKSDB_MAX_BYTES_FOR_LEVEL_BASE
      Max size of level base, and by default the next level size will be 10 times bigger;
      See Also:

    • ROCKSDB_PLAIN_TABLE_FORMAT_ENABLED

      public static final String ROCKSDB_PLAIN_TABLE_FORMAT_ENABLED
      See Also:

    • CAPPED_PREFIX_EXTRACTOR_LENGTH

      public static final String CAPPED_PREFIX_EXTRACTOR_LENGTH
      Length of the capped prefix extractor used with RocksDB Plain Table. Be cautious when tweaking this config because it might prevent reading old data written with a different extractor length. Therefore, changing it requires wiping the server data and offset.
      See Also:

    • ROCKSDB_STORE_INDEX_IN_FILE

      public static final String ROCKSDB_STORE_INDEX_IN_FILE
      See Also:

    • ROCKSDB_HUGE_PAGE_TLB_SIZE

      public static final String ROCKSDB_HUGE_PAGE_TLB_SIZE
      See Also:

    • ROCKSDB_BLOOM_BITS_PER_KEY

      public static final String ROCKSDB_BLOOM_BITS_PER_KEY
      See Also:

    • ROCKSDB_HASH_TABLE_RATIO

      public static final String ROCKSDB_HASH_TABLE_RATIO
      See Also:

    • ROCKSDB_MAX_OPEN_FILES

      public static final String ROCKSDB_MAX_OPEN_FILES
      See Also:

    • ROCKSDB_TARGET_FILE_SIZE_IN_BYTES

      public static final String ROCKSDB_TARGET_FILE_SIZE_IN_BYTES
      Target file size, and this will only apply to hybrid store since batch-only push from VPJ is using SSTFileWriter directly.
      See Also:

    • ROCKSDB_MEM_TABLE_HUGE_PAGE_SIZE_BYTES

      public static final String ROCKSDB_MEM_TABLE_HUGE_PAGE_SIZE_BYTES
      Page size for huge page for the arena used by the memtable in rocksdb. If <=0, it won't allocate from huge page but from malloc. Users are responsible to reserve huge pages for it to be allocated. For example: sysctl -w vm.nr_hugepages=20 See linux doc Documentation/vm/hugetlbpage.txt If there isn't enough free huge page available, rocksdb will fall back to malloc.
      See Also:

    • ROCKSDB_BYTES_PER_SYNC

      public static final String ROCKSDB_BYTES_PER_SYNC
      Comments from rocksdb c++ code: Allows OS to incrementally sync files to disk while they are being written, asynchronously, in the background. This operation can be used to smooth out write I/Os over time. Users shouldn't rely on it for persistency guarantee. Issue one request for every bytes_per_sync written. 0 turns it off. Default: 0
      See Also:

    • ROCKSDB_STATISTICS_ENABLED

      public static final String ROCKSDB_STATISTICS_ENABLED
      Whether to enable rocksdb statistics. The reason to make it configurable is that there is about 5%-10% overhead by enabling statistics. https://github.com/facebook/rocksdb/wiki/Statistics
      See Also:

    • ROCKSDB_TOTAL_MEMTABLE_USAGE_CAP_IN_BYTES

      public static final String ROCKSDB_TOTAL_MEMTABLE_USAGE_CAP_IN_BYTES
      https://github.com/facebook/rocksdb/wiki/Write-Buffer-Manager The total memory usage cap of all the memtables for every RocksDB database.
      See Also:

    • ROCKSDB_MAX_FILE_OPENING_THREADS

      public static final String ROCKSDB_MAX_FILE_OPENING_THREADS
      Currently, the max file open thread cnt per open operation is 16 by default. We could tune this param to reduce the possible maximum thread cnt;
      See Also:

    • ROCKSDB_LEVEL0_FILE_NUM_COMPACTION_TRIGGER

      public static final String ROCKSDB_LEVEL0_FILE_NUM_COMPACTION_TRIGGER
      When the number of level-0 SST files reaches level0_slowdown_writes_trigger, writes are stalled. When the number of level-0 SST files reaches level0_stop_writes_trigger, writes are fully stopped to wait for level-0 to level-1 compaction reduce the number of level-0 files.
      See Also:

    • ROCKSDB_LEVEL0_SLOWDOWN_WRITES_TRIGGER

      public static final String ROCKSDB_LEVEL0_SLOWDOWN_WRITES_TRIGGER
      See Also:

    • ROCKSDB_LEVEL0_STOPS_WRITES_TRIGGER

      public static final String ROCKSDB_LEVEL0_STOPS_WRITES_TRIGGER
      See Also:

    • ROCKSDB_LEVEL0_FILE_NUM_COMPACTION_TRIGGER_WRITE_ONLY_VERSION

      public static final String ROCKSDB_LEVEL0_FILE_NUM_COMPACTION_TRIGGER_WRITE_ONLY_VERSION
      See Also:

    • ROCKSDB_LEVEL0_SLOWDOWN_WRITES_TRIGGER_WRITE_ONLY_VERSION

      public static final String ROCKSDB_LEVEL0_SLOWDOWN_WRITES_TRIGGER_WRITE_ONLY_VERSION
      See Also:

    • ROCKSDB_LEVEL0_STOPS_WRITES_TRIGGER_WRITE_ONLY_VERSION

      public static final String ROCKSDB_LEVEL0_STOPS_WRITES_TRIGGER_WRITE_ONLY_VERSION
      See Also:

    • ROCKSDB_LEVEL0_FILE_NUM_COMPACTION_TRIGGER_FOR_READ_WRITE_LEADER

      public static final String ROCKSDB_LEVEL0_FILE_NUM_COMPACTION_TRIGGER_FOR_READ_WRITE_LEADER
      See Also:

    • ROCKSDB_LEVEL0_SLOWDOWN_WRITES_TRIGGER_FOR_READ_WRITE_LEADER

      public static final String ROCKSDB_LEVEL0_SLOWDOWN_WRITES_TRIGGER_FOR_READ_WRITE_LEADER
      See Also:

    • ROCKSDB_LEVEL0_STOPS_WRITES_TRIGGER_FOR_READ_WRITE_LEADER

      public static final String ROCKSDB_LEVEL0_STOPS_WRITES_TRIGGER_FOR_READ_WRITE_LEADER
      See Also:

    • ROCKSDB_LEVEL0_COMPACTION_TUNING_FOR_READ_WRITE_LEADER_ENABLED

      public static final String ROCKSDB_LEVEL0_COMPACTION_TUNING_FOR_READ_WRITE_LEADER_ENABLED
      See Also:

    • ROCKSDB_PUT_REUSE_BYTE_BUFFER

      public static final String ROCKSDB_PUT_REUSE_BYTE_BUFFER
      See Also:

    • ROCKSDB_DB_OPEN_OPERATION_THROTTLE

      public static final String ROCKSDB_DB_OPEN_OPERATION_THROTTLE
      Every time, when RocksDB tries to open a database, it will spin up multiple threads to load the file metadata in parallel, and the application could hit the thread limit issue if there are many RocksDB open operations at the same time. The following config is used to throttle the RocksDB open operations.
      See Also:

    • ROCKSDB_WRITE_QUOTA_BYTES_PER_SECOND

      public static final String ROCKSDB_WRITE_QUOTA_BYTES_PER_SECOND
      Check the following link for more details: https://github.com/facebook/rocksdb/wiki/Rate-Limiter This is used to throttle flush and compaction.
      See Also:

    • ROCKSDB_AUTO_TUNED_RATE_LIMITER_ENABLED

      public static final String ROCKSDB_AUTO_TUNED_RATE_LIMITER_ENABLED
      See Also:

    • ROCKSDB_ATOMIC_FLUSH_ENABLED

      public static final String ROCKSDB_ATOMIC_FLUSH_ENABLED
      See Also:

    • ROCKSDB_SEPARATE_RMD_CACHE_ENABLED

      public static final String ROCKSDB_SEPARATE_RMD_CACHE_ENABLED
      See Also:

    • ROCKSDB_BLOCK_BASE_FORMAT_VERSION

      public static final String ROCKSDB_BLOCK_BASE_FORMAT_VERSION
      See Also:

    • ROCKSDB_MAX_LOG_FILE_NUM

      public static final String ROCKSDB_MAX_LOG_FILE_NUM
      See Also:

    • ROCKSDB_MAX_LOG_FILE_SIZE

      public static final String ROCKSDB_MAX_LOG_FILE_SIZE
      See Also:

    • RECORD_TRANSFORMER_VALUE_SCHEMA

      public static final String RECORD_TRANSFORMER_VALUE_SCHEMA
      See Also:

    • ROCKSDB_BLOB_FILES_ENABLED

      public static final String ROCKSDB_BLOB_FILES_ENABLED
      Check this page to find more details: https://github.com/facebook/rocksdb/wiki/BlobDB
      See Also:

    • ROCKSDB_MIN_BLOB_SIZE_IN_BYTES

      public static final String ROCKSDB_MIN_BLOB_SIZE_IN_BYTES
      See Also:

    • ROCKSDB_BLOB_FILE_SIZE_IN_BYTES

      public static final String ROCKSDB_BLOB_FILE_SIZE_IN_BYTES
      See Also:

    • ROCKSDB_BLOB_GARBAGE_COLLECTION_AGE_CUTOFF

      public static final String ROCKSDB_BLOB_GARBAGE_COLLECTION_AGE_CUTOFF
      See Also:

    • ROCKSDB_BLOB_GARBAGE_COLLECTION_FORCE_THRESHOLD

      public static final String ROCKSDB_BLOB_GARBAGE_COLLECTION_FORCE_THRESHOLD
      See Also:

    • ROCKSDB_BLOB_FILE_STARTING_LEVEL

      public static final String ROCKSDB_BLOB_FILE_STARTING_LEVEL
      See Also:

    • ROCKSDB_BLOCK_CACHE_MEMORY_LIMIT

      public static final String ROCKSDB_BLOCK_CACHE_MEMORY_LIMIT
      See Also:

    • ROCKSDB_ITERATOR_READ_AHEAD_SIZE_IN_BYTES

      public static final String ROCKSDB_ITERATOR_READ_AHEAD_SIZE_IN_BYTES
      When this config is set to a value > 0, the RocksDB iterator will pre-fetch data asynchronously leading to better iteration performance. From testing, setting this to a value larger than 2MB doesn't result in any performance gain. Hypothetically, if the records are large, setting this to a higher number may see noticeable gains.
      See Also:

  • Constructor Details

  • Method Details

    • getLevel0FileNumCompactionTriggerWriteOnlyVersion

      public int getLevel0FileNumCompactionTriggerWriteOnlyVersion()

    • getLevel0SlowdownWritesTriggerWriteOnlyVersion

      public int getLevel0SlowdownWritesTriggerWriteOnlyVersion()

    • getLevel0StopWritesTriggerWriteOnlyVersion

      public int getLevel0StopWritesTriggerWriteOnlyVersion()

    • getLevel0FileNumCompactionTrigger

      public int getLevel0FileNumCompactionTrigger()

    • getLevel0SlowdownWritesTrigger

      public int getLevel0SlowdownWritesTrigger()

    • getLevel0StopWritesTrigger

      public int getLevel0StopWritesTrigger()

    • getLevel0FileNumCompactionTriggerForReadWriteLeader

      public int getLevel0FileNumCompactionTriggerForReadWriteLeader()

    • getLevel0SlowdownWritesTriggerForReadWriteLeader

      public int getLevel0SlowdownWritesTriggerForReadWriteLeader()

    • getLevel0StopWritesTriggerForReadWriteLeader

      public int getLevel0StopWritesTriggerForReadWriteLeader()

    • isLevel0CompactionTuningForReadWriteLeaderEnabled

      public boolean isLevel0CompactionTuningForReadWriteLeaderEnabled()

    • getRocksDBUseDirectReads

      public boolean getRocksDBUseDirectReads()

    • getRocksDBEnvFlushPoolSize

      public int getRocksDBEnvFlushPoolSize()

    • getRocksDBEnvCompactionPoolSize

      public int getRocksDBEnvCompactionPoolSize()

    • getRocksDBOptionsCompressionType

      public org.rocksdb.CompressionType getRocksDBOptionsCompressionType()

    • getRocksDBOptionsCompactionStyle

      public org.rocksdb.CompactionStyle getRocksDBOptionsCompactionStyle()

    • getRocksDBBlockCacheSizeInBytes

      public long getRocksDBBlockCacheSizeInBytes()

    • getRocksDBRMDBlockCacheSizeInBytes

      public long getRocksDBRMDBlockCacheSizeInBytes()

    • getRocksDBBlockCacheImplementation

      public RocksDBBlockCacheImplementations getRocksDBBlockCacheImplementation()

    • getRocksDBBlockCacheStrictCapacityLimit

      public boolean getRocksDBBlockCacheStrictCapacityLimit()

    • isRocksDBSetCacheIndexAndFilterBlocks

      public boolean isRocksDBSetCacheIndexAndFilterBlocks()

    • getRocksDBBlockCacheShardBits

      public int getRocksDBBlockCacheShardBits()

    • getRocksDBSSTFileBlockSizeInBytes

      public long getRocksDBSSTFileBlockSizeInBytes()

    • getRocksDBMemtableSizeInBytes

      public long getRocksDBMemtableSizeInBytes()

    • getRocksDBMaxMemtableCount

      public int getRocksDBMaxMemtableCount()

    • getRocksDBMinWriteBufferNumberToMerge

      public int getRocksDBMinWriteBufferNumberToMerge()

    • getRocksDBMaxTotalWalSizeInBytes

      public long getRocksDBMaxTotalWalSizeInBytes()

    • getRocksDBMaxBytesForLevelBase

      public long getRocksDBMaxBytesForLevelBase()

    • getMemTableHugePageSize

      public long getMemTableHugePageSize()

    • getRocksDBBytesPerSync

      public long getRocksDBBytesPerSync()

    • isRocksDBStatisticsEnabled

      public boolean isRocksDBStatisticsEnabled()

    • isRocksDBPlainTableFormatEnabled

      public boolean isRocksDBPlainTableFormatEnabled()
      DO NOT ENABLE! This is still experimental. PlainTable gives ultra low latency on smaller DB partition (< 2GB), but its not backward compatible so enabling would require storing the format type into metadata and enable based on the format or some similar approach. For details about PlainTable https://github.com/facebook/rocksdb/wiki/PlainTable-Format

    • isRocksDBStoreIndexInFile

      public boolean isRocksDBStoreIndexInFile()

    • getRocksDBHugePageTlbSize

      public int getRocksDBHugePageTlbSize()

    • getRocksDBBloomBitsPerKey

      public int getRocksDBBloomBitsPerKey()

    • getRocksDBTotalMemtableUsageCapInBytes

      public long getRocksDBTotalMemtableUsageCapInBytes()

    • getMaxOpenFiles

      public int getMaxOpenFiles()

    • getTargetFileSizeInBytes

      public int getTargetFileSizeInBytes()

    • getMaxFileOpeningThreads

      public int getMaxFileOpeningThreads()

    • getDatabaseOpenOperationThrottle

      public int getDatabaseOpenOperationThrottle()

    • getCappedPrefixExtractorLength

      public int getCappedPrefixExtractorLength()

    • getWriteQuotaBytesPerSecond

      public long getWriteQuotaBytesPerSecond()

    • isAutoTunedRateLimiterEnabled

      public boolean isAutoTunedRateLimiterEnabled()

    • isPutReuseByteBufferEnabled

      public boolean isPutReuseByteBufferEnabled()

    • isAtomicFlushEnabled

      public boolean isAtomicFlushEnabled()

    • isUseSeparateRMDCacheEnabled

      public boolean isUseSeparateRMDCacheEnabled()

    • getBlockBaseFormatVersion

      public int getBlockBaseFormatVersion()

    • setBlockBaseFormatVersion

      public void setBlockBaseFormatVersion(int version)

    • setRocksdbOptionsCompressionType

      public void setRocksdbOptionsCompressionType(String compressionType)

    • getMaxLogFileNum

      public int getMaxLogFileNum()

    • getMaxLogFileSize

      public long getMaxLogFileSize()

    • getTransformerValueSchema

      public String getTransformerValueSchema()

    • isBlobFilesEnabled

      public boolean isBlobFilesEnabled()

    • getMinBlobSizeInBytes

      public long getMinBlobSizeInBytes()

    • getBlobFileSizeInBytes

      public long getBlobFileSizeInBytes()

    • getBlobGarbageCollectionAgeCutOff

      public double getBlobGarbageCollectionAgeCutOff()

    • getBlobGarbageCollectionForceThreshold

      public double getBlobGarbageCollectionForceThreshold()

    • getBlobFileStartingLevel

      public int getBlobFileStartingLevel()

    • getRocksdbBlockCacheMemoryLimit

      public double getRocksdbBlockCacheMemoryLimit()

    • getIteratorReadAheadSizeInBytes

      public long getIteratorReadAheadSizeInBytes()