Package com.linkedin.davinci.store.rocksdb

Class RocksDBServerConfig

  • public class RocksDBServerConfig
extends java.lang.Object

    • Field Detail

      • ROCKSDB_OPTIONS_USE_DIRECT_READS

        public static final java.lang.String ROCKSDB_OPTIONS_USE_DIRECT_READS
        Ability to use direct IO for disk reads, might yield better performance on Azure disks. Also makes caching behavior more consistent, by limiting the caching to only RocksDB. This also reduces number of mem copies which might yield improved performance.
        See Also:
        Constant Field Values

      • ROCKSDB_ENV_FLUSH_POOL_SIZE

        public static final java.lang.String ROCKSDB_ENV_FLUSH_POOL_SIZE
        Thread pool being used by all the RocksDB databases. https://github.com/facebook/rocksdb/wiki/RocksDB-Tuning-Guide The maximum number of concurrent flush operations. It is usually good enough to set this to 1.
        See Also:
        Constant Field Values

      • ROCKSDB_ENV_COMPACTION_POOL_SIZE

        public static final java.lang.String ROCKSDB_ENV_COMPACTION_POOL_SIZE
        Thread pool being used by all the RocksDB databases. https://github.com/facebook/rocksdb/wiki/RocksDB-Tuning-Guide The maximum number of concurrent background compactions. The default is 1, but to fully utilize your CPU and storage you might want to increase this to the minimum of (the number of cores in the system, the disk throughput divided by the average throughput of one compaction thread).
        See Also:
        Constant Field Values

      • ROCKSDB_OPTIONS_COMPRESSION_TYPE

        public static final java.lang.String ROCKSDB_OPTIONS_COMPRESSION_TYPE
        Compression type, and please check this enum class to find out all the available options: CompressionType. For now, the default option is to disable compression, and use Venice native compression support if necessary.
        See Also:
        Constant Field Values

      • ROCKSDB_OPTIONS_COMPACTION_STYLE

        public static final java.lang.String ROCKSDB_OPTIONS_COMPACTION_STYLE
        Please check CompactionStyle to find all the available options.
        See Also:
        Constant Field Values

      • ROCKSDB_BLOCK_CACHE_SIZE_IN_BYTES

        public static final java.lang.String ROCKSDB_BLOCK_CACHE_SIZE_IN_BYTES
        Shared block cache across all the RocksDB databases.
        See Also:
        Constant Field Values

      • ROCKSDB_RMD_BLOCK_CACHE_SIZE_IN_BYTES

        public static final java.lang.String ROCKSDB_RMD_BLOCK_CACHE_SIZE_IN_BYTES
        Shared block cache used by RMD column family across all the RocksDB databases.
        See Also:
        Constant Field Values

      • ROCKSDB_BLOCK_CACHE_COMPRESSED_SIZE_IN_BYTES

        public static final java.lang.String ROCKSDB_BLOCK_CACHE_COMPRESSED_SIZE_IN_BYTES
        Shared block cache for compressed data.
        See Also:
        Constant Field Values

      • ROCKSDB_BLOCK_CACHE_SHARD_BITS

        public static final java.lang.String ROCKSDB_BLOCK_CACHE_SHARD_BITS
        number of bits to count cache shards, total shard count would be 2 to the power of this number.
        See Also:
        Constant Field Values

      • ROCKSDB_BLOCK_CACHE_IMPLEMENTATION

        public static final java.lang.String ROCKSDB_BLOCK_CACHE_IMPLEMENTATION
        the implementation of block cache that the venice server should use. For supported implementations @see RocksDBBlockCacheImplementations Defaults to LRU
        See Also:
        Constant Field Values

      • ROCKSDB_BLOCK_CACHE_STRICT_CAPACITY_LIMIT

        public static final java.lang.String ROCKSDB_BLOCK_CACHE_STRICT_CAPACITY_LIMIT
        if set to True, Cache size will strictly stay within set bounds, by allocating space for indexes and metadata within cache size. This needs to be set to true to make OHC behavior and memory sizing predictable.
        See Also:
        Constant Field Values

      • ROCKSDB_SET_CACHE_INDEX_AND_FILTER_BLOCKS

        public static final java.lang.String ROCKSDB_SET_CACHE_INDEX_AND_FILTER_BLOCKS
        If set to true, we will put index/filter blocks to the block cache. Otherwise, each "table reader" object will pre-load index/filter block during table initialization.
        See Also:
        Constant Field Values

      • ROCKSDB_SST_FILE_BLOCK_SIZE_IN_BYTES

        public static final java.lang.String ROCKSDB_SST_FILE_BLOCK_SIZE_IN_BYTES
        File block size, and this config has impact to the index size and read performance.
        See Also:
        Constant Field Values

      • ROCKSDB_MEMTABLE_SIZE_IN_BYTES

        public static final java.lang.String ROCKSDB_MEMTABLE_SIZE_IN_BYTES
        Max memtable size per database;
        See Also:
        Constant Field Values

      • ROCKSDB_MAX_MEMTABLE_COUNT

        public static final java.lang.String ROCKSDB_MAX_MEMTABLE_COUNT
        Max memtable count per database;
        See Also:
        Constant Field Values

      • ROCKSDB_MAX_TOTAL_WAL_SIZE_IN_BYTES

        public static final java.lang.String ROCKSDB_MAX_TOTAL_WAL_SIZE_IN_BYTES
        Max total WAL log size per database;
        See Also:
        Constant Field Values

      • ROCKSDB_MAX_BYTES_FOR_LEVEL_BASE

        public static final java.lang.String ROCKSDB_MAX_BYTES_FOR_LEVEL_BASE
        Max size of level base, and by default the next level size will be 10 times bigger;
        See Also:
        Constant Field Values

      • ROCKSDB_PLAIN_TABLE_FORMAT_ENABLED

        public static final java.lang.String ROCKSDB_PLAIN_TABLE_FORMAT_ENABLED
        See Also:
        Constant Field Values

      • CAPPED_PREFIX_EXTRACTOR_LENGTH

        public static final java.lang.String CAPPED_PREFIX_EXTRACTOR_LENGTH
        Length of the capped prefix extractor used with RocksDB Plain Table. Be cautious when tweaking this config because it might prevent reading old data written with a different extractor length. Therefore, changing it requires wiping the server data and offset.
        See Also:
        Constant Field Values

      • ROCKSDB_STORE_INDEX_IN_FILE

        public static final java.lang.String ROCKSDB_STORE_INDEX_IN_FILE
        See Also:
        Constant Field Values

      • ROCKSDB_HUGE_PAGE_TLB_SIZE

        public static final java.lang.String ROCKSDB_HUGE_PAGE_TLB_SIZE
        See Also:
        Constant Field Values

      • ROCKSDB_BLOOM_BITS_PER_KEY

        public static final java.lang.String ROCKSDB_BLOOM_BITS_PER_KEY
        See Also:
        Constant Field Values

      • ROCKSDB_HASH_TABLE_RATIO

        public static final java.lang.String ROCKSDB_HASH_TABLE_RATIO
        See Also:
        Constant Field Values

      • ROCKSDB_MAX_OPEN_FILES

        public static final java.lang.String ROCKSDB_MAX_OPEN_FILES
        See Also:
        Constant Field Values

      • ROCKSDB_TARGET_FILE_SIZE_IN_BYTES

        public static final java.lang.String ROCKSDB_TARGET_FILE_SIZE_IN_BYTES
        Target file size, and this will only apply to hybrid store since batch-only push from VPJ is using SSTFileWriter directly.
        See Also:
        Constant Field Values

      • ROCKSDB_MEM_TABLE_HUGE_PAGE_SIZE_BYTES

        public static final java.lang.String ROCKSDB_MEM_TABLE_HUGE_PAGE_SIZE_BYTES
        Page size for huge page for the arena used by the memtable in rocksdb. If <=0, it won't allocate from huge page but from malloc. Users are responsible to reserve huge pages for it to be allocated. For example: sysctl -w vm.nr_hugepages=20 See linux doc Documentation/vm/hugetlbpage.txt If there isn't enough free huge page available, rocksdb will fall back to malloc.
        See Also:
        Constant Field Values

      • ROCKSDB_BYTES_PER_SYNC

        public static final java.lang.String ROCKSDB_BYTES_PER_SYNC
        Comments from rocksdb c++ code: Allows OS to incrementally sync files to disk while they are being written, asynchronously, in the background. This operation can be used to smooth out write I/Os over time. Users shouldn't rely on it for persistency guarantee. Issue one request for every bytes_per_sync written. 0 turns it off. Default: 0
        See Also:
        Constant Field Values

      • ROCKSDB_STATISTICS_ENABLED

        public static final java.lang.String ROCKSDB_STATISTICS_ENABLED
        Whether to enable rocksdb statistics. The reason to make it configurable is that there is about 5%-10% overhead by enabling statistics. https://github.com/facebook/rocksdb/wiki/Statistics
        See Also:
        Constant Field Values

      • ROCKSDB_TOTAL_MEMTABLE_USAGE_CAP_IN_BYTES

        public static final java.lang.String ROCKSDB_TOTAL_MEMTABLE_USAGE_CAP_IN_BYTES
        https://github.com/facebook/rocksdb/wiki/Write-Buffer-Manager The total memory usage cap of all the memtables for every RocksDB database.
        See Also:
        Constant Field Values

      • ROCKSDB_MAX_FILE_OPENING_THREADS

        public static final java.lang.String ROCKSDB_MAX_FILE_OPENING_THREADS
        Currently, the max file open thread cnt per open operation is 16 by default. We could tune this param to reduce the possible maximum thread cnt;
        See Also:
        Constant Field Values

      • ROCKSDB_LEVEL0_FILE_NUM_COMPACTION_TRIGGER

        public static final java.lang.String ROCKSDB_LEVEL0_FILE_NUM_COMPACTION_TRIGGER
        When the number of level-0 SST files reaches level0_slowdown_writes_trigger, writes are stalled. When the number of level-0 SST files reaches level0_stop_writes_trigger, writes are fully stopped to wait for level-0 to level-1 compaction reduce the number of level-0 files.
        See Also:
        Constant Field Values

      • ROCKSDB_LEVEL0_SLOWDOWN_WRITES_TRIGGER

        public static final java.lang.String ROCKSDB_LEVEL0_SLOWDOWN_WRITES_TRIGGER
        See Also:
        Constant Field Values

      • ROCKSDB_LEVEL0_STOPS_WRITES_TRIGGER

        public static final java.lang.String ROCKSDB_LEVEL0_STOPS_WRITES_TRIGGER
        See Also:
        Constant Field Values

      • ROCKSDB_LEVEL0_FILE_NUM_COMPACTION_TRIGGER_WRITE_ONLY_VERSION

        public static final java.lang.String ROCKSDB_LEVEL0_FILE_NUM_COMPACTION_TRIGGER_WRITE_ONLY_VERSION
        See Also:
        Constant Field Values

      • ROCKSDB_LEVEL0_SLOWDOWN_WRITES_TRIGGER_WRITE_ONLY_VERSION

        public static final java.lang.String ROCKSDB_LEVEL0_SLOWDOWN_WRITES_TRIGGER_WRITE_ONLY_VERSION
        See Also:
        Constant Field Values

      • ROCKSDB_LEVEL0_STOPS_WRITES_TRIGGER_WRITE_ONLY_VERSION

        public static final java.lang.String ROCKSDB_LEVEL0_STOPS_WRITES_TRIGGER_WRITE_ONLY_VERSION
        See Also:
        Constant Field Values

      • ROCKSDB_LEVEL0_FILE_NUM_COMPACTION_TRIGGER_FOR_READ_WRITE_LEADER

        public static final java.lang.String ROCKSDB_LEVEL0_FILE_NUM_COMPACTION_TRIGGER_FOR_READ_WRITE_LEADER
        See Also:
        Constant Field Values

      • ROCKSDB_LEVEL0_SLOWDOWN_WRITES_TRIGGER_FOR_READ_WRITE_LEADER

        public static final java.lang.String ROCKSDB_LEVEL0_SLOWDOWN_WRITES_TRIGGER_FOR_READ_WRITE_LEADER
        See Also:
        Constant Field Values

      • ROCKSDB_LEVEL0_STOPS_WRITES_TRIGGER_FOR_READ_WRITE_LEADER

        public static final java.lang.String ROCKSDB_LEVEL0_STOPS_WRITES_TRIGGER_FOR_READ_WRITE_LEADER
        See Also:
        Constant Field Values

      • ROCKSDB_LEVEL0_COMPACTION_TUNING_FOR_READ_WRITE_LEADER_ENABLED

        public static final java.lang.String ROCKSDB_LEVEL0_COMPACTION_TUNING_FOR_READ_WRITE_LEADER_ENABLED
        See Also:
        Constant Field Values

      • ROCKSDB_PUT_REUSE_BYTE_BUFFER

        public static final java.lang.String ROCKSDB_PUT_REUSE_BYTE_BUFFER
        See Also:
        Constant Field Values

      • ROCKSDB_DB_OPEN_OPERATION_THROTTLE

        public static final java.lang.String ROCKSDB_DB_OPEN_OPERATION_THROTTLE
        Every time, when RocksDB tries to open a database, it will spin up multiple threads to load the file metadata in parallel, and the application could hit the thread limit issue if there are many RocksDB open operations at the same time. The following config is used to throttle the RocksDB open operations.
        See Also:
        Constant Field Values

      • ROCKSDB_WRITE_QUOTA_BYTES_PER_SECOND

        public static final java.lang.String ROCKSDB_WRITE_QUOTA_BYTES_PER_SECOND
        Check the following link for more details: https://github.com/facebook/rocksdb/wiki/Rate-Limiter This is used to throttle flush and compaction.
        See Also:
        Constant Field Values

      • ROCKSDB_AUTO_TUNED_RATE_LIMITER_ENABLED

        public static final java.lang.String ROCKSDB_AUTO_TUNED_RATE_LIMITER_ENABLED
        See Also:
        Constant Field Values

      • ROCKSDB_ATOMIC_FLUSH_ENABLED

        public static final java.lang.String ROCKSDB_ATOMIC_FLUSH_ENABLED
        See Also:
        Constant Field Values

      • ROCKSDB_SEPARATE_RMD_CACHE_ENABLED

        public static final java.lang.String ROCKSDB_SEPARATE_RMD_CACHE_ENABLED
        See Also:
        Constant Field Values

      • ROCKSDB_BLOCK_BASE_FORMAT_VERSION

        public static final java.lang.String ROCKSDB_BLOCK_BASE_FORMAT_VERSION
        See Also:
        Constant Field Values

      • ROCKSDB_MAX_LOG_FILE_NUM

        public static final java.lang.String ROCKSDB_MAX_LOG_FILE_NUM
        See Also:
        Constant Field Values

      • ROCKSDB_MAX_LOG_FILE_SIZE

        public static final java.lang.String ROCKSDB_MAX_LOG_FILE_SIZE
        See Also:
        Constant Field Values

      • RECORD_TRANSFORMER_VALUE_SCHEMA

        public static final java.lang.String RECORD_TRANSFORMER_VALUE_SCHEMA
        See Also:
        Constant Field Values

    • Constructor Detail

    • Method Detail

      • getLevel0FileNumCompactionTriggerWriteOnlyVersion

        public int getLevel0FileNumCompactionTriggerWriteOnlyVersion()

      • getLevel0SlowdownWritesTriggerWriteOnlyVersion

        public int getLevel0SlowdownWritesTriggerWriteOnlyVersion()

      • getLevel0StopWritesTriggerWriteOnlyVersion

        public int getLevel0StopWritesTriggerWriteOnlyVersion()

      • getLevel0FileNumCompactionTrigger

        public int getLevel0FileNumCompactionTrigger()

      • getLevel0SlowdownWritesTrigger

        public int getLevel0SlowdownWritesTrigger()

      • getLevel0StopWritesTrigger

        public int getLevel0StopWritesTrigger()

      • getLevel0FileNumCompactionTriggerForReadWriteLeader

        public int getLevel0FileNumCompactionTriggerForReadWriteLeader()

      • getLevel0SlowdownWritesTriggerForReadWriteLeader

        public int getLevel0SlowdownWritesTriggerForReadWriteLeader()

      • getLevel0StopWritesTriggerForReadWriteLeader

        public int getLevel0StopWritesTriggerForReadWriteLeader()

      • isLevel0CompactionTuningForReadWriteLeaderEnabled

        public boolean isLevel0CompactionTuningForReadWriteLeaderEnabled()

      • getRocksDBUseDirectReads

        public boolean getRocksDBUseDirectReads()

      • getRocksDBEnvFlushPoolSize

        public int getRocksDBEnvFlushPoolSize()

      • getRocksDBEnvCompactionPoolSize

        public int getRocksDBEnvCompactionPoolSize()

      • getRocksDBOptionsCompressionType

        public org.rocksdb.CompressionType getRocksDBOptionsCompressionType()

      • getRocksDBOptionsCompactionStyle

        public org.rocksdb.CompactionStyle getRocksDBOptionsCompactionStyle()

      • getRocksDBBlockCacheSizeInBytes

        public long getRocksDBBlockCacheSizeInBytes()

      • getRocksDBRMDBlockCacheSizeInBytes

        public long getRocksDBRMDBlockCacheSizeInBytes()

      • getRocksDBBlockCacheStrictCapacityLimit

        public boolean getRocksDBBlockCacheStrictCapacityLimit()

      • isRocksDBSetCacheIndexAndFilterBlocks

        public boolean isRocksDBSetCacheIndexAndFilterBlocks()

      • getRocksDBBlockCacheShardBits

        public int getRocksDBBlockCacheShardBits()

      • getRocksDBSSTFileBlockSizeInBytes

        public long getRocksDBSSTFileBlockSizeInBytes()

      • getRocksDBMemtableSizeInBytes

        public long getRocksDBMemtableSizeInBytes()

      • getRocksDBMaxMemtableCount

        public int getRocksDBMaxMemtableCount()

      • getRocksDBMaxTotalWalSizeInBytes

        public long getRocksDBMaxTotalWalSizeInBytes()

      • getRocksDBMaxBytesForLevelBase

        public long getRocksDBMaxBytesForLevelBase()

      • getMemTableHugePageSize

        public long getMemTableHugePageSize()

      • getRocksDBBytesPerSync

        public long getRocksDBBytesPerSync()

      • isRocksDBStatisticsEnabled

        public boolean isRocksDBStatisticsEnabled()

      • isRocksDBPlainTableFormatEnabled

        public boolean isRocksDBPlainTableFormatEnabled()
        DO NOT ENABLE! This is still experimental. PlainTable gives ultra low latency on smaller DB partition (< 2GB), but its not backward compatible so enabling would require storing the format type into metadata and enable based on the format or some similar approach. For details about PlainTable https://github.com/facebook/rocksdb/wiki/PlainTable-Format

      • isRocksDBStoreIndexInFile

        public boolean isRocksDBStoreIndexInFile()

      • getRocksDBHugePageTlbSize

        public int getRocksDBHugePageTlbSize()

      • getRocksDBBloomBitsPerKey

        public int getRocksDBBloomBitsPerKey()

      • getRocksDBTotalMemtableUsageCapInBytes

        public long getRocksDBTotalMemtableUsageCapInBytes()

      • getMaxOpenFiles

        public int getMaxOpenFiles()

      • getTargetFileSizeInBytes

        public int getTargetFileSizeInBytes()

      • getMaxFileOpeningThreads

        public int getMaxFileOpeningThreads()

      • getDatabaseOpenOperationThrottle

        public int getDatabaseOpenOperationThrottle()

      • getCappedPrefixExtractorLength

        public int getCappedPrefixExtractorLength()

      • getWriteQuotaBytesPerSecond

        public long getWriteQuotaBytesPerSecond()

      • isAutoTunedRateLimiterEnabled

        public boolean isAutoTunedRateLimiterEnabled()

      • isPutReuseByteBufferEnabled

        public boolean isPutReuseByteBufferEnabled()

      • isAtomicFlushEnabled

        public boolean isAtomicFlushEnabled()

      • isUseSeparateRMDCacheEnabled

        public boolean isUseSeparateRMDCacheEnabled()

      • getBlockBaseFormatVersion

        public int getBlockBaseFormatVersion()

      • setBlockBaseFormatVersion

        public void setBlockBaseFormatVersion​(int version)

      • setRocksdbOptionsCompressionType

        public void setRocksdbOptionsCompressionType​(java.lang.String compressionType)

      • getMaxLogFileNum

        public int getMaxLogFileNum()

      • getMaxLogFileSize

        public long getMaxLogFileSize()

      • getTransformerValueSchema

        public java.lang.String getTransformerValueSchema()