Package com.linkedin.davinci.utils

Class RocksDBChunkAssembler

java.lang.Object

com.linkedin.davinci.utils.ChunkAssembler

com.linkedin.davinci.utils.RocksDBChunkAssembler

public class RocksDBChunkAssembler
extends ChunkAssembler

Similar to InMemoryChunkAssembler except is uses a RocksDBStorageEngine as the buffer to buffer chunks before the full record can be assembled. The evict policy is also different. Only keys belonging to the assembled record's manifest are removed. This could result in larger buffer size, but it's necessary for consuming topics that could have multiple ongoing chunks such as the materialized view topics. TTL or eviction by source partition (VT partition) could be adopted as an improvement to clean up any leaked chunks due to producer failures.

Field Summary

Fields inherited from class com.linkedin.davinci.utils.ChunkAssembler

bufferStorageEngine

Constructor Summary

Constructors

Constructor	Description
RocksDBChunkAssembler(StorageEngine bufferStorageEngine, boolean skipFailedToAssembleRecords)

Method Summary

Methods inherited from class com.linkedin.davinci.utils.ChunkAssembler

bufferAndAssembleRecord, clearBuffer, isChunkedRecord

Methods inherited from class java.lang.Object

clone, equals, finalize, getClass, hashCode, notify, notifyAll, toString, wait, wait, wait

Constructor Details

RocksDBChunkAssembler

public RocksDBChunkAssembler(StorageEngine bufferStorageEngine,
 boolean skipFailedToAssembleRecords)