Package com.linkedin.davinci.utils

Class ChunkAssembler

java.lang.Object

com.linkedin.davinci.utils.ChunkAssembler

Direct Known Subclasses:

InMemoryChunkAssembler, RocksDBChunkAssembler

public abstract class ChunkAssembler
extends Object

Field Summary

Fields

Modifier and Type	Field	Description
protected final StorageEngine	bufferStorageEngine

Constructor Summary

Constructors

Constructor	Description
ChunkAssembler(StorageEngine bufferStorageEngine, boolean skipFailedToAssembleRecords)
ChunkAssembler(StorageEngine bufferStorageEngine, boolean skipFailedToAssembleRecords, boolean isRmdChunkingEnabled)

Method Summary

Modifier and Type	Method	Description
ByteBufferValueRecord<ByteBuffer>	bufferAndAssembleRecord(PubSubTopicPartition pubSubTopicPartition, int schemaId, byte[] keyBytes, ByteBuffer valueBytes, PubSubPosition recordOffset, VeniceCompressor compressor)	Buffers and assembles chunks of a record.
ByteBufferValueRecord<ByteBuffer>	bufferAndAssembleRecord(PubSubTopicPartition pubSubTopicPartition, int schemaId, byte[] keyBytes, ByteBuffer valueBytes, ByteBuffer replicationMetadataPayload, PubSubPosition recordOffset, VeniceCompressor compressor)	Buffers and assembles chunks of a record, including RMD chunks when RMD chunking is enabled.
void	clearBuffer()
static ByteBuffer	decompressValueIfNeeded(ByteBuffer value, int schemaId, VeniceCompressor compressor)	For chunked records, the chunk assembler already decompresses the assembled value via RawBytesChunkingAdapter's decompressingInputStreamDecoder.
static boolean	isChunkedRecord(int schemaId)

Methods inherited from class java.lang.Object

clone, equals, finalize, getClass, hashCode, notify, notifyAll, toString, wait, wait, wait

Field Details

bufferStorageEngine

protected final StorageEngine bufferStorageEngine

Constructor Details

ChunkAssembler

public ChunkAssembler(StorageEngine bufferStorageEngine,
 boolean skipFailedToAssembleRecords)

ChunkAssembler

public ChunkAssembler(StorageEngine bufferStorageEngine,
 boolean skipFailedToAssembleRecords,
 boolean isRmdChunkingEnabled)

Method Details

bufferAndAssembleRecord

public ByteBufferValueRecord<ByteBuffer> bufferAndAssembleRecord(PubSubTopicPartition pubSubTopicPartition,
 int schemaId,
 byte[] keyBytes,
 ByteBuffer valueBytes,
 PubSubPosition recordOffset,
 VeniceCompressor compressor)

Buffers and assembles chunks of a record. If the record is chunked, it stores the chunks and returns null. Once all chunks of a record are received, it returns the compressed and serialized assembled record.

bufferAndAssembleRecord

public ByteBufferValueRecord<ByteBuffer> bufferAndAssembleRecord(PubSubTopicPartition pubSubTopicPartition,
 int schemaId,
 byte[] keyBytes,
 ByteBuffer valueBytes,
 ByteBuffer replicationMetadataPayload,
 PubSubPosition recordOffset,
 VeniceCompressor compressor)

Buffers and assembles chunks of a record, including RMD chunks when RMD chunking is enabled. For CHUNK messages: - If valueBytes is non-empty: this is a value chunk; store valueBytes. - Else if replicationMetadataPayload is non-empty: this is an RMD chunk; store replicationMetadataPayload. For CHUNKED_VALUE_MANIFEST messages: - Assembles the value from stored value chunks. - If isRmdChunkingEnabled, also assembles the RMD from stored RMD chunks using the RMD manifest in replicationMetadataPayload, and sets it on the returned record.

clearBuffer

public void clearBuffer()

isChunkedRecord

public static boolean isChunkedRecord(int schemaId)

decompressValueIfNeeded

public static ByteBuffer decompressValueIfNeeded(ByteBuffer value,
 int schemaId,
 VeniceCompressor compressor)
                                          throws IOException

For chunked records, the chunk assembler already decompresses the assembled value via RawBytesChunkingAdapter's decompressingInputStreamDecoder. For non-chunked records, the value is still compressed and needs explicit decompression here.

Throws:

IOException