Package com.linkedin.venice.hadoop.input.kafka.chunk

Class ChunkAssembler

java.lang.Object

com.linkedin.venice.hadoop.input.kafka.chunk.ChunkAssembler

public class ChunkAssembler
extends Object

This class accumulates all mapper values and assemble them to provide assembled complete large values or regular message.

Nested Class Summary

Nested Classes

Modifier and Type	Class	Description
static class	ChunkAssembler.ValueBytesAndSchemaId

Constructor Summary

Constructors

Constructor	Description
ChunkAssembler(boolean isRmdChunkingEnabled)

Method Summary

Modifier and Type	Method	Description
ChunkAssembler.ValueBytesAndSchemaId	assembleAndGetValue(byte[] keyBytes, Iterator<byte[]> valueIterator)	The `valueIterator` of this function is supposed to be in descending order by offset.

Methods inherited from class java.lang.Object

clone, equals, finalize, getClass, hashCode, notify, notifyAll, toString, wait, wait, wait

Constructor Details

ChunkAssembler

public ChunkAssembler(boolean isRmdChunkingEnabled)

Method Details

assembleAndGetValue

public ChunkAssembler.ValueBytesAndSchemaId assembleAndGetValue(byte[] keyBytes,
 Iterator<byte[]> valueIterator)

The `valueIterator` of this function is supposed to be in descending order by offset. Here is the high-level algo: 1. If the latest event is a `DELETE`, return; 2. If the latest event is a regular `PUT`, return; 3. If the latest event is a manifest, capture all the chunk info from the latest one and ignore the older ones. 4. For chunks: (a). If there is no manifest captured yet, ignore. (b). If there is a manifest captured previously, check whether the current chunk belongs to it or not.