Class ChunkAssembler
- java.lang.Object
-
- com.linkedin.venice.hadoop.input.kafka.chunk.ChunkAssembler
-
public class ChunkAssembler extends java.lang.Object
This class accumulates all mapper values and assemble them to provide assembled complete large values or regular message.
-
-
Nested Class Summary
Nested Classes Modifier and Type Class Description static class
ChunkAssembler.ValueBytesAndSchemaId
-
Constructor Summary
Constructors Constructor Description ChunkAssembler(boolean isRmdChunkingEnabled)
-
Method Summary
All Methods Instance Methods Concrete Methods Modifier and Type Method Description ChunkAssembler.ValueBytesAndSchemaId
assembleAndGetValue(byte[] keyBytes, java.util.Iterator<byte[]> valueIterator)
The `valueIterator` of this function is supposed to be in descending order by offset.
-
-
-
Method Detail
-
assembleAndGetValue
public ChunkAssembler.ValueBytesAndSchemaId assembleAndGetValue(byte[] keyBytes, java.util.Iterator<byte[]> valueIterator)
The `valueIterator` of this function is supposed to be in descending order by offset. Here is the high-level algo: 1. If the latest event is a `DELETE`, return; 2. If the latest event is a regular `PUT`, return; 3. If the latest event is a manifest, capture all the chunk info from the latest one and ignore the older ones. 4. For chunks: (a). If there is no manifest captured yet, ignore. (b). If there is a manifest captured previously, check whether the current chunk belongs to it or not.
-
-