This class and the rest of this package encapsulate the complexity of assembling chunked values
from the storage engine. At a high level, value chunking in Venice works this way:
The
VeniceWriter performs the chunking, and the ingestion code completely ignores it,
treating chunks and full values exactly the same way. Re-assembly then happens at read time.
The reason the above strategy works is that when a store-version has chunking enabled, there
is a
ChunkedKeySuffix appended to the end of every key. This suffix indicates, via
ChunkedKeySuffix.isChunk, whether the corresponding value is a chunk or a "top-level"
key. The suffix is carefully designed to achieve the following goals:
1. Chunks and top-level keys should never collide, so that the storage engine and Kafka log
compaction never inadvertently overwrite a chunk with a top-level key or vice-versa.
2. Byte ordering is preserved assuming the
VeniceWriter writes chunks in order and
then writes the top-level key/value at the end. This is important because Venice is optimized
for ordered ingestion.
A top-level key can correspond either to a full value, or to a
ChunkedValueManifest.
This is disambiguated by looking at the
Put.schemaId field, which is set to a specific
negative value in the case of manifests.