Package com.linkedin.davinci.storage.chunking

Class ChunkingUtils

java.lang.Object

com.linkedin.davinci.storage.chunking.ChunkingUtils

public class ChunkingUtils
extends Object

This class and the rest of this package encapsulate the complexity of assembling chunked values from the storage engine. At a high level, value chunking in Venice works this way: The VeniceWriter performs the chunking, and the ingestion code completely ignores it, treating chunks and full values exactly the same way. Re-assembly then happens at read time. The reason the above strategy works is that when a store-version has chunking enabled, there is a ChunkedKeySuffix appended to the end of every key. This suffix indicates, via ChunkedKeySuffix.isChunk, whether the corresponding value is a chunk or a "top-level" key. The suffix is carefully designed to achieve the following goals: 1. Chunks and top-level keys should never collide, so that the storage engine and Kafka log compaction never inadvertently overwrite a chunk with a top-level key or vice-versa. 2. Byte ordering is preserved assuming the VeniceWriter writes chunks in order and then writes the top-level key/value at the end. This is important because Venice is optimized for ordered ingestion. A top-level key can correspond either to a full value, or to a ChunkedValueManifest. This is disambiguated by looking at the Put.schemaId field, which is set to a specific negative value in the case of manifests.

See Also:

• for the specific ID Therefore, at read time, the following steps are executed: 1. The top-level key is queried. 2. The top-level key's value's schema ID is checked. a) If it is positive, then it's a full value, and is returned immediately. b) If it is negative, then it's a ChunkedValueManifest, and we continue to the next steps. 3. The ChunkedValueManifest is deserialized, and its chunk keys are extracted. 4. Each chunk key is queried. 5. The chunks are stitched back together using the various adapter interfaces of this package, depending on whether it is the single get or batch get/compute path that needs to re-assemble a chunked value.

Field Summary

Fields

Modifier and Type	Field	Description
static final KeyWithChunkingSuffixSerializer	KEY_WITH_CHUNKING_SUFFIX_SERIALIZER

Constructor Summary

Constructors

Constructor	Description
ChunkingUtils()

Method Summary

Methods inherited from class java.lang.Object

clone, equals, finalize, getClass, hashCode, notify, notifyAll, toString, wait, wait, wait

Field Details

KEY_WITH_CHUNKING_SUFFIX_SERIALIZER

public static final KeyWithChunkingSuffixSerializer KEY_WITH_CHUNKING_SUFFIX_SERIALIZER

Constructor Details

ChunkingUtils

public ChunkingUtils()