Package com.linkedin.davinci.storage.chunking

Class AbstractAvroChunkingAdapter<T>

java.lang.Object

com.linkedin.davinci.storage.chunking.AbstractAvroChunkingAdapter<T>

All Implemented Interfaces:

ChunkingAdapter<ChunkedValueInputStream,T>

Direct Known Subclasses:

GenericChunkingAdapter, RawBytesChunkingAdapter, SpecificRecordChunkingAdapter

public abstract class AbstractAvroChunkingAdapter<T>
extends Object
implements ChunkingAdapter<ChunkedValueInputStream,T>

Read compute and write compute chunking adapter

Field Summary

Fields

Modifier and Type	Field	Description
static final int	DO_NOT_USE_READER_SCHEMA_ID

Constructor Summary

Constructors

Constructor	Description
AbstractAvroChunkingAdapter()

Method Summary

Modifier and Type	Method	Description
void	addChunkIntoContainer(ChunkedValueInputStream chunkedValueInputStream, int chunkIndex, byte[] valueChunk)	Used to incrementally add a into the container.
ChunkedValueInputStream	constructChunksContainer(ChunkedValueManifest chunkedValueManifest)	Used to construct the right kind of container (according to the query code) to hold a large value which needs to be incrementally re-assembled from many smaller chunks.
T	constructValue(byte[] valueOnlyBytes, int offset, int bytesLength, RecordDeserializer<T> recordDeserializer, VeniceCompressor veniceCompressor)
T	constructValue(byte[] fullBytes, int bytesLength, T reusedValue, org.apache.avro.io.BinaryDecoder reusedDecoder, ReadResponseStats responseStats, int writerSchemaId, int readerSchemaId, StoreDeserializerCache<T> storeDeserializerCache, VeniceCompressor compressor)	Used to wrap a small value fetched from the storage engine into the right type of class needed by the query code.
T	constructValue(ChunkedValueInputStream chunkedValueInputStream, T reusedValue, org.apache.avro.io.BinaryDecoder reusedDecoder, ReadResponseStats responseStats, int writerSchemaId, int readerSchemaId, StoreDeserializerCache<T> storeDeserializerCache, VeniceCompressor compressor)	Used to wrap a large value re-assembled with the use of a into the right type of class needed by the query code.
T	get(StorageEngine store, int partition, byte[] key, ByteBuffer reusedRawValue, T reusedValue, org.apache.avro.io.BinaryDecoder reusedDecoder, boolean isChunked, ReadResponseStats responseStats, int readerSchemaId, StoreDeserializerCache<T> storeDeserializerCache, VeniceCompressor compressor)
T	get(StorageEngine store, int partition, ByteBuffer key, boolean isChunked, T reusedValue, org.apache.avro.io.BinaryDecoder reusedDecoder, ReadResponseStats responseStats, int readerSchemaId, StoreDeserializerCache<T> storeDeserializerCache, VeniceCompressor compressor, ChunkedValueManifestContainer manifestContainer)
void	getByPartialKey(StorageEngine store, int userPartition, byte[] keyPrefixBytes, T reusedValue, org.apache.avro.io.BinaryDecoder reusedDecoder, RecordDeserializer<org.apache.avro.generic.GenericRecord> keyRecordDeserializer, boolean isChunked, int readerSchemaId, StoreDeserializerCache<T> storeDeserializerCache, VeniceCompressor compressor, StreamingCallback<org.apache.avro.generic.GenericRecord,org.apache.avro.generic.GenericRecord> computingCallback)
ByteBufferValueRecord<T>	getWithSchemaId(StorageEngine store, int partition, ByteBuffer key, boolean isChunked, T reusedValue, org.apache.avro.io.BinaryDecoder reusedDecoder, StoreDeserializerCache<T> storeDeserializerCache, VeniceCompressor compressor, ChunkedValueManifestContainer manifestContainer)

Methods inherited from class java.lang.Object

clone, equals, finalize, getClass, hashCode, notify, notifyAll, toString, wait, wait, wait

Methods inherited from interface com.linkedin.davinci.storage.chunking.ChunkingAdapter

constructValue, constructValue

Field Details

DO_NOT_USE_READER_SCHEMA_ID

public static final int DO_NOT_USE_READER_SCHEMA_ID

See Also:

• Constant Field Values

Constructor Details

AbstractAvroChunkingAdapter

public AbstractAvroChunkingAdapter()

Method Details

constructValue

public T constructValue(byte[] fullBytes,
 int bytesLength,
 T reusedValue,
 org.apache.avro.io.BinaryDecoder reusedDecoder,
 ReadResponseStats responseStats,
 int writerSchemaId,
 int readerSchemaId,
 StoreDeserializerCache<T> storeDeserializerCache,
 VeniceCompressor compressor)

Description copied from interface: ChunkingAdapter

Used to wrap a small value fetched from the storage engine into the right type of class needed by the query code.

The following parameters are mandatory:

Specified by:

constructValue in interface ChunkingAdapter<ChunkedValueInputStream,T>

Parameters:

fullBytes - includes both the schema ID header and the payload

The following parameters can be ignored, by implementing ChunkingAdapter.constructValue(int, byte[]):

reusedValue - a previous instance of to be re-used in order to minimize GC

reusedDecoder - a previous instance of BinaryDecoder to be re-used in order to minimize GC

responseStats - the ReadResponseStats which carries certain metrics to be recorded at the end

writerSchemaId - schema used to serialize the value

constructValue

public T constructValue(byte[] valueOnlyBytes,
 int offset,
 int bytesLength,
 RecordDeserializer<T> recordDeserializer,
 VeniceCompressor veniceCompressor)

Specified by:

constructValue in interface ChunkingAdapter<ChunkedValueInputStream,T>

addChunkIntoContainer

public void addChunkIntoContainer(ChunkedValueInputStream chunkedValueInputStream,
 int chunkIndex,
 byte[] valueChunk)

Description copied from interface: ChunkingAdapter

Used to incrementally add a into the container.

Specified by:

addChunkIntoContainer in interface ChunkingAdapter<ChunkedValueInputStream,T>

constructChunksContainer

public ChunkedValueInputStream constructChunksContainer(ChunkedValueManifest chunkedValueManifest)

Description copied from interface: ChunkingAdapter

Used to construct the right kind of container (according to the query code) to hold a large value which needs to be incrementally re-assembled from many smaller chunks.

Specified by:

constructChunksContainer in interface ChunkingAdapter<ChunkedValueInputStream,T>

constructValue

public T constructValue(ChunkedValueInputStream chunkedValueInputStream,
 T reusedValue,
 org.apache.avro.io.BinaryDecoder reusedDecoder,
 ReadResponseStats responseStats,
 int writerSchemaId,
 int readerSchemaId,
 StoreDeserializerCache<T> storeDeserializerCache,
 VeniceCompressor compressor)

Description copied from interface: ChunkingAdapter

Used to wrap a large value re-assembled with the use of a into the right type of class needed by the query code.

The following parameters are mandatory:

Specified by:

constructValue in interface ChunkingAdapter<ChunkedValueInputStream,T>

Parameters:

chunkedValueInputStream - temporary , obtained from ChunkingAdapter.constructChunksContainer(ChunkedValueManifest)

The following parameters can be ignored, by implementing ChunkingAdapter.constructValue(int, Object):

reusedValue - a previous instance of to be re-used in order to minimize GC

reusedDecoder - a previous instance of BinaryDecoder to be re-used in order to minimize GC

responseStats - the ReadResponseStats which carries certain metrics to be recorded at the end

writerSchemaId - of the user's value

get

public T get(StorageEngine store,
 int partition,
 ByteBuffer key,
 boolean isChunked,
 T reusedValue,
 org.apache.avro.io.BinaryDecoder reusedDecoder,
 ReadResponseStats responseStats,
 int readerSchemaId,
 StoreDeserializerCache<T> storeDeserializerCache,
 VeniceCompressor compressor,
 ChunkedValueManifestContainer manifestContainer)

getWithSchemaId

public ByteBufferValueRecord<T> getWithSchemaId(StorageEngine store,
 int partition,
 ByteBuffer key,
 boolean isChunked,
 T reusedValue,
 org.apache.avro.io.BinaryDecoder reusedDecoder,
 StoreDeserializerCache<T> storeDeserializerCache,
 VeniceCompressor compressor,
 ChunkedValueManifestContainer manifestContainer)

get

public T get(StorageEngine store,
 int partition,
 byte[] key,
 ByteBuffer reusedRawValue,
 T reusedValue,
 org.apache.avro.io.BinaryDecoder reusedDecoder,
 boolean isChunked,
 ReadResponseStats responseStats,
 int readerSchemaId,
 StoreDeserializerCache<T> storeDeserializerCache,
 VeniceCompressor compressor)

getByPartialKey

public void getByPartialKey(StorageEngine store,
 int userPartition,
 byte[] keyPrefixBytes,
 T reusedValue,
 org.apache.avro.io.BinaryDecoder reusedDecoder,
 RecordDeserializer<org.apache.avro.generic.GenericRecord> keyRecordDeserializer,
 boolean isChunked,
 int readerSchemaId,
 StoreDeserializerCache<T> storeDeserializerCache,
 VeniceCompressor compressor,
 StreamingCallback<org.apache.avro.generic.GenericRecord,org.apache.avro.generic.GenericRecord> computingCallback)