Package com.linkedin.davinci.storage.chunking

Class AbstractAvroChunkingAdapter<T>

java.lang.Object

com.linkedin.davinci.storage.chunking.AbstractAvroChunkingAdapter<T>

All Implemented Interfaces:

ChunkingAdapter<ChunkedValueInputStream,T>

Direct Known Subclasses:

GenericChunkingAdapter, RawBytesChunkingAdapter, SpecificRecordChunkingAdapter

public abstract class AbstractAvroChunkingAdapter<T>
extends java.lang.Object
implements ChunkingAdapter<ChunkedValueInputStream,T>

Read compute and write compute chunking adapter

Constructor Summary

Constructors

Constructor	Description
AbstractAvroChunkingAdapter()

Method Summary

Modifier and Type	Method	Description
void	addChunkIntoContainer(ChunkedValueInputStream chunkedValueInputStream, int chunkIndex, byte[] valueChunk)	Used to incrementally add a {@param valueChunk} into the {@param CHUNKS_CONTAINER} container.
ChunkedValueInputStream	constructChunksContainer(ChunkedValueManifest chunkedValueManifest)	Used to construct the right kind of {@param CHUNKS_CONTAINER} container (according to the query code) to hold a large value which needs to be incrementally re-assembled from many smaller chunks.
T	constructValue(byte[] valueOnlyBytes, int offset, int bytesLength, RecordDeserializer<T> recordDeserializer, VeniceCompressor veniceCompressor)
T	constructValue(byte[] fullBytes, int bytesLength, T reusedValue, org.apache.avro.io.BinaryDecoder reusedDecoder, ReadResponse response, int writerSchemaId, int readerSchemaId, StoreDeserializerCache<T> storeDeserializerCache, VeniceCompressor compressor)	Used to wrap a small {@param fullBytes} value fetched from the storage engine into the right type of {@param VALUE} class needed by the query code.
T	constructValue(ChunkedValueInputStream chunkedValueInputStream, T reusedValue, org.apache.avro.io.BinaryDecoder reusedDecoder, ReadResponse response, int writerSchemaId, int readerSchemaId, StoreDeserializerCache<T> storeDeserializerCache, VeniceCompressor compressor)	Used to wrap a large value re-assembled with the use of a {@param CHUNKS_CONTAINER} into the right type of {@param VALUE} class needed by the query code.
T	get(AbstractStorageEngine store, int partition, byte[] key, java.nio.ByteBuffer reusedRawValue, T reusedValue, org.apache.avro.io.BinaryDecoder reusedDecoder, boolean isChunked, ReadResponse response, int readerSchemaId, StoreDeserializerCache<T> storeDeserializerCache, VeniceCompressor compressor)
T	get(AbstractStorageEngine store, int userPartition, VenicePartitioner partitioner, PartitionerConfig partitionerConfig, byte[] key, java.nio.ByteBuffer reusedRawValue, T reusedValue, org.apache.avro.io.BinaryDecoder reusedDecoder, boolean isChunked, ReadResponse response, int readerSchemaId, StoreDeserializerCache<T> storeDeserializerCache, VeniceCompressor compressor)
T	get(AbstractStorageEngine store, int partition, java.nio.ByteBuffer key, boolean isChunked, T reusedValue, org.apache.avro.io.BinaryDecoder reusedDecoder, ReadResponse response, int readerSchemaId, StoreDeserializerCache<T> storeDeserializerCache, VeniceCompressor compressor, ChunkedValueManifestContainer manifestContainer)
void	getByPartialKey(AbstractStorageEngine store, int userPartition, PartitionerConfig partitionerConfig, byte[] keyPrefixBytes, T reusedValue, org.apache.avro.io.BinaryDecoder reusedDecoder, RecordDeserializer<org.apache.avro.generic.GenericRecord> keyRecordDeserializer, boolean isChunked, ReadResponse response, int readerSchemaId, StoreDeserializerCache<T> storeDeserializerCache, VeniceCompressor compressor, StreamingCallback<org.apache.avro.generic.GenericRecord,org.apache.avro.generic.GenericRecord> computingCallback)
ByteBufferValueRecord<T>	getWithSchemaId(AbstractStorageEngine store, int partition, java.nio.ByteBuffer key, boolean isChunked, T reusedValue, org.apache.avro.io.BinaryDecoder reusedDecoder, StoreDeserializerCache<T> storeDeserializerCache, VeniceCompressor compressor, ChunkedValueManifestContainer manifestContainer)

Methods inherited from class java.lang.Object

clone, equals, finalize, getClass, hashCode, notify, notifyAll, toString, wait, wait, wait

Methods inherited from interface com.linkedin.davinci.storage.chunking.ChunkingAdapter

constructValue, constructValue

Constructor Detail

AbstractAvroChunkingAdapter

public AbstractAvroChunkingAdapter()

Method Detail

constructValue

public T constructValue(byte[] fullBytes,
                        int bytesLength,
                        T reusedValue,
                        org.apache.avro.io.BinaryDecoder reusedDecoder,
                        ReadResponse response,
                        int writerSchemaId,
                        int readerSchemaId,
                        StoreDeserializerCache<T> storeDeserializerCache,
                        VeniceCompressor compressor)

Description copied from interface: ChunkingAdapter

Used to wrap a small {@param fullBytes} value fetched from the storage engine into the right type of {@param VALUE} class needed by the query code.

The following parameters are mandatory:

Specified by:

constructValue in interface ChunkingAdapter<ChunkedValueInputStream,T>

Parameters:

fullBytes - includes both the schema ID header and the payload

The following parameters can be ignored, by implementing ChunkingAdapter.constructValue(int, byte[]):

reusedValue - a previous instance of to be re-used in order to minimize GC

reusedDecoder - a previous instance of BinaryDecoder to be re-used in order to minimize GC

response - the response returned by the query path, which carries certain metrics to be recorded at the end

writerSchemaId - schema used to serialize the value

constructValue

public T constructValue(byte[] valueOnlyBytes,
                        int offset,
                        int bytesLength,
                        RecordDeserializer<T> recordDeserializer,
                        VeniceCompressor veniceCompressor)

Specified by:

constructValue in interface ChunkingAdapter<ChunkedValueInputStream,T>

addChunkIntoContainer

public void addChunkIntoContainer(ChunkedValueInputStream chunkedValueInputStream,
                                  int chunkIndex,
                                  byte[] valueChunk)

Description copied from interface: ChunkingAdapter

Used to incrementally add a {@param valueChunk} into the {@param CHUNKS_CONTAINER} container.

Specified by:

addChunkIntoContainer in interface ChunkingAdapter<ChunkedValueInputStream,T>

constructChunksContainer

public ChunkedValueInputStream constructChunksContainer(ChunkedValueManifest chunkedValueManifest)

Description copied from interface: ChunkingAdapter

Used to construct the right kind of {@param CHUNKS_CONTAINER} container (according to the query code) to hold a large value which needs to be incrementally re-assembled from many smaller chunks.

Specified by:

constructChunksContainer in interface ChunkingAdapter<ChunkedValueInputStream,T>

constructValue

public T constructValue(ChunkedValueInputStream chunkedValueInputStream,
                        T reusedValue,
                        org.apache.avro.io.BinaryDecoder reusedDecoder,
                        ReadResponse response,
                        int writerSchemaId,
                        int readerSchemaId,
                        StoreDeserializerCache<T> storeDeserializerCache,
                        VeniceCompressor compressor)

Description copied from interface: ChunkingAdapter

Used to wrap a large value re-assembled with the use of a {@param CHUNKS_CONTAINER} into the right type of {@param VALUE} class needed by the query code.

The following parameters are mandatory:

Specified by:

constructValue in interface ChunkingAdapter<ChunkedValueInputStream,T>

Parameters:

chunkedValueInputStream - temporary , obtained from ChunkingAdapter.constructChunksContainer(ChunkedValueManifest)

The following parameters can be ignored, by implementing ChunkingAdapter.constructValue(int, Object):

reusedValue - a previous instance of to be re-used in order to minimize GC

reusedDecoder - a previous instance of BinaryDecoder to be re-used in order to minimize GC

response - the response returned by the query path, which carries certain metrics to be recorded at the end

writerSchemaId - of the user's value

get

public T get(AbstractStorageEngine store,
             int partition,
             java.nio.ByteBuffer key,
             boolean isChunked,
             T reusedValue,
             org.apache.avro.io.BinaryDecoder reusedDecoder,
             ReadResponse response,
             int readerSchemaId,
             StoreDeserializerCache<T> storeDeserializerCache,
             VeniceCompressor compressor,
             ChunkedValueManifestContainer manifestContainer)

getWithSchemaId

public ByteBufferValueRecord<T> getWithSchemaId(AbstractStorageEngine store,
                                                int partition,
                                                java.nio.ByteBuffer key,
                                                boolean isChunked,
                                                T reusedValue,
                                                org.apache.avro.io.BinaryDecoder reusedDecoder,
                                                StoreDeserializerCache<T> storeDeserializerCache,
                                                VeniceCompressor compressor,
                                                ChunkedValueManifestContainer manifestContainer)

get

public T get(AbstractStorageEngine store,
             int partition,
             byte[] key,
             java.nio.ByteBuffer reusedRawValue,
             T reusedValue,
             org.apache.avro.io.BinaryDecoder reusedDecoder,
             boolean isChunked,
             ReadResponse response,
             int readerSchemaId,
             StoreDeserializerCache<T> storeDeserializerCache,
             VeniceCompressor compressor)

get

public T get(AbstractStorageEngine store,
             int userPartition,
             VenicePartitioner partitioner,
             PartitionerConfig partitionerConfig,
             byte[] key,
             java.nio.ByteBuffer reusedRawValue,
             T reusedValue,
             org.apache.avro.io.BinaryDecoder reusedDecoder,
             boolean isChunked,
             ReadResponse response,
             int readerSchemaId,
             StoreDeserializerCache<T> storeDeserializerCache,
             VeniceCompressor compressor)

getByPartialKey

public void getByPartialKey(AbstractStorageEngine store,
                            int userPartition,
                            PartitionerConfig partitionerConfig,
                            byte[] keyPrefixBytes,
                            T reusedValue,
                            org.apache.avro.io.BinaryDecoder reusedDecoder,
                            RecordDeserializer<org.apache.avro.generic.GenericRecord> keyRecordDeserializer,
                            boolean isChunked,
                            ReadResponse response,
                            int readerSchemaId,
                            StoreDeserializerCache<T> storeDeserializerCache,
                            VeniceCompressor compressor,
                            StreamingCallback<org.apache.avro.generic.GenericRecord,org.apache.avro.generic.GenericRecord> computingCallback)