Package com.linkedin.davinci.client

Class AvroGenericDaVinciClient<K,V>

java.lang.Object

com.linkedin.davinci.client.AvroGenericDaVinciClient<K,V>

All Implemented Interfaces:

DaVinciClient<K,V>, AvroGenericReadComputeStoreClient<K,V>, AvroGenericStoreClient<K,V>, Closeable, AutoCloseable

Direct Known Subclasses:

AvroGenericSeekableDaVinciClient, AvroSpecificDaVinciClient, AvroSpecificSeekableDaVinciClient

public class AvroGenericDaVinciClient<K,V>
extends Object
implements DaVinciClient<K,V>, AvroGenericReadComputeStoreClient<K,V>

Field Summary

Fields

Modifier and Type	Field	Description
static final int	DEFAULT_CHUNK_SPLIT_THRESHOLD
protected final org.apache.logging.log4j.Logger	logger
static final ExecutorService	READ_CHUNK_EXECUTOR	The following two fields are used to speed up the requests with a big number of keys: 1.
protected StoreBackend	storeBackend
protected final ComplementSet<Integer>	subscription

Constructor Summary

Constructors

Modifier	Constructor	Description
	AvroGenericDaVinciClient(DaVinciConfig daVinciConfig, ClientConfig clientConfig, VeniceProperties backendConfig, Optional<Set<String>> managedClients)
protected	AvroGenericDaVinciClient(DaVinciConfig daVinciConfig, ClientConfig clientConfig, VeniceProperties backendConfig, Optional<Set<String>> managedClients, ICProvider icProvider, AbstractAvroChunkingAdapter<V> chunkingAdapter, Runnable preValidation, Executor readChunkExecutorForLargeRequest, Integer storeVersion)
	AvroGenericDaVinciClient(DaVinciConfig daVinciConfig, ClientConfig clientConfig, VeniceProperties backendConfig, Optional<Set<String>> managedClients, ICProvider icProvider, Executor readChunkExecutorForLargeRequest)

Method Summary

Modifier and Type	Method	Description
protected void	addPartitionsToSubscription(ComplementSet<Integer> partitions)
CompletableFuture<Map<K,V>>	batchGet(Set<K> keys)	Get the values associated with the given keys and return them in a map of keys to values.
void	close()	Release the internal resources.
void	compute(ComputeRequestWrapper computeRequestWrapper, Set<K> keys, org.apache.avro.Schema resultSchema, StreamingCallback<K,ComputeGenericRecord> callback, long preRequestTimeInNS)
ComputeRequestBuilder<K>	compute(Optional<ClientStats> stats, AvroGenericReadComputeStoreClient computeStoreClient)
void	computeWithKeyPrefixFilter(byte[] keyPrefix, ComputeRequestWrapper computeRequestWrapper, StreamingCallback<org.apache.avro.generic.GenericRecord,org.apache.avro.generic.GenericRecord> callback)
protected D2ServiceDiscoveryResponse	discoverService()
CompletableFuture<V>	get(K key)	Lookup the value by given key, and get(key).get() will return null if it doesn't exist.
CompletableFuture<V>	get(K key, V reusableValue)	Similar to AvroGenericStoreClient.get(Object) except that it allows passing in a instance, to minimize GC.
protected AbstractAvroChunkingAdapter<V>	getAvroChunkingAdapter()
static DaVinciBackend	getBackend()
protected GenericRecordChunkingAdapter	getGenericRecordChunkingAdapter()
org.apache.avro.Schema	getKeySchema()	Get key schema.
org.apache.avro.Schema	getLatestValueSchema()	Deprecated.
int	getPartitionCount()	Get partition count of a store.
SchemaReader	getSchemaReader()
String	getStoreName()
protected Integer	getStoreVersion()
protected void	initBackend(ClientConfig clientConfig, VeniceConfigLoader configLoader, Optional<Set<String>> managedClients, ICProvider icProvider, Optional<ObjectCacheConfig> cacheConfig)
protected boolean	isPartitionReadyToServe(VersionBackend versionBackend, int partition)	Check if user partition is ready to serve traffic.
protected boolean	isPartitionSubscribed(VersionBackend versionBackend, int partition)
boolean	isProjectionFieldValidationEnabled()
boolean	isReady()
protected CompletableFuture<Void>	seekToCheckpoint(Set<VeniceChangeCoordinate> checkpoints)
protected CompletableFuture<Void>	seekToTimestamps(Long timestamps)
protected CompletableFuture<Void>	seekToTimestamps(Map<Integer,Long> timestamps)
static <K> List<List<K>>	split(Set<K> keySet, int threshold)
void	start()
void	streamingBatchGet(Set<K> keys, StreamingCallback<K,V> callback)	Streaming interface for AvroGenericStoreClient.batchGet(Set).
protected CompletableFuture<Void>	subscribe(ComplementSet<Integer> partitions)
CompletableFuture<Void>	subscribe(Set<Integer> partitions)	Ingest specific partition/partitions locally.
CompletableFuture<Void>	subscribeAll()	Ingest the entire data (i.e.
protected void	throwIfNotReady()
String	toString()
protected void	unsubscribe(ComplementSet<Integer> partitions)
void	unsubscribe(Set<Integer> partitions)	Stop ingesting a partition locally, and drop its associated local states/data.
void	unsubscribeAll()	Stop ingesting all subscribed partition locally, and drop their associated local states/data.

Methods inherited from class java.lang.Object

clone, equals, finalize, getClass, hashCode, notify, notifyAll, wait, wait, wait

Methods inherited from interface com.linkedin.venice.client.store.AvroGenericReadComputeStoreClient

compute, compute, compute, computeAggregation

Methods inherited from interface com.linkedin.venice.client.store.AvroGenericStoreClient

streamingBatchGet

Field Details

logger

protected final org.apache.logging.log4j.Logger logger

READ_CHUNK_EXECUTOR

public static final ExecutorService READ_CHUNK_EXECUTOR

The following two fields are used to speed up the requests with a big number of keys: 1. Split the big request into smaller chunks. 2. Execute these chunks concurrently.

DEFAULT_CHUNK_SPLIT_THRESHOLD

public static final int DEFAULT_CHUNK_SPLIT_THRESHOLD

See Also:

• Constant Field Values

subscription

protected final ComplementSet<Integer> subscription

storeBackend

protected StoreBackend storeBackend

Constructor Details

AvroGenericDaVinciClient

public AvroGenericDaVinciClient(DaVinciConfig daVinciConfig,
 ClientConfig clientConfig,
 VeniceProperties backendConfig,
 Optional<Set<String>> managedClients)

AvroGenericDaVinciClient

public AvroGenericDaVinciClient(DaVinciConfig daVinciConfig,
 ClientConfig clientConfig,
 VeniceProperties backendConfig,
 Optional<Set<String>> managedClients,
 ICProvider icProvider,
 Executor readChunkExecutorForLargeRequest)

AvroGenericDaVinciClient

protected AvroGenericDaVinciClient(DaVinciConfig daVinciConfig,
 ClientConfig clientConfig,
 VeniceProperties backendConfig,
 Optional<Set<String>> managedClients,
 ICProvider icProvider,
 AbstractAvroChunkingAdapter<V> chunkingAdapter,
 Runnable preValidation,
 Executor readChunkExecutorForLargeRequest,
 Integer storeVersion)

Method Details

getSchemaReader

public SchemaReader getSchemaReader()

Specified by:

getSchemaReader in interface AvroGenericReadComputeStoreClient<K,V>

getStoreName

public String getStoreName()

Specified by:

getStoreName in interface AvroGenericStoreClient<K,V>

getKeySchema

public org.apache.avro.Schema getKeySchema()

Description copied from interface: AvroGenericStoreClient

Get key schema.

Specified by:

getKeySchema in interface AvroGenericStoreClient<K,V>

getLatestValueSchema

@Deprecated
public org.apache.avro.Schema getLatestValueSchema()

Deprecated.

Description copied from interface: AvroGenericStoreClient

Get the latest value schema known in current store client. This function doesn't guarantee it will return the latest schema if you add a new value schema when current store client is running.

Specified by:

getLatestValueSchema in interface AvroGenericStoreClient<K,V>

getPartitionCount

public int getPartitionCount()

Description copied from interface: DaVinciClient

Get partition count of a store.

Specified by:

getPartitionCount in interface DaVinciClient<K,V>

Returns:

partition count

subscribeAll

public CompletableFuture<Void> subscribeAll()

Description copied from interface: DaVinciClient

Ingest the entire data (i.e. all partitions) locally.

Specified by:

subscribeAll in interface DaVinciClient<K,V>

Returns:

a future which completes when the data is ready to serve

subscribe

public CompletableFuture<Void> subscribe(Set<Integer> partitions)

Description copied from interface: DaVinciClient

Ingest specific partition/partitions locally.

Specified by:

subscribe in interface DaVinciClient<K,V>

Parameters:

partitions - the set of partition IDs to subscribe to

Returns:

a future which completes when the partitions are ready to serve

seekToCheckpoint

protected CompletableFuture<Void> seekToCheckpoint(Set<VeniceChangeCoordinate> checkpoints)

seekToTimestamps

protected CompletableFuture<Void> seekToTimestamps(Map<Integer,Long> timestamps)

seekToTimestamps

protected CompletableFuture<Void> seekToTimestamps(Long timestamps)

subscribe

protected CompletableFuture<Void> subscribe(ComplementSet<Integer> partitions)

unsubscribeAll

public void unsubscribeAll()

Description copied from interface: DaVinciClient

Stop ingesting all subscribed partition locally, and drop their associated local states/data. If applications intend to keep the states/data for future use, no need to invoke this function before calling AvroGenericStoreClient.close().

Specified by:

unsubscribeAll in interface DaVinciClient<K,V>

unsubscribe

public void unsubscribe(Set<Integer> partitions)

Description copied from interface: DaVinciClient

Stop ingesting a partition locally, and drop its associated local states/data. If applications intend to keep the states/data for future use, no need to invoke this function before calling AvroGenericStoreClient.close().

Specified by:

unsubscribe in interface DaVinciClient<K,V>

Parameters:

partitions - the set of partition IDs to unsubscribe from

unsubscribe

protected void unsubscribe(ComplementSet<Integer> partitions)

get

public CompletableFuture<V> get(K key)

Description copied from interface: AvroGenericStoreClient

Lookup the value by given key, and get(key).get() will return null if it doesn't exist. For now, if any backend error/exception happens, get(Object key).get() will throw ExecutionException, which is a wrapper of the real exception.

Specified by:

get in interface AvroGenericStoreClient<K,V>

Returns:

get

public CompletableFuture<V> get(K key,
 V reusableValue)

Description copied from interface: AvroGenericStoreClient

Similar to AvroGenericStoreClient.get(Object) except that it allows passing in a instance, to minimize GC.

Specified by:

get in interface AvroGenericStoreClient<K,V>

split

public static <K> List<List<K>> split(Set<K> keySet,
 int threshold)

batchGet

public CompletableFuture<Map<K,V>> batchGet(Set<K> keys)
                                     throws VeniceClientException

Description copied from interface: AvroGenericStoreClient

Get the values associated with the given keys and return them in a map of keys to values. Note that the returned map will only contain entries for the keys which have a value associated with them.

Specified by:

batchGet in interface AvroGenericStoreClient<K,V>

Returns:

Throws:

VeniceClientException

streamingBatchGet

public void streamingBatchGet(Set<K> keys,
 StreamingCallback<K,V> callback)
                       throws VeniceClientException

Description copied from interface: AvroGenericStoreClient

Streaming interface for AvroGenericStoreClient.batchGet(Set). You can find more info in StreamingCallback.

Specified by:

streamingBatchGet in interface AvroGenericStoreClient<K,V>

Throws:

VeniceClientException

isProjectionFieldValidationEnabled

public boolean isProjectionFieldValidationEnabled()

Specified by:

isProjectionFieldValidationEnabled in interface AvroGenericReadComputeStoreClient<K,V>

compute

public ComputeRequestBuilder<K> compute(Optional<ClientStats> stats,
 AvroGenericReadComputeStoreClient computeStoreClient)
                                 throws VeniceClientException

Specified by:

compute in interface AvroGenericReadComputeStoreClient<K,V>

Throws:

VeniceClientException

compute

public void compute(ComputeRequestWrapper computeRequestWrapper,
 Set<K> keys,
 org.apache.avro.Schema resultSchema,
 StreamingCallback<K,ComputeGenericRecord> callback,
 long preRequestTimeInNS)
             throws VeniceClientException

Specified by:

compute in interface AvroGenericReadComputeStoreClient<K,V>

Throws:

VeniceClientException

computeWithKeyPrefixFilter

public void computeWithKeyPrefixFilter(byte[] keyPrefix,
 ComputeRequestWrapper computeRequestWrapper,
 StreamingCallback<org.apache.avro.generic.GenericRecord,org.apache.avro.generic.GenericRecord> callback)

Specified by:

computeWithKeyPrefixFilter in interface AvroGenericReadComputeStoreClient<K,V>

isReady

public boolean isReady()

isPartitionReadyToServe

protected boolean isPartitionReadyToServe(VersionBackend versionBackend,
 int partition)

Check if user partition is ready to serve traffic.

isPartitionSubscribed

protected boolean isPartitionSubscribed(VersionBackend versionBackend,
 int partition)

throwIfNotReady

protected void throwIfNotReady()

getAvroChunkingAdapter

protected AbstractAvroChunkingAdapter<V> getAvroChunkingAdapter()

getGenericRecordChunkingAdapter

protected GenericRecordChunkingAdapter getGenericRecordChunkingAdapter()

discoverService

protected D2ServiceDiscoveryResponse discoverService()

initBackend

protected void initBackend(ClientConfig clientConfig,
 VeniceConfigLoader configLoader,
 Optional<Set<String>> managedClients,
 ICProvider icProvider,
 Optional<ObjectCacheConfig> cacheConfig)

getBackend

public static DaVinciBackend getBackend()

start

public void start()

Specified by:

start in interface AvroGenericStoreClient<K,V>

close

public void close()

Description copied from interface: AvroGenericStoreClient

Release the internal resources.

Specified by:

close in interface AutoCloseable

Specified by:

close in interface AvroGenericStoreClient<K,V>

Specified by:

close in interface Closeable

toString

public String toString()

Overrides:

toString in class Object

getStoreVersion

protected Integer getStoreVersion()

Returns:

the version this client is specifically subscribed to. If it's null, it's a regular client.

addPartitionsToSubscription

protected void addPartitionsToSubscription(ComplementSet<Integer> partitions)