Package com.linkedin.davinci.kafka.consumer
package com.linkedin.davinci.kafka.consumer
-
ClassDescriptionAbstract class capturing the responsibilities of drainers threads doing store ingestion.This class contains logic that SNs must perform if a store-version is running in Active/Active mode.This class contains service to periodically refresh all the signals for throttlers and update all registered throttler based on new signal values.
AggKafkaConsumerServicesupports Kafka consumer pool for multiple Kafka clusters from different data centers; for each Kafka bootstrap server url,AggKafkaConsumerServicewill create oneKafkaConsumerService.The class is used to asynchronously trigger behavior changes in theStoreIngestionTask.An Enum enumerating all valid types ofConsumerAction.Encapsulates the logic for deciding if a given topic-partition should be unsubscribed and, in that case, executing the unsubscription.This class is used to process the AA/WC messages in parallel to address the long-tail partition lagging issues.This throttler has the following functionality: 1.This class is used to throttle records consumed per Kafka clusterKafkaConsumerServiceis used to manage a pool of consumption-related resources connected to a specific Kafka cluster.This consumer assignment strategy specify how consumers from consumer pool are allocated.This delegator impl is used to distribute different partition requests into different consumer service.Assumes: One to One mapping between a Venice Store and Kafka Topic.A helper class to return the same lock for the same raw key bytes.This class contains the state transition work between leader and follower; both leader and follower will keep track of information like which topic leader is consuming from and the corresponding offset as well as the latest successfully consumed or produced offset in the version topic (VT).This class holds all the necessary context information which is passed from kafka consumer thread -> kafka producer callback thread -> drainer thread.This wrapper is used to keep the context after handling Active/Active messages.This class is used to maintain internal state for consumption of each partition.This immutable class holds a association between a key and value and the source offset of the consumed message.This class is for wrapping the information about the role of the partition replica on that host toAggKafkaConsumerServiceto achieve finer granularity of consumer assignment.PartitionWiseKafkaConsumerServiceis used to allocate share consumer from consumer pool at partition granularity.PubSubMessageProcessedResultWrapper<K,V, OFFSET> Singleton class meant to keep track of subscription failures for ingestion tasks where the ingestion task needs to keep running despite cluster failures.This store buffer services maintains two separate drainer queues for store ingestions.This class has the following responsibilities: 1.StoreAwarePartitionWiseKafkaConsumerServiceis used to allocate share consumer from consumer pool at partition granularity.This class is serving as aPubSubMessagebuffer with an accompanying pool of drainer threads.An interface for Store Ingestion Service for Venice.A runnable Kafka Consumer consuming messages from all the partition assigned to current node for a Kafka Topic.This enum represents all potential results after callingStoreIngestionTask.delegateConsumerRecord(PubSubMessageProcessedResultWrapper, int, String, int, long, long).Builder class forStoreIngestionTaskFactory; once the build() function is invoked, no modification is allowed on any parameters.This class handles Write Compute operations related to a specific store.This class is a wrapper of pair of <Version Topic, Version /Real time topic partition>.This class precomputes the conversion from Avro'sCharSequenceto String, and also hangs on to the correct instance ofPubSubTopic, so that the lookup need not be performed in many locations.TopicWiseKafkaConsumerServiceis used to allocate share consumer from consumer pool at topic granularity.This result wrapper is for non-Active-Active stores.