Package com.linkedin.davinci.kafka.consumer

Class TopicWiseKafkaConsumerService

java.lang.Object

com.linkedin.venice.service.AbstractVeniceService

com.linkedin.davinci.kafka.consumer.AbstractKafkaConsumerService

com.linkedin.davinci.kafka.consumer.KafkaConsumerService

com.linkedin.davinci.kafka.consumer.TopicWiseKafkaConsumerService

All Implemented Interfaces:

java.io.Closeable, java.lang.AutoCloseable

public class TopicWiseKafkaConsumerService
extends KafkaConsumerService

TopicWiseKafkaConsumerService is used to allocate share consumer from consumer pool at topic granularity. One shared consumer may have multiple topics, while each topic can only take one consumer at most. All the partitions from same topic will always be subscribed in the chosen consumer. Before consumer assignment happen, the consumer with least partitions subscribed will be chosen ideally.

Nested Class Summary

Nested classes/interfaces inherited from class com.linkedin.davinci.kafka.consumer.KafkaConsumerService

KafkaConsumerService.ConsumerAssignmentStrategy

Nested classes/interfaces inherited from class com.linkedin.venice.service.AbstractVeniceService

AbstractVeniceService.ServiceState

Field Summary

Fields inherited from class com.linkedin.davinci.kafka.consumer.KafkaConsumerService

aggStats, consumerToConsumptionTask, kafkaUrl, kafkaUrlForLogger, versionTopicToTopicPartitionToConsumer

Fields inherited from class com.linkedin.venice.service.AbstractVeniceService

logger, serviceState

Method Summary

Modifier and Type	Method	Description
protected com.linkedin.davinci.kafka.consumer.SharedKafkaConsumer	pickConsumerForPartition(PubSubTopic versionTopic, PubSubTopicPartition topicPartition)	This function will return a consumer for the passed StoreIngestionTask.
void	unsubscribeAll(PubSubTopic versionTopic)	Detach the messages processing belonging to the topics of the passed {@param ingestionTask}

Methods inherited from class com.linkedin.davinci.kafka.consumer.KafkaConsumerService

assignConsumerFor, batchUnsubscribe, getConsumerAssignedToVersionTopicPartition, getIngestionInfoFromConsumer, getLatestOffsetBasedOnMetrics, getMaxElapsedTimeMSSinceLastPollInConsumerPool, getOffsetLagBasedOnMetrics, hasAnySubscriptionFor, removeTopicPartitionFromConsumptionTask, startConsumptionIntoDataReceiver, startInner, stopInner, unSubscribe

Methods inherited from class com.linkedin.venice.service.AbstractVeniceService

close, getName, isRunning, start, stop

Methods inherited from class java.lang.Object

clone, equals, finalize, getClass, hashCode, notify, notifyAll, toString, wait, wait, wait

Method Detail

pickConsumerForPartition

protected com.linkedin.davinci.kafka.consumer.SharedKafkaConsumer pickConsumerForPartition(PubSubTopic versionTopic,
                                                                                           PubSubTopicPartition topicPartition)

This function will return a consumer for the passed StoreIngestionTask. If the version topic of the passed StoreIngestionTask has been attached before, the previously assigned consumer will be returned. This function will also try to avoid assigning the same consumer to the version topics, which are belonging to the same store since for Hybrid store, the ingestion tasks for different store versions will subscribe the same Real-time topic, which won't work if they are using the same shared consumer.

Specified by:

pickConsumerForPartition in class KafkaConsumerService

unsubscribeAll

public void unsubscribeAll(PubSubTopic versionTopic)

Detach the messages processing belonging to the topics of the passed {@param ingestionTask}

Overrides:

unsubscribeAll in class KafkaConsumerService