Package com.linkedin.davinci.kafka.consumer

Class TopicWiseKafkaConsumerService

java.lang.Object
com.linkedin.venice.service.AbstractVeniceService
com.linkedin.davinci.kafka.consumer.AbstractKafkaConsumerService
com.linkedin.davinci.kafka.consumer.KafkaConsumerService
com.linkedin.davinci.kafka.consumer.TopicWiseKafkaConsumerService
All Implemented Interfaces:
Closeable, AutoCloseable
public class TopicWiseKafkaConsumerService extends KafkaConsumerService
TopicWiseKafkaConsumerService is used to allocate share consumer from consumer pool at topic granularity. One shared consumer may have multiple topics, while each topic can only take one consumer at most. All the partitions from same topic will always be subscribed in the chosen consumer. Before consumer assignment happen, the consumer with least partitions subscribed will be chosen ideally.

  • Method Details

    • pickConsumerForPartition

      protected com.linkedin.davinci.kafka.consumer.SharedKafkaConsumer pickConsumerForPartition(PubSubTopic versionTopic, PubSubTopicPartition topicPartition)
      This function will return a consumer for the passed StoreIngestionTask. If the version topic of the passed StoreIngestionTask has been attached before, the previously assigned consumer will be returned. This function will also try to avoid assigning the same consumer to the version topics, which are belonging to the same store since for Hybrid store, the ingestion tasks for different store versions will subscribe the same Real-time topic, which won't work if they are using the same shared consumer.
      Specified by:
      pickConsumerForPartition in class KafkaConsumerService

    • unsubscribeAll

      public void unsubscribeAll(PubSubTopic versionTopic)
      Detach the messages processing belonging to the topics of the passed
      Overrides:
      unsubscribeAll in class KafkaConsumerService