Package com.linkedin.davinci.kafka.consumer

Class TopicWiseKafkaConsumerService

  • All Implemented Interfaces:
    java.io.Closeable, java.lang.AutoCloseable
    public class TopicWiseKafkaConsumerService
extends KafkaConsumerService
    TopicWiseKafkaConsumerService is used to allocate share consumer from consumer pool at topic granularity. One shared consumer may have multiple topics, while each topic can only take one consumer at most. All the partitions from same topic will always be subscribed in the chosen consumer. Before consumer assignment happen, the consumer with least partitions subscribed will be chosen ideally.

    • Method Detail

      • pickConsumerForPartition

        protected com.linkedin.davinci.kafka.consumer.SharedKafkaConsumer pickConsumerForPartition​(PubSubTopic versionTopic,
                                                                                           PubSubTopicPartition topicPartition)
        This function will return a consumer for the passed StoreIngestionTask. If the version topic of the passed StoreIngestionTask has been attached before, the previously assigned consumer will be returned. This function will also try to avoid assigning the same consumer to the version topics, which are belonging to the same store since for Hybrid store, the ingestion tasks for different store versions will subscribe the same Real-time topic, which won't work if they are using the same shared consumer.
        Specified by:
        pickConsumerForPartition in class KafkaConsumerService

      • unsubscribeAll

        public void unsubscribeAll​(PubSubTopic versionTopic)
        Detach the messages processing belonging to the topics of the passed {@param ingestionTask}
        Overrides:
        unsubscribeAll in class KafkaConsumerService