Package com.linkedin.davinci.kafka.consumer

Class PartitionWiseKafkaConsumerService

java.lang.Object
com.linkedin.venice.service.AbstractVeniceService
com.linkedin.davinci.kafka.consumer.AbstractKafkaConsumerService
com.linkedin.davinci.kafka.consumer.KafkaConsumerService
com.linkedin.davinci.kafka.consumer.PartitionWiseKafkaConsumerService
All Implemented Interfaces:
Closeable, AutoCloseable
Direct Known Subclasses:
StoreAwarePartitionWiseKafkaConsumerService
public class PartitionWiseKafkaConsumerService extends KafkaConsumerService
PartitionWiseKafkaConsumerService is used to allocate share consumer from consumer pool at partition granularity. One shared consumer may have multiple topics, and each topic may have multiple consumers. For this basic implementation, we rely on round-robin to allocate next consumer from pool to achieve efficient and balanced shared consumer partition assignment load. We can improve this allocation strategy if we need to.

  • Field Details

    • rtTopicPartitionToConsumerMap

      protected final Map<PubSubTopicPartition,Set<PubSubConsumerAdapter>> rtTopicPartitionToConsumerMap
      Mapping from real-time topic partition to consumers. For hybrid store, different version topics from one store have same real-time topics, we should avoid same real-time topic partition from different version topics sharing the same consumer from consumer pool.

  • Method Details