Package com.linkedin.davinci.kafka.consumer

Class PartitionWiseKafkaConsumerService

  • All Implemented Interfaces:
    java.io.Closeable, java.lang.AutoCloseable
    Direct Known Subclasses:
    StoreAwarePartitionWiseKafkaConsumerService
    public class PartitionWiseKafkaConsumerService
extends KafkaConsumerService
    PartitionWiseKafkaConsumerService is used to allocate share consumer from consumer pool at partition granularity. One shared consumer may have multiple topics, and each topic may have multiple consumers. For this basic implementation, we rely on round-robin to allocate next consumer from pool to achieve efficient and balanced shared consumer partition assignment load. We can improve this allocation strategy if we need to.

    • Field Detail

      • rtTopicPartitionToConsumerMap

        protected final java.util.Map<PubSubTopicPartition,​java.util.Set<PubSubConsumerAdapter>> rtTopicPartitionToConsumerMap
        Mapping from real-time topic partition to consumers. For hybrid store, different version topics from one store have same real-time topics, we should avoid same real-time topic partition from different version topics sharing the same consumer from consumer pool.