Package com.linkedin.davinci.kafka.consumer

Class StoreAwarePartitionWiseKafkaConsumerService

  • All Implemented Interfaces:
    java.io.Closeable, java.lang.AutoCloseable
    public class StoreAwarePartitionWiseKafkaConsumerService
extends PartitionWiseKafkaConsumerService
    StoreAwarePartitionWiseKafkaConsumerService is used to allocate share consumer from consumer pool at partition granularity. One shared consumer may have multiple topics, and each topic may have multiple consumers. This is store-aware version of topic-wise shared consumer service. The topic partition assignment in this service has a heuristic that we should distribute the all the subscriptions related to a same store / version as even as possible. The load calculation for each consumer will be: Consumer assignment size + IMPOSSIBLE_MAX_PARTITION_COUNT_PER_CONSUMER * subscription count for the same store; and we will pick the least loaded consumer for a new topic partition request. If there is no eligible consumer, it will throw IllegalStateException