Package com.linkedin.davinci.kafka.consumer

Class StoreBufferService

  • All Implemented Interfaces:
    java.io.Closeable, java.lang.AutoCloseable
    public class StoreBufferService
extends AbstractStoreBufferService
    This class is serving as a PubSubMessage buffer with an accompanying pool of drainer threads. The drainers pull records out of the buffer and delegate the persistence and validation to the appropriate StoreIngestionTask. High-level idea: 1. StoreBufferService will be maintaining a fixed number (configurable) of StoreBufferService.StoreBufferDrainer pool; 2. For each StoreBufferService.StoreBufferDrainer, there is a corresponding BlockingQueue, which will buffer StoreBufferService.QueueNode; 3. All the records belonging to the same topic+partition will be allocated to the same drainer thread, otherwise DIV will fail; 4. The logic to assign topic+partition to drainer, please check getDrainerIndexForConsumerRecord(PubSubMessage, int); 5. There is still a thread executing StoreIngestionTask for each topic, which will handle admin actions, such as subscribe, unsubscribe, kill and so on, and also poll consumer records from Kafka and put them into blockingQueueArr maintained by StoreBufferService; For now, the assumption is that one-consumer-polling-thread should be fast enough to catch up with Kafka MM replication, and data processing is the slowest part. If we find that polling is also slow later on, we may consider to adopt a consumer thread pool to speed up polling from local Kafka brokers.