Package com.linkedin.davinci.kafka.consumer

Class SeparatedStoreBufferService

java.lang.Object

com.linkedin.venice.service.AbstractVeniceService

com.linkedin.davinci.kafka.consumer.AbstractStoreBufferService

com.linkedin.davinci.kafka.consumer.SeparatedStoreBufferService

All Implemented Interfaces:

Closeable, AutoCloseable

public class SeparatedStoreBufferService
extends AbstractStoreBufferService

This store buffer services maintains two separate drainer queues for store ingestions. For the sorted messages, Venice SN could use SSTFileWriter to ingest into RocksDB, and the performance is constant and stable. But for the unsorted messages, RocksDB behavior is not constant because of RocksDB compaction and sometimes write compute. Since there are very different characteristics, it will be helpful to decouple these two types of ingestions to avoid one blocking the other.

Nested Class Summary

Nested classes/interfaces inherited from class com.linkedin.venice.service.AbstractVeniceService

AbstractVeniceService.ServiceState

Field Summary

Fields

Modifier and Type	Field	Description
protected final StoreBufferService	sortedStoreBufferServiceDelegate
protected final StoreBufferService	unsortedStoreBufferServiceDelegate

Fields inherited from class com.linkedin.venice.service.AbstractVeniceService

logger, serviceState

Method Summary

Modifier and Type	Method	Description
void	drainBufferedRecordsFromTopicPartition(PubSubTopicPartition topicPartition)	This method will wait for all the messages to be processed (persisted to disk) that are already queued up to drainer till now.
CompletableFuture<Void>	execSyncOffsetCommandAsync(PubSubTopicPartition topicPartition, StoreIngestionTask ingestionTask)
void	execSyncOffsetFromSnapshotAsync(PubSubTopicPartition topicPartition, PartitionTracker vtDivSnapshot, StoreIngestionTask ingestionTask)
long	getMaxMemoryUsagePerDrainer()
long	getMinMemoryUsagePerDrainer()
long	getTotalMemoryUsage()
long	getTotalRemainingMemory()
void	putConsumerRecord(DefaultPubSubMessage consumerRecord, StoreIngestionTask ingestionTask, LeaderProducedRecordContext leaderProducedRecordContext, int partition, String kafkaUrl, long beforeProcessingRecordTimestampNs)
boolean	startInner()
void	stopInner()

Methods inherited from class com.linkedin.venice.service.AbstractVeniceService

close, getName, isRunning, start, stop

Methods inherited from class java.lang.Object

clone, equals, finalize, getClass, hashCode, notify, notifyAll, toString, wait, wait, wait

Field Details

sortedStoreBufferServiceDelegate

protected final StoreBufferService sortedStoreBufferServiceDelegate

unsortedStoreBufferServiceDelegate

protected final StoreBufferService unsortedStoreBufferServiceDelegate

Method Details

putConsumerRecord

public void putConsumerRecord(DefaultPubSubMessage consumerRecord,
 StoreIngestionTask ingestionTask,
 LeaderProducedRecordContext leaderProducedRecordContext,
 int partition,
 String kafkaUrl,
 long beforeProcessingRecordTimestampNs)
                       throws InterruptedException

Specified by:

putConsumerRecord in class AbstractStoreBufferService

Throws:

InterruptedException

drainBufferedRecordsFromTopicPartition

public void drainBufferedRecordsFromTopicPartition(PubSubTopicPartition topicPartition)
                                            throws InterruptedException

Description copied from class: AbstractStoreBufferService

This method will wait for all the messages to be processed (persisted to disk) that are already queued up to drainer till now.

Specified by:

drainBufferedRecordsFromTopicPartition in class AbstractStoreBufferService

Throws:

InterruptedException

execSyncOffsetCommandAsync

public CompletableFuture<Void> execSyncOffsetCommandAsync(PubSubTopicPartition topicPartition,
 StoreIngestionTask ingestionTask)
                                                   throws InterruptedException

Specified by:

execSyncOffsetCommandAsync in class AbstractStoreBufferService

Throws:

InterruptedException

execSyncOffsetFromSnapshotAsync

public void execSyncOffsetFromSnapshotAsync(PubSubTopicPartition topicPartition,
 PartitionTracker vtDivSnapshot,
 StoreIngestionTask ingestionTask)
                                     throws InterruptedException

Specified by:

execSyncOffsetFromSnapshotAsync in class AbstractStoreBufferService

Throws:

InterruptedException

startInner

public boolean startInner()
                   throws Exception

Specified by:

startInner in class AbstractVeniceService

Returns:

true if the service is completely started, false if it is still starting asynchronously (in this case, it is the implementer's responsibility to set AbstractVeniceService.serviceState to AbstractVeniceService.ServiceState.STARTED upon completion of the async work).

Throws:

Exception

stopInner

public void stopInner()
               throws Exception

Specified by:

stopInner in class AbstractVeniceService

Throws:

Exception

getTotalMemoryUsage

public long getTotalMemoryUsage()

Specified by:

getTotalMemoryUsage in class AbstractStoreBufferService

getTotalRemainingMemory

public long getTotalRemainingMemory()

Specified by:

getTotalRemainingMemory in class AbstractStoreBufferService

getMaxMemoryUsagePerDrainer

public long getMaxMemoryUsagePerDrainer()

Specified by:

getMaxMemoryUsagePerDrainer in class AbstractStoreBufferService

getMinMemoryUsagePerDrainer

public long getMinMemoryUsagePerDrainer()

Specified by:

getMinMemoryUsagePerDrainer in class AbstractStoreBufferService