Package com.linkedin.davinci.ingestion

Class DefaultIngestionBackend

java.lang.Object

com.linkedin.davinci.ingestion.DefaultIngestionBackend

All Implemented Interfaces:

IngestionBackend, java.io.Closeable, java.lang.AutoCloseable

Direct Known Subclasses:

IsolatedIngestionBackend

public class DefaultIngestionBackend
extends java.lang.Object
implements IngestionBackend

The default ingestion backend implementation. Ingestion will be done in the same JVM as the application.

Constructor Summary

Constructors

Constructor	Description
DefaultIngestionBackend(StorageMetadataService storageMetadataService, KafkaStoreIngestionService storeIngestionService, StorageService storageService, BlobTransferManager blobTransferManager, VeniceServerConfig serverConfig)

Method Summary

Modifier and Type	Method	Description
void	addIngestionNotifier(VeniceNotifier ingestionListener)
void	close()
java.util.concurrent.CompletableFuture<java.lang.Void>	dropStoragePartitionGracefully(VeniceStoreVersionConfig storeConfig, int partition, int timeoutInSeconds, boolean removeEmptyStorageEngine)	This method stops to subscribe the specified topic partition and delete partition data from storage.
KafkaStoreIngestionService	getStoreIngestionService()
boolean	hasCurrentVersionBootstrapping()	Check whether there are any current version bootstrapping or not.
boolean	isOffsetLagged(java.lang.String store, int versionNumber, int partition, long blobTransferDisabledOffsetLagThreshold)	A helper method to check if the offset lag is within the allowed threshold.
void	killConsumptionTask(java.lang.String topicName)
void	removeStorageEngine(java.lang.String topicName)
void	setStorageEngineReference(java.lang.String topicName, java.util.concurrent.atomic.AtomicReference<AbstractStorageEngine> storageEngineReference)
void	shutdownIngestionTask(java.lang.String topicName)
void	startConsumption(VeniceStoreVersionConfig storeConfig, int partition)
java.util.concurrent.CompletableFuture<java.lang.Void>	stopConsumption(VeniceStoreVersionConfig storeConfig, int partition)

Methods inherited from class java.lang.Object

clone, equals, finalize, getClass, hashCode, notify, notifyAll, toString, wait, wait, wait

Methods inherited from interface com.linkedin.davinci.ingestion.IngestionBackend

dropStoragePartitionGracefully

Constructor Detail

DefaultIngestionBackend

public DefaultIngestionBackend(StorageMetadataService storageMetadataService,
                               KafkaStoreIngestionService storeIngestionService,
                               StorageService storageService,
                               BlobTransferManager blobTransferManager,
                               VeniceServerConfig serverConfig)

Method Detail

startConsumption

public void startConsumption(VeniceStoreVersionConfig storeConfig,
                             int partition)

Specified by:

startConsumption in interface IngestionBackend

isOffsetLagged

public boolean isOffsetLagged(java.lang.String store,
                              int versionNumber,
                              int partition,
                              long blobTransferDisabledOffsetLagThreshold)

A helper method to check if the offset lag is within the allowed threshold. If the offset lag is smaller than the `blobTransferDisabledOffsetLagThreshold`, bootstrapping from Kafka firstly, even if blob transfer is enabled.

Parameters:

store - the store name

versionNumber - the version number

partition - the partition number

blobTransferDisabledOffsetLagThreshold - the maximum allowed offset lag threshold. If the offset lag is within this threshold, bootstrapping from Kafka is allowed, even if blob transfer is enabled. If the lag exceeds this threshold, bootstrapping should happen from blobs transfer firstly.

Returns:

true if the offset lag exceeds the threshold or if the lag is 0, indicating bootstrapping should happen from blobs transfer. false otherwise

stopConsumption

public java.util.concurrent.CompletableFuture<java.lang.Void> stopConsumption(VeniceStoreVersionConfig storeConfig,
                                                                              int partition)

Specified by:

stopConsumption in interface IngestionBackend

killConsumptionTask

public void killConsumptionTask(java.lang.String topicName)

Specified by:

killConsumptionTask in interface IngestionBackend

shutdownIngestionTask

public void shutdownIngestionTask(java.lang.String topicName)

Specified by:

shutdownIngestionTask in interface IngestionBackend

removeStorageEngine

public void removeStorageEngine(java.lang.String topicName)

Specified by:

removeStorageEngine in interface IngestionBackend

dropStoragePartitionGracefully

public java.util.concurrent.CompletableFuture<java.lang.Void> dropStoragePartitionGracefully(VeniceStoreVersionConfig storeConfig,
                                                                                             int partition,
                                                                                             int timeoutInSeconds,
                                                                                             boolean removeEmptyStorageEngine)

Description copied from interface: IngestionBackend

This method stops to subscribe the specified topic partition and delete partition data from storage.

Specified by:

dropStoragePartitionGracefully in interface IngestionBackend

Parameters:

storeConfig - Store version config

partition - Partition number to be dropped in the store version.

timeoutInSeconds - Number of seconds to wait before timeout.

removeEmptyStorageEngine - Whether to drop storage engine when dropping the last partition.

Returns:

a future for the drop partition action.

addIngestionNotifier

public void addIngestionNotifier(VeniceNotifier ingestionListener)

Specified by:

addIngestionNotifier in interface IngestionBackend

setStorageEngineReference

public void setStorageEngineReference(java.lang.String topicName,
                                      java.util.concurrent.atomic.AtomicReference<AbstractStorageEngine> storageEngineReference)

Specified by:

setStorageEngineReference in interface IngestionBackend

hasCurrentVersionBootstrapping

public boolean hasCurrentVersionBootstrapping()

Description copied from interface: IngestionBackend

Check whether there are any current version bootstrapping or not.

Specified by:

hasCurrentVersionBootstrapping in interface IngestionBackend

getStoreIngestionService

public KafkaStoreIngestionService getStoreIngestionService()

Specified by:

getStoreIngestionService in interface IngestionBackend

close

public void close()

Specified by:

close in interface java.lang.AutoCloseable

Specified by:

close in interface java.io.Closeable