Package com.linkedin.davinci.ingestion.main

Class MainIngestionStorageMetadataService

java.lang.Object

com.linkedin.venice.service.AbstractVeniceService

com.linkedin.davinci.ingestion.main.MainIngestionStorageMetadataService

All Implemented Interfaces:

StorageMetadataService, OffsetManager, Closeable, AutoCloseable

public class MainIngestionStorageMetadataService
extends AbstractVeniceService
implements StorageMetadataService

MainIngestionStorageMetadataService is an in-memory storage metadata service for IsolatedIngestionBackend. It keeps storage metadata in the memory so RocksDB metadata partitions can be opened by isolated ingestion process only. For metadata update generated by hybrid ingestion, it will sync and persist the update to the RocksDB metadata partition through IPC protocol.

Nested Class Summary

Nested classes/interfaces inherited from class com.linkedin.venice.service.AbstractVeniceService

AbstractVeniceService.ServiceState

Field Summary

Fields inherited from class com.linkedin.venice.service.AbstractVeniceService

logger, serviceState

Constructor Summary

Constructors

Constructor	Description
MainIngestionStorageMetadataService(int targetPort, InternalAvroSpecificSerializer<PartitionState> partitionStateSerializer, MetadataUpdateStats metadataUpdateStats, VeniceConfigLoader configLoader, BiConsumer<String,StoreVersionState> storeVersionStateSyncer)

Method Summary

Modifier and Type	Method	Description
void	clearOffset(String topicName, int partitionId)
void	clearStoreVersionState(String topicName)	This will clear all metadata, including store-version state and partition states, tied to .
StoreVersionState	computeStoreVersionState(String topicName, Function<StoreVersionState,StoreVersionState> mapFunction)
OffsetRecord	getLastOffset(String topicName, int partitionId, PubSubContext pubSubContext)	Gets the Last Known persisted offset of this consumer.
StoreVersionState	getStoreVersionState(String topicName)	Gets the currently-persisted StoreVersionState for this topic.
void	put(String topicName, int partitionId, OffsetRecord record)	Records the offset with underlying/external storage.
void	putOffsetRecord(String topicName, int partitionId, OffsetRecord record)	putOffsetRecord will only put OffsetRecord into in-memory state, without persisting into metadata RocksDB partition.
void	putStoreVersionState(String topicName, StoreVersionState record)	putStoreVersionState will only put StoreVersionState into in-memory state, without persisting into metadata RocksDB partition.
boolean	startInner()
void	stopInner()

Methods inherited from class com.linkedin.venice.service.AbstractVeniceService

close, getName, isRunning, start, stop

Methods inherited from class java.lang.Object

clone, equals, finalize, getClass, hashCode, notify, notifyAll, toString, wait, wait, wait

Methods inherited from interface com.linkedin.davinci.storage.StorageMetadataService

getStoreVersionCompressionDictionary

Constructor Details

MainIngestionStorageMetadataService

public MainIngestionStorageMetadataService(int targetPort,
 InternalAvroSpecificSerializer<PartitionState> partitionStateSerializer,
 MetadataUpdateStats metadataUpdateStats,
 VeniceConfigLoader configLoader,
 BiConsumer<String,StoreVersionState> storeVersionStateSyncer)

Method Details

startInner

public boolean startInner()
                   throws Exception

Specified by:

startInner in class AbstractVeniceService

Returns:

true if the service is completely started, false if it is still starting asynchronously (in this case, it is the implementer's responsibility to set AbstractVeniceService.serviceState to AbstractVeniceService.ServiceState.STARTED upon completion of the async work).

Throws:

Exception

stopInner

public void stopInner()
               throws Exception

Specified by:

stopInner in class AbstractVeniceService

Throws:

Exception

computeStoreVersionState

public StoreVersionState computeStoreVersionState(String topicName,
 Function<StoreVersionState,StoreVersionState> mapFunction)
                                           throws VeniceException

Specified by:

computeStoreVersionState in interface StorageMetadataService

Throws:

VeniceException

clearStoreVersionState

public void clearStoreVersionState(String topicName)

Description copied from interface: StorageMetadataService

This will clear all metadata, including store-version state and partition states, tied to .

Specified by:

clearStoreVersionState in interface StorageMetadataService

Parameters:

topicName - to be cleared

getStoreVersionState

public StoreVersionState getStoreVersionState(String topicName)
                                       throws VeniceException

Description copied from interface: StorageMetadataService

Gets the currently-persisted StoreVersionState for this topic.

Specified by:

getStoreVersionState in interface StorageMetadataService

Parameters:

topicName - kafka topic to which the consumer thread is registered to.

Returns:

an instance of StoreVersionState corresponding to this topic, or null if there isn't any.

Throws:

VeniceException

put

public void put(String topicName,
 int partitionId,
 OffsetRecord record)
         throws VeniceException

Description copied from interface: OffsetManager

Records the offset with underlying/external storage. Persistence to disk happens in configurable time interval by a background thread. For example in case of BDB the check pointer thread can be configured to do this.

Specified by:

put in interface OffsetManager

Parameters:

topicName - kafka topic to which the consumer thread is registered to.

partitionId - kafka partition id for which the consumer thread is registered to.

record - OffSetRecord containing last read offset for the topic and partition combination.

Throws:

VeniceException

clearOffset

public void clearOffset(String topicName,
 int partitionId)

Specified by:

clearOffset in interface OffsetManager

getLastOffset

public OffsetRecord getLastOffset(String topicName,
 int partitionId,
 PubSubContext pubSubContext)
                           throws VeniceException

Description copied from interface: OffsetManager

Gets the Last Known persisted offset of this consumer.

Specified by:

getLastOffset in interface OffsetManager

Parameters:

topicName - kafka topic to which the consumer thread is registered to.

partitionId - kafka partition id for which the consumer thread is registered to.

pubSubContext - PubSubContext for passing PubSub related dependencies

Returns:

OffsetRecord - contains offset and time when it was recorded before the consumer thread went down. consumer

Throws:

VeniceException

putOffsetRecord

public void putOffsetRecord(String topicName,
 int partitionId,
 OffsetRecord record)

putOffsetRecord will only put OffsetRecord into in-memory state, without persisting into metadata RocksDB partition.

putStoreVersionState

public void putStoreVersionState(String topicName,
 StoreVersionState record)

putStoreVersionState will only put StoreVersionState into in-memory state, without persisting into metadata RocksDB partition.