Package com.linkedin.davinci.blobtransfer

Class BlobSnapshotManager

java.lang.Object

com.linkedin.davinci.blobtransfer.BlobSnapshotManager

public class BlobSnapshotManager
extends Object

This class will manage the snapshot creation, for batch store and hybrid store.

Field Summary

Fields

Modifier and Type	Field	Description
static final int	DEFAULT_MAX_CONCURRENT_USERS

Constructor Summary

Constructors

Constructor	Description
BlobSnapshotManager(ReadOnlyStoreRepository readOnlyStoreRepository, StorageEngineRepository storageEngineRepository, StorageMetadataService storageMetadataService)	The constructor for the BlobSnapshotManager, with default max concurrent users and snapshot retention time
BlobSnapshotManager(ReadOnlyStoreRepository readOnlyStoreRepository, StorageEngineRepository storageEngineRepository, StorageMetadataService storageMetadataService, int maxConcurrentUsers, int snapshotRetentionTimeInMin, BlobTransferUtils.BlobTransferTableFormat transferTableFormat)	Constructor for the BlobSnapshotManager

Method Summary

Modifier and Type	Method	Description
void	createSnapshot(String kafkaVersionTopic, int partitionId)	Create a snapshot for a particular partition
static void	createSnapshot(org.rocksdb.RocksDB rocksDB, String fullPathForPartitionDBSnapshot)	util method to create a snapshot It will check the snapshot directory and delete it if it exists, then generate a new snapshot
void	decreaseConcurrentUserCount(BlobTransferPayload blobTransferRequest)	Decrease the count of hosts using the snapshot
BlobTransferUtils.BlobTransferTableFormat	getBlobTransferTableFormat()	Get the current snapshot format, which is a config value.
protected int	getConcurrentSnapshotUsers(String topicName, int partitionId)
BlobTransferPartitionMetadata	getTransferMetadata(BlobTransferPayload payload)	Get the transfer metadata for a particular payload 0.
BlobTransferPartitionMetadata	getTransferredSnapshotMetadata(String topicName, int partitionId)	Get the snapshot metadata for a particular topic and partition
boolean	isStoreHybrid(String storeName, int versionNum)	Check if the store is hybrid
BlobTransferPartitionMetadata	prepareMetadata(BlobTransferPayload blobTransferRequest)	Prepare the metadata for a blob transfer request

Methods inherited from class java.lang.Object

clone, equals, finalize, getClass, hashCode, notify, notifyAll, toString, wait, wait, wait

Field Details

DEFAULT_MAX_CONCURRENT_USERS

public static final int DEFAULT_MAX_CONCURRENT_USERS

See Also:

• Constant Field Values

Constructor Details

BlobSnapshotManager

public BlobSnapshotManager(ReadOnlyStoreRepository readOnlyStoreRepository,
 StorageEngineRepository storageEngineRepository,
 StorageMetadataService storageMetadataService,
 int maxConcurrentUsers,
 int snapshotRetentionTimeInMin,
 BlobTransferUtils.BlobTransferTableFormat transferTableFormat)

Constructor for the BlobSnapshotManager

BlobSnapshotManager

public BlobSnapshotManager(ReadOnlyStoreRepository readOnlyStoreRepository,
 StorageEngineRepository storageEngineRepository,
 StorageMetadataService storageMetadataService)

The constructor for the BlobSnapshotManager, with default max concurrent users and snapshot retention time

Method Details

getTransferMetadata

public BlobTransferPartitionMetadata getTransferMetadata(BlobTransferPayload payload)
                                                  throws VeniceException

Get the transfer metadata for a particular payload 0. pre-check: throttle the request if many concurrent users. 1. the store is not hybrid, it will prepare the metadata and return it. 2. the store is hybrid: 2. 1. check snapshot staleness 2. 1. 1. if stale, recreate the snapshot and metadata, then return the metadata 2. 1. 2. if not stale, directly return the metadata

Parameters:

payload - the blob transfer payload

Returns:

the need transfer metadata to client

Throws:

VeniceException

decreaseConcurrentUserCount

public void decreaseConcurrentUserCount(BlobTransferPayload blobTransferRequest)

Decrease the count of hosts using the snapshot

getConcurrentSnapshotUsers

protected int getConcurrentSnapshotUsers(String topicName,
 int partitionId)

isStoreHybrid

public boolean isStoreHybrid(String storeName,
 int versionNum)

Check if the store is hybrid

Parameters:

storeName - the name of the store

versionNum - the version number

Returns:

true if the store is hybrid, false otherwise

createSnapshot

public static void createSnapshot(org.rocksdb.RocksDB rocksDB,
 String fullPathForPartitionDBSnapshot)

util method to create a snapshot It will check the snapshot directory and delete it if it exists, then generate a new snapshot

createSnapshot

public void createSnapshot(String kafkaVersionTopic,
 int partitionId)

Create a snapshot for a particular partition

getTransferredSnapshotMetadata

public BlobTransferPartitionMetadata getTransferredSnapshotMetadata(String topicName,
 int partitionId)

Get the snapshot metadata for a particular topic and partition

Parameters:

topicName - the topic name

partitionId - the partition id

Returns:

the snapshot metadata

prepareMetadata

public BlobTransferPartitionMetadata prepareMetadata(BlobTransferPayload blobTransferRequest)

Prepare the metadata for a blob transfer request

Parameters:

blobTransferRequest - the blob transfer request

Returns:

the metadata for the blob transfer request

getBlobTransferTableFormat

public BlobTransferUtils.BlobTransferTableFormat getBlobTransferTableFormat()

Get the current snapshot format, which is a config value.

Returns:

the transfer table format, BLOCK_BASED_TABLE or PLAIN_TABLE.