Package com.linkedin.davinci.blobtransfer

Class BlobSnapshotManager

java.lang.Object

com.linkedin.davinci.blobtransfer.BlobSnapshotManager

public class BlobSnapshotManager
extends Object

This class will manage the snapshot creation, for batch store and hybrid store.

Field Summary

Fields

Modifier and Type	Field	Description
static final int	DEFAULT_MAX_CONCURRENT_USERS
static final int	DEFAULT_SNAPSHOT_CLEANUP_INTERVAL_IN_MINS

Constructor Summary

Constructors

Constructor	Description
BlobSnapshotManager(StorageEngineRepository storageEngineRepository, StorageMetadataService storageMetadataService)	The constructor for the BlobSnapshotManager, with default max concurrent users and snapshot retention time
BlobSnapshotManager(StorageEngineRepository storageEngineRepository, StorageMetadataService storageMetadataService, int maxConcurrentUsers, int snapshotRetentionTimeInMin, BlobTransferUtils.BlobTransferTableFormat transferTableFormat, int snapshotCleanupIntervalInMins)	Constructor for the BlobSnapshotManager

Method Summary

Modifier and Type	Method	Description
void	cleanupOutOfRetentionSnapshot(String topicName, int partitionId)	A regular cleanup task to clean up the snapshot folder which is out of retention time.
void	cleanupSnapshot(String kafkaVersionTopic, int partitionId)	Cleanup the snapshot for a particular partition
void	createSnapshot(String kafkaVersionTopic, int partitionId)	Create a snapshot for a particular partition
void	decreaseConcurrentUserCount(BlobTransferPayload blobTransferRequest)	Decrease the count of hosts using the snapshot
BlobTransferUtils.BlobTransferTableFormat	getBlobTransferTableFormat()	Get the current snapshot format, which is a config value.
protected int	getConcurrentSnapshotUsers(String topicName, int partitionId)
BlobTransferPartitionMetadata	getTransferMetadata(BlobTransferPayload payload, AtomicBoolean successCountedAsActiveCurrentUser)	Get the transfer metadata for a particular payload 1.
BlobTransferPartitionMetadata	getTransferredSnapshotMetadata(String topicName, int partitionId)	Get the snapshot metadata for a particular topic and partition
BlobTransferPartitionMetadata	prepareMetadata(BlobTransferPayload blobTransferRequest)	Prepare the metadata for a blob transfer request
void	removeTrackingValues(String topicName, int partitionId)	Remove tracking values for a topic-partition when the snapshot is cleaned up
void	shutdown()

Methods inherited from class java.lang.Object

clone, equals, finalize, getClass, hashCode, notify, notifyAll, toString, wait, wait, wait

Field Details

DEFAULT_MAX_CONCURRENT_USERS

public static final int DEFAULT_MAX_CONCURRENT_USERS

See Also:

• Constant Field Values

DEFAULT_SNAPSHOT_CLEANUP_INTERVAL_IN_MINS

public static final int DEFAULT_SNAPSHOT_CLEANUP_INTERVAL_IN_MINS

See Also:

• Constant Field Values

Constructor Details

BlobSnapshotManager

public BlobSnapshotManager(StorageEngineRepository storageEngineRepository,
 StorageMetadataService storageMetadataService,
 int maxConcurrentUsers,
 int snapshotRetentionTimeInMin,
 BlobTransferUtils.BlobTransferTableFormat transferTableFormat,
 int snapshotCleanupIntervalInMins)

Constructor for the BlobSnapshotManager

BlobSnapshotManager

public BlobSnapshotManager(StorageEngineRepository storageEngineRepository,
 StorageMetadataService storageMetadataService)

The constructor for the BlobSnapshotManager, with default max concurrent users and snapshot retention time

Method Details

getTransferMetadata

public BlobTransferPartitionMetadata getTransferMetadata(BlobTransferPayload payload,
 AtomicBoolean successCountedAsActiveCurrentUser)
                                                  throws VeniceException

Get the transfer metadata for a particular payload 1. throttle the request if many concurrent users. 2. check snapshot staleness 2.1. if stale: 2.1.1. if it does not have active users: recreate the snapshot and metadata, then return the metadata 2.1.2. if it has active users: no need to recreate the snapshot, throw an exception to let the client move to next candidate. 2.2. if not stale, directly return the metadata

Parameters:

payload - the blob transfer payload

successCountedAsActiveCurrentUser - Indicates whether this request has been successfully counted as an active user. Typically, we should increment the active concurrent user count at the beginning of receiving request on the server handler side. However, since we need to check the count of active users before recreating a snapshot for ensuring no active users are present, we move the increment logic to here. We also need to decrement the active user count when the request is completed or fails at the server side handler. This flag (successCountedCurrentUser) lets us know if this request was counted as an active user, so we can accurately decrement the count later.

Returns:

the need transfer metadata to client

Throws:

VeniceException

decreaseConcurrentUserCount

public void decreaseConcurrentUserCount(BlobTransferPayload blobTransferRequest)

Decrease the count of hosts using the snapshot

getConcurrentSnapshotUsers

protected int getConcurrentSnapshotUsers(String topicName,
 int partitionId)

createSnapshot

public void createSnapshot(String kafkaVersionTopic,
 int partitionId)

Create a snapshot for a particular partition

cleanupSnapshot

public void cleanupSnapshot(String kafkaVersionTopic,
 int partitionId)

Cleanup the snapshot for a particular partition

Parameters:

kafkaVersionTopic - the topic name

partitionId - the partition id

getTransferredSnapshotMetadata

public BlobTransferPartitionMetadata getTransferredSnapshotMetadata(String topicName,
 int partitionId)

Get the snapshot metadata for a particular topic and partition

Parameters:

topicName - the topic name

partitionId - the partition id

Returns:

the snapshot metadata

prepareMetadata

public BlobTransferPartitionMetadata prepareMetadata(BlobTransferPayload blobTransferRequest)

Prepare the metadata for a blob transfer request

Parameters:

blobTransferRequest - the blob transfer request

Returns:

the metadata for the blob transfer request

getBlobTransferTableFormat

public BlobTransferUtils.BlobTransferTableFormat getBlobTransferTableFormat()

Get the current snapshot format, which is a config value.

Returns:

the transfer table format, BLOCK_BASED_TABLE or PLAIN_TABLE.

removeTrackingValues

public void removeTrackingValues(String topicName,
 int partitionId)

Remove tracking values for a topic-partition when the snapshot is cleaned up

cleanupOutOfRetentionSnapshot

public void cleanupOutOfRetentionSnapshot(String topicName,
 int partitionId)

A regular cleanup task to clean up the snapshot folder which is out of retention time.

shutdown

public void shutdown()