Interface VeniceChangelogConsumer<K,V>
- Type Parameters:
K- The Type for keyV- The Type for value
- All Superinterfaces:
AutoCloseable
- All Known Implementing Classes:
VeniceAfterImageConsumerImpl,VeniceChangelogConsumerDaVinciRecordTransformerImpl,VeniceChangelogConsumerImpl
-
Method Summary
Modifier and TypeMethodDescriptionvoidclose()Releases the internal resources.default voidCommits the consumer's current positions back to the underlying pub-sub broker so external monitoring tools (e.g.Returns the timestamp of the last heartbeat received for each subscribed partition.intbooleanChecks whether all subscribed partitions are caught up during bootstrap.voidpause()Deprecated.voidDeprecated.poll(long timeoutInMs) Polling function to get any available messages from the underlying system for all partitions subscribed.voidresume()Deprecated.voidDeprecated.Seek to the beginning of the push for subscribed partitions.seekToBeginningOfPush(Set<Integer> partitions) Seek to the beginning of the push for a set of partitions.seekToCheckpoint(Set<VeniceChangeCoordinate> checkpoints) Seek the provided checkpoints for the specified partitions.Deprecated.seekToEndOfPush(Set<Integer> partitions) Deprecated.Seek to the end of events which have been transmitted to Venice for all subscribed partitions.seekToTail(Set<Integer> partitions) Seek to the end of events which have been transmitted to Venice and start consuming new events.seekToTimestamp(Long timestamp) Seek to the specified timestamp for all subscribed partitions.seekToTimestamps(Map<Integer, Long> timestamps) Seek to the provided timestamps for the specified partitions based on wall clock time for when this message was processed by Venice and produced to change capture.default voidRegisters a listener told which topic each partition is consumed from, replacing any previous listener.Subscribe a set of partitions for a store to this VeniceChangelogConsumer, with the earliest position in the topic for each partition.Subscribe all partitions belonging to a specific store, with the earliest position in the topic for each partition.voidunsubscribe(Set<Integer> partitions) Stop ingesting messages from a set of partitions for a specific store.voidStop ingesting messages from all partitions.
-
Method Details
-
getPartitionCount
int getPartitionCount()- Returns:
- total number of store partitions
-
subscribe
Subscribe a set of partitions for a store to this VeniceChangelogConsumer, with the earliest position in the topic for each partition. The VeniceChangelogConsumer should try to consume messages from all partitions that are subscribed to it.- Parameters:
partitions- the set of partition to subscribe and consume- Returns:
- a future which completes when data from the partitions are ready to be consumed
- Throws:
VeniceException- if subscribe operation failed for any of the partitions
-
subscribeAll
CompletableFuture<Void> subscribeAll()Subscribe all partitions belonging to a specific store, with the earliest position in the topic for each partition.- Returns:
- a future which completes when all partitions are ready to be consumed data
- Throws:
VeniceException- if subscribe operation failed for any of the partitions
-
unsubscribe
Stop ingesting messages from a set of partitions for a specific store.- Parameters:
partitions- The set of topic partitions to unsubscribe- Throws:
VeniceException- if unsubscribe operation failed for any of the partitions
-
unsubscribeAll
void unsubscribeAll()Stop ingesting messages from all partitions.- Throws:
VeniceException- if unsubscribe operation failed for any of the partitions
-
seekToBeginningOfPush
Seek to the beginning of the push for a set of partitions. This is analogous to doing a bootstrap of data for the consumer. This seek will ONLY seek to the beginning of the version which is currently serving data, and the consumer will switch to reading data from a new version (should one get created) once it has read up to the point in the change capture stream that indicates the version swap (which can only occur after consuming all the data in the last push). This instructs the consumer to consume data from the batch push.- Parameters:
partitions- the set of partitions to seek with- Returns:
- a future which completes when the operation has succeeded for all partitions.
- Throws:
VeniceException- if seek operation failed for any of the partitions, or seeking was performed on unsubscribed partitions
-
seekToBeginningOfPush
CompletableFuture<Void> seekToBeginningOfPush()Seek to the beginning of the push for subscribed partitions. SeeseekToBeginningOfPush(Set)for more information.- Returns:
- a future which completes when the partitions are ready to be consumed data
- Throws:
VeniceException- if seek operation failed for any of the partitions.
-
seekToEndOfPush
Deprecated.Seek to the end of the last push for a given set of partitions. This instructs the consumer to begin consuming events which are transmitted to Venice following the last batch push.- Parameters:
partitions- the set of partitions to seek with- Returns:
- a future which completes when the operation has succeeded for all partitions.
- Throws:
VeniceException- if seek operation failed for any of the partitions, or seeking was performed on unsubscribed partitions
-
seekToEndOfPush
Deprecated.Seek to the end of the push for all subscribed partitions. SeeseekToEndOfPush(Set)for more information.- Returns:
- a future which completes when the operation has succeeded for all partitions.
- Throws:
VeniceException- if seek operation failed for any of the partitions.
-
seekToTail
Seek to the end of events which have been transmitted to Venice and start consuming new events. This will ONLY consume events transmitted via nearline and incremental push. It will not read batch push data.- Parameters:
partitions- the set of partitions to seek with- Returns:
- a future which completes when the operation has succeeded for all partitions.
- Throws:
VeniceException- if seek operation failed for any of the partitions, or seeking was performed on unsubscribed partitions
-
seekToTail
CompletableFuture<Void> seekToTail()Seek to the end of events which have been transmitted to Venice for all subscribed partitions. SeeseekToTail(Set)for more information.- Returns:
- a future which completes when the operation has succeeded for all partitions.
- Throws:
VeniceException- if seek operation failed for any of the partitions.
-
seekToCheckpoint
CompletableFuture<Void> seekToCheckpoint(Set<VeniceChangeCoordinate> checkpoints) throws VeniceCoordinateOutOfRangeException Seek the provided checkpoints for the specified partitions. Note about checkpoints: Checkpoints have the following properties and should be considered: - Checkpoints are NOT comparable or valid across partitions. - Checkpoints are NOT comparable or valid across regions - Checkpoints are NOT comparable across store versions - It is not possible to determine the number of events between two checkpoints - It is possible that a checkpoint is no longer on retention. In such case, we will return an exception to the caller.- Parameters:
checkpoints-- Returns:
- a future which completes when seek has completed for all partitions
- Throws:
VeniceException- if seek operation failed for any of the partitionsVeniceCoordinateOutOfRangeException- if passed checkpoint is no longer valid
-
seekToTimestamps
Seek to the provided timestamps for the specified partitions based on wall clock time for when this message was processed by Venice and produced to change capture. Note, this API can only be used to seek on nearline data applied to the current serving version in Venice. This will not seek on data transmitted via Batch Push. If the provided timestamp is lower than the earliest timestamp on a given stream, the earliest event will be returned. THIS WILL NOT SEEK TO DATA WHICH WAS APPLIED ON A PREVIOUS VERSION. You should never seek back in time to a timestamp which is smaller than the current time - rewindTimeInSeconds configured in the hybrid settings for this Venice store. The timestamp passed to this function should be associated to timestamps processed by this interface. The timestamp returned byPubSubMessage.getPubSubMessageTime()is the best-available message timestamp: the pub-sub system timestamp when available, otherwise the Venice producer timestamp. Calls to this method will seek based on that sequence of events. Note: the precision of the seek depends on the underlying pub-sub system — some systems may return a position that is close to, but not exactly at, the requested timestamp.- Parameters:
timestamps- a map keyed by a partition ID, and the timestamp checkpoints to seek for each partition.- Returns:
- Throws:
VeniceException- if seek operations failed for any of the specified partitions.
-
seekToTimestamp
Seek to the specified timestamp for all subscribed partitions. SeeseekToTimestamps(Map)for more information.- Returns:
- a future which completes when the operation has succeeded for all partitions.
- Throws:
VeniceException- if seek operation failed for any of the partitions.
-
pause
Deprecated.Pause the client on all or subset of partitions this client is subscribed to. Calls topoll(long)will not return results from paused partitions untilresume(Set)is called again later for those partitions.- Throws:
VeniceException- if operation failed for any of the partitions.
-
pause
Deprecated.Pause the client on all subscriptions. Seepause(Set)for more information.- Throws:
VeniceException- if operation failed for any of the partitions.
-
resume
Deprecated.Resume the client on all or a subset of partitions this client is subscribed to and has paused.- Throws:
VeniceException- if operation failed for any of the partitions.
-
resume
Deprecated.Pause the client on all subscriptions. Seeresume(Set)for more information.- Throws:
VeniceException- if operation failed for any of the partitions.
-
poll
Polling function to get any available messages from the underlying system for all partitions subscribed.- Parameters:
timeoutInMs- The maximum time to block/wait in between two polling requests (must not be greater thanLong.MAX_VALUEmilliseconds)- Returns:
- a collection of messages since the last fetch for the subscribed list of topic partitions
- Throws:
VeniceException- if polling operation fails
-
commitOffsets
default void commitOffsets()Commits the consumer's current positions back to the underlying pub-sub broker so external monitoring tools (e.g. Kafka consumer-group lag dashboards) can observe progress. This is intended for monitoring purposes only — the caller's checkpoint state (e.g. Flink'sVeniceChangeCoordinatestate) remains the source of truth for offsets. A failed commit must not affect correctness; impls are expected to log and swallow transient broker failures.No-op unless
ChangelogClientConfig.setConsumerOffsetCommitEnabled(boolean)is set and a consumer group id is configured onChangelogClientConfig.getConsumerProperties()(for Kafka,kafka.group.id).Default is no-op; impls that wrap a Kafka pub-sub consumer override.
-
setVersionSwapListener
Registers a listener told which topic each partition is consumed from, replacing any previous listener.A notification is produced whenever partitions are (re)subscribed: by
subscribe(java.util.Set<java.lang.Integer>), by any seek, or by an internal version swap (including a rollback to a lower version). It carries the exact position each partition was moved to by a version swap; positions of (re)subscriptions are included only when the consumer knows them exactly (e.g. not forsubscribe(java.util.Set<java.lang.Integer>)). Only the latest topic of each partition is reported, and explicitly unsubscribed partitions are dropped.The listener is invoked synchronously on the thread calling
poll(long), insidepoll(long), beforepoll(long)returns any record from the new topic. Records from the old topic that were already fetched or buffered may still be returned afterwards. Notifications produced before registration are delivered by the first poll after it. If the listener throws,poll(long)throws aVeniceExceptionwithout fetching any records, and the failed and remaining notifications are delivered again by the nextpoll(long).The listener runs while the consumer blocks subscription changes and buffering, so it should return quickly and must not wait on other threads that use this consumer.
- Parameters:
listener- the listener to invoke, must not be null- Throws:
UnsupportedOperationException- if this consumer cannot report version swaps
-
isCaughtUp
boolean isCaughtUp()Checks whether all subscribed partitions are caught up during bootstrap. If a partition's (currentTimestamp - latestMessageTimestamp) is smaller or equal to 1 min, we consider this partition is caught up.- Returns:
- True if all subscribed partitions have caught up.
-
getLastHeartbeatPerPartition
Returns the timestamp of the last heartbeat received for each subscribed partition. Heartbeats are messages sent periodically by Venice servers to measure lag.- Returns:
- a map of partition number to the timestamp, in milliseconds, of the last heartbeat received for that partition.
-
close
void close()Releases the internal resources. This consumer instance cannot be restarted or sought after close. Create a new consumer fromVeniceChangelogConsumerClientFactoryto resume consumption.- Specified by:
closein interfaceAutoCloseable
-