Package com.linkedin.venice.vpj.pubsub.input

Class PubSubSplitIterator

java.lang.Object

com.linkedin.venice.vpj.pubsub.input.PubSubSplitIterator

All Implemented Interfaces:

AutoCloseable

public class PubSubSplitIterator
extends Object
implements AutoCloseable

PubSubSplitIterator provides an iterator-like abstraction over a bounded range of messages from a PubSubConsumerAdapter. It is designed for Venice Push Job (VPJ) repush use cases where a split of a topic-partition needs to be consumed deterministically between a start and an end position.

This class:

• Subscribes a PubSubConsumerAdapter to a specific PubSubTopicPartition.
• Starts reading from a caller-specified PubSubPosition (inclusive) and stops when reaching the end-exclusive position or the record count limit.
• Supports both numeric offsets and logical index–based offsets for progress tracking.
• Skips control messages transparently while tracking statistics about consumption.

Nested Class Summary

Nested Classes

Modifier and Type	Class	Description
static enum	PubSubSplitIterator.OffsetMode
static class	PubSubSplitIterator.PubSubInputRecord

Field Summary

Fields

Modifier and Type	Field	Description
static final int	DEFAULT_EMPTY_RESULT_RETRIES
static final long	DEFAULT_EMPTY_SLEEP_MS
static final long	DEFAULT_POLL_TIMEOUT_MS

Constructor Summary

Constructors

Constructor	Description
PubSubSplitIterator(PubSubConsumerAdapter pubSubConsumer, PubSubPartitionSplit split, boolean useLogicalIndexOffset)
PubSubSplitIterator(PubSubConsumerAdapter pubSubConsumer, PubSubPartitionSplit split, boolean useLogicalIndexOffset, long pollTimeoutMs, int emptyRetryTimes, long emptySleepMs)

Method Summary

Modifier and Type	Method	Description
void	close()
PubSubPosition	getCurrentPosition()
float	getProgress()
PubSubTopicPartition	getTopicPartition()
boolean	hasNext()
PubSubSplitIterator.PubSubInputRecord	next()
long	recordsRead()

Methods inherited from class java.lang.Object

clone, equals, finalize, getClass, hashCode, notify, notifyAll, toString, wait, wait, wait

Field Details

DEFAULT_POLL_TIMEOUT_MS

public static final long DEFAULT_POLL_TIMEOUT_MS

DEFAULT_EMPTY_RESULT_RETRIES

public static final int DEFAULT_EMPTY_RESULT_RETRIES

See Also:

• Constant Field Values

DEFAULT_EMPTY_SLEEP_MS

public static final long DEFAULT_EMPTY_SLEEP_MS

Constructor Details

PubSubSplitIterator

public PubSubSplitIterator(PubSubConsumerAdapter pubSubConsumer,
 PubSubPartitionSplit split,
 boolean useLogicalIndexOffset,
 long pollTimeoutMs,
 int emptyRetryTimes,
 long emptySleepMs)

PubSubSplitIterator

public PubSubSplitIterator(PubSubConsumerAdapter pubSubConsumer,
 PubSubPartitionSplit split,
 boolean useLogicalIndexOffset)

Method Details

hasNext

public boolean hasNext()

next

public PubSubSplitIterator.PubSubInputRecord next()
                                           throws IOException

Throws:

IOException

getProgress

public float getProgress()

recordsRead

public long recordsRead()

getCurrentPosition

public PubSubPosition getCurrentPosition()

getTopicPartition

public PubSubTopicPartition getTopicPartition()

close

public void close()

Specified by:

close in interface AutoCloseable