Package com.linkedin.venice.pubsub.api

Interface PubSubConsumerAdapter

  • All Superinterfaces:
    java.lang.AutoCloseable, java.io.Closeable
    All Known Implementing Classes:
    ApacheKafkaConsumerAdapter, MockInMemoryConsumer
    public interface PubSubConsumerAdapter
extends java.lang.AutoCloseable, java.io.Closeable
    An adapter for consuming messages from a Pub-Sub topic. Implementations of this interface are not expected to be thread safe. However, they are expected to provide the following guarantees: 1) Honor the timeout parameter for all methods that have one. 2) Non-blocking behavior for methods that do not have an explicit timeout parameter. In other words, they should timeout after the default timeout period: PubSubConstants.PUBSUB_CONSUMER_API_DEFAULT_TIMEOUT_MS.

    • Method Detail

      • subscribe

        void subscribe​(PubSubTopicPartition pubSubTopicPartition,
               long lastReadOffset)
        Subscribes to a topic-partition if it is not already subscribed. If the topic-partition is already subscribed, this method is a no-op. The method assumes that the topic-partition exists.
        Parameters:
        pubSubTopicPartition - The topic-partition to subscribe to.
        lastReadOffset - The last read offset for the topic-partition. A poll call following a subscribe call will return messages from the offset (lastReadOffset + 1).
        Throws:
        java.lang.IllegalArgumentException - If the topic-partition is null or if the partition number is negative.
        PubSubTopicDoesNotExistException - If the topic does not exist.

      • unSubscribe

        void unSubscribe​(PubSubTopicPartition pubSubTopicPartition)
        Unsubscribes the consumer from a specified topic-partition. If the consumer was previously subscribed to the given partition, it will be unsubscribed, and the associated partition assignments and tracked offsets will be updated accordingly.
        Parameters:
        pubSubTopicPartition - The PubSub topic-partition to unsubscribe from.

      • batchUnsubscribe

        void batchUnsubscribe​(java.util.Set<PubSubTopicPartition> pubSubTopicPartitionSet)
        Unsubscribes the consumer from a batch of topic-partitions.
        Parameters:
        pubSubTopicPartitionSet - A set of topic-partitions to unsubscribe from.

      • close

        void close()
        Closes the PubSub consumer and releases any associated resources.
        Specified by:
        close in interface java.lang.AutoCloseable
        Specified by:
        close in interface java.io.Closeable

      • poll

        java.util.Map<PubSubTopicPartition,​java.util.List<PubSubMessage<KafkaKey,​KafkaMessageEnvelope,​java.lang.Long>>> poll​(long timeoutMs)
        Polls the Kafka consumer for messages from the subscribed topic partitions within the specified time duration.
        Parameters:
        timeoutMs - The maximum time, in milliseconds, to wait for messages to be polled.
        Returns:
        A mapping of PubSub topic partitions to lists of PubSub messages retrieved from Kafka.
        Throws:
        PubSubClientException - If there is an error during message retrieval from Kafka.
        PubSubClientRetriableException - If a retriable exception occurs during polling attempts, with retries as configured.

      • hasAnySubscription

        boolean hasAnySubscription()
        Checks if the consumer has any active topic-partition subscriptions.
        Returns:
        true if the consumer is subscribed to one or more topic partitions, false otherwise.

      • hasSubscription

        boolean hasSubscription​(PubSubTopicPartition pubSubTopicPartition)
        Checks if the consumer is currently subscribed to the specified PubSub topic-partition.
        Parameters:
        pubSubTopicPartition - The PubSub topic-partition to check for subscription.
        Returns:
        true if the consumer is subscribed to the given topic-partition, false otherwise.

      • pause

        void pause​(PubSubTopicPartition pubSubTopicPartition)
        Pauses message consumption for the specified PubSub topic-partition. If the partition was not previously subscribed, this method is a no-op.
        Parameters:
        pubSubTopicPartition - The PubSub topic-partition for which to pause message consumption.

      • resume

        void resume​(PubSubTopicPartition pubSubTopicPartition)
        Resumes message consumption for the specified PubSub topic-partition. If the partition was not previously paused or if they were not subscribed at all, this method is a no-op.
        Parameters:
        pubSubTopicPartition - The PubSub topic partition for which to resume message consumption.

      • getAssignment

        java.util.Set<PubSubTopicPartition> getAssignment()
        Retrieves the set of PubSub topic-partitions currently assigned to the consumer.
        Returns:
        A set of PubSub topic-partitions representing the current assignment of the consumer.

      • getOffsetLag

        default long getOffsetLag​(PubSubTopicPartition pubSubTopicPartition)
        Retrieves the consuming offset lag for a PubSub topic partition. The offset lag represents the difference between the last consumed message offset and the latest available message offset for the partition.
        Parameters:
        pubSubTopicPartition - The PubSub topic partition for which to fetch the offset lag.
        Returns:
        The offset lag, which is zero or a positive value if a valid lag was collected by the consumer, or -1 if the lag cannot be determined or is not applicable.

      • getLatestOffset

        default long getLatestOffset​(PubSubTopicPartition pubSubTopicPartition)
        Retrieves the latest available offset for a PubSub topic partition.
        Parameters:
        pubSubTopicPartition - The PubSub topic partition for which to fetch the latest offset.
        Returns:
        The latest offset, which is zero or a positive value if an offset was collected by the consumer, or -1 if the offset cannot be determined or is not applicable.

      • offsetForTime

        java.lang.Long offsetForTime​(PubSubTopicPartition pubSubTopicPartition,
                             long timestamp,
                             java.time.Duration timeout)
        Retrieves the offset of the first message with a timestamp greater than or equal to the target timestamp for the specified PubSub topic-partition. If no such message is found, null will be returned for the partition.
        Parameters:
        pubSubTopicPartition - The PubSub topic-partition for which to fetch the offset.
        timestamp - The target timestamp to search for in milliseconds since the Unix epoch.
        timeout - The maximum duration to wait for the operation to complete.
        Returns:
        The offset of the first message with a timestamp greater than or equal to the target timestamp, or null if no such message is found for the partition.
        Throws:
        PubSubOpTimeoutException - If the operation times out while fetching the offset.
        PubSubClientException - If there is an error while attempting to fetch the offset.

      • offsetForTime

        java.lang.Long offsetForTime​(PubSubTopicPartition pubSubTopicPartition,
                             long timestamp)
        Retrieves the offset of the first message with a timestamp greater than or equal to the target timestamp for the specified PubSub topic-partition. If no such message is found, null will be returned for the partition.
        Parameters:
        pubSubTopicPartition - The PubSub topic-partition for which to fetch the offset.
        timestamp - The target timestamp to search for in milliseconds since the Unix epoch.
        Returns:
        The offset of the first message with a timestamp greater than or equal to the target timestamp, or null if no such message is found for the partition.
        Throws:
        PubSubOpTimeoutException - If the operation times out while fetching the offset.
        PubSubClientException - If there is an error while attempting to fetch the offset.

      • beginningOffset

        java.lang.Long beginningOffset​(PubSubTopicPartition pubSubTopicPartition,
                               java.time.Duration timeout)
        Retrieves the beginning offset for the specified PubSub topic-partition.
        Parameters:
        pubSubTopicPartition - The PubSub topic-partition for which to fetch the beginning offset.
        timeout - The maximum duration to wait for the operation to complete.
        Returns:
        The beginning offset of the specified topic-partition. If topic-partition exists but has no messages, the offset will be 0.
        Throws:
        PubSubOpTimeoutException - If the operation times out while fetching the beginning offset.
        PubSubClientException - If there is an error while attempting to fetch the beginning offset.

      • endOffsets

        java.util.Map<PubSubTopicPartition,​java.lang.Long> endOffsets​(java.util.Collection<PubSubTopicPartition> partitions,
                                                                    java.time.Duration timeout)
        Retrieves the end offsets for a collection of PubSub topic-partitions. The end offset represents the highest offset available in each specified partition, i.e., offset of the last message + 1. If there are no messages in a partition, the end offset will be 0.
        Parameters:
        partitions - A collection of PubSub topic-partitions for which to fetch the end offsets.
        timeout - The maximum duration to wait for the operation to complete.
        Returns:
        A mapping of PubSub topic partitions to their respective end offsets, or an empty map if the offsets cannot be determined.
        Throws:
        PubSubOpTimeoutException - If the operation times out while fetching the end offsets.
        PubSubClientException - If there is an error while attempting to fetch the end offsets.

      • endOffset

        java.lang.Long endOffset​(PubSubTopicPartition pubSubTopicPartition)
        Retrieves the end offset for the specified PubSub topic-partition. The end offset represents the highest offset available in each specified partition, i.e., offset of the last message + 1. If there are no messages in a partition, the end offset will be 0.
        Parameters:
        pubSubTopicPartition - The PubSub topic partition for which to fetch the end offset.
        Returns:
        The end offset of the specified topic partition.
        Throws:
        PubSubOpTimeoutException - If the operation times out while fetching the end offset.
        PubSubClientException - If there is an error while attempting to fetch the end offset.

      • partitionsFor

        java.util.List<PubSubTopicPartitionInfo> partitionsFor​(PubSubTopic pubSubTopic)
        Retrieves the list of partitions associated with a given Pub-Sub topic.
        Parameters:
        pubSubTopic - The Pub-Sub topic for which partition information is requested.
        Returns:
        A list of PubSubTopicPartitionInfo representing the partitions of the topic, or null if the topic does not exist.