Package com.linkedin.venice.pubsub.api

Interface PubSubConsumerAdapter

All Superinterfaces:
AutoCloseable, Closeable
All Known Implementing Classes:
ApacheKafkaConsumerAdapter, MockInMemoryConsumer
public interface PubSubConsumerAdapter extends AutoCloseable, Closeable
An adapter for consuming messages from a Pub-Sub topic. Implementations of this interface are not expected to be thread safe. However, they are expected to provide the following guarantees: 1) Honor the timeout parameter for all methods that have one. 2) Non-blocking behavior for methods that do not have an explicit timeout parameter. In other words, they should timeout after the default timeout period: PubSubConstants.PUBSUB_CONSUMER_API_DEFAULT_TIMEOUT_MS.

  • Method Details

    • subscribe

      void subscribe(PubSubTopicPartition pubSubTopicPartition, long lastReadOffset)
      Subscribes to a topic-partition if it is not already subscribed. If the topic-partition is already subscribed, this method is a no-op. The method assumes that the topic-partition exists.
      Parameters:
      pubSubTopicPartition - The topic-partition to subscribe to.
      lastReadOffset - The last read offset for the topic-partition. A poll call following a subscribe call will return messages from the offset (lastReadOffset + 1).
      Throws:
      IllegalArgumentException - If the topic-partition is null or if the partition number is negative.
      PubSubTopicDoesNotExistException - If the topic does not exist.

    • unSubscribe

      void unSubscribe(PubSubTopicPartition pubSubTopicPartition)
      Unsubscribes the consumer from a specified topic-partition. If the consumer was previously subscribed to the given partition, it will be unsubscribed, and the associated partition assignments and tracked offsets will be updated accordingly.
      Parameters:
      pubSubTopicPartition - The PubSub topic-partition to unsubscribe from.

    • batchUnsubscribe

      void batchUnsubscribe(Set<PubSubTopicPartition> pubSubTopicPartitionSet)
      Unsubscribes the consumer from a batch of topic-partitions.
      Parameters:
      pubSubTopicPartitionSet - A set of topic-partitions to unsubscribe from.

    • resetOffset

      void resetOffset(PubSubTopicPartition pubSubTopicPartition) throws PubSubUnsubscribedTopicPartitionException
      Resets the offset for a specific topic-partition, seeking it to the beginning of the topic. If the topic partition is not currently subscribed, a PubSubUnsubscribedTopicPartitionException is thrown.
      Parameters:
      pubSubTopicPartition - The PubSub topic-partition for which to reset the offset.
      Throws:
      PubSubUnsubscribedTopicPartitionException - If the specified topic-partition is not currently subscribed.

    • close

      void close()
      Closes the PubSub consumer and releases any associated resources.
      Specified by:
      close in interface AutoCloseable
      Specified by:
      close in interface Closeable

    • poll

      Map<PubSubTopicPartition,List<PubSubMessage<KafkaKey,KafkaMessageEnvelope,Long>>> poll(long timeoutMs)
      Polls the Kafka consumer for messages from the subscribed topic partitions within the specified time duration.
      Parameters:
      timeoutMs - The maximum time, in milliseconds, to wait for messages to be polled.
      Returns:
      A mapping of PubSub topic partitions to lists of PubSub messages retrieved from Kafka.
      Throws:
      PubSubClientException - If there is an error during message retrieval from Kafka.
      PubSubClientRetriableException - If a retriable exception occurs during polling attempts, with retries as configured.

    • hasAnySubscription

      boolean hasAnySubscription()
      Checks if the consumer has any active topic-partition subscriptions.
      Returns:
      true if the consumer is subscribed to one or more topic partitions, false otherwise.

    • hasSubscription

      boolean hasSubscription(PubSubTopicPartition pubSubTopicPartition)
      Checks if the consumer is currently subscribed to the specified PubSub topic-partition.
      Parameters:
      pubSubTopicPartition - The PubSub topic-partition to check for subscription.
      Returns:
      true if the consumer is subscribed to the given topic-partition, false otherwise.

    • pause

      void pause(PubSubTopicPartition pubSubTopicPartition)
      Pauses message consumption for the specified PubSub topic-partition. If the partition was not previously subscribed, this method is a no-op.
      Parameters:
      pubSubTopicPartition - The PubSub topic-partition for which to pause message consumption.

    • resume

      void resume(PubSubTopicPartition pubSubTopicPartition)
      Resumes message consumption for the specified PubSub topic-partition. If the partition was not previously paused or if they were not subscribed at all, this method is a no-op.
      Parameters:
      pubSubTopicPartition - The PubSub topic partition for which to resume message consumption.

    • getAssignment

      Set<PubSubTopicPartition> getAssignment()
      Retrieves the set of PubSub topic-partitions currently assigned to the consumer.
      Returns:
      A set of PubSub topic-partitions representing the current assignment of the consumer.

    • getOffsetLag

      default long getOffsetLag(PubSubTopicPartition pubSubTopicPartition)
      Retrieves the consuming offset lag for a PubSub topic partition. The offset lag represents the difference between the last consumed message offset and the latest available message offset for the partition.
      Parameters:
      pubSubTopicPartition - The PubSub topic partition for which to fetch the offset lag.
      Returns:
      The offset lag, which is zero or a positive value if a valid lag was collected by the consumer, or -1 if the lag cannot be determined or is not applicable.

    • getLatestOffset

      default long getLatestOffset(PubSubTopicPartition pubSubTopicPartition)
      Retrieves the latest available offset for a PubSub topic partition.
      Parameters:
      pubSubTopicPartition - The PubSub topic partition for which to fetch the latest offset.
      Returns:
      The latest offset, which is zero or a positive value if an offset was collected by the consumer, or -1 if the offset cannot be determined or is not applicable.

    • offsetForTime

      Long offsetForTime(PubSubTopicPartition pubSubTopicPartition, long timestamp, Duration timeout)
      Retrieves the offset of the first message with a timestamp greater than or equal to the target timestamp for the specified PubSub topic-partition. If no such message is found, null will be returned for the partition.
      Parameters:
      pubSubTopicPartition - The PubSub topic-partition for which to fetch the offset.
      timestamp - The target timestamp to search for in milliseconds since the Unix epoch.
      timeout - The maximum duration to wait for the operation to complete.
      Returns:
      The offset of the first message with a timestamp greater than or equal to the target timestamp, or null if no such message is found for the partition.
      Throws:
      PubSubOpTimeoutException - If the operation times out while fetching the offset.
      PubSubClientException - If there is an error while attempting to fetch the offset.

    • offsetForTime

      Long offsetForTime(PubSubTopicPartition pubSubTopicPartition, long timestamp)
      Retrieves the offset of the first message with a timestamp greater than or equal to the target timestamp for the specified PubSub topic-partition. If no such message is found, null will be returned for the partition.
      Parameters:
      pubSubTopicPartition - The PubSub topic-partition for which to fetch the offset.
      timestamp - The target timestamp to search for in milliseconds since the Unix epoch.
      Returns:
      The offset of the first message with a timestamp greater than or equal to the target timestamp, or null if no such message is found for the partition.
      Throws:
      PubSubOpTimeoutException - If the operation times out while fetching the offset.
      PubSubClientException - If there is an error while attempting to fetch the offset.

    • beginningOffset

      Long beginningOffset(PubSubTopicPartition pubSubTopicPartition, Duration timeout)
      Retrieves the beginning offset for the specified PubSub topic-partition.
      Parameters:
      pubSubTopicPartition - The PubSub topic-partition for which to fetch the beginning offset.
      timeout - The maximum duration to wait for the operation to complete.
      Returns:
      The beginning offset of the specified topic-partition. If topic-partition exists but has no messages, the offset will be 0.
      Throws:
      PubSubOpTimeoutException - If the operation times out while fetching the beginning offset.
      PubSubClientException - If there is an error while attempting to fetch the beginning offset.

    • endOffsets

      Map<PubSubTopicPartition,Long> endOffsets(Collection<PubSubTopicPartition> partitions, Duration timeout)
      Retrieves the end offsets for a collection of PubSub topic-partitions. The end offset represents the highest offset available in each specified partition, i.e., offset of the last message + 1. If there are no messages in a partition, the end offset will be 0.
      Parameters:
      partitions - A collection of PubSub topic-partitions for which to fetch the end offsets.
      timeout - The maximum duration to wait for the operation to complete.
      Returns:
      A mapping of PubSub topic partitions to their respective end offsets, or an empty map if the offsets cannot be determined.
      Throws:
      PubSubOpTimeoutException - If the operation times out while fetching the end offsets.
      PubSubClientException - If there is an error while attempting to fetch the end offsets.

    • endOffset

      Long endOffset(PubSubTopicPartition pubSubTopicPartition)
      Retrieves the end offset for the specified PubSub topic-partition. The end offset represents the highest offset available in each specified partition, i.e., offset of the last message + 1. If there are no messages in a partition, the end offset will be 0.
      Parameters:
      pubSubTopicPartition - The PubSub topic partition for which to fetch the end offset.
      Returns:
      The end offset of the specified topic partition.
      Throws:
      PubSubOpTimeoutException - If the operation times out while fetching the end offset.
      PubSubClientException - If there is an error while attempting to fetch the end offset.

    • partitionsFor

      List<PubSubTopicPartitionInfo> partitionsFor(PubSubTopic pubSubTopic)
      Retrieves the list of partitions associated with a given Pub-Sub topic.
      Parameters:
      pubSubTopic - The Pub-Sub topic for which partition information is requested.
      Returns:
      A list of PubSubTopicPartitionInfo representing the partitions of the topic, or null if the topic does not exist.