Package com.linkedin.venice.pubsub.adapter.kafka.consumer

Class ApacheKafkaConsumerAdapter

  • All Implemented Interfaces:
    PubSubConsumerAdapter, java.io.Closeable, java.lang.AutoCloseable
    public class ApacheKafkaConsumerAdapter
extends java.lang.Object
implements PubSubConsumerAdapter
    This class is not thread safe because of the internal KafkaConsumer is not thread safe. It is the responsibility of the caller to ensure that the methods are called in a thread safe manner.

    • Method Detail

      • subscribe

        public void subscribe​(PubSubTopicPartition pubSubTopicPartition,
                      long lastReadOffset)
        Subscribe to a topic-partition if not already subscribed. If the topic-partition is already subscribed, this method is a no-op. This method requires the topic-partition to exist.
        Specified by:
        subscribe in interface PubSubConsumerAdapter
        Parameters:
        pubSubTopicPartition - the topic-partition to subscribe to
        lastReadOffset - the last read offset for the topic-partition
        Throws:
        java.lang.IllegalArgumentException - if the topic-partition is null or the partition number is negative
        PubSubTopicDoesNotExistException - if the topic does not exist

      • unSubscribe

        public void unSubscribe​(PubSubTopicPartition pubSubTopicPartition)
        Description copied from interface: PubSubConsumerAdapter
        Unsubscribes the consumer from a specified topic-partition. If the consumer was previously subscribed to the given partition, it will be unsubscribed, and the associated partition assignments and tracked offsets will be updated accordingly.
        Specified by:
        unSubscribe in interface PubSubConsumerAdapter
        Parameters:
        pubSubTopicPartition - The PubSub topic-partition to unsubscribe from.

      • batchUnsubscribe

        public void batchUnsubscribe​(java.util.Set<PubSubTopicPartition> pubSubTopicPartitionsToUnsubscribe)
        Description copied from interface: PubSubConsumerAdapter
        Unsubscribes the consumer from a batch of topic-partitions.
        Specified by:
        batchUnsubscribe in interface PubSubConsumerAdapter
        Parameters:
        pubSubTopicPartitionsToUnsubscribe - A set of topic-partitions to unsubscribe from.

      • poll

        public java.util.Map<PubSubTopicPartition,​java.util.List<PubSubMessage<KafkaKey,​KafkaMessageEnvelope,​java.lang.Long>>> poll​(long timeoutMs)
        Description copied from interface: PubSubConsumerAdapter
        Polls the Kafka consumer for messages from the subscribed topic partitions within the specified time duration.
        Specified by:
        poll in interface PubSubConsumerAdapter
        Parameters:
        timeoutMs - The maximum time, in milliseconds, to wait for messages to be polled.
        Returns:
        A mapping of PubSub topic partitions to lists of PubSub messages retrieved from Kafka.

      • hasAnySubscription

        public boolean hasAnySubscription()
        Description copied from interface: PubSubConsumerAdapter
        Checks if the consumer has any active topic-partition subscriptions.
        Specified by:
        hasAnySubscription in interface PubSubConsumerAdapter
        Returns:
        true if the consumer is subscribed to one or more topic partitions, false otherwise.

      • hasSubscription

        public boolean hasSubscription​(PubSubTopicPartition pubSubTopicPartition)
        Description copied from interface: PubSubConsumerAdapter
        Checks if the consumer is currently subscribed to the specified PubSub topic-partition.
        Specified by:
        hasSubscription in interface PubSubConsumerAdapter
        Parameters:
        pubSubTopicPartition - The PubSub topic-partition to check for subscription.
        Returns:
        true if the consumer is subscribed to the given topic-partition, false otherwise.

      • pause

        public void pause​(PubSubTopicPartition pubSubTopicPartition)
        If the partitions were not previously subscribed, this method is a no-op.
        Specified by:
        pause in interface PubSubConsumerAdapter
        Parameters:
        pubSubTopicPartition - The PubSub topic-partition for which to pause message consumption.

      • resume

        public void resume​(PubSubTopicPartition pubSubTopicPartition)
        If the partitions were not previously paused or if they were not subscribed at all, this method is a no-op.
        Specified by:
        resume in interface PubSubConsumerAdapter
        Parameters:
        pubSubTopicPartition - The PubSub topic partition for which to resume message consumption.

      • getAssignment

        public java.util.Set<PubSubTopicPartition> getAssignment()
        Description copied from interface: PubSubConsumerAdapter
        Retrieves the set of PubSub topic-partitions currently assigned to the consumer.
        Specified by:
        getAssignment in interface PubSubConsumerAdapter
        Returns:
        A set of PubSub topic-partitions representing the current assignment of the consumer.

      • close

        public void close()
        Description copied from interface: PubSubConsumerAdapter
        Closes the PubSub consumer and releases any associated resources.
        Specified by:
        close in interface java.lang.AutoCloseable
        Specified by:
        close in interface java.io.Closeable
        Specified by:
        close in interface PubSubConsumerAdapter

      • getOffsetLag

        public long getOffsetLag​(PubSubTopicPartition pubSubTopicPartition)
        Description copied from interface: PubSubConsumerAdapter
        Retrieves the consuming offset lag for a PubSub topic partition. The offset lag represents the difference between the last consumed message offset and the latest available message offset for the partition.
        Specified by:
        getOffsetLag in interface PubSubConsumerAdapter
        Parameters:
        pubSubTopicPartition - The PubSub topic partition for which to fetch the offset lag.
        Returns:
        The offset lag, which is zero or a positive value if a valid lag was collected by the consumer, or -1 if the lag cannot be determined or is not applicable.

      • getLatestOffset

        public long getLatestOffset​(PubSubTopicPartition pubSubTopicPartition)
        Description copied from interface: PubSubConsumerAdapter
        Retrieves the latest available offset for a PubSub topic partition.
        Specified by:
        getLatestOffset in interface PubSubConsumerAdapter
        Parameters:
        pubSubTopicPartition - The PubSub topic partition for which to fetch the latest offset.
        Returns:
        The latest offset, which is zero or a positive value if an offset was collected by the consumer, or -1 if the offset cannot be determined or is not applicable.

      • offsetForTime

        public java.lang.Long offsetForTime​(PubSubTopicPartition pubSubTopicPartition,
                                    long timestamp,
                                    java.time.Duration timeout)
        Description copied from interface: PubSubConsumerAdapter
        Retrieves the offset of the first message with a timestamp greater than or equal to the target timestamp for the specified PubSub topic-partition. If no such message is found, null will be returned for the partition.
        Specified by:
        offsetForTime in interface PubSubConsumerAdapter
        Parameters:
        pubSubTopicPartition - The PubSub topic-partition for which to fetch the offset.
        timestamp - The target timestamp to search for in milliseconds since the Unix epoch.
        timeout - The maximum duration to wait for the operation to complete.
        Returns:
        get the offset of the first message with timestamp greater than or equal to the target timestamp. null will be returned for the partition if there is no such message.

      • offsetForTime

        public java.lang.Long offsetForTime​(PubSubTopicPartition pubSubTopicPartition,
                                    long timestamp)
        Description copied from interface: PubSubConsumerAdapter
        Retrieves the offset of the first message with a timestamp greater than or equal to the target timestamp for the specified PubSub topic-partition. If no such message is found, null will be returned for the partition.
        Specified by:
        offsetForTime in interface PubSubConsumerAdapter
        Parameters:
        pubSubTopicPartition - The PubSub topic-partition for which to fetch the offset.
        timestamp - The target timestamp to search for in milliseconds since the Unix epoch.
        Returns:
        The offset of the first message with a timestamp greater than or equal to the target timestamp, or null if no such message is found for the partition.

      • beginningOffset

        public java.lang.Long beginningOffset​(PubSubTopicPartition pubSubTopicPartition,
                                      java.time.Duration timeout)
        Description copied from interface: PubSubConsumerAdapter
        Retrieves the beginning offset for the specified PubSub topic-partition.
        Specified by:
        beginningOffset in interface PubSubConsumerAdapter
        Parameters:
        pubSubTopicPartition - The PubSub topic-partition for which to fetch the beginning offset.
        timeout - The maximum duration to wait for the operation to complete.
        Returns:
        The beginning offset of the specified topic-partition. If topic-partition exists but has no messages, the offset will be 0.

      • endOffsets

        public java.util.Map<PubSubTopicPartition,​java.lang.Long> endOffsets​(java.util.Collection<PubSubTopicPartition> partitions,
                                                                           java.time.Duration timeout)
        Description copied from interface: PubSubConsumerAdapter
        Retrieves the end offsets for a collection of PubSub topic-partitions. The end offset represents the highest offset available in each specified partition, i.e., offset of the last message + 1. If there are no messages in a partition, the end offset will be 0.
        Specified by:
        endOffsets in interface PubSubConsumerAdapter
        Parameters:
        partitions - A collection of PubSub topic-partitions for which to fetch the end offsets.
        timeout - The maximum duration to wait for the operation to complete.
        Returns:
        A mapping of PubSub topic partitions to their respective end offsets, or an empty map if the offsets cannot be determined.

      • endOffset

        public java.lang.Long endOffset​(PubSubTopicPartition pubSubTopicPartition)
        Description copied from interface: PubSubConsumerAdapter
        Retrieves the end offset for the specified PubSub topic-partition. The end offset represents the highest offset available in each specified partition, i.e., offset of the last message + 1. If there are no messages in a partition, the end offset will be 0.
        Specified by:
        endOffset in interface PubSubConsumerAdapter
        Parameters:
        pubSubTopicPartition - The PubSub topic partition for which to fetch the end offset.
        Returns:
        The end offset of the specified topic partition.