Package com.linkedin.venice

Class KafkaTopicDumper

  • All Implemented Interfaces:
    java.lang.AutoCloseable
    public class KafkaTopicDumper
extends java.lang.Object
implements java.lang.AutoCloseable
    This class contains logic to dump Venice Kafka topics. It has several modes: (1) Log metadata: Print out Kafka message metadata to console. (2) Log data record: Print out data record's key/value and optionally RMD to console. For now it only supports version topic and realtime topic. (3) Save data record to file: If both (1)&(2) is not enabled, it will save all the data record value payload to local disk.

    • Method Summary

      All Methods Instance Methods Concrete Methods 
      Modifier and Type Method Description
      void close()  
      int fetchAndProcess​(long startOffset, long endOffset, int messageCount)
      Fetches and processes messages from a given PubSub topic partition between specified offsets.

      • Methods inherited from class java.lang.Object

        clone, equals, finalize, getClass, hashCode, notify, notifyAll, toString, wait, wait, wait

    • Constructor Detail

      • KafkaTopicDumper

        public KafkaTopicDumper​(ControllerClient controllerClient,
                        PubSubConsumerAdapter consumer,
                        PubSubTopicPartition topicPartition,
                        java.lang.String parentDir,
                        int maxConsumeAttempts,
                        boolean logMetadata,
                        boolean logDataRecord,
                        boolean logRmdRecord,
                        boolean logTsRecord)

    • Method Detail

      • fetchAndProcess

        public int fetchAndProcess​(long startOffset,
                           long endOffset,
                           int messageCount)
        Fetches and processes messages from a given PubSub topic partition between specified offsets.

        The method polls messages from the specified `startOffset` to `endOffset` and processes up to `messageCount` messages. It uses the consumer to fetch records in batches, processes them via processRecord, and stops under the following conditions:

        • The number of processed messages reaches messageCount.
        • The offset of the last processed message is greater than or equal to endOffset.
        • No new records are fetched within the allowed number of attempts.
        Parameters:
        startOffset - the starting offset (inclusive) to begin processing messages
        endOffset - the ending offset (exclusive) to stop processing messages
        messageCount - the maximum number of messages to process
        Returns:
        the total number of messages processed
        Throws:
        java.lang.IllegalArgumentException - if messageCount is less than or equal to zero, or if startOffset is greater than endOffset

      • close

        public void close()
           throws java.lang.Exception
        Specified by:
        close in interface java.lang.AutoCloseable
        Throws:
        java.lang.Exception