Package com.linkedin.venice

Class KafkaTopicDumper

java.lang.Object

com.linkedin.venice.KafkaTopicDumper

All Implemented Interfaces:

AutoCloseable

public class KafkaTopicDumper
extends Object
implements AutoCloseable

This class contains logic to dump Venice Kafka topics. It has several modes: (1) Log metadata: Print out Kafka message metadata to console. (2) Log data record: Print out data record's key/value and optionally RMD to console. For now it only supports version topic and realtime topic. (3) Save data record to file: If both (1)&(2) is not enabled, it will save all the data record value payload to local disk.

Constructor Summary

Constructors

Constructor	Description
KafkaTopicDumper(ControllerClient controllerClient, PubSubConsumerAdapter consumer, PubSubTopicPartition topicPartition, String parentDir, int maxConsumeAttempts, boolean logMetadata, boolean logDataRecord, boolean logRmdRecord, boolean logTsRecord)

Method Summary

Modifier and Type	Method	Description
void	close()
int	fetchAndProcess(long startOffset, long endOffset, int messageCount)	Fetches and processes messages from a given PubSub topic partition between specified offsets.

Methods inherited from class java.lang.Object

clone, equals, finalize, getClass, hashCode, notify, notifyAll, toString, wait, wait, wait

Constructor Details

KafkaTopicDumper

public KafkaTopicDumper(ControllerClient controllerClient,
 PubSubConsumerAdapter consumer,
 PubSubTopicPartition topicPartition,
 String parentDir,
 int maxConsumeAttempts,
 boolean logMetadata,
 boolean logDataRecord,
 boolean logRmdRecord,
 boolean logTsRecord)

Method Details

fetchAndProcess

public int fetchAndProcess(long startOffset,
 long endOffset,
 int messageCount)

Fetches and processes messages from a given PubSub topic partition between specified offsets.

The method polls messages from the specified `startOffset` to `endOffset` and processes up to `messageCount` messages. It uses the consumer to fetch records in batches, processes them via processRecord, and stops under the following conditions:

• The number of processed messages reaches messageCount.
• The offset of the last processed message is greater than or equal to endOffset.
• No new records are fetched within the allowed number of attempts.

Parameters:

startOffset - the starting offset (inclusive) to begin processing messages

endOffset - the ending offset (exclusive) to stop processing messages

messageCount - the maximum number of messages to process

Returns:

the total number of messages processed

Throws:

IllegalArgumentException - if messageCount is less than or equal to zero, or if startOffset is greater than endOffset

close

public void close()
           throws Exception

Specified by:

close in interface AutoCloseable

Throws:

Exception