Package com.linkedin.venice

Class KafkaTopicDumper

java.lang.Object

com.linkedin.venice.KafkaTopicDumper

All Implemented Interfaces:

AutoCloseable

public class KafkaTopicDumper
extends Object
implements AutoCloseable

This class contains logic to dump Venice Kafka topics. It has several modes: (1) Log metadata: Print out Kafka message metadata to console. (2) Log data record: Print out data record's key/value and optionally RMD to console. For now it only supports version topic and realtime topic. (3) Save data record to file: If both (1)&(2) is not enabled, it will save all the data record value payload to local disk.

Constructor Summary

Constructors

Constructor	Description
KafkaTopicDumper(ControllerClient controllerClient, PubSubConsumerAdapter consumer, PubSubTopicPartition topicPartition, String parentDir, int maxConsumeAttempts, boolean logMetadata, boolean logDataRecord, boolean logRmdRecord, boolean logTsRecord)

Method Summary

Modifier and Type	Method	Description
void	close()
int	fetchAndProcess(PubSubPosition startPosition, PubSubPosition endPosition, long messageCount)	Fetches and processes messages from a given PubSub topic partition between specified positions.

Methods inherited from class java.lang.Object

clone, equals, finalize, getClass, hashCode, notify, notifyAll, toString, wait, wait, wait

Constructor Details

KafkaTopicDumper

public KafkaTopicDumper(ControllerClient controllerClient,
 PubSubConsumerAdapter consumer,
 PubSubTopicPartition topicPartition,
 String parentDir,
 int maxConsumeAttempts,
 boolean logMetadata,
 boolean logDataRecord,
 boolean logRmdRecord,
 boolean logTsRecord)

Method Details

fetchAndProcess

public int fetchAndProcess(PubSubPosition startPosition,
 PubSubPosition endPosition,
 long messageCount)

Fetches and processes messages from a given PubSub topic partition between specified positions.

The method polls messages from the specified `startPosition` to `endPosition` and processes up to `messageCount` messages. It uses the consumer to fetch records in batches, processes them via processRecord, and stops under the following conditions:

• The number of processed messages reaches messageCount.
• The position of the last processed message is greater than or equal to endPosition.
• No new records are fetched within the allowed number of attempts.

Parameters:

startPosition - the starting position (inclusive) to begin processing messages

endPosition - the ending position (exclusive) to stop processing messages

messageCount - the maximum number of messages to process

Returns:

the total number of messages processed

Throws:

IllegalArgumentException - if messageCount is less than or equal to zero, or if startPosition is greater than endPosition

close

public void close()
           throws Exception

Specified by:

close in interface AutoCloseable

Throws:

Exception