Package com.linkedin.venice.hadoop.input.kafka

Class KafkaInputSplit

java.lang.Object

com.linkedin.venice.hadoop.input.kafka.KafkaInputSplit

All Implemented Interfaces:

org.apache.hadoop.io.Writable, org.apache.hadoop.mapred.InputSplit

public class KafkaInputSplit
extends Object
implements org.apache.hadoop.mapred.InputSplit

We borrowed some idea from the open-sourced attic-crunch lib: https://github.com/apache/attic-crunch/blob/master/crunch-kafka/src/main/java/org/apache/crunch/kafka/record/KafkaInputSplit.java InputSplit that represent retrieving data from a single TopicPartition between the specified start and end offsets.

Constructor Summary

Constructors

Constructor	Description
KafkaInputSplit()	Nullary Constructor for creating the instance inside the Mapper instance.
KafkaInputSplit(String topic, int partition, long startingOffset, long endingOffset)	Constructs an input split for the provided and restricting data to be between the and

Method Summary

Modifier and Type	Method	Description
long	getEndingOffset()	Returns the ending offset for the split
long	getLength()
String[]	getLocations()
long	getStartingOffset()	Returns the starting offset for the split
org.apache.kafka.common.TopicPartition	getTopicPartition()	Returns the topic and partition for the split
void	readFields(DataInput dataInput)
String	toString()
void	write(DataOutput dataOutput)

Methods inherited from class java.lang.Object

clone, equals, finalize, getClass, hashCode, notify, notifyAll, wait, wait, wait

Constructor Details

KafkaInputSplit

public KafkaInputSplit()

Nullary Constructor for creating the instance inside the Mapper instance.

KafkaInputSplit

public KafkaInputSplit(String topic,
 int partition,
 long startingOffset,
 long endingOffset)

Constructs an input split for the provided and restricting data to be between the and

Parameters:

topic - the topic for the split

partition - the partition for the topic

startingOffset - the start of the split

endingOffset - the end of the split

Method Details

getLength

public long getLength()
               throws IOException

Specified by:

getLength in interface org.apache.hadoop.mapred.InputSplit

Throws:

IOException

getLocations

public String[] getLocations()
                      throws IOException

Specified by:

getLocations in interface org.apache.hadoop.mapred.InputSplit

Throws:

IOException

getTopicPartition

public org.apache.kafka.common.TopicPartition getTopicPartition()

Returns the topic and partition for the split

Returns:

the topic and partition for the split

getStartingOffset

public long getStartingOffset()

Returns the starting offset for the split

Returns:

the starting offset for the split

getEndingOffset

public long getEndingOffset()

Returns the ending offset for the split

Returns:

the ending offset for the split

write

public void write(DataOutput dataOutput)
           throws IOException

Specified by:

write in interface org.apache.hadoop.io.Writable

Throws:

IOException

readFields

public void readFields(DataInput dataInput)
                throws IOException

Specified by:

readFields in interface org.apache.hadoop.io.Writable

Throws:

IOException

toString

public String toString()

Overrides:

toString in class Object