Class KafkaInputSplit
java.lang.Object
com.linkedin.venice.hadoop.input.kafka.KafkaInputSplit
- All Implemented Interfaces:
org.apache.hadoop.io.Writable
,org.apache.hadoop.mapred.InputSplit
We borrowed some idea from the open-sourced attic-crunch lib:
https://github.com/apache/attic-crunch/blob/master/crunch-kafka/src/main/java/org/apache/crunch/kafka/record/KafkaInputSplit.java
InputSplit that represent retrieving data from a single
PubSubTopicPartition
between the specified start
and end offsets.-
Constructor Summary
ConstructorsConstructorDescriptionNullary Constructor for creating the instance inside the Mapper instance.KafkaInputSplit
(PubSubTopicRepository topicRepository, PubSubTopicPartition topicPartition, long startingOffset, long endingOffset) Constructs an input split for the provided and restricting data to be between the and -
Method Summary
Modifier and TypeMethodDescriptionlong
Returns the ending offset for the splitlong
String[]
long
Returns the starting offset for the splitReturns the topic and partition for the splitvoid
readFields
(DataInput dataInput) toString()
void
write
(DataOutput dataOutput)
-
Constructor Details
-
KafkaInputSplit
public KafkaInputSplit()Nullary Constructor for creating the instance inside the Mapper instance. -
KafkaInputSplit
public KafkaInputSplit(PubSubTopicRepository topicRepository, PubSubTopicPartition topicPartition, long startingOffset, long endingOffset) Constructs an input split for the provided and restricting data to be between the and- Parameters:
topicPartition
- the topic-partition for the splitstartingOffset
- the start of the splitendingOffset
- the end of the split
-
-
Method Details
-
getLength
- Specified by:
getLength
in interfaceorg.apache.hadoop.mapred.InputSplit
- Throws:
IOException
-
getLocations
- Specified by:
getLocations
in interfaceorg.apache.hadoop.mapred.InputSplit
- Throws:
IOException
-
getTopicPartition
Returns the topic and partition for the split- Returns:
- the topic and partition for the split
-
getStartingOffset
public long getStartingOffset()Returns the starting offset for the split- Returns:
- the starting offset for the split
-
getEndingOffset
public long getEndingOffset()Returns the ending offset for the split- Returns:
- the ending offset for the split
-
write
- Specified by:
write
in interfaceorg.apache.hadoop.io.Writable
- Throws:
IOException
-
readFields
- Specified by:
readFields
in interfaceorg.apache.hadoop.io.Writable
- Throws:
IOException
-
toString
-