Package com.linkedin.venice.hadoop.input.kafka

Class KafkaInputSplit

  • All Implemented Interfaces:
    org.apache.hadoop.io.Writable, org.apache.hadoop.mapred.InputSplit
    public class KafkaInputSplit
extends java.lang.Object
implements org.apache.hadoop.mapred.InputSplit
    We borrowed some idea from the open-sourced attic-crunch lib: https://github.com/apache/attic-crunch/blob/master/crunch-kafka/src/main/java/org/apache/crunch/kafka/record/KafkaInputSplit.java InputSplit that represent retrieving data from a single TopicPartition between the specified start and end offsets.

    • Constructor Summary

      Constructors 
      Constructor Description
      KafkaInputSplit()
      Nullary Constructor for creating the instance inside the Mapper instance.
      KafkaInputSplit​(java.lang.String topic, int partition, long startingOffset, long endingOffset)
      Constructs an input split for the provided {@param topic} and {@param partition} restricting data to be between the {@param startingOffset} and {@param endingOffset}

    • Method Summary

      All Methods Instance Methods Concrete Methods 
      Modifier and Type Method Description
      long getEndingOffset()
      Returns the ending offset for the split
      long getLength()  
      java.lang.String[] getLocations()  
      long getStartingOffset()
      Returns the starting offset for the split
      org.apache.kafka.common.TopicPartition getTopicPartition()
      Returns the topic and partition for the split
      void readFields​(java.io.DataInput dataInput)  
      java.lang.String toString()  
      void write​(java.io.DataOutput dataOutput)  

      • Methods inherited from class java.lang.Object

        clone, equals, finalize, getClass, hashCode, notify, notifyAll, wait, wait, wait

    • Constructor Detail

      • KafkaInputSplit

        public KafkaInputSplit()
        Nullary Constructor for creating the instance inside the Mapper instance.

      • KafkaInputSplit

        public KafkaInputSplit​(java.lang.String topic,
                       int partition,
                       long startingOffset,
                       long endingOffset)
        Constructs an input split for the provided {@param topic} and {@param partition} restricting data to be between the {@param startingOffset} and {@param endingOffset}
        Parameters:
        topic - the topic for the split
        partition - the partition for the topic
        startingOffset - the start of the split
        endingOffset - the end of the split

    • Method Detail

      • getLength

        public long getLength()
               throws java.io.IOException
        Specified by:
        getLength in interface org.apache.hadoop.mapred.InputSplit
        Throws:
        java.io.IOException

      • getLocations

        public java.lang.String[] getLocations()
                                throws java.io.IOException
        Specified by:
        getLocations in interface org.apache.hadoop.mapred.InputSplit
        Throws:
        java.io.IOException

      • getTopicPartition

        public org.apache.kafka.common.TopicPartition getTopicPartition()
        Returns the topic and partition for the split
        Returns:
        the topic and partition for the split

      • getStartingOffset

        public long getStartingOffset()
        Returns the starting offset for the split
        Returns:
        the starting offset for the split

      • getEndingOffset

        public long getEndingOffset()
        Returns the ending offset for the split
        Returns:
        the ending offset for the split

      • write

        public void write​(java.io.DataOutput dataOutput)
           throws java.io.IOException
        Specified by:
        write in interface org.apache.hadoop.io.Writable
        Throws:
        java.io.IOException

      • readFields

        public void readFields​(java.io.DataInput dataInput)
                throws java.io.IOException
        Specified by:
        readFields in interface org.apache.hadoop.io.Writable
        Throws:
        java.io.IOException

      • toString

        public java.lang.String toString()
        Overrides:
        toString in class java.lang.Object