Package com.linkedin.venice.hadoop.input.kafka

Class KafkaInputRecordReader

java.lang.Object

com.linkedin.venice.hadoop.input.kafka.KafkaInputRecordReader

All Implemented Interfaces:

AutoCloseable, org.apache.hadoop.mapred.RecordReader<KafkaInputMapperKey,KafkaInputMapperValue>

public class KafkaInputRecordReader
extends Object
implements org.apache.hadoop.mapred.RecordReader<KafkaInputMapperKey,KafkaInputMapperValue>, AutoCloseable

Reads data from a Kafka-backed PubSub topic partition and converts each message into KafkaInputMapperKey/KafkaInputMapperValue. All generic iteration, polling, end-bound checks, and progress tracking are delegated to PubSubSplitIterator.

Constructor Summary

Constructors

Constructor	Description
KafkaInputRecordReader(org.apache.hadoop.mapred.InputSplit split, org.apache.hadoop.mapred.JobConf job, DataWriterTaskTracker taskTracker)
KafkaInputRecordReader(org.apache.hadoop.mapred.InputSplit split, org.apache.hadoop.mapred.JobConf job, DataWriterTaskTracker taskTracker, PubSubConsumerAdapter consumer)

Method Summary

Modifier and Type	Method	Description
void	close()
KafkaInputMapperKey	createKey()
KafkaInputMapperValue	createValue()
long	getPos()
float	getProgress()
boolean	next(KafkaInputMapperKey mapperKey, KafkaInputMapperValue mapperValue)

Methods inherited from class java.lang.Object

clone, equals, finalize, getClass, hashCode, notify, notifyAll, toString, wait, wait, wait

Constructor Details

KafkaInputRecordReader

public KafkaInputRecordReader(org.apache.hadoop.mapred.InputSplit split,
 org.apache.hadoop.mapred.JobConf job,
 DataWriterTaskTracker taskTracker)

KafkaInputRecordReader

public KafkaInputRecordReader(org.apache.hadoop.mapred.InputSplit split,
 org.apache.hadoop.mapred.JobConf job,
 DataWriterTaskTracker taskTracker,
 PubSubConsumerAdapter consumer)

Method Details

next

public boolean next(KafkaInputMapperKey mapperKey,
 KafkaInputMapperValue mapperValue)
             throws IOException

Specified by:

next in interface org.apache.hadoop.mapred.RecordReader<KafkaInputMapperKey,KafkaInputMapperValue>

Throws:

IOException

createKey

public KafkaInputMapperKey createKey()

Specified by:

createKey in interface org.apache.hadoop.mapred.RecordReader<KafkaInputMapperKey,KafkaInputMapperValue>

createValue

public KafkaInputMapperValue createValue()

Specified by:

createValue in interface org.apache.hadoop.mapred.RecordReader<KafkaInputMapperKey,KafkaInputMapperValue>

getPos

public long getPos()

Specified by:

getPos in interface org.apache.hadoop.mapred.RecordReader<KafkaInputMapperKey,KafkaInputMapperValue>

close

public void close()

Specified by:

close in interface AutoCloseable

Specified by:

close in interface org.apache.hadoop.mapred.RecordReader<KafkaInputMapperKey,KafkaInputMapperValue>

getProgress

public float getProgress()

Specified by:

getProgress in interface org.apache.hadoop.mapred.RecordReader<KafkaInputMapperKey,KafkaInputMapperValue>