Package com.linkedin.venice.hadoop.input.kafka

Class KafkaInputRecordReader

java.lang.Object

com.linkedin.venice.hadoop.input.kafka.KafkaInputRecordReader

All Implemented Interfaces:

java.lang.AutoCloseable, org.apache.hadoop.mapred.RecordReader<KafkaInputMapperKey,KafkaInputMapperValue>

public class KafkaInputRecordReader
extends java.lang.Object
implements org.apache.hadoop.mapred.RecordReader<KafkaInputMapperKey,KafkaInputMapperValue>, java.lang.AutoCloseable

We borrowed some idea from the open-sourced attic-crunch lib: https://github.com/apache/attic-crunch/blob/master/crunch-kafka/src/main/java/org/apache/crunch/kafka/record/KafkaRecordReader.java This class is used to read data off a Kafka topic partition. It will return the key bytes as unchanged, and extract the following fields and wrap them up as KafkaInputMapperValue as the value: 1. Value bytes. 2. Schema Id. 3. Offset. 4. Value type, which could be 'PUT' or 'DELETE'.

Field Summary

Fields

Modifier and Type	Field	Description
static java.lang.String	KIF_RECORD_READER_KAFKA_CONFIG_PREFIX

Constructor Summary

Constructors

Constructor	Description
KafkaInputRecordReader(org.apache.hadoop.mapred.InputSplit split, org.apache.hadoop.mapred.JobConf job, DataWriterTaskTracker taskTracker)
KafkaInputRecordReader(org.apache.hadoop.mapred.InputSplit split, org.apache.hadoop.mapred.JobConf job, DataWriterTaskTracker taskTracker, PubSubConsumerAdapter consumer)

Method Summary

Modifier and Type	Method	Description
void	close()
KafkaInputMapperKey	createKey()
KafkaInputMapperValue	createValue()
long	getPos()
float	getProgress()
boolean	next(KafkaInputMapperKey key, KafkaInputMapperValue value)	This function will skip all the Control Messages right now.

Methods inherited from class java.lang.Object

clone, equals, finalize, getClass, hashCode, notify, notifyAll, toString, wait, wait, wait

Field Detail

KIF_RECORD_READER_KAFKA_CONFIG_PREFIX

public static final java.lang.String KIF_RECORD_READER_KAFKA_CONFIG_PREFIX

See Also:

Constant Field Values

Constructor Detail

KafkaInputRecordReader

public KafkaInputRecordReader(org.apache.hadoop.mapred.InputSplit split,
                              org.apache.hadoop.mapred.JobConf job,
                              DataWriterTaskTracker taskTracker)

KafkaInputRecordReader

public KafkaInputRecordReader(org.apache.hadoop.mapred.InputSplit split,
                              org.apache.hadoop.mapred.JobConf job,
                              DataWriterTaskTracker taskTracker,
                              PubSubConsumerAdapter consumer)

Method Detail

next

public boolean next(KafkaInputMapperKey key,
                    KafkaInputMapperValue value)
             throws java.io.IOException

This function will skip all the Control Messages right now.

Specified by:

next in interface org.apache.hadoop.mapred.RecordReader<KafkaInputMapperKey,KafkaInputMapperValue>

Throws:

java.io.IOException

createKey

public KafkaInputMapperKey createKey()

Specified by:

createKey in interface org.apache.hadoop.mapred.RecordReader<KafkaInputMapperKey,KafkaInputMapperValue>

createValue

public KafkaInputMapperValue createValue()

Specified by:

createValue in interface org.apache.hadoop.mapred.RecordReader<KafkaInputMapperKey,KafkaInputMapperValue>

getPos

public long getPos()

Specified by:

getPos in interface org.apache.hadoop.mapred.RecordReader<KafkaInputMapperKey,KafkaInputMapperValue>

close

public void close()

Specified by:

close in interface java.lang.AutoCloseable

Specified by:

close in interface org.apache.hadoop.mapred.RecordReader<KafkaInputMapperKey,KafkaInputMapperValue>

getProgress

public float getProgress()

Specified by:

getProgress in interface org.apache.hadoop.mapred.RecordReader<KafkaInputMapperKey,KafkaInputMapperValue>