Package com.linkedin.venice.hadoop.input.kafka

Class VeniceKafkaInputReducer

java.lang.Object

com.linkedin.venice.hadoop.task.datawriter.AbstractDataWriterTask

com.linkedin.venice.hadoop.task.datawriter.AbstractPartitionWriter

com.linkedin.venice.hadoop.mapreduce.datawriter.reduce.VeniceReducer

com.linkedin.venice.hadoop.input.kafka.VeniceKafkaInputReducer

All Implemented Interfaces:

Closeable, AutoCloseable, org.apache.hadoop.io.Closeable, org.apache.hadoop.mapred.JobConfigurable, org.apache.hadoop.mapred.Reducer<org.apache.hadoop.io.BytesWritable,org.apache.hadoop.io.BytesWritable,org.apache.hadoop.io.BytesWritable,org.apache.hadoop.io.BytesWritable>

public class VeniceKafkaInputReducer
extends VeniceReducer

This class is designed specifically for KafkaInputFormat, and right now, it will pick up the latest entry according to the associated offset, and produce it to Kafka.

Nested Class Summary

Nested classes/interfaces inherited from class com.linkedin.venice.hadoop.task.datawriter.AbstractPartitionWriter

AbstractPartitionWriter.ChildWriterProducerCallback, AbstractPartitionWriter.DuplicateKeyPrinter, AbstractPartitionWriter.PartitionWriterProducerCallback, AbstractPartitionWriter.VeniceRecordWithMetadata, AbstractPartitionWriter.VeniceWriterMessage

Field Summary

Fields

Modifier and Type	Field	Description
protected FilterChain<ChunkAssembler.ValueBytesAndSchemaId>	veniceFilterChain

Fields inherited from class com.linkedin.venice.hadoop.mapreduce.datawriter.reduce.VeniceReducer

MAP_REDUCE_JOB_ID_PROP

Fields inherited from class com.linkedin.venice.hadoop.task.datawriter.AbstractDataWriterTask

TASK_ID_NOT_SET

Constructor Summary

Constructors

Constructor	Description
VeniceKafkaInputReducer()

Method Summary

Modifier and Type	Method	Description
void	close()
protected byte[]	compress(byte[] valueBytesFromSourceVersion)
protected void	configureTask(VeniceProperties props)	Allow implementations of this class to configure task-specific stuff.
protected AbstractPartitionWriter.VeniceWriterMessage	extract(byte[] key, Iterator<AbstractPartitionWriter.VeniceRecordWithMetadata> values, DataWriterTaskTracker dataWriterTaskTracker)
protected AbstractPartitionWriter.DuplicateKeyPrinter	initDuplicateKeyPrinter(VeniceProperties properties)	No need to print out duplicate keys since duplicate keys are expected in Kafka topics.
protected void	setChunkingEnabled(boolean isChunkingEnabled)
protected void	setDestVersionCompressor(VeniceCompressor compressor)
protected void	setSourceVersionCompressor(VeniceCompressor compressor)

Methods inherited from class com.linkedin.venice.hadoop.mapreduce.datawriter.reduce.VeniceReducer

configure, createBasicVeniceWriter, getCallback, getDataWriterTaskTracker, getExceedQuotaFlag, getJobConf, getTotalIncomingDataSizeInBytes, hasReportedFailure, reduce, setExceedQuota, setHadoopJobClientProvider, setVeniceWriter, setVeniceWriterFactory

Methods inherited from class com.linkedin.venice.hadoop.task.datawriter.AbstractPartitionWriter

getDerivedValueSchemaId, getRmdSchema, getVeniceWriterFactory, isEnableWriteCompute, logMessageProgress, processValuesForKey, recordMessageErrored

Methods inherited from class com.linkedin.venice.hadoop.task.datawriter.AbstractDataWriterTask

configure, getEngineTaskConfigProvider, getPartitionCount, getTaskId, isChunkingEnabled, isRmdChunkingEnabled

Methods inherited from class java.lang.Object

clone, equals, finalize, getClass, hashCode, notify, notifyAll, toString, wait, wait, wait

Field Details

veniceFilterChain

protected FilterChain<ChunkAssembler.ValueBytesAndSchemaId> veniceFilterChain

Constructor Details

VeniceKafkaInputReducer

public VeniceKafkaInputReducer()

Method Details

configureTask

protected void configureTask(VeniceProperties props)

Description copied from class: AbstractDataWriterTask

Allow implementations of this class to configure task-specific stuff.

Overrides:

configureTask in class VeniceReducer

Parameters:

props - the job props that the task was configured with.

setSourceVersionCompressor

protected void setSourceVersionCompressor(VeniceCompressor compressor)

setDestVersionCompressor

protected void setDestVersionCompressor(VeniceCompressor compressor)

compress

protected byte[] compress(byte[] valueBytesFromSourceVersion)

initDuplicateKeyPrinter

protected AbstractPartitionWriter.DuplicateKeyPrinter initDuplicateKeyPrinter(VeniceProperties properties)

No need to print out duplicate keys since duplicate keys are expected in Kafka topics.

Overrides:

initDuplicateKeyPrinter in class AbstractPartitionWriter

extract

protected AbstractPartitionWriter.VeniceWriterMessage extract(byte[] key,
 Iterator<AbstractPartitionWriter.VeniceRecordWithMetadata> values,
 DataWriterTaskTracker dataWriterTaskTracker)

Overrides:

extract in class AbstractPartitionWriter

setChunkingEnabled

protected void setChunkingEnabled(boolean isChunkingEnabled)

Overrides:

setChunkingEnabled in class AbstractDataWriterTask

close

public void close()
           throws IOException

Specified by:

close in interface AutoCloseable

Specified by:

close in interface Closeable

Overrides:

close in class AbstractPartitionWriter

Throws:

IOException