Package com.linkedin.venice.hadoop.task.datawriter

Class AbstractInputRecordProcessor<INPUT_KEY,INPUT_VALUE>

java.lang.Object

com.linkedin.venice.hadoop.task.datawriter.AbstractDataWriterTask

com.linkedin.venice.hadoop.task.datawriter.AbstractInputRecordProcessor<INPUT_KEY,INPUT_VALUE>

Type Parameters:

INPUT_KEY - type of the input key read from InputFormat

INPUT_VALUE - type of the input value read from InputFormat

All Implemented Interfaces:

java.io.Closeable, java.lang.AutoCloseable

Direct Known Subclasses:

AbstractVeniceMapper, SparkInputRecordProcessor

public abstract class AbstractInputRecordProcessor<INPUT_KEY,INPUT_VALUE>
extends AbstractDataWriterTask
implements java.io.Closeable

An abstraction of the task that processes each record from the input, and returns serialized, and potentially compressed, Avro key/value pairs.

Field Summary

Fields

Modifier and Type	Field	Description
protected AbstractVeniceRecordReader<INPUT_KEY,INPUT_VALUE>	veniceRecordReader

Fields inherited from class com.linkedin.venice.hadoop.task.datawriter.AbstractDataWriterTask

TASK_ID_NOT_SET

Constructor Summary

Constructors

Constructor	Description
AbstractInputRecordProcessor()

Method Summary

Modifier and Type	Method	Description
void	close()
protected void	configureTask(VeniceProperties props)	Allow implementations of this class to configure task-specific stuff.
protected abstract AbstractVeniceRecordReader<INPUT_KEY,INPUT_VALUE>	getRecordReader(VeniceProperties props)	A method for child classes to setup veniceRecordReader.
protected boolean	process(INPUT_KEY inputKey, INPUT_VALUE inputValue, java.util.concurrent.atomic.AtomicReference<byte[]> keyRef, java.util.concurrent.atomic.AtomicReference<byte[]> valueRef, DataWriterTaskTracker dataWriterTaskTracker)	This function will return true if the input key/value pair is valid.
protected void	processRecord(INPUT_KEY inputKey, INPUT_VALUE inputValue, java.util.function.BiConsumer<byte[],byte[]> recordEmitter, DataWriterTaskTracker dataWriterTaskTracker)
protected java.nio.ByteBuffer	readDictionaryFromKafka(java.lang.String topicName, VeniceProperties props)	This function is added to allow it to be mocked for tests.

Methods inherited from class com.linkedin.venice.hadoop.task.datawriter.AbstractDataWriterTask

configure, getEngineTaskConfigProvider, getPartitionCount, getTaskId, isChunkingEnabled, isRmdChunkingEnabled, setChunkingEnabled

Methods inherited from class java.lang.Object

clone, equals, finalize, getClass, hashCode, notify, notifyAll, toString, wait, wait, wait

Field Detail

veniceRecordReader

protected AbstractVeniceRecordReader<INPUT_KEY,INPUT_VALUE> veniceRecordReader

Constructor Detail

AbstractInputRecordProcessor

public AbstractInputRecordProcessor()

Method Detail

processRecord

protected final void processRecord(INPUT_KEY inputKey,
                                   INPUT_VALUE inputValue,
                                   java.util.function.BiConsumer<byte[],byte[]> recordEmitter,
                                   DataWriterTaskTracker dataWriterTaskTracker)

process

protected boolean process(INPUT_KEY inputKey,
                          INPUT_VALUE inputValue,
                          java.util.concurrent.atomic.AtomicReference<byte[]> keyRef,
                          java.util.concurrent.atomic.AtomicReference<byte[]> valueRef,
                          DataWriterTaskTracker dataWriterTaskTracker)

This function will return true if the input key/value pair is valid.

getRecordReader

protected abstract AbstractVeniceRecordReader<INPUT_KEY,INPUT_VALUE> getRecordReader(VeniceProperties props)

A method for child classes to setup veniceRecordReader.

configureTask

protected void configureTask(VeniceProperties props)

Description copied from class: AbstractDataWriterTask

Allow implementations of this class to configure task-specific stuff.

Specified by:

configureTask in class AbstractDataWriterTask

Parameters:

props - the job props that the task was configured with.

readDictionaryFromKafka

protected java.nio.ByteBuffer readDictionaryFromKafka(java.lang.String topicName,
                                                      VeniceProperties props)

This function is added to allow it to be mocked for tests. Since mocking this function of an actual object in AbstractTestVeniceMapper#getMapper(int, int, Consumer) ended up hitting the original function always, added an override for this in TestVeniceAvroMapperClass.

close

public void close()

Specified by:

close in interface java.lang.AutoCloseable

Specified by:

close in interface java.io.Closeable