Class VeniceKafkaInputMapper
java.lang.Object
com.linkedin.venice.hadoop.task.datawriter.AbstractDataWriterTask
com.linkedin.venice.hadoop.task.datawriter.AbstractInputRecordProcessor<INPUT_KEY,INPUT_VALUE>
com.linkedin.venice.hadoop.mapreduce.datawriter.map.AbstractVeniceMapper<KafkaInputMapperKey,KafkaInputMapperValue>
com.linkedin.venice.hadoop.input.kafka.VeniceKafkaInputMapper
- All Implemented Interfaces:
Closeable
,AutoCloseable
,org.apache.hadoop.io.Closeable
,org.apache.hadoop.mapred.JobConfigurable
,org.apache.hadoop.mapred.Mapper<KafkaInputMapperKey,
KafkaInputMapperValue, org.apache.hadoop.io.BytesWritable, org.apache.hadoop.io.BytesWritable>
public class VeniceKafkaInputMapper
extends AbstractVeniceMapper<KafkaInputMapperKey,KafkaInputMapperValue>
This class is designed specifically for
KafkaInputFormat
, and right now, it is doing simple pass-through.-
Field Summary
Fields inherited from class com.linkedin.venice.hadoop.task.datawriter.AbstractInputRecordProcessor
veniceRecordReader
Fields inherited from class com.linkedin.venice.hadoop.task.datawriter.AbstractDataWriterTask
TASK_ID_NOT_SET
-
Constructor Summary
Constructors -
Method Summary
Modifier and TypeMethodDescriptionvoid
close()
protected void
configureTask
(VeniceProperties props) Allow implementations of this class to configure task-specific stuff.protected FilterChain<KafkaInputMapperValue>
getFilterChain
(VeniceProperties props) getRecordReader
(VeniceProperties props) A method for child classes to setupAbstractInputRecordProcessor.veniceRecordReader
.protected boolean
process
(KafkaInputMapperKey inputKey, KafkaInputMapperValue inputValue, Long timestamp, AtomicReference<byte[]> keyRef, AtomicReference<byte[]> valueRef, AtomicReference<Long> timestampRef, DataWriterTaskTracker dataWriterTaskTracker) This function compresses the record and checks whether its uncompressed size exceeds the maximum allowed size.Methods inherited from class com.linkedin.venice.hadoop.mapreduce.datawriter.map.AbstractVeniceMapper
configure, map
Methods inherited from class com.linkedin.venice.hadoop.task.datawriter.AbstractInputRecordProcessor
processRecord, readDictionaryFromKafka
Methods inherited from class com.linkedin.venice.hadoop.task.datawriter.AbstractDataWriterTask
configure, getEngineTaskConfigProvider, getPartitionCount, getTaskId, isChunkingEnabled, isRmdChunkingEnabled, setChunkingEnabled
-
Constructor Details
-
VeniceKafkaInputMapper
public VeniceKafkaInputMapper()
-
-
Method Details
-
getRecordReader
protected AbstractVeniceRecordReader<KafkaInputMapperKey,KafkaInputMapperValue> getRecordReader(VeniceProperties props) Description copied from class:AbstractInputRecordProcessor
A method for child classes to setupAbstractInputRecordProcessor.veniceRecordReader
.- Specified by:
getRecordReader
in classAbstractInputRecordProcessor<KafkaInputMapperKey,
KafkaInputMapperValue>
-
getFilterChain
-
configureTask
Description copied from class:AbstractDataWriterTask
Allow implementations of this class to configure task-specific stuff.- Overrides:
configureTask
in classAbstractInputRecordProcessor<KafkaInputMapperKey,
KafkaInputMapperValue> - Parameters:
props
- the job props that the task was configured with.
-
process
protected boolean process(KafkaInputMapperKey inputKey, KafkaInputMapperValue inputValue, Long timestamp, AtomicReference<byte[]> keyRef, AtomicReference<byte[]> valueRef, AtomicReference<Long> timestampRef, DataWriterTaskTracker dataWriterTaskTracker) Description copied from class:AbstractInputRecordProcessor
This function compresses the record and checks whether its uncompressed size exceeds the maximum allowed size. Regardless of the configuration, it tracks uncompressed record size violations in theDataWriterTaskTracker
. IfenableUncompressedMaxRecordSizeLimit
is enabled, any record that exceeds the limit will be dropped from further processing.The metrics collected by this function will be exposed in the PushJobDetails system store. Downstream, the
trackUncompressedRecordTooLargeFailure
metric is used to verify that the job does not violate the maximum uncompressed record size constraint.If
trackUncompressedRecordTooLargeFailure
is non-zero andenableUncompressedMaxRecordSizeLimit
is enabled, the job will throw aVeniceException
inVenicePushJob.runJobAndUpdateStatus()
, using the output ofVenicePushJob.updatePushJobDetailsWithJobDetails(DataWriterTaskTracker)
.When
enableUncompressedMaxRecordSizeLimit
is enabled, no records will be produced to Kafka inAbstractPartitionWriter.processValuesForKey(byte[], Iterator, Iterator, DataWriterTaskTracker)
.- Overrides:
process
in classAbstractInputRecordProcessor<KafkaInputMapperKey,
KafkaInputMapperValue>
-
close
public void close()- Specified by:
close
in interfaceAutoCloseable
- Specified by:
close
in interfaceCloseable
- Overrides:
close
in classAbstractInputRecordProcessor<KafkaInputMapperKey,
KafkaInputMapperValue>
-