Class AbstractInputRecordProcessor<INPUT_KEY,​INPUT_VALUE>

  • Type Parameters:
    INPUT_KEY - type of the input key read from InputFormat
    INPUT_VALUE - type of the input value read from InputFormat
    All Implemented Interfaces:, java.lang.AutoCloseable
    Direct Known Subclasses:
    AbstractVeniceMapper, SparkInputRecordProcessor

    public abstract class AbstractInputRecordProcessor<INPUT_KEY,​INPUT_VALUE>
    extends AbstractDataWriterTask
    An abstraction of the task that processes each record from the input, and returns serialized, and potentially compressed, Avro key/value pairs.
    • Constructor Detail

      • AbstractInputRecordProcessor

        public AbstractInputRecordProcessor()
    • Method Detail

      • processRecord

        protected final void processRecord​(INPUT_KEY inputKey,
                                           INPUT_VALUE inputValue,
                                           java.util.function.BiConsumer<byte[],​byte[]> recordEmitter,
                                           DataWriterTaskTracker dataWriterTaskTracker)
      • process

        protected boolean process​(INPUT_KEY inputKey,
                                  INPUT_VALUE inputValue,
                                  java.util.concurrent.atomic.AtomicReference<byte[]> keyRef,
                                  java.util.concurrent.atomic.AtomicReference<byte[]> valueRef,
                                  DataWriterTaskTracker dataWriterTaskTracker)
        This function will return true if the input key/value pair is valid.
      • readDictionaryFromKafka

        protected java.nio.ByteBuffer readDictionaryFromKafka​(java.lang.String topicName,
                                                              VeniceProperties props)
        This function is added to allow it to be mocked for tests. Since mocking this function of an actual object in AbstractTestVeniceMapper#getMapper(int, int, Consumer) ended up hitting the original function always, added an override for this in TestVeniceAvroMapperClass.
      • close

        public void close()
        Specified by:
        close in interface java.lang.AutoCloseable
        Specified by:
        close in interface