Package com.linkedin.venice.hadoop.input.kafka.ttl

Class VeniceRmdTTLFilter<INPUT_VALUE>

  • Type Parameters:
    INPUT_VALUE - , the value contains schemaID, rmdId and rmdPayload that are required to retrieve RMD timestamp.
    All Implemented Interfaces:
    java.io.Closeable, java.lang.AutoCloseable
    Direct Known Subclasses:
    VeniceChunkedPayloadTTLFilter, VeniceKafkaInputTTLFilter
    public abstract class VeniceRmdTTLFilter<INPUT_VALUE>
extends AbstractVeniceFilter<INPUT_VALUE>
    This class is responsible to filter records based on the RMD information and the ttl config. It requires RMD schemas for a given store from an existing HDFS directory to be able to parse timestamp information in RMD.

    • Field Detail

      • rmdSchemaMap

        protected final java.util.Map<RmdVersionId,​org.apache.avro.Schema> rmdSchemaMap

      • valueSchemaMap

        protected final java.util.Map<java.lang.Integer,​org.apache.avro.Schema> valueSchemaMap

    • Constructor Detail

      • VeniceRmdTTLFilter

        public VeniceRmdTTLFilter​(VeniceProperties props)
                   throws java.io.IOException
        Throws:
        java.io.IOException

    • Method Detail

      • checkAndMaybeFilterValue

        public boolean checkAndMaybeFilterValue​(INPUT_VALUE value)
        Description copied from class: AbstractVeniceFilter
        This function implements how to parse the value and determine if filtering is needed. For certain value from Active/Active partial update enabled stores, it might filter out part of its input value and only keep the remaining fresh part based on filter timestamp.
        Specified by:
        checkAndMaybeFilterValue in class AbstractVeniceFilter<INPUT_VALUE>
        Returns:
        true if the value should be filtered out, otherwise false.

      • close

        public void close()

      • getSchemaId

        protected abstract int getSchemaId​(INPUT_VALUE value)

      • getRmdProtocolId

        protected abstract int getRmdProtocolId​(INPUT_VALUE value)

      • getRmdPayload

        protected abstract java.nio.ByteBuffer getRmdPayload​(INPUT_VALUE value)

      • getValuePayload

        protected abstract java.nio.ByteBuffer getValuePayload​(INPUT_VALUE value)

      • updateRmdPayload

        protected abstract void updateRmdPayload​(INPUT_VALUE value,
                                         java.nio.ByteBuffer payload)

      • updateValuePayload

        protected abstract void updateValuePayload​(INPUT_VALUE value,
                                           byte[] payload)

      • skipRmdRecord

        protected boolean skipRmdRecord​(INPUT_VALUE value)
        Define how records could be skipped if certain conditions are met. Do not skip by default.
        Parameters:
        value -
        Returns:
        true if this record should not be filtered and skipped.