Package com.linkedin.venice.hadoop.input.kafka.ttl

Class VeniceRmdTTLFilter<INPUT_VALUE>

java.lang.Object
com.linkedin.venice.hadoop.AbstractVeniceFilter<INPUT_VALUE>
com.linkedin.venice.hadoop.input.kafka.ttl.VeniceRmdTTLFilter<INPUT_VALUE>
Type Parameters:
INPUT_VALUE - , the value contains schemaID, rmdId and rmdPayload that are required to retrieve RMD timestamp.
All Implemented Interfaces:
Closeable, AutoCloseable
Direct Known Subclasses:
VeniceChunkedPayloadTTLFilter, VeniceKafkaInputTTLFilter
public abstract class VeniceRmdTTLFilter<INPUT_VALUE> extends AbstractVeniceFilter<INPUT_VALUE>
This class is responsible to filter records based on the RMD information and the ttl config. It requires RMD schemas for a given store from an existing HDFS directory to be able to parse timestamp information in RMD.

  • Field Details

    • rmdSchemaMap

      protected final Map<RmdVersionId,org.apache.avro.Schema> rmdSchemaMap

    • valueSchemaMap

      protected final Map<Integer,org.apache.avro.Schema> valueSchemaMap

  • Constructor Details

  • Method Details

    • checkAndMaybeFilterValue

      public boolean checkAndMaybeFilterValue(INPUT_VALUE value)
      Description copied from class: AbstractVeniceFilter
      This function implements how to parse the value and determine if filtering is needed. For certain value from Active/Active partial update enabled stores, it might filter out part of its input value and only keep the remaining fresh part based on filter timestamp.
      Specified by:
      checkAndMaybeFilterValue in class AbstractVeniceFilter<INPUT_VALUE>
      Returns:
      true if the value should be filtered out, otherwise false.

    • close

      public void close()

    • getSchemaId

      protected abstract int getSchemaId(INPUT_VALUE value)

    • getRmdProtocolId

      protected abstract int getRmdProtocolId(INPUT_VALUE value)

    • getRmdPayload

      protected abstract ByteBuffer getRmdPayload(INPUT_VALUE value)

    • getValuePayload

      protected abstract ByteBuffer getValuePayload(INPUT_VALUE value)

    • updateRmdPayload

      protected abstract void updateRmdPayload(INPUT_VALUE value, ByteBuffer payload)

    • updateValuePayload

      protected abstract void updateValuePayload(INPUT_VALUE value, byte[] payload)

    • skipRmdRecord

      protected boolean skipRmdRecord(INPUT_VALUE value)
      Define how records could be skipped if certain conditions are met. Do not skip by default.
      Parameters:
      value -
      Returns:
      true if this record should not be filtered and skipped.