Package com.linkedin.davinci.client

Class InternalDaVinciRecordTransformer<K,V,O>

java.lang.Object
com.linkedin.davinci.client.DaVinciRecordTransformer<K,V,O>
com.linkedin.davinci.client.InternalDaVinciRecordTransformer<K,V,O>
Type Parameters:
K - type of the input key
V - type of the input value
O - type of the output value
All Implemented Interfaces:
Closeable, AutoCloseable
@Experimental public class InternalDaVinciRecordTransformer<K,V,O> extends DaVinciRecordTransformer<K,V,O>
This is an implementation of DaVinciRecordTransformer that is used internally inside the StoreIngestionTask.

  • Constructor Details

    • InternalDaVinciRecordTransformer

      public InternalDaVinciRecordTransformer(DaVinciRecordTransformer recordTransformer, org.apache.avro.Schema keySchema, org.apache.avro.Schema inputValueSchema, org.apache.avro.Schema outputValueSchema, DaVinciRecordTransformerConfig recordTransformerConfig)

  • Method Details

    • transform

      public DaVinciRecordTransformerResult<O> transform(Lazy<K> key, Lazy<V> value, int partitionId)
      Description copied from class: DaVinciRecordTransformer
      Implement this method to transform records before they are stored. This can be useful for tasks such as filtering out unused fields to save storage space.
      Specified by:
      transform in class DaVinciRecordTransformer<K,V,O>
      Parameters:
      key - the key of the record to be transformed
      value - the value of the record to be transformed
      partitionId - what partition the record came from
      Returns:
      DaVinciRecordTransformerResult

    • processPut

      public void processPut(Lazy<K> key, Lazy<O> value, int partitionId)
      Description copied from class: DaVinciRecordTransformer
      Implement this method to manage custom state outside the Da Vinci Client.
      Specified by:
      processPut in class DaVinciRecordTransformer<K,V,O>
      Parameters:
      key - the key of the record to be put
      value - the value of the record to be put, derived from the output of DaVinciRecordTransformer.transform(Lazy key, Lazy value, int partitionId)
      partitionId - what partition the record came from

    • processDelete

      public void processDelete(Lazy<K> key, int partitionId)
      Description copied from class: DaVinciRecordTransformer
      Override this method to customize the behavior for record deletions. For example, you can use this method to delete records from a custom storage outside the Da Vinci Client. By default, it performs no operation.
      Overrides:
      processDelete in class DaVinciRecordTransformer<K,V,O>
      Parameters:
      key - the key of the record to be deleted
      partitionId - what partition the record is being deleted from

    • onStartVersionIngestion

      public void onStartVersionIngestion(boolean isCurrentVersion)
      Description copied from class: DaVinciRecordTransformer
      Lifecycle event triggered before consuming records for DaVinciRecordTransformer.storeVersion. Use this method to perform setup operations such as opening database connections or creating tables. By default, it performs no operation.
      Overrides:
      onStartVersionIngestion in class DaVinciRecordTransformer<K,V,O>

    • onEndVersionIngestion

      public void onEndVersionIngestion(int currentVersion)
      Description copied from class: DaVinciRecordTransformer
      Lifecycle event triggered when record consumption is stopped for DaVinciRecordTransformer.storeVersion. Use this method to perform cleanup operations such as closing database connections or dropping tables. By default, it performs no operation.
      Overrides:
      onEndVersionIngestion in class DaVinciRecordTransformer<K,V,O>

    • useUniformInputValueSchema

      public boolean useUniformInputValueSchema()
      Overrides:
      useUniformInputValueSchema in class DaVinciRecordTransformer<K,V,O>

    • onVersionSwap

      public void onVersionSwap(int currentVersion, int futureVersion, int partitionId)
      Lifecycle event triggered when a version swap is detected for partitionId It is used for DVRT CDC.

    • internalOnRecovery

      public void internalOnRecovery(StorageEngine storageEngine, int partitionId, InternalAvroSpecificSerializer<PartitionState> partitionStateSerializer, Lazy<VeniceCompressor> compressor)
      Using a wrapper around onRecovery because when calculating the class hash it grabs the name of the current class that is invoking it. If we directly invoke onRecovery from this class, the class hash will be calculated based on the contents of InternalDaVinciRecordTransformer, not the user's implementation of DVRT. We also can't override onRecovery like the other methods because this method is final and the implementation should never be overridden.

    • getCountDownStartConsumptionLatchCount

      public long getCountDownStartConsumptionLatchCount()

    • countDownStartConsumptionLatch

      public void countDownStartConsumptionLatch()
      This method gets invoked when local RocksDB scan has completed for a partition.

    • close

      public void close() throws IOException
      Throws:
      IOException