Package com.linkedin.davinci.consumer

Class VeniceChangelogConsumerDaVinciRecordTransformerImpl.DaVinciRecordTransformerChangelogConsumer

java.lang.Object

com.linkedin.davinci.client.DaVinciRecordTransformer<K,V,V>

com.linkedin.davinci.consumer.VeniceChangelogConsumerDaVinciRecordTransformerImpl.DaVinciRecordTransformerChangelogConsumer

All Implemented Interfaces:

Closeable, AutoCloseable

Enclosing class:

VeniceChangelogConsumerDaVinciRecordTransformerImpl<K,V>

public class VeniceChangelogConsumerDaVinciRecordTransformerImpl.DaVinciRecordTransformerChangelogConsumer
extends DaVinciRecordTransformer<K,V,V>

Constructor Summary

Constructors

Constructor	Description
DaVinciRecordTransformerChangelogConsumer(String storeName, int storeVersion, org.apache.avro.Schema keySchema, org.apache.avro.Schema inputValueSchema, org.apache.avro.Schema outputValueSchema, DaVinciRecordTransformerConfig recordTransformerConfig)

Method Summary

Modifier and Type	Method	Description
void	addControlMessageToBuffer(int partitionId, PubSubPosition offset, ControlMessage controlMessage)	Add ControlMessage to the buffer.
void	addMessageToBuffer(K key, V value, int partitionId, DaVinciRecordTransformerRecordMetadata recordMetadata)	Add ChangeCapturePubSubMessage to the buffer based on record metadata.
void	close()
void	onControlMessage(int partitionId, PubSubPosition offset, ControlMessage controlMessage)	If includeControlMessages is FALSE, or the client is NOT version specific, this method is a no-op.
void	onHeartbeat(int partitionId, long heartbeatTimestamp)	Receive heartbeat timestamp for a partition and update latest seen time.
void	onStartVersionIngestion(int partitionId, boolean isCurrentVersion)	Callback invoked before consuming records for DaVinciRecordTransformer.storeVersion.
void	onVersionSwap(int currentVersion, int futureVersion, int partitionId)
void	processDelete(Lazy<K> key, int partitionId, DaVinciRecordTransformerRecordMetadata recordMetadata)	Optional callback for delete events (for example, remove from an external system).
void	processPut(Lazy<K> key, Lazy<V> value, int partitionId, DaVinciRecordTransformerRecordMetadata recordMetadata)	Callback for put/update events (for example, write to an external system).
DaVinciRecordTransformerResult<V>	transform(Lazy<K> key, Lazy<V> value, int partitionId, DaVinciRecordTransformerRecordMetadata recordMetadata)	Callback to transform records before they are stored.

Methods inherited from class com.linkedin.davinci.client.DaVinciRecordTransformer

getAlwaysBootstrapFromVersionTopic, getClassHash, getInputValueSchema, getKeySchema, getOutputValueSchema, getRecordTransformerUtility, getStoreName, getStoreRecordsInDaVinci, getStoreVersion, onEndVersionIngestion, onRecovery, prependSchemaIdToHeader, prependSchemaIdToHeader, transformAndProcessPut, useUniformInputValueSchema

Methods inherited from class java.lang.Object

clone, equals, finalize, getClass, hashCode, notify, notifyAll, toString, wait, wait, wait

Constructor Details

DaVinciRecordTransformerChangelogConsumer

public DaVinciRecordTransformerChangelogConsumer(String storeName,
 int storeVersion,
 org.apache.avro.Schema keySchema,
 org.apache.avro.Schema inputValueSchema,
 org.apache.avro.Schema outputValueSchema,
 DaVinciRecordTransformerConfig recordTransformerConfig)

Method Details

onStartVersionIngestion

public void onStartVersionIngestion(int partitionId,
 boolean isCurrentVersion)

Description copied from class: DaVinciRecordTransformer

Callback invoked before consuming records for DaVinciRecordTransformer.storeVersion. Use this to open connections, create tables, or initialize resources. By default, it's a no-op.

Overrides:

onStartVersionIngestion in class DaVinciRecordTransformer<K,V,V>

Parameters:

partitionId - what partition is being subscribed

isCurrentVersion - whether the version being started is the current serving version; if not it's a future version

transform

public DaVinciRecordTransformerResult<V> transform(Lazy<K> key,
 Lazy<V> value,
 int partitionId,
 DaVinciRecordTransformerRecordMetadata recordMetadata)

Description copied from class: DaVinciRecordTransformer

Callback to transform records before they are stored. This can be useful for tasks such as filtering out unused fields to save storage space. Result types: - SKIP: drop the record - UNCHANGED: keep the original value; useful when only callbacks/notifications are needed - TRANSFORMED: record has been transformed and should be persisted to disk

Specified by:

transform in class DaVinciRecordTransformer<K,V,V>

Parameters:

key - the key of the record to be transformed

value - the value of the record to be transformed

partitionId - what partition the record came from

recordMetadata - returns DaVinciRecordTransformerRecordMetadata if enabled in DaVinciRecordTransformerConfig, null otherwise

Returns:

DaVinciRecordTransformerResult

addMessageToBuffer

public void addMessageToBuffer(K key,
 V value,
 int partitionId,
 DaVinciRecordTransformerRecordMetadata recordMetadata)

Add ChangeCapturePubSubMessage to the buffer based on record metadata.

addControlMessageToBuffer

public void addControlMessageToBuffer(int partitionId,
 PubSubPosition offset,
 ControlMessage controlMessage)

Add ControlMessage to the buffer. Key and Value are both null. Details for control message are in controlMessage param.

processPut

public void processPut(Lazy<K> key,
 Lazy<V> value,
 int partitionId,
 DaVinciRecordTransformerRecordMetadata recordMetadata)

Description copied from class: DaVinciRecordTransformer

Callback for put/update events (for example, write to an external system). Invoked after DaVinciRecordTransformer.transform(Lazy, Lazy, int, DaVinciRecordTransformerRecordMetadata) when the result is UNCHANGED or TRANSFORMED. Also invoked during startup/recovery when Da Vinci replays records persisted on disk to rehydrate external systems.

Specified by:

processPut in class DaVinciRecordTransformer<K,V,V>

Parameters:

key - the key of the record to be put

value - the value of the record to be put, either the original value or the transformed value

partitionId - what partition the record came from

recordMetadata - returns DaVinciRecordTransformerRecordMetadata if enabled in DaVinciRecordTransformerConfig, null otherwise.

processDelete

public void processDelete(Lazy<K> key,
 int partitionId,
 DaVinciRecordTransformerRecordMetadata recordMetadata)

Description copied from class: DaVinciRecordTransformer

Optional callback for delete events (for example, remove from an external system). By default, this is a no-op.

Overrides:

processDelete in class DaVinciRecordTransformer<K,V,V>

Parameters:

key - the key of the record to be deleted

partitionId - what partition the record is being deleted from

recordMetadata - returns DaVinciRecordTransformerRecordMetadata if enabled in DaVinciRecordTransformerConfig, null otherwise.

onControlMessage

public void onControlMessage(int partitionId,
 PubSubPosition offset,
 ControlMessage controlMessage)

If includeControlMessages is FALSE, or the client is NOT version specific, this method is a no-op. If includeControlMessages is true, AND the client is version specific, it adds the control message to the buffer.

onVersionSwap

public void onVersionSwap(int currentVersion,
 int futureVersion,
 int partitionId)

onHeartbeat

public void onHeartbeat(int partitionId,
 long heartbeatTimestamp)

Receive heartbeat timestamp for a partition and update latest seen time.

close

public void close()
           throws IOException

Throws:

IOException