Package com.linkedin.venice.duckdb

Class DuckDBDaVinciRecordTransformer

java.lang.Object

com.linkedin.davinci.client.DaVinciRecordTransformer<org.apache.avro.generic.GenericRecord,org.apache.avro.generic.GenericRecord,org.apache.avro.generic.GenericRecord>

com.linkedin.venice.duckdb.DuckDBDaVinciRecordTransformer

All Implemented Interfaces:

Closeable, AutoCloseable

public class DuckDBDaVinciRecordTransformer
extends DaVinciRecordTransformer<org.apache.avro.generic.GenericRecord,org.apache.avro.generic.GenericRecord,org.apache.avro.generic.GenericRecord>

Constructor Summary

Constructors

Constructor	Description
DuckDBDaVinciRecordTransformer(int storeVersion, org.apache.avro.Schema keySchema, org.apache.avro.Schema inputValueSchema, org.apache.avro.Schema outputValueSchema, boolean storeRecordsInDaVinci, String baseDir, String storeNameWithoutVersionInfo, Set<String> columnsToProject)

Method Summary

Modifier and Type	Method	Description
String	buildStoreNameWithVersion(int version)
void	close()
String	getDuckDBUrl()
void	onEndVersionIngestion(int currentVersion)	Lifecycle event triggered when record consumption is stopped for DaVinciRecordTransformer.storeVersion.
void	onStartVersionIngestion(boolean isCurrentVersion)	Lifecycle event triggered before consuming records for DaVinciRecordTransformer.storeVersion.
void	processDelete(Lazy<org.apache.avro.generic.GenericRecord> key)	Override this method to customize the behavior for record deletions.
void	processPut(Lazy<org.apache.avro.generic.GenericRecord> key, Lazy<org.apache.avro.generic.GenericRecord> value)	Implement this method to manage custom state outside the Da Vinci Client.
DaVinciRecordTransformerResult<org.apache.avro.generic.GenericRecord>	transform(Lazy<org.apache.avro.generic.GenericRecord> key, Lazy<org.apache.avro.generic.GenericRecord> value)	Implement this method to transform records before they are stored.
boolean	useUniformInputValueSchema()

Methods inherited from class com.linkedin.davinci.client.DaVinciRecordTransformer

getClassHash, getInputValueSchema, getKeySchema, getOutputValueSchema, getRecordTransformerUtility, getStoreRecordsInDaVinci, getStoreVersion, onRecovery, prependSchemaIdToHeader, prependSchemaIdToHeader, transformAndProcessPut

Methods inherited from class java.lang.Object

clone, equals, finalize, getClass, hashCode, notify, notifyAll, toString, wait, wait, wait

Constructor Details

DuckDBDaVinciRecordTransformer

public DuckDBDaVinciRecordTransformer(int storeVersion,
 org.apache.avro.Schema keySchema,
 org.apache.avro.Schema inputValueSchema,
 org.apache.avro.Schema outputValueSchema,
 boolean storeRecordsInDaVinci,
 String baseDir,
 String storeNameWithoutVersionInfo,
 Set<String> columnsToProject)

Method Details

transform

public DaVinciRecordTransformerResult<org.apache.avro.generic.GenericRecord> transform(Lazy<org.apache.avro.generic.GenericRecord> key,
 Lazy<org.apache.avro.generic.GenericRecord> value)

Description copied from class: DaVinciRecordTransformer

Implement this method to transform records before they are stored. This can be useful for tasks such as filtering out unused fields to save storage space.

Specified by:

transform in class DaVinciRecordTransformer<org.apache.avro.generic.GenericRecord,org.apache.avro.generic.GenericRecord,org.apache.avro.generic.GenericRecord>

Parameters:

key - the key of the record to be transformed

value - the value of the record to be transformed

Returns:

DaVinciRecordTransformerResult

processPut

public void processPut(Lazy<org.apache.avro.generic.GenericRecord> key,
 Lazy<org.apache.avro.generic.GenericRecord> value)

Description copied from class: DaVinciRecordTransformer

Implement this method to manage custom state outside the Da Vinci Client.

Specified by:

processPut in class DaVinciRecordTransformer<org.apache.avro.generic.GenericRecord,org.apache.avro.generic.GenericRecord,org.apache.avro.generic.GenericRecord>

Parameters:

key - the key of the record to be put

value - the value of the record to be put, derived from the output of DaVinciRecordTransformer.transform(Lazy key, Lazy value)

processDelete

public void processDelete(Lazy<org.apache.avro.generic.GenericRecord> key)

Description copied from class: DaVinciRecordTransformer

Override this method to customize the behavior for record deletions. For example, you can use this method to delete records from a custom storage outside the Da Vinci Client. By default, it performs no operation.

Overrides:

processDelete in class DaVinciRecordTransformer<org.apache.avro.generic.GenericRecord,org.apache.avro.generic.GenericRecord,org.apache.avro.generic.GenericRecord>

Parameters:

key - the key of the record to be deleted

onStartVersionIngestion

public void onStartVersionIngestion(boolean isCurrentVersion)

Description copied from class: DaVinciRecordTransformer

Lifecycle event triggered before consuming records for DaVinciRecordTransformer.storeVersion. Use this method to perform setup operations such as opening database connections or creating tables. By default, it performs no operation.

Overrides:

onStartVersionIngestion in class DaVinciRecordTransformer<org.apache.avro.generic.GenericRecord,org.apache.avro.generic.GenericRecord,org.apache.avro.generic.GenericRecord>

onEndVersionIngestion

public void onEndVersionIngestion(int currentVersion)

Description copied from class: DaVinciRecordTransformer

Lifecycle event triggered when record consumption is stopped for DaVinciRecordTransformer.storeVersion. Use this method to perform cleanup operations such as closing database connections or dropping tables. By default, it performs no operation.

Overrides:

onEndVersionIngestion in class DaVinciRecordTransformer<org.apache.avro.generic.GenericRecord,org.apache.avro.generic.GenericRecord,org.apache.avro.generic.GenericRecord>

useUniformInputValueSchema

public boolean useUniformInputValueSchema()

Overrides:

useUniformInputValueSchema in class DaVinciRecordTransformer<org.apache.avro.generic.GenericRecord,org.apache.avro.generic.GenericRecord,org.apache.avro.generic.GenericRecord>

getDuckDBUrl

public String getDuckDBUrl()

buildStoreNameWithVersion

public String buildStoreNameWithVersion(int version)

close

public void close()
           throws IOException

Throws:

IOException