Package com.linkedin.venice.spark.utils

Class RowToAvroConverter

java.lang.Object

com.linkedin.venice.spark.utils.RowToAvroConverter

public final class RowToAvroConverter
extends java.lang.Object

A utility class to convert Spark SQL Row to an Avro GenericRecord with the specified schema. This has been written in accordance with the following resources:

• Spark Avro data source documentation
• Avro specification
• Spark's internal Catalyst type to Avro bytes implementation

Spark's implementation is not ideal to be used directly for two reasons:

• It cannot handle complex unions in the version of Spark that we use (3.3.3). The support was added in 3.4.0.
• It converts directly to Avro binary that we need to deserialize, and that incurs an additional serde cost.

Method Summary

Modifier and Type	Method	Description
static org.apache.avro.generic.GenericRecord	convert(org.apache.spark.sql.Row row, org.apache.avro.Schema schema)

Methods inherited from class java.lang.Object

clone, equals, finalize, getClass, hashCode, notify, notifyAll, toString, wait, wait, wait

Method Detail

convert

public static org.apache.avro.generic.GenericRecord convert(org.apache.spark.sql.Row row,
                                                            org.apache.avro.Schema schema)