Package com.linkedin.venice.spark.input.pubsub

Interface PubSubMessageConverter

All Known Implementing Classes:

VenicePubSubMessageToRow

Functional Interface:

This is a functional interface and can therefore be used as the assignment target for a lambda expression or method reference.

@FunctionalInterface
public interface PubSubMessageConverter

Functional interface for converting PubSub messages to Spark InternalRows.

Method Summary

Modifier and Type	Method	Description
org.apache.spark.sql.catalyst.InternalRow	convert(@NotNull PubSubMessage<KafkaKey,KafkaMessageEnvelope,PubSubPosition> pubSubMessage, String region, int partitionNumber)	Converts a PubSub message to a Spark InternalRow.

Method Details

convert

org.apache.spark.sql.catalyst.InternalRow convert(@NotNull
 @NotNull PubSubMessage<KafkaKey,KafkaMessageEnvelope,PubSubPosition> pubSubMessage,
 String region,
 int partitionNumber)

Converts a PubSub message to a Spark InternalRow.

Parameters:

pubSubMessage - The PubSub message to process. Contains key, value, and metadata.

region - The region identifier to include in the row.

partitionNumber - The partition number to include in the row.

Returns:

An InternalRow containing the processed message data. The row includes the following fields: 1. Region (String) 2. Partition number (int) 3. Message type (int) 4. Offset (long) 5. Schema ID (int) 6. Key bytes (byte[]) 7. Value bytes (byte[]) 8. Replication metadata payload bytes (byte[]) 9. Replication metadata version ID (int) See SparkConstants.RAW_PUBSUB_INPUT_TABLE_SCHEMA for the schema definition.