Package com.linkedin.venice.spark.datawriter.jobs

Class DataWriterSparkJob

java.lang.Object

com.linkedin.venice.jobs.DataWriterComputeJob

com.linkedin.venice.spark.datawriter.jobs.AbstractDataWriterSparkJob

com.linkedin.venice.spark.datawriter.jobs.DataWriterSparkJob

All Implemented Interfaces:

ComputeJob, Closeable, AutoCloseable

public class DataWriterSparkJob
extends AbstractDataWriterSparkJob

The default implementation of AbstractDataWriterSparkJob for Avro and Vson file input formats.

Nested Class Summary

Nested classes/interfaces inherited from class com.linkedin.venice.jobs.DataWriterComputeJob

DataWriterComputeJob.ConfigSetter

Nested classes/interfaces inherited from interface com.linkedin.venice.jobs.ComputeJob

ComputeJob.Status

Field Summary

Fields inherited from class com.linkedin.venice.jobs.DataWriterComputeJob

PASS_THROUGH_CONFIG_PREFIXES

Constructor Summary

Constructors

Constructor	Description
DataWriterSparkJob()

Method Summary

Modifier and Type	Method	Description
protected org.apache.spark.sql.Dataset<org.apache.spark.sql.Row>	getKafkaInputDataFrame()	Get the input DataFrame from Kafka/PubSub source for repush workloads.
protected org.apache.spark.sql.Dataset<org.apache.spark.sql.Row>	getUserInputDataFrame()	Get the data frame based on the user's input data.

Methods inherited from class com.linkedin.venice.spark.datawriter.jobs.AbstractDataWriterSparkJob

applyChunkAssembly, applyCompaction, applyCompressionReEncoding, applyTTLFilter, close, configure, createPartitionWriterFactory, getAccumulatorsForDataWriterJob, getJobProperties, getPushJobSetting, getSparkSession, getStageMetricsRegistry, getStageMetricsSnapshot, getTaskTracker, kill, performsPreWriteQuotaCheck, runComputeJob, setInputConf, validateRmdSchema

Methods inherited from class com.linkedin.venice.jobs.DataWriterComputeJob

configure, getCurrentStorageQuotaSupplier, getFailureReason, getStatus, populateWithPassThroughConfigs, populateWithPassThroughConfigs, runJob, setCurrentStorageQuotaSupplier, validateJob

Methods inherited from class java.lang.Object

clone, equals, finalize, getClass, hashCode, notify, notifyAll, toString, wait, wait, wait

Constructor Details

DataWriterSparkJob

public DataWriterSparkJob()

Method Details

getUserInputDataFrame

protected org.apache.spark.sql.Dataset<org.apache.spark.sql.Row> getUserInputDataFrame()

Description copied from class: AbstractDataWriterSparkJob

Get the data frame based on the user's input data. The schema of the Row has the following constraints:

• Must contain a field "key" with the schema: DataTypes.BinaryType. This is the key of the record represented in serialized Avro.
• Must contain a field "value" with the schema: DataTypes.BinaryType. This is the value of the record represented in serialized Avro.
• Must not contain fields with names beginning with "_". These are reserved for internal use.
• Can contain fields that do not violate the above constraints

Specified by:

getUserInputDataFrame in class AbstractDataWriterSparkJob

Returns:

The data frame based on the user's input data

getKafkaInputDataFrame

protected org.apache.spark.sql.Dataset<org.apache.spark.sql.Row> getKafkaInputDataFrame()

Description copied from class: AbstractDataWriterSparkJob

Get the input DataFrame from Kafka/PubSub source for repush workloads. This method reads from a Venice version topic (Kafka) and returns a DataFrame with the RAW_PUBSUB_INPUT_TABLE_SCHEMA.

Specified by:

getKafkaInputDataFrame in class AbstractDataWriterSparkJob

Returns:

DataFrame containing the Kafka input data