Package com.linkedin.venice.spark.datawriter.recordprocessor

Class SparkInputRecordProcessor

java.lang.Object

com.linkedin.venice.hadoop.task.datawriter.AbstractDataWriterTask

com.linkedin.venice.hadoop.task.datawriter.AbstractInputRecordProcessor<ByteBuffer,ByteBuffer>

com.linkedin.venice.spark.datawriter.recordprocessor.SparkInputRecordProcessor

All Implemented Interfaces:

Closeable, AutoCloseable

public class SparkInputRecordProcessor
extends AbstractInputRecordProcessor<ByteBuffer,ByteBuffer>

An implementation of AbstractInputRecordProcessor for Spark that processes input records from the dataframe and emits an Iterator of Row with DEFAULT_SCHEMA as the schema.

Field Summary

Fields inherited from class com.linkedin.venice.hadoop.task.datawriter.AbstractInputRecordProcessor

EMPTY_BYTES, veniceRecordReader

Fields inherited from class com.linkedin.venice.hadoop.task.datawriter.AbstractDataWriterTask

TASK_ID_NOT_SET

Constructor Summary

Constructors

Constructor	Description
SparkInputRecordProcessor(Properties jobProperties, DataWriterAccumulators accumulators)

Method Summary

Modifier and Type	Method	Description
protected AbstractVeniceRecordReader<ByteBuffer,ByteBuffer>	getRecordReader(VeniceProperties props)	A method for child classes to setup AbstractInputRecordProcessor.veniceRecordReader.
Iterator<org.apache.spark.sql.Row>	processRecord(org.apache.spark.sql.Row record)

Methods inherited from class com.linkedin.venice.hadoop.task.datawriter.AbstractInputRecordProcessor

close, configureTask, process, processRecord, readDictionaryFromKafka

Methods inherited from class com.linkedin.venice.hadoop.task.datawriter.AbstractDataWriterTask

configure, getEngineTaskConfigProvider, getPartitionCount, getTaskId, isChunkingEnabled, isRmdChunkingEnabled, setChunkingEnabled

Methods inherited from class java.lang.Object

clone, equals, finalize, getClass, hashCode, notify, notifyAll, toString, wait, wait, wait

Constructor Details

SparkInputRecordProcessor

public SparkInputRecordProcessor(Properties jobProperties,
 DataWriterAccumulators accumulators)

Method Details

processRecord

public Iterator<org.apache.spark.sql.Row> processRecord(org.apache.spark.sql.Row record)

getRecordReader

protected AbstractVeniceRecordReader<ByteBuffer,ByteBuffer> getRecordReader(VeniceProperties props)

Description copied from class: AbstractInputRecordProcessor

A method for child classes to setup AbstractInputRecordProcessor.veniceRecordReader.

Specified by:

getRecordReader in class AbstractInputRecordProcessor<ByteBuffer,ByteBuffer>