SparkInputRecordProcessor (venice API)

java.lang.Object
- com.linkedin.venice.hadoop.task.datawriter.AbstractDataWriterTask
- - com.linkedin.venice.hadoop.task.datawriter.AbstractInputRecordProcessor<java.nio.ByteBuffer,java.nio.ByteBuffer>
  - - com.linkedin.venice.hadoop.spark.datawriter.recordprocessor.SparkInputRecordProcessor

All Implemented Interfaces:

java.io.Closeable, java.lang.AutoCloseable
```
public class SparkInputRecordProcessor
extends AbstractInputRecordProcessor<java.nio.ByteBuffer,java.nio.ByteBuffer>
```
An implementation of AbstractInputRecordProcessor for Spark that processes input records from the dataframe and emits an Iterator of Row with DEFAULT_SCHEMA as the schema.

Field Summary
- Fields inherited from class com.linkedin.venice.hadoop.task.datawriter.AbstractInputRecordProcessor
  veniceRecordReader
- Fields inherited from class com.linkedin.venice.hadoop.task.datawriter.AbstractDataWriterTask
  TASK_ID_NOT_SET

Constructor Summary

Constructors
Constructor Description

SparkInputRecordProcessor(java.util.Properties jobProperties, DataWriterAccumulators accumulators)

Method Summary

All Methods Instance Methods Concrete Methods
Modifier and Type	Method	Description
`protected AbstractVeniceRecordReader<java.nio.ByteBuffer,java.nio.ByteBuffer>`	`getRecordReader(VeniceProperties props)`	A method for child classes to setup `AbstractInputRecordProcessor.veniceRecordReader`.
`java.util.Iterator<org.apache.spark.sql.Row>`	`processRecord(org.apache.spark.sql.Row record)`

Methods inherited from class com.linkedin.venice.hadoop.task.datawriter.AbstractInputRecordProcessor
close, configureTask, process, processRecord, readDictionaryFromKafka

Methods inherited from class com.linkedin.venice.hadoop.task.datawriter.AbstractDataWriterTask
configure, getEngineTaskConfigProvider, getPartitionCount, getTaskId, isChunkingEnabled, isRmdChunkingEnabled, setChunkingEnabled

Methods inherited from class java.lang.Object
clone, equals, finalize, getClass, hashCode, notify, notifyAll, toString, wait, wait, wait

Constructor Detail

SparkInputRecordProcessor

public SparkInputRecordProcessor(java.util.Properties jobProperties,
                                 DataWriterAccumulators accumulators)

Method Detail
- processRecord
```
public java.util.Iterator<org.apache.spark.sql.Row> processRecord(org.apache.spark.sql.Row record)
```
- getRecordReader
```
protected AbstractVeniceRecordReader<java.nio.ByteBuffer,java.nio.ByteBuffer> getRecordReader(VeniceProperties props)
```
  Description copied from class: AbstractInputRecordProcessor
  
  A method for child classes to setup AbstractInputRecordProcessor.veniceRecordReader.
  
  Specified by:
  
  getRecordReader in class AbstractInputRecordProcessor<java.nio.ByteBuffer,java.nio.ByteBuffer>