Package com.linkedin.venice.hadoop.input.recordreader.avro

Class AbstractAvroRecordReader<INPUT_KEY,INPUT_VALUE>

java.lang.Object

com.linkedin.venice.hadoop.input.recordreader.AbstractVeniceRecordReader<INPUT_KEY,INPUT_VALUE>

com.linkedin.venice.hadoop.input.recordreader.avro.AbstractAvroRecordReader<INPUT_KEY,INPUT_VALUE>

Type Parameters:

INPUT_KEY - The format of the key as controlled by the input format

INPUT_VALUE - The format of the value as controlled by the input format

Direct Known Subclasses:

VeniceAvroRecordReader

public abstract class AbstractAvroRecordReader<INPUT_KEY,INPUT_VALUE>
extends AbstractVeniceRecordReader<INPUT_KEY,INPUT_VALUE>

An abstraction for a record reader that reads records from input into Avro-serialized keys and values.

Constructor Summary

Constructors

Constructor	Description
AbstractAvroRecordReader(org.apache.avro.Schema dataSchema, String keyFieldStr, String valueFieldStr, String rmdFieldStr, ETLValueSchemaTransformation etlValueSchemaTransformation, org.apache.avro.Schema updateSchema)	This constructor is used when data is read from HDFS.
AbstractAvroRecordReader(org.apache.avro.Schema dataSchema, String keyFieldStr, String valueFieldStr, String rmdFieldStr, ETLValueSchemaTransformation etlValueSchemaTransformation, org.apache.avro.Schema updateSchema, org.apache.avro.Schema writerValueSchema, org.apache.avro.Schema writerRmdSchema)

Method Summary

Modifier and Type	Method	Description
Object	getAvroKey(INPUT_KEY inputKey, INPUT_VALUE inputValue)	Return an Avro output key
Object	getAvroValue(INPUT_KEY inputKey, INPUT_VALUE inputValue)	return an Avro output value
org.apache.avro.Schema	getDataSchema()
protected abstract org.apache.avro.generic.IndexedRecord	getRecordDatum(INPUT_KEY inputKey, INPUT_VALUE inputValue)
Object	getRmdValue(INPUT_KEY inputKey, INPUT_VALUE inputValue)

Methods inherited from class com.linkedin.venice.hadoop.input.recordreader.AbstractVeniceRecordReader

configure, configure, getKeyBytes, getKeySchema, getRmdBytes, getRmdSchema, getValueBytes, getValueSchema

Methods inherited from class java.lang.Object

clone, equals, finalize, getClass, hashCode, notify, notifyAll, toString, wait, wait, wait

Constructor Details

AbstractAvroRecordReader

public AbstractAvroRecordReader(org.apache.avro.Schema dataSchema,
 String keyFieldStr,
 String valueFieldStr,
 String rmdFieldStr,
 ETLValueSchemaTransformation etlValueSchemaTransformation,
 org.apache.avro.Schema updateSchema)

This constructor is used when data is read from HDFS.

Parameters:

dataSchema - Schema of the avro file

keyFieldStr - Field name of the key field

valueFieldStr - Field name of the value field

etlValueSchemaTransformation - The type of transformation that was applied to this schema during ETL. When source data set is not an ETL job, use NONE.

AbstractAvroRecordReader

public AbstractAvroRecordReader(org.apache.avro.Schema dataSchema,
 String keyFieldStr,
 String valueFieldStr,
 String rmdFieldStr,
 ETLValueSchemaTransformation etlValueSchemaTransformation,
 org.apache.avro.Schema updateSchema,
 org.apache.avro.Schema writerValueSchema,
 org.apache.avro.Schema writerRmdSchema)

Parameters:

writerValueSchema - when non-null, input value records are projected down to this registered writer (target) value schema via VeniceSchemaProjector and serialized against it.

writerRmdSchema - when non-null, input RMD records are projected down to this writer RMD schema (the RMD schema generated against writerValueSchema) in lockstep with the value, and serialized against it.

Method Details

getDataSchema

public org.apache.avro.Schema getDataSchema()

getRecordDatum

protected abstract org.apache.avro.generic.IndexedRecord getRecordDatum(INPUT_KEY inputKey,
 INPUT_VALUE inputValue)

getAvroKey

public Object getAvroKey(INPUT_KEY inputKey,
 INPUT_VALUE inputValue)

Description copied from class: AbstractVeniceRecordReader

Return an Avro output key

Specified by:

getAvroKey in class AbstractVeniceRecordReader<INPUT_KEY,INPUT_VALUE>

getRmdValue

public Object getRmdValue(INPUT_KEY inputKey,
 INPUT_VALUE inputValue)

Specified by:

getRmdValue in class AbstractVeniceRecordReader<INPUT_KEY,INPUT_VALUE>

getAvroValue

public Object getAvroValue(INPUT_KEY inputKey,
 INPUT_VALUE inputValue)

Description copied from class: AbstractVeniceRecordReader

return an Avro output value

Specified by:

getAvroValue in class AbstractVeniceRecordReader<INPUT_KEY,INPUT_VALUE>