Package com.linkedin.venice.hadoop.mapreduce.datawriter.partition

Class VeniceMRPartitioner

java.lang.Object

com.linkedin.venice.hadoop.mapreduce.datawriter.partition.VeniceMRPartitioner

All Implemented Interfaces:

org.apache.hadoop.mapred.JobConfigurable, org.apache.hadoop.mapred.Partitioner<org.apache.hadoop.io.BytesWritable,org.apache.hadoop.io.BytesWritable>

Direct Known Subclasses:

KafkaInputMRPartitioner

public class VeniceMRPartitioner
extends Object
implements org.apache.hadoop.mapred.Partitioner<org.apache.hadoop.io.BytesWritable,org.apache.hadoop.io.BytesWritable>

VeniceMRPartitioner will use the exact same partitioner as the one that will be used inside VeniceWriter (e.g. DefaultVenicePartitioner) to make sure the messages inside each partition belong to the same PubSub topic partition.

Field Summary

Fields

Modifier and Type	Field	Description
static final int	EMPTY_KEY_LENGTH
protected VenicePartitioner	venicePartitioner

Constructor Summary

Constructors

Constructor	Description
VeniceMRPartitioner()

Method Summary

Modifier and Type	Method	Description
void	configure(org.apache.hadoop.mapred.JobConf job)
protected int	getPartition(org.apache.hadoop.io.BytesWritable key, int numPartitions)
int	getPartition(org.apache.hadoop.io.BytesWritable key, org.apache.hadoop.io.BytesWritable value, int numPartitions)

Methods inherited from class java.lang.Object

clone, equals, finalize, getClass, hashCode, notify, notifyAll, toString, wait, wait, wait

Field Details

venicePartitioner

protected VenicePartitioner venicePartitioner

EMPTY_KEY_LENGTH

public static final int EMPTY_KEY_LENGTH

See Also:

• Constant Field Values

Constructor Details

VeniceMRPartitioner

public VeniceMRPartitioner()

Method Details

getPartition

public int getPartition(org.apache.hadoop.io.BytesWritable key,
 org.apache.hadoop.io.BytesWritable value,
 int numPartitions)

Specified by:

getPartition in interface org.apache.hadoop.mapred.Partitioner<org.apache.hadoop.io.BytesWritable,org.apache.hadoop.io.BytesWritable>

getPartition

protected int getPartition(org.apache.hadoop.io.BytesWritable key,
 int numPartitions)

configure

public void configure(org.apache.hadoop.mapred.JobConf job)

Specified by:

configure in interface org.apache.hadoop.mapred.JobConfigurable