Package com.linkedin.venice.hadoop.input.kafka

Class KafkaInputUtils

java.lang.Object

com.linkedin.venice.hadoop.input.kafka.KafkaInputUtils

public class KafkaInputUtils
extends Object

Constructor Summary

Constructors

Constructor	Description
KafkaInputUtils()

Method Summary

Modifier and Type	Method	Description
static VeniceCompressor	getCompressor(CompressorFactory compressorFactory, CompressionStrategy strategy, String kafkaUrl, String topic, VeniceProperties properties)
static VeniceProperties	getConsumerProperties(org.apache.hadoop.mapred.JobConf config)
static VeniceProperties	getConsumerProperties(org.apache.hadoop.mapred.JobConf config, Properties overrideProperties)	Extracts and prepares the Kafka consumer properties for a Venice input job.
static KafkaValueSerializer	getKafkaValueSerializer(org.apache.hadoop.mapred.JobConf config)
static Map<String,String>	putSchemaMapIntoProperties(Map<Integer,String> schemaMap)	Puts a Map of schema ID to schema string into Properties using the specified prefix.

Methods inherited from class java.lang.Object

clone, equals, finalize, getClass, hashCode, notify, notifyAll, toString, wait, wait, wait

Constructor Details

KafkaInputUtils

public KafkaInputUtils()

Method Details

getConsumerProperties

public static VeniceProperties getConsumerProperties(org.apache.hadoop.mapred.JobConf config,
 Properties overrideProperties)

Extracts and prepares the Kafka consumer properties for a Venice input job.

This method:

• Copies all Hadoop job configurations into a Properties object.
• If an SSL configurator is specified, applies SSL settings and merges SSL properties into the consumer properties.
• Clips and merges any VenicePushJobConstants.KIF_RECORD_READER_KAFKA_CONFIG_PREFIX prefixed properties into the consumer properties.
• Sets a large receive buffer size (4MB) to support remote Kafka re-push scenarios.
• Sets the PubSub bootstrap server address

Parameters:

config - the Hadoop JobConf containing the job configurations

Returns:

a VeniceProperties object containing the prepared Kafka consumer properties

Throws:

VeniceException - if SSL configuration setup fails

getConsumerProperties

public static VeniceProperties getConsumerProperties(org.apache.hadoop.mapred.JobConf config)

getKafkaValueSerializer

public static KafkaValueSerializer getKafkaValueSerializer(org.apache.hadoop.mapred.JobConf config)

getCompressor

public static VeniceCompressor getCompressor(CompressorFactory compressorFactory,
 CompressionStrategy strategy,
 String kafkaUrl,
 String topic,
 VeniceProperties properties)

putSchemaMapIntoProperties

public static Map<String,String> putSchemaMapIntoProperties(Map<Integer,String> schemaMap)

Puts a Map of schema ID to schema string into Properties using the specified prefix. Each entry is stored as: prefix + schemaId = schemaString

Parameters:

schemaMap - the Map containing schema ID to schema string mappings