Class KafkaInputUtils
java.lang.Object
com.linkedin.venice.hadoop.input.kafka.KafkaInputUtils
-
Constructor Summary
Constructors -
Method Summary
Modifier and TypeMethodDescriptionstatic VeniceCompressor
getCompressor
(CompressorFactory compressorFactory, CompressionStrategy strategy, String kafkaUrl, String topic, VeniceProperties properties) static VeniceProperties
getConsumerProperties
(org.apache.hadoop.mapred.JobConf config) static VeniceProperties
getConsumerProperties
(org.apache.hadoop.mapred.JobConf config, Properties overrideProperties) Extracts and prepares the Kafka consumer properties for a Venice input job.static KafkaValueSerializer
getKafkaValueSerializer
(org.apache.hadoop.mapred.JobConf config)
-
Constructor Details
-
KafkaInputUtils
public KafkaInputUtils()
-
-
Method Details
-
getConsumerProperties
public static VeniceProperties getConsumerProperties(org.apache.hadoop.mapred.JobConf config, Properties overrideProperties) Extracts and prepares the Kafka consumer properties for a Venice input job.This method:
- Copies all Hadoop job configurations into a
Properties
object. - If an SSL configurator is specified, applies SSL settings and merges SSL properties into the consumer properties.
- Clips and merges any
VenicePushJobConstants.KIF_RECORD_READER_KAFKA_CONFIG_PREFIX
prefixed properties into the consumer properties. - Sets a large receive buffer size (4MB) to support remote Kafka re-push scenarios.
- Sets the PubSub bootstrap server address
- Parameters:
config
- the HadoopJobConf
containing the job configurations- Returns:
- a
VeniceProperties
object containing the prepared Kafka consumer properties - Throws:
VeniceException
- if SSL configuration setup fails
- Copies all Hadoop job configurations into a
-
getConsumerProperties
-
getKafkaValueSerializer
-
getCompressor
public static VeniceCompressor getCompressor(CompressorFactory compressorFactory, CompressionStrategy strategy, String kafkaUrl, String topic, VeniceProperties properties)
-