Package com.linkedin.venice.hadoop

Interface Summary

Interface	Description
InputDataInfoProvider	This interface lets users get input data information
JobClientWrapper
SentPushJobDetailsTracker	Interface of class that is used to keep track of push job details sent to the Venice controller.

Class Summary

Class	Description
AbstractMapReduceTask	Class for commonalities between AbstractVeniceMapper and VeniceReducer.
AbstractVeniceFilter<INPUT_VALUE>	An abstraction to filter given data type.
AbstractVeniceMapper<INPUT_KEY,INPUT_VALUE>	An abstraction of the mapper that would return serialized, and potentially compressed, Avro key/value pairs.
AbstractVeniceRecordReader<INPUT_KEY,INPUT_VALUE>
DefaultHadoopJobClientProvider
DefaultInputDataInfoProvider
DefaultJobClientWrapper
FilterChain<INPUT_VALUE>	The FilterChain class takes a list of AbstractVeniceFilter to assemble a filter chain to manage the life cycles' of filters and perform filtering based on the order of filters.
InputDataInfoProvider.InputDataInfo
MRJobCounterHelper	All MapReducer job counters (identified by a group name + a counter name) are defined here and this util class provides method to increment counters and read counters
NoOpSentPushJobDetailsTracker
PushJobSchemaInfo
PushJobZstdConfig
ValidateSchemaAndBuildDictMapper	Mapper only MR to Validate Schema, Build compression dictionary if needed and persist some data (total file size and compression dictionary) in HDFS to be used by the VPJ Driver Note: processing all the files in this split are done sequentially and if it results in significant increase in the mapper time or resulting in timeouts, this needs to be revisited to be done via a thread pool.
ValidateSchemaAndBuildDictMapperOutputReader	This class reads the data(total input size in bytes and zstd dictionary) persisted in HDFS by ValidateSchemaAndBuildDictMapper based on the schema ValidateSchemaAndBuildDictMapperOutput
ValidateSchemaAndBuildDictOutputFormat	This class provides a way to: 1.
VeniceAvroMapper
VeniceAvroRecordReader
VeniceFileInputFormat	Custom Input Format with the following specs to be used for the feature VenicePushJob.PushJobSetting.useMapperToBuildDict with ValidateSchemaAndBuildDictMapper 1.
VeniceMRPartitioner	VeniceMRPartitioner will use the exact same partitioner: DefaultVenicePartitioner inside VeniceWriter to make sure the messages inside each reducer belong to the same Kafka topic partition.
VenicePushJob	This class sets up the Hadoop job used to push data to Venice.
VenicePushJob.PushJobSetting
VenicePushJob.StoreSetting
VenicePushJob.TopicInfo
VeniceReducer	VeniceReducer will be in charge of producing the messages to Kafka broker.
VeniceReducer.DuplicateKeyPrinter	Using Avro Json encoder to print duplicate keys in case there are tons of duplicate keys, only print first VeniceReducer.DuplicateKeyPrinter.MAX_NUM_OF_LOG of them so that it won't pollute Reducer's log.
VeniceReducer.VeniceWriterMessage
VeniceVsonMapper	Mapper that reads Vson input and deserializes it as Avro object and then Avro binary
VeniceVsonRecordReader
VsonSequenceFileInputFormat	This class was originally from Voldemort.

Enum Summary

Enum	Description
VenicePushJob.PushJobCheckpoints	Different successful checkpoints and known error scenarios of the VPJ flow.