Package com.linkedin.venice.hadoop.task.datawriter
-
Interface Summary Interface Description DataWriterTaskTracker An interface to report and retrieve metrics related to data writer tasks. -
Class Summary Class Description AbstractDataWriterTask Class for commonalities betweenAbstractInputRecordProcessor
,AbstractPartitionWriter
,KafkaInputFormatCombiner
andValidateSchemaAndBuildDictMapper
.AbstractInputRecordProcessor<INPUT_KEY,INPUT_VALUE> An abstraction of the task that processes each record from the input, and returns serialized, and potentially compressed, Avro key/value pairs.AbstractPartitionWriter An abstraction of the task that processes all key/value pairs, checks for duplicates and emits the final key/value pairs to Venice's PubSub.AbstractPartitionWriter.DuplicateKeyPrinter Using Avro Json encoder to print duplicate keys in case there are tons of duplicate keys, only print firstAbstractPartitionWriter.DuplicateKeyPrinter.MAX_NUM_OF_LOG
of them so that it won't pollute Reducer's log.AbstractPartitionWriter.VeniceWriterMessage