Package com.linkedin.venice.hadoop.task.datawriter

Class AbstractPartitionWriter.DuplicateKeyPrinter

java.lang.Object

com.linkedin.venice.hadoop.task.datawriter.AbstractPartitionWriter.DuplicateKeyPrinter

All Implemented Interfaces:

java.io.Closeable, java.lang.AutoCloseable

Enclosing class:

AbstractPartitionWriter

public static class AbstractPartitionWriter.DuplicateKeyPrinter
extends java.lang.Object
implements java.lang.AutoCloseable, java.io.Closeable

Using Avro Json encoder to print duplicate keys in case there are tons of duplicate keys, only print first MAX_NUM_OF_LOG of them so that it won't pollute Reducer's log. N.B. We assume that this is an Avro record here. (Vson is considered as Avro as well from Reducer's perspective) We should update this method once Venice supports other format in the future

Method Summary

Modifier and Type	Method	Description
void	close()
protected void	detectAndHandleDuplicateKeys(byte[] keyBytes, byte[] valueBytes, java.util.Iterator<byte[]> values, DataWriterTaskTracker dataWriterTaskTracker)

Methods inherited from class java.lang.Object

clone, equals, finalize, getClass, hashCode, notify, notifyAll, toString, wait, wait, wait

Method Detail

detectAndHandleDuplicateKeys

protected void detectAndHandleDuplicateKeys(byte[] keyBytes,
                                            byte[] valueBytes,
                                            java.util.Iterator<byte[]> values,
                                            DataWriterTaskTracker dataWriterTaskTracker)

close

public void close()

Specified by:

close in interface java.lang.AutoCloseable

Specified by:

close in interface java.io.Closeable