Package com.linkedin.venice.hadoop.mapreduce.datawriter.task

Class CounterBackedMapReduceDataWriterTaskTracker

java.lang.Object

com.linkedin.venice.hadoop.mapreduce.datawriter.task.CounterBackedMapReduceDataWriterTaskTracker

All Implemented Interfaces:

DataWriterTaskTracker, TaskTracker

public class CounterBackedMapReduceDataWriterTaskTracker
extends Object
implements DataWriterTaskTracker

An interface to retrieve metrics related to data writer tasks after a MapReduce job has completed.

Field Summary

Fields inherited from interface com.linkedin.venice.hadoop.task.TaskTracker

PROGRESS_COMPLETED, PROGRESS_NOT_SUPPORTED

Constructor Summary

Constructors

Constructor	Description
CounterBackedMapReduceDataWriterTaskTracker(org.apache.hadoop.mapred.Counters counters)

Method Summary

Modifier and Type	Method	Description
long	getDuplicateKeyWithDistinctValueCount()
long	getExternalStorageWriteTimeMs()
Set<String>	getFailedExternalStorageRegions()	Returns the set of regions whose external writers exhausted retries and were reported by one or more data-writer tasks.
long	getIncrementalPushThrottledTimeMs()
long	getOutputRecordsCount()
long	getPartitionWriterCloseCount()
long	getRecordTooLargeFailureCount()
long	getRepushTtlFilterCount()
long	getSprayAllPartitionsCount()
long	getTotalGzipCompressedValueSize()
long	getTotalKeySize()
long	getTotalPutOrDeleteRecordsCount()
long	getTotalUncompressedValueSize()
long	getTotalValueSize()
long	getTotalZstdCompressedValueSize()
long	getUncompressedRecordTooLargeFailureCount()
long	getVeniceWriteTimeMs()
long	getWriteAclAuthorizationFailureCount()

Methods inherited from class java.lang.Object

clone, equals, finalize, getClass, hashCode, notify, notifyAll, toString, wait, wait, wait

Methods inherited from interface com.linkedin.venice.hadoop.task.datawriter.DataWriterTaskTracker

getLargestUncompressedValueSize, getPerPartitionRecordCounts, trackCompressedValueSize, trackDuplicateKeyWithDistinctValue, trackDuplicateKeyWithIdenticalValue, trackEmptyRecord, trackExternalStorageWriteTime, trackFailedExternalStorageRegion, trackGzipCompressedValueSize, trackIncrementalPushThrottledTime, trackKeySize, trackLargestUncompressedValueSize, trackPartitionWriterClose, trackPutOrDeleteRecord, trackRecordSentToPubSub, trackRecordTooLargeFailure, trackRepushTtlFilteredRecord, trackSprayAllPartitions, trackUncompressedRecordTooLargeFailure, trackUncompressedValueSize, trackVeniceWriteTime, trackWriteAclAuthorizationFailure, trackZstdCompressedValueSize

Methods inherited from interface com.linkedin.venice.hadoop.task.TaskTracker

getProgress, heartbeat

Constructor Details

CounterBackedMapReduceDataWriterTaskTracker

public CounterBackedMapReduceDataWriterTaskTracker(org.apache.hadoop.mapred.Counters counters)

Method Details

getSprayAllPartitionsCount

public long getSprayAllPartitionsCount()

Specified by:

getSprayAllPartitionsCount in interface DataWriterTaskTracker

getTotalKeySize

public long getTotalKeySize()

Specified by:

getTotalKeySize in interface DataWriterTaskTracker

getTotalValueSize

public long getTotalValueSize()

Specified by:

getTotalValueSize in interface DataWriterTaskTracker

getTotalUncompressedValueSize

public long getTotalUncompressedValueSize()

Specified by:

getTotalUncompressedValueSize in interface DataWriterTaskTracker

getTotalGzipCompressedValueSize

public long getTotalGzipCompressedValueSize()

Specified by:

getTotalGzipCompressedValueSize in interface DataWriterTaskTracker

getTotalZstdCompressedValueSize

public long getTotalZstdCompressedValueSize()

Specified by:

getTotalZstdCompressedValueSize in interface DataWriterTaskTracker

getRecordTooLargeFailureCount

public long getRecordTooLargeFailureCount()

Specified by:

getRecordTooLargeFailureCount in interface DataWriterTaskTracker

getUncompressedRecordTooLargeFailureCount

public long getUncompressedRecordTooLargeFailureCount()

Specified by:

getUncompressedRecordTooLargeFailureCount in interface DataWriterTaskTracker

getWriteAclAuthorizationFailureCount

public long getWriteAclAuthorizationFailureCount()

Specified by:

getWriteAclAuthorizationFailureCount in interface DataWriterTaskTracker

getDuplicateKeyWithDistinctValueCount

public long getDuplicateKeyWithDistinctValueCount()

Specified by:

getDuplicateKeyWithDistinctValueCount in interface DataWriterTaskTracker

getOutputRecordsCount

public long getOutputRecordsCount()

Specified by:

getOutputRecordsCount in interface DataWriterTaskTracker

getPartitionWriterCloseCount

public long getPartitionWriterCloseCount()

Specified by:

getPartitionWriterCloseCount in interface DataWriterTaskTracker

getRepushTtlFilterCount

public long getRepushTtlFilterCount()

Specified by:

getRepushTtlFilterCount in interface DataWriterTaskTracker

getTotalPutOrDeleteRecordsCount

public long getTotalPutOrDeleteRecordsCount()

Specified by:

getTotalPutOrDeleteRecordsCount in interface DataWriterTaskTracker

getIncrementalPushThrottledTimeMs

public long getIncrementalPushThrottledTimeMs()

Specified by:

getIncrementalPushThrottledTimeMs in interface DataWriterTaskTracker

getFailedExternalStorageRegions

public Set<String> getFailedExternalStorageRegions()

Description copied from interface: DataWriterTaskTracker

Returns the set of regions whose external writers exhausted retries and were reported by one or more data-writer tasks. Implementations should return an immutable or defensive-copy snapshot.

Specified by:

getFailedExternalStorageRegions in interface DataWriterTaskTracker

getExternalStorageWriteTimeMs

public long getExternalStorageWriteTimeMs()

Specified by:

getExternalStorageWriteTimeMs in interface DataWriterTaskTracker

Returns:

the summed per-task duration (ms) spent in the external-storage write path across all successful task outputs of this push. See DataWriterTaskTracker.trackExternalStorageWriteTime(long) for exactly what is included; this is not the push's wall-clock time.

getVeniceWriteTimeMs

public long getVeniceWriteTimeMs()

Specified by:

getVeniceWriteTimeMs in interface DataWriterTaskTracker

Returns:

the summed per-task duration (ms) spent in the Venice/Kafka write path across all successful task outputs of this push. See DataWriterTaskTracker.trackVeniceWriteTime(long); this is not the push's wall-clock time.