Package com.linkedin.venice.hadoop.mapreduce.datawriter.task

Class ReporterBackedMapReduceDataWriterTaskTracker

java.lang.Object

com.linkedin.venice.hadoop.mapreduce.datawriter.task.ReporterBackedMapReduceDataWriterTaskTracker

All Implemented Interfaces:

DataWriterTaskTracker, TaskTracker

public class ReporterBackedMapReduceDataWriterTaskTracker
extends Object
implements DataWriterTaskTracker

An interface to report and retrieve metrics related to data writer tasks during the execution of a MapReduce job.

Field Summary

Fields inherited from interface com.linkedin.venice.hadoop.task.TaskTracker

PROGRESS_COMPLETED, PROGRESS_NOT_SUPPORTED

Constructor Summary

Constructors

Constructor	Description
ReporterBackedMapReduceDataWriterTaskTracker(org.apache.hadoop.mapred.Reporter reporter)

Method Summary

Modifier and Type	Method	Description
long	getDuplicateKeyWithDistinctValueCount()
long	getExternalStorageWriteTimeMs()
Set<String>	getFailedExternalStorageRegions()	Returns the set of regions whose external writers exhausted retries and were reported by one or more data-writer tasks.
long	getIncrementalPushThrottledTimeMs()
float	getProgress()	Get the progress of the task.
long	getRecordTooLargeFailureCount()
org.apache.hadoop.mapred.Reporter	getReporter()
long	getTotalKeySize()
long	getTotalPutOrDeleteRecordsCount()
long	getTotalValueSize()
long	getUncompressedRecordTooLargeFailureCount()
long	getVeniceWriteTimeMs()
long	getWriteAclAuthorizationFailureCount()
void	heartbeat()	Report task heartbeat if the execution engine requires it
void	trackCompressedValueSize(int size)
void	trackDuplicateKeyWithDistinctValue(int count)
void	trackDuplicateKeyWithIdenticalValue(int count)
void	trackEmptyRecord()
void	trackExternalStorageWriteTime(long timeMs)	Report elapsed wall-clock time this task spent inside the external-storage write path.
void	trackFailedExternalStorageRegion(String regionName)	Report that the external writer for regionName exhausted its retry budget and was disabled for the remainder of the task.
void	trackGzipCompressedValueSize(int size)
void	trackIncrementalPushThrottledTime(long timeMs)
void	trackKeySize(int size)
void	trackPartitionWriterClose()
void	trackPutOrDeleteRecord()
void	trackRecordSentToPubSub()
void	trackRecordTooLargeFailure()
void	trackRepushTtlFilteredRecord()
void	trackSprayAllPartitions()
void	trackUncompressedRecordTooLargeFailure()
void	trackUncompressedValueSize(int size)
void	trackVeniceWriteTime(long timeMs)	Report elapsed wall-clock time this task spent invoking the Venice/Kafka write path: the VeniceWriter.put invocations plus flushing and closing the Venice writer.
void	trackWriteAclAuthorizationFailure()
void	trackZstdCompressedValueSize(int size)

Methods inherited from class java.lang.Object

clone, equals, finalize, getClass, hashCode, notify, notifyAll, toString, wait, wait, wait

Methods inherited from interface com.linkedin.venice.hadoop.task.datawriter.DataWriterTaskTracker

getLargestUncompressedValueSize, getOutputRecordsCount, getPartitionWriterCloseCount, getPerPartitionRecordCounts, getRepushTtlFilterCount, getSprayAllPartitionsCount, getTotalGzipCompressedValueSize, getTotalUncompressedValueSize, getTotalZstdCompressedValueSize, trackLargestUncompressedValueSize

Constructor Details

ReporterBackedMapReduceDataWriterTaskTracker

public ReporterBackedMapReduceDataWriterTaskTracker(org.apache.hadoop.mapred.Reporter reporter)

Method Details

getReporter

public org.apache.hadoop.mapred.Reporter getReporter()

heartbeat

public void heartbeat()

Description copied from interface: TaskTracker

Report task heartbeat if the execution engine requires it

Specified by:

heartbeat in interface TaskTracker

getProgress

public float getProgress()

Description copied from interface: TaskTracker

Get the progress of the task. If the engine supports fetching the progress, progress is represented as a number between 0 and 1 (inclusive). If the engine doesn't support fetching the progress, a value of -1 is returned.

Specified by:

getProgress in interface TaskTracker

trackSprayAllPartitions

public void trackSprayAllPartitions()

Specified by:

trackSprayAllPartitions in interface DataWriterTaskTracker

trackEmptyRecord

public void trackEmptyRecord()

Specified by:

trackEmptyRecord in interface DataWriterTaskTracker

trackKeySize

public void trackKeySize(int size)

Specified by:

trackKeySize in interface DataWriterTaskTracker

trackUncompressedValueSize

public void trackUncompressedValueSize(int size)

Specified by:

trackUncompressedValueSize in interface DataWriterTaskTracker

trackCompressedValueSize

public void trackCompressedValueSize(int size)

Specified by:

trackCompressedValueSize in interface DataWriterTaskTracker

trackGzipCompressedValueSize

public void trackGzipCompressedValueSize(int size)

Specified by:

trackGzipCompressedValueSize in interface DataWriterTaskTracker

trackZstdCompressedValueSize

public void trackZstdCompressedValueSize(int size)

Specified by:

trackZstdCompressedValueSize in interface DataWriterTaskTracker

trackWriteAclAuthorizationFailure

public void trackWriteAclAuthorizationFailure()

Specified by:

trackWriteAclAuthorizationFailure in interface DataWriterTaskTracker

trackRecordTooLargeFailure

public void trackRecordTooLargeFailure()

Specified by:

trackRecordTooLargeFailure in interface DataWriterTaskTracker

trackUncompressedRecordTooLargeFailure

public void trackUncompressedRecordTooLargeFailure()

Specified by:

trackUncompressedRecordTooLargeFailure in interface DataWriterTaskTracker

trackRecordSentToPubSub

public void trackRecordSentToPubSub()

Specified by:

trackRecordSentToPubSub in interface DataWriterTaskTracker

trackDuplicateKeyWithDistinctValue

public void trackDuplicateKeyWithDistinctValue(int count)

Specified by:

trackDuplicateKeyWithDistinctValue in interface DataWriterTaskTracker

trackDuplicateKeyWithIdenticalValue

public void trackDuplicateKeyWithIdenticalValue(int count)

Specified by:

trackDuplicateKeyWithIdenticalValue in interface DataWriterTaskTracker

trackRepushTtlFilteredRecord

public void trackRepushTtlFilteredRecord()

Specified by:

trackRepushTtlFilteredRecord in interface DataWriterTaskTracker

trackPartitionWriterClose

public void trackPartitionWriterClose()

Specified by:

trackPartitionWriterClose in interface DataWriterTaskTracker

trackPutOrDeleteRecord

public void trackPutOrDeleteRecord()

Specified by:

trackPutOrDeleteRecord in interface DataWriterTaskTracker

trackIncrementalPushThrottledTime

public void trackIncrementalPushThrottledTime(long timeMs)

Specified by:

trackIncrementalPushThrottledTime in interface DataWriterTaskTracker

trackFailedExternalStorageRegion

public void trackFailedExternalStorageRegion(String regionName)

Description copied from interface: DataWriterTaskTracker

Report that the external writer for regionName exhausted its retry budget and was disabled for the remainder of the task. Callers should report each region at most once per task.

Specified by:

trackFailedExternalStorageRegion in interface DataWriterTaskTracker

getIncrementalPushThrottledTimeMs

public long getIncrementalPushThrottledTimeMs()

Specified by:

getIncrementalPushThrottledTimeMs in interface DataWriterTaskTracker

trackExternalStorageWriteTime

public void trackExternalStorageWriteTime(long timeMs)

Description copied from interface: DataWriterTaskTracker

Report elapsed wall-clock time this task spent inside the external-storage write path. This covers the complete external leg: the per-region write throttling wait, the ExternalStorageWriter.batchPut calls including their retries and retry backoff sleeps, the external flush, and the external close. It excludes anything spent producing to Venice/Kafka.

Callers report monotonic deltas as they accrue, so the aggregate is the sum of per-task durations, not the push's wall-clock duration: with N concurrent data-writer tasks the total can be up to N times the push's elapsed time.

Specified by:

trackExternalStorageWriteTime in interface DataWriterTaskTracker

trackVeniceWriteTime

public void trackVeniceWriteTime(long timeMs)

Description copied from interface: DataWriterTaskTracker

Report elapsed wall-clock time this task spent invoking the Venice/Kafka write path: the VeniceWriter.put invocations plus flushing and closing the Venice writer. It excludes anything spent in the external-storage path.

Same aggregation semantics as DataWriterTaskTracker.trackExternalStorageWriteTime(long): summed task durations, not push wall-clock time.

Specified by:

trackVeniceWriteTime in interface DataWriterTaskTracker

getExternalStorageWriteTimeMs

public long getExternalStorageWriteTimeMs()

Specified by:

getExternalStorageWriteTimeMs in interface DataWriterTaskTracker

Returns:

the summed per-task duration (ms) spent in the external-storage write path across all successful task outputs of this push. See DataWriterTaskTracker.trackExternalStorageWriteTime(long) for exactly what is included; this is not the push's wall-clock time.

getVeniceWriteTimeMs

public long getVeniceWriteTimeMs()

Specified by:

getVeniceWriteTimeMs in interface DataWriterTaskTracker

Returns:

the summed per-task duration (ms) spent in the Venice/Kafka write path across all successful task outputs of this push. See DataWriterTaskTracker.trackVeniceWriteTime(long); this is not the push's wall-clock time.

getTotalKeySize

public long getTotalKeySize()

Specified by:

getTotalKeySize in interface DataWriterTaskTracker

getTotalValueSize

public long getTotalValueSize()

Specified by:

getTotalValueSize in interface DataWriterTaskTracker

getRecordTooLargeFailureCount

public long getRecordTooLargeFailureCount()

Specified by:

getRecordTooLargeFailureCount in interface DataWriterTaskTracker

getUncompressedRecordTooLargeFailureCount

public long getUncompressedRecordTooLargeFailureCount()

Specified by:

getUncompressedRecordTooLargeFailureCount in interface DataWriterTaskTracker

getWriteAclAuthorizationFailureCount

public long getWriteAclAuthorizationFailureCount()

Specified by:

getWriteAclAuthorizationFailureCount in interface DataWriterTaskTracker

getDuplicateKeyWithDistinctValueCount

public long getDuplicateKeyWithDistinctValueCount()

Specified by:

getDuplicateKeyWithDistinctValueCount in interface DataWriterTaskTracker

getTotalPutOrDeleteRecordsCount

public long getTotalPutOrDeleteRecordsCount()

Specified by:

getTotalPutOrDeleteRecordsCount in interface DataWriterTaskTracker

getFailedExternalStorageRegions

public Set<String> getFailedExternalStorageRegions()

Description copied from interface: DataWriterTaskTracker

Returns the set of regions whose external writers exhausted retries and were reported by one or more data-writer tasks. Implementations should return an immutable or defensive-copy snapshot.

Specified by:

getFailedExternalStorageRegions in interface DataWriterTaskTracker