Package com.linkedin.venice.hadoop.task.datawriter

Interface DataWriterTaskTracker

All Superinterfaces:
TaskTracker
All Known Implementing Classes:
CounterBackedMapReduceDataWriterTaskTracker, ReporterBackedMapReduceDataWriterTaskTracker, SparkDataWriterTaskTracker
public interface DataWriterTaskTracker extends TaskTracker
An interface to report and retrieve metrics related to data writer tasks.

  • Method Details

    • trackSprayAllPartitions

      default void trackSprayAllPartitions()

    • trackEmptyRecord

      default void trackEmptyRecord()

    • trackKeySize

      default void trackKeySize(int size)

    • trackUncompressedValueSize

      default void trackUncompressedValueSize(int size)

    • trackLargestUncompressedValueSize

      default void trackLargestUncompressedValueSize(int size)
      This accumulator performs a 'max' operation, which is not natively supported by either Spark or Hadoop. It is implemented using a custom accumulator in Spark.

    • trackCompressedValueSize

      default void trackCompressedValueSize(int size)

    • trackGzipCompressedValueSize

      default void trackGzipCompressedValueSize(int size)

    • trackZstdCompressedValueSize

      default void trackZstdCompressedValueSize(int size)

    • trackWriteAclAuthorizationFailure

      default void trackWriteAclAuthorizationFailure()

    • trackRecordTooLargeFailure

      default void trackRecordTooLargeFailure()

    • trackUncompressedRecordTooLargeFailure

      default void trackUncompressedRecordTooLargeFailure()

    • trackRecordSentToPubSub

      default void trackRecordSentToPubSub()

    • trackDuplicateKeyWithDistinctValue

      default void trackDuplicateKeyWithDistinctValue(int count)

    • trackDuplicateKeyWithIdenticalValue

      default void trackDuplicateKeyWithIdenticalValue(int count)

    • trackRepushTtlFilteredRecord

      default void trackRepushTtlFilteredRecord()

    • trackPutOrDeleteRecord

      default void trackPutOrDeleteRecord()

    • trackPartitionWriterClose

      default void trackPartitionWriterClose()

    • trackIncrementalPushThrottledTime

      default void trackIncrementalPushThrottledTime(long timeMs)

    • trackFailedExternalStorageRegion

      default void trackFailedExternalStorageRegion(String regionName)
      Report that the external writer for regionName exhausted its retry budget and was disabled for the remainder of the task. Callers should report each region at most once per task.

    • getSprayAllPartitionsCount

      default long getSprayAllPartitionsCount()

    • getTotalKeySize

      default long getTotalKeySize()

    • getTotalValueSize

      default long getTotalValueSize()

    • getTotalUncompressedValueSize

      default long getTotalUncompressedValueSize()

    • getLargestUncompressedValueSize

      default int getLargestUncompressedValueSize()
      This accumulator performs a 'max' operation, which is not natively supported by either Spark or Hadoop. It is implemented using a custom accumulator in Spark.

    • getTotalGzipCompressedValueSize

      default long getTotalGzipCompressedValueSize()

    • getTotalZstdCompressedValueSize

      default long getTotalZstdCompressedValueSize()

    • getRecordTooLargeFailureCount

      default long getRecordTooLargeFailureCount()

    • getUncompressedRecordTooLargeFailureCount

      default long getUncompressedRecordTooLargeFailureCount()

    • getWriteAclAuthorizationFailureCount

      default long getWriteAclAuthorizationFailureCount()

    • getDuplicateKeyWithDistinctValueCount

      default long getDuplicateKeyWithDistinctValueCount()

    • getOutputRecordsCount

      default long getOutputRecordsCount()

    • getPartitionWriterCloseCount

      default long getPartitionWriterCloseCount()

    • getRepushTtlFilterCount

      default long getRepushTtlFilterCount()

    • getTotalPutOrDeleteRecordsCount

      default long getTotalPutOrDeleteRecordsCount()

    • getIncrementalPushThrottledTimeMs

      default long getIncrementalPushThrottledTimeMs()

    • getPerPartitionRecordCounts

      default Map<Integer,Long> getPerPartitionRecordCounts()
      Returns per-partition record counts collected during the data writer job. For the Spark path, these are collected via collect() on the DAG output
      Returns:
      Map of partition ID to record count, or empty map if not available.

    • getFailedExternalStorageRegions

      default Set<String> getFailedExternalStorageRegions()
      Returns the set of regions whose external writers exhausted retries and were reported by one or more data-writer tasks. Implementations should return an immutable or defensive-copy snapshot.