Package com.linkedin.venice.hadoop.task.datawriter

Interface DataWriterTaskTracker

All Superinterfaces:
TaskTracker
All Known Implementing Classes:
CounterBackedMapReduceDataWriterTaskTracker, ReporterBackedMapReduceDataWriterTaskTracker, SparkDataWriterTaskTracker
public interface DataWriterTaskTracker extends TaskTracker
An interface to report and retrieve metrics related to data writer tasks.

  • Method Details

    • trackSprayAllPartitions

      default void trackSprayAllPartitions()

    • trackEmptyRecord

      default void trackEmptyRecord()

    • trackKeySize

      default void trackKeySize(int size)

    • trackUncompressedValueSize

      default void trackUncompressedValueSize(int size)

    • trackLargestUncompressedValueSize

      default void trackLargestUncompressedValueSize(int size)
      This accumulator performs a 'max' operation, which is not natively supported by either Spark or Hadoop. It is implemented using a custom accumulator in Spark.

    • trackCompressedValueSize

      default void trackCompressedValueSize(int size)

    • trackGzipCompressedValueSize

      default void trackGzipCompressedValueSize(int size)

    • trackZstdCompressedValueSize

      default void trackZstdCompressedValueSize(int size)

    • trackWriteAclAuthorizationFailure

      default void trackWriteAclAuthorizationFailure()

    • trackRecordTooLargeFailure

      default void trackRecordTooLargeFailure()

    • trackUncompressedRecordTooLargeFailure

      default void trackUncompressedRecordTooLargeFailure()

    • trackRecordSentToPubSub

      default void trackRecordSentToPubSub()

    • trackDuplicateKeyWithDistinctValue

      default void trackDuplicateKeyWithDistinctValue(int count)

    • trackDuplicateKeyWithIdenticalValue

      default void trackDuplicateKeyWithIdenticalValue(int count)

    • trackRepushTtlFilteredRecord

      default void trackRepushTtlFilteredRecord()

    • trackPutOrDeleteRecord

      default void trackPutOrDeleteRecord()

    • trackPartitionWriterClose

      default void trackPartitionWriterClose()

    • trackIncrementalPushThrottledTime

      default void trackIncrementalPushThrottledTime(long timeMs)

    • trackExternalStorageWriteTime

      default void trackExternalStorageWriteTime(long timeMs)
      Report elapsed wall-clock time this task spent inside the external-storage write path. This covers the complete external leg: the per-region write throttling wait, the ExternalStorageWriter.batchPut calls including their retries and retry backoff sleeps, the external flush, and the external close. It excludes anything spent producing to Venice/Kafka.

      Callers report monotonic deltas as they accrue, so the aggregate is the sum of per-task durations, not the push's wall-clock duration: with N concurrent data-writer tasks the total can be up to N times the push's elapsed time.

    • trackVeniceWriteTime

      default void trackVeniceWriteTime(long timeMs)
      Report elapsed wall-clock time this task spent invoking the Venice/Kafka write path: the VeniceWriter.put invocations plus flushing and closing the Venice writer. It excludes anything spent in the external-storage path.

      Same aggregation semantics as trackExternalStorageWriteTime(long): summed task durations, not push wall-clock time.

    • trackFailedExternalStorageRegion

      default void trackFailedExternalStorageRegion(String regionName)
      Report that the external writer for regionName exhausted its retry budget and was disabled for the remainder of the task. Callers should report each region at most once per task.

    • getSprayAllPartitionsCount

      default long getSprayAllPartitionsCount()

    • getTotalKeySize

      default long getTotalKeySize()

    • getTotalValueSize

      default long getTotalValueSize()

    • getTotalUncompressedValueSize

      default long getTotalUncompressedValueSize()

    • getLargestUncompressedValueSize

      default int getLargestUncompressedValueSize()
      This accumulator performs a 'max' operation, which is not natively supported by either Spark or Hadoop. It is implemented using a custom accumulator in Spark.

    • getTotalGzipCompressedValueSize

      default long getTotalGzipCompressedValueSize()

    • getTotalZstdCompressedValueSize

      default long getTotalZstdCompressedValueSize()

    • getRecordTooLargeFailureCount

      default long getRecordTooLargeFailureCount()

    • getUncompressedRecordTooLargeFailureCount

      default long getUncompressedRecordTooLargeFailureCount()

    • getWriteAclAuthorizationFailureCount

      default long getWriteAclAuthorizationFailureCount()

    • getDuplicateKeyWithDistinctValueCount

      default long getDuplicateKeyWithDistinctValueCount()

    • getOutputRecordsCount

      default long getOutputRecordsCount()

    • getPartitionWriterCloseCount

      default long getPartitionWriterCloseCount()

    • getRepushTtlFilterCount

      default long getRepushTtlFilterCount()

    • getTotalPutOrDeleteRecordsCount

      default long getTotalPutOrDeleteRecordsCount()

    • getIncrementalPushThrottledTimeMs

      default long getIncrementalPushThrottledTimeMs()

    • getExternalStorageWriteTimeMs

      default long getExternalStorageWriteTimeMs()
      Returns:
      the summed per-task duration (ms) spent in the external-storage write path across all successful task outputs of this push. See trackExternalStorageWriteTime(long) for exactly what is included; this is not the push's wall-clock time.

    • getVeniceWriteTimeMs

      default long getVeniceWriteTimeMs()
      Returns:
      the summed per-task duration (ms) spent in the Venice/Kafka write path across all successful task outputs of this push. See trackVeniceWriteTime(long); this is not the push's wall-clock time.

    • getPerPartitionRecordCounts

      default Map<Integer,Long> getPerPartitionRecordCounts()
      Returns per-partition record counts collected during the data writer job. For the Spark path, these are collected via collect() on the DAG output
      Returns:
      Map of partition ID to record count, or empty map if not available.

    • getFailedExternalStorageRegions

      default Set<String> getFailedExternalStorageRegions()
      Returns the set of regions whose external writers exhausted retries and were reported by one or more data-writer tasks. Implementations should return an immutable or defensive-copy snapshot.