Package com.linkedin.venice.spark.datawriter.task

Class SparkDataWriterTaskTracker

java.lang.Object

com.linkedin.venice.spark.datawriter.task.SparkDataWriterTaskTracker

All Implemented Interfaces:

DataWriterTaskTracker, TaskTracker

public class SparkDataWriterTaskTracker
extends Object
implements DataWriterTaskTracker

Field Summary

Fields inherited from interface com.linkedin.venice.hadoop.task.TaskTracker

PROGRESS_COMPLETED, PROGRESS_NOT_SUPPORTED

Constructor Summary

Constructors

Constructor	Description
SparkDataWriterTaskTracker(DataWriterAccumulators accumulators)

Method Summary

Modifier and Type	Method	Description
long	getDuplicateKeyWithDistinctValueCount()
int	getLargestUncompressedValueSize()	This accumulator performs a 'max' operation, which is not natively supported by either Spark or Hadoop.
long	getOutputRecordsCount()
long	getPartitionWriterCloseCount()
long	getRecordTooLargeFailureCount()
long	getRepushTtlFilterCount()
long	getSprayAllPartitionsCount()
long	getTotalGzipCompressedValueSize()
long	getTotalKeySize()
long	getTotalUncompressedValueSize()
long	getTotalValueSize()
long	getTotalZstdCompressedValueSize()
long	getUncompressedRecordTooLargeFailureCount()
long	getWriteAclAuthorizationFailureCount()
void	trackCompressedValueSize(int size)
void	trackDuplicateKeyWithDistinctValue(int count)
void	trackDuplicateKeyWithIdenticalValue(int count)
void	trackEmptyRecord()
void	trackGzipCompressedValueSize(int size)
void	trackKeySize(int size)
void	trackLargestUncompressedValueSize(int size)	This accumulator performs a 'max' operation, which is not natively supported by either Spark or Hadoop.
void	trackPartitionWriterClose()
void	trackRecordSentToPubSub()
void	trackRecordTooLargeFailure()
void	trackRepushTtlFilteredRecord()
void	trackSprayAllPartitions()
void	trackUncompressedRecordTooLargeFailure()
void	trackUncompressedValueSize(int size)
void	trackWriteAclAuthorizationFailure()
void	trackZstdCompressedValueSize(int size)

Methods inherited from class java.lang.Object

clone, equals, finalize, getClass, hashCode, notify, notifyAll, toString, wait, wait, wait

Methods inherited from interface com.linkedin.venice.hadoop.task.datawriter.DataWriterTaskTracker

getTotalPutOrDeleteRecordsCount, trackPutOrDeleteRecord

Methods inherited from interface com.linkedin.venice.hadoop.task.TaskTracker

getProgress, heartbeat

Constructor Details

SparkDataWriterTaskTracker

public SparkDataWriterTaskTracker(DataWriterAccumulators accumulators)

Method Details

trackSprayAllPartitions

public void trackSprayAllPartitions()

Specified by:

trackSprayAllPartitions in interface DataWriterTaskTracker

trackEmptyRecord

public void trackEmptyRecord()

Specified by:

trackEmptyRecord in interface DataWriterTaskTracker

trackKeySize

public void trackKeySize(int size)

Specified by:

trackKeySize in interface DataWriterTaskTracker

trackUncompressedValueSize

public void trackUncompressedValueSize(int size)

Specified by:

trackUncompressedValueSize in interface DataWriterTaskTracker

trackLargestUncompressedValueSize

public void trackLargestUncompressedValueSize(int size)

Description copied from interface: DataWriterTaskTracker

This accumulator performs a 'max' operation, which is not natively supported by either Spark or Hadoop. It is implemented using a custom accumulator in Spark.

Specified by:

trackLargestUncompressedValueSize in interface DataWriterTaskTracker

trackCompressedValueSize

public void trackCompressedValueSize(int size)

Specified by:

trackCompressedValueSize in interface DataWriterTaskTracker

trackGzipCompressedValueSize

public void trackGzipCompressedValueSize(int size)

Specified by:

trackGzipCompressedValueSize in interface DataWriterTaskTracker

trackZstdCompressedValueSize

public void trackZstdCompressedValueSize(int size)

Specified by:

trackZstdCompressedValueSize in interface DataWriterTaskTracker

trackWriteAclAuthorizationFailure

public void trackWriteAclAuthorizationFailure()

Specified by:

trackWriteAclAuthorizationFailure in interface DataWriterTaskTracker

trackRecordTooLargeFailure

public void trackRecordTooLargeFailure()

Specified by:

trackRecordTooLargeFailure in interface DataWriterTaskTracker

trackUncompressedRecordTooLargeFailure

public void trackUncompressedRecordTooLargeFailure()

Specified by:

trackUncompressedRecordTooLargeFailure in interface DataWriterTaskTracker

trackRecordSentToPubSub

public void trackRecordSentToPubSub()

Specified by:

trackRecordSentToPubSub in interface DataWriterTaskTracker

trackDuplicateKeyWithDistinctValue

public void trackDuplicateKeyWithDistinctValue(int count)

Specified by:

trackDuplicateKeyWithDistinctValue in interface DataWriterTaskTracker

trackDuplicateKeyWithIdenticalValue

public void trackDuplicateKeyWithIdenticalValue(int count)

Specified by:

trackDuplicateKeyWithIdenticalValue in interface DataWriterTaskTracker

trackRepushTtlFilteredRecord

public void trackRepushTtlFilteredRecord()

Specified by:

trackRepushTtlFilteredRecord in interface DataWriterTaskTracker

trackPartitionWriterClose

public void trackPartitionWriterClose()

Specified by:

trackPartitionWriterClose in interface DataWriterTaskTracker

getSprayAllPartitionsCount

public long getSprayAllPartitionsCount()

Specified by:

getSprayAllPartitionsCount in interface DataWriterTaskTracker

getTotalKeySize

public long getTotalKeySize()

Specified by:

getTotalKeySize in interface DataWriterTaskTracker

getTotalValueSize

public long getTotalValueSize()

Specified by:

getTotalValueSize in interface DataWriterTaskTracker

getTotalUncompressedValueSize

public long getTotalUncompressedValueSize()

Specified by:

getTotalUncompressedValueSize in interface DataWriterTaskTracker

getLargestUncompressedValueSize

public int getLargestUncompressedValueSize()

Description copied from interface: DataWriterTaskTracker

This accumulator performs a 'max' operation, which is not natively supported by either Spark or Hadoop. It is implemented using a custom accumulator in Spark.

Specified by:

getLargestUncompressedValueSize in interface DataWriterTaskTracker

getTotalGzipCompressedValueSize

public long getTotalGzipCompressedValueSize()

Specified by:

getTotalGzipCompressedValueSize in interface DataWriterTaskTracker

getTotalZstdCompressedValueSize

public long getTotalZstdCompressedValueSize()

Specified by:

getTotalZstdCompressedValueSize in interface DataWriterTaskTracker

getRecordTooLargeFailureCount

public long getRecordTooLargeFailureCount()

Specified by:

getRecordTooLargeFailureCount in interface DataWriterTaskTracker

getUncompressedRecordTooLargeFailureCount

public long getUncompressedRecordTooLargeFailureCount()

Specified by:

getUncompressedRecordTooLargeFailureCount in interface DataWriterTaskTracker

getWriteAclAuthorizationFailureCount

public long getWriteAclAuthorizationFailureCount()

Specified by:

getWriteAclAuthorizationFailureCount in interface DataWriterTaskTracker

getDuplicateKeyWithDistinctValueCount

public long getDuplicateKeyWithDistinctValueCount()

Specified by:

getDuplicateKeyWithDistinctValueCount in interface DataWriterTaskTracker

getOutputRecordsCount

public long getOutputRecordsCount()

Specified by:

getOutputRecordsCount in interface DataWriterTaskTracker

getPartitionWriterCloseCount

public long getPartitionWriterCloseCount()

Specified by:

getPartitionWriterCloseCount in interface DataWriterTaskTracker

getRepushTtlFilterCount

public long getRepushTtlFilterCount()

Specified by:

getRepushTtlFilterCount in interface DataWriterTaskTracker