Package com.linkedin.venice.spark.datawriter.task

Class DataWriterAccumulators

java.lang.Object
com.linkedin.venice.spark.datawriter.task.DataWriterAccumulators
All Implemented Interfaces:
Serializable
public class DataWriterAccumulators extends Object implements Serializable
All the AccumulatorV2 objects that are used in the Spark DataWriter jobs.
See Also:

  • Field Details

    • sprayAllPartitionsTriggeredCount

      public final org.apache.spark.util.LongAccumulator sprayAllPartitionsTriggeredCount

    • emptyRecordCounter

      public final org.apache.spark.util.LongAccumulator emptyRecordCounter

    • totalKeySizeCounter

      public final org.apache.spark.util.LongAccumulator totalKeySizeCounter

    • uncompressedValueSizeCounter

      public final org.apache.spark.util.LongAccumulator uncompressedValueSizeCounter

    • largestUncompressedValueSize

      public final MaxAccumulator<Integer> largestUncompressedValueSize

    • compressedValueSizeCounter

      public final org.apache.spark.util.LongAccumulator compressedValueSizeCounter

    • gzipCompressedValueSizeCounter

      public final org.apache.spark.util.LongAccumulator gzipCompressedValueSizeCounter

    • zstdCompressedValueSizeCounter

      public final org.apache.spark.util.LongAccumulator zstdCompressedValueSizeCounter

    • outputRecordCounter

      public final org.apache.spark.util.LongAccumulator outputRecordCounter

    • duplicateKeyWithIdenticalValueCounter

      public final org.apache.spark.util.LongAccumulator duplicateKeyWithIdenticalValueCounter

    • writeAclAuthorizationFailureCounter

      public final org.apache.spark.util.LongAccumulator writeAclAuthorizationFailureCounter

    • recordTooLargeFailureCounter

      public final org.apache.spark.util.LongAccumulator recordTooLargeFailureCounter

    • uncompressedRecordTooLargeFailureCounter

      public final org.apache.spark.util.LongAccumulator uncompressedRecordTooLargeFailureCounter

    • duplicateKeyWithDistinctValueCounter

      public final org.apache.spark.util.LongAccumulator duplicateKeyWithDistinctValueCounter

    • partitionWriterCloseCounter

      public final org.apache.spark.util.LongAccumulator partitionWriterCloseCounter

    • repushTtlFilteredRecordCounter

      public final org.apache.spark.util.LongAccumulator repushTtlFilteredRecordCounter

  • Constructor Details

    • DataWriterAccumulators

      public DataWriterAccumulators(org.apache.spark.sql.SparkSession session)