Package com.linkedin.davinci.stats

Class IngestionStats

java.lang.Object
com.linkedin.davinci.stats.IngestionStats
public class IngestionStats extends Object
This class contains stats for store ingestion. The stat class is used in VeniceVersionedStats to serve for a single store version or total of all store versions. This class does not contain reporting logic as reporting is done by the IngestionStatsReporter.

  • Field Details

    • INGESTION_TASK_ERROR_GAUGE

      protected static final String INGESTION_TASK_ERROR_GAUGE
      See Also:

    • INGESTION_TASK_PUSH_TIMEOUT_GAUGE

      protected static final String INGESTION_TASK_PUSH_TIMEOUT_GAUGE
      See Also:

    • WRITE_COMPUTE_OPERATION_FAILURE

      protected static final String WRITE_COMPUTE_OPERATION_FAILURE
      See Also:

    • RECORDS_CONSUMED_METRIC_NAME

      protected static final String RECORDS_CONSUMED_METRIC_NAME
      See Also:

    • BYTES_CONSUMED_METRIC_NAME

      protected static final String BYTES_CONSUMED_METRIC_NAME
      See Also:

    • LEADER_RECORDS_CONSUMED_METRIC_NAME

      protected static final String LEADER_RECORDS_CONSUMED_METRIC_NAME
      See Also:

    • LEADER_BYTES_CONSUMED_METRIC_NAME

      protected static final String LEADER_BYTES_CONSUMED_METRIC_NAME
      See Also:

    • FOLLOWER_RECORDS_CONSUMED_METRIC_NAME

      protected static final String FOLLOWER_RECORDS_CONSUMED_METRIC_NAME
      See Also:

    • FOLLOWER_BYTES_CONSUMED_METRIC_NAME

      protected static final String FOLLOWER_BYTES_CONSUMED_METRIC_NAME
      See Also:

    • LEADER_RECORDS_PRODUCED_METRIC_NAME

      protected static final String LEADER_RECORDS_PRODUCED_METRIC_NAME
      See Also:

    • LEADER_BYTES_PRODUCED_METRIC_NAME

      protected static final String LEADER_BYTES_PRODUCED_METRIC_NAME
      See Also:

    • SUBSCRIBE_ACTION_PREP_LATENCY

      protected static final String SUBSCRIBE_ACTION_PREP_LATENCY
      See Also:

    • UPDATE_IGNORED_DCR

      protected static final String UPDATE_IGNORED_DCR
      See Also:

    • TOTAL_DCR

      protected static final String TOTAL_DCR
      See Also:

    • TOTAL_DUPLICATE_KEY_UPDATE_COUNT

      protected static final String TOTAL_DUPLICATE_KEY_UPDATE_COUNT
      See Also:

    • TIMESTAMP_REGRESSION_DCR_ERROR

      protected static final String TIMESTAMP_REGRESSION_DCR_ERROR
      See Also:

    • OFFSET_REGRESSION_DCR_ERROR

      protected static final String OFFSET_REGRESSION_DCR_ERROR
      See Also:

    • TOMBSTONE_CREATION_DCR

      protected static final String TOMBSTONE_CREATION_DCR
      See Also:

    • CONSUMED_RECORD_END_TO_END_PROCESSING_LATENCY

      protected static final String CONSUMED_RECORD_END_TO_END_PROCESSING_LATENCY
      Consumer metric: Measures the total time from when a record starts being processed (after polling from Kafka and schema checks) until it completes all processing stages. This includes leader preprocessing, producing to local Kafka (for leaders), queueing to drainer, drainer processing (persisting to storage), and offset updates.
      See Also:

    • NEARLINE_PRODUCER_TO_LOCAL_BROKER_LATENCY

      public static final String NEARLINE_PRODUCER_TO_LOCAL_BROKER_LATENCY
      Leader metric: Measures the latency from when a nearline producer originally produced a message (with its producer timestamp) to when that message is successfully written to the local broker's version topic by the leader.
      See Also:

    • PRODUCER_TO_SOURCE_BROKER_LATENCY

      public static final String PRODUCER_TO_SOURCE_BROKER_LATENCY
      Leader metric: Measures the latency from when a producer created a message (producer timestamp) to when the source broker (remote region Kafka) received it.
      See Also:

    • SOURCE_BROKER_TO_LEADER_CONSUMER_LATENCY

      public static final String SOURCE_BROKER_TO_LEADER_CONSUMER_LATENCY
      Leader metric: Measures the latency from when the source broker (remote region Kafka) received a message to when the leader consumer fetched it.
      See Also:

    • PRODUCER_TO_LOCAL_BROKER_LATENCY

      public static final String PRODUCER_TO_LOCAL_BROKER_LATENCY
      Follower metric: Measures the latency from when the leader produced a message to when the local broker (local region Kafka) received it.
      See Also:

    • LOCAL_BROKER_TO_FOLLOWER_CONSUMER_LATENCY

      public static final String LOCAL_BROKER_TO_FOLLOWER_CONSUMER_LATENCY
      Follower metric: Measures the latency from when the local broker (local region Kafka) received a message to when the follower consumer fetched it.
      See Also:

    • LEADER_PRODUCER_COMPLETION_LATENCY

      public static final String LEADER_PRODUCER_COMPLETION_LATENCY
      Leader metric: Measures the time from when a produce call is made to when the producer callback is invoked, indicating how long Kafka took to write the message to the broker and invoke the callback.
      See Also:

    • IDLE_TIME

      public static final String IDLE_TIME
      See Also:

    • PRODUCER_CALLBACK_LATENCY

      public static final String PRODUCER_CALLBACK_LATENCY
      Leader metric: Measures the time spent within the leader's producer callback processing after a message is successfully produced to the local broker. This includes chunking processing, producing to drainer buffer service, producing deprecated chunk deletions, and recording stats.
      See Also:

    • LEADER_PREPROCESSING_LATENCY

      public static final String LEADER_PREPROCESSING_LATENCY
      Leader metric: Measures the time from when keys are locked for a batch of records to just before producing to Kafka. When batch processing is enabled: includes batch processing (with partial update operations), record validation, and for real-time topics, recording hybrid consumption stats. When batch processing is disabled: includes only record validation and stats recording; partial update happens later.
      See Also:

    • INTERNAL_PREPROCESSING_LATENCY

      public static final String INTERNAL_PREPROCESSING_LATENCY
      Drainer metric: Measures the time spent on lightweight preprocessing tasks before heavy message processing begins. This includes recording write path latency stats (producer-to-broker, broker-to-consumer), drainer message validation, and reporting batch end of incremental push status. Recorded at the start of internalProcessConsumerRecord, before control message or data message processing.
      See Also:

    • BATCH_PROCESSING_REQUEST

      public static final String BATCH_PROCESSING_REQUEST
      See Also:

    • BATCH_PROCESSING_REQUEST_SIZE

      public static final String BATCH_PROCESSING_REQUEST_SIZE
      See Also:

    • BATCH_PROCESSING_REQUEST_RECORDS

      public static final String BATCH_PROCESSING_REQUEST_RECORDS
      See Also:

    • BATCH_PROCESSING_REQUEST_LATENCY

      public static final String BATCH_PROCESSING_REQUEST_LATENCY
      See Also:

    • BATCH_PROCESSING_REQUEST_ERROR

      public static final String BATCH_PROCESSING_REQUEST_ERROR
      See Also:

    • STORAGE_QUOTA_USED

      public static final String STORAGE_QUOTA_USED
      See Also:

  • Constructor Details

  • Method Details

    • getIngestionTask

      public StoreIngestionTask getIngestionTask()

    • setIngestionTask

      public void setIngestionTask(StoreIngestionTask ingestionTask)

    • getIngestionTaskErroredGauge

      public int getIngestionTaskErroredGauge()

    • getWriteComputeErrorCode

      public int getWriteComputeErrorCode()

    • getSubscribePrepLatencyAvg

      public double getSubscribePrepLatencyAvg()

    • getSubscribePrepLatencyMax

      public double getSubscribePrepLatencyMax()

    • recordSubscribePrepLatency

      public void recordSubscribePrepLatency(double value, long currentTimeMs)

    • getProducerCallBackLatencyMax

      public double getProducerCallBackLatencyMax()

    • recordProducerCallBackLatency

      public void recordProducerCallBackLatency(double value, long currentTimeMs)

    • getLeaderPreprocessingLatencyMax

      public double getLeaderPreprocessingLatencyMax()

    • getLeaderPreprocessingLatencyAvg

      public double getLeaderPreprocessingLatencyAvg()

    • recordLeaderPreprocessingLatency

      public void recordLeaderPreprocessingLatency(double value, long currentTimeMs)

    • getInternalPreprocessingLatencyAvg

      public double getInternalPreprocessingLatencyAvg()

    • getInternalPreprocessingLatencyMax

      public double getInternalPreprocessingLatencyMax()

    • recordInternalPreprocessingLatency

      public void recordInternalPreprocessingLatency(double value, long currentTimeMs)

    • getConsumedRecordEndToEndProcessingLatencyAvg

      public double getConsumedRecordEndToEndProcessingLatencyAvg()

    • getConsumedRecordEndToEndProcessingLatencyMax

      public double getConsumedRecordEndToEndProcessingLatencyMax()

    • recordConsumedRecordEndToEndProcessingLatency

      public void recordConsumedRecordEndToEndProcessingLatency(double value, long currentTimeMs)

    • getRecordsConsumed

      public double getRecordsConsumed()

    • recordRecordsConsumed

      public void recordRecordsConsumed()

    • getBytesConsumed

      public double getBytesConsumed()

    • recordBytesConsumed

      public void recordBytesConsumed(long value)

    • getLeaderRecordsConsumed

      public double getLeaderRecordsConsumed()

    • recordLeaderRecordsConsumed

      public void recordLeaderRecordsConsumed()

    • getLeaderBytesConsumed

      public double getLeaderBytesConsumed()

    • recordLeaderBytesConsumed

      public void recordLeaderBytesConsumed(long value)

    • getFollowerRecordsConsumed

      public double getFollowerRecordsConsumed()

    • recordFollowerRecordsConsumed

      public void recordFollowerRecordsConsumed()

    • getFollowerBytesConsumed

      public double getFollowerBytesConsumed()

    • recordFollowerBytesConsumed

      public void recordFollowerBytesConsumed(long value)

    • recordUpdateIgnoredDCR

      public void recordUpdateIgnoredDCR()

    • recordTotalDCR

      public void recordTotalDCR()

    • recordTotalDuplicateKeyUpdate

      public void recordTotalDuplicateKeyUpdate()

    • recordTimestampRegressionDCRError

      public void recordTimestampRegressionDCRError()

    • recordOffsetRegressionDCRError

      public void recordOffsetRegressionDCRError()

    • recordTombStoneCreationDCR

      public void recordTombStoneCreationDCR()

    • getRegionHybridBytesConsumed

      public double getRegionHybridBytesConsumed(int regionId)

    • recordRegionHybridBytesConsumed

      public void recordRegionHybridBytesConsumed(int regionId, double value, long currentTimeMs)

    • getRegionHybridRecordsConsumed

      public double getRegionHybridRecordsConsumed(int regionId)

    • recordRegionHybridRecordsConsumed

      public void recordRegionHybridRecordsConsumed(int regionId, double value, long currentTimeMs)

    • getLeaderRecordsProduced

      public double getLeaderRecordsProduced()

    • getUpdateIgnoredRate

      public double getUpdateIgnoredRate()

    • getTotalDCRRate

      public double getTotalDCRRate()

    • getTotalDuplicateKeyUpdateCount

      public double getTotalDuplicateKeyUpdateCount()

    • getTombstoneCreationDCRRate

      public double getTombstoneCreationDCRRate()

    • getTimestampRegressionDCRRate

      public double getTimestampRegressionDCRRate()

    • getOffsetRegressionDCRRate

      public double getOffsetRegressionDCRRate()

    • recordLeaderRecordsProduced

      public void recordLeaderRecordsProduced(long value)

    • getLeaderBytesProduced

      public double getLeaderBytesProduced()

    • recordLeaderBytesProduced

      public void recordLeaderBytesProduced(long value)

    • setIngestionTaskPushTimeoutGauge

      public void setIngestionTaskPushTimeoutGauge(int value)

    • getIngestionTaskPushTimeoutGauge

      public int getIngestionTaskPushTimeoutGauge()

    • getNearlineProducerToLocalBrokerLatencyAvg

      public double getNearlineProducerToLocalBrokerLatencyAvg()

    • getNearlineProducerToLocalBrokerLatencyMax

      public double getNearlineProducerToLocalBrokerLatencyMax()

    • recordNearlineProducerToLocalBrokerLatency

      public void recordNearlineProducerToLocalBrokerLatency(double value, long currentTimeMs)

    • recordIdleTime

      public void recordIdleTime(long value)

    • getIdleTime

      public double getIdleTime()

    • getProducerSourceBrokerLatencySensor

      public WritePathLatencySensor getProducerSourceBrokerLatencySensor()

    • recordProducerSourceBrokerLatencyMs

      public void recordProducerSourceBrokerLatencyMs(double value, long currentTimeMs)

    • recordSourceBrokerLeaderConsumerLatencyMs

      public void recordSourceBrokerLeaderConsumerLatencyMs(double value, long currentTimeMs)

    • getSourceBrokerLeaderConsumerLatencySensor

      public WritePathLatencySensor getSourceBrokerLeaderConsumerLatencySensor()

    • recordProducerLocalBrokerLatencyMs

      public void recordProducerLocalBrokerLatencyMs(double value, long currentTimeMs)

    • getProducerLocalBrokerLatencySensor

      public WritePathLatencySensor getProducerLocalBrokerLatencySensor()

    • recordLocalBrokerFollowerConsumerLatencyMs

      public void recordLocalBrokerFollowerConsumerLatencyMs(double value, long currentTimeMs)

    • getLocalBrokerFollowerConsumerLatencySensor

      public WritePathLatencySensor getLocalBrokerFollowerConsumerLatencySensor()

    • recordLeaderProducerCompletionLatencyMs

      public void recordLeaderProducerCompletionLatencyMs(double value, long currentTimeMs)

    • getLeaderProducerCompletionLatencySensor

      public WritePathLatencySensor getLeaderProducerCompletionLatencySensor()

    • recordBatchProcessingRequest

      public void recordBatchProcessingRequest(int size, long currentTimeMs)

    • getBatchProcessingRequest

      public double getBatchProcessingRequest()

    • getBatchProcessingRequestRecords

      public double getBatchProcessingRequestRecords()

    • recordBatchProcessingRequestError

      public void recordBatchProcessingRequestError()

    • getBatchProcessingRequestError

      public double getBatchProcessingRequestError()

    • getBatchProcessingRequestSizeSensor

      public WritePathLatencySensor getBatchProcessingRequestSizeSensor()

    • recordBatchProcessingRequestLatency

      public void recordBatchProcessingRequestLatency(double latency, long currentTimeMs)

    • getBatchProcessingRequestLatencySensor

      public WritePathLatencySensor getBatchProcessingRequestLatencySensor()

    • unAvailableToZero

      public static double unAvailableToZero(double value)

    • getStorageQuotaUsed

      public double getStorageQuotaUsed()
      Retrieves the storage quota usage for the current ingestion task.
      Returns:
      The disk quota usage as a double value, or 0 if unavailable.