Package com.linkedin.venice.hadoop.task.datawriter

Class IncrementalPushWriteQuotaUtils

java.lang.Object

com.linkedin.venice.hadoop.task.datawriter.IncrementalPushWriteQuotaUtils

public final class IncrementalPushWriteQuotaUtils
extends Object

Driver-side helper for the incremental-push write quota.

The configured quota (incremental.push.write.quota.records.per.second) is a single global job/store budget. Because partition-writer tasks run independently in distributed executors with no shared state, the budget is enforced by a static even split: each of the partitionCount tasks gets globalRecordsPerSecond / partitionCount. That division is computed once here, on the driver, and forwarded to the tasks so enforcement never re-derives (or disagrees about) the split.

Field Summary

Fields

Modifier and Type	Field	Description
static final long	THROTTLING_DISABLED	Sentinel returned (and forwarded to tasks) when incremental-push throttling does not apply.

Method Summary

Modifier and Type	Method	Description
static long	perPartitionQuotaForPush(boolean isIncrementalPush, boolean pushToSeparateRealtimeTopicEnabled, boolean versionSeparateRealTimeTopicEnabled, long globalRecordsPerSecond, int partitionCount)	Compute the per-partition-writer quota for a push, validating on the driver so a misconfiguration fails fast before the data-writer job is launched.

Methods inherited from class java.lang.Object

clone, equals, finalize, getClass, hashCode, notify, notifyAll, toString, wait, wait, wait

Field Details

THROTTLING_DISABLED

public static final long THROTTLING_DISABLED

Sentinel returned (and forwarded to tasks) when incremental-push throttling does not apply.

See Also:

• Constant Field Values

Method Details

perPartitionQuotaForPush

public static long perPartitionQuotaForPush(boolean isIncrementalPush,
 boolean pushToSeparateRealtimeTopicEnabled,
 boolean versionSeparateRealTimeTopicEnabled,
 long globalRecordsPerSecond,
 int partitionCount)

Compute the per-partition-writer quota for a push, validating on the driver so a misconfiguration fails fast before the data-writer job is launched.

Returns THROTTLING_DISABLED when throttling does not apply: batch pushes, incremental pushes to a separate real-time topic (both the push-job and store configs are enabled), or a non-positive global quota. Otherwise returns globalRecordsPerSecond / partitionCount.

Throws:

VeniceException - if a positive quota cannot be split into at least 1 record/sec per partition-writer task (i.e. partitionCount <= 0, or globalRecordsPerSecond < partitionCount).