Package com.linkedin.venice.hadoop.task.datawriter

Class ExternalStorageWriteThrottler

java.lang.Object

com.linkedin.venice.hadoop.task.datawriter.ExternalStorageWriteThrottler

public class ExternalStorageWriteThrottler
extends Object

Per-(partition-writer task, region) rate limiter for the VPJ external-storage dual-write path. A single configured global record rate and/or byte rate is the budget for writing to one external region; because partition-writer tasks run in separate distributed executors with no shared memory, the budget is enforced by static even split: every task gets a local throttler sized to globalRate / partitionCount so the aggregate across all partitionCount tasks for that region does not exceed the global rate.

Each region gets its own ExternalStorageWriteThrottler instance (independent buckets): the dual-write fan-out writes the same batch to every region sequentially, so a shared bucket would let two regions drain one budget and halve each region's effective rate. Per-region instances keep each region at its full per-region budget.

The two dimensions are independent: either can be configured alone (the other limiter is null and that dimension is unthrottled). Both use a blocking VeniceRateLimiter — a push must slow down to fit the budget, never drop records.

Method Summary

Modifier and Type	Method	Description
static void	validateQuota(long globalRecordRatePerSecond, long globalByteRatePerSecond, int partitionCount)	Validate that every configured (> 0) dimension can be split into at least 1/sec per task for partitionCount tasks, throwing VeniceException otherwise.

Methods inherited from class java.lang.Object

clone, equals, finalize, getClass, hashCode, notify, notifyAll, toString, wait, wait, wait

Method Details

validateQuota

public static void validateQuota(long globalRecordRatePerSecond,
 long globalByteRatePerSecond,
 int partitionCount)

Validate that every configured (> 0) dimension can be split into at least 1/sec per task for partitionCount tasks, throwing VeniceException otherwise. Allocates nothing, so the VPJ driver can call it to fail a misconfigured push before launching the data-writer job (and its cluster resources). Partition count is only known after version creation, so the earliest the driver can call this is right after the version is created, not before topic creation.