Class StorageUtilizationManager

  • All Implemented Interfaces:
    StoreDataChangedListener

    public class StorageUtilizationManager
    extends java.lang.Object
    implements StoreDataChangedListener
    This class has the following responsibilities: 1. Keep track of storage utilization, at least on disk, and potentially also in-memory (when hybrid quota enforcement is enabled).. 2. Take action (pause/resume) on the associated consumer if storage quota is breached (and enforcement is enabled). 3. Listen to store config changes related to quota limits and whether enforcement is enabled, and react accordingly. 4: Report replica status changes if the above actions affect them. TODO: Consider whether this is tech debt and if we could/should decouple status reporting from this class. This would allow us to stop passing in the ingestionNotificationDispatcher which in turn may allow us to stop mutating the entries in partitionConsumptionStateMap (in which case, we could pass a map where the values are a read-only interface implemented by PartitionConsumptionState and thus preventing mutations of this state from here). For a Samza RT job, if a partition exhausts the partition-level quota, we will stall the consumption for this partition. i.e. stop polling and processing records for this partition until more space is available (quota gets bumped or db compactions shrink disk usage); if only some partitions violates the quota, the job will pause these partitions while keep processing the other good partitions. Assumptions: 1. every partition in a store shares similar size; 2. no write-compute messages/partial updates/incremental push