Package com.linkedin.venice.utils

Class DiskUsage

java.lang.Object
com.linkedin.venice.utils.DiskUsage
public class DiskUsage extends Object
We define a threshold where we consider the disk to be full. For example, we might declare the disk full at 0.90 usage. Each task that writes to the disk maintains an instance of this class. That instance calculates a "reserve" of space that the task is allowed to write without checking the disk again. For example, a 3TB (or 3000GB) disk with a threshold of 0.90 requires 300G to be free on the disk. We calculate the reserve space as 0.001 of the required free space. In this case 300MB. On every write, the task calls this object's #isDiskFull() method, passing the number of bytes written. Once the task has written 300MB (or whatever the reserve is) then this object will reinspect the disk for it's actual usage and reallocate a reserve. If we have exceeded the disk full threshold, we report the disk is full. We calculate reserve space as 0.001 of required free space to allow 1000 concurrent tasks to write to disk without risk of blowing past the disk full threshold and actually reaching 100% disk usage.

  • Constructor Details

    • DiskUsage

      public DiskUsage(String path, double diskFullThreshold)

  • Method Details

    • isDiskFull

      public boolean isDiskFull(long bytesWritten)
      Each time you write bytes to the disk, call this method. It will determine an appropriate frequency to update its view of the disk so this call will usually be very fast. If the disk is beyond the threshold that was set at construction then it will return true. Otherwise it returns false.

    • getDiskStatus

      public String getDiskStatus()