Package com.linkedin.venice.fastclient.meta

Class InstanceHealthMonitor

java.lang.Object
com.linkedin.venice.fastclient.meta.InstanceHealthMonitor
All Implemented Interfaces:
Closeable, AutoCloseable
public class InstanceHealthMonitor extends Object implements Closeable
This class is used to track the health of the instances Fast Client is talking to. Here is the strategy: 1. It is maintaining a pending request map to track the pending request count per instance. 2. If the pending request count exceeds the configured threshold, the instance will be blocked. 3. If the request is timed out based on the configured timeout threshold, this instance will be put into suspicious instance set. 4. The embedded heart-beat runnable will send heartbeat request to the suspicious/unhealthy instances periodically, and if the heartbeat request still times out or receives error response, the instance will be put into the unhealthy instance set. Otherwise, the instance will be removed from the suspicious and unhealthy instance set. 5. Fast Client won't send user requests to the blocked instances and unhealthy instances.

  • Constructor Details

  • Method Details

    • getTimeoutProcessor

      public TimeoutProcessor getTimeoutProcessor()

    • trackHealthBasedOnRequestToInstance

      public ChainedCompletableFuture<Integer,Integer> trackHealthBasedOnRequestToInstance(String instance)

    • trackHealthBasedOnRequestToInstance

      public ChainedCompletableFuture<Integer,Integer> trackHealthBasedOnRequestToInstance(String instance, CompletableFuture<TransportClientResponse> transportFuture)
      This function tracks the health of an Instance based on the request sent to that Instance: by returning an incomplete completable future for AbstractStoreMetadata which 1. increments pendingRequestCounterMap for each server instances per store. This is done in this function which is called before starting a get() request. 2. whenComplete() of this completable future decrements the above counters once the response for the get() request is received. Using this we can track the number of pending requests for each server instance.

    • isInstanceHealthy

      public boolean isInstanceHealthy(String instance)
      If an instance is marked unhealthy, this instances will be retried again continuously to know if that instance comes back up and start serving requests. Note that these instances will eventually become blocked when it reaches the threshold for pendingRequestCounter. This provides some break between continuously sending requests to these instances.

    • isInstanceBlocked

      public boolean isInstanceBlocked(String instance)
      If an instance is blocked, it won't be considered for new requests until the requests are closed either in a proper manner or closed by {@link #trackHealthBasedOnRequestToInstance#timeoutFuture}

    • getBlockedInstanceCount

      public int getBlockedInstanceCount()

    • getUnhealthyInstanceCount

      public int getUnhealthyInstanceCount()

    • getPendingRequestCounter

      public int getPendingRequestCounter(String instance)

    • close

      public void close() throws IOException
      Specified by:
      close in interface AutoCloseable
      Specified by:
      close in interface Closeable
      Throws:
      IOException