Package com.linkedin.venice.pushmonitor

Interface PushMonitor

All Known Implementing Classes:
AbstractPushMonitor, PartitionStatusBasedPushMonitor, PushMonitorDelegator
public interface PushMonitor
In Venice, Push is a task that asynchronously writes data to Venice. Currently, Venice supports 2 kinds of push schemes -- pushing from Hadoop(offline) and pushing from Samza(nearline). Push Monitor is a critical component in Venice controller that keeps track of ongoing and finished pushes in the cluster. Inheritors are expected to be able to: 1. persist recent push history even after restart. This is the first-hand information for debugging push failures 2. Tell what a push's current status is. 3. Switch store's versions when pushes are finished and new data is ready to serve traffic.

  • Method Details

    • loadAllPushes

      void loadAllPushes()
      load previous push statuses when push monitor restarts

    • stopAllMonitoring

      void stopAllMonitoring()
      Stop monitoring all offline pushes.

    • startMonitorOfflinePush

      void startMonitorOfflinePush(String kafkaTopic, int numberOfPartition, int replicaFactor, OfflinePushStrategy strategy)
      Parameters:
      strategy - on which criteria a push is considered to be completed. check PushStatusDecider for more details Start monitoring a new push

    • stopMonitorOfflinePush

      void stopMonitorOfflinePush(String kafkaTopic, boolean deletePushStatus, boolean isForcedDelete)
      Stop monitoring a push. This function should be called when: 1. Retire a version; 2. Leader controller transits to standby; in this case, controller shouldn't delete any push status.

    • cleanupStoreStatus

      void cleanupStoreStatus(String storeName)
      Clean up all push statuses related to a store including all error pushes. This is called when a store gets deleted.

    • getOfflinePushOrThrow

      OfflinePushStatus getOfflinePushOrThrow(String topic)
      Return a push's status (push status contains the history statuses and current status) and throw exception if it doesn't exist

    • getPushStatusAndDetails

      ExecutionStatusWithDetails getPushStatusAndDetails(String topic)
      Returns:
      return the current status. If it's in error, some debugging info is also presented.

    • getUncompletedPartitions

      List<UncompletedPartition> getUncompletedPartitions(String topic)

    • getIncrementalPushStatusAndDetails

      ExecutionStatusWithDetails getIncrementalPushStatusAndDetails(String kafkaTopic, String incrementalPushVersion, HelixCustomizedViewOfflinePushRepository customizedViewOfflinePushRepository)
      Returns incremental push's status read from (ZooKeeper backed) OfflinePushStatus

    • getIncrementalPushStatusFromPushStatusStore

      ExecutionStatusWithDetails getIncrementalPushStatusFromPushStatusStore(String kafkaTopic, String incrementalPushVersion, HelixCustomizedViewOfflinePushRepository customizedViewRepo, PushStatusStoreReader pushStatusStoreReader)
      Returns incremental push's status read from push status store

    • getOngoingIncrementalPushVersions

      Set<String> getOngoingIncrementalPushVersions(String kafkaTopic)
      Check if there are any ongoing incremental push for the given version topic.
      Parameters:
      kafkaTopic - to check for ongoing incremental push
      Returns:
      ongoing incremental push versions if there are any, otherwise an empty set is returned.

    • getOngoingIncrementalPushVersions

      Set<String> getOngoingIncrementalPushVersions(String kafkaTopic, PushStatusStoreReader pushStatusStoreReader)

    • getTopicsOfOngoingOfflinePushes

      List<String> getTopicsOfOngoingOfflinePushes()
      Find all ongoing pushes then return the topics associated to those pushes.

    • getOfflinePushStatusForStore

      List<OfflinePushStatus> getOfflinePushStatusForStore(String storeName)

    • markOfflinePushAsError

      void markOfflinePushAsError(String topic, String statusDetails)
      Mark a push to be as error. This is usually called when push is killed.

    • refreshAndUpdatePushStatus

      OfflinePushStatus refreshAndUpdatePushStatus(String kafkaTopic, ExecutionStatus newStatus, Optional<String> newStatusDetails)
      Here, we refresh the push status, in order to avoid a race condition where a small job could already be completed. Previously, we would clobber the COMPLETED status with STARTED, which would stall the job forever. Now, since we get the refreshed status, we can validate whether a transition to is valid, before making the change. If if wouldn't be valid (because the job already completed or already failed, for example), then we leave the status as is, rather than adding in the new details.
      Returns:
      the new OfflinePushStatus if it was updated, or null if the update was skipped.

    • recordPushPreparationDuration

      void recordPushPreparationDuration(String topic, long offlinePushWaitTimeInSecond)
      stats related operation. TODO: we may want to move it out of the interface

    • getReadyToServeInstances

      List<Instance> getReadyToServeInstances(PartitionAssignment partitionAssignment, int partitionId)

    • isOfflinePushMonitorDaVinciPushStatusEnabled

      boolean isOfflinePushMonitorDaVinciPushStatusEnabled()