Package com.linkedin.venice.pushmonitor

Interface PushMonitor

  • All Known Implementing Classes:
    AbstractPushMonitor, PartitionStatusBasedPushMonitor, PushMonitorDelegator
    public interface PushMonitor
    In Venice, Push is a task that asynchronously writes data to Venice. Currently, Venice supports 2 kinds of push schemes -- pushing from Hadoop(offline) and pushing from Samza(nearline). Push Monitor is a critical component in Venice controller that keeps track of ongoing and finished pushes in the cluster. Inheritors are expected to be able to: 1. persist recent push history even after restart. This is the first-hand information for debugging push failures 2. Tell what a push's current status is. 3. Switch store's versions when pushes are finished and new data is ready to serve traffic.

    • Method Detail

      • loadAllPushes

        void loadAllPushes()
        load previous push statuses when push monitor restarts

      • stopAllMonitoring

        void stopAllMonitoring()
        Stop monitoring all offline pushes.

      • startMonitorOfflinePush

        void startMonitorOfflinePush​(java.lang.String kafkaTopic,
                             int numberOfPartition,
                             int replicaFactor,
                             OfflinePushStrategy strategy)
        Parameters:
        strategy - on which criteria a push is considered to be completed. check PushStatusDecider for more details Start monitoring a new push

      • stopMonitorOfflinePush

        void stopMonitorOfflinePush​(java.lang.String kafkaTopic,
                            boolean deletePushStatus,
                            boolean isForcedDelete)
        Stop monitoring a push. This function should be called when: 1. Retire a version; 2. Leader controller transits to standby; in this case, controller shouldn't delete any push status.

      • cleanupStoreStatus

        void cleanupStoreStatus​(java.lang.String storeName)
        Clean up all push statuses related to a store including all error pushes. This is called when a store gets deleted.

      • getOfflinePushOrThrow

        OfflinePushStatus getOfflinePushOrThrow​(java.lang.String topic)
        Return a push's status (push status contains the history statuses and current status) and throw exception if it doesn't exist

      • getPushStatusAndDetails

        ExecutionStatusWithDetails getPushStatusAndDetails​(java.lang.String topic)
        Returns:
        return the current status. If it's in error, some debugging info is also presented.

      • getUncompletedPartitions

        java.util.List<UncompletedPartition> getUncompletedPartitions​(java.lang.String topic)

      • getIncrementalPushStatusAndDetails

        ExecutionStatusWithDetails getIncrementalPushStatusAndDetails​(java.lang.String kafkaTopic,
                                                              java.lang.String incrementalPushVersion,
                                                              HelixCustomizedViewOfflinePushRepository customizedViewOfflinePushRepository)
        Returns incremental push's status read from (ZooKeeper backed) OfflinePushStatus

      • getOngoingIncrementalPushVersions

        java.util.Set<java.lang.String> getOngoingIncrementalPushVersions​(java.lang.String kafkaTopic)
        Check if there are any ongoing incremental push for the given version topic.
        Parameters:
        kafkaTopic - to check for ongoing incremental push
        Returns:
        ongoing incremental push versions if there are any, otherwise an empty set is returned.

      • getOngoingIncrementalPushVersions

        java.util.Set<java.lang.String> getOngoingIncrementalPushVersions​(java.lang.String kafkaTopic,
                                                                  PushStatusStoreReader pushStatusStoreReader)

      • getTopicsOfOngoingOfflinePushes

        java.util.List<java.lang.String> getTopicsOfOngoingOfflinePushes()
        Find all ongoing pushes then return the topics associated to those pushes.

      • getOfflinePushStatusForStore

        java.util.List<OfflinePushStatus> getOfflinePushStatusForStore​(java.lang.String storeName)

      • markOfflinePushAsError

        void markOfflinePushAsError​(java.lang.String topic,
                            java.lang.String statusDetails)
        Mark a push to be as error. This is usually called when push is killed.

      • refreshAndUpdatePushStatus

        OfflinePushStatus refreshAndUpdatePushStatus​(java.lang.String kafkaTopic,
                                             ExecutionStatus newStatus,
                                             java.util.Optional<java.lang.String> newStatusDetails)
        Here, we refresh the push status, in order to avoid a race condition where a small job could already be completed. Previously, we would clobber the COMPLETED status with STARTED, which would stall the job forever. Now, since we get the refreshed status, we can validate whether a transition to {@param newStatus} is valid, before making the change. If if wouldn't be valid (because the job already completed or already failed, for example), then we leave the status as is, rather than adding in the new details.
        Returns:
        the new OfflinePushStatus if it was updated, or null if the update was skipped.

      • recordPushPreparationDuration

        void recordPushPreparationDuration​(java.lang.String topic,
                                   long offlinePushWaitTimeInSecond)
        stats related operation. TODO: we may want to move it out of the interface

      • getReadyToServeInstances

        java.util.List<Instance> getReadyToServeInstances​(PartitionAssignment partitionAssignment,
                                                  int partitionId)

      • isOfflinePushMonitorDaVinciPushStatusEnabled

        boolean isOfflinePushMonitorDaVinciPushStatusEnabled()