Package com.linkedin.venice.jobs

Class DataWriterComputeJob

java.lang.Object
com.linkedin.venice.jobs.DataWriterComputeJob
All Implemented Interfaces:
ComputeJob, Closeable, AutoCloseable
Direct Known Subclasses:
AbstractDataWriterSparkJob, DataWriterMRJob
public abstract class DataWriterComputeJob extends Object implements ComputeJob
An abstraction for executing and monitoring a data writer compute job running on any batch compute engine

  • Field Details

  • Constructor Details

    • DataWriterComputeJob

      public DataWriterComputeJob()

  • Method Details

    • populateWithPassThroughConfigs

      public static void populateWithPassThroughConfigs(VeniceProperties props, DataWriterComputeJob.ConfigSetter configSetter, List<String> passThroughConfigPrefix, String overridePrefix)
      Override the configs following the rules:
      • Pass-through the properties whose names start with the prefixes defined in `passThroughConfigPrefix`.
      • Pass-through the properties whose names starts with the prefix defined by another special property ConfigKeys.PASS_THROUGH_CONFIG_PREFIXES_LIST_KEY.
      • Override the properties that are specified with a particular `overridePrefix`.

    • populateWithPassThroughConfigs

      public static void populateWithPassThroughConfigs(VeniceProperties props, DataWriterComputeJob.ConfigSetter configSetter)

    • getTaskTracker

      public abstract DataWriterTaskTracker getTaskTracker()

    • getStageMetricsSnapshot

      public Optional<StageMetricsSnapshot> getStageMetricsSnapshot()
      Returns an immutable snapshot of per-stage diagnostic metrics, or empty if not supported by this compute engine. Overridden by Spark-based implementations.

    • performsPreWriteQuotaCheck

      public boolean performsPreWriteQuotaCheck()
      Whether this engine validates the storage quota before writing any data (see the Spark data writer's pre-write intermediary stage, which measures the serialized input size and fails the job before writing if it exceeds the quota). When this returns true, the driver-side post-write quota check in VenicePushJob is redundant and is skipped. The default is false (e.g. MapReduce), which keeps the driver-side post-write check as the authoritative quota gate.

    • setCurrentStorageQuotaSupplier

      public void setCurrentStorageQuotaSupplier(LongSupplier currentStorageQuotaSupplier)

    • getCurrentStorageQuotaSupplier

      protected LongSupplier getCurrentStorageQuotaSupplier()

    • validateJob

      public void validateJob()

    • runComputeJob

      public abstract void runComputeJob()

    • configure

      public void configure(VeniceProperties properties)
      Specified by:
      configure in interface ComputeJob

    • getPushJobSetting

      protected abstract PushJobSetting getPushJobSetting()

    • configure

      public abstract void configure(VeniceProperties props, PushJobSetting pushJobSetting)

    • runJob

      public void runJob()
      Specified by:
      runJob in interface ComputeJob

    • kill

      public void kill()
      Specified by:
      kill in interface ComputeJob

    • getStatus

      public ComputeJob.Status getStatus()
      Specified by:
      getStatus in interface ComputeJob

    • getFailureReason

      public Throwable getFailureReason()
      Specified by:
      getFailureReason in interface ComputeJob