Package com.linkedin.venice.jobs

Class DataWriterComputeJob

java.lang.Object

com.linkedin.venice.jobs.DataWriterComputeJob

All Implemented Interfaces:

ComputeJob, Closeable, AutoCloseable

Direct Known Subclasses:

AbstractDataWriterSparkJob, DataWriterMRJob

public abstract class DataWriterComputeJob
extends Object
implements ComputeJob

An abstraction for executing and monitoring a data writer compute job running on any batch compute engine

Nested Class Summary

Nested Classes

Modifier and Type	Class	Description
static interface	DataWriterComputeJob.ConfigSetter

Nested classes/interfaces inherited from interface com.linkedin.venice.jobs.ComputeJob

ComputeJob.Status

Field Summary

Fields

Modifier and Type	Field	Description
static final List<String>	PASS_THROUGH_CONFIG_PREFIXES	Pass-through the properties whose names start with: VeniceWriter.VENICE_WRITER_CONFIG_PREFIX ConfigKeys.KAFKA_CONFIG_PREFIX KafkaInputRecordReader.KIF_RECORD_READER_KAFKA_CONFIG_PREFIX

Constructor Summary

Constructors

Constructor	Description
DataWriterComputeJob()

Method Summary

Modifier and Type	Method	Description
void	configure(VeniceProperties properties)
abstract void	configure(VeniceProperties props, PushJobSetting pushJobSetting)
Throwable	getFailureReason()
protected abstract PushJobSetting	getPushJobSetting()
ComputeJob.Status	getStatus()
abstract DataWriterTaskTracker	getTaskTracker()
void	kill()
static void	populateWithPassThroughConfigs(VeniceProperties props, DataWriterComputeJob.ConfigSetter configSetter)
static void	populateWithPassThroughConfigs(VeniceProperties props, DataWriterComputeJob.ConfigSetter configSetter, List<String> passThroughConfigPrefix, String overridePrefix)	Override the configs following the rules: Pass-through the properties whose names start with the prefixes defined in `passThroughConfigPrefix`. Pass-through the properties whose names starts with the prefix defined by another special property ConfigKeys.PASS_THROUGH_CONFIG_PREFIXES_LIST_KEY. Override the properties that are specified with a particular `overridePrefix`.
protected abstract void	runComputeJob()
void	runJob()
protected void	validateJob()

Methods inherited from class java.lang.Object

clone, equals, finalize, getClass, hashCode, notify, notifyAll, toString, wait, wait, wait

Methods inherited from interface com.linkedin.venice.jobs.ComputeJob

close

Field Details

PASS_THROUGH_CONFIG_PREFIXES

public static final List<String> PASS_THROUGH_CONFIG_PREFIXES

Pass-through the properties whose names start with:

• VeniceWriter.VENICE_WRITER_CONFIG_PREFIX
• ConfigKeys.KAFKA_CONFIG_PREFIX
• KafkaInputRecordReader.KIF_RECORD_READER_KAFKA_CONFIG_PREFIX

Constructor Details

DataWriterComputeJob

public DataWriterComputeJob()

Method Details

populateWithPassThroughConfigs

public static void populateWithPassThroughConfigs(VeniceProperties props,
 DataWriterComputeJob.ConfigSetter configSetter,
 List<String> passThroughConfigPrefix,
 String overridePrefix)

Override the configs following the rules:

• Pass-through the properties whose names start with the prefixes defined in `passThroughConfigPrefix`.
• Pass-through the properties whose names starts with the prefix defined by another special property ConfigKeys.PASS_THROUGH_CONFIG_PREFIXES_LIST_KEY.
• Override the properties that are specified with a particular `overridePrefix`.

populateWithPassThroughConfigs

public static void populateWithPassThroughConfigs(VeniceProperties props,
 DataWriterComputeJob.ConfigSetter configSetter)

getTaskTracker

public abstract DataWriterTaskTracker getTaskTracker()

validateJob

protected void validateJob()

runComputeJob

protected abstract void runComputeJob()

configure

public void configure(VeniceProperties properties)

Specified by:

configure in interface ComputeJob

getPushJobSetting

protected abstract PushJobSetting getPushJobSetting()

configure

public abstract void configure(VeniceProperties props,
 PushJobSetting pushJobSetting)

runJob

public void runJob()

Specified by:

runJob in interface ComputeJob

kill

public void kill()

Specified by:

kill in interface ComputeJob

getStatus

public ComputeJob.Status getStatus()

Specified by:

getStatus in interface ComputeJob

getFailureReason

public Throwable getFailureReason()

Specified by:

getFailureReason in interface ComputeJob