Package com.linkedin.venice.hadoop.spark

Class SparkConstants

java.lang.Object

com.linkedin.venice.hadoop.spark.SparkConstants

public class SparkConstants
extends java.lang.Object

Field Summary

Fields

Modifier and Type	Field	Description
static org.apache.spark.sql.types.StructType	DEFAULT_SCHEMA
static org.apache.spark.sql.types.StructType	DEFAULT_SCHEMA_WITH_PARTITION
static java.lang.String	DEFAULT_SPARK_CLUSTER
static java.lang.String	KEY_COLUMN_NAME
static java.lang.String	PARTITION_COLUMN_NAME
static java.lang.String	SPARK_APP_NAME_CONFIG
static java.lang.String	SPARK_CASE_SENSITIVE_CONFIG
static java.lang.String	SPARK_CLUSTER_CONFIG
static java.lang.String	SPARK_DATA_WRITER_CONF_PREFIX	Configs with this prefix will be set when building the data writer spark job and passed as job properties.
static java.lang.String	SPARK_LEADER_CONFIG
static java.lang.String	SPARK_SESSION_CONF_PREFIX	Configs with this prefix will be set when building the spark session.
static java.lang.String	VALUE_COLUMN_NAME

Constructor Summary

Constructors

Constructor	Description
SparkConstants()

Method Summary

Methods inherited from class java.lang.Object

clone, equals, finalize, getClass, hashCode, notify, notifyAll, toString, wait, wait, wait

Field Detail

KEY_COLUMN_NAME

public static final java.lang.String KEY_COLUMN_NAME

See Also:

Constant Field Values

VALUE_COLUMN_NAME

public static final java.lang.String VALUE_COLUMN_NAME

See Also:

Constant Field Values

PARTITION_COLUMN_NAME

public static final java.lang.String PARTITION_COLUMN_NAME

See Also:

Constant Field Values

DEFAULT_SCHEMA

public static final org.apache.spark.sql.types.StructType DEFAULT_SCHEMA

DEFAULT_SCHEMA_WITH_PARTITION

public static final org.apache.spark.sql.types.StructType DEFAULT_SCHEMA_WITH_PARTITION

SPARK_SESSION_CONF_PREFIX

public static final java.lang.String SPARK_SESSION_CONF_PREFIX

Configs with this prefix will be set when building the spark session. These will get applied to all Spark jobs that get triggered as a part of VPJ. It can be used to configure arbitrary cluster properties like cluster address.

See Also:

Constant Field Values

SPARK_APP_NAME_CONFIG

public static final java.lang.String SPARK_APP_NAME_CONFIG

See Also:

Constant Field Values

SPARK_CASE_SENSITIVE_CONFIG

public static final java.lang.String SPARK_CASE_SENSITIVE_CONFIG

See Also:

Constant Field Values

SPARK_CLUSTER_CONFIG

public static final java.lang.String SPARK_CLUSTER_CONFIG

See Also:

Constant Field Values

SPARK_LEADER_CONFIG

public static final java.lang.String SPARK_LEADER_CONFIG

See Also:

Constant Field Values

DEFAULT_SPARK_CLUSTER

public static final java.lang.String DEFAULT_SPARK_CLUSTER

See Also:

Constant Field Values

SPARK_DATA_WRITER_CONF_PREFIX

public static final java.lang.String SPARK_DATA_WRITER_CONF_PREFIX

Configs with this prefix will be set when building the data writer spark job and passed as job properties. These will only get applied on the DataWriter Spark jobs. It is useful when there are custom input formats which need additional configs to be able to read the data.

See Also:

Constant Field Values

Constructor Detail

SparkConstants

public SparkConstants()