Package com.linkedin.venice.spark

Class SparkConstants

java.lang.Object
com.linkedin.venice.spark.SparkConstants
public class SparkConstants extends Object

  • Field Details

    • KEY_COLUMN_NAME

      public static final String KEY_COLUMN_NAME
      See Also:

    • VALUE_COLUMN_NAME

      public static final String VALUE_COLUMN_NAME
      See Also:

    • RMD_COLUMN_NAME

      public static final String RMD_COLUMN_NAME
      See Also:

    • PARTITION_COLUMN_NAME

      public static final String PARTITION_COLUMN_NAME
      See Also:

    • RECORD_COUNT_COLUMN_NAME

      public static final String RECORD_COUNT_COLUMN_NAME
      See Also:

    • FAILED_EXTERNAL_STORAGE_REGIONS_COLUMN_NAME

      public static final String FAILED_EXTERNAL_STORAGE_REGIONS_COLUMN_NAME
      See Also:

    • EXTERNAL_STORAGE_WRITE_TIME_MS_COLUMN_NAME

      public static final String EXTERNAL_STORAGE_WRITE_TIME_MS_COLUMN_NAME
      See Also:

    • VENICE_WRITE_TIME_MS_COLUMN_NAME

      public static final String VENICE_WRITE_TIME_MS_COLUMN_NAME
      See Also:

    • SCHEMA_ID_COLUMN_NAME

      public static final String SCHEMA_ID_COLUMN_NAME
      See Also:

    • RMD_VERSION_ID_COLUMN_NAME

      public static final String RMD_VERSION_ID_COLUMN_NAME
      See Also:

    • OFFSET_COLUMN_NAME

      public static final String OFFSET_COLUMN_NAME
      See Also:

    • MESSAGE_TYPE_COLUMN_NAME

      public static final String MESSAGE_TYPE_COLUMN_NAME
      See Also:

    • CHUNKED_KEY_SUFFIX_COLUMN_NAME

      public static final String CHUNKED_KEY_SUFFIX_COLUMN_NAME
      See Also:

    • DEFAULT_SCHEMA

      public static final org.apache.spark.sql.types.StructType DEFAULT_SCHEMA

    • PARTITION_RECORD_COUNT_SCHEMA

      public static final org.apache.spark.sql.types.StructType PARTITION_RECORD_COUNT_SCHEMA
      Task output emitted once per Spark partition by the partition writer. Everything the driver needs from a data-writer task that must not be collected via accumulators travels through these columns: Spark speculative execution can run two attempts for the same partition and accumulator updates from both attempts are visible on the driver, which would double count. Exactly one successful task output row per partition survives collect(), so the row-based values stay exact.

      The two timing columns are per-task wall-clock durations (see DataWriterTaskTracker.trackExternalStorageWriteTime(long)), summed by the driver across partitions. They are a sum of task durations, not the push's wall-clock time.

    • DEFAULT_SCHEMA_WITH_PARTITION

      public static final org.apache.spark.sql.types.StructType DEFAULT_SCHEMA_WITH_PARTITION

    • DEFAULT_SCHEMA_WITH_SCHEMA_ID

      public static final org.apache.spark.sql.types.StructType DEFAULT_SCHEMA_WITH_SCHEMA_ID

    • SCHEMA_FOR_CHUNK_ASSEMBLY

      public static final org.apache.spark.sql.types.StructType SCHEMA_FOR_CHUNK_ASSEMBLY

    • SPARK_SESSION_CONF_PREFIX

      public static final String SPARK_SESSION_CONF_PREFIX
      Configs with this prefix will be set when building the spark session. These will get applied to all Spark jobs that get triggered as a part of VPJ. It can be used to configure arbitrary cluster properties like cluster address.
      See Also:

    • SPARK_APP_NAME_CONFIG

      public static final String SPARK_APP_NAME_CONFIG
      See Also:

    • SPARK_CASE_SENSITIVE_CONFIG

      public static final String SPARK_CASE_SENSITIVE_CONFIG
      See Also:

    • SPARK_CLUSTER_CONFIG

      public static final String SPARK_CLUSTER_CONFIG
      See Also:

    • SPARK_LEADER_CONFIG

      public static final String SPARK_LEADER_CONFIG
      See Also:

    • DEFAULT_SPARK_CLUSTER

      public static final String DEFAULT_SPARK_CLUSTER
      See Also:

    • SPARK_DATA_WRITER_CONF_PREFIX

      public static final String SPARK_DATA_WRITER_CONF_PREFIX
      Configs with this prefix will be set when building the data writer spark job and passed as job properties. These will only get applied on the DataWriter Spark jobs. It is useful when there are custom input formats which need additional configs to be able to read the data.
      See Also:

    • REPLICATION_METADATA_PAYLOAD

      public static final String REPLICATION_METADATA_PAYLOAD
      See Also:

    • MESSAGE_TYPE

      public static final String MESSAGE_TYPE
      See Also:

    • OFFSET

      public static final String OFFSET
      See Also:

    • RAW_PUBSUB_INPUT_TABLE_SCHEMA

      public static final org.apache.spark.sql.types.StructType RAW_PUBSUB_INPUT_TABLE_SCHEMA

  • Constructor Details

    • SparkConstants

      public SparkConstants()