Package com.linkedin.venice.serialization.avro

Enum Class AvroProtocolDefinition

java.lang.Object

java.lang.Enum<AvroProtocolDefinition>

com.linkedin.venice.serialization.avro.AvroProtocolDefinition

All Implemented Interfaces:

Serializable, Comparable<AvroProtocolDefinition>, Constable

public enum AvroProtocolDefinition
extends Enum<AvroProtocolDefinition>

This enum lays out the basic specs of the various stateful protocols used in Venice. Having these definitions in a single place makes it easy to ensure that magic bytes are defined only once and do not conflict with each other.

Nested Class Summary

Nested classes/interfaces inherited from class java.lang.Enum

Enum.EnumDesc<E extends Enum<E>>

Enum Constant Summary

Enum Constants

Enum Constant	Description
ADMIN_OPERATION	Used to encode metadata changes about the system as a whole.
BATCH_JOB_HEARTBEAT
CHUNK	Single chunk of a large multi-chunk value.
CHUNKED_KEY_SUFFIX	Suffix appended to the end of all keys in a store-version where chunking is enabled.
CHUNKED_VALUE_MANIFEST	Used to encode the manifest of a multi-chunk large value.
INGESTION_METRICS_REPORT	Used to encode metrics collected from ingestion task, that are reported backed from child process to Storage Node / Da Vinci backend.
INGESTION_STORAGE_METADATA	Used to encode storage metadata updates that are reported backed from Storage Node / Da Vinci backend to child process.
INGESTION_TASK_COMMAND	Used to encode various kinds of ingestion task commands, which are used to control ingestion task in child process.
INGESTION_TASK_REPORT	Used to encode status of ingestion task, that are reported backed from child process to Storage Node / Da Vinci backend.
KAFKA_MESSAGE_ENVELOPE	Used for the Kafka topics, including the main data topics as well as the admin topic.
LOADED_STORE_USER_PARTITION_MAPPING	Used to retrieve the loaded store partition mapping in the isolated process.
METADATA_SYSTEM_SCHEMA_STORE	Value schema for metadata system store.
METADATA_SYSTEM_SCHEMA_STORE_KEY	Key schema for metadata system store.
PARTICIPANT_MESSAGE_SYSTEM_STORE_VALUE	Value schema for participant system stores.
PARTITION_STATE	Used to persist the state of a partition in Storage Nodes, including offset, Data Ingest Validation state, etc.
PROCESS_SHUTDOWN_COMMAND	Used to encode various kinds of ingestion task commands, which are used to control ingestion task in child process.
PUBSUB_POSITION_WIRE_FORMAT	Used to encode the position of a PubSub message.
PUSH_JOB_DETAILS	Used to encode push job details records to be written to the PushJobDetails system store.
PUSH_STATUS_SYSTEM_SCHEMA_STORE	Value schema for push status system store.
PUSH_STATUS_SYSTEM_SCHEMA_STORE_KEY	Key schema for push status system store.
RECORD_CHANGE_EVENT	Value schema for change capture event.
SERVER_ADMIN_RESPONSE	Response record for admin request.
SERVER_METADATA_RESPONSE	Response record for metadata fetch request.
SERVER_STORE_PROPERTIES_RESPONSE	Response record for metadata by client fetch request.
STORE_VERSION_STATE	Used to persist state related to a store-version, including Start of Buffer Replay offsets and whether the input is sorted.

Field Summary

Fields

Modifier and Type	Field	Description
final Optional<Integer>	currentProtocolVersion	The protocol version that is currently in use.

Method Summary

Modifier and Type	Method	Description
String	getClassName()
int	getCurrentProtocolVersion()
org.apache.avro.Schema	getCurrentProtocolVersionSchema()
Optional<Byte>	getMagicByte()
<T extends org.apache.avro.specific.SpecificRecord> InternalAvroSpecificSerializer<T>	getSerializer()
String	getSystemStoreName()
static AvroProtocolDefinition	valueOf(String name)	Returns the enum constant of this class with the specified name.
static AvroProtocolDefinition[]	values()	Returns an array containing the constants of this enum class, in the order they are declared.

Methods inherited from class java.lang.Enum

clone, compareTo, describeConstable, equals, finalize, getDeclaringClass, hashCode, name, ordinal, toString, valueOf

Methods inherited from class java.lang.Object

getClass, notify, notifyAll, wait, wait, wait

Enum Constant Details

KAFKA_MESSAGE_ENVELOPE

public static final AvroProtocolDefinition KAFKA_MESSAGE_ENVELOPE

Used for the Kafka topics, including the main data topics as well as the admin topic.

PARTITION_STATE

public static final AvroProtocolDefinition PARTITION_STATE

Used to persist the state of a partition in Storage Nodes, including offset, Data Ingest Validation state, etc.

STORE_VERSION_STATE

public static final AvroProtocolDefinition STORE_VERSION_STATE

Used to persist state related to a store-version, including Start of Buffer Replay offsets and whether the input is sorted.

PUSH_JOB_DETAILS

public static final AvroProtocolDefinition PUSH_JOB_DETAILS

Used to encode push job details records to be written to the PushJobDetails system store.

ADMIN_OPERATION

public static final AvroProtocolDefinition ADMIN_OPERATION

Used to encode metadata changes about the system as a whole. Records of this type are propagated via the "admin channel" special Kafka topic. It is stored in the Put of a KafkaMessageEnvelope, and thus leverages the envelope for versioning. TODO: Move AdminOperation to venice-common module so that we can properly reference it here.

CHUNK

public static final AvroProtocolDefinition CHUNK

Single chunk of a large multi-chunk value. Just a bunch of bytes. Uses a negative protocol version in order to avoid clashing with user-defined schemas.

CHUNKED_VALUE_MANIFEST

public static final AvroProtocolDefinition CHUNKED_VALUE_MANIFEST

Used to encode the manifest of a multi-chunk large value. It is stored in the Put of a KafkaMessageEnvelope, and thus leverages the envelope for versioning. Uses a negative protocol version in order to avoid clashing with user-defined schemas.

CHUNKED_KEY_SUFFIX

public static final AvroProtocolDefinition CHUNKED_KEY_SUFFIX

Suffix appended to the end of all keys in a store-version where chunking is enabled. This protocol is actually un-evolvable.

INGESTION_TASK_COMMAND

public static final AvroProtocolDefinition INGESTION_TASK_COMMAND

Used to encode various kinds of ingestion task commands, which are used to control ingestion task in child process.

INGESTION_TASK_REPORT

public static final AvroProtocolDefinition INGESTION_TASK_REPORT

Used to encode status of ingestion task, that are reported backed from child process to Storage Node / Da Vinci backend.

INGESTION_METRICS_REPORT

public static final AvroProtocolDefinition INGESTION_METRICS_REPORT

Used to encode metrics collected from ingestion task, that are reported backed from child process to Storage Node / Da Vinci backend.

INGESTION_STORAGE_METADATA

public static final AvroProtocolDefinition INGESTION_STORAGE_METADATA

Used to encode storage metadata updates that are reported backed from Storage Node / Da Vinci backend to child process.

PROCESS_SHUTDOWN_COMMAND

public static final AvroProtocolDefinition PROCESS_SHUTDOWN_COMMAND

Used to encode various kinds of ingestion task commands, which are used to control ingestion task in child process.

BATCH_JOB_HEARTBEAT

public static final AvroProtocolDefinition BATCH_JOB_HEARTBEAT

PUBSUB_POSITION_WIRE_FORMAT

public static final AvroProtocolDefinition PUBSUB_POSITION_WIRE_FORMAT

Used to encode the position of a PubSub message.

LOADED_STORE_USER_PARTITION_MAPPING

public static final AvroProtocolDefinition LOADED_STORE_USER_PARTITION_MAPPING

Used to retrieve the loaded store partition mapping in the isolated process. In theory, we don't need to use magicByte for the communication with II process.

METADATA_SYSTEM_SCHEMA_STORE_KEY

public static final AvroProtocolDefinition METADATA_SYSTEM_SCHEMA_STORE_KEY

Key schema for metadata system store.

METADATA_SYSTEM_SCHEMA_STORE

public static final AvroProtocolDefinition METADATA_SYSTEM_SCHEMA_STORE

Value schema for metadata system store.

PUSH_STATUS_SYSTEM_SCHEMA_STORE_KEY

public static final AvroProtocolDefinition PUSH_STATUS_SYSTEM_SCHEMA_STORE_KEY

Key schema for push status system store.

PUSH_STATUS_SYSTEM_SCHEMA_STORE

public static final AvroProtocolDefinition PUSH_STATUS_SYSTEM_SCHEMA_STORE

Value schema for push status system store.

PARTICIPANT_MESSAGE_SYSTEM_STORE_VALUE

public static final AvroProtocolDefinition PARTICIPANT_MESSAGE_SYSTEM_STORE_VALUE

Value schema for participant system stores.

SERVER_ADMIN_RESPONSE

public static final AvroProtocolDefinition SERVER_ADMIN_RESPONSE

Response record for admin request.

SERVER_METADATA_RESPONSE

public static final AvroProtocolDefinition SERVER_METADATA_RESPONSE

Response record for metadata fetch request.

SERVER_STORE_PROPERTIES_RESPONSE

public static final AvroProtocolDefinition SERVER_STORE_PROPERTIES_RESPONSE

Response record for metadata by client fetch request.

RECORD_CHANGE_EVENT

public static final AvroProtocolDefinition RECORD_CHANGE_EVENT

Value schema for change capture event. TODO: Figure out a way to pull in protocol from different view class.

Field Details

currentProtocolVersion

public final Optional<Integer> currentProtocolVersion

The protocol version that is currently in use. Typically, this should be the greatest number available, but that may not necessarily be true if some new experimental version of a protocol is being designed and is not yet rolled out.

Method Details

values

public static AvroProtocolDefinition[] values()

Returns an array containing the constants of this enum class, in the order they are declared.

Returns:

an array containing the constants of this enum class, in the order they are declared

valueOf

public static AvroProtocolDefinition valueOf(String name)

Returns the enum constant of this class with the specified name. The string must match exactly an identifier used to declare an enum constant in this class. (Extraneous whitespace characters are not permitted.)

Parameters:

name - the name of the enum constant to be returned.

Returns:

the enum constant with the specified name

Throws:

IllegalArgumentException - if this enum class has no constant with the specified name

NullPointerException - if the argument is null

getSerializer

public <T extends org.apache.avro.specific.SpecificRecord>
InternalAvroSpecificSerializer<T> getSerializer()

getCurrentProtocolVersion

public int getCurrentProtocolVersion()

getCurrentProtocolVersionSchema

public org.apache.avro.Schema getCurrentProtocolVersionSchema()

getMagicByte

public Optional<Byte> getMagicByte()

getClassName

public String getClassName()

getSystemStoreName

public String getSystemStoreName()