Package com.linkedin.venice.pubsub.api

Class PubSubMessageHeaders

java.lang.Object

com.linkedin.venice.pubsub.api.PubSubMessageHeaders

All Implemented Interfaces:

Measurable, Iterable<PubSubMessageHeader>

Direct Known Subclasses:

EmptyPubSubMessageHeaders

public class PubSubMessageHeaders
extends Object
implements Measurable, Iterable<PubSubMessageHeader>

Set of key-value pairs to tagged with messages produced to a topic. In case of headers with the same key, only the most recently added headers value will be kept.

Field Summary

Fields

Modifier and Type	Field	Description
static final String	EXECUTION_ID_KEY
static final long	PRC_HEADER_UNAVAILABLE_SENTINEL	Sentinel value stamped into VENICE_PARTITION_RECORD_COUNT_HEADER when the producer cannot supply a count (e.g.
static final String	VENICE_LEADER_COMPLETION_STATE_HEADER	Header to denote whether the leader is completed or not
static final String	VENICE_PARTITION_RECORD_COUNT_HEADER	Header carrying per-partition record count on End-of-Push control messages.
static final String	VENICE_TRANSPORT_PROTOCOL_HEADER
static final String	VENICE_VIEW_PARTITIONS_MAP_HEADER	Header to provide the desired view partition mapping for the given message.

Constructor Summary

Constructors

Constructor	Description
PubSubMessageHeaders()

Method Summary

Modifier and Type	Method	Description
PubSubMessageHeaders	add(PubSubMessageHeader header)
PubSubMessageHeaders	add(String key, byte[] value)
PubSubMessageHeader	get(String key)
int	getHeapSize()
boolean	isEmpty()
Iterator<PubSubMessageHeader>	iterator()
PubSubMessageHeaders	remove(String key)
static PubSubMessageHeaders	stripProtocolSchemaHeader(PubSubMessageHeaders headers)	Returns a PubSubMessageHeaders without the VENICE_TRANSPORT_PROTOCOL_HEADER (a.k.a.
static PubSubMessageHeaders	stripProtocolSchemaHeaderCopy(PubSubMessageHeaders headers)	Always-copy variant of stripProtocolSchemaHeader(com.linkedin.venice.pubsub.api.PubSubMessageHeaders).
List<PubSubMessageHeader>	toList()

Methods inherited from class java.lang.Object

clone, equals, finalize, getClass, hashCode, notify, notifyAll, toString, wait, wait, wait

Methods inherited from interface java.lang.Iterable

forEach, spliterator

Field Details

EXECUTION_ID_KEY

public static final String EXECUTION_ID_KEY

See Also:

• Constant Field Values

VENICE_TRANSPORT_PROTOCOL_HEADER

public static final String VENICE_TRANSPORT_PROTOCOL_HEADER

See Also:

• Constant Field Values

VENICE_LEADER_COMPLETION_STATE_HEADER

public static final String VENICE_LEADER_COMPLETION_STATE_HEADER

Header to denote whether the leader is completed or not

See Also:

• Constant Field Values

VENICE_VIEW_PARTITIONS_MAP_HEADER

public static final String VENICE_VIEW_PARTITIONS_MAP_HEADER

Header to provide the desired view partition mapping for the given message. Example usage: 1. A VPJ containing large messages write chunks with the header {"view1":[0], "view2":[1, 2]}. Partition leaders during NR pass-through in remote fabrics can forward chunks to their destination view partition(s) without any further processing. In the example, this chunk should be sent to view1's partition 0 and view2's partitions 1 & 2.

See Also:

• Constant Field Values

VENICE_PARTITION_RECORD_COUNT_HEADER

public static final String VENICE_PARTITION_RECORD_COUNT_HEADER

Header carrying per-partition record count on End-of-Push control messages. Value is an 8-byte big-endian encoded long representing the number of records written to that partition. Used for server-side batch push verification.

See Also:

• Constant Field Values

PRC_HEADER_UNAVAILABLE_SENTINEL

public static final long PRC_HEADER_UNAVAILABLE_SENTINEL

Sentinel value stamped into VENICE_PARTITION_RECORD_COUNT_HEADER when the producer cannot supply a count (e.g. legacy code paths, partition-level fallback). Consumers must treat a header carrying this value as "count unavailable" and skip the EOP record-count verification rather than reading -1 as a literal record count.

See Also:

• Constant Field Values

Constructor Details

PubSubMessageHeaders

public PubSubMessageHeaders()

Method Details

add

public PubSubMessageHeaders add(PubSubMessageHeader header)

add

public PubSubMessageHeaders add(String key,
 byte[] value)

remove

public PubSubMessageHeaders remove(String key)

get

public PubSubMessageHeader get(String key)

toList

public List<PubSubMessageHeader> toList()

Returns:

the headers as a List. Mutating this list will not affect the PubSubMessageHeaders. If no headers are present an empty list is returned.

isEmpty

public boolean isEmpty()

stripProtocolSchemaHeader

public static PubSubMessageHeaders stripProtocolSchemaHeader(PubSubMessageHeaders headers)

Returns a PubSubMessageHeaders without the VENICE_TRANSPORT_PROTOCOL_HEADER (a.k.a. vtp). The vtp value is the entire ~16 KB Avro JSON for KafkaMessageEnvelope; once the value envelope has been deserialized it is dead weight and pinning it per queued record has been observed to cost upwards of 10 GB on the DaVinci ingestion buffer queue during back-pressure.

Best-effort: when vtp is absent the input is returned as-is (no allocation). When present, the helper attempts an in-place remove — allocation-free for the production hot path where callers construct fresh mutable headers per record. If remove() throws any RuntimeException (e.g. UnsupportedOperationException from an immutable variant), the helper falls back to building a new headers object without vtp. Never throws — the strip is on the deserialization hot path, where an escaping exception would kill the partition's ingestion thread.

Use stripProtocolSchemaHeaderCopy(com.linkedin.venice.pubsub.api.PubSubMessageHeaders) instead when the caller's headers reference is shared across reads (e.g. an in-memory broker that re-serves the same message), since silent in-place mutation would corrupt subsequent observers.

stripProtocolSchemaHeaderCopy

public static PubSubMessageHeaders stripProtocolSchemaHeaderCopy(PubSubMessageHeaders headers)

Always-copy variant of stripProtocolSchemaHeader(com.linkedin.venice.pubsub.api.PubSubMessageHeaders). Returns the input unchanged when vtp is absent (no allocation); otherwise returns a fresh PubSubMessageHeaders omitting vtp and leaves the input untouched. Use this from call sites where the headers reference is shared across reads.

getHeapSize

public int getHeapSize()

Specified by:

getHeapSize in interface Measurable

iterator

public Iterator<PubSubMessageHeader> iterator()

Specified by:

iterator in interface Iterable<PubSubMessageHeader>