Package com.linkedin.venice.kafka.protocol.state

Class PartitionState

java.lang.Object
org.apache.avro.specific.SpecificRecordBase
com.linkedin.venice.kafka.protocol.state.PartitionState
All Implemented Interfaces:
Externalizable, Serializable, Comparable<org.apache.avro.specific.SpecificRecord>, org.apache.avro.generic.GenericContainer, org.apache.avro.generic.GenericRecord, org.apache.avro.generic.IndexedRecord, org.apache.avro.specific.SpecificRecord
public class PartitionState extends org.apache.avro.specific.SpecificRecordBase implements org.apache.avro.specific.SpecificRecord
This record holds the state necessary for a consumer to checkpoint its progress when consuming a Venice partition. When provided the state in this record, a consumer should thus be able to resume consuming midway through a stream.
See Also:

  • Field Details

    • SCHEMA$

      public static final org.apache.avro.Schema SCHEMA$

    • offset

      public long offset
      The last Kafka offset consumed successfully in this partition from version topic.

    • offsetLag

      public long offsetLag
      The last Kafka offset lag in this partition for fast online transition in server restart.

    • endOfPush

      public boolean endOfPush
      Whether the EndOfPush control message was consumed in this partition.

    • lastUpdate

      public long lastUpdate
      The last time this PartitionState was updated. Can be compared against the various messageTimestamp in ProducerPartitionState in order to infer lag time between producers and the consumer maintaining this PartitionState.

    • startOfBufferReplayDestinationOffset

      public Long startOfBufferReplayDestinationOffset
      This is the offset at which the StartOfBufferReplay control message was consumed in the current partition of the destination topic. This is not the same value as the source offsets contained in the StartOfBufferReplay control message itself. The source and destination offsets act together as a synchronization marker. N.B.: null means that the SOBR control message was not received yet.

    • databaseInfo

      public Map<CharSequence,CharSequence> databaseInfo
      A map of string -> string to store database related info, which is necessary to checkpoint

    • incrementalPushInfo

      public IncrementalPush incrementalPushInfo
      metadata of ongoing incremental push in the partition

    • leaderTopic

      public CharSequence leaderTopic
      The topic that leader is consuming from; for leader, leaderTopic can be different from the version topic; for follower, leaderTopic is the same as version topic.

    • leaderOffset

      public long leaderOffset
      The last Kafka offset consumed successfully in this partition from the leader topic. TODO: remove this field once upstreamOffsetMap is used everywhere.

    • upstreamOffsetMap

      public Map<String,Long> upstreamOffsetMap
      A map of upstream Kafka bootstrap server url -> the last Kafka offset consumed from upstream topic.

    • upstreamVersionTopicOffset

      public long upstreamVersionTopicOffset
      The last upstream version topic offset persisted to disk; if the batch native-replication source is the same as local region, this value will always be -1

    • leaderGUID

      public GUID leaderGUID
      This field is deprecated since GUID is no longer able to identify the split-brain issue once 'pass-through' mode is enabled in venice writer. The field is superseded by leaderHostId and will be removed in the future

    • leaderHostId

      public CharSequence leaderHostId
      An unique identifier (such as host name) that stands for each host. It's used to identify if there is a split-brain happened while the leader(s) re-produce records

    • producerStates

      public Map<CharSequence,ProducerPartitionState> producerStates
      A map of producer GUID -> producer state.

    • previousStatuses

      public Map<CharSequence,CharSequence> previousStatuses
      A map of string -> string which stands for previous PartitionStatus

    • pendingReportIncrementalPushVersions

      public List<CharSequence> pendingReportIncrementalPushVersions
      A list of string which stands for incremental push versions which have received EOIP but not yet reported prior to lag caught up, they will be reported in batch

    • realtimeTopicProducerStates

      public Map<String,Map<CharSequence,ProducerPartitionState>> realtimeTopicProducerStates
      A map that maps upstream Kafka bootstrap server url -> to a map of producer GUID -> producer state for real-time data.

  • Constructor Details

    • PartitionState

      public PartitionState()
      Default constructor. Note that this does not initialize fields to their default values from the schema. If that is desired then one should use newBuilder().

    • PartitionState

      public PartitionState(Long offset, Long offsetLag, Boolean endOfPush, Long lastUpdate, Long startOfBufferReplayDestinationOffset, Map<CharSequence,CharSequence> databaseInfo, IncrementalPush incrementalPushInfo, CharSequence leaderTopic, Long leaderOffset, Map<String,Long> upstreamOffsetMap, Long upstreamVersionTopicOffset, GUID leaderGUID, CharSequence leaderHostId, Map<CharSequence,ProducerPartitionState> producerStates, Map<CharSequence,CharSequence> previousStatuses, List<CharSequence> pendingReportIncrementalPushVersions, Map<String,Map<CharSequence,ProducerPartitionState>> realtimeTopicProducerStates)
      All-args constructor.
      Parameters:
      offset - The last Kafka offset consumed successfully in this partition from version topic.
      offsetLag - The last Kafka offset lag in this partition for fast online transition in server restart.
      endOfPush - Whether the EndOfPush control message was consumed in this partition.
      lastUpdate - The last time this PartitionState was updated. Can be compared against the various messageTimestamp in ProducerPartitionState in order to infer lag time between producers and the consumer maintaining this PartitionState.
      startOfBufferReplayDestinationOffset - This is the offset at which the StartOfBufferReplay control message was consumed in the current partition of the destination topic. This is not the same value as the source offsets contained in the StartOfBufferReplay control message itself. The source and destination offsets act together as a synchronization marker. N.B.: null means that the SOBR control message was not received yet.
      databaseInfo - A map of string -> string to store database related info, which is necessary to checkpoint
      incrementalPushInfo - metadata of ongoing incremental push in the partition
      leaderTopic - The topic that leader is consuming from; for leader, leaderTopic can be different from the version topic; for follower, leaderTopic is the same as version topic.
      leaderOffset - The last Kafka offset consumed successfully in this partition from the leader topic. TODO: remove this field once upstreamOffsetMap is used everywhere.
      upstreamOffsetMap - A map of upstream Kafka bootstrap server url -> the last Kafka offset consumed from upstream topic.
      upstreamVersionTopicOffset - The last upstream version topic offset persisted to disk; if the batch native-replication source is the same as local region, this value will always be -1
      leaderGUID - This field is deprecated since GUID is no longer able to identify the split-brain issue once 'pass-through' mode is enabled in venice writer. The field is superseded by leaderHostId and will be removed in the future
      leaderHostId - An unique identifier (such as host name) that stands for each host. It's used to identify if there is a split-brain happened while the leader(s) re-produce records
      producerStates - A map of producer GUID -> producer state.
      previousStatuses - A map of string -> string which stands for previous PartitionStatus
      pendingReportIncrementalPushVersions - A list of string which stands for incremental push versions which have received EOIP but not yet reported prior to lag caught up, they will be reported in batch
      realtimeTopicProducerStates - A map that maps upstream Kafka bootstrap server url -> to a map of producer GUID -> producer state for real-time data.

  • Method Details

    • getClassSchema

      public static org.apache.avro.Schema getClassSchema()

    • getSpecificData

      public org.apache.avro.specific.SpecificData getSpecificData()
      Overrides:
      getSpecificData in class org.apache.avro.specific.SpecificRecordBase

    • getSchema

      public org.apache.avro.Schema getSchema()
      Specified by:
      getSchema in interface org.apache.avro.generic.GenericContainer
      Specified by:
      getSchema in class org.apache.avro.specific.SpecificRecordBase

    • get

      public Object get(int field$)
      Specified by:
      get in interface org.apache.avro.generic.IndexedRecord
      Specified by:
      get in class org.apache.avro.specific.SpecificRecordBase

    • put

      public void put(int field$, Object value$)
      Specified by:
      put in interface org.apache.avro.generic.IndexedRecord
      Specified by:
      put in class org.apache.avro.specific.SpecificRecordBase

    • getOffset

      public long getOffset()
      Gets the value of the 'offset' field.
      Returns:
      The last Kafka offset consumed successfully in this partition from version topic.

    • setOffset

      public void setOffset(long value)
      Sets the value of the 'offset' field. The last Kafka offset consumed successfully in this partition from version topic.
      Parameters:
      value - the value to set.

    • getOffsetLag

      public long getOffsetLag()
      Gets the value of the 'offsetLag' field.
      Returns:
      The last Kafka offset lag in this partition for fast online transition in server restart.

    • setOffsetLag

      public void setOffsetLag(long value)
      Sets the value of the 'offsetLag' field. The last Kafka offset lag in this partition for fast online transition in server restart.
      Parameters:
      value - the value to set.

    • getEndOfPush

      public boolean getEndOfPush()
      Gets the value of the 'endOfPush' field.
      Returns:
      Whether the EndOfPush control message was consumed in this partition.

    • setEndOfPush

      public void setEndOfPush(boolean value)
      Sets the value of the 'endOfPush' field. Whether the EndOfPush control message was consumed in this partition.
      Parameters:
      value - the value to set.

    • getLastUpdate

      public long getLastUpdate()
      Gets the value of the 'lastUpdate' field.
      Returns:
      The last time this PartitionState was updated. Can be compared against the various messageTimestamp in ProducerPartitionState in order to infer lag time between producers and the consumer maintaining this PartitionState.

    • setLastUpdate

      public void setLastUpdate(long value)
      Sets the value of the 'lastUpdate' field. The last time this PartitionState was updated. Can be compared against the various messageTimestamp in ProducerPartitionState in order to infer lag time between producers and the consumer maintaining this PartitionState.
      Parameters:
      value - the value to set.

    • getStartOfBufferReplayDestinationOffset

      public Long getStartOfBufferReplayDestinationOffset()
      Gets the value of the 'startOfBufferReplayDestinationOffset' field.
      Returns:
      This is the offset at which the StartOfBufferReplay control message was consumed in the current partition of the destination topic. This is not the same value as the source offsets contained in the StartOfBufferReplay control message itself. The source and destination offsets act together as a synchronization marker. N.B.: null means that the SOBR control message was not received yet.

    • setStartOfBufferReplayDestinationOffset

      public void setStartOfBufferReplayDestinationOffset(Long value)
      Sets the value of the 'startOfBufferReplayDestinationOffset' field. This is the offset at which the StartOfBufferReplay control message was consumed in the current partition of the destination topic. This is not the same value as the source offsets contained in the StartOfBufferReplay control message itself. The source and destination offsets act together as a synchronization marker. N.B.: null means that the SOBR control message was not received yet.
      Parameters:
      value - the value to set.

    • getDatabaseInfo

      public Map<CharSequence,CharSequence> getDatabaseInfo()
      Gets the value of the 'databaseInfo' field.
      Returns:
      A map of string -> string to store database related info, which is necessary to checkpoint

    • setDatabaseInfo

      public void setDatabaseInfo(Map<CharSequence,CharSequence> value)
      Sets the value of the 'databaseInfo' field. A map of string -> string to store database related info, which is necessary to checkpoint
      Parameters:
      value - the value to set.

    • getIncrementalPushInfo

      public IncrementalPush getIncrementalPushInfo()
      Gets the value of the 'incrementalPushInfo' field.
      Returns:
      metadata of ongoing incremental push in the partition

    • setIncrementalPushInfo

      public void setIncrementalPushInfo(IncrementalPush value)
      Sets the value of the 'incrementalPushInfo' field. metadata of ongoing incremental push in the partition
      Parameters:
      value - the value to set.

    • getLeaderTopic

      public CharSequence getLeaderTopic()
      Gets the value of the 'leaderTopic' field.
      Returns:
      The topic that leader is consuming from; for leader, leaderTopic can be different from the version topic; for follower, leaderTopic is the same as version topic.

    • setLeaderTopic

      public void setLeaderTopic(CharSequence value)
      Sets the value of the 'leaderTopic' field. The topic that leader is consuming from; for leader, leaderTopic can be different from the version topic; for follower, leaderTopic is the same as version topic.
      Parameters:
      value - the value to set.

    • getLeaderOffset

      public long getLeaderOffset()
      Gets the value of the 'leaderOffset' field.
      Returns:
      The last Kafka offset consumed successfully in this partition from the leader topic. TODO: remove this field once upstreamOffsetMap is used everywhere.

    • setLeaderOffset

      public void setLeaderOffset(long value)
      Sets the value of the 'leaderOffset' field. The last Kafka offset consumed successfully in this partition from the leader topic. TODO: remove this field once upstreamOffsetMap is used everywhere.
      Parameters:
      value - the value to set.

    • getUpstreamOffsetMap

      public Map<String,Long> getUpstreamOffsetMap()
      Gets the value of the 'upstreamOffsetMap' field.
      Returns:
      A map of upstream Kafka bootstrap server url -> the last Kafka offset consumed from upstream topic.

    • setUpstreamOffsetMap

      public void setUpstreamOffsetMap(Map<String,Long> value)
      Sets the value of the 'upstreamOffsetMap' field. A map of upstream Kafka bootstrap server url -> the last Kafka offset consumed from upstream topic.
      Parameters:
      value - the value to set.

    • getUpstreamVersionTopicOffset

      public long getUpstreamVersionTopicOffset()
      Gets the value of the 'upstreamVersionTopicOffset' field.
      Returns:
      The last upstream version topic offset persisted to disk; if the batch native-replication source is the same as local region, this value will always be -1

    • setUpstreamVersionTopicOffset

      public void setUpstreamVersionTopicOffset(long value)
      Sets the value of the 'upstreamVersionTopicOffset' field. The last upstream version topic offset persisted to disk; if the batch native-replication source is the same as local region, this value will always be -1
      Parameters:
      value - the value to set.

    • getLeaderGUID

      public GUID getLeaderGUID()
      Gets the value of the 'leaderGUID' field.
      Returns:
      This field is deprecated since GUID is no longer able to identify the split-brain issue once 'pass-through' mode is enabled in venice writer. The field is superseded by leaderHostId and will be removed in the future

    • setLeaderGUID

      public void setLeaderGUID(GUID value)
      Sets the value of the 'leaderGUID' field. This field is deprecated since GUID is no longer able to identify the split-brain issue once 'pass-through' mode is enabled in venice writer. The field is superseded by leaderHostId and will be removed in the future
      Parameters:
      value - the value to set.

    • getLeaderHostId

      public CharSequence getLeaderHostId()
      Gets the value of the 'leaderHostId' field.
      Returns:
      An unique identifier (such as host name) that stands for each host. It's used to identify if there is a split-brain happened while the leader(s) re-produce records

    • setLeaderHostId

      public void setLeaderHostId(CharSequence value)
      Sets the value of the 'leaderHostId' field. An unique identifier (such as host name) that stands for each host. It's used to identify if there is a split-brain happened while the leader(s) re-produce records
      Parameters:
      value - the value to set.

    • getProducerStates

      public Map<CharSequence,ProducerPartitionState> getProducerStates()
      Gets the value of the 'producerStates' field.
      Returns:
      A map of producer GUID -> producer state.

    • setProducerStates

      public void setProducerStates(Map<CharSequence,ProducerPartitionState> value)
      Sets the value of the 'producerStates' field. A map of producer GUID -> producer state.
      Parameters:
      value - the value to set.

    • getPreviousStatuses

      public Map<CharSequence,CharSequence> getPreviousStatuses()
      Gets the value of the 'previousStatuses' field.
      Returns:
      A map of string -> string which stands for previous PartitionStatus

    • setPreviousStatuses

      public void setPreviousStatuses(Map<CharSequence,CharSequence> value)
      Sets the value of the 'previousStatuses' field. A map of string -> string which stands for previous PartitionStatus
      Parameters:
      value - the value to set.

    • getPendingReportIncrementalPushVersions

      public List<CharSequence> getPendingReportIncrementalPushVersions()
      Gets the value of the 'pendingReportIncrementalPushVersions' field.
      Returns:
      A list of string which stands for incremental push versions which have received EOIP but not yet reported prior to lag caught up, they will be reported in batch

    • setPendingReportIncrementalPushVersions

      public void setPendingReportIncrementalPushVersions(List<CharSequence> value)
      Sets the value of the 'pendingReportIncrementalPushVersions' field. A list of string which stands for incremental push versions which have received EOIP but not yet reported prior to lag caught up, they will be reported in batch
      Parameters:
      value - the value to set.

    • getRealtimeTopicProducerStates

      public Map<String,Map<CharSequence,ProducerPartitionState>> getRealtimeTopicProducerStates()
      Gets the value of the 'realtimeTopicProducerStates' field.
      Returns:
      A map that maps upstream Kafka bootstrap server url -> to a map of producer GUID -> producer state for real-time data.

    • setRealtimeTopicProducerStates

      public void setRealtimeTopicProducerStates(Map<String,Map<CharSequence,ProducerPartitionState>> value)
      Sets the value of the 'realtimeTopicProducerStates' field. A map that maps upstream Kafka bootstrap server url -> to a map of producer GUID -> producer state for real-time data.
      Parameters:
      value - the value to set.

    • writeExternal

      public void writeExternal(ObjectOutput out) throws IOException
      Specified by:
      writeExternal in interface Externalizable
      Overrides:
      writeExternal in class org.apache.avro.specific.SpecificRecordBase
      Throws:
      IOException

    • readExternal

      public void readExternal(ObjectInput in) throws IOException
      Specified by:
      readExternal in interface Externalizable
      Overrides:
      readExternal in class org.apache.avro.specific.SpecificRecordBase
      Throws:
      IOException