Package com.linkedin.venice.kafka.protocol.state

Class PartitionState

  • All Implemented Interfaces:
    java.io.Externalizable, java.io.Serializable, java.lang.Comparable<org.apache.avro.specific.SpecificRecord>, org.apache.avro.generic.GenericContainer, org.apache.avro.generic.GenericRecord, org.apache.avro.generic.IndexedRecord, org.apache.avro.specific.SpecificRecord
    public class PartitionState
extends org.apache.avro.specific.SpecificRecordBase
implements org.apache.avro.specific.SpecificRecord
    This record holds the state necessary for a consumer to checkpoint its progress when consuming a Venice partition. When provided the state in this record, a consumer should thus be able to resume consuming midway through a stream.
    See Also:
    Serialized Form

    • Field Summary

      Fields 
      Modifier and Type Field Description
      java.util.Map<java.lang.CharSequence,​java.lang.CharSequence> databaseInfo
      A map of string -> string to store database related info, which is necessary to checkpoint
      boolean endOfPush
      Whether the EndOfPush control message was consumed in this partition.
      IncrementalPush incrementalPushInfo
      metadata of ongoing incremental push in the partition
      long lastUpdate
      The last time this PartitionState was updated.
      GUID leaderGUID
      This field is deprecated since GUID is no longer able to identify the split-brain issue once 'pass-through' mode is enabled in venice writer.
      java.lang.CharSequence leaderHostId
      An unique identifier (such as host name) that stands for each host.
      long leaderOffset
      The last Kafka offset consumed successfully in this partition from the leader topic.
      java.lang.CharSequence leaderTopic
      The topic that leader is consuming from; for leader, leaderTopic can be different from the version topic; for follower, leaderTopic is the same as version topic.
      long offset
      The last Kafka offset consumed successfully in this partition from version topic.
      long offsetLag
      The last Kafka offset lag in this partition for fast online transition in server restart.
      java.util.List<java.lang.CharSequence> pendingReportIncrementalPushVersions
      A list of string which stands for incremental push versions which have received EOIP but not yet reported prior to lag caught up, they will be reported in batch
      java.util.Map<java.lang.CharSequence,​java.lang.CharSequence> previousStatuses
      A map of string -> string which stands for previous PartitionStatus
      java.util.Map<java.lang.CharSequence,​ProducerPartitionState> producerStates
      A map of producer GUID -> producer state.
      java.util.Map<java.lang.String,​java.util.Map<java.lang.CharSequence,​ProducerPartitionState>> realtimeTopicProducerStates
      A map that maps upstream Kafka bootstrap server url -> to a map of producer GUID -> producer state for real-time data.
      static org.apache.avro.Schema SCHEMA$  
      java.lang.Long startOfBufferReplayDestinationOffset
      This is the offset at which the StartOfBufferReplay control message was consumed in the current partition of the destination topic.
      java.util.Map<java.lang.String,​java.lang.Long> upstreamOffsetMap
      A map of upstream Kafka bootstrap server url -> the last Kafka offset consumed from upstream topic.
      long upstreamVersionTopicOffset
      The last upstream version topic offset persisted to disk; if the batch native-replication source is the same as local region, this value will always be -1

    • Constructor Summary

      Constructors 
      Constructor Description
      PartitionState()
      Default constructor.
      PartitionState​(java.lang.Long offset, java.lang.Long offsetLag, java.lang.Boolean endOfPush, java.lang.Long lastUpdate, java.lang.Long startOfBufferReplayDestinationOffset, java.util.Map<java.lang.CharSequence,​java.lang.CharSequence> databaseInfo, IncrementalPush incrementalPushInfo, java.lang.CharSequence leaderTopic, java.lang.Long leaderOffset, java.util.Map<java.lang.String,​java.lang.Long> upstreamOffsetMap, java.lang.Long upstreamVersionTopicOffset, GUID leaderGUID, java.lang.CharSequence leaderHostId, java.util.Map<java.lang.CharSequence,​ProducerPartitionState> producerStates, java.util.Map<java.lang.CharSequence,​java.lang.CharSequence> previousStatuses, java.util.List<java.lang.CharSequence> pendingReportIncrementalPushVersions, java.util.Map<java.lang.String,​java.util.Map<java.lang.CharSequence,​ProducerPartitionState>> realtimeTopicProducerStates)
      All-args constructor.

    • Field Detail

      • SCHEMA$

        public static final org.apache.avro.Schema SCHEMA$

      • offset

        public long offset
        The last Kafka offset consumed successfully in this partition from version topic.

      • offsetLag

        public long offsetLag
        The last Kafka offset lag in this partition for fast online transition in server restart.

      • endOfPush

        public boolean endOfPush
        Whether the EndOfPush control message was consumed in this partition.

      • lastUpdate

        public long lastUpdate
        The last time this PartitionState was updated. Can be compared against the various messageTimestamp in ProducerPartitionState in order to infer lag time between producers and the consumer maintaining this PartitionState.

      • startOfBufferReplayDestinationOffset

        public java.lang.Long startOfBufferReplayDestinationOffset
        This is the offset at which the StartOfBufferReplay control message was consumed in the current partition of the destination topic. This is not the same value as the source offsets contained in the StartOfBufferReplay control message itself. The source and destination offsets act together as a synchronization marker. N.B.: null means that the SOBR control message was not received yet.

      • databaseInfo

        public java.util.Map<java.lang.CharSequence,​java.lang.CharSequence> databaseInfo
        A map of string -> string to store database related info, which is necessary to checkpoint

      • incrementalPushInfo

        public IncrementalPush incrementalPushInfo
        metadata of ongoing incremental push in the partition

      • leaderTopic

        public java.lang.CharSequence leaderTopic
        The topic that leader is consuming from; for leader, leaderTopic can be different from the version topic; for follower, leaderTopic is the same as version topic.

      • leaderOffset

        public long leaderOffset
        The last Kafka offset consumed successfully in this partition from the leader topic. TODO: remove this field once upstreamOffsetMap is used everywhere.

      • upstreamOffsetMap

        public java.util.Map<java.lang.String,​java.lang.Long> upstreamOffsetMap
        A map of upstream Kafka bootstrap server url -> the last Kafka offset consumed from upstream topic.

      • upstreamVersionTopicOffset

        public long upstreamVersionTopicOffset
        The last upstream version topic offset persisted to disk; if the batch native-replication source is the same as local region, this value will always be -1

      • leaderGUID

        public GUID leaderGUID
        This field is deprecated since GUID is no longer able to identify the split-brain issue once 'pass-through' mode is enabled in venice writer. The field is superseded by leaderHostId and will be removed in the future

      • leaderHostId

        public java.lang.CharSequence leaderHostId
        An unique identifier (such as host name) that stands for each host. It's used to identify if there is a split-brain happened while the leader(s) re-produce records

      • producerStates

        public java.util.Map<java.lang.CharSequence,​ProducerPartitionState> producerStates
        A map of producer GUID -> producer state.

      • previousStatuses

        public java.util.Map<java.lang.CharSequence,​java.lang.CharSequence> previousStatuses
        A map of string -> string which stands for previous PartitionStatus

      • pendingReportIncrementalPushVersions

        public java.util.List<java.lang.CharSequence> pendingReportIncrementalPushVersions
        A list of string which stands for incremental push versions which have received EOIP but not yet reported prior to lag caught up, they will be reported in batch

      • realtimeTopicProducerStates

        public java.util.Map<java.lang.String,​java.util.Map<java.lang.CharSequence,​ProducerPartitionState>> realtimeTopicProducerStates
        A map that maps upstream Kafka bootstrap server url -> to a map of producer GUID -> producer state for real-time data.

    • Constructor Detail

      • PartitionState

        public PartitionState()
        Default constructor. Note that this does not initialize fields to their default values from the schema. If that is desired then one should use newBuilder().

      • PartitionState

        public PartitionState​(java.lang.Long offset,
                      java.lang.Long offsetLag,
                      java.lang.Boolean endOfPush,
                      java.lang.Long lastUpdate,
                      java.lang.Long startOfBufferReplayDestinationOffset,
                      java.util.Map<java.lang.CharSequence,​java.lang.CharSequence> databaseInfo,
                      IncrementalPush incrementalPushInfo,
                      java.lang.CharSequence leaderTopic,
                      java.lang.Long leaderOffset,
                      java.util.Map<java.lang.String,​java.lang.Long> upstreamOffsetMap,
                      java.lang.Long upstreamVersionTopicOffset,
                      GUID leaderGUID,
                      java.lang.CharSequence leaderHostId,
                      java.util.Map<java.lang.CharSequence,​ProducerPartitionState> producerStates,
                      java.util.Map<java.lang.CharSequence,​java.lang.CharSequence> previousStatuses,
                      java.util.List<java.lang.CharSequence> pendingReportIncrementalPushVersions,
                      java.util.Map<java.lang.String,​java.util.Map<java.lang.CharSequence,​ProducerPartitionState>> realtimeTopicProducerStates)
        All-args constructor.
        Parameters:
        offset - The last Kafka offset consumed successfully in this partition from version topic.
        offsetLag - The last Kafka offset lag in this partition for fast online transition in server restart.
        endOfPush - Whether the EndOfPush control message was consumed in this partition.
        lastUpdate - The last time this PartitionState was updated. Can be compared against the various messageTimestamp in ProducerPartitionState in order to infer lag time between producers and the consumer maintaining this PartitionState.
        startOfBufferReplayDestinationOffset - This is the offset at which the StartOfBufferReplay control message was consumed in the current partition of the destination topic. This is not the same value as the source offsets contained in the StartOfBufferReplay control message itself. The source and destination offsets act together as a synchronization marker. N.B.: null means that the SOBR control message was not received yet.
        databaseInfo - A map of string -> string to store database related info, which is necessary to checkpoint
        incrementalPushInfo - metadata of ongoing incremental push in the partition
        leaderTopic - The topic that leader is consuming from; for leader, leaderTopic can be different from the version topic; for follower, leaderTopic is the same as version topic.
        leaderOffset - The last Kafka offset consumed successfully in this partition from the leader topic. TODO: remove this field once upstreamOffsetMap is used everywhere.
        upstreamOffsetMap - A map of upstream Kafka bootstrap server url -> the last Kafka offset consumed from upstream topic.
        upstreamVersionTopicOffset - The last upstream version topic offset persisted to disk; if the batch native-replication source is the same as local region, this value will always be -1
        leaderGUID - This field is deprecated since GUID is no longer able to identify the split-brain issue once 'pass-through' mode is enabled in venice writer. The field is superseded by leaderHostId and will be removed in the future
        leaderHostId - An unique identifier (such as host name) that stands for each host. It's used to identify if there is a split-brain happened while the leader(s) re-produce records
        producerStates - A map of producer GUID -> producer state.
        previousStatuses - A map of string -> string which stands for previous PartitionStatus
        pendingReportIncrementalPushVersions - A list of string which stands for incremental push versions which have received EOIP but not yet reported prior to lag caught up, they will be reported in batch
        realtimeTopicProducerStates - A map that maps upstream Kafka bootstrap server url -> to a map of producer GUID -> producer state for real-time data.

    • Method Detail

      • getClassSchema

        public static org.apache.avro.Schema getClassSchema()

      • getSpecificData

        public org.apache.avro.specific.SpecificData getSpecificData()
        Overrides:
        getSpecificData in class org.apache.avro.specific.SpecificRecordBase

      • getSchema

        public org.apache.avro.Schema getSchema()
        Specified by:
        getSchema in interface org.apache.avro.generic.GenericContainer
        Specified by:
        getSchema in class org.apache.avro.specific.SpecificRecordBase

      • get

        public java.lang.Object get​(int field$)
        Specified by:
        get in interface org.apache.avro.generic.IndexedRecord
        Specified by:
        get in class org.apache.avro.specific.SpecificRecordBase

      • put

        public void put​(int field$,
                java.lang.Object value$)
        Specified by:
        put in interface org.apache.avro.generic.IndexedRecord
        Specified by:
        put in class org.apache.avro.specific.SpecificRecordBase

      • getOffset

        public long getOffset()
        Gets the value of the 'offset' field.
        Returns:
        The last Kafka offset consumed successfully in this partition from version topic.

      • setOffset

        public void setOffset​(long value)
        Sets the value of the 'offset' field. The last Kafka offset consumed successfully in this partition from version topic.
        Parameters:
        value - the value to set.

      • getOffsetLag

        public long getOffsetLag()
        Gets the value of the 'offsetLag' field.
        Returns:
        The last Kafka offset lag in this partition for fast online transition in server restart.

      • setOffsetLag

        public void setOffsetLag​(long value)
        Sets the value of the 'offsetLag' field. The last Kafka offset lag in this partition for fast online transition in server restart.
        Parameters:
        value - the value to set.

      • getEndOfPush

        public boolean getEndOfPush()
        Gets the value of the 'endOfPush' field.
        Returns:
        Whether the EndOfPush control message was consumed in this partition.

      • setEndOfPush

        public void setEndOfPush​(boolean value)
        Sets the value of the 'endOfPush' field. Whether the EndOfPush control message was consumed in this partition.
        Parameters:
        value - the value to set.

      • getLastUpdate

        public long getLastUpdate()
        Gets the value of the 'lastUpdate' field.
        Returns:
        The last time this PartitionState was updated. Can be compared against the various messageTimestamp in ProducerPartitionState in order to infer lag time between producers and the consumer maintaining this PartitionState.

      • setLastUpdate

        public void setLastUpdate​(long value)
        Sets the value of the 'lastUpdate' field. The last time this PartitionState was updated. Can be compared against the various messageTimestamp in ProducerPartitionState in order to infer lag time between producers and the consumer maintaining this PartitionState.
        Parameters:
        value - the value to set.

      • getStartOfBufferReplayDestinationOffset

        public java.lang.Long getStartOfBufferReplayDestinationOffset()
        Gets the value of the 'startOfBufferReplayDestinationOffset' field.
        Returns:
        This is the offset at which the StartOfBufferReplay control message was consumed in the current partition of the destination topic. This is not the same value as the source offsets contained in the StartOfBufferReplay control message itself. The source and destination offsets act together as a synchronization marker. N.B.: null means that the SOBR control message was not received yet.

      • setStartOfBufferReplayDestinationOffset

        public void setStartOfBufferReplayDestinationOffset​(java.lang.Long value)
        Sets the value of the 'startOfBufferReplayDestinationOffset' field. This is the offset at which the StartOfBufferReplay control message was consumed in the current partition of the destination topic. This is not the same value as the source offsets contained in the StartOfBufferReplay control message itself. The source and destination offsets act together as a synchronization marker. N.B.: null means that the SOBR control message was not received yet.
        Parameters:
        value - the value to set.

      • getDatabaseInfo

        public java.util.Map<java.lang.CharSequence,​java.lang.CharSequence> getDatabaseInfo()
        Gets the value of the 'databaseInfo' field.
        Returns:
        A map of string -> string to store database related info, which is necessary to checkpoint

      • setDatabaseInfo

        public void setDatabaseInfo​(java.util.Map<java.lang.CharSequence,​java.lang.CharSequence> value)
        Sets the value of the 'databaseInfo' field. A map of string -> string to store database related info, which is necessary to checkpoint
        Parameters:
        value - the value to set.

      • getIncrementalPushInfo

        public IncrementalPush getIncrementalPushInfo()
        Gets the value of the 'incrementalPushInfo' field.
        Returns:
        metadata of ongoing incremental push in the partition

      • setIncrementalPushInfo

        public void setIncrementalPushInfo​(IncrementalPush value)
        Sets the value of the 'incrementalPushInfo' field. metadata of ongoing incremental push in the partition
        Parameters:
        value - the value to set.

      • getLeaderTopic

        public java.lang.CharSequence getLeaderTopic()
        Gets the value of the 'leaderTopic' field.
        Returns:
        The topic that leader is consuming from; for leader, leaderTopic can be different from the version topic; for follower, leaderTopic is the same as version topic.

      • setLeaderTopic

        public void setLeaderTopic​(java.lang.CharSequence value)
        Sets the value of the 'leaderTopic' field. The topic that leader is consuming from; for leader, leaderTopic can be different from the version topic; for follower, leaderTopic is the same as version topic.
        Parameters:
        value - the value to set.

      • getLeaderOffset

        public long getLeaderOffset()
        Gets the value of the 'leaderOffset' field.
        Returns:
        The last Kafka offset consumed successfully in this partition from the leader topic. TODO: remove this field once upstreamOffsetMap is used everywhere.

      • setLeaderOffset

        public void setLeaderOffset​(long value)
        Sets the value of the 'leaderOffset' field. The last Kafka offset consumed successfully in this partition from the leader topic. TODO: remove this field once upstreamOffsetMap is used everywhere.
        Parameters:
        value - the value to set.

      • getUpstreamOffsetMap

        public java.util.Map<java.lang.String,​java.lang.Long> getUpstreamOffsetMap()
        Gets the value of the 'upstreamOffsetMap' field.
        Returns:
        A map of upstream Kafka bootstrap server url -> the last Kafka offset consumed from upstream topic.

      • setUpstreamOffsetMap

        public void setUpstreamOffsetMap​(java.util.Map<java.lang.String,​java.lang.Long> value)
        Sets the value of the 'upstreamOffsetMap' field. A map of upstream Kafka bootstrap server url -> the last Kafka offset consumed from upstream topic.
        Parameters:
        value - the value to set.

      • getUpstreamVersionTopicOffset

        public long getUpstreamVersionTopicOffset()
        Gets the value of the 'upstreamVersionTopicOffset' field.
        Returns:
        The last upstream version topic offset persisted to disk; if the batch native-replication source is the same as local region, this value will always be -1

      • setUpstreamVersionTopicOffset

        public void setUpstreamVersionTopicOffset​(long value)
        Sets the value of the 'upstreamVersionTopicOffset' field. The last upstream version topic offset persisted to disk; if the batch native-replication source is the same as local region, this value will always be -1
        Parameters:
        value - the value to set.

      • getLeaderGUID

        public GUID getLeaderGUID()
        Gets the value of the 'leaderGUID' field.
        Returns:
        This field is deprecated since GUID is no longer able to identify the split-brain issue once 'pass-through' mode is enabled in venice writer. The field is superseded by leaderHostId and will be removed in the future

      • setLeaderGUID

        public void setLeaderGUID​(GUID value)
        Sets the value of the 'leaderGUID' field. This field is deprecated since GUID is no longer able to identify the split-brain issue once 'pass-through' mode is enabled in venice writer. The field is superseded by leaderHostId and will be removed in the future
        Parameters:
        value - the value to set.

      • getLeaderHostId

        public java.lang.CharSequence getLeaderHostId()
        Gets the value of the 'leaderHostId' field.
        Returns:
        An unique identifier (such as host name) that stands for each host. It's used to identify if there is a split-brain happened while the leader(s) re-produce records

      • setLeaderHostId

        public void setLeaderHostId​(java.lang.CharSequence value)
        Sets the value of the 'leaderHostId' field. An unique identifier (such as host name) that stands for each host. It's used to identify if there is a split-brain happened while the leader(s) re-produce records
        Parameters:
        value - the value to set.

      • getProducerStates

        public java.util.Map<java.lang.CharSequence,​ProducerPartitionState> getProducerStates()
        Gets the value of the 'producerStates' field.
        Returns:
        A map of producer GUID -> producer state.

      • setProducerStates

        public void setProducerStates​(java.util.Map<java.lang.CharSequence,​ProducerPartitionState> value)
        Sets the value of the 'producerStates' field. A map of producer GUID -> producer state.
        Parameters:
        value - the value to set.

      • getPreviousStatuses

        public java.util.Map<java.lang.CharSequence,​java.lang.CharSequence> getPreviousStatuses()
        Gets the value of the 'previousStatuses' field.
        Returns:
        A map of string -> string which stands for previous PartitionStatus

      • setPreviousStatuses

        public void setPreviousStatuses​(java.util.Map<java.lang.CharSequence,​java.lang.CharSequence> value)
        Sets the value of the 'previousStatuses' field. A map of string -> string which stands for previous PartitionStatus
        Parameters:
        value - the value to set.

      • getPendingReportIncrementalPushVersions

        public java.util.List<java.lang.CharSequence> getPendingReportIncrementalPushVersions()
        Gets the value of the 'pendingReportIncrementalPushVersions' field.
        Returns:
        A list of string which stands for incremental push versions which have received EOIP but not yet reported prior to lag caught up, they will be reported in batch

      • setPendingReportIncrementalPushVersions

        public void setPendingReportIncrementalPushVersions​(java.util.List<java.lang.CharSequence> value)
        Sets the value of the 'pendingReportIncrementalPushVersions' field. A list of string which stands for incremental push versions which have received EOIP but not yet reported prior to lag caught up, they will be reported in batch
        Parameters:
        value - the value to set.

      • getRealtimeTopicProducerStates

        public java.util.Map<java.lang.String,​java.util.Map<java.lang.CharSequence,​ProducerPartitionState>> getRealtimeTopicProducerStates()
        Gets the value of the 'realtimeTopicProducerStates' field.
        Returns:
        A map that maps upstream Kafka bootstrap server url -> to a map of producer GUID -> producer state for real-time data.

      • setRealtimeTopicProducerStates

        public void setRealtimeTopicProducerStates​(java.util.Map<java.lang.String,​java.util.Map<java.lang.CharSequence,​ProducerPartitionState>> value)
        Sets the value of the 'realtimeTopicProducerStates' field. A map that maps upstream Kafka bootstrap server url -> to a map of producer GUID -> producer state for real-time data.
        Parameters:
        value - the value to set.

      • writeExternal

        public void writeExternal​(java.io.ObjectOutput out)
                   throws java.io.IOException
        Specified by:
        writeExternal in interface java.io.Externalizable
        Overrides:
        writeExternal in class org.apache.avro.specific.SpecificRecordBase
        Throws:
        java.io.IOException

      • readExternal

        public void readExternal​(java.io.ObjectInput in)
                  throws java.io.IOException
        Specified by:
        readExternal in interface java.io.Externalizable
        Overrides:
        readExternal in class org.apache.avro.specific.SpecificRecordBase
        Throws:
        java.io.IOException