Package com.linkedin.venice.serialization.avro

Class InternalAvroSpecificSerializer<SPECIFIC_RECORD extends org.apache.avro.specific.SpecificRecord>

  • All Implemented Interfaces:
    VeniceKafkaSerializer<SPECIFIC_RECORD>, java.io.Closeable, java.lang.AutoCloseable, org.apache.kafka.common.serialization.Deserializer<SPECIFIC_RECORD>, org.apache.kafka.common.serialization.Serializer<SPECIFIC_RECORD>
    Direct Known Subclasses:
    ChunkedKeySuffixSerializer, ChunkedValueManifestSerializer, KafkaValueSerializer
    public class InternalAvroSpecificSerializer<SPECIFIC_RECORD extends org.apache.avro.specific.SpecificRecord>
extends java.lang.Object
implements VeniceKafkaSerializer<SPECIFIC_RECORD>
    Serializer for translating a versioned protocol of Avro records. The protocol is the following: 1st byte: The magic byte, should always equal 'magicByte'. 2nd byte: The protocol version 3rd byte and onward: The payload (a single binary-encoded Avro record) encoded with a writer schema determined by the protocol version specified in #2.

    • Field Detail

      • VENICE_SCHEMA_READER_CONFIG

        @Deprecated
public static final java.lang.String VENICE_SCHEMA_READER_CONFIG
        Deprecated.
        Used to configure the schemaReader. Deprecated: This path has now been superseded by setSchemaReader(SchemaReader), which is used everywhere except in com.linkedin.venice.kafka.KafkaClientFactory#getConsumer(Properties, PubSubMessageDeserializer) ()}. Once that usage is also eliminated we could remove the config from here.
        See Also:
        Constant Field Values

      • MAX_ATTEMPTS_FOR_SCHEMA_READER

        public static final int MAX_ATTEMPTS_FOR_SCHEMA_READER
        See Also:
        Constant Field Values

      • WAIT_TIME_BETWEEN_SCHEMA_READER_ATTEMPTS_IN_MS

        public static final int WAIT_TIME_BETWEEN_SCHEMA_READER_ATTEMPTS_IN_MS
        See Also:
        Constant Field Values

      • SENTINEL_PROTOCOL_VERSION_USED_FOR_UNDETECTABLE_COMPILED_SCHEMA

        public static final int SENTINEL_PROTOCOL_VERSION_USED_FOR_UNDETECTABLE_COMPILED_SCHEMA
        See Also:
        Constant Field Values

      • SENTINEL_PROTOCOL_VERSION_USED_FOR_UNVERSIONED_PROTOCOL

        public static final int SENTINEL_PROTOCOL_VERSION_USED_FOR_UNVERSIONED_PROTOCOL
        See Also:
        Constant Field Values

    • Constructor Detail

      • InternalAvroSpecificSerializer

        protected InternalAvroSpecificSerializer​(AvroProtocolDefinition protocolDef)

      • InternalAvroSpecificSerializer

        protected InternalAvroSpecificSerializer​(AvroProtocolDefinition protocolDef,
                                         java.lang.Integer payloadOffsetOverride)

      • InternalAvroSpecificSerializer

        protected InternalAvroSpecificSerializer​(AvroProtocolDefinition protocolDef,
                                         java.lang.Integer payloadOffsetOverride,
                                         java.util.function.BiConsumer<java.lang.Integer,​org.apache.avro.Schema> newSchemaEncountered)

    • Method Detail

      • close

        public void close()
        Close this serializer. This method has to be idempotent if the serializer is used in KafkaProducer because it might be called multiple times.
        Specified by:
        close in interface java.lang.AutoCloseable
        Specified by:
        close in interface java.io.Closeable
        Specified by:
        close in interface org.apache.kafka.common.serialization.Deserializer<SPECIFIC_RECORD extends org.apache.avro.specific.SpecificRecord>
        Specified by:
        close in interface org.apache.kafka.common.serialization.Serializer<SPECIFIC_RECORD extends org.apache.avro.specific.SpecificRecord>
        Specified by:
        close in interface VeniceKafkaSerializer<SPECIFIC_RECORD extends org.apache.avro.specific.SpecificRecord>

      • knownProtocols

        public it.unimi.dsi.fastutil.ints.IntSet knownProtocols()

      • getCompiledProtocol

        public org.apache.avro.Schema getCompiledProtocol()

      • configure

        public void configure​(java.util.Map<java.lang.String,​?> configMap,
                      boolean isKey)
        Configure this class. KafkaConsumer(Properties) would eventually call configure(Map, boolean) which would pass in the customized Kafka config map with schema reader.
        Specified by:
        configure in interface org.apache.kafka.common.serialization.Deserializer<SPECIFIC_RECORD extends org.apache.avro.specific.SpecificRecord>
        Specified by:
        configure in interface org.apache.kafka.common.serialization.Serializer<SPECIFIC_RECORD extends org.apache.avro.specific.SpecificRecord>
        Specified by:
        configure in interface VeniceKafkaSerializer<SPECIFIC_RECORD extends org.apache.avro.specific.SpecificRecord>
        Parameters:
        configMap - configs in key/value pairs
        isKey - whether is for key or value

      • setSchemaReader

        public void setSchemaReader​(SchemaReader schemaReader)

      • serialize

        public byte[] serialize​(java.lang.String topic,
                        SPECIFIC_RECORD object)
        Construct an array of bytes from the given object
        Specified by:
        serialize in interface org.apache.kafka.common.serialization.Serializer<SPECIFIC_RECORD extends org.apache.avro.specific.SpecificRecord>
        Specified by:
        serialize in interface VeniceKafkaSerializer<SPECIFIC_RECORD extends org.apache.avro.specific.SpecificRecord>
        Parameters:
        topic - Topic to which the object belongs (for API compatibility reason only, but unused)
        object - A InternalAvroSpecificSerializer instance to be serialized.
        Returns:
        The Avro binary format bytes which represent the {@param object}

      • serialize

        public java.nio.ByteBuffer serialize​(SPECIFIC_RECORD object)

      • deserialize

        public SPECIFIC_RECORD deserialize​(java.lang.String topic,
                                   byte[] bytes)
        Create an object from an array of bytes This method is used by the Kafka consumer. These calls are always intended to be for protocols which use a magic byte and a protocol version, both of which are stored in the header, before the payload.
        Specified by:
        deserialize in interface org.apache.kafka.common.serialization.Deserializer<SPECIFIC_RECORD extends org.apache.avro.specific.SpecificRecord>
        Specified by:
        deserialize in interface VeniceKafkaSerializer<SPECIFIC_RECORD extends org.apache.avro.specific.SpecificRecord>
        Parameters:
        topic - Topic to which the array of bytes belongs (only there to implement the interface, but otherwise useless)
        bytes - An array of bytes representing the object's data serialized in Avro binary format.
        Returns:
        A InternalAvroSpecificSerializer deserialized from the bytes

      • deserialize

        public SPECIFIC_RECORD deserialize​(byte[] bytes,
                                   int protocolVersion)

      • createBinaryDecoder

        protected org.apache.avro.io.BinaryDecoder createBinaryDecoder​(byte[] bytes,
                                                               int offset,
                                                               int length,
                                                               org.apache.avro.io.BinaryDecoder reuse)

      • removeAllSchemas

        public void removeAllSchemas()