Package com.linkedin.venice.controller.kafka.protocol.serializer

Class SemanticDetector

java.lang.Object
com.linkedin.venice.controller.kafka.protocol.serializer.SemanticDetector
public class SemanticDetector extends Object
SemanticDetector is used to detect the semantic changes between two schemas. It traverses the object based on the current schema and target schema. When we detect new semantic change, we compare object with default value. If the object is NOT default value, we throw @code{VeniceProtocolException}. Semantic changes include: 1. Changing the type of a field 2. Adding a new field 3. Changing the fixed size of a fixed type 4. Adding a new enum value

  • Constructor Summary

    Constructors
    Constructor
    Description
    SemanticDetector()
     

  • Method Summary

    Modifier and Type
    Method
    Description
    static void
    compareObjectToDefaultValue(Object object, org.apache.avro.Schema currentSchema, String name, Object defaultValue)
    Validate the object with default value.
    static org.apache.avro.Schema
    getObjectSchema(Object object, org.apache.avro.Schema unionSchema)
     
    static void
    traverseAndValidate(Object object, org.apache.avro.Schema currentSchema, org.apache.avro.Schema targetSchema, String name, Object defaultValue)
    Traverse the object based on the current schema and target schema.
    static void
    traverseCollections(List<Object> array, org.apache.avro.Schema currentSchema, org.apache.avro.Schema targetSchema, String name, Object defaultArrayEntry)
    Traverse the array to make sure the array elements are not using new semantic.
    static void
    traverseFields(org.apache.avro.generic.GenericRecord record, org.apache.avro.Schema currentSchema, org.apache.avro.Schema targetSchema, String recordName, Object defaultValue)
    Traverse the fields of the record.
    static void
    traverseMap(Map<String,Object> map, org.apache.avro.Schema currentSchema, org.apache.avro.Schema targetSchema, String name, Object defaultValueEntry)
    Traverse the map to make sure the map values are not using new semantic.
    static void
    traverseUnion(Object object, org.apache.avro.Schema currentSchema, org.apache.avro.Schema targetSchema, String name, Object defaultValue)
    Traverse the union to make sure the object is not using new semantic.
    static void
    validateEnum(Object enumValue, org.apache.avro.Schema currentSchema, org.apache.avro.Schema targetSchema, String name)
    Validate the enum value.

    Methods inherited from class java.lang.Object

    clone, equals, finalize, getClass, hashCode, notify, notifyAll, toString, wait, wait, wait

  • Constructor Details

    • SemanticDetector

      public SemanticDetector()

  • Method Details

    • traverseAndValidate

      public static void traverseAndValidate(Object object, org.apache.avro.Schema currentSchema, org.apache.avro.Schema targetSchema, String name, Object defaultValue)
      Traverse the object based on the current schema and target schema. Stop traversing if found a semantic change & using not default value.
      Parameters:
      object - the object containing values
      currentSchema - the object schema
      targetSchema - the target schema we want to compare with
      name - The name of the object from the parent object
      defaultValue - The default value of the object
      Throws:
      VeniceProtocolException - if there is a semantic change between the two schemas & value from

    • traverseFields

      public static void traverseFields(org.apache.avro.generic.GenericRecord record, org.apache.avro.Schema currentSchema, org.apache.avro.Schema targetSchema, String recordName, Object defaultValue)
      Traverse the fields of the record. Stop traversing if found a semantic change & using not default value. We want to loop through fields to find the mismatch field between the current schema and target schema.

      For example,

      • Current schema: field1, field2, field3
      • Target schema: field1, field2
      We want to find the mismatch field field3 between the two schemas => field3 is a new field. And validate the value of field3 against the current schema.

      Parameters:
      record - the record containing values
      currentSchema - the record schema
      targetSchema - the target schema we want to compare with
      recordName - The name of the record from the parent object
      defaultValue - The default value of the record
      Throws:
      VeniceProtocolException - if there is a semantic change between the two schemas & field value from @code is non-default value.

    • traverseCollections

      public static void traverseCollections(List<Object> array, org.apache.avro.Schema currentSchema, org.apache.avro.Schema targetSchema, String name, Object defaultArrayEntry)
      Traverse the array to make sure the array elements are not using new semantic.

      For example,

      • Current schema: List (RecordA - field1, field2, field3)
      • Target schema: List (RecordA - field1, field2)
      In this case, we want to check every element in the array to make sure the element is not using new semantic.

      Parameters:
      array - the array containing values
      currentSchema - the array schema
      targetSchema - the target schema we want to compare with
      name - The name of the array from the parent object
      defaultArrayEntry - The default value of the array
      Throws:
      VeniceProtocolException - if there is a semantic change between the two schemas & element from @code is non-default value.

    • traverseMap

      public static void traverseMap(Map<String,Object> map, org.apache.avro.Schema currentSchema, org.apache.avro.Schema targetSchema, String name, Object defaultValueEntry)
      Traverse the map to make sure the map values are not using new semantic.

      For example,

      • Current schema: Map (String, RecordA - field1, field2, field3)
      • Target schema: Map (String, RecordA - field1, field2)
      In this case, we want to check every value in the map to make sure the value is not using new semantic. We do not need to check the key since the key is a string.

      Parameters:
      map - the map containing values
      currentSchema - the map schema
      targetSchema - the target schema we want to compare with
      name - The name of the map from the parent object
      defaultValueEntry - The default value of the map
      Throws:
      VeniceProtocolException - if there is a semantic change between the two schemas & value from @code is non-default value.

    • traverseUnion

      public static void traverseUnion(Object object, org.apache.avro.Schema currentSchema, org.apache.avro.Schema targetSchema, String name, Object defaultValue)
      Traverse the union to make sure the object is not using new semantic.

      For example,

      • Current schema: Union (RecordA - field1, field2, field3, RecordB - field1, field2)
      • Target schema: Union (RecordA - field1, field2, field3)
      In this case, we want to check the object against the schema within the union schema. If object - RecordA(field1, field2, field3), then we are NOT using new semantic change. If object - RecordB(field1, field2), then we have a semantic change. If the object is using new semantic, we return true.

      Parameters:
      object - the object containing values
      currentSchema - the union schema
      targetSchema - the target schema we want to compare with
      name - The name of the object from the parent object
      defaultValue - The default value of the object
      Throws:
      VeniceProtocolException - if there is a semantic change between the two schemas & value from @code is non-default value OR if the object does not match any schema within the union schemas.

    • getObjectSchema

      public static org.apache.avro.Schema getObjectSchema(Object object, org.apache.avro.Schema unionSchema)

    • compareObjectToDefaultValue

      public static void compareObjectToDefaultValue(Object object, org.apache.avro.Schema currentSchema, String name, Object defaultValue)
      Validate the object with default value. There are two default values:
      1. Default value derived from schema as @code}
      2. Default value derived from the schema type of the object
      Parameters:
      object - the object to validate
      currentSchema - the schema of the object
      name - the name of the object
      defaultValue - the default value of the object

    • validateEnum

      public static void validateEnum(Object enumValue, org.apache.avro.Schema currentSchema, org.apache.avro.Schema targetSchema, String name)
      Validate the enum value. If the enum value is in current schema but NOT in target schema, it is a new enum value.
      Parameters:
      enumValue - the enum value to validate
      currentSchema - the current schema
      targetSchema - the target schema
      name - the name of the enum value
      Throws:
      VeniceProtocolException - if the enum value is not in the target schema OR if the enum value is not a string