Package com.linkedin.venice.helix
Class HelixReadWriteSchemaRepository
java.lang.Object
com.linkedin.venice.helix.HelixReadWriteSchemaRepository
- All Implemented Interfaces:
ReadOnlySchemaRepository,ReadWriteSchemaRepository,VeniceResource
This class is used to add schema entries for stores.
There are 4 types of schema entries
1. Key schema
ZK Path: ${cluster_name}/Stores/${store_name}/key-schema/1
Each store only has 1 key schema and the schema is immutable.
2. Value schema
ZK Path: ${cluster_name}/Stores/${store_name}/value-schema/${value_schema_id}
Value schemas are evolvable. Stores can have multiple value schemas and each
value schema is forwards/backwards compatible with others.
3. Derived schema
ZK Path: ${cluster_name}/Stores/${store_name}/derived-schema/${value_schema_id}_${derived_schema_id}
Each value schema can have multiple derived schemas. check out
DerivedSchemaEntry for more
details.
3. Replication metadata schema
* ZK Path: ${cluster_name}/Stores/${store_name}/timestamp-metadata-schema/${value_schema_id}-${replication_metadata_version_id}
*
Check out SchemaEntrySerializer and DerivedSchemaEntrySerializer
to see how schemas are ser-ded.
ReadWriteSchemaRepository doesn't cache existing schemas locally and it always
queries ZK for currently values. This is a different behavior compared to
ReadOnlyStoreRepository where values always
get cached and future update callbacks are registered.
Notice: Users should not instantiate this class elsewhere than in the leader
Controller and there should be always only 1 ReadWriteSchemaRepository per cluster.
Instantiating multiple ReadWriteSchemaRepository will lead to race conditions in
ZK.-
Constructor Summary
ConstructorsConstructorDescriptionHelixReadWriteSchemaRepository(ReadWriteStoreRepository storeRepository, Optional<MetaStoreWriter> metaStoreWriter, HelixSchemaAccessor accessor) HelixReadWriteSchemaRepository(ReadWriteStoreRepository storeRepository, org.apache.helix.zookeeper.impl.client.ZkClient zkClient, HelixAdapterSerializer adapter, String clusterName, Optional<MetaStoreWriter> metaStoreWriter, int refreshAttemptsForZkReconnect) -
Method Summary
Modifier and TypeMethodDescriptionaddDerivedSchema(String storeName, String schemaStr, int valueSchemaId) Add a new derived schema for the given store and value schema idaddDerivedSchema(String storeName, String schemaStr, int valueSchemaId, int derivedSchemaId) Add a new derived schema for the given store by specifying derived schema id.addReplicationMetadataSchema(String storeName, int valueSchemaId, String replicationMetadataSchemaStr, int replicationMetadataVersionId) addValueSchema(String storeName, String schemaStr, int schemaId) Add a new value schema for the given store by specifying schema id.addValueSchema(String storeName, String schemaStr, DirectionalSchemaCompatibilityType expectedCompatibilityType) Add new value schema for the given store.voidclear()getDerivedSchema(String storeName, int valueSchemaId, int derivedSchemaId) getDerivedSchemaId(String storeName, String derivedSchemaStr) Look up derived schema id and its corresponding value schema id by given store name and derived schema.getDerivedSchemas(String storeName) getKeySchema(String storeName) Get key schema for the given store.getLatestDerivedSchema(String storeName, int valueSchemaId) Get the most recent derived schema added to the given store and value schema idgetReplicationMetadataSchema(String storeName, int valueSchemaId, int replicationMetadataVersionId) getReplicationMetadataSchemas(String storeName) getSupersetOrLatestValueSchema(String storeName) Get the most recent value schema or superset value schema if one exists.getSupersetSchema(String storeName) Get the superset value schema for a given store.getValueSchema(String storeName, int id) Get value schema for the given store and schema id.intgetValueSchemaId(String storeName, String valueSchemaStr) This function is used to retrieve value schema id for the given store and schema.getValueSchemas(String storeName) This function is used to retrieve all the value schemas for the given store.booleanhasValueSchema(String storeName, int id) Check whether the given value schema id exists in the given store or not.initKeySchema(String storeName, String schemaStr) Set up key schema for the given store.intpreCheckDerivedSchemaAndGetNextAvailableId(String storeName, int valueSchemaId, String derivedSchemaStr) intpreCheckValueSchemaAndGetNextAvailableId(String storeName, String valueSchemaStr, DirectionalSchemaCompatibilityType expectedCompatibilityType) Check if the incoming schema is a valid schema and return the next available schema ID.voidrefresh()removeDerivedSchema(String storeName, int valueSchemaId, int derivedSchemaId) Remove an existing derived schemavoidremoveValueSchema(String storeName, int valueSchemaId) Methods inherited from class java.lang.Object
clone, equals, finalize, getClass, hashCode, notify, notifyAll, toString, wait, wait, waitMethods inherited from interface com.linkedin.venice.meta.ReadOnlySchemaRepository
getLatestDerivedSchemaMethods inherited from interface com.linkedin.venice.meta.ReadWriteSchemaRepository
addValueSchema
-
Constructor Details
-
HelixReadWriteSchemaRepository
public HelixReadWriteSchemaRepository(ReadWriteStoreRepository storeRepository, Optional<MetaStoreWriter> metaStoreWriter, HelixSchemaAccessor accessor) -
HelixReadWriteSchemaRepository
public HelixReadWriteSchemaRepository(ReadWriteStoreRepository storeRepository, org.apache.helix.zookeeper.impl.client.ZkClient zkClient, HelixAdapterSerializer adapter, String clusterName, Optional<MetaStoreWriter> metaStoreWriter, int refreshAttemptsForZkReconnect)
-
-
Method Details
-
getKeySchema
Get key schema for the given store. Fetch from zookeeper directly.- Specified by:
getKeySchemain interfaceReadOnlySchemaRepository- Returns:
- null if key schema doesn't exist; schema entry if exists;
-
getValueSchema
Get value schema for the given store and schema id. Fetch from zookeeper directly.- Specified by:
getValueSchemain interfaceReadOnlySchemaRepository- Returns:
- null if the schema doesn't exist; schema entry if exists;
-
hasValueSchema
Check whether the given value schema id exists in the given store or not. Fetch from zookeeper directly.- Specified by:
hasValueSchemain interfaceReadOnlySchemaRepository- Returns:
- null if the schema doesn't exist; schema entry if exists;
-
getValueSchemaId
This function is used to retrieve value schema id for the given store and schema. Attempts to get the schema that matches exactly. If multiple matching schemas are found then the id of the latest added schema is returned.- Specified by:
getValueSchemaIdin interfaceReadOnlySchemaRepository- Returns:
SchemaData.INVALID_VALUE_SCHEMA_ID, if the schema doesn't exist in the given store; schema id (int), if the schema exists in the given store;
-
getValueSchemas
This function is used to retrieve all the value schemas for the given store. Fetch from zookeeper directly.- Specified by:
getValueSchemasin interfaceReadOnlySchemaRepository
-
getDerivedSchemas
- Specified by:
getDerivedSchemasin interfaceReadOnlySchemaRepository
-
getSupersetOrLatestValueSchema
Description copied from interface:ReadOnlySchemaRepositoryGet the most recent value schema or superset value schema if one exists.- Specified by:
getSupersetOrLatestValueSchemain interfaceReadOnlySchemaRepository
-
getSupersetSchema
Description copied from interface:ReadOnlySchemaRepositoryGet the superset value schema for a given store. Each store has at most one active superset schema. Specifically a store must have some features enabled (e.g. read compute, write compute) to have a superset value schema which evolves as new value schemas are added.- Specified by:
getSupersetSchemain interfaceReadOnlySchemaRepository- Returns:
- Superset value schema or
nullif store does not have any superset value schema.
-
getLatestDerivedSchema
Description copied from interface:ReadOnlySchemaRepositoryGet the most recent derived schema added to the given store and value schema id- Specified by:
getLatestDerivedSchemain interfaceReadOnlySchemaRepository
-
getDerivedSchema
public DerivedSchemaEntry getDerivedSchema(String storeName, int valueSchemaId, int derivedSchemaId) - Specified by:
getDerivedSchemain interfaceReadOnlySchemaRepository
-
initKeySchema
Set up key schema for the given store.- Specified by:
initKeySchemain interfaceReadWriteSchemaRepository
-
addValueSchema
public SchemaEntry addValueSchema(String storeName, String schemaStr, DirectionalSchemaCompatibilityType expectedCompatibilityType) Add new value schema for the given store.- Specified by:
addValueSchemain interfaceReadWriteSchemaRepository- Returns:
- schema entry if the schema is successfully added or already exists.
-
addValueSchema
Description copied from interface:ReadWriteSchemaRepositoryAdd a new value schema for the given store by specifying schema id. This API is mostly intended to be used in cross-colo mode. When there are multiple colos, we'd like to have consistent value id across colos, so that deserializer can work properly while reading records. Caller should figure out the schema id number by themselves. TODO: Might want to remove it from the interface and make it invisible from the outside- Specified by:
addValueSchemain interfaceReadWriteSchemaRepository
-
preCheckValueSchemaAndGetNextAvailableId
public int preCheckValueSchemaAndGetNextAvailableId(String storeName, String valueSchemaStr, DirectionalSchemaCompatibilityType expectedCompatibilityType) Check if the incoming schema is a valid schema and return the next available schema ID. Venice pre-checks 3 things: 1. If the store is existing or not 2. If the incoming schema contains any reserved fields 3. If the incoming schema is duplicate with current's.- Specified by:
preCheckValueSchemaAndGetNextAvailableIdin interfaceReadWriteSchemaRepository- Returns:
- next available ID if it's a valid schema or
SchemaData.DUPLICATE_VALUE_SCHEMA_CODEif it's a duplicate
-
preCheckDerivedSchemaAndGetNextAvailableId
public int preCheckDerivedSchemaAndGetNextAvailableId(String storeName, int valueSchemaId, String derivedSchemaStr) - Specified by:
preCheckDerivedSchemaAndGetNextAvailableIdin interfaceReadWriteSchemaRepository
-
addDerivedSchema
Description copied from interface:ReadWriteSchemaRepositoryAdd a new derived schema for the given store and value schema id- Specified by:
addDerivedSchemain interfaceReadWriteSchemaRepository
-
addDerivedSchema
public DerivedSchemaEntry addDerivedSchema(String storeName, String schemaStr, int valueSchemaId, int derivedSchemaId) Description copied from interface:ReadWriteSchemaRepositoryAdd a new derived schema for the given store by specifying derived schema id. Mostly used in cross-colo mode.- Specified by:
addDerivedSchemain interfaceReadWriteSchemaRepository
-
removeDerivedSchema
public DerivedSchemaEntry removeDerivedSchema(String storeName, int valueSchemaId, int derivedSchemaId) Description copied from interface:ReadWriteSchemaRepositoryRemove an existing derived schema- Specified by:
removeDerivedSchemain interfaceReadWriteSchemaRepository- Returns:
- the derived schema that is deleted or null if the schema doesn't exist
-
getDerivedSchemaId
Description copied from interface:ReadOnlySchemaRepositoryLook up derived schema id and its corresponding value schema id by given store name and derived schema. This is likely used by clients that write to Venice- Specified by:
getDerivedSchemaIdin interfaceReadOnlySchemaRepository- Returns:
- a pair where the first value is value schema id and the second value is derived schema id
-
removeValueSchema
- Specified by:
removeValueSchemain interfaceReadWriteSchemaRepository
-
getReplicationMetadataSchemas
- Specified by:
getReplicationMetadataSchemasin interfaceReadOnlySchemaRepository
-
getReplicationMetadataSchema
public RmdSchemaEntry getReplicationMetadataSchema(String storeName, int valueSchemaId, int replicationMetadataVersionId) - Specified by:
getReplicationMetadataSchemain interfaceReadOnlySchemaRepository
-
addReplicationMetadataSchema
public RmdSchemaEntry addReplicationMetadataSchema(String storeName, int valueSchemaId, String replicationMetadataSchemaStr, int replicationMetadataVersionId) - Specified by:
addReplicationMetadataSchemain interfaceReadWriteSchemaRepository
-
refresh
public void refresh()- Specified by:
refreshin interfaceVeniceResource
-
clear
public void clear()- Specified by:
clearin interfaceVeniceResource
-