Package com.linkedin.venice.helix
Class HelixReadWriteSchemaRepository
java.lang.Object
com.linkedin.venice.helix.HelixReadWriteSchemaRepository
- All Implemented Interfaces:
ReadOnlySchemaRepository
,ReadWriteSchemaRepository
,VeniceResource
This class is used to add schema entries for stores.
There are 4 types of schema entries
1. Key schema
ZK Path: ${cluster_name}/Stores/${store_name}/key-schema/1
Each store only has 1 key schema and the schema is immutable.
2. Value schema
ZK Path: ${cluster_name}/Stores/${store_name}/value-schema/${value_schema_id}
Value schemas are evolvable. Stores can have multiple value schemas and each
value schema is forwards/backwards compatible with others.
3. Derived schema
ZK Path: ${cluster_name}/Stores/${store_name}/derived-schema/${value_schema_id}_${derived_schema_id}
Each value schema can have multiple derived schemas. check out
DerivedSchemaEntry
for more
details.
3. Replication metadata schema
* ZK Path: ${cluster_name}/Stores/${store_name}/timestamp-metadata-schema/${value_schema_id}-${replication_metadata_version_id}
*
Check out SchemaEntrySerializer
and DerivedSchemaEntrySerializer
to see how schemas are ser-ded.
ReadWriteSchemaRepository doesn't cache existing schemas locally and it always
queries ZK for currently values. This is a different behavior compared to
ReadOnlyStoreRepository
where values always
get cached and future update callbacks are registered.
Notice: Users should not instantiate this class elsewhere than in the leader
Controller and there should be always only 1 ReadWriteSchemaRepository per cluster.
Instantiating multiple ReadWriteSchemaRepository will lead to race conditions in
ZK.-
Constructor Summary
ConstructorDescriptionHelixReadWriteSchemaRepository
(ReadWriteStoreRepository storeRepository, Optional<MetaStoreWriter> metaStoreWriter, HelixSchemaAccessor accessor) HelixReadWriteSchemaRepository
(ReadWriteStoreRepository storeRepository, org.apache.helix.zookeeper.impl.client.ZkClient zkClient, HelixAdapterSerializer adapter, String clusterName, Optional<MetaStoreWriter> metaStoreWriter) -
Method Summary
Modifier and TypeMethodDescriptionaddDerivedSchema
(String storeName, String schemaStr, int valueSchemaId) Add a new derived schema for the given store and value schema idaddDerivedSchema
(String storeName, String schemaStr, int valueSchemaId, int derivedSchemaId) Add a new derived schema for the given store by specifying derived schema id.addReplicationMetadataSchema
(String storeName, int valueSchemaId, String replicationMetadataSchemaStr, int replicationMetadataVersionId) addValueSchema
(String storeName, String schemaStr, int schemaId) Add a new value schema for the given store by specifying schema id.addValueSchema
(String storeName, String schemaStr, DirectionalSchemaCompatibilityType expectedCompatibilityType) Add new value schema for the given store.void
clear()
getDerivedSchema
(String storeName, int valueSchemaId, int derivedSchemaId) getDerivedSchemaId
(String storeName, String derivedSchemaStr) Look up derived schema id and its corresponding value schema id by given store name and derived schema.getDerivedSchemas
(String storeName) getKeySchema
(String storeName) Get key schema for the given store.getLatestDerivedSchema
(String storeName, int valueSchemaId) Get the most recent derived schema added to the given store and value schema idgetReplicationMetadataSchema
(String storeName, int valueSchemaId, int replicationMetadataVersionId) getReplicationMetadataSchemas
(String storeName) getSupersetOrLatestValueSchema
(String storeName) Get the most recent value schema or superset value schema if one exists.getSupersetSchema
(String storeName) Get the superset value schema for a given store.getValueSchema
(String storeName, int id) Get value schema for the given store and schema id.int
getValueSchemaId
(String storeName, String valueSchemaStr) This function is used to retrieve value schema id for the given store and schema.getValueSchemas
(String storeName) This function is used to retrieve all the value schemas for the given store.boolean
hasValueSchema
(String storeName, int id) Check whether the given value schema id exists in the given store or not.initKeySchema
(String storeName, String schemaStr) Set up key schema for the given store.int
preCheckDerivedSchemaAndGetNextAvailableId
(String storeName, int valueSchemaId, String derivedSchemaStr) int
preCheckValueSchemaAndGetNextAvailableId
(String storeName, String valueSchemaStr, DirectionalSchemaCompatibilityType expectedCompatibilityType) Check if the incoming schema is a valid schema and return the next available schema ID.void
refresh()
removeDerivedSchema
(String storeName, int valueSchemaId, int derivedSchemaId) Remove an existing derived schemavoid
removeValueSchema
(String storeName, int valueSchemaId) Methods inherited from class java.lang.Object
clone, equals, finalize, getClass, hashCode, notify, notifyAll, toString, wait, wait, wait
Methods inherited from interface com.linkedin.venice.meta.ReadOnlySchemaRepository
getLatestDerivedSchema
Methods inherited from interface com.linkedin.venice.meta.ReadWriteSchemaRepository
addValueSchema
-
Constructor Details
-
HelixReadWriteSchemaRepository
public HelixReadWriteSchemaRepository(ReadWriteStoreRepository storeRepository, Optional<MetaStoreWriter> metaStoreWriter, HelixSchemaAccessor accessor) -
HelixReadWriteSchemaRepository
public HelixReadWriteSchemaRepository(ReadWriteStoreRepository storeRepository, org.apache.helix.zookeeper.impl.client.ZkClient zkClient, HelixAdapterSerializer adapter, String clusterName, Optional<MetaStoreWriter> metaStoreWriter)
-
-
Method Details
-
getKeySchema
Get key schema for the given store. Fetch from zookeeper directly.- Specified by:
getKeySchema
in interfaceReadOnlySchemaRepository
- Returns:
- null if key schema doesn't exist; schema entry if exists;
-
getValueSchema
Get value schema for the given store and schema id. Fetch from zookeeper directly.- Specified by:
getValueSchema
in interfaceReadOnlySchemaRepository
- Returns:
- null if the schema doesn't exist; schema entry if exists;
-
hasValueSchema
Check whether the given value schema id exists in the given store or not. Fetch from zookeeper directly.- Specified by:
hasValueSchema
in interfaceReadOnlySchemaRepository
- Returns:
- null if the schema doesn't exist; schema entry if exists;
-
getValueSchemaId
This function is used to retrieve value schema id for the given store and schema. Attempts to get the schema that matches exactly. If multiple matching schemas are found then the id of the latest added schema is returned.- Specified by:
getValueSchemaId
in interfaceReadOnlySchemaRepository
- Returns:
SchemaData.INVALID_VALUE_SCHEMA_ID
, if the schema doesn't exist in the given store; schema id (int), if the schema exists in the given store;
-
getValueSchemas
This function is used to retrieve all the value schemas for the given store. Fetch from zookeeper directly.- Specified by:
getValueSchemas
in interfaceReadOnlySchemaRepository
-
getDerivedSchemas
- Specified by:
getDerivedSchemas
in interfaceReadOnlySchemaRepository
-
getSupersetOrLatestValueSchema
Description copied from interface:ReadOnlySchemaRepository
Get the most recent value schema or superset value schema if one exists.- Specified by:
getSupersetOrLatestValueSchema
in interfaceReadOnlySchemaRepository
-
getSupersetSchema
Description copied from interface:ReadOnlySchemaRepository
Get the superset value schema for a given store. Each store has at most one active superset schema. Specifically a store must have some features enabled (e.g. read compute, write compute) to have a superset value schema which evolves as new value schemas are added.- Specified by:
getSupersetSchema
in interfaceReadOnlySchemaRepository
- Returns:
- Superset value schema or
null
if store does not have any superset value schema.
-
getLatestDerivedSchema
Description copied from interface:ReadOnlySchemaRepository
Get the most recent derived schema added to the given store and value schema id- Specified by:
getLatestDerivedSchema
in interfaceReadOnlySchemaRepository
-
getDerivedSchema
public DerivedSchemaEntry getDerivedSchema(String storeName, int valueSchemaId, int derivedSchemaId) - Specified by:
getDerivedSchema
in interfaceReadOnlySchemaRepository
-
initKeySchema
Set up key schema for the given store.- Specified by:
initKeySchema
in interfaceReadWriteSchemaRepository
-
addValueSchema
public SchemaEntry addValueSchema(String storeName, String schemaStr, DirectionalSchemaCompatibilityType expectedCompatibilityType) Add new value schema for the given store.- Specified by:
addValueSchema
in interfaceReadWriteSchemaRepository
- Returns:
- schema entry if the schema is successfully added or already exists.
-
addValueSchema
Description copied from interface:ReadWriteSchemaRepository
Add a new value schema for the given store by specifying schema id. This API is mostly intended to be used in cross-colo mode. When there are multiple colos, we'd like to have consistent value id across colos, so that deserializer can work properly while reading records. Caller should figure out the schema id number by themselves. TODO: Might want to remove it from the interface and make it invisible from the outside- Specified by:
addValueSchema
in interfaceReadWriteSchemaRepository
-
preCheckValueSchemaAndGetNextAvailableId
public int preCheckValueSchemaAndGetNextAvailableId(String storeName, String valueSchemaStr, DirectionalSchemaCompatibilityType expectedCompatibilityType) Check if the incoming schema is a valid schema and return the next available schema ID. Venice pre-checks 3 things: 1. If the store is existing or not 2. If the incoming schema contains any reserved fields 3. If the incoming schema is duplicate with current's.- Specified by:
preCheckValueSchemaAndGetNextAvailableId
in interfaceReadWriteSchemaRepository
- Returns:
- next available ID if it's a valid schema or
SchemaData.DUPLICATE_VALUE_SCHEMA_CODE
if it's a duplicate
-
preCheckDerivedSchemaAndGetNextAvailableId
public int preCheckDerivedSchemaAndGetNextAvailableId(String storeName, int valueSchemaId, String derivedSchemaStr) - Specified by:
preCheckDerivedSchemaAndGetNextAvailableId
in interfaceReadWriteSchemaRepository
-
addDerivedSchema
Description copied from interface:ReadWriteSchemaRepository
Add a new derived schema for the given store and value schema id- Specified by:
addDerivedSchema
in interfaceReadWriteSchemaRepository
-
addDerivedSchema
public DerivedSchemaEntry addDerivedSchema(String storeName, String schemaStr, int valueSchemaId, int derivedSchemaId) Description copied from interface:ReadWriteSchemaRepository
Add a new derived schema for the given store by specifying derived schema id. Mostly used in cross-colo mode.- Specified by:
addDerivedSchema
in interfaceReadWriteSchemaRepository
-
removeDerivedSchema
public DerivedSchemaEntry removeDerivedSchema(String storeName, int valueSchemaId, int derivedSchemaId) Description copied from interface:ReadWriteSchemaRepository
Remove an existing derived schema- Specified by:
removeDerivedSchema
in interfaceReadWriteSchemaRepository
- Returns:
- the derived schema that is deleted or null if the schema doesn't exist
-
getDerivedSchemaId
Description copied from interface:ReadOnlySchemaRepository
Look up derived schema id and its corresponding value schema id by given store name and derived schema. This is likely used by clients that write to Venice- Specified by:
getDerivedSchemaId
in interfaceReadOnlySchemaRepository
- Returns:
- a pair where the first value is value schema id and the second value is derived schema id
-
removeValueSchema
- Specified by:
removeValueSchema
in interfaceReadWriteSchemaRepository
-
getReplicationMetadataSchemas
- Specified by:
getReplicationMetadataSchemas
in interfaceReadOnlySchemaRepository
-
getReplicationMetadataSchema
public RmdSchemaEntry getReplicationMetadataSchema(String storeName, int valueSchemaId, int replicationMetadataVersionId) - Specified by:
getReplicationMetadataSchema
in interfaceReadOnlySchemaRepository
-
addReplicationMetadataSchema
public RmdSchemaEntry addReplicationMetadataSchema(String storeName, int valueSchemaId, String replicationMetadataSchemaStr, int replicationMetadataVersionId) - Specified by:
addReplicationMetadataSchema
in interfaceReadWriteSchemaRepository
-
refresh
public void refresh()- Specified by:
refresh
in interfaceVeniceResource
-
clear
public void clear()- Specified by:
clear
in interfaceVeniceResource
-