Package com.linkedin.venice.helix
Class HelixReadWriteSchemaRepository
- java.lang.Object
-
- com.linkedin.venice.helix.HelixReadWriteSchemaRepository
-
- All Implemented Interfaces:
ReadOnlySchemaRepository
,ReadWriteSchemaRepository
,VeniceResource
public class HelixReadWriteSchemaRepository extends java.lang.Object implements ReadWriteSchemaRepository
This class is used to add schema entries for stores. There are 4 types of schema entries 1. Key schema ZK Path: ${cluster_name}/Stores/${store_name}/key-schema/1 Each store only has 1 key schema and the schema is immutable. 2. Value schema ZK Path: ${cluster_name}/Stores/${store_name}/value-schema/${value_schema_id} Value schemas are evolvable. Stores can have multiple value schemas and each value schema is forwards/backwards compatible with others. 3. Derived schema ZK Path: ${cluster_name}/Stores/${store_name}/derived-schema/${value_schema_id}_${derived_schema_id} Each value schema can have multiple derived schemas. check outDerivedSchemaEntry
for more details. 3. Replication metadata schema * ZK Path: ${cluster_name}/Stores/${store_name}/timestamp-metadata-schema/${value_schema_id}-${replication_metadata_version_id} * Check outSchemaEntrySerializer
andDerivedSchemaEntrySerializer
to see how schemas are ser-ded. ReadWriteSchemaRepository doesn't cache existing schemas locally and it always queries ZK for currently values. This is a different behavior compared toReadOnlyStoreRepository
where values always get cached and future update callbacks are registered. Notice: Users should not instantiate this class elsewhere than in the leader Controller and there should be always only 1 ReadWriteSchemaRepository per cluster. Instantiating multiple ReadWriteSchemaRepository will lead to race conditions in ZK.
-
-
Constructor Summary
Constructors Constructor Description HelixReadWriteSchemaRepository(ReadWriteStoreRepository storeRepository, java.util.Optional<MetaStoreWriter> metaStoreWriter, HelixSchemaAccessor accessor)
HelixReadWriteSchemaRepository(ReadWriteStoreRepository storeRepository, org.apache.helix.zookeeper.impl.client.ZkClient zkClient, HelixAdapterSerializer adapter, java.lang.String clusterName, java.util.Optional<MetaStoreWriter> metaStoreWriter)
-
Method Summary
All Methods Instance Methods Concrete Methods Modifier and Type Method Description DerivedSchemaEntry
addDerivedSchema(java.lang.String storeName, java.lang.String schemaStr, int valueSchemaId)
Add a new derived schema for the given store and value schema idDerivedSchemaEntry
addDerivedSchema(java.lang.String storeName, java.lang.String schemaStr, int valueSchemaId, int derivedSchemaId)
Add a new derived schema for the given store by specifying derived schema id.RmdSchemaEntry
addReplicationMetadataSchema(java.lang.String storeName, int valueSchemaId, java.lang.String replicationMetadataSchemaStr, int replicationMetadataVersionId)
SchemaEntry
addValueSchema(java.lang.String storeName, java.lang.String schemaStr, int schemaId)
Add a new value schema for the given store by specifying schema id.SchemaEntry
addValueSchema(java.lang.String storeName, java.lang.String schemaStr, DirectionalSchemaCompatibilityType expectedCompatibilityType)
Add new value schema for the given store.void
clear()
DerivedSchemaEntry
getDerivedSchema(java.lang.String storeName, int valueSchemaId, int derivedSchemaId)
GeneratedSchemaID
getDerivedSchemaId(java.lang.String storeName, java.lang.String derivedSchemaStr)
Look up derived schema id and its corresponding value schema id by given store name and derived schema.java.util.Collection<DerivedSchemaEntry>
getDerivedSchemas(java.lang.String storeName)
SchemaEntry
getKeySchema(java.lang.String storeName)
Get key schema for the given store.DerivedSchemaEntry
getLatestDerivedSchema(java.lang.String storeName, int valueSchemaId)
Get the most recent derived schema added to the given store and value schema idRmdSchemaEntry
getReplicationMetadataSchema(java.lang.String storeName, int valueSchemaId, int replicationMetadataVersionId)
java.util.Collection<RmdSchemaEntry>
getReplicationMetadataSchemas(java.lang.String storeName)
SchemaEntry
getSupersetOrLatestValueSchema(java.lang.String storeName)
Get the most recent value schema or superset value schema if one exists.SchemaEntry
getSupersetSchema(java.lang.String storeName)
Get the superset value schema for a given store.SchemaEntry
getValueSchema(java.lang.String storeName, int id)
Get value schema for the given store and schema id.int
getValueSchemaId(java.lang.String storeName, java.lang.String valueSchemaStr)
This function is used to retrieve value schema id for the given store and schema.java.util.Collection<SchemaEntry>
getValueSchemas(java.lang.String storeName)
This function is used to retrieve all the value schemas for the given store.boolean
hasValueSchema(java.lang.String storeName, int id)
Check whether the given value schema id exists in the given store or not.SchemaEntry
initKeySchema(java.lang.String storeName, java.lang.String schemaStr)
Set up key schema for the given store.int
preCheckDerivedSchemaAndGetNextAvailableId(java.lang.String storeName, int valueSchemaId, java.lang.String derivedSchemaStr)
int
preCheckValueSchemaAndGetNextAvailableId(java.lang.String storeName, java.lang.String valueSchemaStr, DirectionalSchemaCompatibilityType expectedCompatibilityType)
Check if the incoming schema is a valid schema and return the next available schema ID.void
refresh()
DerivedSchemaEntry
removeDerivedSchema(java.lang.String storeName, int valueSchemaId, int derivedSchemaId)
Remove an existing derived schemavoid
removeValueSchema(java.lang.String storeName, int valueSchemaId)
-
Methods inherited from class java.lang.Object
clone, equals, finalize, getClass, hashCode, notify, notifyAll, toString, wait, wait, wait
-
Methods inherited from interface com.linkedin.venice.meta.ReadOnlySchemaRepository
getLatestDerivedSchema
-
Methods inherited from interface com.linkedin.venice.meta.ReadWriteSchemaRepository
addValueSchema
-
-
-
-
Constructor Detail
-
HelixReadWriteSchemaRepository
public HelixReadWriteSchemaRepository(ReadWriteStoreRepository storeRepository, java.util.Optional<MetaStoreWriter> metaStoreWriter, HelixSchemaAccessor accessor)
-
HelixReadWriteSchemaRepository
public HelixReadWriteSchemaRepository(ReadWriteStoreRepository storeRepository, org.apache.helix.zookeeper.impl.client.ZkClient zkClient, HelixAdapterSerializer adapter, java.lang.String clusterName, java.util.Optional<MetaStoreWriter> metaStoreWriter)
-
-
Method Detail
-
getKeySchema
public SchemaEntry getKeySchema(java.lang.String storeName)
Get key schema for the given store. Fetch from zookeeper directly.- Specified by:
getKeySchema
in interfaceReadOnlySchemaRepository
- Returns:
- null if key schema doesn't exist; schema entry if exists;
-
getValueSchema
public SchemaEntry getValueSchema(java.lang.String storeName, int id)
Get value schema for the given store and schema id. Fetch from zookeeper directly.- Specified by:
getValueSchema
in interfaceReadOnlySchemaRepository
- Returns:
- null if the schema doesn't exist; schema entry if exists;
-
hasValueSchema
public boolean hasValueSchema(java.lang.String storeName, int id)
Check whether the given value schema id exists in the given store or not. Fetch from zookeeper directly.- Specified by:
hasValueSchema
in interfaceReadOnlySchemaRepository
- Returns:
- null if the schema doesn't exist; schema entry if exists;
-
getValueSchemaId
public int getValueSchemaId(java.lang.String storeName, java.lang.String valueSchemaStr)
This function is used to retrieve value schema id for the given store and schema. Attempts to get the schema that matches exactly. If multiple matching schemas are found then the id of the latest added schema is returned.- Specified by:
getValueSchemaId
in interfaceReadOnlySchemaRepository
- Returns:
SchemaData.INVALID_VALUE_SCHEMA_ID
, if the schema doesn't exist in the given store; schema id (int), if the schema exists in the given store;
-
getValueSchemas
public java.util.Collection<SchemaEntry> getValueSchemas(java.lang.String storeName)
This function is used to retrieve all the value schemas for the given store. Fetch from zookeeper directly.- Specified by:
getValueSchemas
in interfaceReadOnlySchemaRepository
-
getDerivedSchemas
public java.util.Collection<DerivedSchemaEntry> getDerivedSchemas(java.lang.String storeName)
- Specified by:
getDerivedSchemas
in interfaceReadOnlySchemaRepository
-
getSupersetOrLatestValueSchema
public SchemaEntry getSupersetOrLatestValueSchema(java.lang.String storeName)
Description copied from interface:ReadOnlySchemaRepository
Get the most recent value schema or superset value schema if one exists.- Specified by:
getSupersetOrLatestValueSchema
in interfaceReadOnlySchemaRepository
-
getSupersetSchema
public SchemaEntry getSupersetSchema(java.lang.String storeName)
Description copied from interface:ReadOnlySchemaRepository
Get the superset value schema for a given store. Each store has at most one active superset schema. Specifically a store must have some features enabled (e.g. read compute, write compute) to have a superset value schema which evolves as new value schemas are added.- Specified by:
getSupersetSchema
in interfaceReadOnlySchemaRepository
- Returns:
- Superset value schema or
null
if store {@param storeName} does not have any superset value schema.
-
getLatestDerivedSchema
public DerivedSchemaEntry getLatestDerivedSchema(java.lang.String storeName, int valueSchemaId)
Description copied from interface:ReadOnlySchemaRepository
Get the most recent derived schema added to the given store and value schema id- Specified by:
getLatestDerivedSchema
in interfaceReadOnlySchemaRepository
-
getDerivedSchema
public DerivedSchemaEntry getDerivedSchema(java.lang.String storeName, int valueSchemaId, int derivedSchemaId)
- Specified by:
getDerivedSchema
in interfaceReadOnlySchemaRepository
-
initKeySchema
public SchemaEntry initKeySchema(java.lang.String storeName, java.lang.String schemaStr)
Set up key schema for the given store.- Specified by:
initKeySchema
in interfaceReadWriteSchemaRepository
-
addValueSchema
public SchemaEntry addValueSchema(java.lang.String storeName, java.lang.String schemaStr, DirectionalSchemaCompatibilityType expectedCompatibilityType)
Add new value schema for the given store.- Specified by:
addValueSchema
in interfaceReadWriteSchemaRepository
- Returns:
- schema entry if the schema is successfully added or already exists.
-
addValueSchema
public SchemaEntry addValueSchema(java.lang.String storeName, java.lang.String schemaStr, int schemaId)
Description copied from interface:ReadWriteSchemaRepository
Add a new value schema for the given store by specifying schema id. This API is mostly intended to be used in cross-colo mode. When there are multiple colos, we'd like to have consistent value id across colos, so that deserializer can work properly while reading records. Caller should figure out the schema id number by themselves. TODO: Might want to remove it from the interface and make it invisible from the outside- Specified by:
addValueSchema
in interfaceReadWriteSchemaRepository
-
preCheckValueSchemaAndGetNextAvailableId
public int preCheckValueSchemaAndGetNextAvailableId(java.lang.String storeName, java.lang.String valueSchemaStr, DirectionalSchemaCompatibilityType expectedCompatibilityType)
Check if the incoming schema is a valid schema and return the next available schema ID. Venice pre-checks 3 things: 1. If the store is existing or not 2. If the incoming schema contains any reserved fields 3. If the incoming schema is duplicate with current's.- Specified by:
preCheckValueSchemaAndGetNextAvailableId
in interfaceReadWriteSchemaRepository
- Returns:
- next available ID if it's a valid schema or
SchemaData.DUPLICATE_VALUE_SCHEMA_CODE
if it's a duplicate
-
preCheckDerivedSchemaAndGetNextAvailableId
public int preCheckDerivedSchemaAndGetNextAvailableId(java.lang.String storeName, int valueSchemaId, java.lang.String derivedSchemaStr)
- Specified by:
preCheckDerivedSchemaAndGetNextAvailableId
in interfaceReadWriteSchemaRepository
-
addDerivedSchema
public DerivedSchemaEntry addDerivedSchema(java.lang.String storeName, java.lang.String schemaStr, int valueSchemaId)
Description copied from interface:ReadWriteSchemaRepository
Add a new derived schema for the given store and value schema id- Specified by:
addDerivedSchema
in interfaceReadWriteSchemaRepository
-
addDerivedSchema
public DerivedSchemaEntry addDerivedSchema(java.lang.String storeName, java.lang.String schemaStr, int valueSchemaId, int derivedSchemaId)
Description copied from interface:ReadWriteSchemaRepository
Add a new derived schema for the given store by specifying derived schema id. Mostly used in cross-colo mode.- Specified by:
addDerivedSchema
in interfaceReadWriteSchemaRepository
-
removeDerivedSchema
public DerivedSchemaEntry removeDerivedSchema(java.lang.String storeName, int valueSchemaId, int derivedSchemaId)
Description copied from interface:ReadWriteSchemaRepository
Remove an existing derived schema- Specified by:
removeDerivedSchema
in interfaceReadWriteSchemaRepository
- Returns:
- the derived schema that is deleted or null if the schema doesn't exist
-
getDerivedSchemaId
public GeneratedSchemaID getDerivedSchemaId(java.lang.String storeName, java.lang.String derivedSchemaStr)
Description copied from interface:ReadOnlySchemaRepository
Look up derived schema id and its corresponding value schema id by given store name and derived schema. This is likely used by clients that write to Venice- Specified by:
getDerivedSchemaId
in interfaceReadOnlySchemaRepository
- Returns:
- a pair where the first value is value schema id and the second value is derived schema id
-
removeValueSchema
public void removeValueSchema(java.lang.String storeName, int valueSchemaId)
- Specified by:
removeValueSchema
in interfaceReadWriteSchemaRepository
-
getReplicationMetadataSchemas
public java.util.Collection<RmdSchemaEntry> getReplicationMetadataSchemas(java.lang.String storeName)
- Specified by:
getReplicationMetadataSchemas
in interfaceReadOnlySchemaRepository
-
getReplicationMetadataSchema
public RmdSchemaEntry getReplicationMetadataSchema(java.lang.String storeName, int valueSchemaId, int replicationMetadataVersionId)
- Specified by:
getReplicationMetadataSchema
in interfaceReadOnlySchemaRepository
-
addReplicationMetadataSchema
public RmdSchemaEntry addReplicationMetadataSchema(java.lang.String storeName, int valueSchemaId, java.lang.String replicationMetadataSchemaStr, int replicationMetadataVersionId)
- Specified by:
addReplicationMetadataSchema
in interfaceReadWriteSchemaRepository
-
refresh
public void refresh()
- Specified by:
refresh
in interfaceVeniceResource
-
clear
public void clear()
- Specified by:
clear
in interfaceVeniceResource
-
-