Package com.linkedin.davinci.repository
Class NativeMetadataRepository
- java.lang.Object
-
- com.linkedin.davinci.repository.NativeMetadataRepository
-
- All Implemented Interfaces:
ClusterInfoProvider
,ReadOnlySchemaRepository
,ReadOnlyStoreRepository
,SubscriptionBasedReadOnlyStoreRepository
,VeniceResource
- Direct Known Subclasses:
ThinClientMetaStoreBasedRepository
public abstract class NativeMetadataRepository extends java.lang.Object implements SubscriptionBasedReadOnlyStoreRepository, ReadOnlySchemaRepository, ClusterInfoProvider
Venice in-house implementation of a read only metadata repository where callers can retrieve various metadata such as Store objects and their corresponding schemas. The implementers of this abstract class all relies on some flavors of Venice system store to carry the metadata from Venice internal components (source) to external consumers such as a DaVinci client (destination). This abstract class includes the implementation of an in-memory cache for all subscribed stores' metadata. Callers are served by the cache and the cache is refreshed periodically by updating it with methods provided by the implementers.
-
-
Field Summary
Fields Modifier and Type Field Description protected ClientConfig
clientConfig
protected java.util.Map<java.lang.String,Store>
subscribedStoreMap
-
Constructor Summary
Constructors Modifier Constructor Description protected
NativeMetadataRepository(ClientConfig clientConfig, VeniceProperties backendConfig)
protected
NativeMetadataRepository(ClientConfig clientConfig, VeniceProperties backendConfig, java.time.Clock clock)
-
Method Summary
All Methods Static Methods Instance Methods Abstract Methods Concrete Methods Modifier and Type Method Description void
clear()
TODO: we may need to rename this function to be 'close' since this resource should not used any more after calling this function.java.util.List<Store>
getAllStores()
Get all stores in the current repositoryprotected SchemaData
getAndCacheSchemaDataFromSystemStore(java.lang.String storeName)
protected void
getAndSetStoreConfigFromSystemStore(java.lang.String storeName)
Get the store cluster config from system store and update the local cache with it.int
getBatchGetLimit(java.lang.String storeName)
Get batch-get limit for the specified storeDerivedSchemaEntry
getDerivedSchema(java.lang.String storeName, int valueSchemaId, int derivedSchemaId)
GeneratedSchemaID
getDerivedSchemaId(java.lang.String storeName, java.lang.String derivedSchemaStr)
Look up derived schema id and its corresponding value schema id by given store name and derived schema.java.util.Collection<DerivedSchemaEntry>
getDerivedSchemas(java.lang.String storeName)
static NativeMetadataRepository
getInstance(ClientConfig clientConfig, VeniceProperties backendConfig)
static NativeMetadataRepository
getInstance(ClientConfig clientConfig, VeniceProperties backendConfig, ICProvider icProvider)
SchemaEntry
getKeySchema(java.lang.String storeName)
This function is used to retrieve key schema for the given store.DerivedSchemaEntry
getLatestDerivedSchema(java.lang.String storeName, int valueSchemaId)
Get the most recent derived schema added to the given store and value schema idRmdSchemaEntry
getReplicationMetadataSchema(java.lang.String storeName, int valueSchemaId, int replicationMetadataVersionId)
java.util.Collection<RmdSchemaEntry>
getReplicationMetadataSchemas(java.lang.String storeName)
protected SchemaData
getSchemaDataFromMetaSystemStore(java.lang.String storeName)
protected abstract SchemaData
getSchemaDataFromSystemStore(java.lang.String storeName)
Store
getStore(java.lang.String storeName)
Get one store by given name from repository.protected StoreConfig
getStoreConfigFromMetaSystemStore(java.lang.String storeName)
protected abstract StoreConfig
getStoreConfigFromSystemStore(java.lang.String storeName)
protected abstract Store
getStoreFromSystemStore(java.lang.String storeName, java.lang.String clusterName)
protected abstract StoreMetaValue
getStoreMetaValue(java.lang.String storeName, StoreMetaKey key)
Store
getStoreOrThrow(java.lang.String storeName)
SchemaEntry
getSupersetOrLatestValueSchema(java.lang.String storeName)
Get the most recent value schema or superset value schema if one exists.SchemaEntry
getSupersetSchema(java.lang.String storeName)
Get the superset value schema for a given store.long
getTotalStoreReadQuota()
Get total read quota of all stores.SchemaEntry
getValueSchema(java.lang.String storeName, int id)
This function is used to retrieve the value schema for the given store and value schema id.int
getValueSchemaId(java.lang.String storeName, java.lang.String valueSchemaStr)
This function is used to retrieve value schema id for the given store and schema.protected SchemaEntry
getValueSchemaInternally(java.lang.String storeName, int id)
java.util.Collection<SchemaEntry>
getValueSchemas(java.lang.String storeName)
This function is used to retrieve all the value schemas for the given store.java.lang.String
getVeniceCluster(java.lang.String storeName)
Get the associated Venice cluster name given a Venice store name.boolean
hasStore(java.lang.String storeName)
Whether the store exists or not.boolean
hasValueSchema(java.lang.String storeName, int id)
This function is used to check whether the value schema id is valid in the given store.boolean
isReadComputationEnabled(java.lang.String storeName)
Whether computation is enabled for the specified store.protected void
notifyStoreChanged(Store store)
protected void
notifyStoreCreated(Store store)
protected void
notifyStoreDeleted(Store store)
protected Store
putStore(Store newStore)
void
refresh()
This method will be triggered periodically to keep the store/schema information up-to-date.Store
refreshOneStore(java.lang.String storeName)
Selective refresh operation which fetches one store from ZKvoid
registerStoreDataChangedListener(StoreDataChangedListener listener)
Register store data change listener.protected Store
removeStore(java.lang.String storeName)
protected void
removeStoreSchema(java.lang.String storeName)
This function is used to remove schema entry for the given store from local cache, and related listeners as well.void
start()
void
subscribe(java.lang.String storeName)
void
unregisterStoreDataChangedListener(StoreDataChangedListener listener)
Unregister store data change listener.void
unsubscribe(java.lang.String storeName)
-
Methods inherited from class java.lang.Object
clone, equals, finalize, getClass, hashCode, notify, notifyAll, toString, wait, wait, wait
-
Methods inherited from interface com.linkedin.venice.meta.ReadOnlySchemaRepository
getLatestDerivedSchema
-
Methods inherited from interface com.linkedin.venice.meta.ReadOnlyStoreRepository
waitVersion, waitVersion
-
-
-
-
Field Detail
-
clientConfig
protected final ClientConfig clientConfig
-
subscribedStoreMap
protected final java.util.Map<java.lang.String,Store> subscribedStoreMap
-
-
Constructor Detail
-
NativeMetadataRepository
protected NativeMetadataRepository(ClientConfig clientConfig, VeniceProperties backendConfig)
-
NativeMetadataRepository
protected NativeMetadataRepository(ClientConfig clientConfig, VeniceProperties backendConfig, java.time.Clock clock)
-
-
Method Detail
-
start
public void start()
-
getInstance
public static NativeMetadataRepository getInstance(ClientConfig clientConfig, VeniceProperties backendConfig)
-
getInstance
public static NativeMetadataRepository getInstance(ClientConfig clientConfig, VeniceProperties backendConfig, ICProvider icProvider)
-
subscribe
public void subscribe(java.lang.String storeName) throws java.lang.InterruptedException
- Specified by:
subscribe
in interfaceSubscriptionBasedReadOnlyStoreRepository
- Throws:
java.lang.InterruptedException
-
unsubscribe
public void unsubscribe(java.lang.String storeName)
- Specified by:
unsubscribe
in interfaceSubscriptionBasedReadOnlyStoreRepository
-
getStore
public Store getStore(java.lang.String storeName)
Description copied from interface:ReadOnlyStoreRepository
Get one store by given name from repository.- Specified by:
getStore
in interfaceReadOnlyStoreRepository
- Parameters:
storeName
- name of wanted store.- Returns:
- Store for given name.
-
getStoreOrThrow
public Store getStoreOrThrow(java.lang.String storeName) throws VeniceNoStoreException
- Specified by:
getStoreOrThrow
in interfaceReadOnlyStoreRepository
- Throws:
VeniceNoStoreException
-
hasStore
public boolean hasStore(java.lang.String storeName)
Description copied from interface:ReadOnlyStoreRepository
Whether the store exists or not.- Specified by:
hasStore
in interfaceReadOnlyStoreRepository
- Parameters:
storeName
- store name- Returns:
-
refreshOneStore
public Store refreshOneStore(java.lang.String storeName)
Description copied from interface:ReadOnlyStoreRepository
Selective refresh operation which fetches one store from ZK- Specified by:
refreshOneStore
in interfaceReadOnlyStoreRepository
- Parameters:
storeName
- store name- Returns:
- the newly refreshed store
-
getAllStores
public java.util.List<Store> getAllStores()
Description copied from interface:ReadOnlyStoreRepository
Get all stores in the current repository- Specified by:
getAllStores
in interfaceReadOnlyStoreRepository
- Returns:
-
getTotalStoreReadQuota
public long getTotalStoreReadQuota()
Description copied from interface:ReadOnlyStoreRepository
Get total read quota of all stores.- Specified by:
getTotalStoreReadQuota
in interfaceReadOnlyStoreRepository
-
registerStoreDataChangedListener
public void registerStoreDataChangedListener(StoreDataChangedListener listener)
Description copied from interface:ReadOnlyStoreRepository
Register store data change listener.- Specified by:
registerStoreDataChangedListener
in interfaceReadOnlyStoreRepository
-
unregisterStoreDataChangedListener
public void unregisterStoreDataChangedListener(StoreDataChangedListener listener)
Description copied from interface:ReadOnlyStoreRepository
Unregister store data change listener.- Specified by:
unregisterStoreDataChangedListener
in interfaceReadOnlyStoreRepository
-
getBatchGetLimit
public int getBatchGetLimit(java.lang.String storeName)
Description copied from interface:ReadOnlyStoreRepository
Get batch-get limit for the specified store- Specified by:
getBatchGetLimit
in interfaceReadOnlyStoreRepository
- Returns:
-
isReadComputationEnabled
public boolean isReadComputationEnabled(java.lang.String storeName)
Description copied from interface:ReadOnlyStoreRepository
Whether computation is enabled for the specified store.- Specified by:
isReadComputationEnabled
in interfaceReadOnlyStoreRepository
- Parameters:
storeName
- store name- Returns:
-
getKeySchema
public SchemaEntry getKeySchema(java.lang.String storeName)
This function is used to retrieve key schema for the given store. If key schema for the given store doesn't exist, will return null; Otherwise, it will return the key schema;- Specified by:
getKeySchema
in interfaceReadOnlySchemaRepository
- Returns:
- null, if key schema for the given store doesn't exist; key schema entry, otherwise;
-
getValueSchema
public SchemaEntry getValueSchema(java.lang.String storeName, int id)
This function is used to retrieve the value schema for the given store and value schema id. Caller shouldn't modify the returned SchemeEntry- Specified by:
getValueSchema
in interfaceReadOnlySchemaRepository
- Returns:
- null, if the schema doesn't exist; value schema entry, otherwise;
-
hasValueSchema
public boolean hasValueSchema(java.lang.String storeName, int id)
This function is used to check whether the value schema id is valid in the given store.- Specified by:
hasValueSchema
in interfaceReadOnlySchemaRepository
-
getValueSchemaId
public int getValueSchemaId(java.lang.String storeName, java.lang.String valueSchemaStr)
This function is used to retrieve value schema id for the given store and schema.- Specified by:
getValueSchemaId
in interfaceReadOnlySchemaRepository
- Returns:
SchemaData.INVALID_VALUE_SCHEMA_ID
, if the schema doesn't exist in the given store; schema id (int), if the schema exists in the given store
-
getValueSchemas
public java.util.Collection<SchemaEntry> getValueSchemas(java.lang.String storeName)
This function is used to retrieve all the value schemas for the given store.- Specified by:
getValueSchemas
in interfaceReadOnlySchemaRepository
-
getSupersetOrLatestValueSchema
public SchemaEntry getSupersetOrLatestValueSchema(java.lang.String storeName)
Description copied from interface:ReadOnlySchemaRepository
Get the most recent value schema or superset value schema if one exists.- Specified by:
getSupersetOrLatestValueSchema
in interfaceReadOnlySchemaRepository
-
getSupersetSchema
public SchemaEntry getSupersetSchema(java.lang.String storeName)
Description copied from interface:ReadOnlySchemaRepository
Get the superset value schema for a given store. Each store has at most one active superset schema. Specifically a store must have some features enabled (e.g. read compute, write compute) to have a superset value schema which evolves as new value schemas are added.- Specified by:
getSupersetSchema
in interfaceReadOnlySchemaRepository
- Returns:
- Superset value schema or
null
if store {@param storeName} does not have any superset value schema.
-
getDerivedSchemaId
public GeneratedSchemaID getDerivedSchemaId(java.lang.String storeName, java.lang.String derivedSchemaStr)
Description copied from interface:ReadOnlySchemaRepository
Look up derived schema id and its corresponding value schema id by given store name and derived schema. This is likely used by clients that write to Venice- Specified by:
getDerivedSchemaId
in interfaceReadOnlySchemaRepository
- Returns:
- a pair where the first value is value schema id and the second value is derived schema id
-
getDerivedSchema
public DerivedSchemaEntry getDerivedSchema(java.lang.String storeName, int valueSchemaId, int derivedSchemaId)
- Specified by:
getDerivedSchema
in interfaceReadOnlySchemaRepository
-
getDerivedSchemas
public java.util.Collection<DerivedSchemaEntry> getDerivedSchemas(java.lang.String storeName)
- Specified by:
getDerivedSchemas
in interfaceReadOnlySchemaRepository
-
getLatestDerivedSchema
public DerivedSchemaEntry getLatestDerivedSchema(java.lang.String storeName, int valueSchemaId)
Description copied from interface:ReadOnlySchemaRepository
Get the most recent derived schema added to the given store and value schema id- Specified by:
getLatestDerivedSchema
in interfaceReadOnlySchemaRepository
-
getReplicationMetadataSchema
public RmdSchemaEntry getReplicationMetadataSchema(java.lang.String storeName, int valueSchemaId, int replicationMetadataVersionId)
- Specified by:
getReplicationMetadataSchema
in interfaceReadOnlySchemaRepository
-
getReplicationMetadataSchemas
public java.util.Collection<RmdSchemaEntry> getReplicationMetadataSchemas(java.lang.String storeName)
- Specified by:
getReplicationMetadataSchemas
in interfaceReadOnlySchemaRepository
-
refresh
public void refresh()
This method will be triggered periodically to keep the store/schema information up-to-date.- Specified by:
refresh
in interfaceVeniceResource
-
clear
public void clear()
TODO: we may need to rename this function to be 'close' since this resource should not used any more after calling this function.- Specified by:
clear
in interfaceVeniceResource
-
getAndSetStoreConfigFromSystemStore
protected void getAndSetStoreConfigFromSystemStore(java.lang.String storeName)
Get the store cluster config from system store and update the local cache with it. Different implementation will get the data differently but should all populate the store cluster config map.
-
getStoreConfigFromSystemStore
protected abstract StoreConfig getStoreConfigFromSystemStore(java.lang.String storeName)
-
getStoreFromSystemStore
protected abstract Store getStoreFromSystemStore(java.lang.String storeName, java.lang.String clusterName)
-
getStoreMetaValue
protected abstract StoreMetaValue getStoreMetaValue(java.lang.String storeName, StoreMetaKey key)
-
getStoreConfigFromMetaSystemStore
protected StoreConfig getStoreConfigFromMetaSystemStore(java.lang.String storeName)
-
getSchemaDataFromMetaSystemStore
protected SchemaData getSchemaDataFromMetaSystemStore(java.lang.String storeName)
-
removeStore
protected Store removeStore(java.lang.String storeName)
-
notifyStoreCreated
protected void notifyStoreCreated(Store store)
-
notifyStoreDeleted
protected void notifyStoreDeleted(Store store)
-
notifyStoreChanged
protected void notifyStoreChanged(Store store)
-
getAndCacheSchemaDataFromSystemStore
protected SchemaData getAndCacheSchemaDataFromSystemStore(java.lang.String storeName)
-
getValueSchemaInternally
protected SchemaEntry getValueSchemaInternally(java.lang.String storeName, int id)
-
getSchemaDataFromSystemStore
protected abstract SchemaData getSchemaDataFromSystemStore(java.lang.String storeName)
-
removeStoreSchema
protected void removeStoreSchema(java.lang.String storeName)
This function is used to remove schema entry for the given store from local cache, and related listeners as well.
-
getVeniceCluster
public java.lang.String getVeniceCluster(java.lang.String storeName)
Description copied from interface:ClusterInfoProvider
Get the associated Venice cluster name given a Venice store name.- Specified by:
getVeniceCluster
in interfaceClusterInfoProvider
- Returns:
- the cluster name that the store belongs to or null if such information cannot be provided.
-
-