Package com.linkedin.davinci.repository

Class NativeMetadataRepository

java.lang.Object

com.linkedin.davinci.repository.NativeMetadataRepository

All Implemented Interfaces:

ClusterInfoProvider, ReadOnlySchemaRepository, ReadOnlyStoreRepository, SubscriptionBasedReadOnlyStoreRepository, VeniceResource

Direct Known Subclasses:

RequestBasedMetaRepository, ThinClientMetaStoreBasedRepository

public abstract class NativeMetadataRepository
extends Object
implements SubscriptionBasedReadOnlyStoreRepository, ReadOnlySchemaRepository, ClusterInfoProvider

Venice in-house implementation of a read only metadata repository where callers can retrieve various metadata such as Store objects and their corresponding schemas. The implementers of this abstract class all relies on some flavors of Venice system store to carry the metadata from Venice internal components (source) to external consumers such as a DaVinci client (destination). This abstract class includes the implementation of an in-memory cache for all subscribed stores' metadata. Callers are served by the cache and the cache is refreshed periodically by updating it with methods provided by the implementers.

Field Summary

Fields

Modifier and Type	Field	Description
protected final ClientConfig	clientConfig
protected Map<String,SchemaData>	schemaMap
protected final Map<String,Store>	subscribedStoreMap

Constructor Summary

Constructors

Modifier	Constructor	Description
protected	NativeMetadataRepository(ClientConfig clientConfig, VeniceProperties backendConfig)
protected	NativeMetadataRepository(ClientConfig clientConfig, VeniceProperties backendConfig, Clock clock)

Method Summary

Modifier and Type	Method	Description
protected StoreConfig	cacheStoreConfigFromRemote(String storeName)	Get the store cluster config from system store and update the local cache with it.
void	clear()	TODO: we may need to rename this function to be 'close' since this resource should not used any more after calling this function.
protected abstract StoreConfig	fetchStoreConfigFromRemote(String storeName)
protected abstract Store	fetchStoreFromRemote(String storeName, String clusterName)
List<Store>	getAllStores()	Get all stores in the current repository
protected SchemaData	getAndCacheSchemaData(String storeName)
int	getBatchGetLimit(String storeName)	Get batch-get limit for the specified store
DerivedSchemaEntry	getDerivedSchema(String storeName, int valueSchemaId, int derivedSchemaId)
GeneratedSchemaID	getDerivedSchemaId(String storeName, String derivedSchemaStr)	Look up derived schema id and its corresponding value schema id by given store name and derived schema.
Collection<DerivedSchemaEntry>	getDerivedSchemas(String storeName)
static NativeMetadataRepository	getInstance(ClientConfig clientConfig, VeniceProperties backendConfig)
static NativeMetadataRepository	getInstance(ClientConfig clientConfig, VeniceProperties backendConfig, ICProvider icProvider)
SchemaEntry	getKeySchema(String storeName)	This function is used to retrieve key schema for the given store.
DerivedSchemaEntry	getLatestDerivedSchema(String storeName, int valueSchemaId)	Get the most recent derived schema added to the given store and value schema id
RmdSchemaEntry	getReplicationMetadataSchema(String storeName, int valueSchemaId, int replicationMetadataVersionId)
Collection<RmdSchemaEntry>	getReplicationMetadataSchemas(String storeName)
protected abstract SchemaData	getSchemaData(String storeName)
Store	getStore(String storeName)	Get one store by given name from repository.
Store	getStoreOrThrow(String storeName)
SchemaEntry	getSupersetOrLatestValueSchema(String storeName)	Get the most recent value schema or superset value schema if one exists.
SchemaEntry	getSupersetSchema(String storeName)	Get the superset value schema for a given store.
long	getTotalStoreReadQuota()	Get total read quota of all stores.
SchemaEntry	getValueSchema(String storeName, int id)	This function is used to retrieve the value schema for the given store and value schema id.
int	getValueSchemaId(String storeName, String valueSchemaStr)	This function is used to retrieve value schema id for the given store and schema.
protected SchemaEntry	getValueSchemaInternally(String storeName, int id)
Collection<SchemaEntry>	getValueSchemas(String storeName)	This function is used to retrieve all the value schemas for the given store.
String	getVeniceCluster(String storeName)	Get the associated Venice cluster name given a Venice store name.
boolean	hasStore(String storeName)	Whether the store exists or not.
boolean	hasValueSchema(String storeName, int id)	This function is used to check whether the value schema id is valid in the given store.
boolean	isReadComputationEnabled(String storeName)	Whether computation is enabled for the specified store.
protected void	notifyStoreChanged(Store store)
protected void	notifyStoreCreated(Store store)
protected void	notifyStoreDeleted(Store store)
protected Store	putStore(Store newStore)
void	refresh()	This method will be triggered periodically to keep the store/schema information up-to-date.
Store	refreshOneStore(String storeName)	Selective refresh operation which fetches one store from ZK
void	registerStoreDataChangedListener(StoreDataChangedListener listener)	Register store data change listener.
protected Store	removeStore(String storeName)
protected void	removeStoreSchema(String storeName)	This function is used to remove schema entry for the given store from local cache, and related listeners as well.
void	start()
void	subscribe(String storeName)
void	unregisterStoreDataChangedListener(StoreDataChangedListener listener)	Unregister store data change listener.
void	unsubscribe(String storeName)

Methods inherited from class java.lang.Object

clone, equals, finalize, getClass, hashCode, notify, notifyAll, toString, wait, wait, wait

Methods inherited from interface com.linkedin.venice.meta.ReadOnlySchemaRepository

getLatestDerivedSchema

Methods inherited from interface com.linkedin.venice.meta.ReadOnlyStoreRepository

waitVersion, waitVersion

Field Details

clientConfig

protected final ClientConfig clientConfig

subscribedStoreMap

protected final Map<String,Store> subscribedStoreMap

schemaMap

protected Map<String,SchemaData> schemaMap

Constructor Details

NativeMetadataRepository

protected NativeMetadataRepository(ClientConfig clientConfig,
 VeniceProperties backendConfig)

NativeMetadataRepository

protected NativeMetadataRepository(ClientConfig clientConfig,
 VeniceProperties backendConfig,
 Clock clock)

Method Details

start

public void start()

getInstance

public static NativeMetadataRepository getInstance(ClientConfig clientConfig,
 VeniceProperties backendConfig)

getInstance

public static NativeMetadataRepository getInstance(ClientConfig clientConfig,
 VeniceProperties backendConfig,
 ICProvider icProvider)

subscribe

public void subscribe(String storeName)
               throws InterruptedException

Specified by:

subscribe in interface SubscriptionBasedReadOnlyStoreRepository

Throws:

InterruptedException

unsubscribe

public void unsubscribe(String storeName)

Specified by:

unsubscribe in interface SubscriptionBasedReadOnlyStoreRepository

getStore

public Store getStore(String storeName)

Description copied from interface: ReadOnlyStoreRepository

Get one store by given name from repository.

Specified by:

getStore in interface ReadOnlyStoreRepository

Parameters:

storeName - name of wanted store.

Returns:

Store for given name.

getStoreOrThrow

public Store getStoreOrThrow(String storeName)
                      throws VeniceNoStoreException

Specified by:

getStoreOrThrow in interface ReadOnlyStoreRepository

Throws:

VeniceNoStoreException

hasStore

public boolean hasStore(String storeName)

Description copied from interface: ReadOnlyStoreRepository

Whether the store exists or not.

Specified by:

hasStore in interface ReadOnlyStoreRepository

Parameters:

storeName - store name

Returns:

refreshOneStore

public Store refreshOneStore(String storeName)

Description copied from interface: ReadOnlyStoreRepository

Selective refresh operation which fetches one store from ZK

Specified by:

refreshOneStore in interface ReadOnlyStoreRepository

Parameters:

storeName - store name

Returns:

the newly refreshed store

getAllStores

public List<Store> getAllStores()

Description copied from interface: ReadOnlyStoreRepository

Get all stores in the current repository

Specified by:

getAllStores in interface ReadOnlyStoreRepository

Returns:

getTotalStoreReadQuota

public long getTotalStoreReadQuota()

Description copied from interface: ReadOnlyStoreRepository

Get total read quota of all stores.

Specified by:

getTotalStoreReadQuota in interface ReadOnlyStoreRepository

registerStoreDataChangedListener

public void registerStoreDataChangedListener(StoreDataChangedListener listener)

Description copied from interface: ReadOnlyStoreRepository

Register store data change listener.

Specified by:

registerStoreDataChangedListener in interface ReadOnlyStoreRepository

unregisterStoreDataChangedListener

public void unregisterStoreDataChangedListener(StoreDataChangedListener listener)

Description copied from interface: ReadOnlyStoreRepository

Unregister store data change listener.

Specified by:

unregisterStoreDataChangedListener in interface ReadOnlyStoreRepository

getBatchGetLimit

public int getBatchGetLimit(String storeName)

Description copied from interface: ReadOnlyStoreRepository

Get batch-get limit for the specified store

Specified by:

getBatchGetLimit in interface ReadOnlyStoreRepository

Returns:

isReadComputationEnabled

public boolean isReadComputationEnabled(String storeName)

Description copied from interface: ReadOnlyStoreRepository

Whether computation is enabled for the specified store.

Specified by:

isReadComputationEnabled in interface ReadOnlyStoreRepository

Parameters:

storeName - store name

Returns:

getKeySchema

public SchemaEntry getKeySchema(String storeName)

This function is used to retrieve key schema for the given store. If key schema for the given store doesn't exist, will return null; Otherwise, it will return the key schema;

Specified by:

getKeySchema in interface ReadOnlySchemaRepository

Returns:

null, if key schema for the given store doesn't exist; key schema entry, otherwise;

getValueSchema

public SchemaEntry getValueSchema(String storeName,
 int id)

This function is used to retrieve the value schema for the given store and value schema id. Caller shouldn't modify the returned SchemeEntry

Specified by:

getValueSchema in interface ReadOnlySchemaRepository

Returns:

null, if the schema doesn't exist; value schema entry, otherwise;

hasValueSchema

public boolean hasValueSchema(String storeName,
 int id)

This function is used to check whether the value schema id is valid in the given store.

Specified by:

hasValueSchema in interface ReadOnlySchemaRepository

getValueSchemaId

public int getValueSchemaId(String storeName,
 String valueSchemaStr)

This function is used to retrieve value schema id for the given store and schema.

Specified by:

getValueSchemaId in interface ReadOnlySchemaRepository

Returns:

SchemaData.INVALID_VALUE_SCHEMA_ID, if the schema doesn't exist in the given store; schema id (int), if the schema exists in the given store

getValueSchemas

public Collection<SchemaEntry> getValueSchemas(String storeName)

This function is used to retrieve all the value schemas for the given store.

Specified by:

getValueSchemas in interface ReadOnlySchemaRepository

getSupersetOrLatestValueSchema

public SchemaEntry getSupersetOrLatestValueSchema(String storeName)

Description copied from interface: ReadOnlySchemaRepository

Get the most recent value schema or superset value schema if one exists.

Specified by:

getSupersetOrLatestValueSchema in interface ReadOnlySchemaRepository

getSupersetSchema

public SchemaEntry getSupersetSchema(String storeName)

Description copied from interface: ReadOnlySchemaRepository

Get the superset value schema for a given store. Each store has at most one active superset schema. Specifically a store must have some features enabled (e.g. read compute, write compute) to have a superset value schema which evolves as new value schemas are added.

Specified by:

getSupersetSchema in interface ReadOnlySchemaRepository

Returns:

Superset value schema or null if store does not have any superset value schema.

getDerivedSchemaId

public GeneratedSchemaID getDerivedSchemaId(String storeName,
 String derivedSchemaStr)

Description copied from interface: ReadOnlySchemaRepository

Look up derived schema id and its corresponding value schema id by given store name and derived schema. This is likely used by clients that write to Venice

Specified by:

getDerivedSchemaId in interface ReadOnlySchemaRepository

Returns:

a pair where the first value is value schema id and the second value is derived schema id

getDerivedSchema

public DerivedSchemaEntry getDerivedSchema(String storeName,
 int valueSchemaId,
 int derivedSchemaId)

Specified by:

getDerivedSchema in interface ReadOnlySchemaRepository

getDerivedSchemas

public Collection<DerivedSchemaEntry> getDerivedSchemas(String storeName)

Specified by:

getDerivedSchemas in interface ReadOnlySchemaRepository

getLatestDerivedSchema

public DerivedSchemaEntry getLatestDerivedSchema(String storeName,
 int valueSchemaId)

Description copied from interface: ReadOnlySchemaRepository

Get the most recent derived schema added to the given store and value schema id

Specified by:

getLatestDerivedSchema in interface ReadOnlySchemaRepository

getReplicationMetadataSchema

public RmdSchemaEntry getReplicationMetadataSchema(String storeName,
 int valueSchemaId,
 int replicationMetadataVersionId)

Specified by:

getReplicationMetadataSchema in interface ReadOnlySchemaRepository

getReplicationMetadataSchemas

public Collection<RmdSchemaEntry> getReplicationMetadataSchemas(String storeName)

Specified by:

getReplicationMetadataSchemas in interface ReadOnlySchemaRepository

refresh

public void refresh()

This method will be triggered periodically to keep the store/schema information up-to-date.

Specified by:

refresh in interface VeniceResource

clear

public void clear()

TODO: we may need to rename this function to be 'close' since this resource should not used any more after calling this function.

Specified by:

clear in interface VeniceResource

cacheStoreConfigFromRemote

protected StoreConfig cacheStoreConfigFromRemote(String storeName)

Get the store cluster config from system store and update the local cache with it. Different implementation will get the data differently but should all populate the store cluster config map.

fetchStoreConfigFromRemote

protected abstract StoreConfig fetchStoreConfigFromRemote(String storeName)

fetchStoreFromRemote

protected abstract Store fetchStoreFromRemote(String storeName,
 String clusterName)

getSchemaData

protected abstract SchemaData getSchemaData(String storeName)

putStore

protected Store putStore(Store newStore)

removeStore

protected Store removeStore(String storeName)

notifyStoreCreated

protected void notifyStoreCreated(Store store)

notifyStoreDeleted

protected void notifyStoreDeleted(Store store)

notifyStoreChanged

protected void notifyStoreChanged(Store store)

getAndCacheSchemaData

protected SchemaData getAndCacheSchemaData(String storeName)

getValueSchemaInternally

protected SchemaEntry getValueSchemaInternally(String storeName,
 int id)

removeStoreSchema

protected void removeStoreSchema(String storeName)

This function is used to remove schema entry for the given store from local cache, and related listeners as well.

getVeniceCluster

public String getVeniceCluster(String storeName)

Description copied from interface: ClusterInfoProvider

Get the associated Venice cluster name given a Venice store name.

Specified by:

getVeniceCluster in interface ClusterInfoProvider

Returns:

the cluster name that the store belongs to or null if such information cannot be provided.