Package com.linkedin.venice.helix

Class HelixBaseRoutingRepository

java.lang.Object

com.linkedin.venice.helix.HelixBaseRoutingRepository

All Implemented Interfaces:

OnlineInstanceFinder, RoutingDataRepository, VeniceResource, org.apache.helix.api.listeners.ControllerChangeListener, org.apache.helix.api.listeners.RoutingTableChangeListener

Direct Known Subclasses:

HelixCustomizedViewOfflinePushRepository, HelixExternalViewRepository

public abstract class HelixBaseRoutingRepository
extends Object
implements RoutingDataRepository, org.apache.helix.api.listeners.ControllerChangeListener, org.apache.helix.api.listeners.RoutingTableChangeListener

Get routing data from Helix and convert it to our Venice partition and replica objects.

Although Helix RoutingTableProvider already cached routing data in local memory. But it only gets data from /$cluster/EXTERNALVIEW, /$cluster/CONFIGS/PARTICIPANTS, /$cluster/CUSTOMIZEDVIEW. Two parts of data are missed: Additional data in /$cluster/LIVEINSTANCES and partition number in /$cluster/IDEALSTATES. So we cached Venice partitions and instances here to include all of them and also convert them from Helix data structure to Venice data structure.

As this repository is used by Router, so here only cached the online instance at first. If Venice needs some more instances in other state, could add them in the further.

Nested Class Summary

Nested classes/interfaces inherited from interface com.linkedin.venice.meta.RoutingDataRepository

RoutingDataRepository.RoutingDataChangedListener

Field Summary

Fields

Modifier and Type	Field	Description
protected final Map<org.apache.helix.PropertyType,List<String>>	dataSource
protected final org.apache.helix.PropertyKey.Builder	keyBuilder	Builder used to build the data path to access Helix internal data.
protected final ListenerManager<RoutingDataRepository.RoutingDataChangedListener>	listenerManager
protected Map<String,Instance>	liveInstancesMap
protected final Lock	liveInstancesMapLock
protected final SafeHelixManager	manager	Manager used to communicate with Helix.
protected ResourceAssignment	resourceAssignment

Constructor Summary

Constructors

Constructor	Description
HelixBaseRoutingRepository(SafeHelixManager manager)

Method Summary

Modifier and Type	Method	Description
void	clear()
boolean	containsKafkaTopic(String kafkaTopic)	Whether this repository contains routing data for given kafka topic or not.
protected Map<String,Instance>	convertLiveInstances(Collection<org.apache.helix.model.LiveInstance> helixLiveInstances)
boolean	doesResourcesExistInIdealState(String resource)	Whether the resources names exist in ideal state or not.
Map<ExecutionStatus,List<Instance>>	getAllInstances(String kafkaTopic, int partitionId)	This function is mainly used in VeniceVersionFinder#anyOfflinePartitions() when there is no online replica for a specific partition, and it calls this function to get the partition assignment info for error msg.
Instance	getLeaderController()	Query the leader controller of current cluster.
long	getLeaderControllerChangeTimeMs()	Timestamp in milliseconds of the last time leader controller changed.
Instance	getLeaderInstance(String resourceName, int partition)	Used by tests only.
int	getNumberOfPartitions(String resourceName)	Query number of partition in given kafka topic.
PartitionAssignment	getPartitionAssignments(String resourceName)	Query all partitions allocations that belong to given kafka topic.
List<Instance>	getReadyToServeInstances(PartitionAssignment partitionAssignment, int partitionId)	Get ready to serve instances from local memory.
List<Instance>	getReadyToServeInstances(String kafkaTopic, int partitionId)	Get instances from local memory.
ResourceAssignment	getResourceAssignment()
List<Instance>	getWorkingInstances(String kafkaTopic, int partitionId)	Get instances from local memory.
boolean	isLiveInstance(String instanceId)
void	onControllerChange(org.apache.helix.NotificationContext changeContext)
protected abstract void	onCustomizedViewDataChange(org.apache.helix.spectator.RoutingTableSnapshot routingTableSnapshot)
protected abstract void	onExternalViewDataChange(org.apache.helix.spectator.RoutingTableSnapshot routingTableSnapshot)
void	onRoutingTableChange(org.apache.helix.spectator.RoutingTableSnapshot routingTableSnapshot, Object context)
void	refresh()	This method is used to add listener after HelixManager being connected.
void	subscribeRoutingDataChange(String kafkaTopic, RoutingDataRepository.RoutingDataChangedListener listener)	Add a listener on kafka topic to get the notification when routing data is changed.
void	unSubscribeRoutingDataChange(String kafkaTopic, RoutingDataRepository.RoutingDataChangedListener listener)	Remove the listener for given kafka topic.

Methods inherited from class java.lang.Object

clone, equals, finalize, getClass, hashCode, notify, notifyAll, toString, wait, wait, wait

Field Details

manager

protected final SafeHelixManager manager

Manager used to communicate with Helix.

keyBuilder

protected final org.apache.helix.PropertyKey.Builder keyBuilder

Builder used to build the data path to access Helix internal data.

resourceAssignment

protected ResourceAssignment resourceAssignment

listenerManager

protected final ListenerManager<RoutingDataRepository.RoutingDataChangedListener> listenerManager

liveInstancesMapLock

protected final Lock liveInstancesMapLock

liveInstancesMap

protected Map<String,Instance> liveInstancesMap

dataSource

protected final Map<org.apache.helix.PropertyType,List<String>> dataSource

Constructor Details

HelixBaseRoutingRepository

public HelixBaseRoutingRepository(SafeHelixManager manager)

Method Details

refresh

public void refresh()

This method is used to add listener after HelixManager being connected. Otherwise, it will met error because adding listener before connecting.

Specified by:

refresh in interface VeniceResource

clear

public void clear()

Specified by:

clear in interface VeniceResource

getWorkingInstances

public List<Instance> getWorkingInstances(String kafkaTopic,
 int partitionId)

Get instances from local memory. All instances are in HelixState.LEADER or HelixState.STANDBY state.

Specified by:

getWorkingInstances in interface OnlineInstanceFinder

getReadyToServeInstances

public List<Instance> getReadyToServeInstances(String kafkaTopic,
 int partitionId)

Get instances from local memory. All instances are in ExecutionStatus.COMPLETED state.

Specified by:

getReadyToServeInstances in interface OnlineInstanceFinder

getReadyToServeInstances

public List<Instance> getReadyToServeInstances(PartitionAssignment partitionAssignment,
 int partitionId)

Get ready to serve instances from local memory. All instances are in ExecutionStatus.COMPLETED state.

Specified by:

getReadyToServeInstances in interface OnlineInstanceFinder

getAllInstances

public Map<ExecutionStatus,List<Instance>> getAllInstances(String kafkaTopic,
 int partitionId)

This function is mainly used in VeniceVersionFinder#anyOfflinePartitions() when there is no online replica for a specific partition, and it calls this function to get the partition assignment info for error msg. It's valid case that there is no partition assignment for a specific partition and we return EMPTY_MAP.

Specified by:

getAllInstances in interface OnlineInstanceFinder

Returns:

a map that has HelixState as the key and list of instances as the value

getPartitionAssignments

public PartitionAssignment getPartitionAssignments(@Nonnull
 String resourceName)

Description copied from interface: RoutingDataRepository

Query all partitions allocations that belong to given kafka topic. The instances in returned allocations are ready to serve OR being bootstrap.

Specified by:

getPartitionAssignments in interface RoutingDataRepository

Parameters:

resourceName - Name of the resource.

Returns:

PartitionAssignment of the resource from local memory.

getNumberOfPartitions

public int getNumberOfPartitions(@Nonnull
 String resourceName)

Description copied from interface: OnlineInstanceFinder

Query number of partition in given kafka topic.

Specified by:

getNumberOfPartitions in interface OnlineInstanceFinder

Parameters:

resourceName - Name of the resource.

Returns:

The number of partition of the resource from local memory cache.

containsKafkaTopic

public boolean containsKafkaTopic(String kafkaTopic)

Description copied from interface: RoutingDataRepository

Whether this repository contains routing data for given kafka topic or not.

Specified by:

containsKafkaTopic in interface RoutingDataRepository

getLeaderController

public Instance getLeaderController()

Description copied from interface: RoutingDataRepository

Query the leader controller of current cluster.

Specified by:

getLeaderController in interface RoutingDataRepository

subscribeRoutingDataChange

public void subscribeRoutingDataChange(String kafkaTopic,
 RoutingDataRepository.RoutingDataChangedListener listener)

Description copied from interface: RoutingDataRepository

Add a listener on kafka topic to get the notification when routing data is changed.

Specified by:

subscribeRoutingDataChange in interface RoutingDataRepository

unSubscribeRoutingDataChange

public void unSubscribeRoutingDataChange(String kafkaTopic,
 RoutingDataRepository.RoutingDataChangedListener listener)

Description copied from interface: RoutingDataRepository

Remove the listener for given kafka topic.

Specified by:

unSubscribeRoutingDataChange in interface RoutingDataRepository

isLiveInstance

public boolean isLiveInstance(String instanceId)

Specified by:

isLiveInstance in interface RoutingDataRepository

getLeaderControllerChangeTimeMs

public long getLeaderControllerChangeTimeMs()

Description copied from interface: RoutingDataRepository

Timestamp in milliseconds of the last time leader controller changed.

Specified by:

getLeaderControllerChangeTimeMs in interface RoutingDataRepository

onControllerChange

public void onControllerChange(org.apache.helix.NotificationContext changeContext)

Specified by:

onControllerChange in interface org.apache.helix.api.listeners.ControllerChangeListener

getResourceAssignment

public ResourceAssignment getResourceAssignment()

Specified by:

getResourceAssignment in interface RoutingDataRepository

doesResourcesExistInIdealState

public boolean doesResourcesExistInIdealState(String resource)

Description copied from interface: RoutingDataRepository

Whether the resources names exist in ideal state or not.

Specified by:

doesResourcesExistInIdealState in interface RoutingDataRepository

convertLiveInstances

protected Map<String,Instance> convertLiveInstances(Collection<org.apache.helix.model.LiveInstance> helixLiveInstances)

onRoutingTableChange

public void onRoutingTableChange(org.apache.helix.spectator.RoutingTableSnapshot routingTableSnapshot,
 Object context)

Specified by:

onRoutingTableChange in interface org.apache.helix.api.listeners.RoutingTableChangeListener

onExternalViewDataChange

protected abstract void onExternalViewDataChange(org.apache.helix.spectator.RoutingTableSnapshot routingTableSnapshot)

onCustomizedViewDataChange

protected abstract void onCustomizedViewDataChange(org.apache.helix.spectator.RoutingTableSnapshot routingTableSnapshot)

getLeaderInstance

public Instance getLeaderInstance(String resourceName,
 int partition)

Used by tests only. Evaluate carefully if there is an intent to start using this in the main code.

Specified by:

getLeaderInstance in interface RoutingDataRepository

Returns:

the leader Instance or null if there isn't one