Package com.linkedin.davinci.config

Class VeniceConfigLoader

  • public class VeniceConfigLoader
extends java.lang.Object
    VeniceConfigService loads the static configuration that does not change once the Server is started. Modifying these configs once the server is up and running, does not get reflected until the server is restarted. There are 3 configurations provided. 1) Cluster Configuration The configuration that needs to be shared between the Controller and Server goes here. 2) Server Configuration Multiple nodes that should share the same value should go in here. Cluster Name, ZooKeeper Address or any other configuration that is common between many storage nodes go here. 3) Override Configuration These value override the values in Base Configuration ( if present). Node Id for each server goes here. There are two types of overrides supported. 1) Precedence among the files Any Key in Server Configuration overrides the same key in Cluster Configuration. Any Key in Override Configuration overrides the same key in Server Configuration. 2) Overriding keys for specific store only. The following config sets default to all stores and override for foobar store. bdb.max.logfile.size=1000 store-foobar.bdb.max.logfile.size=5000 There are two ways of providing the Configs. 1) Config directory. The Config uses 3 files cluster.properties, server.properties and server-override.properties (optional). 2) java.util.properties. It takes in 3 props for the above 3 configuration. The static config initializes the config and starts the default services. The configuration does not specify what stores Storage Node hosts/serves. The Storage Node waits for Commands from Helix to understand what stores and partitions it was assigned to. This design, assumes that without Helix, the storage node/server will not be able to read/write data as it has no knowledge of the stores. Helix Controller may also provide dynamic properties that can override the properties at the store level. This is not implemented currently. There is no validation in the code, to differentiate between Valid and Invalid Overrides for now. Once the code is evolved and our usage patterns are narrowed down, checks can be enforced if necessary.

    • Field Detail

      • VENICE_CONFIG_DIR

        public static final java.lang.String VENICE_CONFIG_DIR
        See Also:
        Constant Field Values

      • CLUSTER_PROPERTIES_FILE

        public static final java.lang.String CLUSTER_PROPERTIES_FILE
        See Also:
        Constant Field Values

      • SERVER_PROPERTIES_FILE

        public static final java.lang.String SERVER_PROPERTIES_FILE
        See Also:
        Constant Field Values

      • KAFKA_CLUSTER_MAP_FILE

        public static final java.lang.String KAFKA_CLUSTER_MAP_FILE
        See Also:
        Constant Field Values

    • Constructor Detail

      • VeniceConfigLoader

        public VeniceConfigLoader​(VeniceProperties properties)

      • VeniceConfigLoader

        public VeniceConfigLoader​(VeniceProperties clusterProperties,
                          VeniceProperties serverProperties,
                          java.util.Map<java.lang.String,​java.util.Map<java.lang.String,​java.lang.String>> kafkaClusterMap)

    • Method Detail

      • loadFromEnvironmentVariable

        public static VeniceConfigLoader loadFromEnvironmentVariable()
        Initializes the Venice configuration service from known environment variables
        Returns:
        VeniceConfigService object

      • loadFromConfigDirectory

        public static VeniceConfigLoader loadFromConfigDirectory​(java.lang.String configDirPath)

      • parseKafkaClusterMap

        public static java.util.Map<java.lang.String,​java.util.Map<java.lang.String,​java.lang.String>> parseKafkaClusterMap​(java.lang.String configDirectory)
                                                                                                                         throws java.lang.Exception
        Throws:
        java.lang.Exception

      • parseKafkaClusterMap

        public static java.util.Map<java.lang.String,​java.util.Map<java.lang.String,​java.lang.String>> parseKafkaClusterMap​(java.lang.String configDirectory,
                                                                                                                                java.lang.String fileName)
                                                                                                                         throws java.lang.Exception
        The file contains the following info: The otherUrls field is used to translate URLs coming from other processes (e.g. coming from the TopicSwitch emitted by the controller, or by a previous run of the server which was configured differently). The various URLs included in otherUrls fields must be globally unique. The securityProtocol entry must be compatible with the url entry (i.e. have the right port). N.B.: This is actually JSON, not XML, and it has some weird escaping in it. TODO: Clean this up. =
        Throws:
        java.lang.Exception

      • storeKafkaClusterMap

        public static void storeKafkaClusterMap​(java.io.File configDirectory,
                                        java.util.Map<java.lang.String,​java.util.Map<java.lang.String,​java.lang.String>> kafkaClusterMap)
                                 throws java.lang.Exception
        Throws:
        java.lang.Exception

      • storeKafkaClusterMap

        public static void storeKafkaClusterMap​(java.io.File configDirectory,
                                        java.lang.String fileName,
                                        java.util.Map<java.lang.String,​java.util.Map<java.lang.String,​java.lang.String>> kafkaClusterMap)
                                 throws java.lang.Exception
        Throws:
        java.lang.Exception