Package com.linkedin.venice.utils

Class Utils

  • public class Utils
extends java.lang.Object
    Helper functions

    • Field Detail

      • WILDCARD_MATCH_ANY

        public static final java.lang.String WILDCARD_MATCH_ANY
        See Also:
        Constant Field Values

      • NEW_LINE_CHAR

        public static final java.lang.String NEW_LINE_CHAR

      • SUPPRESS_SYSTEM_EXIT

        public static final java.util.concurrent.atomic.AtomicBoolean SUPPRESS_SYSTEM_EXIT

      • SEPARATE_TOPIC_SUFFIX

        public static final java.lang.String SEPARATE_TOPIC_SUFFIX
        See Also:
        Constant Field Values

      • FATAL_DATA_VALIDATION_ERROR

        public static final java.lang.String FATAL_DATA_VALIDATION_ERROR
        See Also:
        Constant Field Values

    • Constructor Detail

      • Utils

        public Utils()

    • Method Detail

      • exit

        public static void exit​(java.lang.String message)
        Print an error and exit with error code 1
        Parameters:
        message - The error to print

      • exit

        public static void exit​(java.lang.String message,
                        int exitCode)
        Print an error and exit with the given error code
        Parameters:
        message - The error to print
        exitCode - The error code to exit with

      • computeIfNotNull

        public static <T> void computeIfNotNull​(T t,
                                        java.util.function.Consumer<T> function)
        Run "function" on "t" if "t" is not null

      • parseProperties

        public static VeniceProperties parseProperties​(java.lang.String configFileName)
                                        throws java.io.IOException
        Given a filePath, reads into a Venice Props object
        Parameters:
        configFileName - - String path to a properties file
        Returns:
        A @Props object with the given configurations
        Throws:
        java.io.IOException

      • parseProperties

        public static VeniceProperties parseProperties​(java.lang.String directory,
                                               java.lang.String fileName,
                                               boolean isFileOptional)
                                        throws java.io.IOException
        Generate VeniceProperties object from a given directory, file.
        Parameters:
        directory - directory that contains the Property file
        fileName - fileName of the Property file
        isFileOptional - set this to true if the file is optional. If file is missing and set to true, empty property will be returned. If file is missing and set to false, this will throw an exception.
        Throws:
        java.io.IOException

      • parseProperties

        public static VeniceProperties parseProperties​(java.io.File propertyFile)
                                        throws java.io.IOException
        Given a .property file, reads into a Venice Props object
        Parameters:
        propertyFile - The .property file
        Returns:
        A @Props object with the given properties
        Throws:
        java.io.IOException - if File not found or not accessible

      • isReadableDir

        public static boolean isReadableDir​(java.io.File d)
        Check if a directory exists and is readable
        Parameters:
        d - The directory
        Returns:
        true iff the argument is a readable directory

      • isReadableDir

        public static boolean isReadableDir​(java.lang.String dirName)
        Check if a directory exists and is readable
        Parameters:
        dirName - The directory name
        Returns:
        true iff the argument is the name of a readable directory

      • isReadableFile

        public static boolean isReadableFile​(java.lang.String fileName)
        Check if a file exists and is readable
        Returns:
        true iff the argument is the name of a readable file

      • isReadableFile

        public static boolean isReadableFile​(java.io.File f)
        Check if a file exists and is readable
        Parameters:
        f - The file
        Returns:
        true iff the argument is a readable file

      • getCanonicalPath

        public static java.lang.String getCanonicalPath​(java.lang.String fileName)
        Get the full Path of the file. Useful in logging/error output
        Returns:
        canonicalPath of the file.

      • directoryExists

        public static boolean directoryExists​(java.lang.String dataDirectory)

      • thisIsLocalhost

        public static void thisIsLocalhost()
        The ssl certificate we have for unit tests has the hostname "localhost". Any tests that rely on this certificate require that the hostname of the machine match the hostname of the certificate. This method lets us globally assert that the hostname for the machine should resolve to "localhost". We can call this method at the start of any tests that require hostnames to resolve to "localhost" It's not ideal to put this as state in a Utils class, we can revisit if we come up with a better way to do it

      • getHostName

        public static java.lang.String getHostName()
        Get the node's host name.
        Returns:
        current node's host name.

      • sleep

        public static boolean sleep​(long millis)
        Sleep until number of milliseconds have passed, or the operation is interrupted. This method will swallow the InterruptedException and terminate, if this is used in a loop it may become difficult to cleanly break out of the loop.
        Returns:
        true on success and false if sleep was interrupted

      • parseIntFromString

        public static int parseIntFromString​(java.lang.String value,
                                     java.lang.String fieldName)

      • parseLongFromString

        public static long parseLongFromString​(java.lang.String value,
                                       java.lang.String fieldName)

      • parseBooleanFromString

        public static boolean parseBooleanFromString​(java.lang.String value,
                                             java.lang.String fieldName)
        Since Boolean.parseBoolean(String) does not throw exception and will always return 'false' for any string that are not equal to 'true', We validate the string by our own.

      • parseJsonMapFromString

        public static java.util.Map<java.lang.String,​java.lang.String> parseJsonMapFromString​(java.lang.String value,
                                                                                            java.lang.String fieldName)
        For String-String key-value map config, we expect the command-line interface users to use JSON format to represent it. This method deserialize it to String-String map.

      • getHelixNodeIdentifier

        public static java.lang.String getHelixNodeIdentifier​(java.lang.String hostname,
                                                      int port)

      • parseHostFromHelixNodeIdentifier

        public static java.lang.String parseHostFromHelixNodeIdentifier​(java.lang.String nodeId)

      • parsePortFromHelixNodeIdentifier

        public static int parsePortFromHelixNodeIdentifier​(java.lang.String nodeId)

      • getSchemaFromResource

        public static org.apache.avro.Schema getSchemaFromResource​(java.lang.String resourcePath)
                                                    throws java.io.IOException
        Utility function to get schemas out of embedded resources.
        Parameters:
        resourcePath - The path of the file under the src/main/resources directory
        Returns:
        the Schema instance corresponding to the file at {@param resourcePath}
        Throws:
        java.io.IOException - if the resourcePath does not exist

      • getAllSchemasFromResources

        public static java.util.Map<java.lang.Integer,​org.apache.avro.Schema> getAllSchemasFromResources​(AvroProtocolDefinition protocolDef)

      • verifyTransition

        public static boolean verifyTransition​(ExecutionStatus newStatus,
                                       ExecutionStatus... allowed)
        Verify that is the new status allowed to be used.

      • parseCommaSeparatedStringToSet

        public static java.util.Set<java.lang.String> parseCommaSeparatedStringToSet​(java.lang.String rawString)

      • parseCommaSeparatedStringToList

        public static java.util.List<java.lang.String> parseCommaSeparatedStringToList​(java.lang.String rawString)

      • round

        public static double round​(double value,
                           int precision)
        Parameters:
        value - the double value to be rounded
        precision - the number of decimal places by which to round
        Returns:
        {@param value} rounded by {@param precision} decimal places

      • makeLargeNumberPretty

        public static java.lang.String makeLargeNumberPretty​(long largeNumber)

      • getUniqueString

        public static java.lang.String getUniqueString()

      • getUniqueString

        public static java.lang.String getUniqueString​(java.lang.String prefix)

      • getUniqueTempPath

        public static java.lang.String getUniqueTempPath()

      • getUniqueTempPath

        public static java.lang.String getUniqueTempPath​(java.lang.String prefix)

      • getTempDataDirectory

        public static java.io.File getTempDataDirectory()

      • getTempDataDirectory

        public static java.io.File getTempDataDirectory​(java.lang.String prefix)

      • getRealTimeTopicName

        public static java.lang.String getRealTimeTopicName​(Store store)
        It follows the following order to search for real time topic name, i) current store-version config, ii) store config, iii) other store-version configs, iv) default name

      • getRealTimeTopicName

        public static java.lang.String getRealTimeTopicName​(StoreInfo storeInfo)

      • getRealTimeTopicName

        public static java.lang.String getRealTimeTopicName​(Version version)

      • makeTimePretty

        public static java.lang.String makeTimePretty​(long nanoSecTime)

      • getCurrentWorkingDirectory

        public static java.lang.String getCurrentWorkingDirectory()

      • getPid

        public static java.lang.String getPid()
        Note: this may fail in some JVM implementations. Lifted from: https://stackoverflow.com/a/7690178
        Returns:
        the pid of the current Java process, or null if unavailable

      • getVeniceVersionFromClassPath

        public static java.lang.String getVeniceVersionFromClassPath()
        This might not work when application is running inside application server like Jetty
        Returns:
        the version of the venice-common jar on the classpath, if available, or null otherwise.

      • getCurrentUser

        public static java.lang.String getCurrentUser()

      • getJavaMajorVersion

        public static int getJavaMajorVersion()

      • getDebugInfo

        public static java.util.Map<java.lang.CharSequence,​java.lang.CharSequence> getDebugInfo()

      • getValueFromCharSequenceMapWithStringKey

        public static java.util.Optional<java.lang.CharSequence> getValueFromCharSequenceMapWithStringKey​(java.util.Map<java.lang.CharSequence,​java.lang.CharSequence> charSequenceMap,
                                                                                                  java.lang.String stringKey)
        Given a key with type String and get a value (if there is one) from a map of type Map which has one of key's toString value equals to the given String key. Note that the worst case runtime of this function is O(n) since the given map is iterated to find the value. The reason why this function exists is that the specific implementation of the CharSequence interface might not be String for the map key type. So, when users try to get a value from such a map with a String key, they might not get the value even when the value of the key which has the same toString representation as the user-given String key since the specific implementation of the CharSequence interface has a different hash function than String's hash function.
        Returns:
        a Value in the CharSequence map or {@link null}

      • setOf

        @SafeVarargs
public static <T> java.util.Set<T> setOf​(T... objs)

      • mutableSetOf

        @SafeVarargs
public static <T> java.util.Set<T> mutableSetOf​(T... objs)

      • closeQuietlyWithErrorLogged

        public static void closeQuietlyWithErrorLogged​(java.io.Closeable... closeables)

      • getReplicasForInstance

        public static java.util.List<Replica> getReplicasForInstance​(RoutingDataRepository routingDataRepo,
                                                             java.lang.String instanceId)

      • isCurrentVersion

        public static boolean isCurrentVersion​(java.lang.String resourceName,
                                       ReadOnlyStoreRepository metadataRepo)

      • isFutureVersion

        public static boolean isFutureVersion​(java.lang.String resourceName,
                                      ReadOnlyStoreRepository metadataRepo)

      • isExtraReplica

        public static boolean isExtraReplica​(ReadOnlyStoreRepository metadataRepo,
                                     Replica replica,
                                     java.util.List<Instance> readyInstances)
        When Helix thinks some host is overloaded or a new host joins the cluster, it might move some replicas from one host to another. The partition to be moved is usually in a healthy state, i.e. 3/3 running replicas. We will now build an extra replica in the new host before dropping one replica in an old host. i.e. replica changes 3 -> 4 -> 3. In this scenario, we may see an extra replica appear. This function determines if a replica is an extra replica.
        Returns:
        true, if the input replica is an extra one. false, otherwise.

      • extractQueryParamsFromRequest

        public static java.util.Map<java.lang.String,​java.lang.String> extractQueryParamsFromRequest​(java.util.Map<java.lang.String,​java.lang.String[]> sparkRequestParams,
                                                                                                   ControllerResponse response)

      • iterateOnMapOfLists

        public static <K,​V> java.util.Iterator<V> iterateOnMapOfLists​(java.util.Map<K,​java.util.List<V>> mapOfLists)

      • getSanitizedStringForLogger

        public static java.lang.String getSanitizedStringForLogger​(java.lang.String orig)
        Log4J's class name logging splits at the last "." and assumes it is the class name. In case where custom strings (e.g. URLs, server addresses, etc.) are added to the logger names, Log4J logs an incomplete string. This function replaces "." with "_" in the string when setting as the input for the logger.
        Parameters:
        orig - The string to sanitize
        Returns:
        A sanitized string that won't get mutated by Log4J

      • getReplicaId

        public static java.lang.String getReplicaId​(PubSubTopicPartition topicPartition)
        Standard logging format for TopicPartition

      • getReplicaId

        public static java.lang.String getReplicaId​(java.lang.String topic,
                                            int partition)

      • getReplicaId

        public static java.lang.String getReplicaId​(PubSubTopic topic,
                                            int partition)

      • escapeFilePathComponent

        public static java.lang.String escapeFilePathComponent​(java.lang.String component)
        Method to escape file path component to make it a valid file path string/substring.
        Parameters:
        component - file path component string
        Returns:
        Escaped file path component string

      • resolveKafkaUrlForSepTopic

        public static java.lang.String resolveKafkaUrlForSepTopic​(java.lang.String kafkaUrl)
        Check whether the given kafka url has "_sep" or not. If it has, return the kafka url without "_sep". Otherwise, return the original kafka url.
        Parameters:
        kafkaUrl -
        Returns:

      • isSeparateTopicRegion

        public static boolean isSeparateTopicRegion​(java.lang.String region)
        Check whether input region is for separate RT topic.

      • resolveLeaderTopicFromPubSubTopic

        public static PubSubTopic resolveLeaderTopicFromPubSubTopic​(PubSubTopicRepository pubSubTopicRepository,
                                                            PubSubTopic pubSubTopic)
        Resolve leader topic from input topic. If input topic is separate RT topic, return the corresponding RT topic. Otherwise, return the original input topic.

      • parseDateTimeToEpoch

        public static long parseDateTimeToEpoch​(java.lang.String dateTime,
                                        java.lang.String dateTimeFormat,
                                        java.lang.String timeZone)
                                 throws java.text.ParseException
        Parses a date-time string to epoch milliseconds using the default format and time zone.
        Parameters:
        dateTime - the date-time string in the format "yyyy-MM-dd hh:mm:ss"
        Returns:
        the epoch time in milliseconds
        Throws:
        java.text.ParseException - if the date-time string cannot be parsed