Package com.linkedin.venice.utils

Class Utils

java.lang.Object
com.linkedin.venice.utils.Utils
public class Utils extends Object
Helper functions

  • Field Details

  • Constructor Details

    • Utils

      public Utils()

  • Method Details

    • exit

      public static void exit(String message)
      Print an error and exit with error code 1
      Parameters:
      message - The error to print

    • exit

      public static void exit(String message, int exitCode)
      Print an error and exit with the given error code
      Parameters:
      message - The error to print
      exitCode - The error code to exit with

    • computeIfNotNull

      public static <T> void computeIfNotNull(T t, Consumer<T> function)
      Run "function" on "t" if "t" is not null

    • parseProperties

      public static VeniceProperties parseProperties(String configFileName) throws IOException
      Given a filePath, reads into a Venice Props object
      Parameters:
      configFileName - - String path to a properties file
      Returns:
      A @Props object with the given configurations
      Throws:
      IOException

    • parseProperties

      public static VeniceProperties parseProperties(String directory, String fileName, boolean isFileOptional) throws IOException
      Generate VeniceProperties object from a given directory, file.
      Parameters:
      directory - directory that contains the Property file
      fileName - fileName of the Property file
      isFileOptional - set this to true if the file is optional. If file is missing and set to true, empty property will be returned. If file is missing and set to false, this will throw an exception.
      Throws:
      IOException

    • parseProperties

      public static VeniceProperties parseProperties(File propertyFile) throws IOException
      Given a .property file, reads into a Venice Props object
      Parameters:
      propertyFile - The .property file
      Returns:
      A @Props object with the given properties
      Throws:
      IOException - if File not found or not accessible

    • isReadableDir

      public static boolean isReadableDir(File d)
      Check if a directory exists and is readable
      Parameters:
      d - The directory
      Returns:
      true iff the argument is a readable directory

    • isReadableDir

      public static boolean isReadableDir(String dirName)
      Check if a directory exists and is readable
      Parameters:
      dirName - The directory name
      Returns:
      true iff the argument is the name of a readable directory

    • isReadableFile

      public static boolean isReadableFile(String fileName)
      Check if a file exists and is readable
      Returns:
      true iff the argument is the name of a readable file

    • isReadableFile

      public static boolean isReadableFile(File f)
      Check if a file exists and is readable
      Parameters:
      f - The file
      Returns:
      true iff the argument is a readable file

    • getCanonicalPath

      public static String getCanonicalPath(String fileName)
      Get the full Path of the file. Useful in logging/error output
      Returns:
      canonicalPath of the file.

    • directoryExists

      public static boolean directoryExists(String dataDirectory)

    • thisIsLocalhost

      public static void thisIsLocalhost()
      The ssl certificate we have for unit tests has the hostname "localhost". Any tests that rely on this certificate require that the hostname of the machine match the hostname of the certificate. This method lets us globally assert that the hostname for the machine should resolve to "localhost". We can call this method at the start of any tests that require hostnames to resolve to "localhost" It's not ideal to put this as state in a Utils class, we can revisit if we come up with a better way to do it

    • getHostName

      public static String getHostName()
      Get the node's host name.
      Returns:
      current node's host name.

    • sleep

      public static boolean sleep(long millis)
      Sleep until number of milliseconds have passed, or the operation is interrupted. This method will swallow the InterruptedException and terminate, if this is used in a loop it may become difficult to cleanly break out of the loop.
      Returns:
      true on success and false if sleep was interrupted

    • parseIntFromString

      public static int parseIntFromString(String value, String fieldName)

    • parseIntOrDefault

      public static int parseIntOrDefault(String value, String fieldName, int defaultValue)
      Parses an integer from a string, ensuring that only null or valid integer values are accepted. Returns defaultValue if the value is null. Throws an exception if the value is invalid.
      Parameters:
      value - the string to parse
      fieldName - the name of the field being validated
      defaultValue - the default value to return if the input is null
      Returns:
      the parsed int value

    • parseLongFromString

      public static long parseLongFromString(String value, String fieldName)

    • parseBooleanOrThrow

      public static boolean parseBooleanOrThrow(String value, String fieldName)
      Parses a boolean from a string, ensuring that only valid boolean values ("true" or "false") are accepted. Throws an exception if the value is null or invalid.
      Parameters:
      value - the string to parse
      fieldName - the name of the field being validated
      Returns:
      the parsed boolean value
      Throws:
      VeniceHttpException - if the value is null or not "true" or "false"

    • parseBooleanOrFalse

      public static boolean parseBooleanOrFalse(String value, String fieldName)
      Parses a boolean from a string, ensuring that only null and valid boolean values ("true" or "false") are accepted. Returns false if the value is null.
      Parameters:
      value - the string to parse
      fieldName - the name of the field being validated
      Returns:
      the parsed boolean value, or false if the input is null
      Throws:
      VeniceHttpException - if the value is not "true" or "false"

    • parseJsonMapFromString

      public static Map<String,String> parseJsonMapFromString(String value, String fieldName)
      For String-String key-value map config, we expect the command-line interface users to use JSON format to represent it. This method deserialize it to String-String map.

    • parseStoreLifecycleHooksListFromString

      public static List<LifecycleHooksRecord> parseStoreLifecycleHooksListFromString(String value, String fieldName)
      For Store Lifecycle Hooks value, we expect the command-line interface users to use JSON format to represent it. This method deserialize it to List.

    • getHelixNodeIdentifier

      public static String getHelixNodeIdentifier(String hostname, int port)

    • parseHostFromHelixNodeIdentifier

      public static String parseHostFromHelixNodeIdentifier(String nodeId)

    • parsePortFromHelixNodeIdentifier

      public static int parsePortFromHelixNodeIdentifier(String nodeId)

    • getSchemaFromResource

      public static org.apache.avro.Schema getSchemaFromResource(String resourcePath) throws IOException
      Utility function to get schemas out of embedded resources.
      Parameters:
      resourcePath - The path of the file under the src/main/resources directory
      Returns:
      the Schema instance corresponding to the file at
      Throws:
      IOException - if the resourcePath does not exist

    • getAllSchemasFromResources

      public static Map<Integer,org.apache.avro.Schema> getAllSchemasFromResources(AvroProtocolDefinition protocolDef)

    • verifyTransition

      public static boolean verifyTransition(ExecutionStatus newStatus, ExecutionStatus... allowed)
      Verify that is the new status allowed to be used.

    • parseCommaSeparatedStringToSet

      public static Set<String> parseCommaSeparatedStringToSet(String rawString)

    • parseCommaSeparatedStringToList

      public static List<String> parseCommaSeparatedStringToList(String rawString)

    • round

      public static double round(double value, int precision)
      Parameters:
      value - the double value to be rounded
      precision - the number of decimal places by which to round
      Returns:
      rounded by decimal places

    • makeLargeNumberPretty

      public static String makeLargeNumberPretty(long largeNumber)

    • getUniqueString

      public static String getUniqueString()

    • getUniqueString

      public static String getUniqueString(String prefix)

    • getUniqueTempPath

      public static String getUniqueTempPath()

    • getUniqueTempPath

      public static String getUniqueTempPath(String prefix)

    • getTempDataDirectory

      public static File getTempDataDirectory()

    • getTempDataDirectory

      public static File getTempDataDirectory(String prefix)

    • composeRealTimeTopic

      public static String composeRealTimeTopic(String storeName)
      This method should only be used for system stores. For other stores, use getRealTimeTopicName(Store), getRealTimeTopicName(StoreInfo) or getRealTimeTopicName(Version) in source code. For tests, use composeRealTimeTopic(String, int)

    • composeRealTimeTopic

      public static String composeRealTimeTopic(String storeName, int versionNumber)

    • getRealTimeTopicName

      public static String getRealTimeTopicName(Store store)
      It follows the following order to search for real time topic name, i) current store-version config, ii) store config, iii) other store-version configs, iv) default name

    • getRealTimeTopicName

      public static String getRealTimeTopicName(StoreInfo storeInfo)

    • isRTVersioningApplicable

      public static boolean isRTVersioningApplicable(String storeName)

    • getRealTimeTopicName

      public static String getRealTimeTopicName(Version version)

    • getAllRealTimeTopicNames

      public static Set<String> getAllRealTimeTopicNames(Store store)

    • getRealTimeTopicNameFromSeparateRealTimeTopic

      public static String getRealTimeTopicNameFromSeparateRealTimeTopic(String separateRealTimeTopicName)

    • getSeparateRealTimeTopicName

      public static String getSeparateRealTimeTopicName(String realTimeTopicName)

    • getSeparateRealTimeTopicName

      public static String getSeparateRealTimeTopicName(Version version)

    • getSeparateRealTimeTopicName

      public static String getSeparateRealTimeTopicName(StoreInfo storeInfo)

    • calculateTopicHashCode

      public static int calculateTopicHashCode(PubSubTopic topic)

    • makeTimePretty

      public static String makeTimePretty(long nanoSecTime)

    • getCurrentWorkingDirectory

      public static String getCurrentWorkingDirectory()

    • getPid

      public static String getPid()
      Note: this may fail in some JVM implementations. Lifted from: https://stackoverflow.com/a/7690178
      Returns:
      the pid of the current Java process, or null if unavailable

    • getVeniceVersionFromClassPath

      public static String getVeniceVersionFromClassPath()
      This might not work when application is running inside application server like Jetty
      Returns:
      the version of the venice-common jar on the classpath, if available, or null otherwise.

    • getCurrentUser

      public static String getCurrentUser()

    • getJavaMajorVersion

      public static int getJavaMajorVersion()

    • getDebugInfo

      public static Map<CharSequence,CharSequence> getDebugInfo()

    • getValueFromCharSequenceMapWithStringKey

      public static Optional<CharSequence> getValueFromCharSequenceMapWithStringKey(Map<CharSequence,CharSequence> charSequenceMap, String stringKey)
      Given a key with type String and get a value (if there is one) from a map of type Map<CharSequence,CharSequence> which has one of key's toString value equals to the given String key. Note that the worst case runtime of this function is O(n) since the given map is iterated to find the value. The reason why this function exists is that the specific implementation of the CharSequence interface might not be String for the map key type. So, when users try to get a value from such a map with a String key, they might not get the value even when the value of the key which has the same toString representation as the user-given String key since the specific implementation of the CharSequence interface has a different hash function than String's hash function.
      Returns:
      a Value in the CharSequence map or {@link null}

    • setOf

      @SafeVarargs public static <T> Set<T> setOf(T... objs)

    • mutableSetOf

      @SafeVarargs public static <T> Set<T> mutableSetOf(T... objs)

    • closeQuietlyWithErrorLogged

      public static void closeQuietlyWithErrorLogged(AutoCloseable... closeables)

    • closeQuietly

      public static void closeQuietly(AutoCloseable closeable, Consumer<Exception> consumer)

    • getReplicasForInstance

      public static List<Replica> getReplicasForInstance(RoutingDataRepository routingDataRepo, String instanceId)

    • isCurrentVersion

      public static boolean isCurrentVersion(String resourceName, ReadOnlyStoreRepository metadataRepo)

    • isFutureVersion

      public static boolean isFutureVersion(String resourceName, ReadOnlyStoreRepository metadataRepo)

    • isFutureVersionReady

      public static boolean isFutureVersionReady(String resourceName, ReadOnlyStoreRepository metadataRepo)
      Checks if the future version is ready to serve. A future version is considered ready to serve if the version status is either PUSHED or ONLINE
      Parameters:
      resourceName -
      metadataRepo -
      Returns:

    • isExtraReplica

      public static boolean isExtraReplica(ReadOnlyStoreRepository metadataRepo, Replica replica, List<Instance> readyInstances)
      When Helix thinks some host is overloaded or a new host joins the cluster, it might move some replicas from one host to another. The partition to be moved is usually in a healthy state, i.e. 3/3 running replicas. We will now build an extra replica in the new host before dropping one replica in an old host. i.e. replica changes 3 -> 4 -> 3. In this scenario, we may see an extra replica appear. This function determines if a replica is an extra replica.
      Returns:
      true, if the input replica is an extra one. false, otherwise.

    • extractQueryParamsFromRequest

      public static Map<String,String> extractQueryParamsFromRequest(Map<String,String[]> sparkRequestParams, ControllerResponse response)

    • waitStoreVersionOrThrow

      public static StoreVersionInfo waitStoreVersionOrThrow(String storeVersionName, ReadOnlyStoreRepository metadataRepo)

    • iterateOnMapOfLists

      public static <K, V> Iterator<V> iterateOnMapOfLists(Map<K,List<V>> mapOfLists)

    • getSanitizedStringForLogger

      public static String getSanitizedStringForLogger(String orig)
      Log4J's class name logging splits at the last "." and assumes it is the class name. In case where custom strings (e.g. URLs, server addresses, etc.) are added to the logger names, Log4J logs an incomplete string. This function replaces "." with "_" in the string when setting as the input for the logger.
      Parameters:
      orig - The string to sanitize
      Returns:
      A sanitized string that won't get mutated by Log4J

    • getReplicaId

      public static String getReplicaId(String storeName, int version, int partition)

    • getReplicaId

      public static String getReplicaId(PubSubTopicPartition topicPartition)
      Standard logging format for TopicPartition

    • getReplicaId

      public static String getReplicaId(String topic, int partition)

    • getReplicaId

      public static String getReplicaId(PubSubTopic topic, int partition)

    • escapeFilePathComponent

      public static String escapeFilePathComponent(String component)
      Method to escape file path component to make it a valid file path string/substring.
      Parameters:
      component - file path component string
      Returns:
      Escaped file path component string

    • resolveKafkaUrlForSepTopic

      public static String resolveKafkaUrlForSepTopic(String kafkaUrl)
      Check whether the given kafka url has "_sep" or not. If it has, return the kafka url without "_sep". Otherwise, return the original kafka url.
      Parameters:
      kafkaUrl -
      Returns:

    • isSeparateTopicRegion

      public static boolean isSeparateTopicRegion(String region)
      Check whether input region is for separate RT topic.

    • resolveLeaderTopicFromPubSubTopic

      public static PubSubTopic resolveLeaderTopicFromPubSubTopic(PubSubTopicRepository pubSubTopicRepository, PubSubTopic pubSubTopic)
      Resolve leader topic from input topic. If input topic is separate RT topic, return the corresponding RT topic. Otherwise, return the original input topic.

    • createPubSubTopicPartitionFromLeaderTopicPartition

      public static PubSubTopicPartition createPubSubTopicPartitionFromLeaderTopicPartition(String pubSubAddress, PubSubTopicPartition leaderTopicPartition)

    • parseDateTimeToEpoch

      public static long parseDateTimeToEpoch(String dateTime, String dateTimeFormat, String timeZone) throws ParseException
      Parses a date-time string to epoch milliseconds using the default format and time zone.
      Parameters:
      dateTime - the date-time string in the format "yyyy-MM-dd hh:mm:ss"
      Returns:
      the epoch time in milliseconds
      Throws:
      ParseException - if the date-time string cannot be parsed

    • getOSMemorySize

      public static long getOSMemorySize()