Package com.linkedin.venice.hadoop.utils

Class HadoopUtils

  • public class HadoopUtils
extends java.lang.Object
    Hadoop-specific utils.

    • Method Summary

      All Methods Static Methods Concrete Methods 
      Modifier and Type Method Description
      static void cleanUpHDFSPath​(java.lang.String path, boolean recursive)
      Silently clean up the given path on HDFS.
      static void createDirectoryWithPermission​(org.apache.hadoop.fs.Path path, org.apache.hadoop.fs.permission.FsPermission permission)
      Create a temporary directory with the given name under the given path and set the specified permissions.
      static java.util.Properties getProps​(org.apache.hadoop.mapred.JobConf conf)  
      static VeniceProperties getVeniceProps​(org.apache.hadoop.mapred.JobConf conf)  
      static boolean shouldPathBeIgnored​(org.apache.hadoop.fs.Path path)
      Check if the path should be ignored.

      • Methods inherited from class java.lang.Object

        clone, equals, finalize, getClass, hashCode, notify, notifyAll, toString, wait, wait, wait

    • Method Detail

      • getVeniceProps

        public static VeniceProperties getVeniceProps​(org.apache.hadoop.mapred.JobConf conf)

      • getProps

        public static java.util.Properties getProps​(org.apache.hadoop.mapred.JobConf conf)

      • shouldPathBeIgnored

        public static boolean shouldPathBeIgnored​(org.apache.hadoop.fs.Path path)
                                   throws java.io.IOException
        Check if the path should be ignored. Currently only paths with "_log" are ignored.
        Throws:
        java.io.IOException

      • cleanUpHDFSPath

        public static void cleanUpHDFSPath​(java.lang.String path,
                                   boolean recursive)
        Silently clean up the given path on HDFS. If fails, it will ignore the failure and log a message.
        Parameters:
        path -
        recursive - - see FileSystem.delete(Path, boolean)

      • createDirectoryWithPermission

        public static void createDirectoryWithPermission​(org.apache.hadoop.fs.Path path,
                                                 org.apache.hadoop.fs.permission.FsPermission permission)
                                          throws java.io.IOException
        Create a temporary directory with the given name under the given path and set the specified permissions. If the directory already exists, and the permissions are different from ones specified, the permissions will be updated. If the directory already exists and the permissions are the same, nothing will be done.
        Throws:
        java.io.IOException