Package com.linkedin.venice.hadoop.utils

Class HadoopUtils

  • public class HadoopUtils
extends java.lang.Object
    Hadoop-specific utils.

    • Method Summary

      All Methods Static Methods Concrete Methods 
      Modifier and Type Method Description
      static void cleanUpHDFSPath​(java.lang.String path, boolean recursive)
      Silently clean up the given path on HDFS.
      static java.util.Properties getProps​(org.apache.hadoop.mapred.JobConf conf)  
      static VeniceProperties getVeniceProps​(org.apache.hadoop.mapred.JobConf conf)  
      static boolean shouldPathBeIgnored​(org.apache.hadoop.fs.Path path)
      Check if the path should be ignored.

      • Methods inherited from class java.lang.Object

        clone, equals, finalize, getClass, hashCode, notify, notifyAll, toString, wait, wait, wait

    • Method Detail

      • getVeniceProps

        public static VeniceProperties getVeniceProps​(org.apache.hadoop.mapred.JobConf conf)

      • getProps

        public static java.util.Properties getProps​(org.apache.hadoop.mapred.JobConf conf)

      • shouldPathBeIgnored

        public static boolean shouldPathBeIgnored​(org.apache.hadoop.fs.Path path)
                                   throws java.io.IOException
        Check if the path should be ignored. Currently only paths with "_log" are ignored.
        Throws:
        java.io.IOException

      • cleanUpHDFSPath

        public static void cleanUpHDFSPath​(java.lang.String path,
                                   boolean recursive)
        Silently clean up the given path on HDFS. If fails, it will ignore the failure and log a message.
        Parameters:
        path -
        recursive - - see FileSystem.delete(Path, boolean)