HadoopUtils (venice API)

java.lang.Object

com.linkedin.venice.hadoop.utils.HadoopUtils

public class HadoopUtils extends Object

Hadoop-specific utils.

Method Summary

Modifier and Type

Method

Description

static void

cleanUpHDFSPath(String path, boolean recursive)

Silently clean up the given path on HDFS.

static void

createDirectoryWithPermission(org.apache.hadoop.fs.Path path, org.apache.hadoop.fs.permission.FsPermission permission)

Create a temporary directory with the given name under the given path and set the specified permissions.

static Properties

getProps(org.apache.hadoop.mapred.JobConf conf)

static VeniceProperties

getVeniceProps(org.apache.hadoop.mapred.JobConf conf)

static boolean

shouldPathBeIgnored(org.apache.hadoop.fs.Path path)

Check if the path should be ignored.

Methods inherited from class java.lang.Object
clone, equals, finalize, getClass, hashCode, notify, notifyAll, toString, wait, wait, wait

Method Details
- getVeniceProps
  
  public static VeniceProperties getVeniceProps(org.apache.hadoop.mapred.JobConf conf)
- getProps
  
  public static Properties getProps(org.apache.hadoop.mapred.JobConf conf)
- shouldPathBeIgnored
  
  public static boolean shouldPathBeIgnored(org.apache.hadoop.fs.Path path) throws IOException
  
  Check if the path should be ignored. Currently only paths with "_log" are ignored.
  
  Throws:
  
  IOException
- cleanUpHDFSPath
  
  public static void cleanUpHDFSPath(String path, boolean recursive)
  
  Silently clean up the given path on HDFS. If fails, it will ignore the failure and log a message.
  
  Parameters:
  
  path -
  
  recursive - - see FileSystem.delete(Path, boolean)
- createDirectoryWithPermission
  
  public static void createDirectoryWithPermission(org.apache.hadoop.fs.Path path, org.apache.hadoop.fs.permission.FsPermission permission) throws IOException
  
  Create a temporary directory with the given name under the given path and set the specified permissions. If the directory already exists, and the permissions are different from ones specified, the permissions will be updated. If the directory already exists and the permissions are the same, nothing will be done.
  
  Throws:
  
  IOException