Package com.linkedin.venice.hadoop.utils

Class HadoopUtils

java.lang.Object

com.linkedin.venice.hadoop.utils.HadoopUtils

public class HadoopUtils
extends Object

Hadoop-specific utils.

Method Summary

Modifier and Type	Method	Description
static void	cleanUpHDFSPath(String path, boolean recursive)	Silently clean up the given path on HDFS.
static void	createDirectoryWithPermission(org.apache.hadoop.fs.Path path, org.apache.hadoop.fs.permission.FsPermission permission)	Create a temporary directory with the given name under the given path and set the specified permissions.
static Properties	getProps(org.apache.hadoop.mapred.JobConf conf)
static VeniceProperties	getVeniceProps(org.apache.hadoop.mapred.JobConf conf)
static boolean	isSequenceFile(org.apache.hadoop.fs.FileSystem fs, org.apache.hadoop.fs.Path path)	This function is a helper function to validate if a given file is a SequenceFile or not.
static boolean	shouldPathBeIgnored(org.apache.hadoop.fs.Path path)	Check if the path should be ignored.

Methods inherited from class java.lang.Object

clone, equals, finalize, getClass, hashCode, notify, notifyAll, toString, wait, wait, wait

Method Details

getVeniceProps

public static VeniceProperties getVeniceProps(org.apache.hadoop.mapred.JobConf conf)

getProps

public static Properties getProps(org.apache.hadoop.mapred.JobConf conf)

shouldPathBeIgnored

public static boolean shouldPathBeIgnored(org.apache.hadoop.fs.Path path)
                                   throws IOException

Check if the path should be ignored. Currently only paths with "_log" are ignored.

Throws:

IOException

cleanUpHDFSPath

public static void cleanUpHDFSPath(String path,
 boolean recursive)

Silently clean up the given path on HDFS. If fails, it will ignore the failure and log a message.

Parameters:

path -

recursive - - see FileSystem.delete(Path, boolean)

createDirectoryWithPermission

public static void createDirectoryWithPermission(org.apache.hadoop.fs.Path path,
 org.apache.hadoop.fs.permission.FsPermission permission)
                                          throws IOException

Create a temporary directory with the given name under the given path and set the specified permissions. If the directory already exists, and the permissions are different from ones specified, the permissions will be updated. If the directory already exists and the permissions are the same, nothing will be done.

Throws:

IOException

isSequenceFile

public static boolean isSequenceFile(org.apache.hadoop.fs.FileSystem fs,
 org.apache.hadoop.fs.Path path)
                              throws IOException

This function is a helper function to validate if a given file is a SequenceFile or not. There is no utility function that can help fetch this info. This code has been mostly copied from the Apache Hadoop project: https://github.com/apache/hadoop/blob/trunk/hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/io/SequenceFile.java#L2022

Returns:

true if the is a SequenceFile; false otherwise

Throws:

IOException