Package com.linkedin.venice.hadoop

Class DefaultInputDataInfoProvider

java.lang.Object

com.linkedin.venice.hadoop.DefaultInputDataInfoProvider

All Implemented Interfaces:

InputDataInfoProvider, Closeable, AutoCloseable

public class DefaultInputDataInfoProvider
extends Object
implements InputDataInfoProvider

Nested Class Summary

Nested classes/interfaces inherited from interface com.linkedin.venice.hadoop.InputDataInfoProvider

InputDataInfoProvider.InputDataInfo

Field Summary

Fields

Modifier and Type	Field	Description
protected PushJobZstdConfig	pushJobZstdConfig

Constructor Summary

Constructors

Constructor	Description
DefaultInputDataInfoProvider(PushJobSetting pushJobSetting, VeniceProperties props)

Method Summary

Modifier and Type	Method	Description
void	close()
org.apache.avro.Schema	extractAvroSubSchema(org.apache.avro.Schema origin, String fieldName)
protected Pair<org.apache.avro.Schema,org.apache.avro.Schema>	getAvroFileHeader(org.apache.hadoop.fs.FileSystem fs, org.apache.hadoop.fs.Path path, boolean isZstdDictCreationRequired)
long	getInputLastModificationTime(String inputUri)
protected Pair<VsonSchema,VsonSchema>	getVsonFileHeader(org.apache.hadoop.fs.FileSystem fs, org.apache.hadoop.fs.Path path, boolean isZstdDictCreationRequired)
void	initZstdConfig(int numFiles)
byte[]	trainZstdDictionary()
InputDataInfoProvider.InputDataInfo	validateInputAndGetInfo(String inputUri)	1.

Methods inherited from class java.lang.Object

clone, equals, finalize, getClass, hashCode, notify, notifyAll, toString, wait, wait, wait

Field Details

pushJobZstdConfig

protected PushJobZstdConfig pushJobZstdConfig

Constructor Details

DefaultInputDataInfoProvider

public DefaultInputDataInfoProvider(PushJobSetting pushJobSetting,
 VeniceProperties props)

Method Details

validateInputAndGetInfo

public InputDataInfoProvider.InputDataInfo validateInputAndGetInfo(String inputUri)
                                                            throws Exception

1. Check whether it's Vson input or Avro input 2. Check schema consistency; 3. Populate key schema, value schema; 4. Load samples for dictionary compression if enabled

Specified by:

validateInputAndGetInfo in interface InputDataInfoProvider

Parameters:

inputUri -

Returns:

a InputDataInfoProvider.InputDataInfo that contains input data information

Throws:

Exception

initZstdConfig

public void initZstdConfig(int numFiles)

Specified by:

initZstdConfig in interface InputDataInfoProvider

getVsonFileHeader

protected Pair<VsonSchema,VsonSchema> getVsonFileHeader(org.apache.hadoop.fs.FileSystem fs,
 org.apache.hadoop.fs.Path path,
 boolean isZstdDictCreationRequired)

trainZstdDictionary

public byte[] trainZstdDictionary()

Specified by:

trainZstdDictionary in interface InputDataInfoProvider

extractAvroSubSchema

public org.apache.avro.Schema extractAvroSubSchema(org.apache.avro.Schema origin,
 String fieldName)

Specified by:

extractAvroSubSchema in interface InputDataInfoProvider

getInputLastModificationTime

public long getInputLastModificationTime(String inputUri)
                                  throws IOException

Specified by:

getInputLastModificationTime in interface InputDataInfoProvider

Throws:

IOException

getAvroFileHeader

protected Pair<org.apache.avro.Schema,org.apache.avro.Schema> getAvroFileHeader(org.apache.hadoop.fs.FileSystem fs,
 org.apache.hadoop.fs.Path path,
 boolean isZstdDictCreationRequired)

close

public void close()

Specified by:

close in interface AutoCloseable

Specified by:

close in interface Closeable