Package com.linkedin.venice.hadoop

Class DefaultInputDataInfoProvider

java.lang.Object

com.linkedin.venice.hadoop.DefaultInputDataInfoProvider

All Implemented Interfaces:

InputDataInfoProvider, Closeable, AutoCloseable

public class DefaultInputDataInfoProvider
extends Object
implements InputDataInfoProvider

Nested Class Summary

Nested classes/interfaces inherited from interface com.linkedin.venice.hadoop.InputDataInfoProvider

InputDataInfoProvider.InputDataInfo

Constructor Summary

Constructors

Constructor	Description
DefaultInputDataInfoProvider(PushJobSetting pushJobSetting, VeniceProperties props)

Method Summary

Modifier and Type	Method	Description
void	close()
org.apache.avro.Schema	extractAvroSubSchema(org.apache.avro.Schema origin, String fieldName)
long	getInputLastModificationTime(String inputUri)
PushJobZstdConfig	initZstdConfig(int numFiles)
byte[]	trainZstdDictionary()
InputDataInfoProvider.InputDataInfo	validateInputAndGetInfo(String inputUri)	1.

Methods inherited from class java.lang.Object

clone, equals, finalize, getClass, hashCode, notify, notifyAll, toString, wait, wait, wait

Constructor Details

DefaultInputDataInfoProvider

public DefaultInputDataInfoProvider(PushJobSetting pushJobSetting,
 VeniceProperties props)

Method Details

validateInputAndGetInfo

public InputDataInfoProvider.InputDataInfo validateInputAndGetInfo(String inputUri)
                                                            throws Exception

1. Check whether it's Vson input or Avro input 2. Check schema consistency; 3. Populate key schema, value schema; 4. Load samples for dictionary compression if enabled

Specified by:

validateInputAndGetInfo in interface InputDataInfoProvider

Parameters:

inputUri -

Returns:

a InputDataInfoProvider.InputDataInfo that contains input data information

Throws:

Exception

initZstdConfig

public PushJobZstdConfig initZstdConfig(int numFiles)

Specified by:

initZstdConfig in interface InputDataInfoProvider

trainZstdDictionary

public byte[] trainZstdDictionary()

Specified by:

trainZstdDictionary in interface InputDataInfoProvider

extractAvroSubSchema

public org.apache.avro.Schema extractAvroSubSchema(org.apache.avro.Schema origin,
 String fieldName)

Specified by:

extractAvroSubSchema in interface InputDataInfoProvider

getInputLastModificationTime

public long getInputLastModificationTime(String inputUri)
                                  throws IOException

Specified by:

getInputLastModificationTime in interface InputDataInfoProvider

Throws:

IOException

close

public void close()

Specified by:

close in interface AutoCloseable

Specified by:

close in interface Closeable