InputDataInfoProvider (venice API)

All Superinterfaces:: AutoCloseable, Closeable

All Known Implementing Classes:: DefaultInputDataInfoProvider, KafkaInputDataInfoProvider

public interface InputDataInfoProvider extends Closeable

This interface lets users get input data information

Nested Class Summary

Nested Classes

Modifier and Type

Interface

Description

static class

InputDataInfoProvider.InputDataInfo

A POJO that contains input data information (schema information and input data file size)
Method Summary

Modifier and Type

Method

Description

org.apache.avro.Schema

extractAvroSubSchema(org.apache.avro.Schema origin, String fieldName)

long

getInputLastModificationTime(String inputUri)

PushJobZstdConfig

initZstdConfig(int numFiles)

static void

loadZstdTrainingSamples(VeniceRecordIterator recordIterator, PushJobZstdConfig pushJobZstdConfig)

This function loads training samples from recordReader abstraction for building the Zstd dictionary.

byte[]

trainZstdDictionary()

InputDataInfoProvider.InputDataInfo

validateInputAndGetInfo(String inputUri)

Methods inherited from interface java.io.Closeable
close

Method Details
- validateInputAndGetInfo
  
  InputDataInfoProvider.InputDataInfo validateInputAndGetInfo(String inputUri) throws Exception
  
  Throws:
  
  Exception
- initZstdConfig
  
  PushJobZstdConfig initZstdConfig(int numFiles)
- loadZstdTrainingSamples
  
  static void loadZstdTrainingSamples(VeniceRecordIterator recordIterator, PushJobZstdConfig pushJobZstdConfig)
  
  This function loads training samples from recordReader abstraction for building the Zstd dictionary.
  
  Parameters:
  
  recordIterator - The data accessor of input records.
- trainZstdDictionary
  
  byte[] trainZstdDictionary()
- extractAvroSubSchema
  
  org.apache.avro.Schema extractAvroSubSchema(org.apache.avro.Schema origin, String fieldName)
- getInputLastModificationTime
  
  long getInputLastModificationTime(String inputUri) throws IOException
  
  Throws:
  
  IOException