Package com.linkedin.venice.hadoop.schema

Class HDFSSchemaSource

java.lang.Object

com.linkedin.venice.hadoop.schema.HDFSSchemaSource

All Implemented Interfaces:

SchemaSource, AutoCloseable

public class HDFSSchemaSource
extends Object
implements SchemaSource, AutoCloseable

HDFSSchemaSource caches the Value schemes and RMD schemas for a given store on the HDFS and serve them during the Mapper stage. It will make API calls on Venice controller to retrieve all RMD schemes and all value schemas.

Constructor Summary

Constructors

Constructor	Description
HDFSSchemaSource(String valueSchemaDir, String rmdSchemaDir)
HDFSSchemaSource(String valueSchemaDir, String rmdSchemaDir, String storeName)
HDFSSchemaSource(org.apache.hadoop.fs.Path valueSchemaDir, org.apache.hadoop.fs.Path rmdSchemaDir, String storeName)
HDFSSchemaSource(org.apache.hadoop.fs.Path valueSchemaDir, org.apache.hadoop.fs.Path rmdSchemaDir, org.apache.hadoop.fs.Path keySchemaDir, String storeName)

Method Summary

Modifier and Type	Method	Description
void	close()
org.apache.avro.Schema	fetchKeySchema()
Map<RmdVersionId,org.apache.avro.Schema>	fetchRmdSchemas()	Fetch all rmd schemas under the rmdSchemaDir if available.
Map<Integer,org.apache.avro.Schema>	fetchValueSchemas()	Fetch all value schemas under the rmdSchemaDir if available.
String	getKeySchemaPath()
String	getRmdSchemaPath()
String	getValueSchemaPath()
void	saveSchemasOnDisk(ControllerClient controllerClient)	This method loads related configs, gets store value and RMD schemas from Venice controllers, and then write them on HDFS.

Methods inherited from class java.lang.Object

clone, equals, finalize, getClass, hashCode, notify, notifyAll, toString, wait, wait, wait

Constructor Details

HDFSSchemaSource

public HDFSSchemaSource(org.apache.hadoop.fs.Path valueSchemaDir,
 org.apache.hadoop.fs.Path rmdSchemaDir,
 org.apache.hadoop.fs.Path keySchemaDir,
 String storeName)
                 throws IOException

Throws:

IOException

HDFSSchemaSource

public HDFSSchemaSource(org.apache.hadoop.fs.Path valueSchemaDir,
 org.apache.hadoop.fs.Path rmdSchemaDir,
 String storeName)
                 throws IOException

Throws:

IOException

HDFSSchemaSource

public HDFSSchemaSource(String valueSchemaDir,
 String rmdSchemaDir,
 String storeName)
                 throws IOException

Throws:

IOException

HDFSSchemaSource

public HDFSSchemaSource(String valueSchemaDir,
 String rmdSchemaDir)
                 throws IOException

Throws:

IOException

Method Details

getKeySchemaPath

public String getKeySchemaPath()

getRmdSchemaPath

public String getRmdSchemaPath()

getValueSchemaPath

public String getValueSchemaPath()

saveSchemasOnDisk

public void saveSchemasOnDisk(ControllerClient controllerClient)
                       throws IOException,
IllegalStateException

This method loads related configs, gets store value and RMD schemas from Venice controllers, and then write them on HDFS.

Throws:

IOException

IllegalStateException

fetchRmdSchemas

public Map<RmdVersionId,org.apache.avro.Schema> fetchRmdSchemas()
                                                         throws IOException

Fetch all rmd schemas under the rmdSchemaDir if available.

Specified by:

fetchRmdSchemas in interface SchemaSource

Returns:

RMD schemas mapping

Throws:

IOException

fetchValueSchemas

public Map<Integer,org.apache.avro.Schema> fetchValueSchemas()
                                                      throws IOException

Fetch all value schemas under the rmdSchemaDir if available.

Specified by:

fetchValueSchemas in interface SchemaSource

Returns:

RMD schemas mapping

Throws:

IOException

fetchKeySchema

public org.apache.avro.Schema fetchKeySchema()
                                      throws IOException

Specified by:

fetchKeySchema in interface SchemaSource

Throws:

IOException

close

public void close()

Specified by:

close in interface AutoCloseable