Package com.linkedin.venice.hadoop.schema

Class HDFSSchemaSource

java.lang.Object

com.linkedin.venice.hadoop.schema.HDFSSchemaSource

All Implemented Interfaces:

SchemaSource, java.lang.AutoCloseable

public class HDFSSchemaSource
extends java.lang.Object
implements SchemaSource, java.lang.AutoCloseable

HDFSSchemaSource caches the Value schemes and RMD schemas for a given store on the HDFS and serve them during the Mapper stage. It will make API calls on Venice controller to retrieve all RMD schemes and all value schemas.

Constructor Summary

Constructors

Constructor	Description
HDFSSchemaSource(java.lang.String valueSchemaDir, java.lang.String rmdSchemaDir)
HDFSSchemaSource(java.lang.String valueSchemaDir, java.lang.String rmdSchemaDir, java.lang.String storeName)
HDFSSchemaSource(org.apache.hadoop.fs.Path valueSchemaDir, org.apache.hadoop.fs.Path rmdSchemaDir, java.lang.String storeName)
HDFSSchemaSource(org.apache.hadoop.fs.Path valueSchemaDir, org.apache.hadoop.fs.Path rmdSchemaDir, org.apache.hadoop.fs.Path keySchemaDir, java.lang.String storeName)

Method Summary

Modifier and Type	Method	Description
void	close()
org.apache.avro.Schema	fetchKeySchema()
java.util.Map<RmdVersionId,org.apache.avro.Schema>	fetchRmdSchemas()	Fetch all rmd schemas under the rmdSchemaDir if available.
java.util.Map<java.lang.Integer,org.apache.avro.Schema>	fetchValueSchemas()	Fetch all value schemas under the rmdSchemaDir if available.
java.lang.String	getKeySchemaPath()
java.lang.String	getRmdSchemaPath()
java.lang.String	getValueSchemaPath()
void	saveSchemasOnDisk(ControllerClient controllerClient)	This method loads related configs, gets store value and RMD schemas from Venice controllers, and then write them on HDFS.

Methods inherited from class java.lang.Object

clone, equals, finalize, getClass, hashCode, notify, notifyAll, toString, wait, wait, wait

Constructor Detail

HDFSSchemaSource

public HDFSSchemaSource(org.apache.hadoop.fs.Path valueSchemaDir,
                        org.apache.hadoop.fs.Path rmdSchemaDir,
                        org.apache.hadoop.fs.Path keySchemaDir,
                        java.lang.String storeName)
                 throws java.io.IOException

Throws:

java.io.IOException

HDFSSchemaSource

public HDFSSchemaSource(org.apache.hadoop.fs.Path valueSchemaDir,
                        org.apache.hadoop.fs.Path rmdSchemaDir,
                        java.lang.String storeName)
                 throws java.io.IOException

Throws:

java.io.IOException

HDFSSchemaSource

public HDFSSchemaSource(java.lang.String valueSchemaDir,
                        java.lang.String rmdSchemaDir,
                        java.lang.String storeName)
                 throws java.io.IOException

Throws:

java.io.IOException

HDFSSchemaSource

public HDFSSchemaSource(java.lang.String valueSchemaDir,
                        java.lang.String rmdSchemaDir)
                 throws java.io.IOException

Throws:

java.io.IOException

Method Detail

getKeySchemaPath

public java.lang.String getKeySchemaPath()

getRmdSchemaPath

public java.lang.String getRmdSchemaPath()

getValueSchemaPath

public java.lang.String getValueSchemaPath()

saveSchemasOnDisk

public void saveSchemasOnDisk(ControllerClient controllerClient)
                       throws java.io.IOException,
                              java.lang.IllegalStateException

This method loads related configs, gets store value and RMD schemas from Venice controllers, and then write them on HDFS.

Throws:

java.io.IOException

java.lang.IllegalStateException

fetchRmdSchemas

public java.util.Map<RmdVersionId,org.apache.avro.Schema> fetchRmdSchemas()
                                                                         throws java.io.IOException

Fetch all rmd schemas under the rmdSchemaDir if available.

Specified by:

fetchRmdSchemas in interface SchemaSource

Returns:

RMD schemas mapping

Throws:

java.io.IOException

fetchValueSchemas

public java.util.Map<java.lang.Integer,org.apache.avro.Schema> fetchValueSchemas()
                                                                                throws java.io.IOException

Fetch all value schemas under the rmdSchemaDir if available.

Specified by:

fetchValueSchemas in interface SchemaSource

Returns:

RMD schemas mapping

Throws:

java.io.IOException

fetchKeySchema

public org.apache.avro.Schema fetchKeySchema()
                                      throws java.io.IOException

Specified by:

fetchKeySchema in interface SchemaSource

Throws:

java.io.IOException

close

public void close()

Specified by:

close in interface java.lang.AutoCloseable