Class HDFSSchemaSource
- java.lang.Object
-
- com.linkedin.venice.hadoop.schema.HDFSSchemaSource
-
- All Implemented Interfaces:
SchemaSource
,java.lang.AutoCloseable
public class HDFSSchemaSource extends java.lang.Object implements SchemaSource, java.lang.AutoCloseable
HDFSSchemaSource caches the Value schemes and RMD schemas for a given store on the HDFS and serve them during the Mapper stage. It will make API calls on Venice controller to retrieve all RMD schemes and all value schemas.
-
-
Constructor Summary
Constructors Constructor Description HDFSSchemaSource(java.lang.String valueSchemaDir, java.lang.String rmdSchemaDir)
HDFSSchemaSource(java.lang.String valueSchemaDir, java.lang.String rmdSchemaDir, java.lang.String storeName)
HDFSSchemaSource(org.apache.hadoop.fs.Path valueSchemaDir, org.apache.hadoop.fs.Path rmdSchemaDir, java.lang.String storeName)
HDFSSchemaSource(org.apache.hadoop.fs.Path valueSchemaDir, org.apache.hadoop.fs.Path rmdSchemaDir, org.apache.hadoop.fs.Path keySchemaDir, java.lang.String storeName)
-
Method Summary
All Methods Instance Methods Concrete Methods Modifier and Type Method Description void
close()
org.apache.avro.Schema
fetchKeySchema()
java.util.Map<RmdVersionId,org.apache.avro.Schema>
fetchRmdSchemas()
Fetch all rmd schemas under thermdSchemaDir
if available.java.util.Map<java.lang.Integer,org.apache.avro.Schema>
fetchValueSchemas()
Fetch all value schemas under thermdSchemaDir
if available.java.lang.String
getKeySchemaPath()
java.lang.String
getRmdSchemaPath()
java.lang.String
getValueSchemaPath()
void
saveSchemasOnDisk(ControllerClient controllerClient)
This method loads related configs, gets store value and RMD schemas from Venice controllers, and then write them on HDFS.
-
-
-
Constructor Detail
-
HDFSSchemaSource
public HDFSSchemaSource(org.apache.hadoop.fs.Path valueSchemaDir, org.apache.hadoop.fs.Path rmdSchemaDir, org.apache.hadoop.fs.Path keySchemaDir, java.lang.String storeName) throws java.io.IOException
- Throws:
java.io.IOException
-
HDFSSchemaSource
public HDFSSchemaSource(org.apache.hadoop.fs.Path valueSchemaDir, org.apache.hadoop.fs.Path rmdSchemaDir, java.lang.String storeName) throws java.io.IOException
- Throws:
java.io.IOException
-
HDFSSchemaSource
public HDFSSchemaSource(java.lang.String valueSchemaDir, java.lang.String rmdSchemaDir, java.lang.String storeName) throws java.io.IOException
- Throws:
java.io.IOException
-
HDFSSchemaSource
public HDFSSchemaSource(java.lang.String valueSchemaDir, java.lang.String rmdSchemaDir) throws java.io.IOException
- Throws:
java.io.IOException
-
-
Method Detail
-
getKeySchemaPath
public java.lang.String getKeySchemaPath()
-
getRmdSchemaPath
public java.lang.String getRmdSchemaPath()
-
getValueSchemaPath
public java.lang.String getValueSchemaPath()
-
saveSchemasOnDisk
public void saveSchemasOnDisk(ControllerClient controllerClient) throws java.io.IOException, java.lang.IllegalStateException
This method loads related configs, gets store value and RMD schemas from Venice controllers, and then write them on HDFS.- Throws:
java.io.IOException
java.lang.IllegalStateException
-
fetchRmdSchemas
public java.util.Map<RmdVersionId,org.apache.avro.Schema> fetchRmdSchemas() throws java.io.IOException
Fetch all rmd schemas under thermdSchemaDir
if available.- Specified by:
fetchRmdSchemas
in interfaceSchemaSource
- Returns:
- RMD schemas mapping
- Throws:
java.io.IOException
-
fetchValueSchemas
public java.util.Map<java.lang.Integer,org.apache.avro.Schema> fetchValueSchemas() throws java.io.IOException
Fetch all value schemas under thermdSchemaDir
if available.- Specified by:
fetchValueSchemas
in interfaceSchemaSource
- Returns:
- RMD schemas mapping
- Throws:
java.io.IOException
-
fetchKeySchema
public org.apache.avro.Schema fetchKeySchema() throws java.io.IOException
- Specified by:
fetchKeySchema
in interfaceSchemaSource
- Throws:
java.io.IOException
-
close
public void close()
- Specified by:
close
in interfacejava.lang.AutoCloseable
-
-