Class HDFSSchemaSource

  • All Implemented Interfaces:
    SchemaSource, java.lang.AutoCloseable

    public class HDFSSchemaSource
    extends java.lang.Object
    implements SchemaSource, java.lang.AutoCloseable
    HDFSSchemaSource caches the Value schemes and RMD schemas for a given store on the HDFS and serve them during the Mapper stage. It will make API calls on Venice controller to retrieve all RMD schemes and all value schemas.
    • Constructor Summary

      Constructors 
      Constructor Description
      HDFSSchemaSource​(java.lang.String valueSchemaDir, java.lang.String rmdSchemaDir)  
      HDFSSchemaSource​(java.lang.String valueSchemaDir, java.lang.String rmdSchemaDir, java.lang.String storeName)  
      HDFSSchemaSource​(org.apache.hadoop.fs.Path valueSchemaDir, org.apache.hadoop.fs.Path rmdSchemaDir, java.lang.String storeName)  
      HDFSSchemaSource​(org.apache.hadoop.fs.Path valueSchemaDir, org.apache.hadoop.fs.Path rmdSchemaDir, org.apache.hadoop.fs.Path keySchemaDir, java.lang.String storeName)  
    • Constructor Detail

      • HDFSSchemaSource

        public HDFSSchemaSource​(org.apache.hadoop.fs.Path valueSchemaDir,
                                org.apache.hadoop.fs.Path rmdSchemaDir,
                                org.apache.hadoop.fs.Path keySchemaDir,
                                java.lang.String storeName)
                         throws java.io.IOException
        Throws:
        java.io.IOException
      • HDFSSchemaSource

        public HDFSSchemaSource​(org.apache.hadoop.fs.Path valueSchemaDir,
                                org.apache.hadoop.fs.Path rmdSchemaDir,
                                java.lang.String storeName)
                         throws java.io.IOException
        Throws:
        java.io.IOException
      • HDFSSchemaSource

        public HDFSSchemaSource​(java.lang.String valueSchemaDir,
                                java.lang.String rmdSchemaDir,
                                java.lang.String storeName)
                         throws java.io.IOException
        Throws:
        java.io.IOException
      • HDFSSchemaSource

        public HDFSSchemaSource​(java.lang.String valueSchemaDir,
                                java.lang.String rmdSchemaDir)
                         throws java.io.IOException
        Throws:
        java.io.IOException
    • Method Detail

      • getKeySchemaPath

        public java.lang.String getKeySchemaPath()
      • getRmdSchemaPath

        public java.lang.String getRmdSchemaPath()
      • getValueSchemaPath

        public java.lang.String getValueSchemaPath()
      • saveSchemasOnDisk

        public void saveSchemasOnDisk​(ControllerClient controllerClient)
                               throws java.io.IOException,
                                      java.lang.IllegalStateException
        This method loads related configs, gets store value and RMD schemas from Venice controllers, and then write them on HDFS.
        Throws:
        java.io.IOException
        java.lang.IllegalStateException
      • fetchRmdSchemas

        public java.util.Map<RmdVersionId,​org.apache.avro.Schema> fetchRmdSchemas()
                                                                                 throws java.io.IOException
        Fetch all rmd schemas under the rmdSchemaDir if available.
        Specified by:
        fetchRmdSchemas in interface SchemaSource
        Returns:
        RMD schemas mapping
        Throws:
        java.io.IOException
      • fetchValueSchemas

        public java.util.Map<java.lang.Integer,​org.apache.avro.Schema> fetchValueSchemas()
                                                                                        throws java.io.IOException
        Fetch all value schemas under the rmdSchemaDir if available.
        Specified by:
        fetchValueSchemas in interface SchemaSource
        Returns:
        RMD schemas mapping
        Throws:
        java.io.IOException
      • fetchKeySchema

        public org.apache.avro.Schema fetchKeySchema()
                                              throws java.io.IOException
        Specified by:
        fetchKeySchema in interface SchemaSource
        Throws:
        java.io.IOException
      • close

        public void close()
        Specified by:
        close in interface java.lang.AutoCloseable