Package com.linkedin.venice.hadoop.spark.input.hdfs

Class VeniceHdfsSource

  • All Implemented Interfaces:
    org.apache.spark.sql.connector.catalog.TableProvider
    public class VeniceHdfsSource
extends java.lang.Object
implements org.apache.spark.sql.connector.catalog.TableProvider
    This is the entrypoint of the Avro input source. It is used by Spark to create a DataFrame from a directory on HDFS. The directory must contain either Avro or Vson files. The format of input files must be homogenous, i.e., it cannot contain mixed formats or schemas.

    • Method Summary

      All Methods Instance Methods Concrete Methods 
      Modifier and Type Method Description
      org.apache.spark.sql.connector.catalog.Table getTable​(org.apache.spark.sql.types.StructType schema, org.apache.spark.sql.connector.expressions.Transform[] partitioning, java.util.Map<java.lang.String,​java.lang.String> configs)  
      org.apache.spark.sql.types.StructType inferSchema​(org.apache.spark.sql.util.CaseInsensitiveStringMap options)  

      • Methods inherited from class java.lang.Object

        clone, equals, finalize, getClass, hashCode, notify, notifyAll, toString, wait, wait, wait

      • Methods inherited from interface org.apache.spark.sql.connector.catalog.TableProvider

        inferPartitioning, supportsExternalMetadata

    • Constructor Detail

      • VeniceHdfsSource

        public VeniceHdfsSource()

    • Method Detail

      • inferSchema

        public org.apache.spark.sql.types.StructType inferSchema​(org.apache.spark.sql.util.CaseInsensitiveStringMap options)
        Specified by:
        inferSchema in interface org.apache.spark.sql.connector.catalog.TableProvider

      • getTable

        public org.apache.spark.sql.connector.catalog.Table getTable​(org.apache.spark.sql.types.StructType schema,
                                                             org.apache.spark.sql.connector.expressions.Transform[] partitioning,
                                                             java.util.Map<java.lang.String,​java.lang.String> configs)
        Specified by:
        getTable in interface org.apache.spark.sql.connector.catalog.TableProvider