Package com.linkedin.venice.spark.input.hdfs

Class VeniceHdfsSource

java.lang.Object

com.linkedin.venice.spark.input.hdfs.VeniceHdfsSource

All Implemented Interfaces:

org.apache.spark.sql.connector.catalog.TableProvider

public class VeniceHdfsSource
extends Object
implements org.apache.spark.sql.connector.catalog.TableProvider

This is the entrypoint of the Avro input source. It is used by Spark to create a DataFrame from a directory on HDFS. The directory must contain either Avro or Vson files. The format of input files must be homogenous, i.e., it cannot contain mixed formats or schemas.

Field Summary

Fields

Modifier and Type	Field	Description
static Map<String,String>	lastReceivedConfigs	Records the configs map (i.e.

Constructor Summary

Constructors

Constructor	Description
VeniceHdfsSource()

Method Summary

Modifier and Type	Method	Description
org.apache.spark.sql.connector.catalog.Table	getTable(org.apache.spark.sql.types.StructType schema, org.apache.spark.sql.connector.expressions.Transform[] partitioning, Map<String,String> configs)
org.apache.spark.sql.types.StructType	inferSchema(org.apache.spark.sql.util.CaseInsensitiveStringMap options)
static void	recordLastReceivedConfigs(Map<String,String> configs)	Writes lastReceivedConfigs.

Methods inherited from class java.lang.Object

clone, equals, finalize, getClass, hashCode, notify, notifyAll, toString, wait, wait, wait

Methods inherited from interface org.apache.spark.sql.connector.catalog.TableProvider

inferPartitioning, supportsExternalMetadata

Field Details

lastReceivedConfigs

public static volatile Map<String,String> lastReceivedConfigs

Records the configs map (i.e. the DataFrameReader/reader options Spark resolved for this DataSource V2 table) most recently passed to getTable(org.apache.spark.sql.types.StructType, org.apache.spark.sql.connector.expressions.Transform[], java.util.Map<java.lang.String, java.lang.String>). This is the only config channel visible to VeniceHdfsInputTable, VeniceHdfsInputScanBuilder, and executor partition readers -- unlike SparkSession.conf(), which is visible on the driver but never threaded through to this map. Exposed purely for tests to verify (or disprove) that a given config actually reached the custom DataSource, since there is no public Spark API to inspect a DataFrameReader's options after the fact.

Constructor Details

VeniceHdfsSource

public VeniceHdfsSource()

Method Details

inferSchema

public org.apache.spark.sql.types.StructType inferSchema(org.apache.spark.sql.util.CaseInsensitiveStringMap options)

Specified by:

inferSchema in interface org.apache.spark.sql.connector.catalog.TableProvider

getTable

public org.apache.spark.sql.connector.catalog.Table getTable(org.apache.spark.sql.types.StructType schema,
 org.apache.spark.sql.connector.expressions.Transform[] partitioning,
 Map<String,String> configs)

Specified by:

getTable in interface org.apache.spark.sql.connector.catalog.TableProvider

recordLastReceivedConfigs

public static void recordLastReceivedConfigs(Map<String,String> configs)

Writes lastReceivedConfigs. Kept as a dedicated static method (rather than assigning the field directly from the instance method getTable(org.apache.spark.sql.types.StructType, org.apache.spark.sql.connector.expressions.Transform[], java.util.Map<java.lang.String, java.lang.String>)) to avoid SpotBugs' ST_WRITE_TO_STATIC_FROM_INSTANCE_METHOD.