Package com.linkedin.venice.spark.input.pubsub.raw

Class VeniceRawPubsubSource

java.lang.Object
com.linkedin.venice.spark.input.pubsub.raw.VeniceRawPubsubSource
All Implemented Interfaces:
org.apache.spark.sql.connector.catalog.TableProvider
public class VeniceRawPubsubSource extends Object implements org.apache.spark.sql.connector.catalog.TableProvider
This is the entrypoint of the PubSub input source. It is used by Spark to create a DataFrame from a PubSub topic The Topic must conform to Venice PubSub Version-Topic Schema. This source does not support Realtime or versioned realtime topics at this point.

  • Constructor Summary

    Constructors
    Constructor
    Description
    VeniceRawPubsubSource()
     

  • Method Summary

    Modifier and Type
    Method
    Description
    org.apache.spark.sql.connector.catalog.Table
    getTable(org.apache.spark.sql.types.StructType schema, org.apache.spark.sql.connector.expressions.Transform[] partitioning, Map<String,String> configs)
     
    org.apache.spark.sql.types.StructType
    inferSchema(org.apache.spark.sql.util.CaseInsensitiveStringMap options)
     

    Methods inherited from class java.lang.Object

    clone, equals, finalize, getClass, hashCode, notify, notifyAll, toString, wait, wait, wait

    Methods inherited from interface org.apache.spark.sql.connector.catalog.TableProvider

    inferPartitioning, supportsExternalMetadata

  • Constructor Details

    • VeniceRawPubsubSource

      public VeniceRawPubsubSource()

  • Method Details

    • inferSchema

      public org.apache.spark.sql.types.StructType inferSchema(org.apache.spark.sql.util.CaseInsensitiveStringMap options)
      Specified by:
      inferSchema in interface org.apache.spark.sql.connector.catalog.TableProvider

    • getTable

      public org.apache.spark.sql.connector.catalog.Table getTable(org.apache.spark.sql.types.StructType schema, org.apache.spark.sql.connector.expressions.Transform[] partitioning, Map<String,String> configs)
      Specified by:
      getTable in interface org.apache.spark.sql.connector.catalog.TableProvider