Package com.linkedin.venice.samza

Class VeniceSystemProducer

java.lang.Object
com.linkedin.venice.samza.VeniceSystemProducer
All Implemented Interfaces:
Closeable, AutoCloseable, org.apache.samza.system.SystemProducer
public class VeniceSystemProducer extends Object implements org.apache.samza.system.SystemProducer, Closeable
VeniceSystemProducer defines the interfaces for Samza jobs to send data to Venice stores. Samza jobs talk to either parent or child controller depending on the aggregate mode config. The decision of which controller should be used is made in VeniceSystemFactory. The "Primary Controller" term is used to refer to whichever controller the Samza job should talk to. The primary controller should be: 1. The parent controller when the Venice system is deployed in a multi-colo mode and either: a. Version.PushType is Version.PushType.BATCH or Version.PushType.STREAM_REPROCESSING; 2. The child controller when either: a. The Venice system is deployed in a single-colo mode; or b. The Version.PushType is Version.PushType.STREAM and the job is configured to write data in NON_AGGREGATE mode

  • Constructor Details

  • Method Details

    • getRunningFabric

      public String getRunningFabric()

    • controllerRequestWithRetry

      protected ControllerResponse controllerRequestWithRetry(Supplier<ControllerResponse> supplier, int retryLimit)

    • getTopicName

      public String getTopicName()

    • getVeniceWriter

      protected AbstractVeniceWriter<byte[],byte[],byte[]> getVeniceWriter(VersionCreationResponse store)
      This method is overridden and not used by LinkedIn internally. Please update the overridden method accordingly after modifying this method.

    • extractConcurrentProducerConfig

      protected static void extractConcurrentProducerConfig(Properties veniceWriterProperties, VeniceWriterOptions.Builder builder)

    • getVeniceWriter

      protected AbstractVeniceWriter<byte[],byte[],byte[]> getVeniceWriter(VersionCreationResponse store, Properties veniceWriterProperties)
      Please don't remove this method since it is still being used by LinkedIn internally.

    • setupClientsAndReInitProvider

      protected void setupClientsAndReInitProvider()

    • start

      public void start()
      Specified by:
      start in interface org.apache.samza.system.SystemProducer

    • close

      public void close()
      Specified by:
      close in interface AutoCloseable
      Specified by:
      close in interface Closeable

    • stop

      public void stop()
      Specified by:
      stop in interface org.apache.samza.system.SystemProducer

    • register

      public void register(String source)
      Specified by:
      register in interface org.apache.samza.system.SystemProducer

    • send

      public void send(String source, org.apache.samza.system.OutgoingMessageEnvelope outgoingMessageEnvelope)
      Specified by:
      send in interface org.apache.samza.system.SystemProducer

    • send

      protected CompletableFuture<Void> send(Object keyObject, Object valueObject)
      Validates, serializes, and writes a key-value pair to Venice. This method handles the full write lifecycle: key schema validation, key/value serialization, schema ID resolution, write-compute conversion for versioned topics, and dispatch to the underlying VeniceWriter (put, delete, or update). For real-time topics, the value may be wrapped in a VeniceObjectWithTimestamp to supply a logical timestamp.
      Parameters:
      keyObject - the key object; must conform to the store's registered key schema
      valueObject - the value object, or null for a delete. For real-time topics, may be a VeniceObjectWithTimestamp wrapping the value with a timestamp.
      Returns:
      a CompletableFuture that completes when the write is acknowledged

    • put

      public CompletableFuture<Void> put(Object keyObject, Object valueObject)

    • delete

      public CompletableFuture<Void> delete(Object keyObject)

    • flush

      public void flush(String s)
      Flushing the data to Venice store in case VeniceSystemProducer buffers message.
      Specified by:
      flush in interface org.apache.samza.system.SystemProducer
      Parameters:
      s - String representing the source of the message. Currently, VeniceSystemProducer is not using this param.

    • convertPartialUpdateToFullPut

      protected Object convertPartialUpdateToFullPut(Pair<Integer,Integer> schemaIds, Object incomingWriteValueObject)

    • setExitMode

      public void setExitMode(SamzaExitMode exitMode)

    • getKafkaBootstrapServers

      public String getKafkaBootstrapServers()
      Test methods

    • getInternalWriter

      public AbstractVeniceWriter<byte[],byte[],byte[]> getInternalWriter()

    • setControllerClient

      protected void setControllerClient(D2ControllerClient controllerClient)