Package com.linkedin.venice.writer

Class ComplexVeniceWriter<K,V,U>

java.lang.Object
com.linkedin.venice.writer.AbstractVeniceWriter<K,V,U>
com.linkedin.venice.writer.VeniceWriter<K,V,U>
com.linkedin.venice.writer.ComplexVeniceWriter<K,V,U>
All Implemented Interfaces:
Closeable, AutoCloseable
public class ComplexVeniceWriter<K,V,U> extends VeniceWriter<K,V,U>
Provide more complex and sophisticated writer APIs for writing to MaterializedView. Specifically when a ComplexVenicePartitioner is involved. Otherwise, use the VeniceWriter APIs.

  • Constructor Details

  • Method Details

    • complexPut

      public CompletableFuture<Void> complexPut(K key, V value, int valueSchemaId, Lazy<org.apache.avro.generic.GenericRecord> valueProvider)

    • complexPut

      public CompletableFuture<Void> complexPut(K key, V value, int valueSchemaId, Lazy<org.apache.avro.generic.GenericRecord> valueProvider, Consumer<int[]> partitionConsumer, PubSubProducerCallback callback, PutMetadata putMetadata)
      ComplexVenicePartitioner offers a more sophisticated getPartitionId API. It also takes value as a parameter, and could return a single, multiple or no partition(s). The API also accepts a partition consumer to offer the resulting partition(s) of this complexPut.

    • forwardPut

      public CompletableFuture<Void> forwardPut(K key, V value, int valueSchemaId, Set<Integer> partitions)
      Used during NR pass-through in remote region to forward records or chunks of records to corresponding view partition based on provided view partition map. This way the producing leader don't need to worry about large record assembly or chunking for view topic(s) when ingesting from source VT during NR pass-through. It's also expected to receive an empty partition set and in which case it's a no-op and we simply return a completed future. This is a valid use case since certain complex partitioner implementation could filter out records based on value fields and return an empty partition.

    • complexDelete

      public CompletableFuture<Void> complexDelete(K key, Lazy<org.apache.avro.generic.GenericRecord> valueProvider)
      Perform "delete" on the given key. If a ComplexVenicePartitioner is involved then it will be a best effort attempt to delete the record using the valueProvider. It's best effort because: 1. Nothing we can do if value is null or not provided via valueProvider. 2. Previous writes

    • put

      public CompletableFuture<PubSubProduceResult> put(K key, V value, int valueSchemaId, PubSubProducerCallback callback, LeaderMetadataWrapper leaderMetadataWrapper, long logicalTs, PutMetadata putMetadata, ChunkedValueManifest oldValueManifest, ChunkedValueManifest oldRmdManifest)
      Description copied from class: VeniceWriter
      Execute a standard "put" on the key. VeniceReducer and VeniceSystemProducer should call this API.
      Overrides:
      put in class VeniceWriter<K,V,U>
      Parameters:
      key - - The key to put in storage.
      value - - The value to be associated with the given key
      valueSchemaId - - value schema id for the given value
      callback - - Callback function invoked by Kafka producer after sending the message
      leaderMetadataWrapper - - The leader Metadata of this message in the source topic: -1: VeniceWriter is sending this message in a Samza app to the real-time topic; or it's sending the message in VPJ plugin to the version topic. If views are enabled the metadata wrapper can also contain view partition map to be sent as PubSubMessageHeader. >=0: Leader replica consumes a put message from real-time topic, VeniceWriter in leader is sending this message to version topic with extra info: offset in the real-time topic.
      logicalTs - - An timestamp field to indicate when this record was produced from apps view.
      putMetadata - - a PutMetadata containing replication metadata related fields (can be null).
      Returns:
      a java.util.concurrent.Future Future for the RecordMetadata that will be assigned to this record. Invoking java.util.concurrent.Future's get() on this future will block until the associated request completes and then return the metadata for the record or throw any exception that occurred while sending the record.

    • delete

      public CompletableFuture<PubSubProduceResult> delete(K key, PubSubProducerCallback callback, LeaderMetadataWrapper leaderMetadataWrapper, long logicalTs, DeleteMetadata deleteMetadata, ChunkedValueManifest oldValueManifest, ChunkedValueManifest oldRmdManifest)
      Description copied from class: VeniceWriter
      Execute a standard "delete" on the key.
      Overrides:
      delete in class VeniceWriter<K,V,U>
      Parameters:
      key - - The key to delete in storage.
      callback - - Callback function invoked by Kafka producer after sending the message.
      leaderMetadataWrapper - - The leader Metadata of this message in the source topic: -1: VeniceWriter is sending this message in a Samza app to the real-time topic; or it's sending the message in VPJ plugin to the version topic; >=0: Leader replica consumes a DELETE message from real-time topic, VeniceWriter in leader is sending this message to version topic with extra info: offset in the real-time topic.
      logicalTs - - An timestamp field to indicate when this record was produced from apps point of view.
      deleteMetadata - - a DeleteMetadata containing replication metadata related fields (can be null).
      Returns:
      a java.util.concurrent.CompletableFuture. Future for the RecordMetadata that will be assigned to this record. Invoking java.util.concurrent.CompletableFuture's get() on this future will block until the associated request completes and then return the metadata for the record or throw any exception that occurred while sending the record.

    • update

      public Future<PubSubProduceResult> update(K key, U update, int valueSchemaId, int derivedSchemaId, PubSubProducerCallback callback, long logicalTs)
      Overrides:
      update in class VeniceWriter<K,V,U>

    • getViewName

      public String getViewName()