Package com.linkedin.venice.writer

Class ComplexVeniceWriter<K,V,U>

java.lang.Object
com.linkedin.venice.writer.AbstractVeniceWriter<K,V,U>
com.linkedin.venice.writer.VeniceWriter<K,V,U>
com.linkedin.venice.writer.ComplexVeniceWriter<K,V,U>
All Implemented Interfaces:
Closeable, AutoCloseable
public class ComplexVeniceWriter<K,V,U> extends VeniceWriter<K,V,U>
Provide more complex and sophisticated writer APIs for writing to MaterializedView. Specifically when a ComplexVenicePartitioner is involved. Otherwise, use the VeniceWriter APIs.

  • Constructor Details

  • Method Details

    • complexPut

      public CompletableFuture<Void> complexPut(K key, V value, int valueSchemaId, Lazy<org.apache.avro.generic.GenericRecord> valueProvider)
      ComplexVenicePartitioner offers a more sophisticated getPartitionId API. It also takes value as a parameter, and could return a single, multiple or no partition(s).

    • complexDelete

      public CompletableFuture<Void> complexDelete(K key, Lazy<org.apache.avro.generic.GenericRecord> valueProvider)
      Perform "delete" on the given key. If a ComplexVenicePartitioner is involved then it will be a best effort attempt to delete the record using the valueProvider. It's best effort because: 1. Nothing we can do if value is null or not provided via valueProvider. 2. Previous writes

    • putLargeValue

      protected CompletableFuture<PubSubProduceResult> putLargeValue(byte[] serializedKey, byte[] serializedValue, int valueSchemaId, PubSubProducerCallback callback, int partition, LeaderMetadataWrapper leaderMetadataWrapper, long logicalTs, PutMetadata putMetadata, ChunkedValueManifest oldValueManifest, ChunkedValueManifest oldRmdManifest)
      Prevent the ComplexVeniceWriter from writing any actual chunks for large values. This is because we are only using ComplexVeniceWriter for writing to materialized view. The consumers of materialized view do not fully support assembling the chunks correctly yet. The behavior is the same for both large values from VPJ and leader replicas.
      Overrides:
      putLargeValue in class VeniceWriter<K,V,U>

    • put

      public CompletableFuture<PubSubProduceResult> put(K key, V value, int valueSchemaId, PubSubProducerCallback callback, LeaderMetadataWrapper leaderMetadataWrapper, long logicalTs, PutMetadata putMetadata, ChunkedValueManifest oldValueManifest, ChunkedValueManifest oldRmdManifest)
      Description copied from class: VeniceWriter
      Execute a standard "put" on the key. VeniceReducer and VeniceSystemProducer should call this API.
      Overrides:
      put in class VeniceWriter<K,V,U>
      Parameters:
      key - - The key to put in storage.
      value - - The value to be associated with the given key
      valueSchemaId - - value schema id for the given value
      callback - - Callback function invoked by Kafka producer after sending the message
      leaderMetadataWrapper - - The leader Metadata of this message in the source topic: -1: VeniceWriter is sending this message in a Samza app to the real-time topic; or it's sending the message in VPJ plugin to the version topic; >=0: Leader replica consumes a put message from real-time topic, VeniceWriter in leader is sending this message to version topic with extra info: offset in the real-time topic.
      logicalTs - - An timestamp field to indicate when this record was produced from apps view.
      putMetadata - - a PutMetadata containing replication metadata related fields (can be null).
      Returns:
      a java.util.concurrent.Future Future for the RecordMetadata that will be assigned to this record. Invoking java.util.concurrent.Future's get() on this future will block until the associated request completes and then return the metadata for the record or throw any exception that occurred while sending the record.

    • delete

      public CompletableFuture<PubSubProduceResult> delete(K key, PubSubProducerCallback callback, LeaderMetadataWrapper leaderMetadataWrapper, long logicalTs, DeleteMetadata deleteMetadata, ChunkedValueManifest oldValueManifest, ChunkedValueManifest oldRmdManifest)
      Description copied from class: VeniceWriter
      Execute a standard "delete" on the key.
      Overrides:
      delete in class VeniceWriter<K,V,U>
      Parameters:
      key - - The key to delete in storage.
      callback - - Callback function invoked by Kafka producer after sending the message.
      leaderMetadataWrapper - - The leader Metadata of this message in the source topic: -1: VeniceWriter is sending this message in a Samza app to the real-time topic; or it's sending the message in VPJ plugin to the version topic; >=0: Leader replica consumes a DELETE message from real-time topic, VeniceWriter in leader is sending this message to version topic with extra info: offset in the real-time topic.
      logicalTs - - An timestamp field to indicate when this record was produced from apps point of view.
      deleteMetadata - - a DeleteMetadata containing replication metadata related fields (can be null).
      Returns:
      a java.util.concurrent.CompletableFuture. Future for the RecordMetadata that will be assigned to this record. Invoking java.util.concurrent.CompletableFuture's get() on this future will block until the associated request completes and then return the metadata for the record or throw any exception that occurred while sending the record.

    • update

      public Future<PubSubProduceResult> update(K key, U update, int valueSchemaId, int derivedSchemaId, PubSubProducerCallback callback, long logicalTs)
      Overrides:
      update in class VeniceWriter<K,V,U>