Package com.linkedin.venice.producer

Class AbstractVeniceProducer<K,V>

java.lang.Object
com.linkedin.venice.producer.AbstractVeniceProducer<K,V>
All Implemented Interfaces:
VeniceProducer<K,V>, Closeable, AutoCloseable
Direct Known Subclasses:
OnlineVeniceProducer
public abstract class AbstractVeniceProducer<K,V> extends Object implements VeniceProducer<K,V>
Async Venice producer with configurable parallelism.

Flow: asyncPut(key) → Worker[partition % workers] → serialize → veniceWriter.put() → PubSub callback → Callback executor → complete future

Both worker pool and callback executor are optional (set count=0 for inline execution). Per-key ordering is preserved since same key always routes to same worker.

See Also:

  • Field Details

    • needsPartitionRouting

      protected boolean needsPartitionRouting

  • Constructor Details

    • AbstractVeniceProducer

      public AbstractVeniceProducer()

  • Method Details

    • configure

      protected void configure(String storeName, VeniceProperties producerConfigs, io.tehuti.metrics.MetricsRepository metricsRepository, SchemaReader schemaReader, SchemaReader kmeSchemaReader)

    • createDispatcher

      protected PartitionedProducerExecutor createDispatcher(String storeName, VeniceProperties configs, io.tehuti.metrics.MetricsRepository metricsRepository)
      Factory method for creating the dispatcher. Can be overridden for testing.

      Execution modes based on configuration:

      • workerCount = 0, callbackThreadCount = 0: Fully inline - all tasks execute on caller thread
      • workerCount > 0, callbackThreadCount = 0: Parallel workers, callbacks on PubSub thread
      • workerCount = 0, callbackThreadCount > 0: Inline preprocessing, callbacks on dedicated threads
      • workerCount > 0, callbackThreadCount > 0: Full async - parallel workers + callback isolation

      Set workerCount = 0 to disable worker threads and execute all preprocessing inline on the caller thread. This is useful when the caller is already managing concurrency externally or wants minimal overhead.

      See Also:

    • constructVeniceWriter

      protected VeniceWriter<byte[],byte[],byte[]> constructVeniceWriter(Properties properties, VeniceWriterOptions writerOptions)

    • getSerializer

      protected RecordSerializer<Object> getSerializer(org.apache.avro.Schema schema)

    • asyncPut

      public CompletableFuture<DurableWrite> asyncPut(K key, V value)
      Description copied from interface: VeniceProducer
      A write operation where a full value is written to replace the existing value.
      Specified by:
      asyncPut in interface VeniceProducer<K,V>
      Parameters:
      key - Key of the record that needs to be updated
      value - The full value that needs to be written
      Returns:
      A CompletableFuture that completes when the write operation is durable. It does not imply that the data is available to readers.

    • asyncPut

      public CompletableFuture<DurableWrite> asyncPut(long logicalTime, K key, V value)
      Description copied from interface: VeniceProducer
      A write operation where a full value is written to replace the existing value. It offers the writers to specify a logical time. This value is used to specify the ordering of operations and perform conflict resolution in Active/Active replication.
      Specified by:
      asyncPut in interface VeniceProducer<K,V>
      Parameters:
      logicalTime - The value used during conflict resolution in Active/Active replication
      key - Key of the record that needs to be updated
      value - The full value that needs to be written
      Returns:
      A CompletableFuture that completes when the write operation is durable. It does not imply that the data is available to readers.

    • asyncDelete

      public CompletableFuture<DurableWrite> asyncDelete(K key)
      Description copied from interface: VeniceProducer
      A write operation to delete the record for a key.
      Specified by:
      asyncDelete in interface VeniceProducer<K,V>
      Parameters:
      key - The key associated with the record that should be deleted
      Returns:
      A CompletableFuture that completes when the write operation is durable. It does not imply that the data is available to readers.

    • asyncDelete

      public CompletableFuture<DurableWrite> asyncDelete(long logicalTime, K key)
      Description copied from interface: VeniceProducer
      A write operation to delete the record for a key. It offers the writers to specify a logical time. This value is used to specify the ordering of operations and perform conflict resolution in Active/Active replication.
      Specified by:
      asyncDelete in interface VeniceProducer<K,V>
      Parameters:
      logicalTime - The value used during conflict resolution in Active/Active replication
      key - Key of the record that needs to be deleted
      Returns:
      A CompletableFuture that completes when the write operation is durable. It does not imply that the data is available to readers.

    • asyncUpdate

      public CompletableFuture<DurableWrite> asyncUpdate(K key, Consumer<UpdateBuilder> updateFunction)
      Description copied from interface: VeniceProducer
      A write operation to modify a subset of fields in the record for a key.
      Specified by:
      asyncUpdate in interface VeniceProducer<K,V>
      Parameters:
      key - Key of the record that needs to be updated
      updateFunction - A Consumer that takes in an UpdateBuilder object and updates it to specify which fields to modify and the operations that must be done on them.
      Returns:
      A CompletableFuture that completes when the write operation is durable. It does not imply that the data is available to readers.

    • asyncUpdate

      public CompletableFuture<DurableWrite> asyncUpdate(long logicalTime, K key, Consumer<UpdateBuilder> updateFunction)
      Description copied from interface: VeniceProducer
      A write operation to modify a subset of fields in the record for a key. It offers the writers to specify a logical time. This value is used to specify the ordering of operations and perform conflict resolution in Active/Active replication.
      Specified by:
      asyncUpdate in interface VeniceProducer<K,V>
      Parameters:
      logicalTime - The value used during conflict resolution in Active/Active replication
      key - Key of the record that needs to be updated
      updateFunction - A Consumer that takes in an UpdateBuilder object and updates it to specify which fields to modify and the operations that must be done on them.
      Returns:
      A CompletableFuture that completes when the write operation is durable. It does not imply that the data is available to readers.

    • requestTopic

      protected abstract VersionCreationResponse requestTopic()
      This function should return a VersionCreationResponse to determine the PubSub topic and the characteristics that the producer should follow.

    • close

      public void close() throws IOException
      Specified by:
      close in interface AutoCloseable
      Specified by:
      close in interface Closeable
      Throws:
      IOException

    • isClosed

      protected boolean isClosed()