Package com.linkedin.venice.writer

Class BatchingVeniceWriter<K,V,U>

java.lang.Object

com.linkedin.venice.writer.AbstractVeniceWriter<K,V,U>

com.linkedin.venice.writer.BatchingVeniceWriter<K,V,U>

All Implemented Interfaces:

Closeable, AutoCloseable

public class BatchingVeniceWriter<K,V,U>
extends AbstractVeniceWriter<K,V,U>

This class is a batching implementation of VeniceWriter. The intention of this class is to: (1) Reduce message volumes sent to Venice backend. (2) Avoid generating messages for the same key with the same timestamp for the same writer. Today's Active/Active DCR algorithm will compare field value to break tie when two messages arrive with the same timestamp. This makes sure this case will not happen if each key is only produced by single writer and user will always see the latest value in the producing order. There are two configs that control the batching behavior: (1) Max batch interval: Maximum delay of a batch of records before it is produced. (2) Max batch buffer size: Maximum size of buffer records before it is produced. If any of the limit is reached, the buffered batch will be flushed and produced. For messages within the same batch, only the last one will be produced into the topic, except for those mentioned below. (1) UPDATE message: It will be supported in the future. (2) Message with logical timestamp: It will be sent out individually. When the last message is produced, its callback will be completed (either successfully or exceptionally), all the related messages' callbacks will also be completed with the same result.

Field Summary

Fields

Modifier and Type	Field	Description
static final org.apache.logging.log4j.Logger	LOGGER

Fields inherited from class com.linkedin.venice.writer.AbstractVeniceWriter

topicName

Constructor Summary

Constructors

Constructor	Description
BatchingVeniceWriter(VeniceWriterOptions params, VeniceProperties props, PubSubProducerAdapter producerAdapter)

Method Summary

Modifier and Type	Method	Description
void	close()
void	close(boolean gracefulClose)
CompletableFuture<PubSubProduceResult>	delete(K key, long logicalTs, PubSubProducerCallback callback)
CompletableFuture<PubSubProduceResult>	delete(K key, PubSubProducerCallback callback)
CompletableFuture<PubSubProduceResult>	delete(K key, PubSubProducerCallback callback, DeleteMetadata deleteMetadata)
void	flush()
CompletableFuture<PubSubProduceResult>	put(K key, V value, int valueSchemaId, long logicalTs, PubSubProducerCallback callback)
CompletableFuture<PubSubProduceResult>	put(K key, V value, int valueSchemaId, long logicalTimestamp, PubSubProducerCallback callback, PutMetadata putMetadata)
CompletableFuture<PubSubProduceResult>	put(K key, V value, int valueSchemaId, PubSubProducerCallback callback)
CompletableFuture<PubSubProduceResult>	put(K key, V value, int valueSchemaId, PubSubProducerCallback callback, PutMetadata putMetadata)
CompletableFuture<PubSubProduceResult>	update(K key, U update, int valueSchemaId, int derivedSchemaId, long logicalTimestamp, PubSubProducerCallback callback)
CompletableFuture<PubSubProduceResult>	update(K key, U update, int valueSchemaId, int derivedSchemaId, PubSubProducerCallback callback)

Methods inherited from class com.linkedin.venice.writer.AbstractVeniceWriter

getTopicName, put

Methods inherited from class java.lang.Object

clone, equals, finalize, getClass, hashCode, notify, notifyAll, toString, wait, wait, wait

Field Details

LOGGER

public static final org.apache.logging.log4j.Logger LOGGER

Constructor Details

BatchingVeniceWriter

public BatchingVeniceWriter(VeniceWriterOptions params,
 VeniceProperties props,
 PubSubProducerAdapter producerAdapter)

Method Details

delete

public CompletableFuture<PubSubProduceResult> delete(K key,
 long logicalTs,
 PubSubProducerCallback callback)

Specified by:

delete in class AbstractVeniceWriter<K,V,U>

put

public CompletableFuture<PubSubProduceResult> put(K key,
 V value,
 int valueSchemaId,
 PubSubProducerCallback callback,
 PutMetadata putMetadata)

Specified by:

put in class AbstractVeniceWriter<K,V,U>

put

public CompletableFuture<PubSubProduceResult> put(K key,
 V value,
 int valueSchemaId,
 long logicalTimestamp,
 PubSubProducerCallback callback,
 PutMetadata putMetadata)

Specified by:

put in class AbstractVeniceWriter<K,V,U>

delete

public CompletableFuture<PubSubProduceResult> delete(K key,
 PubSubProducerCallback callback,
 DeleteMetadata deleteMetadata)

Specified by:

delete in class AbstractVeniceWriter<K,V,U>

delete

public CompletableFuture<PubSubProduceResult> delete(K key,
 PubSubProducerCallback callback)

Specified by:

delete in class AbstractVeniceWriter<K,V,U>

put

public CompletableFuture<PubSubProduceResult> put(K key,
 V value,
 int valueSchemaId,
 long logicalTs,
 PubSubProducerCallback callback)

Specified by:

put in class AbstractVeniceWriter<K,V,U>

put

public CompletableFuture<PubSubProduceResult> put(K key,
 V value,
 int valueSchemaId,
 PubSubProducerCallback callback)

Specified by:

put in class AbstractVeniceWriter<K,V,U>

update

public CompletableFuture<PubSubProduceResult> update(K key,
 U update,
 int valueSchemaId,
 int derivedSchemaId,
 PubSubProducerCallback callback)

Specified by:

update in class AbstractVeniceWriter<K,V,U>

update

public CompletableFuture<PubSubProduceResult> update(K key,
 U update,
 int valueSchemaId,
 int derivedSchemaId,
 long logicalTimestamp,
 PubSubProducerCallback callback)

Specified by:

update in class AbstractVeniceWriter<K,V,U>

flush

public void flush()

Specified by:

flush in class AbstractVeniceWriter<K,V,U>

close

public void close(boolean gracefulClose)

Specified by:

close in class AbstractVeniceWriter<K,V,U>

close

public void close()
           throws IOException

Throws:

IOException