Package com.linkedin.venice.hadoop.task.datawriter

Class DualWriteVeniceWriter

java.lang.Object

com.linkedin.venice.writer.AbstractVeniceWriter<byte[],byte[],byte[]>

com.linkedin.venice.hadoop.task.datawriter.DualWriteVeniceWriter

All Implemented Interfaces:

Closeable, AutoCloseable

public class DualWriteVeniceWriter
extends AbstractVeniceWriter<byte[],byte[],byte[]>

Wraps a Kafka-backed AbstractVeniceWriter and an ExternalStorageWriter so each batch-push record is written to the external sink first and then produced to Kafka. The Kafka future is what the caller waits on for at-least-once semantics; external-sink durability is synchronous from the caller's point of view, so a failed external write throws before the Kafka produce starts and the Spark task is retried.

The writer buffers up to batchSize consecutive put records and flushes them as a single ExternalStorageWriter.batchPut(List) before the corresponding Kafka produces fire. batchSize = 1 (the default) disables buffering: every record is forwarded immediately as a one-element batch, matching pre-batching semantics. flush() and close() drain any pending buffer before doing their own work, so the producer's record ordering is preserved.

update and delete are not supported — batch pushes from clean input never call either. Both throw UnsupportedOperationException so a stray invocation fails the Spark task loudly rather than silently leaving the external sink and Venice in divergent states.

Field Summary

Fields inherited from class com.linkedin.venice.writer.AbstractVeniceWriter

topicName

Constructor Summary

Constructors

Constructor	Description
DualWriteVeniceWriter(String topicName, AbstractVeniceWriter<byte[],byte[],byte[]> kafkaWriter, ExternalStorageWriter externalWriter, int batchSize)	Convenience constructor for callers that don't need the buffered retry policy (e.g.
DualWriteVeniceWriter(String topicName, AbstractVeniceWriter<byte[],byte[],byte[]> kafkaWriter, ExternalStorageWriter externalWriter, int batchSize, int batchPutRetries, long batchPutRetryBackoffMs)

Method Summary

Modifier and Type	Method	Description
void	close()
void	close(boolean gracefulClose)
CompletableFuture<PubSubProduceResult>	delete(byte[] key, long logicalTimestamp, PubSubProducerCallback callback)
CompletableFuture<PubSubProduceResult>	delete(byte[] key, PubSubProducerCallback callback)
CompletableFuture<PubSubProduceResult>	delete(byte[] key, PubSubProducerCallback callback, DeleteMetadata deleteMetadata)
void	flush()
CompletableFuture<PubSubProduceResult>	put(byte[] key, byte[] value, int valueSchemaId, long logicalTimestamp, PubSubProducerCallback callback)
CompletableFuture<PubSubProduceResult>	put(byte[] key, byte[] value, int valueSchemaId, long logicalTimestamp, PubSubProducerCallback callback, PutMetadata putMetadata)
CompletableFuture<PubSubProduceResult>	put(byte[] key, byte[] value, int valueSchemaId, PubSubProducerCallback callback)
CompletableFuture<PubSubProduceResult>	put(byte[] key, byte[] value, int valueSchemaId, PubSubProducerCallback callback, PutMetadata putMetadata)
CompletableFuture<PubSubProduceResult>	update(byte[] key, byte[] update, int valueSchemaId, int derivedSchemaId, long logicalTimestamp, PubSubProducerCallback callback)
Future<PubSubProduceResult>	update(byte[] key, byte[] update, int valueSchemaId, int derivedSchemaId, PubSubProducerCallback callback)

Methods inherited from class com.linkedin.venice.writer.AbstractVeniceWriter

getTopicName, put

Methods inherited from class java.lang.Object

clone, equals, finalize, getClass, hashCode, notify, notifyAll, toString, wait, wait, wait

Constructor Details

DualWriteVeniceWriter

public DualWriteVeniceWriter(String topicName,
 AbstractVeniceWriter<byte[],byte[],byte[]> kafkaWriter,
 ExternalStorageWriter externalWriter,
 int batchSize)

Convenience constructor for callers that don't need the buffered retry policy (e.g. unit tests that exercise the wrapper's other behaviors). batchPutRetries = 0 means one attempt, original throw propagates immediately.

DualWriteVeniceWriter

public DualWriteVeniceWriter(String topicName,
 AbstractVeniceWriter<byte[],byte[],byte[]> kafkaWriter,
 ExternalStorageWriter externalWriter,
 int batchSize,
 int batchPutRetries,
 long batchPutRetryBackoffMs)

Method Details

put

public CompletableFuture<PubSubProduceResult> put(byte[] key,
 byte[] value,
 int valueSchemaId,
 PubSubProducerCallback callback)

Specified by:

put in class AbstractVeniceWriter<byte[],byte[],byte[]>

put

public CompletableFuture<PubSubProduceResult> put(byte[] key,
 byte[] value,
 int valueSchemaId,
 long logicalTimestamp,
 PubSubProducerCallback callback)

Specified by:

put in class AbstractVeniceWriter<byte[],byte[],byte[]>

put

public CompletableFuture<PubSubProduceResult> put(byte[] key,
 byte[] value,
 int valueSchemaId,
 PubSubProducerCallback callback,
 PutMetadata putMetadata)

Specified by:

put in class AbstractVeniceWriter<byte[],byte[],byte[]>

put

public CompletableFuture<PubSubProduceResult> put(byte[] key,
 byte[] value,
 int valueSchemaId,
 long logicalTimestamp,
 PubSubProducerCallback callback,
 PutMetadata putMetadata)

Specified by:

put in class AbstractVeniceWriter<byte[],byte[],byte[]>

delete

public CompletableFuture<PubSubProduceResult> delete(byte[] key,
 PubSubProducerCallback callback)

Specified by:

delete in class AbstractVeniceWriter<byte[],byte[],byte[]>

delete

public CompletableFuture<PubSubProduceResult> delete(byte[] key,
 long logicalTimestamp,
 PubSubProducerCallback callback)

Specified by:

delete in class AbstractVeniceWriter<byte[],byte[],byte[]>

delete

public CompletableFuture<PubSubProduceResult> delete(byte[] key,
 PubSubProducerCallback callback,
 DeleteMetadata deleteMetadata)

Specified by:

delete in class AbstractVeniceWriter<byte[],byte[],byte[]>

update

public Future<PubSubProduceResult> update(byte[] key,
 byte[] update,
 int valueSchemaId,
 int derivedSchemaId,
 PubSubProducerCallback callback)

Specified by:

update in class AbstractVeniceWriter<byte[],byte[],byte[]>

update

public CompletableFuture<PubSubProduceResult> update(byte[] key,
 byte[] update,
 int valueSchemaId,
 int derivedSchemaId,
 long logicalTimestamp,
 PubSubProducerCallback callback)

Specified by:

update in class AbstractVeniceWriter<byte[],byte[],byte[]>

flush

public void flush()

Specified by:

flush in class AbstractVeniceWriter<byte[],byte[],byte[]>

close

public void close()
           throws IOException

Throws:

IOException

close

public void close(boolean gracefulClose)
           throws IOException

Specified by:

close in class AbstractVeniceWriter<byte[],byte[],byte[]>

Throws:

IOException