Package com.linkedin.venice.hadoop.task.datawriter

Class CompositeVeniceWriter<K,V,U>

java.lang.Object

com.linkedin.venice.writer.AbstractVeniceWriter<K,V,U>

com.linkedin.venice.hadoop.task.datawriter.CompositeVeniceWriter<K,V,U>

All Implemented Interfaces:

Closeable, AutoCloseable

public class CompositeVeniceWriter<K,V,U>
extends AbstractVeniceWriter<K,V,U>

The composite writer contains a main writer and multiple child writers. The main writer will only perform the write once all of its child writers are complete. Child writers are ComplexVeniceWriter. This is to provide chunking support during NR pass-through. All records produced by the ComplexVeniceWriter's main writer is expected to carry the PubSubMessageHeaders.VENICE_VIEW_PARTITIONS_MAP_HEADER.

Field Summary

Fields inherited from class com.linkedin.venice.writer.AbstractVeniceWriter

topicName

Constructor Summary

Constructors

Constructor	Description
CompositeVeniceWriter(String topicName, VeniceWriter<K,V,U> mainWriter, ComplexVeniceWriter<K,V,U>[] childWriters, PubSubProducerCallback childCallback, BiFunction<V,Integer,org.apache.avro.generic.GenericRecord> valueExtractor)

Method Summary

Modifier and Type	Method	Description
void	close()
void	close(boolean gracefulClose)
CompletableFuture<PubSubProduceResult>	delete(K key, long logicalTimestamp, PubSubProducerCallback callback)
CompletableFuture<PubSubProduceResult>	delete(K key, PubSubProducerCallback callback)
CompletableFuture<PubSubProduceResult>	delete(K key, PubSubProducerCallback callback, DeleteMetadata deleteMetadata)	In VPJ, only re-push can trigger this function.
void	flush()
CompletableFuture<PubSubProduceResult>	put(K key, V value, int valueSchemaId, long logicalTimestamp, PubSubProducerCallback callback)
CompletableFuture<PubSubProduceResult>	put(K key, V value, int valueSchemaId, long logicalTimestamp, PubSubProducerCallback callback, PutMetadata putMetadata)
CompletableFuture<PubSubProduceResult>	put(K key, V value, int valueSchemaId, PubSubProducerCallback callback)
CompletableFuture<PubSubProduceResult>	put(K key, V value, int valueSchemaId, PubSubProducerCallback callback, PutMetadata putMetadata)
CompletableFuture<PubSubProduceResult>	update(K key, U update, int valueSchemaId, int derivedSchemaId, long logicalTimestamp, PubSubProducerCallback callback)
Future<PubSubProduceResult>	update(K key, U update, int valueSchemaId, int derivedSchemaId, PubSubProducerCallback callback)	The main use of the CompositeVeniceWriter for now is to write batch portion of a store version to VT and materialized view topic in the NR fabric.

Methods inherited from class com.linkedin.venice.writer.AbstractVeniceWriter

getTopicName, put

Methods inherited from class java.lang.Object

clone, equals, finalize, getClass, hashCode, notify, notifyAll, toString, wait, wait, wait

Constructor Details

CompositeVeniceWriter

public CompositeVeniceWriter(String topicName,
 VeniceWriter<K,V,U> mainWriter,
 ComplexVeniceWriter<K,V,U>[] childWriters,
 PubSubProducerCallback childCallback,
 BiFunction<V,Integer,org.apache.avro.generic.GenericRecord> valueExtractor)

Method Details

close

public void close(boolean gracefulClose)
           throws IOException

Specified by:

close in class AbstractVeniceWriter<K,V,U>

Throws:

IOException

put

public CompletableFuture<PubSubProduceResult> put(K key,
 V value,
 int valueSchemaId,
 PubSubProducerCallback callback)

Specified by:

put in class AbstractVeniceWriter<K,V,U>

put

public CompletableFuture<PubSubProduceResult> put(K key,
 V value,
 int valueSchemaId,
 long logicalTimestamp,
 PubSubProducerCallback callback)

Specified by:

put in class AbstractVeniceWriter<K,V,U>

put

public CompletableFuture<PubSubProduceResult> put(K key,
 V value,
 int valueSchemaId,
 PubSubProducerCallback callback,
 PutMetadata putMetadata)

Specified by:

put in class AbstractVeniceWriter<K,V,U>

put

public CompletableFuture<PubSubProduceResult> put(K key,
 V value,
 int valueSchemaId,
 long logicalTimestamp,
 PubSubProducerCallback callback,
 PutMetadata putMetadata)

Specified by:

put in class AbstractVeniceWriter<K,V,U>

delete

public CompletableFuture<PubSubProduceResult> delete(K key,
 PubSubProducerCallback callback,
 DeleteMetadata deleteMetadata)

In VPJ, only re-push can trigger this function. During re-push the deletion to view topics are useless and should be ignored.

Specified by:

delete in class AbstractVeniceWriter<K,V,U>

update

public Future<PubSubProduceResult> update(K key,
 U update,
 int valueSchemaId,
 int derivedSchemaId,
 PubSubProducerCallback callback)

The main use of the CompositeVeniceWriter for now is to write batch portion of a store version to VT and materialized view topic in the NR fabric. Updates should never go through the CompositeVeniceWriter because it should be written to RT (hybrid writes or incremental push) and handled by view writers in L/F or A/A SIT.

Specified by:

update in class AbstractVeniceWriter<K,V,U>

update

public CompletableFuture<PubSubProduceResult> update(K key,
 U update,
 int valueSchemaId,
 int derivedSchemaId,
 long logicalTimestamp,
 PubSubProducerCallback callback)

Specified by:

update in class AbstractVeniceWriter<K,V,U>

delete

public CompletableFuture<PubSubProduceResult> delete(K key,
 PubSubProducerCallback callback)

Specified by:

delete in class AbstractVeniceWriter<K,V,U>

delete

public CompletableFuture<PubSubProduceResult> delete(K key,
 long logicalTimestamp,
 PubSubProducerCallback callback)

Specified by:

delete in class AbstractVeniceWriter<K,V,U>

flush

public void flush()

Specified by:

flush in class AbstractVeniceWriter<K,V,U>

close

public void close()
           throws IOException

Throws:

IOException