Package com.linkedin.venice.samza

Class VeniceSystemProducer

  • All Implemented Interfaces:
    java.io.Closeable, java.lang.AutoCloseable, org.apache.samza.system.SystemProducer
    public class VeniceSystemProducer
extends java.lang.Object
implements org.apache.samza.system.SystemProducer, java.io.Closeable
    VeniceSystemProducer defines the interfaces for Samza jobs to send data to Venice stores. Samza jobs talk to either parent or child controller depending on the aggregate mode config. The decision of which controller should be used is made in VeniceSystemFactory. The "Primary Controller" term is used to refer to whichever controller the Samza job should talk to. The primary controller should be: 1. The parent controller when the Venice system is deployed in a multi-colo mode and either: a. Version.PushType is PushType.BATCH or PushType.STREAM_REPROCESSING; or b. @deprecated Version.PushType is PushType.STREAM and the job is configured to write data in AGGREGATE mode 2. The child controller when either: a. The Venice system is deployed in a single-colo mode; or b. The Version.PushType is PushType.STREAM and the job is configured to write data in NON_AGGREGATE mode

    • Constructor Detail

      • VeniceSystemProducer

        @Deprecated
public VeniceSystemProducer​(java.lang.String primaryControllerColoD2ZKHost,
                            java.lang.String primaryControllerD2ServiceName,
                            java.lang.String storeName,
                            Version.PushType pushType,
                            java.lang.String samzaJobId,
                            java.lang.String runningFabric,
                            boolean verifyLatestProtocolPresent,
                            VeniceSystemFactory factory,
                            java.util.Optional<SSLFactory> sslFactory,
                            java.util.Optional<java.lang.String> partitioners)
        Deprecated.

      • VeniceSystemProducer

        public VeniceSystemProducer​(java.lang.String veniceChildD2ZkHost,
                            java.lang.String primaryControllerColoD2ZKHost,
                            java.lang.String primaryControllerD2ServiceName,
                            java.lang.String storeName,
                            Version.PushType pushType,
                            java.lang.String samzaJobId,
                            java.lang.String runningFabric,
                            boolean verifyLatestProtocolPresent,
                            VeniceSystemFactory factory,
                            java.util.Optional<SSLFactory> sslFactory,
                            java.util.Optional<java.lang.String> partitioners)
        Construct a new instance of VeniceSystemProducer. Equivalent to VeniceSystemProducer(veniceChildD2ZkHost, primaryControllerColoD2ZKHost, primaryControllerD2ServiceName, storeName, pushType, samzaJobId, runningFabric, verifyLatestProtocolPresent, factory, sslFactory, partitioners, SystemTime.INSTANCE)

      • VeniceSystemProducer

        @Deprecated
public VeniceSystemProducer​(java.lang.String primaryControllerColoD2ZKHost,
                            java.lang.String primaryControllerD2ServiceName,
                            java.lang.String storeName,
                            Version.PushType pushType,
                            java.lang.String samzaJobId,
                            java.lang.String runningFabric,
                            boolean verifyLatestProtocolPresent,
                            VeniceSystemFactory factory,
                            java.util.Optional<SSLFactory> sslFactory,
                            java.util.Optional<java.lang.String> partitioners,
                            Time time)
        Deprecated.

      • VeniceSystemProducer

        public VeniceSystemProducer​(java.lang.String veniceChildD2ZkHost,
                            java.lang.String primaryControllerColoD2ZKHost,
                            java.lang.String primaryControllerD2ServiceName,
                            java.lang.String storeName,
                            Version.PushType pushType,
                            java.lang.String samzaJobId,
                            java.lang.String runningFabric,
                            boolean verifyLatestProtocolPresent,
                            VeniceSystemFactory factory,
                            java.util.Optional<SSLFactory> sslFactory,
                            java.util.Optional<java.lang.String> partitioners,
                            Time time)
        Construct a new instance of VeniceSystemProducer
        Parameters:
        veniceChildD2ZkHost - D2 Zk Address where the components in the child colo are announcing themselves
        primaryControllerColoD2ZKHost - D2 Zk Address of the colo where the primary controller resides
        primaryControllerD2ServiceName - The service name that the primary controller uses to announce itself to D2
        storeName - The store to write to
        pushType - The Version.PushType to use to write to the store
        samzaJobId - A unique id used to identify jobs that can concurrently write to the same store
        runningFabric - The colo where the job is running. It is used to find the best destination for the data to be written to
        verifyLatestProtocolPresent - Config to check whether the protocol versions used at runtime are valid in Venice backend
        factory - The VeniceSystemFactory object that was used to create this object
        sslFactory - An optional SSLFactory that is used to communicate with other components using SSL
        partitioners - A list of comma-separated partitioners class names that are supported.
        time - An object of type Time. It is helpful to be configurable for testing.

      • VeniceSystemProducer

        public VeniceSystemProducer​(java.lang.String discoveryUrl,
                            java.lang.String storeName,
                            Version.PushType pushType,
                            java.lang.String samzaJobId,
                            java.lang.String runningFabric,
                            boolean verifyLatestProtocolPresent,
                            VeniceSystemFactory factory,
                            java.util.Optional<SSLFactory> sslFactory,
                            java.util.Optional<java.lang.String> partitioners,
                            Time time)

    • Method Detail

      • applyAdditionalWriterConfigs

        public void applyAdditionalWriterConfigs​(java.util.Map<java.lang.String,​java.lang.String> additionalWriterConfigs)

      • setRouterUrl

        public void setRouterUrl​(java.lang.String routerUrl)

      • getRunningFabric

        public java.lang.String getRunningFabric()

      • getTopicName

        public java.lang.String getTopicName()

      • getVeniceWriter

        protected VeniceWriter<byte[],​byte[],​byte[]> getVeniceWriter​(VersionCreationResponse store)
        This method is overridden and not used by LinkedIn internally. Please update the overridden method accordingly after modifying this method.

      • extractConcurrentProducerConfig

        protected static void extractConcurrentProducerConfig​(java.util.Properties veniceWriterProperties,
                                                      VeniceWriterOptions.Builder builder)

      • getVeniceWriter

        protected VeniceWriter<byte[],​byte[],​byte[]> getVeniceWriter​(VersionCreationResponse store,
                                                                         java.util.Properties veniceWriterProperties)
        Please don't remove this method since it is still being used by LinkedIn internally.

      • setupClientsAndReInitProvider

        protected void setupClientsAndReInitProvider()

      • start

        public void start()
        Specified by:
        start in interface org.apache.samza.system.SystemProducer

      • close

        public void close()
        Specified by:
        close in interface java.lang.AutoCloseable
        Specified by:
        close in interface java.io.Closeable

      • stop

        public void stop()
        Specified by:
        stop in interface org.apache.samza.system.SystemProducer

      • register

        public void register​(java.lang.String source)
        Specified by:
        register in interface org.apache.samza.system.SystemProducer

      • send

        public void send​(java.lang.String source,
                 org.apache.samza.system.OutgoingMessageEnvelope outgoingMessageEnvelope)
        Specified by:
        send in interface org.apache.samza.system.SystemProducer

      • send

        protected java.util.concurrent.CompletableFuture<java.lang.Void> send​(java.lang.Object keyObject,
                                                                      java.lang.Object valueObject)

      • put

        public java.util.concurrent.CompletableFuture<java.lang.Void> put​(java.lang.Object keyObject,
                                                                  java.lang.Object valueObject)

      • delete

        public java.util.concurrent.CompletableFuture<java.lang.Void> delete​(java.lang.Object keyObject)

      • flush

        public void flush​(java.lang.String s)
        Flushing the data to Venice store in case VeniceSystemProducer buffers message.
        Specified by:
        flush in interface org.apache.samza.system.SystemProducer
        Parameters:
        s - String representing the source of the message. Currently, VeniceSystemProducer is not using this param.

      • convertPartialUpdateToFullPut

        protected java.lang.Object convertPartialUpdateToFullPut​(Pair<java.lang.Integer,​java.lang.Integer> schemaIds,
                                                         java.lang.Object incomingWriteValueObject)

      • setExitMode

        public void setExitMode​(SamzaExitMode exitMode)

      • getKafkaBootstrapServers

        public java.lang.String getKafkaBootstrapServers()
        Test methods

      • getInternalProducer

        public VeniceWriter<byte[],​byte[],​byte[]> getInternalProducer()

      • setControllerClient

        protected void setControllerClient​(D2ControllerClient controllerClient)