Package com.linkedin.davinci.ingestion.utils

Class IsolatedIngestionUtils

  • public class IsolatedIngestionUtils
extends java.lang.Object
    This class contains methods used for communication between ingestion client and server.

    • Field Detail

      • INGESTION_ISOLATION_CONFIG_PREFIX

        public static final java.lang.String INGESTION_ISOLATION_CONFIG_PREFIX
        See Also:
        Constant Field Values

      • ISOLATED_INGESTION_CONFIG_FILENAME

        public static final java.lang.String ISOLATED_INGESTION_CONFIG_FILENAME
        See Also:
        Constant Field Values

      • ISOLATED_INGESTION_KAFKA_CLUSTER_MAP_FILENAME

        public static final java.lang.String ISOLATED_INGESTION_KAFKA_CLUSTER_MAP_FILENAME
        See Also:
        Constant Field Values

      • FORKED_PROCESS_METADATA_FILENAME

        public static final java.lang.String FORKED_PROCESS_METADATA_FILENAME
        See Also:
        Constant Field Values

    • Constructor Detail

      • IsolatedIngestionUtils

        public IsolatedIngestionUtils()

    • Method Detail

      • serializeIngestionActionRequest

        public static <T extends org.apache.avro.specific.SpecificRecordBase> byte[] serializeIngestionActionRequest​(IngestionAction action,
                                                                                                             T param)

      • deserializeIngestionActionRequest

        public static <T extends org.apache.avro.specific.SpecificRecordBase> T deserializeIngestionActionRequest​(IngestionAction action,
                                                                                                          byte[] content)

      • serializeIngestionActionResponse

        public static <T extends org.apache.avro.specific.SpecificRecordBase> byte[] serializeIngestionActionResponse​(IngestionAction action,
                                                                                                              T param)

      • deserializeIngestionActionResponse

        public static <T extends org.apache.avro.specific.SpecificRecordBase> T deserializeIngestionActionResponse​(IngestionAction action,
                                                                                                           byte[] content)

      • getDummyContent

        public static byte[] getDummyContent()

      • serializeStoreVersionState

        public static byte[] serializeStoreVersionState​(java.lang.String topicName,
                                                StoreVersionState storeVersionState)

      • deserializeStoreVersionState

        public static StoreVersionState deserializeStoreVersionState​(java.lang.String topicName,
                                                             byte[] content)

      • buildHttpResponse

        public static io.netty.handler.codec.http.HttpResponse buildHttpResponse​(io.netty.handler.codec.http.HttpResponseStatus status,
                                                                         java.lang.String msg)

      • buildHttpResponse

        public static io.netty.handler.codec.http.HttpResponse buildHttpResponse​(io.netty.handler.codec.http.HttpResponseStatus status,
                                                                         byte[] content)

      • buildHttpResponse

        public static io.netty.handler.codec.http.HttpResponse buildHttpResponse​(io.netty.handler.codec.http.HttpResponseStatus status,
                                                                         io.netty.buffer.ByteBuf contentBuf)

      • readHttpRequestContent

        public static byte[] readHttpRequestContent​(io.netty.handler.codec.http.FullHttpRequest response)

      • getIngestionActionFromRequest

        public static IngestionAction getIngestionActionFromRequest​(io.netty.handler.codec.http.HttpRequest req)

      • waitPortBinding

        public static void waitPortBinding​(int port,
                                   int maxAttempt)
                            throws java.lang.Exception
        waitPortBinding is used to test server port binding in other process. Since we cannot control the connection setup on other process, we can only test by trying to establish a connection to the target port.
        Parameters:
        port - Target port to test connection.
        maxAttempt - Max number of connection retries before it announces fail to connect.
        Throws:
        java.lang.Exception

      • releaseTargetPortBinding

        public static void releaseTargetPortBinding​(int port)
        releaseTargetPortBinding aims to release the target port by killing lingering ingestion process bound to the port, which is created from previous deployment and was not killed due to unexpected failures.

      • getLingeringIngestionProcessId

        public static java.util.Optional<java.lang.Integer> getLingeringIngestionProcessId​(int port)
        This method returns lingering forked ingestion process PID if it exists. Since the lsof command will return all processes associated with the port number(including main process), we will need to iterate all PIDs and filter out ingestion process by name.

      • executeShellCommand

        public static java.lang.String executeShellCommand​(java.lang.String command)

      • createIngestionTaskReport

        public static IngestionTaskReport createIngestionTaskReport​(IngestionReportType ingestionReportType,
                                                            java.lang.String kafkaTopic,
                                                            int partitionId,
                                                            long offset,
                                                            java.lang.String message)

      • createIngestionTaskReport

        public static IngestionTaskReport createIngestionTaskReport​(java.lang.String kafkaTopic,
                                                            int partitionId)

      • destroyIsolatedIngestionProcess

        public static void destroyIsolatedIngestionProcess​(java.lang.Process isolatedIngestionServiceProcess)

      • destroyIsolatedIngestionProcessByPid

        public static void destroyIsolatedIngestionProcessByPid​(long pid)
        Kill isolated ingestion process by provided PID.

      • destroyLingeringIsolatedIngestionProcess

        public static void destroyLingeringIsolatedIngestionProcess​(VeniceConfigLoader configLoader)
        This method takes two steps to kill any lingering forked ingestion process previously created by the same user. It first tries to locate the PID of the lingering forked ingestion process stored in the metadata file. It then uses lsof command to locate isolated ingestion process binding to the same port and kill it.

      • createIngestionTaskReport

        public static IngestionTaskReport createIngestionTaskReport​(IngestionReportType ingestionReportType,
                                                            java.lang.String kafkaTopic,
                                                            int partitionId,
                                                            java.lang.String message)

      • buildAndSaveConfigsForForkedIngestionProcess

        public static java.lang.String buildAndSaveConfigsForForkedIngestionProcess​(VeniceConfigLoader configLoader)

      • saveForkedIngestionKafkaClusterMapConfig

        public static void saveForkedIngestionKafkaClusterMapConfig​(VeniceConfigLoader configLoader)

      • loadForkedIngestionKafkaClusterMapConfig

        public static java.util.Map<java.lang.String,​java.util.Map<java.lang.String,​java.lang.String>> loadForkedIngestionKafkaClusterMapConfig​(java.lang.String configBasePath)

      • loadVenicePropertiesFromFile

        public static VeniceProperties loadVenicePropertiesFromFile​(java.lang.String configPath)
                                                     throws java.io.FileNotFoundException
        Throws:
        java.io.FileNotFoundException

      • loadVenicePropertiesFromFile

        public static VeniceProperties loadVenicePropertiesFromFile​(java.lang.String basePath,
                                                            java.lang.String fileName)
                                                     throws java.io.FileNotFoundException
        Throws:
        java.io.FileNotFoundException

      • getSSLFactoryForIngestion

        public static java.util.Optional<SSLFactory> getSSLFactoryForIngestion​(VeniceConfigLoader configLoader)
        Create SSLFactory for D2Client in ClientConfig, which will be used by different ingestion components.

      • isolatedIngestionServerSslEnabled

        public static boolean isolatedIngestionServerSslEnabled​(VeniceConfigLoader configLoader)

      • isolatedIngestionServerAclEnabled

        public static boolean isolatedIngestionServerAclEnabled​(VeniceConfigLoader configLoader)