Package com.linkedin.davinci.ingestion.utils

Class IsolatedIngestionUtils

java.lang.Object
com.linkedin.davinci.ingestion.utils.IsolatedIngestionUtils
public class IsolatedIngestionUtils extends Object
This class contains methods used for communication between ingestion client and server.

  • Field Details

  • Constructor Details

    • IsolatedIngestionUtils

      public IsolatedIngestionUtils()

  • Method Details

    • serializeIngestionActionRequest

      public static <T extends org.apache.avro.specific.SpecificRecordBase> byte[] serializeIngestionActionRequest(IngestionAction action, T param)

    • deserializeIngestionActionRequest

      public static <T extends org.apache.avro.specific.SpecificRecordBase> T deserializeIngestionActionRequest(IngestionAction action, byte[] content)

    • serializeIngestionActionResponse

      public static <T extends org.apache.avro.specific.SpecificRecordBase> byte[] serializeIngestionActionResponse(IngestionAction action, T param)

    • deserializeIngestionActionResponse

      public static <T extends org.apache.avro.specific.SpecificRecordBase> T deserializeIngestionActionResponse(IngestionAction action, byte[] content)

    • getDummyCommand

      public static IngestionTaskCommand getDummyCommand()

    • getDummyContent

      public static byte[] getDummyContent()

    • serializeStoreVersionState

      public static byte[] serializeStoreVersionState(String topicName, StoreVersionState storeVersionState)

    • deserializeStoreVersionState

      public static StoreVersionState deserializeStoreVersionState(String topicName, byte[] content)

    • buildHttpResponse

      public static io.netty.handler.codec.http.HttpResponse buildHttpResponse(io.netty.handler.codec.http.HttpResponseStatus status, String msg)

    • buildHttpResponse

      public static io.netty.handler.codec.http.HttpResponse buildHttpResponse(io.netty.handler.codec.http.HttpResponseStatus status, byte[] content)

    • buildHttpResponse

      public static io.netty.handler.codec.http.HttpResponse buildHttpResponse(io.netty.handler.codec.http.HttpResponseStatus status, io.netty.buffer.ByteBuf contentBuf)

    • readHttpRequestContent

      public static byte[] readHttpRequestContent(io.netty.handler.codec.http.FullHttpRequest response)

    • getIngestionActionFromRequest

      public static IngestionAction getIngestionActionFromRequest(io.netty.handler.codec.http.HttpRequest req)

    • waitPortBinding

      public static void waitPortBinding(int port, int maxAttempt) throws Exception
      waitPortBinding is used to test server port binding in other process. Since we cannot control the connection setup on other process, we can only test by trying to establish a connection to the target port.
      Parameters:
      port - Target port to test connection.
      maxAttempt - Max number of connection retries before it announces fail to connect.
      Throws:
      Exception

    • releaseTargetPortBinding

      public static void releaseTargetPortBinding(int port)
      releaseTargetPortBinding aims to release the target port by killing lingering ingestion process bound to the port, which is created from previous deployment and was not killed due to unexpected failures.

    • getLingeringIngestionProcessId

      public static Optional<Integer> getLingeringIngestionProcessId(int port)
      This method returns lingering forked ingestion process PID if it exists. Since the lsof command will return all processes associated with the port number(including main process), we will need to iterate all PIDs and filter out ingestion process by name.

    • executeShellCommand

      public static String executeShellCommand(String command)

    • createIngestionTaskReport

      public static IngestionTaskReport createIngestionTaskReport(IngestionReportType ingestionReportType, String kafkaTopic, int partitionId, long offset, String message)

    • createIngestionTaskReport

      public static IngestionTaskReport createIngestionTaskReport(String kafkaTopic, int partitionId)

    • destroyIsolatedIngestionProcess

      public static void destroyIsolatedIngestionProcess(Process isolatedIngestionServiceProcess)

    • destroyIsolatedIngestionProcessByPid

      public static void destroyIsolatedIngestionProcessByPid(long pid)
      Kill isolated ingestion process by provided PID.

    • destroyLingeringIsolatedIngestionProcess

      public static void destroyLingeringIsolatedIngestionProcess(VeniceConfigLoader configLoader)
      This method takes two steps to kill any lingering forked ingestion process previously created by the same user. It first tries to locate the PID of the lingering forked ingestion process stored in the metadata file. It then uses lsof command to locate isolated ingestion process binding to the same port and kill it.

    • createIngestionTaskReport

      public static IngestionTaskReport createIngestionTaskReport()

    • createIngestionTaskReport

      public static IngestionTaskReport createIngestionTaskReport(IngestionReportType ingestionReportType, String kafkaTopic, int partitionId, String message)

    • buildAndSaveConfigsForForkedIngestionProcess

      public static String buildAndSaveConfigsForForkedIngestionProcess(VeniceConfigLoader configLoader)

    • saveForkedIngestionKafkaClusterMapConfig

      public static void saveForkedIngestionKafkaClusterMapConfig(VeniceConfigLoader configLoader)

    • loadForkedIngestionKafkaClusterMapConfig

      public static Map<String,Map<String,String>> loadForkedIngestionKafkaClusterMapConfig(String configBasePath)

    • saveForkedIngestionProcessMetadata

      public static void saveForkedIngestionProcessMetadata(VeniceConfigLoader configLoader, ForkedJavaProcess forkedJavaProcess)

    • loadVenicePropertiesFromFile

      public static VeniceProperties loadVenicePropertiesFromFile(String configPath) throws FileNotFoundException
      Throws:
      FileNotFoundException

    • loadVenicePropertiesFromFile

      public static VeniceProperties loadVenicePropertiesFromFile(String basePath, String fileName) throws FileNotFoundException
      Throws:
      FileNotFoundException

    • getSSLFactory

      public static Optional<SSLFactory> getSSLFactory(VeniceConfigLoader configLoader)

    • getSSLFactoryForIngestion

      public static Optional<SSLFactory> getSSLFactoryForIngestion(VeniceConfigLoader configLoader)
      Create SSLFactory for D2Client in ClientConfig, which will be used by different ingestion components.

    • getAclHandler

      public static Optional<IsolatedIngestionServerAclHandler> getAclHandler(VeniceConfigLoader configLoader)

    • isolatedIngestionServerSslEnabled

      public static boolean isolatedIngestionServerSslEnabled(VeniceConfigLoader configLoader)

    • isolatedIngestionServerAclEnabled

      public static boolean isolatedIngestionServerAclEnabled(VeniceConfigLoader configLoader)

    • sslEnabled

      public static boolean sslEnabled(VeniceConfigLoader configLoader)