Package com.linkedin.venice.spark.chunk

Class SparkChunkAssembler

java.lang.Object

com.linkedin.venice.spark.chunk.SparkChunkAssembler

All Implemented Interfaces:

Serializable, AutoCloseable

public class SparkChunkAssembler
extends Object
implements Serializable, AutoCloseable

Spark adapter for ChunkAssembler that handles chunked values and RMDs. Converts Spark Rows to the format expected by ChunkAssembler, assembles chunks, and returns the result as a Spark Row.

See Also:

• Serialized Form

Constructor Summary

Constructors

Constructor	Description
SparkChunkAssembler(boolean isRmdChunkingEnabled)
SparkChunkAssembler(boolean isRmdChunkingEnabled, boolean isTTLFilteringEnabled, VeniceProperties filterProperties)

Method Summary

Modifier and Type	Method	Description
org.apache.spark.sql.Row	assembleChunks(byte[] keyBytes, Iterator<org.apache.spark.sql.Row> rows)	Assemble chunks for a single key.
void	close()	Releases resources (e.g.

Methods inherited from class java.lang.Object

clone, equals, finalize, getClass, hashCode, notify, notifyAll, toString, wait, wait, wait

Constructor Details

SparkChunkAssembler

public SparkChunkAssembler(boolean isRmdChunkingEnabled)

SparkChunkAssembler

public SparkChunkAssembler(boolean isRmdChunkingEnabled,
 boolean isTTLFilteringEnabled,
 VeniceProperties filterProperties)

Method Details

assembleChunks

public org.apache.spark.sql.Row assembleChunks(byte[] keyBytes,
 Iterator<org.apache.spark.sql.Row> rows)

Assemble chunks for a single key. If TTL filtering is enabled, also filters the assembled record.

Parameters:

keyBytes - The key bytes

rows - Iterator of rows for this key (MUST be sorted by offset DESC - highest offset first)

Returns:

Assembled row with DEFAULT_SCHEMA_WITH_SCHEMA_ID schema, or null if DELETE, orphan chunks (chunks with no corresponding manifest), or filtered by TTL

Throws:

RuntimeException - (e.g. VeniceException, IllegalArgumentException, IllegalStateException) if the assembled record is malformed - for example missing value/RMD chunks, a byte-count mismatch against the manifest, an offset-ordering violation, or an unrecognized schema id/message type. This intentionally mirrors VeniceKafkaInputReducer#extractChunkedMessage, which lets the same exceptions from ChunkAssembler#assembleAndGetValue propagate and fail the MR task rather than silently dropping the record.

close

public void close()

Releases resources (e.g. the post-assembly TTL filter's schema reader) held by this assembler. Safe to call even if the assembler was never used (transient fields still null).

Specified by:

close in interface AutoCloseable