Package com.linkedin.davinci.store

Class AbstractStorageEngine<Partition extends AbstractStoragePartition>

java.lang.Object
com.linkedin.davinci.store.AbstractStorageEngine<Partition>
All Implemented Interfaces:
Closeable, AutoCloseable
Direct Known Subclasses:
BlackHoleStorageEngine, InMemoryStorageEngine, RocksDBStorageEngine, VeniceStoreCacheStorageEngine
public abstract class AbstractStorageEngine<Partition extends AbstractStoragePartition> extends Object implements Closeable
A base storage abstract class which is actually responsible for data persistence. This abstract class implies all the usual responsibilities of a Store implementation,

There are several proposals for storage-partition model:

1. One storage engine for all stores 1.1 One store uses one database, i.e. all partitions of the store will be in one database. 1.2 One store uses multiple databases, i.e. one partition per database. 2. Each store handled by one storage engine 2.1 All partitions of the store will be handled in one database (current Voldemort implementation) 2.2 One partition per database (Sudha suggests) 3. Each partition handled by one storage engine (original proposal before today’s discussion, super high overhead)

The point of having one storage engine(environment) or one database for one partition, is to simplify the complexity of rebalancing/partition migration/host swap. The team agreed to take (2.2) as default storage-partition model for now, and run performance tests to see if it goes well.

  • Field Details

  • Constructor Details

  • Method Details

    • getRWLockForPartitionOrThrow

      public ReadWriteLock getRWLockForPartitionOrThrow(int partitionId)
      Making it public is for testing purpose.

    • getStoreVersionName

      public String getStoreVersionName()

    • toString

      public String toString()
      Overrides:
      toString in class Object

    • getType

      public abstract PersistenceType getType()

    • getStoreSizeInBytes

      public abstract long getStoreSizeInBytes()

    • getCachedStoreSizeInBytes

      public long getCachedStoreSizeInBytes()

    • getRMDSizeInBytes

      public long getRMDSizeInBytes()

    • getCachedRMDSizeInBytes

      public long getCachedRMDSizeInBytes()

    • getPersistedPartitionIds

      public abstract Set<Integer> getPersistedPartitionIds()

    • createStoragePartition

      public abstract Partition createStoragePartition(StoragePartitionConfig partitionConfig)

    • restoreStoragePartitions

      protected void restoreStoragePartitions(boolean restoreMetadataPartition, boolean restoreDataPartitions)
      Load the existing storage partitions. The implementation should decide when to call this function properly to restore partitions.

    • restoreStoragePartitions

      protected final void restoreStoragePartitions()

    • getMetadataPartition

      public AbstractStoragePartition getMetadataPartition()

    • adjustStoragePartition

      public void adjustStoragePartition(int partitionId, AbstractStorageEngine.StoragePartitionAdjustmentTrigger mode, StoragePartitionConfig partitionConfig)
      Adjust the opened storage partition according to the provided storagePartitionConfig. It will throw exception if there is no opened storage partition for the given partition id. The reason to have is mainly used to ease the unit test.

    • addStoragePartition

      public void addStoragePartition(int partitionId)

    • addStoragePartition

      public void addStoragePartition(StoragePartitionConfig storagePartitionConfig)

    • closePartition

      public void closePartition(int partitionId)

    • closeMetadataPartition

      public void closeMetadataPartition()

    • dropPartition

      public void dropPartition(int partitionId)
      Removes and returns a partition from the current store
      Parameters:
      partitionId - - id of partition to retrieve and remove

    • dropMetadataPartition

      public void dropMetadataPartition()

    • drop

      public void drop()
      Drop the whole store

    • sync

      public Map<String,String> sync(int partitionId)

    • close

      public void close() throws VeniceException
      Specified by:
      close in interface AutoCloseable
      Specified by:
      close in interface Closeable
      Throws:
      VeniceException

    • isClosed

      public boolean isClosed()

    • checkDatabaseIntegrity

      public boolean checkDatabaseIntegrity(int partitionId, Map<String,String> checkpointedInfo, StoragePartitionConfig storagePartitionConfig)
      checks whether the current state of the database is valid during the start of ingestion.

    • beginBatchWrite

      public void beginBatchWrite(StoragePartitionConfig storagePartitionConfig, Map<String,String> checkpointedInfo, Optional<Supplier<byte[]>> checksumSupplier)
      A lot of storage engines support efficient methods for performing large number of writes (puts/deletes) against the data source. This method puts the storage engine in this batch write mode

    • endBatchWrite

      public void endBatchWrite(StoragePartitionConfig storagePartitionConfig)

    • createSnapshot

      public void createSnapshot(StoragePartitionConfig storagePartitionConfig)
      Create snapshot for the given partition
      Parameters:
      storagePartitionConfig -

    • reopenStoragePartition

      public void reopenStoragePartition(int partitionId)
      Reopen the underlying database.

    • put

      public void put(int partitionId, byte[] key, byte[] value) throws VeniceException
      Throws:
      VeniceException

    • put

      public void put(int partitionId, byte[] key, ByteBuffer value) throws VeniceException
      Throws:
      VeniceException

    • putWithReplicationMetadata

      public void putWithReplicationMetadata(int partitionId, byte[] key, ByteBuffer value, byte[] replicationMetadata) throws VeniceException
      Throws:
      VeniceException

    • putReplicationMetadata

      public void putReplicationMetadata(int partitionId, byte[] key, byte[] replicationMetadata) throws VeniceException
      Throws:
      VeniceException

    • put

      public <K, V> void put(int partitionId, K key, V value)

    • get

      public byte[] get(int partitionId, byte[] key) throws VeniceException
      Throws:
      VeniceException

    • get

      public ByteBuffer get(int partitionId, byte[] key, ByteBuffer valueToBePopulated) throws VeniceException
      Throws:
      VeniceException

    • get

      public byte[] get(int partitionId, ByteBuffer keyBuffer) throws VeniceException
      Throws:
      VeniceException

    • getByKeyPrefix

      public void getByKeyPrefix(int partitionId, byte[] partialKey, BytesStreamingCallback bytesStreamingCallback)

    • delete

      public void delete(int partitionId, byte[] key) throws VeniceException
      Throws:
      VeniceException

    • deleteWithReplicationMetadata

      public void deleteWithReplicationMetadata(int partitionId, byte[] key, byte[] replicationMetadata) throws VeniceException
      Throws:
      VeniceException

    • getReplicationMetadata

      public byte[] getReplicationMetadata(int partitionId, ByteBuffer key)

    • putPartitionOffset

      public void putPartitionOffset(int partitionId, OffsetRecord offsetRecord)
      Put the offset associated with the partitionId into the metadata partition.

    • getPartitionOffset

      public Optional<OffsetRecord> getPartitionOffset(int partitionId)
      Retrieve the offset associated with the partitionId from the metadata partition.

    • clearPartitionOffset

      public void clearPartitionOffset(int partitionId)
      Clear the offset associated with the partitionId in the metadata partition.

    • isMetadataPartition

      public static boolean isMetadataPartition(int partitionId)

    • putStoreVersionState

      public void putStoreVersionState(StoreVersionState versionState)
      Put the store version state into the metadata partition.

    • updateStoreVersionStateCache

      public void updateStoreVersionStateCache(StoreVersionState versionState)
      Used in ingestion isolation mode update the storage engine's cache in sync with the updates to the state in MainIngestionStorageMetadataService

    • getStoreVersionState

      public StoreVersionState getStoreVersionState()
      Retrieve the store version state from the metadata partition.

    • clearStoreVersionState

      public void clearStoreVersionState()
      Clear the store version state in the metadata partition.

    • containsPartition

      public boolean containsPartition(int partitionId)
      Return true or false based on whether a given partition exists within this storage engine
      Parameters:
      partitionId - The partition to look for
      Returns:
      True/False, does the partition exist on this node

    • getNumberOfPartitions

      public long getNumberOfPartitions()
      A function which behaves like Map.size(), in the sense that it ignores empty (null) slots in the list.
      Returns:
      the number of non-null partitions in partitionList

    • getPartitionIds

      public Set<Integer> getPartitionIds()
      Get all Partition Ids which are assigned to the current Node.
      Returns:
      partition Ids that are hosted in the current Storage Engine.

    • getPartitionOrThrow

      public AbstractStoragePartition getPartitionOrThrow(int partitionId)

    • getPartitionSizeInBytes

      public long getPartitionSizeInBytes(int partitionId)

    • getCompressionStrategy

      public CompressionStrategy getCompressionStrategy()

    • isChunked

      public boolean isChunked()

    • suppressLogs

      public void suppressLogs(boolean suppressLogs)

    • hasMemorySpaceLeft

      public boolean hasMemorySpaceLeft()

    • getIterator

      public AbstractStorageIterator getIterator(int partitionId)