Package com.linkedin.davinci.store

Class AbstractStorageEngine<Partition extends AbstractStoragePartition>

  • All Implemented Interfaces:
    java.io.Closeable, java.lang.AutoCloseable
    Direct Known Subclasses:
    BlackHoleStorageEngine, InMemoryStorageEngine, RocksDBStorageEngine, VeniceStoreCacheStorageEngine
    public abstract class AbstractStorageEngine<Partition extends AbstractStoragePartition>
extends java.lang.Object
implements java.io.Closeable
    A base storage abstract class which is actually responsible for data persistence. This abstract class implies all the usual responsibilities of a Store implementation,

    There are several proposals for storage-partition model:

    1. One storage engine for all stores 1.1 One store uses one database, i.e. all partitions of the store will be in one database. 1.2 One store uses multiple databases, i.e. one partition per database. 2. Each store handled by one storage engine 2.1 All partitions of the store will be handled in one database (current Voldemort implementation) 2.2 One partition per database (Sudha suggests) 3. Each partition handled by one storage engine (original proposal before today’s discussion, super high overhead)

    The point of having one storage engine(environment) or one database for one partition, is to simplify the complexity of rebalancing/partition migration/host swap. The team agreed to take (2.2) as default storage-partition model for now, and run performance tests to see if it goes well.

    • Field Detail

    • Method Detail

      • getRWLockForPartitionOrThrow

        public java.util.concurrent.locks.ReadWriteLock getRWLockForPartitionOrThrow​(int partitionId)
        Making it public is for testing purpose.

      • getStoreVersionName

        public java.lang.String getStoreVersionName()

      • toString

        public java.lang.String toString()
        Overrides:
        toString in class java.lang.Object

      • getStoreSizeInBytes

        public abstract long getStoreSizeInBytes()

      • getCachedStoreSizeInBytes

        public long getCachedStoreSizeInBytes()

      • getRMDSizeInBytes

        public long getRMDSizeInBytes()

      • getCachedRMDSizeInBytes

        public long getCachedRMDSizeInBytes()

      • getPersistedPartitionIds

        public abstract java.util.Set<java.lang.Integer> getPersistedPartitionIds()

      • restoreStoragePartitions

        protected void restoreStoragePartitions​(boolean restoreMetadataPartition,
                                        boolean restoreDataPartitions)
        Load the existing storage partitions. The implementation should decide when to call this function properly to restore partitions.

      • restoreStoragePartitions

        protected final void restoreStoragePartitions()

      • adjustStoragePartition

        public void adjustStoragePartition​(int partitionId,
                                   AbstractStorageEngine.StoragePartitionAdjustmentTrigger mode,
                                   StoragePartitionConfig partitionConfig)
        Adjust the opened storage partition according to the provided storagePartitionConfig. It will throw exception if there is no opened storage partition for the given partition id. The reason to have {@param partitionId} is mainly used to ease the unit test.

      • addStoragePartition

        public void addStoragePartition​(int partitionId)

      • closePartition

        public void closePartition​(int partitionId)

      • closeMetadataPartition

        public void closeMetadataPartition()

      • dropPartition

        public void dropPartition​(int partitionId)
        Removes and returns a partition from the current store
        Parameters:
        partitionId - - id of partition to retrieve and remove

      • dropMetadataPartition

        public void dropMetadataPartition()

      • drop

        public void drop()
        Drop the whole store

      • sync

        public java.util.Map<java.lang.String,​java.lang.String> sync​(int partitionId)

      • close

        public void close()
           throws VeniceException
        Specified by:
        close in interface java.lang.AutoCloseable
        Specified by:
        close in interface java.io.Closeable
        Throws:
        VeniceException

      • isClosed

        public boolean isClosed()

      • checkDatabaseIntegrity

        public boolean checkDatabaseIntegrity​(int partitionId,
                                      java.util.Map<java.lang.String,​java.lang.String> checkpointedInfo,
                                      StoragePartitionConfig storagePartitionConfig)
        checks whether the current state of the database is valid during the start of ingestion.

      • beginBatchWrite

        public void beginBatchWrite​(StoragePartitionConfig storagePartitionConfig,
                            java.util.Map<java.lang.String,​java.lang.String> checkpointedInfo,
                            java.util.Optional<java.util.function.Supplier<byte[]>> checksumSupplier)
        A lot of storage engines support efficient methods for performing large number of writes (puts/deletes) against the data source. This method puts the storage engine in this batch write mode

      • createSnapshot

        public void createSnapshot​(StoragePartitionConfig storagePartitionConfig)
        Create snapshot for the given partition
        Parameters:
        storagePartitionConfig -

      • reopenStoragePartition

        public void reopenStoragePartition​(int partitionId)
        Reopen the underlying database.

      • putWithReplicationMetadata

        public void putWithReplicationMetadata​(int partitionId,
                                       byte[] key,
                                       java.nio.ByteBuffer value,
                                       byte[] replicationMetadata)
                                throws VeniceException
        Throws:
        VeniceException

      • putReplicationMetadata

        public void putReplicationMetadata​(int partitionId,
                                   byte[] key,
                                   byte[] replicationMetadata)
                            throws VeniceException
        Throws:
        VeniceException

      • put

        public <K,​V> void put​(int partitionId,
                            K key,
                            V value)

      • get

        public java.nio.ByteBuffer get​(int partitionId,
                               byte[] key,
                               java.nio.ByteBuffer valueToBePopulated)
                        throws VeniceException
        Throws:
        VeniceException

      • getByKeyPrefix

        public void getByKeyPrefix​(int partitionId,
                           byte[] partialKey,
                           BytesStreamingCallback bytesStreamingCallback)

      • deleteWithReplicationMetadata

        public void deleteWithReplicationMetadata​(int partitionId,
                                          byte[] key,
                                          byte[] replicationMetadata)
                                   throws VeniceException
        Throws:
        VeniceException

      • getReplicationMetadata

        public byte[] getReplicationMetadata​(int partitionId,
                                     java.nio.ByteBuffer key)

      • putPartitionOffset

        public void putPartitionOffset​(int partitionId,
                               OffsetRecord offsetRecord)
        Put the offset associated with the partitionId into the metadata partition.

      • getPartitionOffset

        public java.util.Optional<OffsetRecord> getPartitionOffset​(int partitionId)
        Retrieve the offset associated with the partitionId from the metadata partition.

      • clearPartitionOffset

        public void clearPartitionOffset​(int partitionId)
        Clear the offset associated with the partitionId in the metadata partition.

      • isMetadataPartition

        public static boolean isMetadataPartition​(int partitionId)

      • putStoreVersionState

        public void putStoreVersionState​(StoreVersionState versionState)
        Put the store version state into the metadata partition.

      • getStoreVersionState

        public StoreVersionState getStoreVersionState()
        Retrieve the store version state from the metadata partition.

      • clearStoreVersionState

        public void clearStoreVersionState()
        Clear the store version state in the metadata partition.

      • containsPartition

        public boolean containsPartition​(int partitionId)
        Return true or false based on whether a given partition exists within this storage engine
        Parameters:
        partitionId - The partition to look for
        Returns:
        True/False, does the partition exist on this node

      • getNumberOfPartitions

        public long getNumberOfPartitions()
        A function which behaves like Map.size(), in the sense that it ignores empty (null) slots in the list.
        Returns:
        the number of non-null partitions in partitionList

      • getPartitionIds

        public java.util.Set<java.lang.Integer> getPartitionIds()
        Get all Partition Ids which are assigned to the current Node.
        Returns:
        partition Ids that are hosted in the current Storage Engine.

      • getPartitionSizeInBytes

        public long getPartitionSizeInBytes​(int partitionId)

      • isChunked

        public boolean isChunked()

      • suppressLogs

        public void suppressLogs​(boolean suppressLogs)

      • hasMemorySpaceLeft

        public boolean hasMemorySpaceLeft()