FsVolumeImpl (Apache Hadoop HDFS 3.5.0 API)
Description: declaration: package: org.apache.hadoop.hdfs.server.datanode.fsdataset.impl, class: FsVolumeImpl

Package org.apache.hadoop.hdfs.server.datanode.fsdataset.impl

# Class FsVolumeImpl

java.lang.Object
org.apache.hadoop.hdfs.server.datanode.fsdataset.impl.FsVolumeImpl

All Implemented Interfaces:
  `Checkable<FsVolumeSpi.VolumeCheckContext,VolumeCheckResult>`, `FsVolumeSpi`

@Private @VisibleForTesting public class FsVolumeImpl extends Object implements FsVolumeSpi
The underlying volume used to store replica. It uses the `FsDatasetImpl` object for synchronization.

- ## Nested Class Summary

  Nested Classes
  Modifier and Type
  Class
  Description
  `static enum`
  `FsVolumeImpl.BlockDirFilter`
  Filter for block file names stored on the file system volumes.

  ## Nested classes/interfaces inherited from interface org.apache.hadoop.hdfs.server.datanode.fsdataset.FsVolumeSpi

  `FsVolumeSpi.BlockIterator, FsVolumeSpi.ScanInfo, FsVolumeSpi.VolumeCheckContext`

- ## Field Summary

  Fields
  Modifier and Type
  Field
  Description
  `protected ThreadPoolExecutor`
  `cacheExecutor`
  Per-volume worker pool that processes new blocks to cache.
  `protected long`
  `configuredCapacity`
  `static final org.slf4j.Logger`
  `LOG`

- ## Method Summary

  Modifier and Type
  Method
  Description
  `ReplicaInfo`
  `activateSavedReplica(String bpid, ReplicaInfo replicaInfo, org.apache.hadoop.hdfs.server.datanode.fsdataset.impl.RamDiskReplicaTracker.RamDiskReplica replicaState)`
  `ReplicaInPipeline`
  `append(String bpid, ReplicaInfo replicaInfo, long newGS, long estimateBlockLen)`
  `VolumeCheckResult`
  `check(FsVolumeSpi.VolumeCheckContext ignored)`
  Query the health of this object.
  `void`
  `compileReport(String bpid, Collection<FsVolumeSpi.ScanInfo> report, DirectoryScanner.ReportCompiler reportCompiler)`
  Compile a list of `FsVolumeSpi.ScanInfo` for the blocks in the block pool with id `bpid`.
  `ReplicaInPipeline`
  `convertTemporaryToRbw(org.apache.hadoop.hdfs.protocol.ExtendedBlock b, ReplicaInfo temp)`
  `File[]`
  `copyBlockToLazyPersistLocation(String bpId, long blockId, long genStamp, ReplicaInfo replicaInfo, int smallBufferSize, org.apache.hadoop.conf.Configuration conf)`
  `ReplicaInPipeline`
  `createRbw(org.apache.hadoop.hdfs.protocol.ExtendedBlock b)`
  `ReplicaInPipeline`
  `createTemporary(org.apache.hadoop.hdfs.protocol.ExtendedBlock b)`
  `long`
  `getAvailable()`
  Calculate the available space of the filesystem, excluding space reserved for non-HDFS and space reserved for RBW.
  `URI`
  `getBaseURI()`
  `String[]`
  `getBlockPoolList()`
  Make a deep copy of the list of currently active BPIDs.
  `Map<String,BlockPoolSlice>`
  `getBlockPoolSlices()`
  `long`
  `getCapacity()`
  Return either the configured capacity of the file system if configured; or the capacity of the file system excluding space reserved for non-HDFS.
  `File`
  `getCurrentDir()`
  `FsDatasetSpi<? extends FsVolumeSpi>`
  `getDataset()`
  Get the FSDatasetSpi which this volume is a part of.
  `long`
  `getDfsUsed()`
  `long`
  `getDfUsed()`
  This function is only used for Mock.
  `FileIoProvider`
  `getFileIoProvider()`
  `File`
  `getFinalizedDir(String bpid)`
  `protected File`
  `getLazyPersistDir(String bpid)`
  `DataNodeVolumeMetrics`
  `getMetrics()`
  `long`
  `getNonDfsUsed()`
  Unplanned Non-DFS usage, i.e.
  `protected File`
  `getRbwDir(String bpid)`
  `int`
  `getReferenceCount()`
  `long`
  `getReservedForReplicas()`
  `String`
  `getStorageID()`
  `StorageLocation`
  `getStorageLocation()`
  `org.apache.hadoop.fs.StorageType`
  `getStorageType()`
  `protected File`
  `getTmpDir(String bpid)`
  `org.apache.hadoop.fs.DF`
  `getUsageStats(org.apache.hadoop.conf.Configuration conf)`
  `ReplicaInfo`
  `hardLinkBlockToTmpLocation(org.apache.hadoop.hdfs.protocol.ExtendedBlock block, ReplicaInfo replicaInfo)`
  `void`
  `incrNumBlocks(String bpid)`
  `protected ThreadPoolExecutor`
  `initializeCacheExecutor(File parent)`
  `boolean`
  `isRAMStorage()`
  Returns true if the volume is backed by RAM storage.
  `boolean`
  `isTransientStorage()`
  Returns true if the volume is NOT backed by persistent storage.
  `FsVolumeSpi.BlockIterator`
  `loadBlockIterator(String bpid, String name)`
  Load a saved block iterator.
  `byte[]`
  `loadLastPartialChunkChecksum(File blockFile, File metaFile)`
  Load last partial chunk checksum from checksum file.
  `ReplicaInfo`
  `moveBlockToTmpLocation(org.apache.hadoop.hdfs.protocol.ExtendedBlock block, ReplicaInfo replicaInfo, int smallBufferSize, org.apache.hadoop.conf.Configuration conf)`
  `FsVolumeSpi.BlockIterator`
  `newBlockIterator(String bpid, String name)`
  Create a new block iterator.
  `static String`
  `nextSorted(List<String> arr, String prev)`
  `FsVolumeReference`
  `obtainReference()`
  Obtain a reference object that had increased 1 reference count of the volume.
  `void`
  `releaseLockedMemory(long bytesToRelease)`
  Release reserved memory for an RBW block written to transient storage i.e.
  `void`
  `releaseReservedSpace(long bytesToRelease)`
  Release disk space previously reserved for block opened for write.
  `void`
  `reserveSpaceForReplica(long bytesToReserve)`
  Reserve disk space for a block (RBW or Re-replicating) so a writer does not run out of space before the block is full.
  `void`
  `resolveDuplicateReplicas(String bpid, ReplicaInfo memBlockInfo, ReplicaInfo diskBlockInfo, org.apache.hadoop.hdfs.server.datanode.fsdataset.impl.ReplicaMap volumeMap)`
  `void`
  `setCapacityForTesting(long capacity)`
  This function MUST NOT be used outside of tests.
  `String`
  `toString()`
  `ReplicaInPipeline`
  `updateRURCopyOnTruncate(ReplicaInfo rur, String bpid, long newBlockId, long recoveryId, long newlength)`

  ### Methods inherited from class java.lang.Object

  `clone, equals, finalize, getClass, hashCode, notify, notifyAll, wait, wait, wait`

- ## Field Details

  - ### LOG

    public static final org.slf4j.Logger LOG

  - ### configuredCapacity

    protected volatile long configuredCapacity

  - ### cacheExecutor

    protected ThreadPoolExecutor cacheExecutor
    Per-volume worker pool that processes new blocks to cache. The maximum number of workers per volume is bounded (configurable via dfs.datanode.fsdatasetcache.max.threads.per.volume) to limit resource contention.

- ## Method Details

  - ### initializeCacheExecutor

    protected ThreadPoolExecutor initializeCacheExecutor(File parent)

  - ### obtainReference

    public FsVolumeReference obtainReference() throws ClosedChannelException
    Description copied from interface: `FsVolumeSpi`
    Obtain a reference object that had increased 1 reference count of the volume. It is caller's responsibility to close `FsVolumeReference` to decrease the reference count on the volume.
    Specified by:
      `obtainReference` in interface `FsVolumeSpi`
    Throws:
      `ClosedChannelException`

  - ### getReferenceCount

    @VisibleForTesting public int getReferenceCount()

  - ### getCurrentDir

    @VisibleForTesting public File getCurrentDir()

  - ### getRbwDir

    protected File getRbwDir(String bpid) throws IOException
    Throws:
      `IOException`

  - ### getLazyPersistDir

    protected File getLazyPersistDir(String bpid) throws IOException
    Throws:
      `IOException`

  - ### getTmpDir

    protected File getTmpDir(String bpid) throws IOException
    Throws:
      `IOException`

  - ### getDfsUsed

    @VisibleForTesting public long getDfsUsed() throws IOException
    Throws:
      `IOException`

  - ### getCapacity

    @VisibleForTesting public long getCapacity()
    Return either the configured capacity of the file system if configured; or the capacity of the file system excluding space reserved for non-HDFS. When same-disk-tiering is turned on, the reported capacity will take reservedForArchive value into consideration of.
    Returns:
      the unreserved number of bytes left in this filesystem. May be zero.

  - ### setCapacityForTesting

    @VisibleForTesting public void setCapacityForTesting(long capacity)
    This function MUST NOT be used outside of tests.
    Parameters:
      `capacity` -

  - ### getAvailable

    public long getAvailable() throws IOException
    Calculate the available space of the filesystem, excluding space reserved for non-HDFS and space reserved for RBW.
    Specified by:
      `getAvailable` in interface `FsVolumeSpi`
    Returns:
      the available number of bytes left in this filesystem. May be zero.
    Throws:
      `IOException`

  - ### getDfUsed

    @VisibleForTesting public long getDfUsed()
    This function is only used for Mock.

  - ### getNonDfsUsed

    public long getNonDfsUsed() throws IOException
    Unplanned Non-DFS usage, i.e. Extra usage beyond reserved.
    Returns:
      Disk usage excluding space used by HDFS and excluding space reserved for blocks open for write.
    Throws:
      `IOException`

  - ### getReservedForReplicas

    @VisibleForTesting public long getReservedForReplicas()

  - ### getBlockPoolSlices

    public Map<String,BlockPoolSlice> getBlockPoolSlices()

  - ### getBaseURI

    public URI getBaseURI()
    Specified by:
      `getBaseURI` in interface `FsVolumeSpi`
    Returns:
      the base path to the volume

  - ### getUsageStats

    public org.apache.hadoop.fs.DF getUsageStats(org.apache.hadoop.conf.Configuration conf)
    Specified by:
      `getUsageStats` in interface `FsVolumeSpi`

  - ### getStorageLocation

    public StorageLocation getStorageLocation()
    Specified by:
      `getStorageLocation` in interface `FsVolumeSpi`
    Returns:
      the `StorageLocation` to the volume

  - ### isTransientStorage

    public boolean isTransientStorage()
    Description copied from interface: `FsVolumeSpi`
    Returns true if the volume is NOT backed by persistent storage.
    Specified by:
      `isTransientStorage` in interface `FsVolumeSpi`

  - ### isRAMStorage

    public boolean isRAMStorage()
    Description copied from interface: `FsVolumeSpi`
    Returns true if the volume is backed by RAM storage.
    Specified by:
      `isRAMStorage` in interface `FsVolumeSpi`

  - ### getFinalizedDir

    @VisibleForTesting public File getFinalizedDir(String bpid) throws IOException
    Throws:
      `IOException`

  - ### getBlockPoolList

    public String[] getBlockPoolList()
    Make a deep copy of the list of currently active BPIDs.
    Specified by:
      `getBlockPoolList` in interface `FsVolumeSpi`
    Returns:
      a list of block pools.

  - ### reserveSpaceForReplica

    public void reserveSpaceForReplica(long bytesToReserve)
    Description copied from interface: `FsVolumeSpi`
    Reserve disk space for a block (RBW or Re-replicating) so a writer does not run out of space before the block is full.
    Specified by:
      `reserveSpaceForReplica` in interface `FsVolumeSpi`

  - ### releaseReservedSpace

    public void releaseReservedSpace(long bytesToRelease)
    Description copied from interface: `FsVolumeSpi`
    Release disk space previously reserved for block opened for write.
    Specified by:
      `releaseReservedSpace` in interface `FsVolumeSpi`

  - ### releaseLockedMemory

    public void releaseLockedMemory(long bytesToRelease)
    Description copied from interface: `FsVolumeSpi`
    Release reserved memory for an RBW block written to transient storage i.e. RAM. bytesToRelease will be rounded down to the OS page size since locked memory reservation must always be a multiple of the page size.
    Specified by:
      `releaseLockedMemory` in interface `FsVolumeSpi`

  - ### nextSorted

    @VisibleForTesting public static String nextSorted(List<String> arr, String prev)

  - ### newBlockIterator

    public FsVolumeSpi.BlockIterator newBlockIterator(String bpid, String name)
    Description copied from interface: `FsVolumeSpi`
    Create a new block iterator. It will start at the beginning of the block set.
    Specified by:
      `newBlockIterator` in interface `FsVolumeSpi`
    Parameters:
      `bpid` - The block pool id to iterate over.
      `name` - The name of the block iterator to create.
    Returns:
      The new block iterator.

  - ### loadBlockIterator

    public FsVolumeSpi.BlockIterator loadBlockIterator(String bpid, String name) throws IOException
    Description copied from interface: `FsVolumeSpi`
    Load a saved block iterator.
    Specified by:
      `loadBlockIterator` in interface `FsVolumeSpi`
    Parameters:
      `bpid` - The block pool id to iterate over.
      `name` - The name of the block iterator to load.
    Returns:
      The saved block iterator.
    Throws:
      `IOException` - If there was an IO error loading the saved block iterator.

  - ### getDataset

    public FsDatasetSpi<? extends FsVolumeSpi> getDataset()
    Description copied from interface: `FsVolumeSpi`
    Get the FSDatasetSpi which this volume is a part of.
    Specified by:
      `getDataset` in interface `FsVolumeSpi`

  - ### check

    public VolumeCheckResult check(FsVolumeSpi.VolumeCheckContext ignored) throws org.apache.hadoop.util.DiskChecker.DiskErrorException
    Description copied from interface: `Checkable`
    Query the health of this object. This method may hang indefinitely depending on the status of the target resource.
    Specified by:
      `check` in interface `Checkable<FsVolumeSpi.VolumeCheckContext,VolumeCheckResult>`
    Parameters:
      `ignored` - for the probe operation. May be null depending on the implementation.
    Returns:
      result of the check operation.
    Throws:
      `org.apache.hadoop.util.DiskChecker.DiskErrorException`

  - ### toString

    public String toString()
    Overrides:
      `toString` in class `Object`

  - ### getStorageID

    public String getStorageID()
    Specified by:
      `getStorageID` in interface `FsVolumeSpi`
    Returns:
      the StorageUuid of the volume

  - ### getStorageType

    public org.apache.hadoop.fs.StorageType getStorageType()
    Specified by:
      `getStorageType` in interface `FsVolumeSpi`
    Returns:
      the `StorageType` of the volume

  - ### loadLastPartialChunkChecksum

    public byte[] loadLastPartialChunkChecksum(File blockFile, File metaFile) throws IOException
    Description copied from interface: `FsVolumeSpi`
    Load last partial chunk checksum from checksum file. Need to be called with FsDataset lock acquired.
    Specified by:
      `loadLastPartialChunkChecksum` in interface `FsVolumeSpi`
    Returns:
      the last partial checksum
    Throws:
      `IOException`

  - ### append

    public ReplicaInPipeline append(String bpid, ReplicaInfo replicaInfo, long newGS, long estimateBlockLen) throws IOException
    Throws:
      `IOException`

  - ### createRbw

    public ReplicaInPipeline createRbw(org.apache.hadoop.hdfs.protocol.ExtendedBlock b) throws IOException
    Throws:
      `IOException`

  - ### convertTemporaryToRbw

    public ReplicaInPipeline convertTemporaryToRbw(org.apache.hadoop.hdfs.protocol.ExtendedBlock b, ReplicaInfo temp) throws IOException
    Throws:
      `IOException`

  - ### createTemporary

    public ReplicaInPipeline createTemporary(org.apache.hadoop.hdfs.protocol.ExtendedBlock b) throws IOException
    Throws:
      `IOException`

  - ### updateRURCopyOnTruncate

    public ReplicaInPipeline updateRURCopyOnTruncate(ReplicaInfo rur, String bpid, long newBlockId, long recoveryId, long newlength) throws IOException
    Throws:
      `IOException`

  - ### compileReport

    public void compileReport(String bpid, Collection<FsVolumeSpi.ScanInfo> report, DirectoryScanner.ReportCompiler reportCompiler) throws InterruptedException, IOException
    Description copied from interface: `FsVolumeSpi`
    Compile a list of `FsVolumeSpi.ScanInfo` for the blocks in the block pool with id `bpid`.
    Specified by:
      `compileReport` in interface `FsVolumeSpi`
    Parameters:
      `bpid` - block pool id to scan
      `report` - the list onto which blocks reports are placed
    Throws:
      `InterruptedException`
      `IOException`

  - ### getFileIoProvider

    public FileIoProvider getFileIoProvider()
    Specified by:
      `getFileIoProvider` in interface `FsVolumeSpi`

  - ### getMetrics

    public DataNodeVolumeMetrics getMetrics()
    Specified by:
      `getMetrics` in interface `FsVolumeSpi`

  - ### moveBlockToTmpLocation

    public ReplicaInfo moveBlockToTmpLocation(org.apache.hadoop.hdfs.protocol.ExtendedBlock block, ReplicaInfo replicaInfo, int smallBufferSize, org.apache.hadoop.conf.Configuration conf) throws IOException
    Throws:
      `IOException`

  - ### hardLinkBlockToTmpLocation

    public ReplicaInfo hardLinkBlockToTmpLocation(org.apache.hadoop.hdfs.protocol.ExtendedBlock block, ReplicaInfo replicaInfo) throws IOException
    Throws:
      `IOException`

  - ### copyBlockToLazyPersistLocation

    public File[] copyBlockToLazyPersistLocation(String bpId, long blockId, long genStamp, ReplicaInfo replicaInfo, int smallBufferSize, org.apache.hadoop.conf.Configuration conf) throws IOException
    Throws:
      `IOException`

  - ### incrNumBlocks

    public void incrNumBlocks(String bpid) throws IOException
    Throws:
      `IOException`

  - ### resolveDuplicateReplicas

    public void resolveDuplicateReplicas(String bpid, ReplicaInfo memBlockInfo, ReplicaInfo diskBlockInfo, org.apache.hadoop.hdfs.server.datanode.fsdataset.impl.ReplicaMap volumeMap) throws IOException
    Throws:
      `IOException`

  - ### activateSavedReplica

    public ReplicaInfo activateSavedReplica(String bpid, ReplicaInfo replicaInfo, org.apache.hadoop.hdfs.server.datanode.fsdataset.impl.RamDiskReplicaTracker.RamDiskReplica replicaState) throws IOException
    Throws:
      `IOException`