Storage (Apache Hadoop HDFS 3.5.0 API)
Description: declaration: package: org.apache.hadoop.hdfs.server.common, class: Storage

Package org.apache.hadoop.hdfs.server.common

# Class Storage

java.lang.Object
org.apache.hadoop.hdfs.server.common.StorageInfo
org.apache.hadoop.hdfs.server.common.Storage

Direct Known Subclasses:
  `BlockPoolSliceStorage`, `DataStorage`, `NNStorage`

@Private public abstract class Storage extends StorageInfo
Storage information file.

Local storage information is stored in a separate file VERSION. It contains type of the node, the storage layout version, the namespace id, and the fs state creation time.

Local storage can reside in multiple directories. Each directory should contain the same VERSION file as the others. During startup Hadoop servers (name-node and data-nodes) read their local storage information from them.

The servers hold a lock for each storage directory while they run so that other nodes were not able to startup sharing the same storage. The locks are released when the servers stop (normally or abnormally).

- ## Nested Class Summary

  Nested Classes
  Modifier and Type
  Class
  Description
  `static interface`
  `Storage.FormatConfirmable`
  Interface for classes which need to have the user confirm their formatting during NameNode -format and other similar operations.
  `static class`
  `Storage.StorageDirectory`
  One of the storage directories.
  `static interface`
  `Storage.StorageDirType`
  An interface to denote storage directory type Implementations can define a type for storage directory by implementing this interface.
  `static enum`
  `Storage.StorageState`

- ## Field Summary

  Fields
  Modifier and Type
  Field
  Description
  `static final int`
  `LAST_PRE_UPGRADE_LAYOUT_VERSION`
  `protected static final String`
  `LAST_UPGRADABLE_HADOOP_VERSION`
  `static final int`
  `LAST_UPGRADABLE_LAYOUT_VERSION`
  `static final int[]`
  `LAYOUT_VERSIONS_203`
  Layout versions of 0.20.203 release
  `static final org.slf4j.Logger`
  `LOG`
  `static final String`
  `STORAGE_1_BBW`
  The blocksBeingWritten directory which was used in some 1.x and earlier releases.
  `static final String`
  `STORAGE_DIR_CURRENT`
  `static final String`
  `STORAGE_DIR_PREVIOUS`
  `static final String`
  `STORAGE_FILE_LOCK`
  `static final String`
  `STORAGE_PREVIOUS_CKPT`
  `static final String`
  `STORAGE_TMP_FINALIZED`
  `static final String`
  `STORAGE_TMP_LAST_CKPT`
  `static final String`
  `STORAGE_TMP_PREVIOUS`
  `static final String`
  `STORAGE_TMP_REMOVED`

  ### Fields inherited from class org.apache.hadoop.hdfs.server.common.StorageInfo

  `clusterID, cTime, layoutVersion, namespaceID, STORAGE_FILE_VERSION, storageType`

- ## Constructor Summary

  Constructors
  Modifier
  Constructor
  Description
  `protected`
  `Storage(HdfsServerConstants.NodeType type)`
  Create empty storage info of the specified type
  `protected`
  `Storage(StorageInfo storageInfo)`

- ## Method Summary

  Modifier and Type
  Method
  Description
  `protected void`
  `addStorageDir(Storage.StorageDirectory sd)`
  `static void`
  `checkVersionUpgradable(int oldVersion)`
  Checks if the upgrade from `oldVersion` is supported.
  `static boolean`
  `confirmFormat(Iterable<? extends Storage.FormatConfirmable> items, boolean force, boolean interactive)`
  Iterate over each of the `Storage.FormatConfirmable` objects, potentially checking with the user whether it should be formatted.
  `protected boolean`
  `containsStorageDir(File root)`
  Returns true if the storage directory on the given directory is already loaded.
  `protected boolean`
  `containsStorageDir(StorageLocation location)`
  Returns true if the storage directory on the given directory is already loaded.
  `protected boolean`
  `containsStorageDir(StorageLocation location, String bpid)`
  Returns true if the storage directory on the given location is already loaded.
  `static void`
  `deleteDir(File dir)`
  Recursively delete all the content of the directory first and then the directory itself from the local filesystem.
  `Iterable<Storage.StorageDirectory>`
  `dirIterable(Storage.StorageDirType dirType)`
  `Iterator<Storage.StorageDirectory>`
  `dirIterator()`
  Return default iterator This iterator returns all entries in storageDirs
  `Iterator<Storage.StorageDirectory>`
  `dirIterator(boolean includeShared)`
  Return all entries in storageDirs, potentially excluding shared dirs.
  `Iterator<Storage.StorageDirectory>`
  `dirIterator(Storage.StorageDirType dirType)`
  Return iterator based on Storage Directory Type This iterator selects entries in storageDirs of type dirType and returns them via the Iterator
  `Iterator<Storage.StorageDirectory>`
  `dirIterator(Storage.StorageDirType dirType, boolean includeShared)`
  `static String`
  `getBuildVersion()`
  `List<File>`
  `getFiles(Storage.StorageDirType dirType, String fileName)`
  `NamespaceInfo`
  `getNamespaceInfo()`
  `int`
  `getNumStorageDirs()`
  `static String`
  `getRegistrationID(StorageInfo storage)`
  `Storage.StorageDirectory`
  `getSingularStorageDir()`
  `Storage.StorageDirectory`
  `getStorageDir(int idx)`
  `List<Storage.StorageDirectory>`
  `getStorageDirs()`
  `static boolean`
  `is203LayoutVersion(int layoutVersion)`
  `abstract boolean`
  `isPreUpgradableLayout(Storage.StorageDirectory sd)`
  Return true if the layout of the given storage directory is from a version of Hadoop prior to the introduction of the "current" and "previous" directories which allow upgrade and rollback.
  `String`
  `listStorageDirectories()`
  generate storage list (debug line)
  `static void`
  `nativeCopyFileUnbuffered(File srcFile, File destFile, boolean preserveFileDate)`
  Copies a file (usually large) to a new location using native unbuffered IO.
  `static void`
  `rename(File from, File to)`
  `protected void`
  `setPropertiesFromFields(Properties props, Storage.StorageDirectory sd)`
  Set common storage fields into the given properties object.
  `void`
  `unlockAll()`
  Unlock all storage directories.
  `void`
  `writeAll()`
  Write all data storage files.
  `static void`
  `writeProperties(File to, Properties props)`
  `void`
  `writeProperties(File to, Storage.StorageDirectory sd)`
  `void`
  `writeProperties(Storage.StorageDirectory sd)`
  Write properties to the VERSION file in the given storage directory.

  ### Methods inherited from class org.apache.hadoop.hdfs.server.common.StorageInfo

  `checkStorageType, getClusterID, getClusterIdFromColonSeparatedString, getCTime, getLayoutVersion, getNamespaceID, getNsIdFromColonSeparatedString, getProperty, getServiceLayoutFeatureMap, getServiceLayoutVersion, readPreviousVersionProperties, readProperties, readPropertiesFile, setClusterId, setcTime, setFieldsFromProperties, setLayoutVersion, setNamespaceID, setServiceLayoutVersion, setStorageInfo, toColonSeparatedString, toMapString, toString, versionSupportsFederation`

  ### Methods inherited from class java.lang.Object

  `clone, equals, finalize, getClass, hashCode, notify, notifyAll, wait, wait, wait`

- ## Field Details

  - ### LOG

    public static final org.slf4j.Logger LOG

  - ### LAST_PRE_UPGRADE_LAYOUT_VERSION

    public static final int LAST_PRE_UPGRADE_LAYOUT_VERSION
    See Also:
      - Constant Field Values

  - ### LAST_UPGRADABLE_LAYOUT_VERSION

    public static final int LAST_UPGRADABLE_LAYOUT_VERSION
    See Also:
      - Constant Field Values

  - ### LAST_UPGRADABLE_HADOOP_VERSION

    protected static final String LAST_UPGRADABLE_HADOOP_VERSION
    See Also:
      - Constant Field Values

  - ### LAYOUT_VERSIONS_203

    public static final int[] LAYOUT_VERSIONS_203
    Layout versions of 0.20.203 release

  - ### STORAGE_FILE_LOCK

    public static final String STORAGE_FILE_LOCK
    See Also:
      - Constant Field Values

  - ### STORAGE_DIR_CURRENT

    public static final String STORAGE_DIR_CURRENT
    See Also:
      - Constant Field Values

  - ### STORAGE_DIR_PREVIOUS

    public static final String STORAGE_DIR_PREVIOUS
    See Also:
      - Constant Field Values

  - ### STORAGE_TMP_REMOVED

    public static final String STORAGE_TMP_REMOVED
    See Also:
      - Constant Field Values

  - ### STORAGE_TMP_PREVIOUS

    public static final String STORAGE_TMP_PREVIOUS
    See Also:
      - Constant Field Values

  - ### STORAGE_TMP_FINALIZED

    public static final String STORAGE_TMP_FINALIZED
    See Also:
      - Constant Field Values

  - ### STORAGE_TMP_LAST_CKPT

    public static final String STORAGE_TMP_LAST_CKPT
    See Also:
      - Constant Field Values

  - ### STORAGE_PREVIOUS_CKPT

    public static final String STORAGE_PREVIOUS_CKPT
    See Also:
      - Constant Field Values

  - ### STORAGE_1_BBW

    public static final String STORAGE_1_BBW
    The blocksBeingWritten directory which was used in some 1.x and earlier releases.
    See Also:
      - Constant Field Values

- ## Constructor Details

  - ### Storage

    protected Storage(HdfsServerConstants.NodeType type)
    Create empty storage info of the specified type

  - ### Storage

    protected Storage(StorageInfo storageInfo)

- ## Method Details

  - ### getFiles

    public List<File> getFiles(Storage.StorageDirType dirType, String fileName)
    Returns:
      A list of the given File in every available storage directory, regardless of whether it might exist.

  - ### dirIterator

    public Iterator<Storage.StorageDirectory> dirIterator()
    Return default iterator This iterator returns all entries in storageDirs

  - ### dirIterator

    public Iterator<Storage.StorageDirectory> dirIterator(Storage.StorageDirType dirType)
    Return iterator based on Storage Directory Type This iterator selects entries in storageDirs of type dirType and returns them via the Iterator

  - ### dirIterator

    public Iterator<Storage.StorageDirectory> dirIterator(boolean includeShared)
    Return all entries in storageDirs, potentially excluding shared dirs.
    Parameters:
      `includeShared` - whether or not to include shared dirs.
    Returns:
      an iterator over the configured storage dirs.

  - ### dirIterator

    public Iterator<Storage.StorageDirectory> dirIterator(Storage.StorageDirType dirType, boolean includeShared)
    Parameters:
      `dirType` - all entries will be of this type of dir
      `includeShared` - true to include any shared directories, false otherwise
    Returns:
      an iterator over the configured storage dirs.

  - ### dirIterable

    public Iterable<Storage.StorageDirectory> dirIterable(Storage.StorageDirType dirType)

  - ### listStorageDirectories

    public String listStorageDirectories()
    generate storage list (debug line)

  - ### getNumStorageDirs

    public int getNumStorageDirs()

  - ### getStorageDirs

    public List<Storage.StorageDirectory> getStorageDirs()

  - ### getStorageDir

    public Storage.StorageDirectory getStorageDir(int idx)

  - ### getSingularStorageDir

    public Storage.StorageDirectory getSingularStorageDir()
    Returns:
      the storage directory, with the precondition that this storage has exactly one storage directory

  - ### addStorageDir

    protected void addStorageDir(Storage.StorageDirectory sd)

  - ### containsStorageDir

    protected boolean containsStorageDir(File root) throws IOException
    Returns true if the storage directory on the given directory is already loaded.
    Parameters:
      `root` - the root directory of a `Storage.StorageDirectory`
    Throws:
      `IOException` - if failed to get canonical path.

  - ### containsStorageDir

    protected boolean containsStorageDir(StorageLocation location) throws IOException
    Returns true if the storage directory on the given directory is already loaded.
    Parameters:
      `location` - the `StorageLocation`
    Throws:
      `IOException` - if failed to get canonical path.

  - ### containsStorageDir

    protected boolean containsStorageDir(StorageLocation location, String bpid) throws IOException
    Returns true if the storage directory on the given location is already loaded.
    Parameters:
      `location` - the `StorageLocation`
      `bpid` - the block pool id
    Returns:
      true if the location matches to any existing storage directories
    Throws:
      `IOException` - IOException if failed to read location or storage directory path

  - ### getNamespaceInfo

    public NamespaceInfo getNamespaceInfo()

  - ### isPreUpgradableLayout

    public abstract boolean isPreUpgradableLayout(Storage.StorageDirectory sd) throws IOException
    Return true if the layout of the given storage directory is from a version of Hadoop prior to the introduction of the "current" and "previous" directories which allow upgrade and rollback.
    Throws:
      `IOException`

  - ### checkVersionUpgradable

    public static void checkVersionUpgradable(int oldVersion) throws IOException
    Checks if the upgrade from `oldVersion` is supported.
    Parameters:
      `oldVersion` - the version of the metadata to check with the current version
    Throws:
      `IOException` - if upgrade is not supported

  - ### confirmFormat

    public static boolean confirmFormat(Iterable<? extends Storage.FormatConfirmable> items, boolean force, boolean interactive) throws IOException
    Iterate over each of the `Storage.FormatConfirmable` objects, potentially checking with the user whether it should be formatted. If running in interactive mode, will prompt the user for each directory to allow them to format anyway. Otherwise, returns false, unless 'force' is specified.
    Parameters:
      `force` - format regardless of whether dirs exist
      `interactive` - prompt the user when a dir exists
    Returns:
      true if formatting should proceed
    Throws:
      `IOException` - if some storage cannot be accessed

  - ### setPropertiesFromFields

    protected void setPropertiesFromFields(Properties props, Storage.StorageDirectory sd) throws IOException
    Set common storage fields into the given properties object. Should be overloaded if additional fields need to be set.
    Parameters:
      `props` - the Properties object to write into
    Throws:
      `IOException`

  - ### writeProperties

    public void writeProperties(Storage.StorageDirectory sd) throws IOException
    Write properties to the VERSION file in the given storage directory.
    Throws:
      `IOException`

  - ### writeProperties

    public void writeProperties(File to, Storage.StorageDirectory sd) throws IOException
    Throws:
      `IOException`

  - ### writeProperties

    public static void writeProperties(File to, Properties props) throws IOException
    Throws:
      `IOException`

  - ### rename

    public static void rename(File from, File to) throws IOException
    Throws:
      `IOException`

  - ### nativeCopyFileUnbuffered

    public static void nativeCopyFileUnbuffered(File srcFile, File destFile, boolean preserveFileDate) throws IOException
    Copies a file (usually large) to a new location using native unbuffered IO.

    This method copies the contents of the specified source file to the specified destination file using OS specific unbuffered IO. The goal is to avoid churning the file system buffer cache when copying large files. We can't use FileUtils#copyFile from apache-commons-io because it is a buffered IO based on FileChannel#transferFrom, which uses MmapByteBuffer internally. The directory holding the destination file is created if it does not exist. If the destination file exists, then this method will delete it first.

    Note: Setting `preserveFileDate` to `true` tries to preserve the file's last modified date/times using `File.setLastModified(long)`, however it is not guaranteed that the operation will succeed. If the modification operation fails, no indication is provided.

    Parameters:
      `srcFile` - an existing file to copy, must not be `null`
      `destFile` - the new file, must not be `null`
      `preserveFileDate` - true if the file date of the copy should be the same as the original
    Throws:
      `NullPointerException` - if source or destination is `null`
      `IOException` - if source or destination is invalid
      `IOException` - if an IO error occurs during copying

  - ### deleteDir

    public static void deleteDir(File dir) throws IOException
    Recursively delete all the content of the directory first and then the directory itself from the local filesystem.
    Parameters:
      `dir` - The directory to delete
    Throws:
      `IOException`

  - ### writeAll

    public void writeAll() throws IOException
    Write all data storage files.
    Throws:
      `IOException`

  - ### unlockAll

    public void unlockAll() throws IOException
    Unlock all storage directories.
    Throws:
      `IOException`

  - ### getBuildVersion

    public static String getBuildVersion()

  - ### getRegistrationID

    public static String getRegistrationID(StorageInfo storage)

  - ### is203LayoutVersion

    public static boolean is203LayoutVersion(int layoutVersion)