FileIoProvider (Apache Hadoop HDFS 3.5.0 API)
Description: declaration: package: org.apache.hadoop.hdfs.server.datanode, class: FileIoProvider

Package org.apache.hadoop.hdfs.server.datanode

# Class FileIoProvider

java.lang.Object
org.apache.hadoop.hdfs.server.datanode.FileIoProvider

@Private @Unstable public class FileIoProvider extends Object
This class abstracts out various file IO operations performed by the DataNode and invokes profiling (for collecting stats) and fault injection (for testing) event hooks before and after each file IO. Behavior can be injected into these events by enabling the profiling and/or fault injection event hooks through `DFSConfigKeys.DFS_DATANODE_FILEIO_PROFILING_SAMPLING_PERCENTAGE_KEY` and `DFSConfigKeys.DFS_DATANODE_ENABLE_FILEIO_FAULT_INJECTION_KEY`. These event hooks are disabled by default. Most functions accept an optional `FsVolumeSpi` parameter for instrumentation/logging. Some methods may look redundant, especially the multiple variations of move/rename/list. They exist to retain behavior compatibility for existing code.

- ## Nested Class Summary

  Nested Classes
  Modifier and Type
  Class
  Description
  `static enum`
  `FileIoProvider.OPERATION`
  Lists the types of file system operations.

- ## Field Summary

  Fields
  Modifier and Type
  Field
  Description
  `static final org.slf4j.Logger`
  `LOG`

- ## Constructor Summary

  Constructors
  Constructor
  Description
  `FileIoProvider(org.apache.hadoop.conf.Configuration conf, DataNode datanode)`

- ## Method Summary

  Modifier and Type
  Method
  Description
  `boolean`
  `createFile(FsVolumeSpi volume, File f)`
  Create a file.
  `boolean`
  `delete(FsVolumeSpi volume, File f)`
  Delete a file.
  `boolean`
  `deleteWithExistsCheck(FsVolumeSpi volume, File f)`
  Delete a file, first checking to see if it exists.
  `void`
  `dirSync(FsVolumeSpi volume, File dir)`
  Sync the given directory changes to durable device.
  `boolean`
  `exists(FsVolumeSpi volume, File f)`
  Check for file existence using `File.exists()`.
  `void`
  `flush(FsVolumeSpi volume, Flushable f)`
  See `Flushable.flush()`.
  `boolean`
  `fullyDelete(FsVolumeSpi volume, File dir)`
  Delete the given directory using `FileUtil.fullyDelete(File)`.
  `FileInputStream`
  `getFileInputStream(FsVolumeSpi volume, File f)`
  Create a FileInputStream using `FileInputStream(File)`.
  `FileOutputStream`
  `getFileOutputStream(FsVolumeSpi volume, File f)`
  Create a FileOutputStream using `FileOutputStream(File, boolean)`.
  `FileOutputStream`
  `getFileOutputStream(FsVolumeSpi volume, FileDescriptor fd)`
  Create a FileOutputStream using `FileOutputStream(FileDescriptor)`.
  `FileOutputStream`
  `getFileOutputStream(FsVolumeSpi volume, File f, boolean append)`
  Create a FileOutputStream using `FileOutputStream(File, boolean)`.
  `int`
  `getHardLinkCount(FsVolumeSpi volume, File f)`
  Retrieves the number of links to the specified file.
  `org.apache.hadoop.hdfs.server.datanode.ProfilingFileIoEvents`
  `getProfilingEventHook()`
  `RandomAccessFile`
  `getRandomAccessFile(FsVolumeSpi volume, File f, String mode)`
  Create a RandomAccessFile using `RandomAccessFile(File, String)`.
  `FileInputStream`
  `getShareDeleteFileInputStream(FsVolumeSpi volume, File f, long offset)`
  Create a FileInputStream using `NativeIO.getShareDeleteFileDescriptor(java.io.File, long)`.
  `String[]`
  `list(FsVolumeSpi volume, File dir)`
  Get a listing of the given directory using `FileUtil.listFiles(File)`.
  `List<String>`
  `listDirectory(FsVolumeSpi volume, File dir, FilenameFilter filter)`
  Get a listing of the given directory using `IOUtils.listDirectory(File, FilenameFilter)`.
  `File[]`
  `listFiles(FsVolumeSpi volume, File dir)`
  Get a listing of the given directory using `FileUtil.listFiles(File)`.
  `boolean`
  `mkdirs(FsVolumeSpi volume, File dir)`
  See `File.mkdirs()`.
  `void`
  `mkdirsWithExistsCheck(FsVolumeSpi volume, File dir)`
  Create the target directory using `File.mkdirs()` only if it doesn't exist already.
  `void`
  `move(FsVolumeSpi volume, Path src, Path target, CopyOption... options)`
  Move the src file to the target using `Files.move(Path, Path, CopyOption...)`.
  `void`
  `moveFile(FsVolumeSpi volume, File src, File target)`
  Move the src file to the target using `FileUtils.moveFile(File, File)`.
  `void`
  `nativeCopyFileUnbuffered(FsVolumeSpi volume, File src, File target, boolean preserveFileDate)`
  See `Storage.nativeCopyFileUnbuffered(File, File, boolean)`.
  `FileInputStream`
  `openAndSeek(FsVolumeSpi volume, File f, long offset)`
  Create a FileInputStream using `FileInputStream(File)` and position it at the given offset.
  `void`
  `posixFadvise(FsVolumeSpi volume, String identifier, FileDescriptor outFd, long offset, long length, int flags)`
  Call posix_fadvise on the given file descriptor.
  `void`
  `rename(FsVolumeSpi volume, File src, File target)`
  Move the src file to the target using `Storage.rename(File, File)`.
  `void`
  `replaceFile(FsVolumeSpi volume, File src, File target)`
  Move the src file to the target using `FileUtil.replaceFile(File, File)`.
  `void`
  `sync(FsVolumeSpi volume, FileOutputStream fos)`
  Sync the given `FileOutputStream`.
  `void`
  `syncFileRange(FsVolumeSpi volume, FileDescriptor outFd, long offset, long numBytes, int flags)`
  Call sync_file_range on the given file descriptor.
  `void`
  `transferToSocketFully(FsVolumeSpi volume, org.apache.hadoop.net.SocketOutputStream sockOut, FileChannel fileCh, long position, int count, org.apache.hadoop.io.LongWritable waitTime, org.apache.hadoop.io.LongWritable transferTime)`
  Transfer data from a FileChannel to a SocketOutputStream.

  ### Methods inherited from class java.lang.Object

  `clone, equals, finalize, getClass, hashCode, notify, notifyAll, toString, wait, wait, wait`

- ## Field Details

  - ### LOG

    public static final org.slf4j.Logger LOG

- ## Constructor Details

  - ### FileIoProvider

    public FileIoProvider(@Nullable org.apache.hadoop.conf.Configuration conf, DataNode datanode)
    Parameters:
      `conf` - Configuration object. May be null. When null, the event handlers are no-ops.
      `datanode` - datanode that owns this FileIoProvider. Used for IO error based volume checker callback

- ## Method Details

  - ### flush

    public void flush(@Nullable FsVolumeSpi volume, Flushable f) throws IOException
    See `Flushable.flush()`.
    Parameters:
      `volume` - target volume. null if unavailable.
    Throws:
      `IOException`

  - ### sync

    public void sync(@Nullable FsVolumeSpi volume, FileOutputStream fos) throws IOException
    Sync the given `FileOutputStream`.
    Parameters:
      `volume` - target volume. null if unavailable.
    Throws:
      `IOException`

  - ### dirSync

    public void dirSync(@Nullable FsVolumeSpi volume, File dir) throws IOException
    Sync the given directory changes to durable device.
    Throws:
      `IOException`

  - ### syncFileRange

    public void syncFileRange(@Nullable FsVolumeSpi volume, FileDescriptor outFd, long offset, long numBytes, int flags) throws org.apache.hadoop.io.nativeio.NativeIOException
    Call sync_file_range on the given file descriptor.
    Parameters:
      `volume` - target volume. null if unavailable.
    Throws:
      `org.apache.hadoop.io.nativeio.NativeIOException`

  - ### posixFadvise

    public void posixFadvise(@Nullable FsVolumeSpi volume, String identifier, FileDescriptor outFd, long offset, long length, int flags) throws org.apache.hadoop.io.nativeio.NativeIOException
    Call posix_fadvise on the given file descriptor.
    Parameters:
      `volume` - target volume. null if unavailable.
    Throws:
      `org.apache.hadoop.io.nativeio.NativeIOException`

  - ### delete

    public boolean delete(@Nullable FsVolumeSpi volume, File f)
    Delete a file.
    Parameters:
      `volume` - target volume. null if unavailable.
      `f` - File to delete.
    Returns:
      true if the file was successfully deleted.

  - ### deleteWithExistsCheck

    public boolean deleteWithExistsCheck(@Nullable FsVolumeSpi volume, File f)
    Delete a file, first checking to see if it exists.
    Parameters:
      `volume` - target volume. null if unavailable.
      `f` - File to delete
    Returns:
      true if the file was successfully deleted or if it never existed.

  - ### transferToSocketFully

    public void transferToSocketFully(@Nullable FsVolumeSpi volume, org.apache.hadoop.net.SocketOutputStream sockOut, FileChannel fileCh, long position, int count, org.apache.hadoop.io.LongWritable waitTime, org.apache.hadoop.io.LongWritable transferTime) throws IOException
    Transfer data from a FileChannel to a SocketOutputStream.
    Parameters:
      `volume` - target volume. null if unavailable.
      `sockOut` - SocketOutputStream to write the data.
      `fileCh` - FileChannel from which to read data.
      `position` - position within the channel where the transfer begins.
      `count` - number of bytes to transfer.
      `waitTime` - returns the nanoseconds spent waiting for the socket to become writable.
      `transferTime` - returns the nanoseconds spent transferring data.
    Throws:
      `IOException`

  - ### createFile

    public boolean createFile(@Nullable FsVolumeSpi volume, File f) throws IOException
    Create a file.
    Parameters:
      `volume` - target volume. null if unavailable.
      `f` - File to be created.
    Returns:
      true if the file does not exist and was successfully created. false if the file already exists.
    Throws:
      `IOException`

  - ### getFileInputStream

    public FileInputStream getFileInputStream(@Nullable FsVolumeSpi volume, File f) throws FileNotFoundException
    Create a FileInputStream using `FileInputStream(File)`. Wraps the created input stream to intercept read calls before delegating to the wrapped stream.
    Parameters:
      `volume` - target volume. null if unavailable.
      `f` - File object.
    Returns:
      FileInputStream to the given file.
    Throws:
      `FileNotFoundException`

  - ### getFileOutputStream

    public FileOutputStream getFileOutputStream(@Nullable FsVolumeSpi volume, File f, boolean append) throws FileNotFoundException
    Create a FileOutputStream using `FileOutputStream(File, boolean)`. Wraps the created output stream to intercept write calls before delegating to the wrapped stream.
    Parameters:
      `volume` - target volume. null if unavailable.
      `f` - File object.
      `append` - if true, then bytes will be written to the end of the file rather than the beginning.
    Returns:
      FileOutputStream to the given file object.
    Throws:
      `FileNotFoundException`

  - ### getFileOutputStream

    public FileOutputStream getFileOutputStream(@Nullable FsVolumeSpi volume, File f) throws FileNotFoundException
    Create a FileOutputStream using `FileOutputStream(File, boolean)`. Wraps the created output stream to intercept write calls before delegating to the wrapped stream.
    Parameters:
      `volume` - target volume. null if unavailable.
      `f` - File object.
    Returns:
      FileOutputStream to the given file object.
    Throws:
      `FileNotFoundException`

  - ### getFileOutputStream

    public FileOutputStream getFileOutputStream(@Nullable FsVolumeSpi volume, FileDescriptor fd)
    Create a FileOutputStream using `FileOutputStream(FileDescriptor)`. Wraps the created output stream to intercept write calls before delegating to the wrapped stream.
    Parameters:
      `volume` - target volume. null if unavailable.
      `fd` - File descriptor object.
    Returns:
      FileOutputStream to the given file object.

  - ### getShareDeleteFileInputStream

    public FileInputStream getShareDeleteFileInputStream(@Nullable FsVolumeSpi volume, File f, long offset) throws IOException
    Create a FileInputStream using `NativeIO.getShareDeleteFileDescriptor(java.io.File, long)`. Wraps the created input stream to intercept input calls before delegating to the wrapped stream.
    Parameters:
      `volume` - target volume. null if unavailable.
      `f` - File object.
      `offset` - the offset position, measured in bytes from the beginning of the file, at which to set the file pointer.
    Returns:
      FileOutputStream to the given file object.
    Throws:
      `FileNotFoundException`
      `IOException`

  - ### openAndSeek

    public FileInputStream openAndSeek(@Nullable FsVolumeSpi volume, File f, long offset) throws IOException
    Create a FileInputStream using `FileInputStream(File)` and position it at the given offset. Wraps the created input stream to intercept read calls before delegating to the wrapped stream.
    Parameters:
      `volume` - target volume. null if unavailable.
      `f` - File object.
      `offset` - the offset position, measured in bytes from the beginning of the file, at which to set the file pointer.
    Throws:
      `FileNotFoundException`
      `IOException`

  - ### getRandomAccessFile

    public RandomAccessFile getRandomAccessFile(@Nullable FsVolumeSpi volume, File f, String mode) throws FileNotFoundException
    Create a RandomAccessFile using `RandomAccessFile(File, String)`. Wraps the created input stream to intercept IO calls before delegating to the wrapped RandomAccessFile.
    Parameters:
      `volume` - target volume. null if unavailable.
      `f` - File object.
      `mode` - See `RandomAccessFile` for a description of the mode string.
    Returns:
      RandomAccessFile representing the given file.
    Throws:
      `FileNotFoundException`

  - ### fullyDelete

    public boolean fullyDelete(@Nullable FsVolumeSpi volume, File dir)
    Delete the given directory using `FileUtil.fullyDelete(File)`.
    Parameters:
      `volume` - target volume. null if unavailable.
      `dir` - directory to be deleted.
    Returns:
      true on success false on failure.

  - ### replaceFile

    public void replaceFile(@Nullable FsVolumeSpi volume, File src, File target) throws IOException
    Move the src file to the target using `FileUtil.replaceFile(File, File)`.
    Parameters:
      `volume` - target volume. null if unavailable.
      `src` - source path.
      `target` - target path.
    Throws:
      `IOException`

  - ### rename

    public void rename(@Nullable FsVolumeSpi volume, File src, File target) throws IOException
    Move the src file to the target using `Storage.rename(File, File)`.
    Parameters:
      `volume` - target volume. null if unavailable.
      `src` - source path.
      `target` - target path.
    Throws:
      `IOException`

  - ### moveFile

    public void moveFile(@Nullable FsVolumeSpi volume, File src, File target) throws IOException
    Move the src file to the target using `FileUtils.moveFile(File, File)`.
    Parameters:
      `volume` - target volume. null if unavailable.
      `src` - source path.
      `target` - target path.
    Throws:
      `IOException`

  - ### move

    public void move(@Nullable FsVolumeSpi volume, Path src, Path target, CopyOption... options) throws IOException
    Move the src file to the target using `Files.move(Path, Path, CopyOption...)`.
    Parameters:
      `volume` - target volume. null if unavailable.
      `src` - source path.
      `target` - target path.
      `options` - See `Files.move(java.nio.file.Path, java.nio.file.Path, java.nio.file.CopyOption...)` for a description of the options.
    Throws:
      `IOException`

  - ### nativeCopyFileUnbuffered

    public void nativeCopyFileUnbuffered(@Nullable FsVolumeSpi volume, File src, File target, boolean preserveFileDate) throws IOException
    See `Storage.nativeCopyFileUnbuffered(File, File, boolean)`.
    Parameters:
      `volume` - target volume. null if unavailable.
      `src` - an existing file to copy, must not be `null`
      `target` - the new file, must not be `null`
      `preserveFileDate` - true if the file date of the copy should be the same as the original
    Throws:
      `IOException`

  - ### mkdirs

    public boolean mkdirs(@Nullable FsVolumeSpi volume, File dir) throws IOException
    See `File.mkdirs()`.
    Parameters:
      `volume` - target volume. null if unavailable.
      `dir` - directory to be created.
    Returns:
      true only if the directory was created. false if the directory already exists.
    Throws:
      `IOException` - if a directory with the given name does not exist and could not be created.

  - ### mkdirsWithExistsCheck

    public void mkdirsWithExistsCheck(@Nullable FsVolumeSpi volume, File dir) throws IOException
    Create the target directory using `File.mkdirs()` only if it doesn't exist already.
    Parameters:
      `volume` - target volume. null if unavailable.
      `dir` - directory to be created.
    Throws:
      `IOException` - if the directory could not created

  - ### listFiles

    public File[] listFiles(@Nullable FsVolumeSpi volume, File dir) throws IOException
    Get a listing of the given directory using `FileUtil.listFiles(File)`.
    Parameters:
      `volume` - target volume. null if unavailable.
      `dir` - Directory to be listed.
    Returns:
      array of file objects representing the directory entries.
    Throws:
      `IOException`

  - ### list

    public String[] list(@Nullable FsVolumeSpi volume, File dir) throws IOException
    Get a listing of the given directory using `FileUtil.listFiles(File)`.
    Parameters:
      `volume` - target volume. null if unavailable.
      `dir` - directory to be listed.
    Returns:
      array of strings representing the directory entries.
    Throws:
      `IOException`

  - ### listDirectory

    public List<String> listDirectory(@Nullable FsVolumeSpi volume, File dir, FilenameFilter filter) throws IOException
    Get a listing of the given directory using `IOUtils.listDirectory(File, FilenameFilter)`.
    Parameters:
      `volume` - target volume. null if unavailable.
      `dir` - Directory to list.
      `filter` - `FilenameFilter` to filter the directory entries.
    Throws:
      `IOException`

  - ### getHardLinkCount

    public int getHardLinkCount(@Nullable FsVolumeSpi volume, File f) throws IOException
    Retrieves the number of links to the specified file.
    Parameters:
      `volume` - target volume. null if unavailable.
      `f` - file whose link count is being queried.
    Returns:
      number of hard-links to the given file, including the given path itself.
    Throws:
      `IOException`

  - ### exists

    public boolean exists(@Nullable FsVolumeSpi volume, File f)
    Check for file existence using `File.exists()`.
    Parameters:
      `volume` - target volume. null if unavailable.
      `f` - file object.
    Returns:
      true if the file exists.

  - ### getProfilingEventHook

    public org.apache.hadoop.hdfs.server.datanode.ProfilingFileIoEvents getProfilingEventHook()