IOUtils (Apache Hadoop Common 3.5.0 API)
Description: declaration: package: org.apache.hadoop.io, class: IOUtils

Package org.apache.hadoop.io

# Class IOUtils

java.lang.Object
org.apache.hadoop.io.IOUtils

@Public @Evolving public class IOUtils extends Object
An utility class for I/O related functionality.

- ## Nested Class Summary

  Nested Classes
  Modifier and Type
  Class
  Description
  `static class`
  `IOUtils.NullOutputStream`
  The /dev/null of OutputStreams.

- ## Field Summary

  Fields
  Modifier and Type
  Field
  Description
  `static final org.slf4j.Logger`
  `LOG`

- ## Constructor Summary

  Constructors
  Constructor
  Description
  `IOUtils()`

- ## Method Summary

  Modifier and Type
  Method
  Description
  `static void`
  `cleanupWithLogger(org.slf4j.Logger logger, Closeable... closeables)`
  Close the Closeable objects and ignore any `Throwable` or null pointers.
  `static void`
  `closeSocket(Socket sock)`
  Closes the socket ignoring `IOException`
  `static void`
  `closeStream(Closeable stream)`
  Closes the stream ignoring `Throwable`.
  `static void`
  `closeStreams(Closeable... streams)`
  Closes the streams ignoring `Throwable`.
  `static void`
  `copyBytes(InputStream in, OutputStream out, int buffSize)`
  Copies from one stream to another.
  `static void`
  `copyBytes(InputStream in, OutputStream out, int buffSize, boolean close)`
  Copies from one stream to another.
  `static void`
  `copyBytes(InputStream in, OutputStream out, long count, boolean close)`
  Copies count bytes from one stream to another.
  `static void`
  `copyBytes(InputStream in, OutputStream out, Configuration conf)`
  Copies from one stream to another.
  `static void`
  `copyBytes(InputStream in, OutputStream out, Configuration conf, boolean close)`
  Copies from one stream to another.
  `static void`
  `fsync(File fileToSync)`
  Ensure that any writes to the given file is written to the storage device that contains it.
  `static void`
  `fsync(FileChannel channel, boolean isDir)`
  Ensure that any writes to the given file is written to the storage device that contains it.
  `static List<String>`
  `listDirectory(File dir, FilenameFilter filter)`
  Return the complete list of files in a directory as strings.
  `static void`
  `readFully(InputStream in, byte[] buf, int off, int len)`
  Reads len bytes in a loop.
  `static byte[]`
  `readFullyToByteArray(DataInput in)`
  Reads a DataInput until EOF and returns a byte array.
  `static void`
  `skipFully(InputStream in, long len)`
  Similar to readFully().
  `static IOException`
  `wrapException(String path, String methodName, IOException exception)`
  Takes an IOException, file/directory path, and method name and returns an IOException with the input exception as the cause and also include the file,method details.
  `static int`
  `wrappedReadForCompressedData(InputStream is, byte[] buf, int off, int len)`
  Utility wrapper for reading from `InputStream`.
  `static void`
  `writeFully(FileChannel fc, ByteBuffer buf, long offset)`
  Write a ByteBuffer to a FileChannel at a given offset, handling short writes.
  `static void`
  `writeFully(WritableByteChannel bc, ByteBuffer buf)`
  Write a ByteBuffer to a WritableByteChannel, handling short writes.

  ### Methods inherited from class java.lang.Object

  `clone, equals, finalize, getClass, hashCode, notify, notifyAll, toString, wait, wait, wait`

- ## Field Details

  - ### LOG

    public static final org.slf4j.Logger LOG

- ## Constructor Details

  - ### IOUtils

    public IOUtils()

- ## Method Details

  - ### copyBytes

    public static void copyBytes(InputStream in, OutputStream out, int buffSize, boolean close) throws IOException
    Copies from one stream to another.
    Parameters:
      `in` - InputStrem to read from
      `out` - OutputStream to write to
      `buffSize` - the size of the buffer
      `close` - whether or not close the InputStream and OutputStream at the end. The streams are closed in the finally clause.
    Throws:
      `IOException` - raised on errors performing I/O.

  - ### copyBytes

    public static void copyBytes(InputStream in, OutputStream out, int buffSize) throws IOException
    Copies from one stream to another.
    Parameters:
      `in` - InputStrem to read from
      `out` - OutputStream to write to
      `buffSize` - the size of the buffer.
    Throws:
      `IOException` - raised on errors performing I/O.

  - ### copyBytes

    public static void copyBytes(InputStream in, OutputStream out, Configuration conf) throws IOException
    Copies from one stream to another. closes the input and output streams at the end.
    Parameters:
      `in` - InputStrem to read from
      `out` - OutputStream to write to
      `conf` - the Configuration object.
    Throws:
      `IOException` - raised on errors performing I/O.

  - ### copyBytes

    public static void copyBytes(InputStream in, OutputStream out, Configuration conf, boolean close) throws IOException
    Copies from one stream to another.
    Parameters:
      `in` - InputStream to read from
      `out` - OutputStream to write to
      `conf` - the Configuration object
      `close` - whether or not close the InputStream and OutputStream at the end. The streams are closed in the finally clause.
    Throws:
      `IOException` - raised on errors performing I/O.

  - ### copyBytes

    public static void copyBytes(InputStream in, OutputStream out, long count, boolean close) throws IOException
    Copies count bytes from one stream to another.
    Parameters:
      `in` - InputStream to read from
      `out` - OutputStream to write to
      `count` - number of bytes to copy
      `close` - whether to close the streams
    Throws:
      `IOException` - if bytes can not be read or written

  - ### wrappedReadForCompressedData

    public static int wrappedReadForCompressedData(InputStream is, byte[] buf, int off, int len) throws IOException
    Utility wrapper for reading from `InputStream`. It catches any errors thrown by the underlying stream (either IO or decompression-related), and re-throws as an IOException.
    Parameters:
      `is` - - InputStream to be read from
      `buf` - - buffer the data is read into
      `off` - - offset within buf
      `len` - - amount of data to be read
    Returns:
      number of bytes read
    Throws:
      `IOException` - raised on errors performing I/O.

  - ### readFully

    public static void readFully(InputStream in, byte[] buf, int off, int len) throws IOException
    Reads len bytes in a loop.
    Parameters:
      `in` - InputStream to read from
      `buf` - The buffer to fill
      `off` - offset from the buffer
      `len` - the length of bytes to read
    Throws:
      `IOException` - if it could not read requested number of bytes for any reason (including EOF)

  - ### skipFully

    public static void skipFully(InputStream in, long len) throws IOException
    Similar to readFully(). Skips bytes in a loop.
    Parameters:
      `in` - The InputStream to skip bytes from
      `len` - number of bytes to skip.
    Throws:
      `IOException` - if it could not skip requested number of bytes for any reason (including EOF)

  - ### cleanupWithLogger

    public static void cleanupWithLogger(org.slf4j.Logger logger, Closeable... closeables)
    Close the Closeable objects and ignore any `Throwable` or null pointers. Must only be used for cleanup in exception handlers.
    Parameters:
      `logger` - the log to record problems to at debug level. Can be null.
      `closeables` - the objects to close

  - ### closeStream

    public static void closeStream(Closeable stream)
    Closes the stream ignoring `Throwable`. Must only be called in cleaning up from exception handlers.
    Parameters:
      `stream` - the Stream to close

  - ### closeStreams

    public static void closeStreams(Closeable... streams)
    Closes the streams ignoring `Throwable`. Must only be called in cleaning up from exception handlers.
    Parameters:
      `streams` - the Streams to close

  - ### closeSocket

    public static void closeSocket(Socket sock)
    Closes the socket ignoring `IOException`
    Parameters:
      `sock` - the Socket to close

  - ### writeFully

    public static void writeFully(WritableByteChannel bc, ByteBuffer buf) throws IOException
    Write a ByteBuffer to a WritableByteChannel, handling short writes.
    Parameters:
      `bc` - The WritableByteChannel to write to
      `buf` - The input buffer
    Throws:
      `IOException` - On I/O error

  - ### writeFully

    public static void writeFully(FileChannel fc, ByteBuffer buf, long offset) throws IOException
    Write a ByteBuffer to a FileChannel at a given offset, handling short writes.
    Parameters:
      `fc` - The FileChannel to write to
      `buf` - The input buffer
      `offset` - The offset in the file to start writing at
    Throws:
      `IOException` - On I/O error

  - ### listDirectory

    public static List<String> listDirectory(File dir, FilenameFilter filter) throws IOException
    Return the complete list of files in a directory as strings.

    This is better than File#listDir because it does not ignore IOExceptions.

    Parameters:
      `dir` - The directory to list.
      `filter` - If non-null, the filter to use when listing this directory.
    Returns:
      The list of files in the directory.
    Throws:
      `IOException` - On I/O error

  - ### fsync

    public static void fsync(File fileToSync) throws IOException
    Ensure that any writes to the given file is written to the storage device that contains it. This method opens channel on given File and closes it once the sync is done.
    Borrowed from Uwe Schindler in LUCENE-5588
    Parameters:
      `fileToSync` - the file to fsync
    Throws:
      `IOException` - raised on errors performing I/O.

  - ### fsync

    public static void fsync(FileChannel channel, boolean isDir) throws IOException
    Ensure that any writes to the given file is written to the storage device that contains it. This method opens channel on given File and closes it once the sync is done. Borrowed from Uwe Schindler in LUCENE-5588
    Parameters:
      `channel` - Channel to sync
      `isDir` - if true, the given file is a directory (Channel should be opened for read and ignore IOExceptions, because not all file systems and operating systems allow to fsync on a directory)
    Throws:
      `IOException` - raised on errors performing I/O.

  - ### wrapException

    public static IOException wrapException(String path, String methodName, IOException exception)
    Takes an IOException, file/directory path, and method name and returns an IOException with the input exception as the cause and also include the file,method details. The new exception provides the stack trace of the place where the exception is thrown and some extra diagnostics information. Return instance of same exception if exception class has a public string constructor; Otherwise return an PathIOException. InterruptedIOException and PathIOException are returned unwrapped.
    Parameters:
      `path` - file/directory path
      `methodName` - method name
      `exception` - the caught exception.
    Returns:
      an exception to throw

  - ### readFullyToByteArray

    public static byte[] readFullyToByteArray(DataInput in) throws IOException
    Reads a DataInput until EOF and returns a byte array. Make sure not to pass in an infinite DataInput or this will never return.
    Parameters:
      `in` - A DataInput
    Returns:
      a byte array containing the data from the DataInput
    Throws:
      `IOException` - on I/O error, other than EOF