DFSOutputStream (Apache Hadoop HDFS Client 3.5.0 API)
Description: declaration: package: org.apache.hadoop.hdfs, class: DFSOutputStream

Package org.apache.hadoop.hdfs

# Class DFSOutputStream

java.lang.Object
java.io.OutputStream
org.apache.hadoop.fs.FSOutputSummer
org.apache.hadoop.hdfs.DFSOutputStream

All Implemented Interfaces:
  `Closeable`, `Flushable`, `AutoCloseable`, `org.apache.hadoop.fs.CanSetDropBehind`, `org.apache.hadoop.fs.StreamCapabilities`, `org.apache.hadoop.fs.Syncable`

Direct Known Subclasses:
  `DFSStripedOutputStream`

@Private public class DFSOutputStream extends org.apache.hadoop.fs.FSOutputSummer implements org.apache.hadoop.fs.Syncable, org.apache.hadoop.fs.CanSetDropBehind, org.apache.hadoop.fs.StreamCapabilities
DFSOutputStream creates files from a stream of bytes. The client application writes data that is cached internally by this stream. Data is broken up into packets, each packet is typically 64K in size. A packet comprises of chunks. Each chunk is typically 512 bytes and has an associated checksum with it. When a client application fills up the currentPacket, it is enqueued into the dataQueue of DataStreamer. DataStreamer is a thread that picks up packets from the dataQueue and sends it to the first datanode in the pipeline.

- ## Nested Class Summary

  ## Nested classes/interfaces inherited from interface org.apache.hadoop.fs.StreamCapabilities

  `org.apache.hadoop.fs.StreamCapabilities.StreamCapability`

- ## Field Summary

  Fields
  Modifier and Type
  Field
  Description
  `protected final long`
  `blockSize`
  `protected final ByteArrayManager`
  `byteArrayManager`
  `protected final int`
  `bytesPerChecksum`
  `protected final AtomicReference<CachingStrategy>`
  `cachingStrategy`
  `protected int`
  `chunksPerPacket`
  `protected boolean`
  `closed`
  `protected DFSPacket`
  `currentPacket`
  `protected final DFSClient`
  `dfsClient`
  `protected final long`
  `fileId`
  `protected long`
  `initialFileSize`
  `protected long`
  `lastFlushOffset`
  `protected int`
  `packetSize`
  `protected boolean`
  `shouldSyncBlock`
  `protected final String`
  `src`
  `protected org.apache.hadoop.hdfs.DataStreamer`
  `streamer`

  ### Fields inherited from interface org.apache.hadoop.fs.StreamCapabilities

  `ABORTABLE_STREAM, DROPBEHIND, HFLUSH, HSYNC, IOSTATISTICS, IOSTATISTICS_CONTEXT, PREADBYTEBUFFER, READAHEAD, READBYTEBUFFER, UNBUFFER, VECTOREDIO, VECTOREDIO_BUFFERS_SLICED`

- ## Constructor Summary

  Constructors
  Modifier
  Constructor
  Description
  `protected`
  `DFSOutputStream(DFSClient dfsClient, String src, HdfsFileStatus stat, EnumSet<org.apache.hadoop.fs.CreateFlag> flag, org.apache.hadoop.util.Progressable progress, org.apache.hadoop.util.DataChecksum checksum, String[] favoredNodes, boolean createStreamer)`
  Construct a new output stream for creating a file.

- ## Method Summary

  Modifier and Type
  Method
  Description
  `protected void`
  `adjustChunkBoundary()`
  If the reopened file did not end at chunk boundary and the above write filled up its partial chunk.
  `protected void`
  `checkClosed()`
  `void`
  `close()`
  Closes this output stream and releases any system resources associated with this stream.
  `protected void`
  `closeImpl()`
  `protected void`
  `closeThreads(boolean force)`
  `protected void`
  `completeFile(ExtendedBlock last)`
  `protected void`
  `computePacketChunkSize(int psize, int csize)`
  `protected DFSPacket`
  `createPacket(int packetSize, int chunksPerPkt, long offsetInBlock, long seqno, boolean lastPacketInBlock)`
  Use `ByteArrayManager` to create buffer for non-heartbeat packets.
  `protected org.apache.hadoop.tracing.TraceScope`
  `createWriteTraceScope()`
  `protected void`
  `flushInternal()`
  Waits till all existing data is flushed and confirmations received from datanodes.
  `protected long`
  `flushInternalWithoutWaitingAck()`
  `protected EnumSet<AddBlockFlag>`
  `getAddBlockFlags()`
  `int`
  `getCurrentBlockReplication()`
  Note that this is not a public API; use `HdfsDataOutputStream.getCurrentBlockReplication()` instead.
  `org.apache.hadoop.fs.FileEncryptionInfo`
  `getFileEncryptionInfo()`
  `long`
  `getFileId()`
  `long`
  `getInitialLen()`
  Returns the size of a file as it was when this stream was opened
  `String`
  `getNamespace()`
  `int`
  `getNumCurrentReplicas()`
  Deprecated.
  use `HdfsDataOutputStream.getCurrentBlockReplication()`.
  `DatanodeInfo[]`
  `getPipeline()`
  `protected org.apache.hadoop.hdfs.DataStreamer`
  `getStreamer()`
  Returns the data streamer object.
  `String`
  `getUniqKey()`
  `boolean`
  `hasCapability(String capability)`
  `void`
  `hflush()`
  Flushes out to all replicas of the block.
  `void`
  `hsync()`
  `void`
  `hsync(EnumSet<HdfsDataOutputStream.SyncFlag> syncFlags)`
  The expected semantics is all data have flushed out to all replicas and all replicas have done posix fsync equivalent - ie the OS has flushed it to the disk device (but the disk may have it in its cache).
  `protected void`
  `recoverLease(boolean recoverLeaseOnCloseException)`
  If recoverLeaseOnCloseException is true and an exception occurs when closing a file, recover lease.
  `void`
  `setArtificialSlowdown(long period)`
  `void`
  `setChunksPerPacket(int value)`
  `void`
  `setDropBehind(Boolean dropBehind)`
  `protected void`
  `start()`
  `String`
  `toString()`
  `protected void`
  `writeChunk(byte[] b, int offset, int len, byte[] checksum, int ckoff, int cklen)`
  `protected void`
  `writeChunk(ByteBuffer buffer, int len, byte[] checksum, int ckoff, int cklen)`

  ### Methods inherited from class org.apache.hadoop.fs.FSOutputSummer

  `convertToByteStream, flush, flushBuffer, flushBuffer, getBufferedDataSize, getChecksumSize, getDataChecksum, resetChecksumBufSize, setChecksumBufSize, write, write`

  ### Methods inherited from class java.io.OutputStream

  `nullOutputStream, write`

  ### Methods inherited from class java.lang.Object

  `clone, equals, finalize, getClass, hashCode, notify, notifyAll, wait, wait, wait`

- ## Field Details

  - ### dfsClient

    protected final DFSClient dfsClient

  - ### byteArrayManager

    protected final ByteArrayManager byteArrayManager

  - ### closed

    protected volatile boolean closed

  - ### src

    protected final String src

  - ### fileId

    protected final long fileId

  - ### blockSize

    protected final long blockSize

  - ### bytesPerChecksum

    protected final int bytesPerChecksum

  - ### currentPacket

    protected DFSPacket currentPacket

  - ### streamer

    protected org.apache.hadoop.hdfs.DataStreamer streamer

  - ### packetSize

    protected int packetSize

  - ### chunksPerPacket

    protected int chunksPerPacket

  - ### lastFlushOffset

    protected long lastFlushOffset

  - ### initialFileSize

    protected long initialFileSize

  - ### shouldSyncBlock

    protected boolean shouldSyncBlock

  - ### cachingStrategy

    protected final AtomicReference<CachingStrategy> cachingStrategy

- ## Constructor Details

  - ### DFSOutputStream

    protected DFSOutputStream(DFSClient dfsClient, String src, HdfsFileStatus stat, EnumSet<org.apache.hadoop.fs.CreateFlag> flag, org.apache.hadoop.util.Progressable progress, org.apache.hadoop.util.DataChecksum checksum, String[] favoredNodes, boolean createStreamer)
    Construct a new output stream for creating a file.

- ## Method Details

  - ### createPacket

    protected DFSPacket createPacket(int packetSize, int chunksPerPkt, long offsetInBlock, long seqno, boolean lastPacketInBlock) throws InterruptedIOException
    Use `ByteArrayManager` to create buffer for non-heartbeat packets.
    Throws:
      `InterruptedIOException`

  - ### checkClosed

    protected void checkClosed() throws IOException
    Specified by:
      `checkClosed` in class `org.apache.hadoop.fs.FSOutputSummer`
    Throws:
      `IOException`

  - ### getPipeline

    @VisibleForTesting public DatanodeInfo[] getPipeline()

  - ### computePacketChunkSize

    protected void computePacketChunkSize(int psize, int csize)

  - ### createWriteTraceScope

    protected org.apache.hadoop.tracing.TraceScope createWriteTraceScope()
    Overrides:
      `createWriteTraceScope` in class `org.apache.hadoop.fs.FSOutputSummer`

  - ### writeChunk

    protected void writeChunk(byte[] b, int offset, int len, byte[] checksum, int ckoff, int cklen) throws IOException
    Specified by:
      `writeChunk` in class `org.apache.hadoop.fs.FSOutputSummer`
    Throws:
      `IOException`

  - ### writeChunk

    protected void writeChunk(ByteBuffer buffer, int len, byte[] checksum, int ckoff, int cklen) throws IOException
    Throws:
      `IOException`

  - ### adjustChunkBoundary

    protected void adjustChunkBoundary()
    If the reopened file did not end at chunk boundary and the above write filled up its partial chunk. Tell the summer to generate full crc chunks from now on.

  - ### hasCapability

    public boolean hasCapability(String capability)
    Specified by:
      `hasCapability` in interface `org.apache.hadoop.fs.StreamCapabilities`
    Overrides:
      `hasCapability` in class `org.apache.hadoop.fs.FSOutputSummer`

  - ### hflush

    public void hflush() throws IOException
    Flushes out to all replicas of the block. The data is in the buffers of the DNs but not necessarily in the DN's OS buffers. It is a synchronous operation. When it returns, it guarantees that flushed data become visible to new readers. It is not guaranteed that data has been flushed to persistent store on the datanode. Block allocations are persisted on namenode.
    Specified by:
      `hflush` in interface `org.apache.hadoop.fs.Syncable`
    Throws:
      `IOException`

  - ### hsync

    public void hsync() throws IOException
    Specified by:
      `hsync` in interface `org.apache.hadoop.fs.Syncable`
    Throws:
      `IOException`

  - ### hsync

    public void hsync(EnumSet<HdfsDataOutputStream.SyncFlag> syncFlags) throws IOException
    The expected semantics is all data have flushed out to all replicas and all replicas have done posix fsync equivalent - ie the OS has flushed it to the disk device (but the disk may have it in its cache). Note that only the current block is flushed to the disk device. To guarantee durable sync across block boundaries the stream should be created with `CreateFlag.SYNC_BLOCK`.
    Parameters:
      `syncFlags` - Indicate the semantic of the sync. Currently used to specify whether or not to update the block length in NameNode.
    Throws:
      `IOException`

  - ### getNumCurrentReplicas

    @Deprecated public int getNumCurrentReplicas() throws IOException
    Deprecated.
    use `HdfsDataOutputStream.getCurrentBlockReplication()`.
    Throws:
      `IOException`

  - ### getCurrentBlockReplication

    public int getCurrentBlockReplication() throws IOException
    Note that this is not a public API; use `HdfsDataOutputStream.getCurrentBlockReplication()` instead.
    Returns:
      the number of valid replicas of the current block
    Throws:
      `IOException`

  - ### flushInternal

    protected void flushInternal() throws IOException
    Waits till all existing data is flushed and confirmations received from datanodes.
    Throws:
      `IOException`

  - ### start

    protected void start()

  - ### closeThreads

    protected void closeThreads(boolean force) throws IOException
    Throws:
      `IOException`

  - ### close

    public void close() throws IOException
    Closes this output stream and releases any system resources associated with this stream.
    Specified by:
      `close` in interface `AutoCloseable`
    Specified by:
      `close` in interface `Closeable`
    Overrides:
      `close` in class `OutputStream`
    Throws:
      `IOException`

  - ### closeImpl

    protected void closeImpl() throws IOException
    Throws:
      `IOException`

  - ### recoverLease

    protected void recoverLease(boolean recoverLeaseOnCloseException)
    If recoverLeaseOnCloseException is true and an exception occurs when closing a file, recover lease.

  - ### completeFile

    protected void completeFile(ExtendedBlock last) throws IOException
    Throws:
      `IOException`

  - ### setArtificialSlowdown

    @VisibleForTesting public void setArtificialSlowdown(long period)

  - ### setChunksPerPacket

    @VisibleForTesting public void setChunksPerPacket(int value)

  - ### getInitialLen

    public long getInitialLen()
    Returns the size of a file as it was when this stream was opened

  - ### getAddBlockFlags

    protected EnumSet<AddBlockFlag> getAddBlockFlags()

  - ### getFileEncryptionInfo

    public org.apache.hadoop.fs.FileEncryptionInfo getFileEncryptionInfo()
    Returns:
      the FileEncryptionInfo for this stream, or null if not encrypted.

  - ### flushInternalWithoutWaitingAck

    protected long flushInternalWithoutWaitingAck() throws IOException
    Throws:
      `IOException`

  - ### setDropBehind

    public void setDropBehind(Boolean dropBehind) throws IOException
    Specified by:
      `setDropBehind` in interface `org.apache.hadoop.fs.CanSetDropBehind`
    Throws:
      `IOException`

  - ### getFileId

    @VisibleForTesting public long getFileId()

  - ### getNamespace

    @VisibleForTesting public String getNamespace()

  - ### getUniqKey

    @VisibleForTesting public String getUniqKey()

  - ### getStreamer

    protected org.apache.hadoop.hdfs.DataStreamer getStreamer()
    Returns the data streamer object.

  - ### toString

    public String toString()
    Overrides:
      `toString` in class `Object`