StripedBlockUtil (Apache Hadoop HDFS Client 3.5.0 API)
Description: declaration: package: org.apache.hadoop.hdfs.util, class: StripedBlockUtil

Package org.apache.hadoop.hdfs.util

# Class StripedBlockUtil

java.lang.Object
org.apache.hadoop.hdfs.util.StripedBlockUtil

@Private public class StripedBlockUtil extends Object
When accessing a file in striped layout, operations on logical byte ranges in the file need to be mapped to physical byte ranges on block files stored on DataNodes. This utility class facilities this mapping by defining and exposing a number of striping-related concepts. The most basic ones are illustrated in the following diagram. Unless otherwise specified, all range-related calculations are inclusive (the end offset of the previous range should be 1 byte lower than the start offset of the next one).

- ## Nested Class Summary

  Nested Classes
  Modifier and Type
  Class
  Description
  `static class`
  `StripedBlockUtil.AlignedStripe`
  Given a requested byte range on a striped block group, an AlignedStripe represents an inclusive `StripedBlockUtil.VerticalRange` that is aligned with both the byte range and boundaries of all internal blocks.
  `static class`
  `StripedBlockUtil.BlockReadStats`
  Struct holding the read statistics.
  `static class`
  `StripedBlockUtil.ChunkByteBuffer`
  A utility to manage ByteBuffer slices for a reader.
  `static class`
  `StripedBlockUtil.StripeRange`
  Used to indicate the buffered data's range in the block group.
  `static class`
  `StripedBlockUtil.StripingCell`
  Cell is the unit of encoding used in `DFSStripedOutputStream`.
  `static class`
  `StripedBlockUtil.StripingChunk`
  Indicates the coverage of an `StripedBlockUtil.AlignedStripe` on an internal block, and the state of the chunk in the context of the read request.
  `static class`
  `StripedBlockUtil.StripingChunkReadResult`
  This class represents result from a striped read request.
  `static class`
  `StripedBlockUtil.VerticalRange`
  A simple utility class representing an arbitrary vertical inclusive range starting at `StripedBlockUtil.VerticalRange.offsetInBlock` and lasting for `StripedBlockUtil.VerticalRange.spanInBlock` bytes in an internal block.

- ## Field Summary

  Fields
  Modifier and Type
  Field
  Description
  `static final org.slf4j.Logger`
  `LOG`

- ## Constructor Summary

  Constructors
  Constructor
  Description
  `StripedBlockUtil()`

- ## Method Summary

  Modifier and Type
  Method
  Description
  `static void`
  `checkBlocks(ExtendedBlock blockGroup, int i, ExtendedBlock blocki)`
  Check if the information such as IDs and generation stamps in block-i match the block group.
  `static ExtendedBlock`
  `constructInternalBlock(ExtendedBlock blockGroup, int cellSize, int dataBlkNum, int idxInBlockGroup)`
  This method creates an internal `ExtendedBlock` at the given index of a block group.
  `static ExtendedBlock`
  `constructInternalBlock(ExtendedBlock blockGroup, ErasureCodingPolicy ecPolicy, int idxInBlockGroup)`
  `static LocatedBlock`
  `constructInternalBlock(LocatedStripedBlock bg, int idxInReturnedLocs, int cellSize, int dataBlkNum, int idxInBlockGroup)`
  This method creates an internal block at the given index of a block group.
  `static StripedBlockUtil.AlignedStripe[]`
  `divideByteRangeIntoStripes(ErasureCodingPolicy ecPolicy, int cellSize, LocatedStripedBlock blockGroup, long rangeStartInBlockGroup, long rangeEndInBlockGroup, ByteBuffer buf)`
  This method divides a requested byte range into an array of inclusive `StripedBlockUtil.AlignedStripe`.
  `static StripedBlockUtil.AlignedStripe[]`
  `divideOneStripe(ErasureCodingPolicy ecPolicy, int cellSize, LocatedStripedBlock blockGroup, long rangeStartInBlockGroup, long rangeEndInBlockGroup, ByteBuffer buf)`
  Similar functionality with `divideByteRangeIntoStripes(org.apache.hadoop.hdfs.protocol.ErasureCodingPolicy, int, org.apache.hadoop.hdfs.protocol.LocatedStripedBlock, long, long, java.nio.ByteBuffer)`, but is used by stateful read and uses ByteBuffer as reading target buffer.
  `static int`
  `getBlockIndex(Block reportedBlock)`
  `static long`
  `getInternalBlockLength(long dataSize, int cellSize, int numDataBlocks, int idxInBlockGroup)`
  Get the size of an internal block at the given index of a block group.
  `static long`
  `getInternalBlockLength(long dataSize, ErasureCodingPolicy ecPolicy, int idxInBlockGroup)`
  `static StripedBlockUtil.StripingChunkReadResult`
  `getNextCompletedStripedRead(CompletionService<StripedBlockUtil.BlockReadStats> readService, Map<Future<StripedBlockUtil.BlockReadStats>,Integer> futures, long timeoutMillis)`
  Get the next completed striped read task.
  `static long`
  `getSafeLength(ErasureCodingPolicy ecPolicy, long[] blockLens)`
  Compute the safe length given the internal block lengths.
  `static long`
  `offsetInBlkToOffsetInBG(int cellSize, int dataBlkNum, long offsetInBlk, int idxInBlockGroup)`
  Given a byte's offset in an internal block, calculate the offset in the block group.
  `static LocatedBlock[]`
  `parseStripedBlockGroup(LocatedStripedBlock bg, int cellSize, int dataBlkNum, int parityBlkNum)`
  This method parses a striped block group into individual blocks.
  `static long`
  `spaceConsumedByStripedBlock(long numDataBlkBytes, int dataBlkNum, int parityBlkNum, int cellSize)`
  Get the total usage of the striped blocks, which is the total of data blocks and parity blocks.

  ### Methods inherited from class java.lang.Object

  `clone, equals, finalize, getClass, hashCode, notify, notifyAll, toString, wait, wait, wait`

- ## Field Details

  - ### LOG

    public static final org.slf4j.Logger LOG

- ## Constructor Details

  - ### StripedBlockUtil

    public StripedBlockUtil()

- ## Method Details

  - ### parseStripedBlockGroup

    public static LocatedBlock[] parseStripedBlockGroup(LocatedStripedBlock bg, int cellSize, int dataBlkNum, int parityBlkNum)
    This method parses a striped block group into individual blocks.
    Parameters:
      `bg` - The striped block group
      `cellSize` - The size of a striping cell
      `dataBlkNum` - The number of data blocks
    Returns:
      An array containing the blocks in the group

  - ### constructInternalBlock

    public static LocatedBlock constructInternalBlock(LocatedStripedBlock bg, int idxInReturnedLocs, int cellSize, int dataBlkNum, int idxInBlockGroup)
    This method creates an internal block at the given index of a block group.
    Parameters:
      `idxInReturnedLocs` - The index in the stored locations in the `LocatedStripedBlock` object
      `idxInBlockGroup` - The logical index in the striped block group
    Returns:
      The constructed internal block

  - ### constructInternalBlock

    public static ExtendedBlock constructInternalBlock(ExtendedBlock blockGroup, ErasureCodingPolicy ecPolicy, int idxInBlockGroup)

  - ### constructInternalBlock

    public static ExtendedBlock constructInternalBlock(ExtendedBlock blockGroup, int cellSize, int dataBlkNum, int idxInBlockGroup)
    This method creates an internal `ExtendedBlock` at the given index of a block group.

  - ### getInternalBlockLength

    public static long getInternalBlockLength(long dataSize, ErasureCodingPolicy ecPolicy, int idxInBlockGroup)

  - ### getInternalBlockLength

    public static long getInternalBlockLength(long dataSize, int cellSize, int numDataBlocks, int idxInBlockGroup)
    Get the size of an internal block at the given index of a block group.
    Parameters:
      `dataSize` - Size of the block group only counting data blocks
      `cellSize` - The size of a striping cell
      `numDataBlocks` - The number of data blocks
      `idxInBlockGroup` - The logical index in the striped block group
    Returns:
      The size of the internal block at the specified index

  - ### getSafeLength

    public static long getSafeLength(ErasureCodingPolicy ecPolicy, long[] blockLens)
    Compute the safe length given the internal block lengths.
    Parameters:
      `ecPolicy` - The EC policy used for the block group
      `blockLens` - The lengths of internal blocks
    Returns:
      The safe length

  - ### offsetInBlkToOffsetInBG

    public static long offsetInBlkToOffsetInBG(int cellSize, int dataBlkNum, long offsetInBlk, int idxInBlockGroup)
    Given a byte's offset in an internal block, calculate the offset in the block group.

  - ### getNextCompletedStripedRead

    public static StripedBlockUtil.StripingChunkReadResult getNextCompletedStripedRead(CompletionService<StripedBlockUtil.BlockReadStats> readService, Map<Future<StripedBlockUtil.BlockReadStats>,Integer> futures, long timeoutMillis) throws InterruptedException
    Get the next completed striped read task.
    Returns:
      `StripedBlockUtil.StripingChunkReadResult` indicating the status of the read task succeeded, and the block index of the task. If the method times out without getting any completed read tasks, -1 is returned as block index.
    Throws:
      `InterruptedException`

  - ### spaceConsumedByStripedBlock

    public static long spaceConsumedByStripedBlock(long numDataBlkBytes, int dataBlkNum, int parityBlkNum, int cellSize)
    Get the total usage of the striped blocks, which is the total of data blocks and parity blocks.
    Parameters:
      `numDataBlkBytes` - Size of the block group only counting data blocks
      `dataBlkNum` - The number of data blocks
      `parityBlkNum` - The number of parity blocks
      `cellSize` - The size of a striping cell
    Returns:
      The total usage of data blocks and parity blocks

  - ### divideOneStripe

    public static StripedBlockUtil.AlignedStripe[] divideOneStripe(ErasureCodingPolicy ecPolicy, int cellSize, LocatedStripedBlock blockGroup, long rangeStartInBlockGroup, long rangeEndInBlockGroup, ByteBuffer buf)
    Similar functionality with `divideByteRangeIntoStripes(org.apache.hadoop.hdfs.protocol.ErasureCodingPolicy, int, org.apache.hadoop.hdfs.protocol.LocatedStripedBlock, long, long, java.nio.ByteBuffer)`, but is used by stateful read and uses ByteBuffer as reading target buffer. Besides the read range is within a single stripe thus the calculation logic is simpler.

  - ### divideByteRangeIntoStripes

    public static StripedBlockUtil.AlignedStripe[] divideByteRangeIntoStripes(ErasureCodingPolicy ecPolicy, int cellSize, LocatedStripedBlock blockGroup, long rangeStartInBlockGroup, long rangeEndInBlockGroup, ByteBuffer buf)
    This method divides a requested byte range into an array of inclusive `StripedBlockUtil.AlignedStripe`.
    Parameters:
      `ecPolicy` - The codec policy for the file, which carries the numbers of data / parity blocks
      `cellSize` - Cell size of stripe
      `blockGroup` - The striped block group
      `rangeStartInBlockGroup` - The byte range's start offset in block group
      `rangeEndInBlockGroup` - The byte range's end offset in block group
      `buf` - Destination buffer of the read operation for the byte range At most 5 stripes will be generated from each logical range, as demonstrated in the header of `StripedBlockUtil.AlignedStripe`.

  - ### checkBlocks

    public static void checkBlocks(ExtendedBlock blockGroup, int i, ExtendedBlock blocki) throws IOException
    Check if the information such as IDs and generation stamps in block-i match the block group.
    Throws:
      `IOException`

  - ### getBlockIndex

    public static int getBlockIndex(Block reportedBlock)