StoragePolicySatisfier (Apache Hadoop HDFS 3.5.0 API)
Description: declaration: package: org.apache.hadoop.hdfs.server.namenode.sps, class: StoragePolicySatisfier

Package org.apache.hadoop.hdfs.server.namenode.sps

# Class StoragePolicySatisfier

java.lang.Object
org.apache.hadoop.hdfs.server.namenode.sps.StoragePolicySatisfier

All Implemented Interfaces:
  `Runnable`, `SPSService`

@Private public class StoragePolicySatisfier extends Object implements SPSService, Runnable
Setting storagePolicy on a file after the file write will only update the new storage policy type in Namespace, but physical block storage movement will not happen until user runs "Mover Tool" explicitly for such files. The StoragePolicySatisfier Daemon thread implemented for addressing the case where users may want to physically move the blocks by a dedicated daemon (can run inside Namenode or stand alone) instead of running mover tool explicitly. Just calling client API to satisfyStoragePolicy on a file/dir will automatically trigger to move its physical storage locations as expected in asynchronous manner. Here SPS will pick the file blocks which are expecting to change its storages, then it will build the mapping of source block location and expected storage type and location to move. After that this class will also prepare requests to send to Datanode for processing the physical block movements.

- ## Nested Class Summary

  Nested Classes
  Modifier and Type
  Class
  Description
  `static final class`
  `StoragePolicySatisfier.AttemptedItemInfo`
  This class contains information of an attempted blocks and its last attempted or reported time stamp.
  `static class`
  `StoragePolicySatisfier.DatanodeMap`
  Maintains storage type map with the available datanodes in the cluster.
  `static final class`
  `StoragePolicySatisfier.DatanodeWithStorage`
  Keeps datanode with its respective set of supported storage types.

- ## Field Summary

  Fields
  Modifier and Type
  Field
  Description
  `static final org.slf4j.Logger`
  `LOG`

- ## Constructor Summary

  Constructors
  Constructor
  Description
  `StoragePolicySatisfier(org.apache.hadoop.conf.Configuration conf)`

- ## Method Summary

  Modifier and Type
  Method
  Description
  `void`
  `addAllFilesToProcess(long startPath, List<ItemInfo> itemInfoList, boolean scanCompleted)`
  Adds all the Item information(file etc) to processing queue.
  `void`
  `addFileToProcess(ItemInfo trackInfo, boolean scanCompleted)`
  Adds the Item information(file etc) to processing queue.
  `void`
  `clearQueues()`
  Clear the queues from to be storage movement needed lists and items tracked in storage movement monitor.
  `BlockStorageMovementAttemptedItems`
  `getAttemptedItemsMonitor()`
  `org.apache.hadoop.conf.Configuration`
  `getConf()`
  `BlockStorageMovementNeeded`
  `getStorageMovementQueue()`
  `void`
  `init(Context context)`
  Initializes the helper services.
  `boolean`
  `isRunning()`
  Check whether StoragePolicySatisfier is running.
  `void`
  `join()`
  Join main SPS thread.
  `void`
  `markScanCompletedForPath(long inodeId)`
  Marks the scanning of directory if finished.
  `void`
  `notifyStorageMovementAttemptFinishedBlk(org.apache.hadoop.hdfs.protocol.DatanodeInfo dnInfo, org.apache.hadoop.fs.StorageType storageType, org.apache.hadoop.hdfs.protocol.Block block)`
  Receives storage movement attempt finished block report.
  `int`
  `processingQueueSize()`
  `void`
  `run()`
  `void`
  `start(org.apache.hadoop.hdfs.protocol.HdfsConstants.StoragePolicySatisfierMode serviceMode)`
  Start storage policy satisfier demon thread.
  `void`
  `stop(boolean forceStop)`
  Stops the SPS service.
  `void`
  `stopGracefully()`
  Stops the SPS service gracefully.

  ### Methods inherited from class java.lang.Object

  `clone, equals, finalize, getClass, hashCode, notify, notifyAll, toString, wait, wait, wait`

- ## Field Details

  - ### LOG

    public static final org.slf4j.Logger LOG

- ## Constructor Details

  - ### StoragePolicySatisfier

    public StoragePolicySatisfier(org.apache.hadoop.conf.Configuration conf)

- ## Method Details

  - ### init

    public void init(Context context)
    Description copied from interface: `SPSService`
    Initializes the helper services.
    Specified by:
      `init` in interface `SPSService`
    Parameters:
      `context` - - context is an helper service to provide communication channel between NN and SPS

  - ### start

    public void start(org.apache.hadoop.hdfs.protocol.HdfsConstants.StoragePolicySatisfierMode serviceMode)
    Start storage policy satisfier demon thread. Also start block storage movements monitor for retry the attempts if needed.
    Specified by:
      `start` in interface `SPSService`
    Parameters:
      `serviceMode` - sps service mode

  - ### stop

    public void stop(boolean forceStop)
    Description copied from interface: `SPSService`
    Stops the SPS service.
    Specified by:
      `stop` in interface `SPSService`
    Parameters:
      `forceStop` - true represents to clear all the sps path's hint, false otherwise.

  - ### stopGracefully

    public void stopGracefully()
    Description copied from interface: `SPSService`
    Stops the SPS service gracefully. Timed wait to stop storage policy satisfier daemon threads.
    Specified by:
      `stopGracefully` in interface `SPSService`

  - ### isRunning

    public boolean isRunning()
    Description copied from interface: `SPSService`
    Check whether StoragePolicySatisfier is running.
    Specified by:
      `isRunning` in interface `SPSService`
    Returns:
      true if running

  - ### run

    public void run()
    Specified by:
      `run` in interface `Runnable`

  - ### notifyStorageMovementAttemptFinishedBlk

    public void notifyStorageMovementAttemptFinishedBlk(org.apache.hadoop.hdfs.protocol.DatanodeInfo dnInfo, org.apache.hadoop.fs.StorageType storageType, org.apache.hadoop.hdfs.protocol.Block block)
    Receives storage movement attempt finished block report.
    Specified by:
      `notifyStorageMovementAttemptFinishedBlk` in interface `SPSService`
    Parameters:
      `dnInfo` - reported datanode
      `storageType` - - storage type
      `block` - movement attempt finished block.

  - ### getAttemptedItemsMonitor

    @VisibleForTesting public BlockStorageMovementAttemptedItems getAttemptedItemsMonitor()

  - ### clearQueues

    public void clearQueues()
    Clear the queues from to be storage movement needed lists and items tracked in storage movement monitor.

  - ### addFileToProcess

    public void addFileToProcess(ItemInfo trackInfo, boolean scanCompleted)
    Description copied from interface: `SPSService`
    Adds the Item information(file etc) to processing queue.
    Specified by:
      `addFileToProcess` in interface `SPSService`
    Parameters:
      `trackInfo` - file info object for which need to satisfy the policy

  - ### addAllFilesToProcess

    public void addAllFilesToProcess(long startPath, List<ItemInfo> itemInfoList, boolean scanCompleted)
    Description copied from interface: `SPSService`
    Adds all the Item information(file etc) to processing queue.
    Specified by:
      `addAllFilesToProcess` in interface `SPSService`
    Parameters:
      `startPath` - - directoryId/fileId, on which SPS was called.
      `itemInfoList` - - list of item infos
      `scanCompleted` - - whether the scanning of directory fully done with itemInfoList

  - ### processingQueueSize

    public int processingQueueSize()
    Specified by:
      `processingQueueSize` in interface `SPSService`
    Returns:
      current processing queue size.

  - ### getConf

    public org.apache.hadoop.conf.Configuration getConf()
    Specified by:
      `getConf` in interface `SPSService`
    Returns:
      the configuration.

  - ### getStorageMovementQueue

    @VisibleForTesting public BlockStorageMovementNeeded getStorageMovementQueue()

  - ### markScanCompletedForPath

    public void markScanCompletedForPath(long inodeId)
    Description copied from interface: `SPSService`
    Marks the scanning of directory if finished.
    Specified by:
      `markScanCompletedForPath` in interface `SPSService`
    Parameters:
      `inodeId` - - satisfier path id

  - ### join

    public void join() throws InterruptedException
    Join main SPS thread.
    Throws:
      `InterruptedException`