AMRMClientAsync (Apache Hadoop Main 3.5.0 API)
Description: declaration: package: org.apache.hadoop.yarn.client.api.async, class: AMRMClientAsync

Package org.apache.hadoop.yarn.client.api.async

# Class AMRMClientAsync<T extends org.apache.hadoop.yarn.client.api.AMRMClient.ContainerRequest>

java.lang.Object
org.apache.hadoop.service.AbstractService
org.apache.hadoop.yarn.client.api.async.AMRMClientAsync<T>

All Implemented Interfaces:
  `Closeable`, `AutoCloseable`, `Service`

@Public @Stable public abstract class AMRMClientAsync<T extends org.apache.hadoop.yarn.client.api.AMRMClient.ContainerRequest> extends AbstractService
`AMRMClientAsync` handles communication with the ResourceManager and provides asynchronous updates on events such as container allocations and completions. It contains a thread that sends periodic heartbeats to the ResourceManager. It should be used by implementing a CallbackHandler:

     class MyCallbackHandler extends AMRMClientAsync.AbstractCallbackHandler {
       public void onContainersAllocated(List<Container> containers) {
         [run tasks on the containers]
       }

       public void onContainersUpdated(List<Container> containers) {
         [determine if resource allocation of containers have been increased in
          the ResourceManager, and if so, inform the NodeManagers to increase the
          resource monitor/enforcement on the containers]
       }

       public void onContainersCompleted(List<ContainerStatus> statuses) {
         [update progress, check whether app is done]
       }

       public void onNodesUpdated(List<NodeReport> updated) {}

       public void onReboot() {}
     }

The client's lifecycle should be managed similarly to the following:

     AMRMClientAsync asyncClient =
         createAMRMClientAsync(appAttId, 1000, new MyCallbackhandler());
     asyncClient.init(conf);
     asyncClient.start();
     RegisterApplicationMasterResponse response = asyncClient
        .registerApplicationMaster(appMasterHostname, appMasterRpcPort,
           appMasterTrackingUrl);
     asyncClient.addContainerRequest(containerRequest);
     [... wait for application to complete]
     asyncClient.unregisterApplicationMaster(status, appMsg, trackingUrl);
     asyncClient.stop();

- ## Nested Class Summary

  Nested Classes
  Modifier and Type
  Class
  Description
  `static class`
  `org.apache.hadoop.yarn.client.api.async.AMRMClientAsync.AbstractCallbackHandler`
  The callback abstract class.
  `static interface`
  `org.apache.hadoop.yarn.client.api.async.AMRMClientAsync.CallbackHandler`
  Deprecated.
  Use `AMRMClientAsync.AbstractCallbackHandler` instead.

  ## Nested classes/interfaces inherited from interface org.apache.hadoop.service.Service

  `Service.STATE`

- ## Field Summary

  Fields
  Modifier and Type
  Field
  Description
  `protected final AMRMClient<T>`
  `client`
  `protected final org.apache.hadoop.yarn.client.api.async.AMRMClientAsync.CallbackHandler`
  `handler`
  `protected final AtomicInteger`
  `heartbeatIntervalMs`

- ## Constructor Summary

  Constructors
  Modifier
  Constructor
  Description
  `protected`
  `AMRMClientAsync(int intervalMs, org.apache.hadoop.yarn.client.api.async.AMRMClientAsync.AbstractCallbackHandler callbackHandler)`
  `protected`
  `AMRMClientAsync(int intervalMs, org.apache.hadoop.yarn.client.api.async.AMRMClientAsync.CallbackHandler callbackHandler)`
  Deprecated.
  `protected`
  `AMRMClientAsync(AMRMClient<T> client, int intervalMs, org.apache.hadoop.yarn.client.api.async.AMRMClientAsync.AbstractCallbackHandler callbackHandler)`
  `protected`
  `AMRMClientAsync(AMRMClient<T> client, int intervalMs, org.apache.hadoop.yarn.client.api.async.AMRMClientAsync.CallbackHandler callbackHandler)`
  Deprecated.

- ## Method Summary

  Modifier and Type
  Method
  Description
  `abstract void`
  `addContainerRequest(T req)`
  Request containers for resources before calling `allocate`
  `void`
  `addSchedulingRequests(Collection<SchedulingRequest> schedulingRequests)`
  Add a Collection of SchedulingRequests.
  `static <T extends org.apache.hadoop.yarn.client.api.AMRMClient.ContainerRequest>
  AMRMClientAsync<T>`
  `createAMRMClientAsync(int intervalMs, org.apache.hadoop.yarn.client.api.async.AMRMClientAsync.AbstractCallbackHandler callbackHandler)`
  Create a new instance of AMRMClientAsync.
  `static <T extends org.apache.hadoop.yarn.client.api.AMRMClient.ContainerRequest>
  AMRMClientAsync<T>`
  `createAMRMClientAsync(int intervalMs, org.apache.hadoop.yarn.client.api.async.AMRMClientAsync.CallbackHandler callbackHandler)`
  Deprecated.
  Use `createAMRMClientAsync(int, AMRMClientAsync.AbstractCallbackHandler)` instead.
  `static <T extends org.apache.hadoop.yarn.client.api.AMRMClient.ContainerRequest>
  AMRMClientAsync<T>`
  `createAMRMClientAsync(AMRMClient<T> client, int intervalMs, org.apache.hadoop.yarn.client.api.async.AMRMClientAsync.AbstractCallbackHandler callbackHandler)`
  Create a new instance of AMRMClientAsync.
  `static <T extends org.apache.hadoop.yarn.client.api.AMRMClient.ContainerRequest>
  AMRMClientAsync<T>`
  `createAMRMClientAsync(AMRMClient<T> client, int intervalMs, org.apache.hadoop.yarn.client.api.async.AMRMClientAsync.CallbackHandler callbackHandler)`
  Deprecated.
  Use `createAMRMClientAsync(AMRMClient, int, AMRMClientAsync.AbstractCallbackHandler)` instead.
  `abstract Resource`
  `getAvailableResources()`
  Get the currently available resources in the cluster.
  `abstract int`
  `getClusterNodeCount()`
  Get the current number of nodes in the cluster.
  `Collection<T>`
  `getMatchingRequests(long allocationRequestId)`
  Returns all matching ContainerRequests that match the given AllocationRequestId.
  `List<? extends Collection<T>>`
  `getMatchingRequests(Priority priority, String resourceName, ExecutionType executionType, Resource capability)`
  Returns all matching ContainerRequests that match the given Priority, ResourceName, ExecutionType and Capability.
  `abstract List<? extends Collection<T>>`
  `getMatchingRequests(Priority priority, String resourceName, Resource capability)`
  `org.apache.hadoop.yarn.client.api.TimelineV2Client`
  `getRegisteredTimelineV2Client()`
  Get registered timeline client.
  `abstract RegisterApplicationMasterResponse`
  `registerApplicationMaster(String appHostName, int appHostPort, String appTrackingUrl)`
  Registers this application master with the resource manager.
  `RegisterApplicationMasterResponse`
  `registerApplicationMaster(String appHostName, int appHostPort, String appTrackingUrl, Map<Set<String>,PlacementConstraint> placementConstraints)`
  Register the application master.
  `void`
  `registerTimelineV2Client(org.apache.hadoop.yarn.client.api.TimelineV2Client timelineClient)`
  Register TimelineClient to AMRMClient.
  `abstract void`
  `releaseAssignedContainer(ContainerId containerId)`
  Release containers assigned by the Resource Manager.
  `abstract void`
  `removeContainerRequest(T req)`
  Remove previous container request.
  `void`
  `requestContainerResourceChange(Container container, Resource capability)`
  Deprecated.
  use `requestContainerUpdate(Container, UpdateContainerRequest)`
  `abstract void`
  `requestContainerUpdate(Container container, UpdateContainerRequest updateContainerRequest)`
  Request a container update before calling `allocate`.
  `void`
  `setHeartbeatInterval(int interval)`
  `abstract void`
  `unregisterApplicationMaster(FinalApplicationStatus appStatus, String appMessage, String appTrackingUrl)`
  Unregister the application master.
  `abstract void`
  `updateBlacklist(List<String> blacklistAdditions, List<String> blacklistRemovals)`
  Update application's blacklist with addition or removal resources.
  `void`
  `updateTrackingUrl(String trackingUrl)`
  Update application's tracking url on next heartbeat.
  `void`
  `waitFor(Supplier<Boolean> check)`
  Wait for `check` to return true for each 1000 ms.
  `void`
  `waitFor(Supplier<Boolean> check, int checkEveryMillis)`
  Wait for `check` to return true for each `checkEveryMillis` ms.
  `void`
  `waitFor(Supplier<Boolean> check, int checkEveryMillis, int logInterval)`
  Wait for `check` to return true for each `checkEveryMillis` ms.

  ### Methods inherited from class org.apache.hadoop.service.AbstractService

  `close, getBlockers, getConfig, getFailureCause, getFailureState, getLifecycleHistory, getName, getServiceState, getStartTime, init, isInState, noteFailure, putBlocker, registerGlobalListener, registerServiceListener, removeBlocker, serviceInit, serviceStart, serviceStop, setConfig, start, stop, toString, unregisterGlobalListener, unregisterServiceListener, waitForServiceToStop`

  ### Methods inherited from class java.lang.Object

  `clone, equals, finalize, getClass, hashCode, notify, notifyAll, wait, wait, wait`

- ## Field Details

  - ### client

    protected final AMRMClient<T extends org.apache.hadoop.yarn.client.api.AMRMClient.ContainerRequest> client

  - ### handler

    protected final org.apache.hadoop.yarn.client.api.async.AMRMClientAsync.CallbackHandler handler

  - ### heartbeatIntervalMs

    protected final AtomicInteger heartbeatIntervalMs

- ## Constructor Details

  - ### AMRMClientAsync

    protected AMRMClientAsync(int intervalMs, org.apache.hadoop.yarn.client.api.async.AMRMClientAsync.AbstractCallbackHandler callbackHandler)

  - ### AMRMClientAsync

    @Private @VisibleForTesting protected AMRMClientAsync(AMRMClient<T> client, int intervalMs, org.apache.hadoop.yarn.client.api.async.AMRMClientAsync.AbstractCallbackHandler callbackHandler)

  - ### AMRMClientAsync

    @Deprecated protected AMRMClientAsync(int intervalMs, org.apache.hadoop.yarn.client.api.async.AMRMClientAsync.CallbackHandler callbackHandler)
    Deprecated.

  - ### AMRMClientAsync

    @Private @VisibleForTesting @Deprecated protected AMRMClientAsync(AMRMClient<T> client, int intervalMs, org.apache.hadoop.yarn.client.api.async.AMRMClientAsync.CallbackHandler callbackHandler)
    Deprecated.

- ## Method Details

  - ### createAMRMClientAsync

    public static <T extends org.apache.hadoop.yarn.client.api.AMRMClient.ContainerRequest> AMRMClientAsync<T> createAMRMClientAsync(int intervalMs, org.apache.hadoop.yarn.client.api.async.AMRMClientAsync.AbstractCallbackHandler callbackHandler)

    Create a new instance of AMRMClientAsync.

    Parameters:
      `intervalMs` - heartbeat interval in milliseconds between AM and RM
      `callbackHandler` - callback handler that processes responses from the `ResourceManager`

  - ### createAMRMClientAsync

    public static <T extends org.apache.hadoop.yarn.client.api.AMRMClient.ContainerRequest> AMRMClientAsync<T> createAMRMClientAsync(AMRMClient<T> client, int intervalMs, org.apache.hadoop.yarn.client.api.async.AMRMClientAsync.AbstractCallbackHandler callbackHandler)

    Create a new instance of AMRMClientAsync.

    Parameters:
      `client` - the AMRMClient instance
      `intervalMs` - heartbeat interval in milliseconds between AM and RM
      `callbackHandler` - callback handler that processes responses from the `ResourceManager`

  - ### createAMRMClientAsync

    @Deprecated public static <T extends org.apache.hadoop.yarn.client.api.AMRMClient.ContainerRequest> AMRMClientAsync<T> createAMRMClientAsync(int intervalMs, org.apache.hadoop.yarn.client.api.async.AMRMClientAsync.CallbackHandler callbackHandler)
    Deprecated.
    Use `createAMRMClientAsync(int, AMRMClientAsync.AbstractCallbackHandler)` instead.

  - ### createAMRMClientAsync

    @Deprecated public static <T extends org.apache.hadoop.yarn.client.api.AMRMClient.ContainerRequest> AMRMClientAsync<T> createAMRMClientAsync(AMRMClient<T> client, int intervalMs, org.apache.hadoop.yarn.client.api.async.AMRMClientAsync.CallbackHandler callbackHandler)
    Deprecated.
    Use `createAMRMClientAsync(AMRMClient, int, AMRMClientAsync.AbstractCallbackHandler)` instead.

  - ### setHeartbeatInterval

    public void setHeartbeatInterval(int interval)

  - ### getMatchingRequests

    public abstract List<? extends Collection<T>> getMatchingRequests(Priority priority, String resourceName, Resource capability)

  - ### addSchedulingRequests

    @Public @Unstable public void addSchedulingRequests(Collection<SchedulingRequest> schedulingRequests)
    Add a Collection of SchedulingRequests. The AMRMClient will ensure that all requests in the same batch are sent in the same allocate call.
    Parameters:
      `schedulingRequests` - Collection of Scheduling Requests.

  - ### getMatchingRequests

    public List<? extends Collection<T>> getMatchingRequests(Priority priority, String resourceName, ExecutionType executionType, Resource capability)
    Returns all matching ContainerRequests that match the given Priority, ResourceName, ExecutionType and Capability. NOTE: This matches only requests that were made by the client WITHOUT the allocationRequestId specified.
    Parameters:
      `priority` - Priority.
      `resourceName` - Location.
      `executionType` - ExecutionType.
      `capability` - Capability.
    Returns:
      All matching ContainerRequests

  - ### getMatchingRequests

    public Collection<T> getMatchingRequests(long allocationRequestId)
    Returns all matching ContainerRequests that match the given AllocationRequestId. NOTE: This matches only requests that were made by the client WITH the allocationRequestId specified.
    Parameters:
      `allocationRequestId` - AllocationRequestId.
    Returns:
      All matching ContainerRequests

  - ### registerApplicationMaster

    public abstract RegisterApplicationMasterResponse registerApplicationMaster(String appHostName, int appHostPort, String appTrackingUrl) throws YarnException, IOException
    Registers this application master with the resource manager. On successful registration, starts the heartbeating thread.
    Throws:
      `YarnException`
      `IOException`

  - ### registerApplicationMaster

    @Public @Unstable public RegisterApplicationMasterResponse registerApplicationMaster(String appHostName, int appHostPort, String appTrackingUrl, Map<Set<String>,PlacementConstraint> placementConstraints) throws YarnException, IOException
    Register the application master. This must be called before any other interaction
    Parameters:
      `appHostName` - Name of the host on which master is running
      `appHostPort` - Port master is listening on
      `appTrackingUrl` - URL at which the master info can be seen
      `placementConstraints` - Placement Constraints mappings.
    Returns:
      `RegisterApplicationMasterResponse`
    Throws:
      `YarnException`
      `IOException`

  - ### unregisterApplicationMaster

    public abstract void unregisterApplicationMaster(FinalApplicationStatus appStatus, String appMessage, String appTrackingUrl) throws YarnException, IOException
    Unregister the application master. This must be called in the end.
    Parameters:
      `appStatus` - Success/Failure status of the master
      `appMessage` - Diagnostics message on failure
      `appTrackingUrl` - New URL to get master info
    Throws:
      `YarnException`
      `IOException`

  - ### addContainerRequest

    public abstract void addContainerRequest(T req)
    Request containers for resources before calling `allocate`
    Parameters:
      `req` - Resource request

  - ### removeContainerRequest

    public abstract void removeContainerRequest(T req)
    Remove previous container request. The previous container request may have already been sent to the ResourceManager. So even after the remove request the app must be prepared to receive an allocation for the previous request even after the remove request
    Parameters:
      `req` - Resource request

  - ### requestContainerResourceChange

    @Deprecated public void requestContainerResourceChange(Container container, Resource capability)
    Deprecated.
    use `requestContainerUpdate(Container, UpdateContainerRequest)`
    Request container resource change before calling `allocate`. Any previous pending resource change request of the same container will be removed. Application that calls this method is expected to maintain the `Container`s that are returned from previous successful allocations or resource changes. By passing in the existing container and a target resource capability to this method, the application requests the ResourceManager to change the existing resource allocation to the target resource allocation.
    Parameters:
      `container` - The container returned from the last successful resource allocation or resource change
      `capability` - The target resource capability of the container

  - ### requestContainerUpdate

    public abstract void requestContainerUpdate(Container container, UpdateContainerRequest updateContainerRequest)
    Request a container update before calling `allocate`. Any previous pending update request of the same container will be removed.
    Parameters:
      `container` - The container returned from the last successful resource allocation or update
      `updateContainerRequest` - The `UpdateContainerRequest`.

  - ### releaseAssignedContainer

    public abstract void releaseAssignedContainer(ContainerId containerId)
    Release containers assigned by the Resource Manager. If the app cannot use the container or wants to give up the container then it can release them. The app needs to make new requests for the released resource capability if it still needs it. eg. it released non-local resources
    Parameters:
      `containerId` -

  - ### getAvailableResources

    public abstract Resource getAvailableResources()
    Get the currently available resources in the cluster. A valid value is available after a call to allocate has been made
    Returns:
      Currently available resources

  - ### getClusterNodeCount

    public abstract int getClusterNodeCount()
    Get the current number of nodes in the cluster. A valid values is available after a call to allocate has been made
    Returns:
      Current number of nodes in the cluster

  - ### registerTimelineV2Client

    public void registerTimelineV2Client(org.apache.hadoop.yarn.client.api.TimelineV2Client timelineClient) throws YarnException
    Register TimelineClient to AMRMClient.
    Parameters:
      `timelineClient` -
    Throws:
      `YarnException` - when this method is invoked even when ATS V2 is not configured.

  - ### getRegisteredTimelineV2Client

    public org.apache.hadoop.yarn.client.api.TimelineV2Client getRegisteredTimelineV2Client()
    Get registered timeline client.
    Returns:
      the registered timeline client

  - ### updateBlacklist

    public abstract void updateBlacklist(List<String> blacklistAdditions, List<String> blacklistRemovals)
    Update application's blacklist with addition or removal resources.
    Parameters:
      `blacklistAdditions` - list of resources which should be added to the application blacklist
      `blacklistRemovals` - list of resources which should be removed from the application blacklist

  - ### updateTrackingUrl

    @Public @Unstable public void updateTrackingUrl(String trackingUrl)
    Update application's tracking url on next heartbeat.
    Parameters:
      `trackingUrl` - new tracking url for this application

  - ### waitFor

    public void waitFor(Supplier<Boolean> check) throws InterruptedException
    Wait for `check` to return true for each 1000 ms. See also `waitFor(java.util.function.Supplier, int)` and `waitFor(java.util.function.Supplier, int, int)`
    Parameters:
      `check` - the condition for which it should wait
    Throws:
      `InterruptedException`

  - ### waitFor

    public void waitFor(Supplier<Boolean> check, int checkEveryMillis) throws InterruptedException
    Wait for `check` to return true for each `checkEveryMillis` ms. See also `waitFor(java.util.function.Supplier, int, int)`
    Parameters:
      `check` - user defined checker
      `checkEveryMillis` - interval to call `check`
    Throws:
      `InterruptedException`

  - ### waitFor

    public void waitFor(Supplier<Boolean> check, int checkEveryMillis, int logInterval) throws InterruptedException
    Wait for `check` to return true for each `checkEveryMillis` ms. In the main loop, this method will log the message "waiting in main loop" for each `logInterval` times iteration to confirm the thread is alive.
    Parameters:
      `check` - user defined checker
      `checkEveryMillis` - interval to call `check`
      `logInterval` - interval to log for each
    Throws:
      `InterruptedException`