RouterRpcClient (Apache Hadoop HDFS-RBF 3.5.0 API)
Description: declaration: package: org.apache.hadoop.hdfs.server.federation.router, class: RouterRpcClient

Package org.apache.hadoop.hdfs.server.federation.router

# Class RouterRpcClient

java.lang.Object
org.apache.hadoop.hdfs.server.federation.router.RouterRpcClient

Direct Known Subclasses:
  `RouterAsyncRpcClient`

public class RouterRpcClient extends Object
A client proxy for Router to NN communication using the NN ClientProtocol.

Provides routers to invoke remote ClientProtocol methods and handle retries/failover.

- invokeSingle Make a single request to a single namespace
- invokeSequential Make a sequential series of requests to multiple ordered namespaces until a condition is met.
- invokeConcurrent Make concurrent requests to multiple namespaces and return all of the results.

Also maintains a cached pool of connections to NNs. Connections are managed by the ConnectionManager and are unique to each user + NN. The size of the connection pool can be configured. Larger pools allow for more simultaneous requests to a single NN from a single user.

- ## Nested Class Summary

  Nested Classes
  Modifier and Type
  Class
  Description
  `protected static class`
  `RouterRpcClient.ExecutionStatus`
  The `RouterRpcClient.ExecutionStatus` class is a utility class used to track the status of execution operations performed by the `RouterRpcClient`.

- ## Constructor Summary

  Constructors
  Constructor
  Description
  `RouterRpcClient(org.apache.hadoop.conf.Configuration conf, Router router, ActiveNamenodeResolver resolver, RouterRpcMonitor monitor, RouterStateIdContext routerStateIdContext)`
  Create a router RPC client to manage remote procedure calls to NNs.

- ## Method Summary

  Modifier and Type
  Method
  Description
  `protected void`
  `acquirePermit(String nsId, org.apache.hadoop.security.UserGroupInformation ugi, String methodName, RouterRpcFairnessPolicyController controller)`
  Acquire permit to continue processing the request for specific nsId.
  `protected void`
  `addClientInfoToCallerContext(org.apache.hadoop.security.UserGroupInformation ugi)`
  For tracking some information about the actual client.
  `Long`
  `getAcceptedPermitForNs(String ns)`
  `String`
  `getAcceptedPermitsPerNsJSON()`
  JSON representation of the accepted permits for each nameservice.
  `String`
  `getAsyncCallerPoolJson()`
  JSON representation of the async caller thread pool.
  `protected static IOException`
  `getCleanException(IOException ioe)`
  Get a clean copy of the exception.
  `protected ConnectionContext`
  `getConnection(org.apache.hadoop.security.UserGroupInformation ugi, String nsId, String rpcAddress, Class<?> proto)`
  Get ClientProtocol proxy client for a NameNode.
  `ThreadPoolExecutor`
  `getExecutorService()`
  Get the executor service used by invoking concurrent calls.
  `String`
  `getJSON()`
  JSON representation of the connection pool.
  `ActiveNamenodeResolver`
  `getNamenodeResolver()`
  Get the active namenode resolver used by this client.
  `int`
  `getNumActiveConnections()`
  Total number of available sockets between the router and NNs.
  `int`
  `getNumActiveConnectionsRecently()`
  Total number of active sockets between the router and NNs.
  `int`
  `getNumConnectionPools()`
  Total number of open connection pools to a NN.
  `int`
  `getNumConnections()`
  Total number of available sockets between the router and NNs.
  `int`
  `getNumCreatingConnections()`
  Number of connections between the router and NNs being created sockets.
  `int`
  `getNumIdleConnections()`
  Total number of idle sockets between the router and NNs.
  `List<? extends FederationNamenodeContext>`
  `getOrderedNamenodes(String nsId, boolean isObserverRead)`
  Get a prioritized list of NNs that share the same nameservice ID (in the same namespace).
  `Long`
  `getRejectedPermitForNs(String ns)`
  `String`
  `getRejectedPermitsPerNsJSON()`
  JSON representation of the rejected permits for each nameservice.
  `protected <T extends RemoteLocationContext, R>
  List<RemoteResult<T,R>>`
  `getRemoteResults(RemoteMethod method, long timeOutMs, RouterRpcFairnessPolicyController controller, List<T> orderedLocations, List<Callable<Object>> callables)`
  Invokes multiple concurrent proxy calls to different clients.
  `RouterRpcFairnessPolicyController`
  `getRouterRpcFairnessPolicyController()`
  `protected void`
  `handleInvokeMethodIOException(FederationNamenodeContext namenode, IOException ioe, RouterRpcClient.ExecutionStatus status, boolean useObserver)`
  The RPC request to the NameNode throws an exception, handle it according to the type of exception.
  `protected Object`
  `handlerAllNamenodeFail(List<? extends FederationNamenodeContext> namenodes, Method method, Map<FederationNamenodeContext,IOException> ioes, Object[] params)`
  All namenodes cannot successfully process the RPC request, throw corresponding exceptions according to the exception type of each namenode.
  `protected Object`
  `handlerInvokeException(FederationNamenodeContext namenode, Boolean listObserverFirst, int retryCount, Method method, Object obj, Throwable e, Object[] params)`
  Handle the exception when an RPC request to the NameNode throws an exception.
  `protected void`
  `incrAcceptedPermitForNs(String ns)`
  `protected void`
  `incrRejectedPermitForNs(String ns)`
  `protected void`
  `initConcurrentCallExecutorService(org.apache.hadoop.conf.Configuration conf)`
  `protected Object`
  `invoke(FederationNamenodeContext namenode, Boolean listObserverFirst, int retryCount, Method method, Object obj, Object... params)`
  Invokes a method on the designated object.
  `<T extends RemoteLocationContext>
  boolean`
  `invokeAll(Collection<T> locations, RemoteMethod method)`
  Invoke method in all locations and return success if any succeeds.
  `<T extends RemoteLocationContext, R>
  void`
  `invokeConcurrent(Collection<T> locations, RemoteMethod method)`
  Invoke multiple concurrent proxy calls to different clients.
  `<T extends RemoteLocationContext, R>
  void`
  `invokeConcurrent(Collection<T> locations, RemoteMethod method, boolean requireResponse, boolean standby)`
  Invoke multiple concurrent proxy calls to different clients.
  `<T extends RemoteLocationContext, R>
  Map<T,R>`
  `invokeConcurrent(Collection<T> locations, RemoteMethod method, boolean requireResponse, boolean standby, long timeOutMs, Class<R> clazz)`
  Invokes multiple concurrent proxy calls to different clients.
  `<T extends RemoteLocationContext, R>
  Map<T,R>`
  `invokeConcurrent(Collection<T> locations, RemoteMethod method, boolean requireResponse, boolean standby, Class<R> clazz)`
  Invokes multiple concurrent proxy calls to different clients.
  `<T extends RemoteLocationContext, R>
  List<RemoteResult<T,R>>`
  `invokeConcurrent(Collection<T> locations, RemoteMethod method, boolean standby, long timeOutMs, Class<R> clazz)`
  Invokes multiple concurrent proxy calls to different clients.
  `<T extends RemoteLocationContext, R>
  Map<T,R>`
  `invokeConcurrent(Collection<T> locations, RemoteMethod method, Class<R> clazz)`
  Invoke multiple concurrent proxy calls to different clients.
  `Object`
  `invokeMethod(org.apache.hadoop.security.UserGroupInformation ugi, List<? extends FederationNamenodeContext> namenodes, boolean useObserver, Class<?> protocol, Method method, Object... params)`
  Invokes a method against the ClientProtocol proxy server.
  `<T> T`
  `invokeSequential(List<? extends RemoteLocationContext> locations, RemoteMethod remoteMethod)`
  Invokes sequential proxy calls to different locations.
  `<T> T`
  `invokeSequential(List<? extends RemoteLocationContext> locations, RemoteMethod remoteMethod, Class<T> expectedResultClass, Object expectedResultValue)`
  Invokes sequential proxy calls to different locations.
  `<R extends RemoteLocationContext, T>
  RemoteResult`
  `invokeSequential(RemoteMethod remoteMethod, List<R> locations, Class<T> expectedResultClass, Object expectedResultValue)`
  Invokes sequential proxy calls to different locations.
  `Object`
  `invokeSingle(String nsId, RemoteMethod method)`
  Invokes a ClientProtocol method against the specified namespace.
  `<T> T`
  `invokeSingle(String nsId, RemoteMethod method, Class<T> clazz)`
  Invokes a remote method against the specified namespace.
  `<T> T`
  `invokeSingle(org.apache.hadoop.hdfs.protocol.ExtendedBlock extendedBlock, RemoteMethod method, Class<T> clazz)`
  Invokes a remote method against the specified extendedBlock.
  `<T> T`
  `invokeSingle(org.apache.hadoop.hdfs.protocol.ExtendedBlock block, RemoteMethod method, List<RemoteLocation> locations, Class<T> clazz)`
  Invokes a ClientProtocol method.
  `<T> T`
  `invokeSingle(RemoteLocationContext location, RemoteMethod remoteMethod, Class<T> clazz)`
  Invokes a single proxy call for a single location.
  `<T extends RemoteLocationContext, R>
  List<RemoteResult<T,R>>`
  `invokeSingle(T location, RemoteMethod method)`
  Invokes a ClientProtocol method against the specified namespace.
  `Object`
  `invokeSingleBlockPool(String bpId, RemoteMethod method)`
  Invokes a ClientProtocol method.
  `protected static boolean`
  `isExpectedClass(Class<?> expectedClass, Object clazz)`
  Checks if a result matches the required result class.
  `protected static boolean`
  `isExpectedValue(Object expectedValue, Object value)`
  Checks if a result matches the expected value.
  `boolean`
  `isNamespaceObserverReadEligible(String nsId)`
  Check if a namespace is eligible for observer reads.
  `protected boolean`
  `isObserverReadEligible(String nsId, Method method)`
  `static boolean`
  `isUnavailableException(IOException ioe)`
  Check if the exception comes from an unavailable subcluster.
  `protected static <T extends RemoteLocationContext, R>
  Map<T,R>`
  `postProcessResult(boolean requireResponse, List<RemoteResult<T,R>> results)`
  Post-process the results returned by `invokeConcurrent(Collection, RemoteMethod, boolean, long, Class)`.
  `protected void`
  `postProcessResult(Method method, RouterRpcClient.ExecutionStatus status, FederationNamenodeContext namenode, String nsId, org.apache.hadoop.hdfs.NameNodeProxiesClient.ProxyAndInfo<?> client)`
  The RPC request is successfully processed by the NameNode, the NameNode status in the router cache is updated according to the ExecutionStatus.
  `protected IOException`
  `processException(IOException ioe, RemoteLocationContext loc)`
  Exception messages might contain local subcluster paths.
  `protected <T extends RemoteLocationContext, R>
  List<RemoteResult<T,R>>`
  `processFutures(RemoteMethod method, Method m, List<T> orderedLocations, List<Future<Object>> futures)`
  Handle all futures during the invokeConcurrent call process.
  `String`
  `refreshFairnessPolicyController(org.apache.hadoop.conf.Configuration conf)`
  Refreshes/changes the fairness policy controller implementation if possible and returns the controller class name.
  `protected void`
  `releasePermit(String nsId, org.apache.hadoop.security.UserGroupInformation ugi, RemoteMethod m, RouterRpcFairnessPolicyController controller)`
  Release permit for specific nsId after processing against downstream nsId is completed.
  `protected org.apache.hadoop.io.retry.RetryPolicy.RetryAction.RetryDecision`
  `shouldRetry(IOException ioe, int retryCount, String nsId, FederationNamenodeContext namenode, boolean listObserverFirst)`
  If we should retry the RPC call.
  `protected boolean`
  `shouldRotateCache(IOException ioe)`
  Determine whether router rotated cache is required when NoNamenodesAvailableException occurs.
  `void`
  `shutdown()`
  Shutdown the client.
  `protected void`
  `transferThreadLocalContext(org.apache.hadoop.ipc.Server.Call originCall, org.apache.hadoop.ipc.CallerContext originContext)`
  Transfer origin thread local context which is necessary to current worker thread when invoking method concurrently by executor service.

  ### Methods inherited from class java.lang.Object

  `clone, equals, finalize, getClass, hashCode, notify, notifyAll, toString, wait, wait, wait`

- ## Constructor Details

  - ### RouterRpcClient

    public RouterRpcClient(org.apache.hadoop.conf.Configuration conf, Router router, ActiveNamenodeResolver resolver, RouterRpcMonitor monitor, RouterStateIdContext routerStateIdContext)
    Create a router RPC client to manage remote procedure calls to NNs.
    Parameters:
      `conf` - Hdfs Configuration.
      `router` - A router using this RPC client.
      `resolver` - A NN resolver to determine the currently active NN in HA.
      `monitor` - Optional performance monitor.
      `routerStateIdContext` - the router state context object to hold the state ids for all namespaces.

- ## Method Details

  - ### initConcurrentCallExecutorService

    protected void initConcurrentCallExecutorService(org.apache.hadoop.conf.Configuration conf)

  - ### getNamenodeResolver

    public ActiveNamenodeResolver getNamenodeResolver()
    Get the active namenode resolver used by this client.
    Returns:
      Active namenode resolver.

  - ### getExecutorService

    @VisibleForTesting public ThreadPoolExecutor getExecutorService()
    Get the executor service used by invoking concurrent calls.
    Returns:
      the executor service.

  - ### shutdown

    public void shutdown()
    Shutdown the client.

  - ### getNumConnections

    public int getNumConnections()
    Total number of available sockets between the router and NNs.
    Returns:
      Number of namenode clients.

  - ### getNumActiveConnections

    public int getNumActiveConnections()
    Total number of available sockets between the router and NNs.
    Returns:
      Number of namenode clients.

  - ### getNumIdleConnections

    public int getNumIdleConnections()
    Total number of idle sockets between the router and NNs.
    Returns:
      Number of namenode clients.

  - ### getNumActiveConnectionsRecently

    public int getNumActiveConnectionsRecently()
    Total number of active sockets between the router and NNs.
    Returns:
      Number of recently active namenode clients.

  - ### getNumConnectionPools

    public int getNumConnectionPools()
    Total number of open connection pools to a NN. Each connection pool. represents one user + one NN.
    Returns:
      Number of connection pools.

  - ### getNumCreatingConnections

    public int getNumCreatingConnections()
    Number of connections between the router and NNs being created sockets.
    Returns:
      Number of connections waiting to be created.

  - ### getJSON

    public String getJSON()
    JSON representation of the connection pool.
    Returns:
      String representation of the JSON.

  - ### getAsyncCallerPoolJson

    public String getAsyncCallerPoolJson()
    JSON representation of the async caller thread pool.
    Returns:
      String representation of the JSON.

  - ### getRejectedPermitsPerNsJSON

    public String getRejectedPermitsPerNsJSON()
    JSON representation of the rejected permits for each nameservice.
    Returns:
      String representation of the rejected permits for each nameservice.

  - ### getAcceptedPermitsPerNsJSON

    public String getAcceptedPermitsPerNsJSON()
    JSON representation of the accepted permits for each nameservice.
    Returns:
      String representation of the accepted permits for each nameservice.

  - ### getConnection

    protected ConnectionContext getConnection(org.apache.hadoop.security.UserGroupInformation ugi, String nsId, String rpcAddress, Class<?> proto) throws IOException
    Get ClientProtocol proxy client for a NameNode. Each combination of user + NN must use a unique proxy client. Previously created clients are cached and stored in a connection pool by the ConnectionManager.
    Parameters:
      `ugi` - User group information.
      `nsId` - Nameservice identifier.
      `rpcAddress` - RPC server address of the NN.
      `proto` - Protocol of the connection.
    Returns:
      ConnectionContext containing a ClientProtocol proxy client for the NN + current user.
    Throws:
      `IOException` - If we cannot get a connection to the NameNode.

  - ### shouldRetry

    protected org.apache.hadoop.io.retry.RetryPolicy.RetryAction.RetryDecision shouldRetry(IOException ioe, int retryCount, String nsId, FederationNamenodeContext namenode, boolean listObserverFirst) throws IOException
    If we should retry the RPC call.
    Parameters:
      `ioe` - IOException reported.
      `retryCount` - Number of retries.
      `nsId` - Nameservice ID.
      `namenode` - namenode context.
      `listObserverFirst` - Observer read case, observer NN will be ranked first.
    Returns:
      Retry decision.
    Throws:
      `IOException` - An IO Error occurred.

  - ### invokeMethod

    @VisibleForTesting public Object invokeMethod(org.apache.hadoop.security.UserGroupInformation ugi, List<? extends FederationNamenodeContext> namenodes, boolean useObserver, Class<?> protocol, Method method, Object... params) throws ConnectException, org.apache.hadoop.ipc.StandbyException, IOException
    Invokes a method against the ClientProtocol proxy server. If a standby exception is generated by the call to the client, retries using the alternate server.

    Re-throws exceptions generated by the remote RPC call as either RemoteException or IOException.

    Parameters:
      `ugi` - User group information.
      `namenodes` - A prioritized list of namenodes within the same nameservice.
      `useObserver` - Whether to use observer namenodes.
      `protocol` - the protocol of the connection.
      `method` - Remote ClientProtocol method to invoke.
      `params` - Variable list of parameters matching the method.
    Returns:
      The result of invoking the method.
    Throws:
      `ConnectException` - If it cannot connect to any Namenode.
      `org.apache.hadoop.ipc.StandbyException` - If all Namenodes are in Standby.
      `IOException` - If it cannot invoke the method.

  - ### handlerAllNamenodeFail

    protected Object handlerAllNamenodeFail(List<? extends FederationNamenodeContext> namenodes, Method method, Map<FederationNamenodeContext,IOException> ioes, Object[] params) throws IOException
    All namenodes cannot successfully process the RPC request, throw corresponding exceptions according to the exception type of each namenode.
    Parameters:
      `namenodes` - A prioritized list of namenodes within the same nameservice.
      `method` - Remote ClientProtocol method to invoke.
      `ioes` - The exception type of each namenode.
      `params` - Variable list of parameters matching the method.
    Returns:
      null
    Throws:
      `IOException` - Corresponding IOException according to the exception type of each namenode.

  - ### postProcessResult

    protected void postProcessResult(Method method, RouterRpcClient.ExecutionStatus status, FederationNamenodeContext namenode, String nsId, org.apache.hadoop.hdfs.NameNodeProxiesClient.ProxyAndInfo<?> client) throws IOException
    The RPC request is successfully processed by the NameNode, the NameNode status in the router cache is updated according to the ExecutionStatus.
    Parameters:
      `method` - Remote method to invoke.
      `status` - Current execution status.
      `namenode` - The namenode that successfully processed this RPC request.
      `nsId` - Nameservice ID.
      `client` - Connection client.
    Throws:
      `IOException` - If the state store cannot be accessed.

  - ### handleInvokeMethodIOException

    protected void handleInvokeMethodIOException(FederationNamenodeContext namenode, IOException ioe, RouterRpcClient.ExecutionStatus status, boolean useObserver) throws IOException
    The RPC request to the NameNode throws an exception, handle it according to the type of exception.
    Parameters:
      `namenode` - The namenode that processed this RPC request.
      `ioe` - The exception thrown by this RPC request.
      `status` - The current execution status.
      `useObserver` - Whether to use observer namenodes.
    Throws:
      `IOException` - If it cannot invoke the method.

  - ### addClientInfoToCallerContext

    protected void addClientInfoToCallerContext(org.apache.hadoop.security.UserGroupInformation ugi)
    For tracking some information about the actual client. It adds trace info "clientIp:ip", "clientPort:port", "clientId:id", "clientCallId:callId" and "realUser:userName" in the caller context, removing the old values if they were already present.
    Parameters:
      `ugi` - User group information.

  - ### invoke

    protected Object invoke(FederationNamenodeContext namenode, Boolean listObserverFirst, int retryCount, Method method, Object obj, Object... params) throws IOException
    Invokes a method on the designated object. Catches exceptions specific to the invocation.

    Re-throws exceptions generated by the remote RPC call as either RemoteException or IOException.

    Parameters:
      `namenode` - namenode context.
      `listObserverFirst` - Observer read case, observer NN will be ranked first.
      `retryCount` - Current retry times
      `method` - Method to invoke
      `obj` - Target object for the method
      `params` - Variable parameters
    Returns:
      Response from the remote server
    Throws:
      `IOException` - If error occurs.

  - ### handlerInvokeException

    protected Object handlerInvokeException(FederationNamenodeContext namenode, Boolean listObserverFirst, int retryCount, Method method, Object obj, Throwable e, Object[] params) throws IOException
    Handle the exception when an RPC request to the NameNode throws an exception.
    Parameters:
      `namenode` - namenode context.
      `listObserverFirst` - Observer read case, observer NN will be ranked first.
      `retryCount` - Current retry times
      `method` - Method to invoke
      `obj` - Target object for the method
      `e` - The exception thrown by the current invocation.
      `params` - Variable parameters
    Returns:
      Response from the remote server
    Throws:
      `IOException` - If error occurs.

  - ### isUnavailableException

    public static boolean isUnavailableException(IOException ioe)
    Check if the exception comes from an unavailable subcluster.
    Parameters:
      `ioe` - IOException to check.
    Returns:
      If the exception comes from an unavailable subcluster.

  - ### getCleanException

    protected static IOException getCleanException(IOException ioe)
    Get a clean copy of the exception. Sometimes the exceptions returned by the server contain the full stack trace in the message.
    Parameters:
      `ioe` - Exception to clean up.
    Returns:
      Copy of the original exception with a clean message.

  - ### invokeSingle

    public <T> T invokeSingle(org.apache.hadoop.hdfs.protocol.ExtendedBlock block, RemoteMethod method, List<RemoteLocation> locations, Class<T> clazz) throws IOException
    Invokes a ClientProtocol method. Determines the target nameservice via a provided block.

    Re-throws exceptions generated by the remote RPC call as either RemoteException or IOException.

    Type Parameters:
      `T` - The type of the remote method return.
    Parameters:
      `block` - Block used to determine appropriate nameservice.
      `method` - The remote method and parameters to invoke.
      `locations` - The remote locations will be used.
      `clazz` - Class for the return type.
    Returns:
      The result of invoking the method.
    Throws:
      `IOException` - If the invoke generated an error.

  - ### invokeSingleBlockPool

    public Object invokeSingleBlockPool(String bpId, RemoteMethod method) throws IOException
    Invokes a ClientProtocol method. Determines the target nameservice using the block pool id.

    Re-throws exceptions generated by the remote RPC call as either RemoteException or IOException.

    Parameters:
      `bpId` - Block pool identifier.
      `method` - The remote method and parameters to invoke.
    Returns:
      The result of invoking the method.
    Throws:
      `IOException` - If the invoke generated an error.

  - ### invokeSingle

    public Object invokeSingle(String nsId, RemoteMethod method) throws IOException
    Invokes a ClientProtocol method against the specified namespace.

    Re-throws exceptions generated by the remote RPC call as either RemoteException or IOException.

    Parameters:
      `nsId` - Target namespace for the method.
      `method` - The remote method and parameters to invoke.
    Returns:
      The result of invoking the method.
    Throws:
      `IOException` - If the invoke generated an error.

  - ### invokeSingle

    public <T> T invokeSingle(String nsId, RemoteMethod method, Class<T> clazz) throws IOException
    Invokes a remote method against the specified namespace.

    Re-throws exceptions generated by the remote RPC call as either RemoteException or IOException.

    Type Parameters:
      `T` - The type of the remote method return.
    Parameters:
      `nsId` - Target namespace for the method.
      `method` - The remote method and parameters to invoke.
      `clazz` - Class for the return type.
    Returns:
      The result of invoking the method.
    Throws:
      `IOException` - If the invoke generated an error.

  - ### invokeSingle

    public <T> T invokeSingle(org.apache.hadoop.hdfs.protocol.ExtendedBlock extendedBlock, RemoteMethod method, Class<T> clazz) throws IOException
    Invokes a remote method against the specified extendedBlock.

    Re-throws exceptions generated by the remote RPC call as either RemoteException or IOException.

    Type Parameters:
      `T` - The type of the remote method return.
    Parameters:
      `extendedBlock` - Target extendedBlock for the method.
      `method` - The remote method and parameters to invoke.
      `clazz` - Class for the return type.
    Returns:
      The result of invoking the method.
    Throws:
      `IOException` - If the invoke generated an error.

  - ### invokeSingle

    public <T> T invokeSingle(RemoteLocationContext location, RemoteMethod remoteMethod, Class<T> clazz) throws IOException
    Invokes a single proxy call for a single location.

    Re-throws exceptions generated by the remote RPC call as either RemoteException or IOException.

    Type Parameters:
      `T` - The type of the remote method return.
    Parameters:
      `location` - RemoteLocation to invoke.
      `remoteMethod` - The remote method and parameters to invoke.
      `clazz` - Class for the return type.
    Returns:
      The result of invoking the method if successful.
    Throws:
      `IOException` - If the invoke generated an error.

  - ### invokeSequential

    public <T> T invokeSequential(List<? extends RemoteLocationContext> locations, RemoteMethod remoteMethod) throws IOException
    Invokes sequential proxy calls to different locations. Continues to invoke calls until a call returns without throwing a remote exception.
    Type Parameters:
      `T` - The type of the remote method return.
    Parameters:
      `locations` - List of locations/nameservices to call concurrently.
      `remoteMethod` - The remote method and parameters to invoke.
    Returns:
      The result of the first successful call, or if no calls are successful, the result of the last RPC call executed.
    Throws:
      `IOException` - if the success condition is not met and one of the RPC calls generated a remote exception.

  - ### invokeSequential

    public <T> T invokeSequential(List<? extends RemoteLocationContext> locations, RemoteMethod remoteMethod, Class<T> expectedResultClass, Object expectedResultValue) throws IOException
    Invokes sequential proxy calls to different locations. Continues to invoke calls until the success condition is met, or until all locations have been attempted. The success condition may be specified by:
    - An expected result class
    - An expected result value
    If no expected result class/values are specified, the success condition is a call that does not throw a remote exception.
    Type Parameters:
      `T` - The type of the remote method return.
    Parameters:
      `locations` - List of locations/nameservices to call concurrently.
      `remoteMethod` - The remote method and parameters to invoke.
      `expectedResultClass` - In order to be considered a positive result, the return type must be of this class.
      `expectedResultValue` - In order to be considered a positive result, the return value must equal the value of this object.
    Returns:
      The result of the first successful call, or if no calls are successful, the result of the first RPC call executed.
    Throws:
      `IOException` - if the success condition is not met, return the first remote exception generated.

  - ### invokeSequential

    public <R extends RemoteLocationContext, T> RemoteResult invokeSequential(RemoteMethod remoteMethod, List<R> locations, Class<T> expectedResultClass, Object expectedResultValue) throws IOException
    Invokes sequential proxy calls to different locations. Continues to invoke calls until the success condition is met, or until all locations have been attempted. The success condition may be specified by:
    - An expected result class
    - An expected result value
    If no expected result class/values are specified, the success condition is a call that does not throw a remote exception. This returns RemoteResult, which contains the invoked location as well as the result.
    Type Parameters:
      `R` - The type of the remote location.
      `T` - The type of the remote method return.
    Parameters:
      `remoteMethod` - The remote method and parameters to invoke.
      `locations` - List of locations/nameservices to call concurrently.
      `expectedResultClass` - In order to be considered a positive result, the return type must be of this class.
      `expectedResultValue` - In order to be considered a positive result, the return value must equal the value of this object.
    Returns:
      The result of the first successful call, or if no calls are successful, the result of the first RPC call executed, along with the invoked location in form of RemoteResult.
    Throws:
      `IOException` - if the success condition is not met, return the first remote exception generated.

  - ### processException

    protected IOException processException(IOException ioe, RemoteLocationContext loc)
    Exception messages might contain local subcluster paths. This method generates a new exception with the proper message.
    Parameters:
      `ioe` - Original IOException.
      `loc` - Location we are processing.
    Returns:
      Exception processed for federation.

  - ### isExpectedClass

    protected static boolean isExpectedClass(Class<?> expectedClass, Object clazz)
    Checks if a result matches the required result class.
    Parameters:
      `expectedClass` - Required result class, null to skip the check.
      `clazz` - The result to check.
    Returns:
      True if the result is an instance of the required class or if the expected class is null.

  - ### isExpectedValue

    protected static boolean isExpectedValue(Object expectedValue, Object value)
    Checks if a result matches the expected value.
    Parameters:
      `expectedValue` - The expected value, null to skip the check.
      `value` - The result to check.
    Returns:
      True if the result is equals to the expected value or if the expected value is null.

  - ### invokeAll

    public <T extends RemoteLocationContext> boolean invokeAll(Collection<T> locations, RemoteMethod method) throws IOException
    Invoke method in all locations and return success if any succeeds.
    Type Parameters:
      `T` - The type of the remote location.
    Parameters:
      `locations` - List of remote locations to call concurrently.
      `method` - The remote method and parameters to invoke.
    Returns:
      If the call succeeds in any location.
    Throws:
      `IOException` - If any of the calls return an exception.

  - ### invokeConcurrent

    public <T extends RemoteLocationContext, R> void invokeConcurrent(Collection<T> locations, RemoteMethod method) throws IOException
    Invoke multiple concurrent proxy calls to different clients. Returns an array of results.

    Re-throws exceptions generated by the remote RPC call as either RemoteException or IOException.

    Type Parameters:
      `T` - The type of the remote location.
      `R` - The type of the remote method return.
    Parameters:
      `locations` - List of remote locations to call concurrently.
      `method` - The remote method and parameters to invoke.
    Throws:
      `IOException` - If all the calls throw an exception.

  - ### invokeConcurrent

    public <T extends RemoteLocationContext, R> Map<T,R> invokeConcurrent(Collection<T> locations, RemoteMethod method, Class<R> clazz) throws IOException
    Invoke multiple concurrent proxy calls to different clients. Returns an array of results.

    Re-throws exceptions generated by the remote RPC call as either RemoteException or IOException.

    Type Parameters:
      `T` - The type of the remote location.
      `R` - The type of the remote method return.
    Parameters:
      `locations` - List of remote locations to call concurrently.
      `method` - The remote method and parameters to invoke.
      `clazz` - Type of the remote return type.
    Returns:
      Result of invoking the method per subcluster: nsId to result.
    Throws:
      `IOException` - If all the calls throw an exception.

  - ### invokeConcurrent

    public <T extends RemoteLocationContext, R> void invokeConcurrent(Collection<T> locations, RemoteMethod method, boolean requireResponse, boolean standby) throws IOException
    Invoke multiple concurrent proxy calls to different clients. Returns an array of results.

    Re-throws exceptions generated by the remote RPC call as either RemoteException or IOException.

    Type Parameters:
      `T` - The type of the remote location.
      `R` - The type of the remote method return.
    Parameters:
      `locations` - List of remote locations to call concurrently.
      `method` - The remote method and parameters to invoke.
      `requireResponse` - If true an exception will be thrown if all calls do not complete. If false exceptions are ignored and all data results successfully received are returned.
      `standby` - If the requests should go to the standby namenodes too.
    Throws:
      `IOException` - If all the calls throw an exception.

  - ### invokeConcurrent

    public <T extends RemoteLocationContext, R> Map<T,R> invokeConcurrent(Collection<T> locations, RemoteMethod method, boolean requireResponse, boolean standby, Class<R> clazz) throws IOException
    Invokes multiple concurrent proxy calls to different clients. Returns an array of results.

    Re-throws exceptions generated by the remote RPC call as either RemoteException or IOException.

    Type Parameters:
      `T` - The type of the remote location.
      `R` - The type of the remote method return.
    Parameters:
      `locations` - List of remote locations to call concurrently.
      `method` - The remote method and parameters to invoke.
      `requireResponse` - If true an exception will be thrown if all calls do not complete. If false exceptions are ignored and all data results successfully received are returned.
      `standby` - If the requests should go to the standby namenodes too.
      `clazz` - Type of the remote return type.
    Returns:
      Result of invoking the method per subcluster: nsId to result.
    Throws:
      `IOException` - If requiredResponse=true and any of the calls throw an exception.

  - ### invokeConcurrent

    public <T extends RemoteLocationContext, R> Map<T,R> invokeConcurrent(Collection<T> locations, RemoteMethod method, boolean requireResponse, boolean standby, long timeOutMs, Class<R> clazz) throws IOException
    Invokes multiple concurrent proxy calls to different clients. Returns an array of results.

    Re-throws exceptions generated by the remote RPC call as either RemoteException or IOException.

    Type Parameters:
      `T` - The type of the remote location.
      `R` - The type of the remote method return.
    Parameters:
      `locations` - List of remote locations to call concurrently.
      `method` - The remote method and parameters to invoke.
      `requireResponse` - If true an exception will be thrown if all calls do not complete. If false exceptions are ignored and all data results successfully received are returned.
      `standby` - If the requests should go to the standby namenodes too.
      `timeOutMs` - Timeout for each individual call.
      `clazz` - Type of the remote return type.
    Returns:
      Result of invoking the method per subcluster: nsId to result.
    Throws:
      `IOException` - If requiredResponse=true and any of the calls throw an exception.

  - ### postProcessResult

    protected static <T extends RemoteLocationContext, R> Map<T,R> postProcessResult(boolean requireResponse, List<RemoteResult<T,R>> results) throws IOException
    Post-process the results returned by `invokeConcurrent(Collection, RemoteMethod, boolean, long, Class)`.
    Type Parameters:
      `T` - The type of the remote location.
      `R` - The type of the remote method return.
    Parameters:
      `requireResponse` - If true an exception will be thrown if all calls do not complete. If false exceptions are ignored and all data results successfully received are returned.
      `results` - Result of invoking the method per subcluster (list of results), This includes the exception for each remote location.
    Returns:
      Result of invoking the method per subcluster: nsId to result.
    Throws:
      `IOException` - If requiredResponse=true and any of the calls throw an exception.

  - ### invokeConcurrent

    public <T extends RemoteLocationContext, R> List<RemoteResult<T,R>> invokeConcurrent(Collection<T> locations, RemoteMethod method, boolean standby, long timeOutMs, Class<R> clazz) throws IOException
    Invokes multiple concurrent proxy calls to different clients. Returns an array of results.

    Re-throws exceptions generated by the remote RPC call as either RemoteException or IOException.

    Type Parameters:
      `T` - The type of the remote location.
      `R` - The type of the remote method return
    Parameters:
      `locations` - List of remote locations to call concurrently.
      `method` - The remote method and parameters to invoke.
      `standby` - If the requests should go to the standby namenodes too.
      `timeOutMs` - Timeout for each individual call.
      `clazz` - Type of the remote return type.
    Returns:
      Result of invoking the method per subcluster (list of results). This includes the exception for each remote location.
    Throws:
      `IOException` - If there are errors invoking the method.

  - ### getRemoteResults

    protected <T extends RemoteLocationContext, R> List<RemoteResult<T,R>> getRemoteResults(RemoteMethod method, long timeOutMs, RouterRpcFairnessPolicyController controller, List<T> orderedLocations, List<Callable<Object>> callables) throws IOException
    Invokes multiple concurrent proxy calls to different clients. Returns an array of results.
    Type Parameters:
      `T` - The type of the remote location.
      `R` - The type of the remote method return.
    Parameters:
      `method` - The remote method and parameters to invoke.
      `timeOutMs` - Timeout for each individual call.
      `controller` - Fairness manager to control handlers assigned per NS.
      `orderedLocations` - List of remote locations to call concurrently.
      `callables` - Invoke method for each NameNode.
    Returns:
      Result of invoking the method per subcluster (list of results), This includes the exception for each remote location.
    Throws:
      `IOException` - If there are errors invoking the method.

  - ### processFutures

    protected <T extends RemoteLocationContext, R> List<RemoteResult<T,R>> processFutures(RemoteMethod method, Method m, List<T> orderedLocations, List<Future<Object>> futures) throws InterruptedException
    Handle all futures during the invokeConcurrent call process.
    Type Parameters:
      `T` - The type of the remote location.
      `R` - The type of the remote method return.
    Parameters:
      `method` - The remote method and parameters to invoke.
      `m` - The method to invoke.
      `orderedLocations` - List of remote locations to call concurrently.
      `futures` - all futures during the invokeConcurrent call process.
    Returns:
      Result of invoking the method per subcluster (list of results), This includes the exception for each remote location.
    Throws:
      `InterruptedException` - if the current thread was interrupted while waiting.

  - ### invokeSingle

    public <T extends RemoteLocationContext, R> List<RemoteResult<T,R>> invokeSingle(T location, RemoteMethod method) throws IOException
    Invokes a ClientProtocol method against the specified namespace.

    Re-throws exceptions generated by the remote RPC call as either RemoteException or IOException.

    Type Parameters:
      `T` - The type of the remote location.
      `R` - The type of the remote method return.
    Parameters:
      `location` - RemoteLocation to invoke.
      `method` - The remote method and parameters to invoke.
    Returns:
      Result of invoking the method per subcluster (list of results), This includes the exception for each remote location.
    Throws:
      `IOException` - If there are errors invoking the method.

  - ### transferThreadLocalContext

    protected void transferThreadLocalContext(org.apache.hadoop.ipc.Server.Call originCall, org.apache.hadoop.ipc.CallerContext originContext)
    Transfer origin thread local context which is necessary to current worker thread when invoking method concurrently by executor service.
    Parameters:
      `originCall` - origin Call required for getting remote client ip.
      `originContext` - origin CallerContext which should be transferred to server side.

  - ### acquirePermit

    protected void acquirePermit(String nsId, org.apache.hadoop.security.UserGroupInformation ugi, String methodName, RouterRpcFairnessPolicyController controller) throws IOException
    Acquire permit to continue processing the request for specific nsId.
    Parameters:
      `nsId` - Identifier of the block pool.
      `ugi` - UserGroupIdentifier associated with the user.
      `methodName` - The name of remote method that needs to be invoked.
      `controller` - fairness policy controller to acquire permit from
    Throws:
      `IOException` - If permit could not be acquired for the nsId.

  - ### releasePermit

    protected void releasePermit(String nsId, org.apache.hadoop.security.UserGroupInformation ugi, RemoteMethod m, RouterRpcFairnessPolicyController controller)
    Release permit for specific nsId after processing against downstream nsId is completed.
    Parameters:
      `nsId` - Identifier of the block pool.
      `ugi` - UserGroupIdentifier associated with the user.
      `m` - Remote method that needs to be invoked.
      `controller` - fairness policy controller to release permit from

  - ### getRouterRpcFairnessPolicyController

    public RouterRpcFairnessPolicyController getRouterRpcFairnessPolicyController()

  - ### incrRejectedPermitForNs

    protected void incrRejectedPermitForNs(String ns)

  - ### getRejectedPermitForNs

    public Long getRejectedPermitForNs(String ns)

  - ### incrAcceptedPermitForNs

    protected void incrAcceptedPermitForNs(String ns)

  - ### getAcceptedPermitForNs

    public Long getAcceptedPermitForNs(String ns)

  - ### refreshFairnessPolicyController

    public String refreshFairnessPolicyController(org.apache.hadoop.conf.Configuration conf)
    Refreshes/changes the fairness policy controller implementation if possible and returns the controller class name.
    Parameters:
      `conf` - Configuration
    Returns:
      New controller class name if successfully refreshed, else old controller class name

  - ### getOrderedNamenodes

    public List<? extends FederationNamenodeContext> getOrderedNamenodes(String nsId, boolean isObserverRead) throws IOException
    Get a prioritized list of NNs that share the same nameservice ID (in the same namespace). In observer read case, OBSERVER NNs will be first in the list. Otherwise, ACTIVE NNs will be first in the list.
    Parameters:
      `nsId` - The nameservice ID for the namespace.
      `isObserverRead` - Read on observer namenode.
    Returns:
      A prioritized list of NNs to use for communication.
    Throws:
      `IOException` - If a NN cannot be located for the nameservice ID.

  - ### isObserverReadEligible

    protected boolean isObserverReadEligible(String nsId, Method method)

  - ### isNamespaceObserverReadEligible

    public boolean isNamespaceObserverReadEligible(String nsId)
    Check if a namespace is eligible for observer reads.
    Parameters:
      `nsId` - namespaceID
    Returns:
      whether the 'namespace' has observer reads enabled.

  - ### shouldRotateCache

    protected boolean shouldRotateCache(IOException ioe)
    Determine whether router rotated cache is required when NoNamenodesAvailableException occurs.
    Parameters:
      `ioe` - cause of the NoNamenodesAvailableException.
    Returns:
      true if NoNamenodesAvailableException occurs due to `unavailable exception`, otherwise false.