MRAppMaster (Apache Hadoop MapReduce App 3.5.0 API)
Description: declaration: package: org.apache.hadoop.mapreduce.v2.app, class: MRAppMaster

Package org.apache.hadoop.mapreduce.v2.app

# Class MRAppMaster

java.lang.Object
org.apache.hadoop.service.AbstractService
org.apache.hadoop.service.CompositeService
org.apache.hadoop.mapreduce.v2.app.MRAppMaster

All Implemented Interfaces:
  `Closeable`, `AutoCloseable`, `org.apache.hadoop.service.Service`

public class MRAppMaster extends org.apache.hadoop.service.CompositeService
The Map-Reduce Application Master. The state machine is encapsulated in the implementation of Job interface. All state changes happens via Job interface. Each event results in a Finite State Transition in Job. MR AppMaster is the composition of loosely coupled services. The services interact with each other via events. The components resembles the Actors model. The component acts on received event and send out the events to other components. This keeps it highly concurrent with no or minimal synchronization needs. The events are dispatched by a central Dispatch mechanism. All components register to the Dispatcher. The information is shared across different components using AppContext.

- ## Nested Class Summary

  Nested Classes
  Modifier and Type
  Class
  Description
  `class`
  `MRAppMaster.RunningAppContext`

  ## Nested classes/interfaces inherited from class org.apache.hadoop.service.CompositeService

  `org.apache.hadoop.service.CompositeService.CompositeServiceShutdownHook`

  ## Nested classes/interfaces inherited from interface org.apache.hadoop.service.Service

  `org.apache.hadoop.service.Service.STATE`

- ## Field Summary

  Fields
  Modifier and Type
  Field
  Description
  `protected org.apache.hadoop.security.UserGroupInformation`
  `currentUser`
  `static final String`
  `INTERMEDIATE_DATA_ENCRYPTION_ALGO`
  `protected boolean`
  `isLastAMRetry`
  `protected org.apache.hadoop.mapreduce.security.token.JobTokenSecretManager`
  `jobTokenSecretManager`
  `protected final MRAppMetrics`
  `metrics`
  `static final int`
  `SHUTDOWN_HOOK_PRIORITY`
  Priority of the MRAppMaster shutdown hook.
  `protected AtomicBoolean`
  `successfullyUnregistered`
  `protected TaskAttemptListener`
  `taskAttemptListener`

  ### Fields inherited from class org.apache.hadoop.service.CompositeService

  `STOP_ONLY_STARTED_SERVICES`

- ## Constructor Summary

  Constructors
  Constructor
  Description
  `MRAppMaster(org.apache.hadoop.yarn.api.records.ApplicationAttemptId applicationAttemptId, org.apache.hadoop.yarn.api.records.ContainerId containerId, String nmHost, int nmPort, int nmHttpPort, long appSubmitTime)`
  `MRAppMaster(org.apache.hadoop.yarn.api.records.ApplicationAttemptId applicationAttemptId, org.apache.hadoop.yarn.api.records.ContainerId containerId, String nmHost, int nmPort, int nmHttpPort, org.apache.hadoop.yarn.util.Clock clock, long appSubmitTime)`

- ## Method Summary

  Modifier and Type
  Method
  Description
  `void`
  `cleanupStagingDir()`
  clean up staging directories for the job.
  `protected ClientService`
  `createClientService(AppContext context)`
  `protected org.apache.hadoop.yarn.event.EventHandler<CommitterEvent>`
  `createCommitterEventHandler(AppContext context, org.apache.hadoop.mapreduce.OutputCommitter committer)`
  `protected ContainerAllocator`
  `createContainerAllocator(ClientService clientService, AppContext context)`
  `protected ContainerLauncher`
  `createContainerLauncher(AppContext context)`
  `protected org.apache.hadoop.yarn.event.Dispatcher`
  `createDispatcher()`
  `protected Job`
  `createJob(org.apache.hadoop.conf.Configuration conf, JobStateInternal forcedState, String diagnostic)`
  Create and initialize (but don't start) a single job.
  `protected org.apache.hadoop.yarn.event.EventHandler<JobFinishEvent>`
  `createJobFinishEventHandler()`
  create an event handler that handles the job finish event.
  `protected org.apache.hadoop.yarn.event.EventHandler<JobHistoryEvent>`
  `createJobHistoryHandler(AppContext context)`
  `protected AMPreemptionPolicy`
  `createPreemptionPolicy(org.apache.hadoop.conf.Configuration conf)`
  `protected Speculator`
  `createSpeculator(org.apache.hadoop.conf.Configuration conf, AppContext context)`
  `protected org.apache.hadoop.service.AbstractService`
  `createStagingDirCleaningService()`
  `protected TaskAttemptFinishingMonitor`
  `createTaskAttemptFinishingMonitor(org.apache.hadoop.yarn.event.EventHandler eventHandler)`
  `protected TaskAttemptListener`
  `createTaskAttemptListener(AppContext context, AMPreemptionPolicy preemptionPolicy)`
  `List<org.apache.hadoop.mapreduce.v2.api.records.AMInfo>`
  `getAllAMInfos()`
  `org.apache.hadoop.yarn.api.records.ApplicationId`
  `getAppID()`
  `org.apache.hadoop.yarn.api.records.ApplicationAttemptId`
  `getAttemptID()`
  `ClientService`
  `getClientService()`
  `org.apache.hadoop.mapreduce.OutputCommitter`
  `getCommitter()`
  `Map<org.apache.hadoop.mapreduce.v2.api.records.TaskId,org.apache.hadoop.mapreduce.jobhistory.JobHistoryParser.TaskInfo>`
  `getCompletedTaskFromPreviousRun()`
  `ContainerAllocator`
  `getContainerAllocator()`
  `ContainerLauncher`
  `getContainerLauncher()`
  `AppContext`
  `getContext()`
  `protected org.apache.hadoop.security.Credentials`
  `getCredentials()`
  `org.apache.hadoop.yarn.event.Dispatcher`
  `getDispatcher()`
  `protected org.apache.hadoop.fs.FileSystem`
  `getFileSystem(org.apache.hadoop.conf.Configuration conf)`
  Create the default file System for this job.
  `org.apache.hadoop.mapreduce.v2.api.records.JobId`
  `getJobId()`
  `protected RMHeartbeatHandler`
  `getRMHeartbeatHandler()`
  `int`
  `getStartCount()`
  `TaskAttemptListener`
  `getTaskAttemptListener()`
  `protected static void`
  `initAndStartAppMaster(MRAppMaster appMaster, org.apache.hadoop.mapred.JobConf conf, String jobUserName)`
  `protected void`
  `initJobCredentialsAndUGI(org.apache.hadoop.conf.Configuration conf)`
  Obtain the tokens needed by the job and put them in the UGI
  `Boolean`
  `isLastAMRetry()`
  `boolean`
  `isNewApiCommitter()`
  `protected boolean`
  `keepJobFiles(org.apache.hadoop.mapred.JobConf conf, String jobTempDir)`
  `static void`
  `main(String[] args)`
  `void`
  `notifyIsLastAMRetry(boolean isLastAMRetry)`
  `boolean`
  `recovered()`
  `protected void`
  `serviceInit(org.apache.hadoop.conf.Configuration conf)`
  `protected void`
  `serviceStart()`
  `protected void`
  `serviceStop()`
  `void`
  `shutDownJob()`
  `protected void`
  `startJobs()`
  This can be overridden to instantiate multiple jobs and create a workflow.
  `void`
  `stop()`
  `protected void`
  `sysexit()`
  Exit call.

  ### Methods inherited from class org.apache.hadoop.service.CompositeService

  `addIfService, addService, getServices, removeService`

  ### Methods inherited from class org.apache.hadoop.service.AbstractService

  `close, getBlockers, getConfig, getFailureCause, getFailureState, getLifecycleHistory, getName, getServiceState, getStartTime, init, isInState, noteFailure, putBlocker, registerGlobalListener, registerServiceListener, removeBlocker, setConfig, start, toString, unregisterGlobalListener, unregisterServiceListener, waitForServiceToStop`

  ### Methods inherited from class java.lang.Object

  `clone, equals, finalize, getClass, hashCode, notify, notifyAll, wait, wait, wait`

- ## Field Details

  - ### SHUTDOWN_HOOK_PRIORITY

    public static final int SHUTDOWN_HOOK_PRIORITY
    Priority of the MRAppMaster shutdown hook.
    See Also:
      - Constant Field Values

  - ### INTERMEDIATE_DATA_ENCRYPTION_ALGO

    public static final String INTERMEDIATE_DATA_ENCRYPTION_ALGO
    See Also:
      - Constant Field Values

  - ### metrics

    protected final MRAppMetrics metrics

  - ### taskAttemptListener

    protected TaskAttemptListener taskAttemptListener

  - ### jobTokenSecretManager

    protected org.apache.hadoop.mapreduce.security.token.JobTokenSecretManager jobTokenSecretManager

  - ### currentUser

    protected org.apache.hadoop.security.UserGroupInformation currentUser

  - ### isLastAMRetry

    @VisibleForTesting protected volatile boolean isLastAMRetry

  - ### successfullyUnregistered

    @VisibleForTesting protected AtomicBoolean successfullyUnregistered

- ## Constructor Details

  - ### MRAppMaster

    public MRAppMaster(org.apache.hadoop.yarn.api.records.ApplicationAttemptId applicationAttemptId, org.apache.hadoop.yarn.api.records.ContainerId containerId, String nmHost, int nmPort, int nmHttpPort, long appSubmitTime)

  - ### MRAppMaster

    public MRAppMaster(org.apache.hadoop.yarn.api.records.ApplicationAttemptId applicationAttemptId, org.apache.hadoop.yarn.api.records.ContainerId containerId, String nmHost, int nmPort, int nmHttpPort, org.apache.hadoop.yarn.util.Clock clock, long appSubmitTime)

- ## Method Details

  - ### createTaskAttemptFinishingMonitor

    protected TaskAttemptFinishingMonitor createTaskAttemptFinishingMonitor(org.apache.hadoop.yarn.event.EventHandler eventHandler)

  - ### serviceInit

    protected void serviceInit(org.apache.hadoop.conf.Configuration conf) throws Exception
    Overrides:
      `serviceInit` in class `org.apache.hadoop.service.CompositeService`
    Throws:
      `Exception`

  - ### createDispatcher

    protected org.apache.hadoop.yarn.event.Dispatcher createDispatcher()

  - ### createPreemptionPolicy

    protected AMPreemptionPolicy createPreemptionPolicy(org.apache.hadoop.conf.Configuration conf)

  - ### keepJobFiles

    protected boolean keepJobFiles(org.apache.hadoop.mapred.JobConf conf, String jobTempDir)

  - ### getFileSystem

    protected org.apache.hadoop.fs.FileSystem getFileSystem(org.apache.hadoop.conf.Configuration conf) throws IOException
    Create the default file System for this job.
    Parameters:
      `conf` - the conf object
    Returns:
      the default filesystem for this job
    Throws:
      `IOException`

  - ### getCredentials

    protected org.apache.hadoop.security.Credentials getCredentials()

  - ### cleanupStagingDir

    public void cleanupStagingDir() throws IOException
    clean up staging directories for the job.
    Throws:
      `IOException`

  - ### sysexit

    protected void sysexit()
    Exit call. Just in a function call to enable testing.

  - ### shutDownJob

    @VisibleForTesting public void shutDownJob()

  - ### createJobFinishEventHandler

    protected org.apache.hadoop.yarn.event.EventHandler<JobFinishEvent> createJobFinishEventHandler()
    create an event handler that handles the job finish event.
    Returns:
      the job finish event handler.

  - ### createJob

    protected Job createJob(org.apache.hadoop.conf.Configuration conf, JobStateInternal forcedState, String diagnostic)
    Create and initialize (but don't start) a single job.
    Parameters:
      `forcedState` - a state to force the job into or null for normal operation.
      `diagnostic` - a diagnostic message to include with the job.

  - ### initJobCredentialsAndUGI

    protected void initJobCredentialsAndUGI(org.apache.hadoop.conf.Configuration conf)
    Obtain the tokens needed by the job and put them in the UGI
    Parameters:
      `conf` -

  - ### createJobHistoryHandler

    protected org.apache.hadoop.yarn.event.EventHandler<JobHistoryEvent> createJobHistoryHandler(AppContext context)

  - ### createStagingDirCleaningService

    protected org.apache.hadoop.service.AbstractService createStagingDirCleaningService()

  - ### createSpeculator

    protected Speculator createSpeculator(org.apache.hadoop.conf.Configuration conf, AppContext context)

  - ### createTaskAttemptListener

    protected TaskAttemptListener createTaskAttemptListener(AppContext context, AMPreemptionPolicy preemptionPolicy)

  - ### createCommitterEventHandler

    protected org.apache.hadoop.yarn.event.EventHandler<CommitterEvent> createCommitterEventHandler(AppContext context, org.apache.hadoop.mapreduce.OutputCommitter committer)

  - ### createContainerAllocator

    protected ContainerAllocator createContainerAllocator(ClientService clientService, AppContext context)

  - ### getRMHeartbeatHandler

    protected RMHeartbeatHandler getRMHeartbeatHandler()

  - ### createContainerLauncher

    protected ContainerLauncher createContainerLauncher(AppContext context)

  - ### createClientService

    protected ClientService createClientService(AppContext context)

  - ### getAppID

    public org.apache.hadoop.yarn.api.records.ApplicationId getAppID()

  - ### getAttemptID

    public org.apache.hadoop.yarn.api.records.ApplicationAttemptId getAttemptID()

  - ### getJobId

    public org.apache.hadoop.mapreduce.v2.api.records.JobId getJobId()

  - ### getCommitter

    public org.apache.hadoop.mapreduce.OutputCommitter getCommitter()

  - ### isNewApiCommitter

    public boolean isNewApiCommitter()

  - ### getStartCount

    public int getStartCount()

  - ### getContext

    public AppContext getContext()

  - ### getDispatcher

    public org.apache.hadoop.yarn.event.Dispatcher getDispatcher()

  - ### getCompletedTaskFromPreviousRun

    public Map<org.apache.hadoop.mapreduce.v2.api.records.TaskId,org.apache.hadoop.mapreduce.jobhistory.JobHistoryParser.TaskInfo> getCompletedTaskFromPreviousRun()

  - ### getAllAMInfos

    public List<org.apache.hadoop.mapreduce.v2.api.records.AMInfo> getAllAMInfos()

  - ### getContainerAllocator

    public ContainerAllocator getContainerAllocator()

  - ### getContainerLauncher

    public ContainerLauncher getContainerLauncher()

  - ### getTaskAttemptListener

    public TaskAttemptListener getTaskAttemptListener()

  - ### isLastAMRetry

    public Boolean isLastAMRetry()

  - ### serviceStart

    protected void serviceStart() throws Exception
    Overrides:
      `serviceStart` in class `org.apache.hadoop.service.CompositeService`
    Throws:
      `Exception`

  - ### stop

    public void stop()
    Specified by:
      `stop` in interface `org.apache.hadoop.service.Service`
    Overrides:
      `stop` in class `org.apache.hadoop.service.AbstractService`

  - ### recovered

    public boolean recovered()

  - ### startJobs

    protected void startJobs()
    This can be overridden to instantiate multiple jobs and create a workflow. TODO: Rework the design to actually support this. Currently much of the job stuff has been moved to init() above to support uberization (MR-1220). In a typical workflow, one presumably would want to uberize only a subset of the jobs (the "small" ones), which is awkward with the current design.

  - ### main

    public static void main(String[] args)

  - ### notifyIsLastAMRetry

    public void notifyIsLastAMRetry(boolean isLastAMRetry)

  - ### initAndStartAppMaster

    protected static void initAndStartAppMaster(MRAppMaster appMaster, org.apache.hadoop.mapred.JobConf conf, String jobUserName) throws IOException, InterruptedException
    Throws:
      `IOException`
      `InterruptedException`

  - ### serviceStop

    protected void serviceStop() throws Exception
    Overrides:
      `serviceStop` in class `org.apache.hadoop.service.CompositeService`
    Throws:
      `Exception`

  - ### getClientService

    public ClientService getClientService()