LogAggregationFileController (Apache Hadoop Main 3.5.0 API)
Description: declaration: package: org.apache.hadoop.yarn.logaggregation.filecontroller, class: LogAggregationFileController

Package org.apache.hadoop.yarn.logaggregation.filecontroller

# Class LogAggregationFileController

java.lang.Object
org.apache.hadoop.yarn.logaggregation.filecontroller.LogAggregationFileController

@Public @Unstable public abstract class LogAggregationFileController extends Object
Base class to implement Log Aggregation File Controller.

- ## Field Summary

  Fields
  Modifier and Type
  Field
  Description
  `protected static final FsPermission`
  `APP_DIR_PERMISSIONS`
  Permissions for the Application directory.
  `protected static final FsPermission`
  `APP_LOG_FILE_UMASK`
  Umask for the log file.
  `protected Configuration`
  `conf`
  `protected String`
  `fileControllerName`
  `protected boolean`
  `fsSupportsChmod`
  `protected Path`
  `remoteRootLogDir`
  `protected String`
  `remoteRootLogDirSuffix`
  `protected int`
  `retentionSize`
  `protected static final FsPermission`
  `TLDIR_PERMISSIONS`
  Permissions for the top level directory under which app directories will be created.

- ## Constructor Summary

  Constructors
  Constructor
  Description
  `LogAggregationFileController()`

- ## Method Summary

  Modifier and Type
  Method
  Description
  `protected String`
  `aggregatedLogSuffix(String fileName)`
  Create the aggregated log suffix.
  `protected boolean`
  `belongsToAppAttempt(ApplicationAttemptId appAttemptId, String containerIdStr)`
  `protected boolean`
  `checkExists(FileSystem fs, Path path, FsPermission fsPerm)`
  `protected void`
  `cleanOldLogs(Path remoteNodeLogFileForApp, NodeId nodeId, UserGroupInformation userUgi)`
  `protected void`
  `closePrintStream(OutputStream out)`
  `abstract void`
  `closeWriter()`
  Close the writer.
  `void`
  `createAppDir(String user, ApplicationId appId, UserGroupInformation userUgi)`
  Create remote Application directory for log aggregation.
  `protected void`
  `createDir(FileSystem fs, Path path, FsPermission fsPerm)`
  `abstract Map<ApplicationAccessType,String>`
  `getApplicationAcls(Path aggregatedLogPath, ApplicationId appId)`
  Returns ACLs for the application.
  `org.apache.hadoop.fs.RemoteIterator<FileStatus>`
  `getApplicationDirectoriesOfUser(String user)`
  Gets all application directories of a user.
  `abstract String`
  `getApplicationOwner(Path aggregatedLogPath, ApplicationId appId)`
  Returns the owner of the application.
  `String`
  `getFileControllerName()`
  Get the name of the file controller.
  `protected FileSystem`
  `getFileSystem(Configuration conf)`
  `Map<String,List<org.apache.hadoop.yarn.logaggregation.ContainerLogFileInfo>>`
  `getLogMetaFilesOfNode(org.apache.hadoop.yarn.logaggregation.ExtendedLogMetaRequest logRequest, FileStatus currentNodeFile, ApplicationId appId)`
  Returns log file metadata for a node grouped by containers.
  `org.apache.hadoop.fs.RemoteIterator<FileStatus>`
  `getNodeFilesOfApplicationDirectory(FileStatus appDir)`
  Gets all node files in an application directory.
  `Path`
  `getOlderRemoteAppLogDir(ApplicationId appId, String appOwner)`
  Get the older remote application directory for log aggregation.
  `Path`
  `getRemoteAppLogDir(ApplicationId appId, String appOwner)`
  Get the remote application directory for log aggregation.
  `Path`
  `getRemoteNodeLogFileForApp(ApplicationId appId, String user, NodeId nodeId)`
  Get the remote aggregated log path.
  `Path`
  `getRemoteRootLogDir()`
  Get the remote root log directory.
  `String`
  `getRemoteRootLogDirSuffix()`
  Get the log aggregation directory suffix.
  `void`
  `initialize(Configuration conf, String controllerName)`
  Initialize the log file controller.
  `abstract void`
  `initializeWriter(org.apache.hadoop.yarn.logaggregation.filecontroller.LogAggregationFileControllerContext context)`
  Initialize the writer.
  `protected abstract void`
  `initInternal(Configuration conf)`
  Derived classes initialize themselves using this method.
  `boolean`
  `isFsSupportsChmod()`
  `abstract void`
  `postWrite(org.apache.hadoop.yarn.logaggregation.filecontroller.LogAggregationFileControllerContext record)`
  Operations needed after write the log content.
  `abstract boolean`
  `readAggregatedLogs(org.apache.hadoop.yarn.logaggregation.ContainerLogsRequest logRequest, OutputStream os)`
  Output container log.
  `abstract List<org.apache.hadoop.yarn.logaggregation.ContainerLogMeta>`
  `readAggregatedLogsMeta(org.apache.hadoop.yarn.logaggregation.ContainerLogsRequest logRequest)`
  Return a list of `ContainerLogMeta` for an application from Remote FileSystem.
  `abstract void`
  `renderAggregatedLogsBlock(org.apache.hadoop.yarn.webapp.view.HtmlBlock.Block html, org.apache.hadoop.yarn.webapp.View.ViewContext context)`
  Render Aggregated Logs block.
  `void`
  `verifyAndCreateRemoteLogDir()`
  Verify and create the remote log directory.
  `abstract void`
  `write(AggregatedLogFormat.LogKey logKey, org.apache.hadoop.yarn.logaggregation.AggregatedLogFormat.LogValue logValue)`
  Write the log content.

  ### Methods inherited from class java.lang.Object

  `clone, equals, finalize, getClass, hashCode, notify, notifyAll, toString, wait, wait, wait`

- ## Field Details

  - ### TLDIR_PERMISSIONS

    protected static final FsPermission TLDIR_PERMISSIONS
    Permissions for the top level directory under which app directories will be created.

  - ### APP_DIR_PERMISSIONS

    protected static final FsPermission APP_DIR_PERMISSIONS
    Permissions for the Application directory.

  - ### APP_LOG_FILE_UMASK

    protected static final FsPermission APP_LOG_FILE_UMASK
    Umask for the log file.

  - ### conf

    protected Configuration conf

  - ### remoteRootLogDir

    protected Path remoteRootLogDir

  - ### remoteRootLogDirSuffix

    protected String remoteRootLogDirSuffix

  - ### retentionSize

    protected int retentionSize

  - ### fileControllerName

    protected String fileControllerName

  - ### fsSupportsChmod

    protected boolean fsSupportsChmod

- ## Constructor Details

  - ### LogAggregationFileController

    public LogAggregationFileController()

- ## Method Details

  - ### initialize

    public void initialize(Configuration conf, String controllerName)
    Initialize the log file controller.
    Parameters:
      `conf` - the Configuration
      `controllerName` - the log controller class name

  - ### initInternal

    protected abstract void initInternal(Configuration conf)
    Derived classes initialize themselves using this method.
    Parameters:
      `conf` - the Configuration

  - ### getRemoteRootLogDir

    public Path getRemoteRootLogDir()
    Get the remote root log directory.
    Returns:
      the remote root log directory path

  - ### getRemoteRootLogDirSuffix

    public String getRemoteRootLogDirSuffix()
    Get the log aggregation directory suffix.
    Returns:
      the log aggregation directory suffix

  - ### getFileControllerName

    public String getFileControllerName()
    Get the name of the file controller.
    Returns:
      name of the file controller.

  - ### initializeWriter

    public abstract void initializeWriter(org.apache.hadoop.yarn.logaggregation.filecontroller.LogAggregationFileControllerContext context) throws IOException
    Initialize the writer.
    Parameters:
      `context` - the `LogAggregationFileControllerContext`
    Throws:
      `IOException` - if fails to initialize the writer

  - ### closeWriter

    public abstract void closeWriter() throws org.apache.hadoop.yarn.logaggregation.filecontroller.LogAggregationDFSException
    Close the writer.
    Throws:
      `org.apache.hadoop.yarn.logaggregation.filecontroller.LogAggregationDFSException` - if the closing of the writer fails (for example due to HDFS quota being exceeded)

  - ### write

    public abstract void write(AggregatedLogFormat.LogKey logKey, org.apache.hadoop.yarn.logaggregation.AggregatedLogFormat.LogValue logValue) throws IOException
    Write the log content.
    Parameters:
      `logKey` - the log key
      `logValue` - the log content
    Throws:
      `IOException` - if fails to write the logs

  - ### postWrite

    public abstract void postWrite(org.apache.hadoop.yarn.logaggregation.filecontroller.LogAggregationFileControllerContext record) throws Exception
    Operations needed after write the log content.
    Parameters:
      `record` - the `LogAggregationFileControllerContext`
    Throws:
      `Exception` - if anything fails

  - ### closePrintStream

    protected void closePrintStream(OutputStream out)

  - ### readAggregatedLogs

    public abstract boolean readAggregatedLogs(org.apache.hadoop.yarn.logaggregation.ContainerLogsRequest logRequest, OutputStream os) throws IOException
    Output container log.
    Parameters:
      `logRequest` - `ContainerLogsRequest`
      `os` - the output stream
    Returns:
      true if we can read the aggregated logs successfully
    Throws:
      `IOException` - if we can not access the log file.

  - ### readAggregatedLogsMeta

    public abstract List<org.apache.hadoop.yarn.logaggregation.ContainerLogMeta> readAggregatedLogsMeta(org.apache.hadoop.yarn.logaggregation.ContainerLogsRequest logRequest) throws IOException
    Return a list of `ContainerLogMeta` for an application from Remote FileSystem.
    Parameters:
      `logRequest` - `ContainerLogsRequest`
    Returns:
      a list of `ContainerLogMeta`
    Throws:
      `IOException` - if there is no available log file

  - ### getLogMetaFilesOfNode

    public Map<String,List<org.apache.hadoop.yarn.logaggregation.ContainerLogFileInfo>> getLogMetaFilesOfNode(org.apache.hadoop.yarn.logaggregation.ExtendedLogMetaRequest logRequest, FileStatus currentNodeFile, ApplicationId appId) throws IOException
    Returns log file metadata for a node grouped by containers.
    Parameters:
      `logRequest` - extended query information holder
      `currentNodeFile` - file status of a node in an application directory
      `appId` - id of the application, which is the same as in node path
    Returns:
      log file metadata
    Throws:
      `IOException` - if there is no node file

  - ### getApplicationDirectoriesOfUser

    public org.apache.hadoop.fs.RemoteIterator<FileStatus> getApplicationDirectoriesOfUser(String user) throws IOException
    Gets all application directories of a user.
    Parameters:
      `user` - name of the user
    Returns:
      a lazy iterator of directories
    Throws:
      `IOException` - if user directory does not exist

  - ### getNodeFilesOfApplicationDirectory

    public org.apache.hadoop.fs.RemoteIterator<FileStatus> getNodeFilesOfApplicationDirectory(FileStatus appDir) throws IOException
    Gets all node files in an application directory.
    Parameters:
      `appDir` - application directory
    Returns:
      a lazy iterator of files
    Throws:
      `IOException` - if file context is not reachable

  - ### renderAggregatedLogsBlock

    public abstract void renderAggregatedLogsBlock(org.apache.hadoop.yarn.webapp.view.HtmlBlock.Block html, org.apache.hadoop.yarn.webapp.View.ViewContext context)
    Render Aggregated Logs block.
    Parameters:
      `html` - the html
      `context` - the ViewContext

  - ### getApplicationOwner

    public abstract String getApplicationOwner(Path aggregatedLogPath, ApplicationId appId) throws IOException
    Returns the owner of the application.
    Parameters:
      `aggregatedLogPath` - the aggregatedLog path
      `appId` - the ApplicationId
    Returns:
      the application owner
    Throws:
      `IOException` - if we can not get the application owner

  - ### getApplicationAcls

    public abstract Map<ApplicationAccessType,String> getApplicationAcls(Path aggregatedLogPath, ApplicationId appId) throws IOException
    Returns ACLs for the application. An empty map is returned if no ACLs are found.
    Parameters:
      `aggregatedLogPath` - the aggregatedLog path.
      `appId` - the ApplicationId
    Returns:
      a map of the Application ACLs.
    Throws:
      `IOException` - if we can not get the application acls

  - ### verifyAndCreateRemoteLogDir

    public void verifyAndCreateRemoteLogDir()
    Verify and create the remote log directory.

  - ### createAppDir

    public void createAppDir(String user, ApplicationId appId, UserGroupInformation userUgi)
    Create remote Application directory for log aggregation.
    Parameters:
      `user` - the user
      `appId` - the application ID
      `userUgi` - the UGI

  - ### getFileSystem

    @VisibleForTesting protected FileSystem getFileSystem(Configuration conf) throws IOException
    Throws:
      `IOException`

  - ### createDir

    protected void createDir(FileSystem fs, Path path, FsPermission fsPerm) throws IOException
    Throws:
      `IOException`

  - ### checkExists

    protected boolean checkExists(FileSystem fs, Path path, FsPermission fsPerm) throws IOException
    Throws:
      `IOException`

  - ### getRemoteNodeLogFileForApp

    public Path getRemoteNodeLogFileForApp(ApplicationId appId, String user, NodeId nodeId)
    Get the remote aggregated log path.
    Parameters:
      `appId` - the ApplicationId
      `user` - the Application Owner
      `nodeId` - the NodeManager Id
    Returns:
      the remote aggregated log path

  - ### getRemoteAppLogDir

    public Path getRemoteAppLogDir(ApplicationId appId, String appOwner) throws IOException
    Get the remote application directory for log aggregation.
    Parameters:
      `appId` - the Application ID
      `appOwner` - the Application Owner
    Returns:
      the remote application directory
    Throws:
      `IOException` - if can not find the remote application directory

  - ### getOlderRemoteAppLogDir

    public Path getOlderRemoteAppLogDir(ApplicationId appId, String appOwner) throws IOException
    Get the older remote application directory for log aggregation.
    Parameters:
      `appId` - the Application ID
      `appOwner` - the Application Owner
    Returns:
      the older remote application directory
    Throws:
      `IOException` - if can not find the remote application directory

  - ### cleanOldLogs

    protected void cleanOldLogs(Path remoteNodeLogFileForApp, NodeId nodeId, UserGroupInformation userUgi)

  - ### aggregatedLogSuffix

    protected String aggregatedLogSuffix(String fileName)
    Create the aggregated log suffix. The LogAggregationFileController should call this to get the suffix and append the suffix to the end of each log. This would keep the aggregated log format consistent.
    Parameters:
      `fileName` - the File Name
    Returns:
      the aggregated log suffix String

  - ### isFsSupportsChmod

    public boolean isFsSupportsChmod()

  - ### belongsToAppAttempt

    protected boolean belongsToAppAttempt(ApplicationAttemptId appAttemptId, String containerIdStr)