LogAggregationIndexedFileController (Apache Hadoop YARN Common 3.5.0 API)
Description: declaration: package: org.apache.hadoop.yarn.logaggregation.filecontroller.ifile, class: LogAggregationIndexedFileController

Package org.apache.hadoop.yarn.logaggregation.filecontroller.ifile

# Class LogAggregationIndexedFileController

java.lang.Object
org.apache.hadoop.yarn.logaggregation.filecontroller.LogAggregationFileController
org.apache.hadoop.yarn.logaggregation.filecontroller.ifile.LogAggregationIndexedFileController

@Private @Unstable public class LogAggregationIndexedFileController extends LogAggregationFileController
The Indexed Log Aggregation File Format implementation.

- ## Nested Class Summary

  Nested Classes
  Modifier and Type
  Class
  Description
  `static class`
  `LogAggregationIndexedFileController.IndexedFileLogMeta`
  This IndexedFileLogMeta includes the meta information for a single file which would be aggregated in one Log aggregation cycle.
  `static class`
  `LogAggregationIndexedFileController.IndexedLogsMeta`
  This IndexedLogsMeta includes all the meta information for the aggregated log file.
  `static class`
  `LogAggregationIndexedFileController.IndexedPerAggregationLogMeta`
  This IndexedPerAggregationLogMeta includes the meta information for all files which would be aggregated in one Log aggregation cycle.

- ## Field Summary

  Fields
  Modifier and Type
  Field
  Description
  `static final String`
  `CHECK_SUM_FILE_SUFFIX`

  ### Fields inherited from class org.apache.hadoop.yarn.logaggregation.filecontroller.LogAggregationFileController

  `APP_DIR_PERMISSIONS, APP_LOG_FILE_UMASK, conf, fileControllerName, fsSupportsChmod, remoteRootLogDir, remoteRootLogDirSuffix, retentionSize, TLDIR_PERMISSIONS`

- ## Constructor Summary

  Constructors
  Constructor
  Description
  `LogAggregationIndexedFileController()`

- ## Method Summary

  Modifier and Type
  Method
  Description
  `void`
  `closeWriter()`
  Close the writer.
  `org.apache.hadoop.fs.FileStatus`
  `getAllChecksumFiles(Map<String,org.apache.hadoop.fs.FileStatus> fileMap, String fileName)`
  `Map<org.apache.hadoop.yarn.api.records.ApplicationAccessType,String>`
  `getApplicationAcls(org.apache.hadoop.fs.Path aggregatedLogPath, org.apache.hadoop.yarn.api.records.ApplicationId appId)`
  Returns ACLs for the application.
  `String`
  `getApplicationOwner(org.apache.hadoop.fs.Path aggregatedLogPath, org.apache.hadoop.yarn.api.records.ApplicationId appId)`
  Returns the owner of the application.
  `static int`
  `getFSInputBufferSize(org.apache.hadoop.conf.Configuration conf)`
  `static int`
  `getFSOutputBufferSize(org.apache.hadoop.conf.Configuration conf)`
  `Map<String,List<ContainerLogFileInfo>>`
  `getLogMetaFilesOfNode(ExtendedLogMetaRequest logRequest, org.apache.hadoop.fs.FileStatus currentNodeFile, org.apache.hadoop.yarn.api.records.ApplicationId appId)`
  Returns log file metadata for a node grouped by containers.
  `List<org.apache.hadoop.fs.FileStatus>`
  `getNodeLogFileToRead(List<org.apache.hadoop.fs.FileStatus> nodeFiles, String nodeId, org.apache.hadoop.yarn.api.records.ApplicationId appId)`
  `org.apache.hadoop.fs.Path`
  `getOlderRemoteAppLogDir(org.apache.hadoop.yarn.api.records.ApplicationId appId, String user)`
  Get the older remote application directory for log aggregation.
  `org.apache.hadoop.fs.Path`
  `getRemoteAppLogDir(org.apache.hadoop.yarn.api.records.ApplicationId appId, String user)`
  Get the remote application directory for log aggregation.
  `long`
  `getRollOverLogMaxSize(org.apache.hadoop.conf.Configuration conf)`
  `Clock`
  `getSystemClock()`
  `void`
  `initializeWriter(LogAggregationFileControllerContext context)`
  Initialize the writer.
  `void`
  `initInternal(org.apache.hadoop.conf.Configuration conf)`
  Derived classes initialize themselves using this method.
  `boolean`
  `isRollover(org.apache.hadoop.fs.FileContext fc, org.apache.hadoop.fs.Path candidate)`
  `LogAggregationIndexedFileController.IndexedLogsMeta`
  `loadIndexedLogsMeta(org.apache.hadoop.fs.Path remoteLogPath, long end, org.apache.hadoop.yarn.api.records.ApplicationId appId)`
  `Map<String,Long>`
  `parseCheckSumFiles(List<org.apache.hadoop.fs.FileStatus> fileList)`
  `void`
  `postWrite(LogAggregationFileControllerContext record)`
  Operations needed after write the log content.
  `boolean`
  `readAggregatedLogs(ContainerLogsRequest logRequest, OutputStream os)`
  Output container log.
  `List<ContainerLogMeta>`
  `readAggregatedLogsMeta(ContainerLogsRequest logRequest)`
  Return a list of `ContainerLogMeta` for an application from Remote FileSystem.
  `void`
  `renderAggregatedLogsBlock(HtmlBlock.Block html, View.ViewContext context)`
  Render Aggregated Logs block.
  `void`
  `write(AggregatedLogFormat.LogKey logKey, AggregatedLogFormat.LogValue logValue)`
  Write the log content.

  ### Methods inherited from class org.apache.hadoop.yarn.logaggregation.filecontroller.LogAggregationFileController

  `aggregatedLogSuffix, belongsToAppAttempt, checkExists, cleanOldLogs, closePrintStream, createAppDir, createDir, getApplicationDirectoriesOfUser, getFileControllerName, getFileSystem, getNodeFilesOfApplicationDirectory, getRemoteNodeLogFileForApp, getRemoteRootLogDir, getRemoteRootLogDirSuffix, initialize, isFsSupportsChmod, verifyAndCreateRemoteLogDir`

  ### Methods inherited from class java.lang.Object

  `clone, equals, finalize, getClass, hashCode, notify, notifyAll, toString, wait, wait, wait`

- ## Field Details

  - ### CHECK_SUM_FILE_SUFFIX

    @VisibleForTesting public static final String CHECK_SUM_FILE_SUFFIX
    See Also:
      - Constant Field Values

- ## Constructor Details

  - ### LogAggregationIndexedFileController

    public LogAggregationIndexedFileController()

- ## Method Details

  - ### initInternal

    public void initInternal(org.apache.hadoop.conf.Configuration conf)
    Description copied from class: `LogAggregationFileController`
    Derived classes initialize themselves using this method.
    Specified by:
      `initInternal` in class `LogAggregationFileController`
    Parameters:
      `conf` - the Configuration

  - ### initializeWriter

    public void initializeWriter(LogAggregationFileControllerContext context) throws IOException
    Description copied from class: `LogAggregationFileController`
    Initialize the writer.
    Specified by:
      `initializeWriter` in class `LogAggregationFileController`
    Parameters:
      `context` - the `LogAggregationFileControllerContext`
    Throws:
      `IOException` - if fails to initialize the writer

  - ### closeWriter

    public void closeWriter()
    Description copied from class: `LogAggregationFileController`
    Close the writer.
    Specified by:
      `closeWriter` in class `LogAggregationFileController`

  - ### write

    public void write(AggregatedLogFormat.LogKey logKey, AggregatedLogFormat.LogValue logValue) throws IOException
    Description copied from class: `LogAggregationFileController`
    Write the log content.
    Specified by:
      `write` in class `LogAggregationFileController`
    Parameters:
      `logKey` - the log key
      `logValue` - the log content
    Throws:
      `IOException` - if fails to write the logs

  - ### postWrite

    public void postWrite(LogAggregationFileControllerContext record) throws Exception
    Description copied from class: `LogAggregationFileController`
    Operations needed after write the log content.
    Specified by:
      `postWrite` in class `LogAggregationFileController`
    Parameters:
      `record` - the `LogAggregationFileControllerContext`
    Throws:
      `Exception` - if anything fails

  - ### readAggregatedLogs

    public boolean readAggregatedLogs(ContainerLogsRequest logRequest, OutputStream os) throws IOException
    Description copied from class: `LogAggregationFileController`
    Output container log.
    Specified by:
      `readAggregatedLogs` in class `LogAggregationFileController`
    Parameters:
      `logRequest` - `ContainerLogsRequest`
      `os` - the output stream
    Returns:
      true if we can read the aggregated logs successfully
    Throws:
      `IOException` - if we can not access the log file.

  - ### getLogMetaFilesOfNode

    public Map<String,List<ContainerLogFileInfo>> getLogMetaFilesOfNode(ExtendedLogMetaRequest logRequest, org.apache.hadoop.fs.FileStatus currentNodeFile, org.apache.hadoop.yarn.api.records.ApplicationId appId) throws IOException
    Description copied from class: `LogAggregationFileController`
    Returns log file metadata for a node grouped by containers.
    Overrides:
      `getLogMetaFilesOfNode` in class `LogAggregationFileController`
    Parameters:
      `logRequest` - extended query information holder
      `currentNodeFile` - file status of a node in an application directory
      `appId` - id of the application, which is the same as in node path
    Returns:
      log file metadata
    Throws:
      `IOException` - if there is no node file

  - ### readAggregatedLogsMeta

    public List<ContainerLogMeta> readAggregatedLogsMeta(ContainerLogsRequest logRequest) throws IOException
    Description copied from class: `LogAggregationFileController`
    Return a list of `ContainerLogMeta` for an application from Remote FileSystem.
    Specified by:
      `readAggregatedLogsMeta` in class `LogAggregationFileController`
    Parameters:
      `logRequest` - `ContainerLogsRequest`
    Returns:
      a list of `ContainerLogMeta`
    Throws:
      `IOException` - if there is no available log file

  - ### parseCheckSumFiles

    @Private public Map<String,Long> parseCheckSumFiles(List<org.apache.hadoop.fs.FileStatus> fileList) throws IOException
    Throws:
      `IOException`

  - ### getNodeLogFileToRead

    @Private public List<org.apache.hadoop.fs.FileStatus> getNodeLogFileToRead(List<org.apache.hadoop.fs.FileStatus> nodeFiles, String nodeId, org.apache.hadoop.yarn.api.records.ApplicationId appId) throws IOException
    Throws:
      `IOException`

  - ### getAllChecksumFiles

    @Private public org.apache.hadoop.fs.FileStatus getAllChecksumFiles(Map<String,org.apache.hadoop.fs.FileStatus> fileMap, String fileName)

  - ### renderAggregatedLogsBlock

    public void renderAggregatedLogsBlock(HtmlBlock.Block html, View.ViewContext context)
    Description copied from class: `LogAggregationFileController`
    Render Aggregated Logs block.
    Specified by:
      `renderAggregatedLogsBlock` in class `LogAggregationFileController`
    Parameters:
      `html` - the html
      `context` - the ViewContext

  - ### getApplicationOwner

    public String getApplicationOwner(org.apache.hadoop.fs.Path aggregatedLogPath, org.apache.hadoop.yarn.api.records.ApplicationId appId) throws IOException
    Description copied from class: `LogAggregationFileController`
    Returns the owner of the application.
    Specified by:
      `getApplicationOwner` in class `LogAggregationFileController`
    Parameters:
      `aggregatedLogPath` - the aggregatedLog path
      `appId` - the ApplicationId
    Returns:
      the application owner
    Throws:
      `IOException` - if we can not get the application owner

  - ### getApplicationAcls

    public Map<org.apache.hadoop.yarn.api.records.ApplicationAccessType,String> getApplicationAcls(org.apache.hadoop.fs.Path aggregatedLogPath, org.apache.hadoop.yarn.api.records.ApplicationId appId) throws IOException
    Description copied from class: `LogAggregationFileController`
    Returns ACLs for the application. An empty map is returned if no ACLs are found.
    Specified by:
      `getApplicationAcls` in class `LogAggregationFileController`
    Parameters:
      `aggregatedLogPath` - the aggregatedLog path.
      `appId` - the ApplicationId
    Returns:
      a map of the Application ACLs.
    Throws:
      `IOException` - if we can not get the application acls

  - ### getRemoteAppLogDir

    public org.apache.hadoop.fs.Path getRemoteAppLogDir(org.apache.hadoop.yarn.api.records.ApplicationId appId, String user) throws IOException
    Description copied from class: `LogAggregationFileController`
    Get the remote application directory for log aggregation.
    Overrides:
      `getRemoteAppLogDir` in class `LogAggregationFileController`
    Parameters:
      `appId` - the Application ID
      `user` - the Application Owner
    Returns:
      the remote application directory
    Throws:
      `IOException` - if can not find the remote application directory

  - ### getOlderRemoteAppLogDir

    public org.apache.hadoop.fs.Path getOlderRemoteAppLogDir(org.apache.hadoop.yarn.api.records.ApplicationId appId, String user) throws IOException
    Description copied from class: `LogAggregationFileController`
    Get the older remote application directory for log aggregation.
    Overrides:
      `getOlderRemoteAppLogDir` in class `LogAggregationFileController`
    Parameters:
      `appId` - the Application ID
      `user` - the Application Owner
    Returns:
      the older remote application directory
    Throws:
      `IOException` - if can not find the remote application directory

  - ### loadIndexedLogsMeta

    @Private public LogAggregationIndexedFileController.IndexedLogsMeta loadIndexedLogsMeta(org.apache.hadoop.fs.Path remoteLogPath, long end, org.apache.hadoop.yarn.api.records.ApplicationId appId) throws IOException
    Throws:
      `IOException`

  - ### getFSOutputBufferSize

    @Private public static int getFSOutputBufferSize(org.apache.hadoop.conf.Configuration conf)

  - ### getFSInputBufferSize

    @Private public static int getFSInputBufferSize(org.apache.hadoop.conf.Configuration conf)

  - ### getRollOverLogMaxSize

    @Private @VisibleForTesting public long getRollOverLogMaxSize(org.apache.hadoop.conf.Configuration conf)

  - ### isRollover

    @Private @VisibleForTesting public boolean isRollover(org.apache.hadoop.fs.FileContext fc, org.apache.hadoop.fs.Path candidate) throws IOException
    Throws:
      `IOException`

  - ### getSystemClock

    @Private @VisibleForTesting public Clock getSystemClock()