TaskPool (Apache Hadoop Common 3.5.0 API)
Description: declaration: package: org.apache.hadoop.util.functional, class: TaskPool

Package org.apache.hadoop.util.functional

# Class TaskPool

java.lang.Object
org.apache.hadoop.util.functional.TaskPool

@Private @Unstable public final class TaskPool extends Object
Utility class for parallel execution, takes closures for the various actions. There is no retry logic: it is expected to be handled by the closures. From `org.apache.hadoop.fs.s3a.commit.Tasks` which came from the Netflix committer patch. Apache Iceberg has its own version of this, with a common ancestor at some point in its history. A key difference with this class is that the iterator is always, internally, an `RemoteIterator`. This is to allow tasks to be scheduled while incremental operations such as paged directory listings are still collecting in results. While awaiting completion, this thread spins and sleeps a time of `SLEEP_INTERVAL_AWAITING_COMPLETION`, which, being a busy-wait, is inefficient. There's an implicit assumption that remote IO is being performed, and so this is not impacting throughput/performance. History: This class came with the Netflix contributions to the S3A committers in HADOOP-13786. It was moved into hadoop-common for use in the manifest committer and anywhere else it is needed, and renamed in the process as "Tasks" has too many meanings in the hadoop source. The iterator was then changed from a normal java iterable to a hadoop `RemoteIterator`. This allows a task pool to be supplied with incremental listings from object stores, scheduling work as pages of listing results come in, rather than blocking until the entire directory/directory tree etc has been enumerated. There is a variant of this in Apache Iceberg in `org.apache.iceberg.util.Tasks` That is not derived from any version in the hadoop codebase, it just shares a common ancestor somewhere in the Netflix codebase. It is the more sophisticated version.

- ## Nested Class Summary

  Nested Classes
  Modifier and Type
  Class
  Description
  `static class`
  `TaskPool.Builder<I>`
  Builder for task execution.
  `static interface`
  `TaskPool.FailureTask<I,E extends Exception>`
  Callback invoked on a failure.
  `static interface`
  `TaskPool.Submitter`
  Interface to whatever lets us submit tasks.
  `static interface`
  `TaskPool.Task<I,E extends Exception>`
  Callback invoked to process an item.

- ## Method Summary

  Modifier and Type
  Method
  Description
  `static <I> TaskPool.Builder<I>`
  `foreach(I[] items)`
  `static <I> TaskPool.Builder<I>`
  `foreach(Iterable<I> items)`
  Create a task builder for the iterable.
  `static <I> TaskPool.Builder<I>`
  `foreach(RemoteIterator<I> items)`
  Create a task builder for the remote iterator.

  ### Methods inherited from class java.lang.Object

  `clone, equals, finalize, getClass, hashCode, notify, notifyAll, toString, wait, wait, wait`

- ## Method Details

  - ### foreach

    public static <I> TaskPool.Builder<I> foreach(Iterable<I> items)
    Create a task builder for the iterable.
    Type Parameters:
      `I` - type of result.
    Parameters:
      `items` - item source.
    Returns:
      builder.

  - ### foreach

    public static <I> TaskPool.Builder<I> foreach(RemoteIterator<I> items)
    Create a task builder for the remote iterator.
    Type Parameters:
      `I` - type of result.
    Parameters:
      `items` - item source.
    Returns:
      builder.

  - ### foreach

    public static <I> TaskPool.Builder<I> foreach(I[] items)