DominantResourceCalculator (Apache Hadoop YARN Common 3.5.0 API)
Description: declaration: package: org.apache.hadoop.yarn.util.resource, class: DominantResourceCalculator

Package org.apache.hadoop.yarn.util.resource

# Class DominantResourceCalculator

java.lang.Object
org.apache.hadoop.yarn.util.resource.ResourceCalculator
org.apache.hadoop.yarn.util.resource.DominantResourceCalculator

@Private @Unstable public class DominantResourceCalculator extends ResourceCalculator
A `ResourceCalculator` which uses the concept of dominant resource to compare multi-dimensional resources. Essentially the idea is that the in a multi-resource environment, the resource allocation should be determined by the dominant share of an entity (user or queue), which is the maximum share that the entity has been allocated of any resource. In a nutshell, it seeks to maximize the minimum dominant share across all entities. For example, if user A runs CPU-heavy tasks and user B runs memory-heavy tasks, it attempts to equalize CPU share of user A with Memory-share of user B. In the single resource case, it reduces to max-min fairness for that resource. See the Dominant Resource Fairness paper for more details: www.cs.berkeley.edu/~matei/papers/2011/nsdi_drf.pdf

- ## Constructor Summary

  Constructors
  Constructor
  Description
  `DominantResourceCalculator()`

- ## Method Summary

  Modifier and Type
  Method
  Description
  `int`
  `compare(org.apache.hadoop.yarn.api.records.Resource clusterResource, org.apache.hadoop.yarn.api.records.Resource lhs, org.apache.hadoop.yarn.api.records.Resource rhs, boolean singleType)`
  On a cluster with capacity `clusterResource`, compare `lhs` and `rhs`.
  `long`
  `computeAvailableContainers(org.apache.hadoop.yarn.api.records.Resource available, org.apache.hadoop.yarn.api.records.Resource required)`
  Compute the number of containers which can be allocated given `available` and `required` resources.
  `float`
  `divide(org.apache.hadoop.yarn.api.records.Resource clusterResource, org.apache.hadoop.yarn.api.records.Resource numerator, org.apache.hadoop.yarn.api.records.Resource denominator)`
  Divide resource `numerator` by resource `denominator` using specified policy (domination, average, fairness etc.); hence overall `clusterResource` is provided for context.
  `org.apache.hadoop.yarn.api.records.Resource`
  `divideAndCeil(org.apache.hadoop.yarn.api.records.Resource numerator, float denominator)`
  Divide-and-ceil `numerator` by `denominator`.
  `org.apache.hadoop.yarn.api.records.Resource`
  `divideAndCeil(org.apache.hadoop.yarn.api.records.Resource numerator, int denominator)`
  Divide-and-ceil `numerator` by `denominator`.
  `org.apache.hadoop.yarn.api.records.Resource`
  `divideAndCeil(org.apache.hadoop.yarn.api.records.Resource numerator, long denominator)`
  `boolean`
  `fitsIn(org.apache.hadoop.yarn.api.records.Resource smaller, org.apache.hadoop.yarn.api.records.Resource bigger)`
  Check if a smaller resource can be contained by bigger resource.
  `Set<String>`
  `getInsufficientResourceNames(org.apache.hadoop.yarn.api.records.Resource required, org.apache.hadoop.yarn.api.records.Resource available)`
  Get insufficient resource names via comparing required resource and capacity resource.
  `boolean`
  `isAllInvalidDivisor(org.apache.hadoop.yarn.api.records.Resource r)`
  Determine if all resources are zero.
  `boolean`
  `isAnyMajorResourceAboveZero(org.apache.hadoop.yarn.api.records.Resource resource)`
  Check if resource has any major resource types (which are all NodeManagers included) has a > 0 value.
  `boolean`
  `isAnyMajorResourceZeroOrNegative(org.apache.hadoop.yarn.api.records.Resource resource)`
  Check if resource has any major resource types (which are all NodeManagers included) a zero value or negative value.
  `boolean`
  `isInvalidDivisor(org.apache.hadoop.yarn.api.records.Resource r)`
  Determine if a resource is not suitable for use as a divisor (will result in divide by 0, etc)
  `float`
  `minRatio(org.apache.hadoop.yarn.api.records.Resource a, org.apache.hadoop.yarn.api.records.Resource b)`
  Computes the ratio of resource a over resource b.
  `org.apache.hadoop.yarn.api.records.Resource`
  `multiplyAndNormalizeDown(org.apache.hadoop.yarn.api.records.Resource r, double by, org.apache.hadoop.yarn.api.records.Resource stepFactor)`
  Multiply resource `r` by factor `by` and normalize down using step-factor `stepFactor`.
  `org.apache.hadoop.yarn.api.records.Resource`
  `multiplyAndNormalizeUp(org.apache.hadoop.yarn.api.records.Resource r, double[] by, org.apache.hadoop.yarn.api.records.Resource stepFactor)`
  Multiply resource `r` by factor `by` and normalize up using step-factor `stepFactor`.
  `org.apache.hadoop.yarn.api.records.Resource`
  `multiplyAndNormalizeUp(org.apache.hadoop.yarn.api.records.Resource r, double by, org.apache.hadoop.yarn.api.records.Resource stepFactor)`
  Multiply resource `r` by factor `by` and normalize up using step-factor `stepFactor`.
  `org.apache.hadoop.yarn.api.records.Resource`
  `normalize(org.apache.hadoop.yarn.api.records.Resource r, org.apache.hadoop.yarn.api.records.Resource minimumResource, org.apache.hadoop.yarn.api.records.Resource maximumResource, org.apache.hadoop.yarn.api.records.Resource stepFactor)`
  Normalize resource `r` given the base `minimumResource` and verify against max allowed `maximumResource` using a step factor for the normalization.
  `org.apache.hadoop.yarn.api.records.Resource`
  `normalizeDown(org.apache.hadoop.yarn.api.records.Resource r, org.apache.hadoop.yarn.api.records.Resource stepFactor)`
  Get resource `r`and normalize down using step-factor `stepFactor`.
  `float`
  `ratio(org.apache.hadoop.yarn.api.records.Resource a, org.apache.hadoop.yarn.api.records.Resource b)`
  Ratio of resource `a` to resource `b`.
  `org.apache.hadoop.yarn.api.records.Resource`
  `roundDown(org.apache.hadoop.yarn.api.records.Resource r, org.apache.hadoop.yarn.api.records.Resource stepFactor)`
  Round-down resource `r` given factor `stepFactor`.
  `org.apache.hadoop.yarn.api.records.Resource`
  `roundUp(org.apache.hadoop.yarn.api.records.Resource r, org.apache.hadoop.yarn.api.records.Resource stepFactor)`
  Round-up resource `r` given factor `stepFactor`.

  ### Methods inherited from class org.apache.hadoop.yarn.util.resource.ResourceCalculator

  `compare, divideAndCeil, divideAndCeil, divideAndCeil, divideAndCeil, divideSafelyAsFloat, roundDown, roundDown, roundUp, roundUp`

  ### Methods inherited from class java.lang.Object

  `clone, equals, finalize, getClass, hashCode, notify, notifyAll, toString, wait, wait, wait`

- ## Constructor Details

  - ### DominantResourceCalculator

    public DominantResourceCalculator()

- ## Method Details

  - ### compare

    public int compare(org.apache.hadoop.yarn.api.records.Resource clusterResource, org.apache.hadoop.yarn.api.records.Resource lhs, org.apache.hadoop.yarn.api.records.Resource rhs, boolean singleType)
    Description copied from class: `ResourceCalculator`
    On a cluster with capacity `clusterResource`, compare `lhs` and `rhs`. Consider all resources unless `singleType` is set to true. When `singleType` is set to true, consider only one resource as per the `ResourceCalculator` implementation; the `DefaultResourceCalculator` considers memory and `DominantResourceCalculator` considers the dominant resource.
    Specified by:
      `compare` in class `ResourceCalculator`
    Parameters:
      `clusterResource` - cluster capacity
      `lhs` - First `Resource` to compare
      `rhs` - Second `Resource` to compare
      `singleType` - Whether to consider a single resource type or all resource types
    Returns:
      -1 if `lhs` is smaller, 0 if equal and 1 if it is larger

  - ### computeAvailableContainers

    public long computeAvailableContainers(org.apache.hadoop.yarn.api.records.Resource available, org.apache.hadoop.yarn.api.records.Resource required)
    Description copied from class: `ResourceCalculator`
    Compute the number of containers which can be allocated given `available` and `required` resources.
    Specified by:
      `computeAvailableContainers` in class `ResourceCalculator`
    Parameters:
      `available` - available resources
      `required` - required resources
    Returns:
      number of containers which can be allocated

  - ### divide

    public float divide(org.apache.hadoop.yarn.api.records.Resource clusterResource, org.apache.hadoop.yarn.api.records.Resource numerator, org.apache.hadoop.yarn.api.records.Resource denominator)
    Description copied from class: `ResourceCalculator`
    Divide resource `numerator` by resource `denominator` using specified policy (domination, average, fairness etc.); hence overall `clusterResource` is provided for context.
    Specified by:
      `divide` in class `ResourceCalculator`
    Parameters:
      `clusterResource` - cluster resources
      `numerator` - numerator
      `denominator` - denominator
    Returns:
      `numerator`/`denominator` using specific policy

  - ### isInvalidDivisor

    public boolean isInvalidDivisor(org.apache.hadoop.yarn.api.records.Resource r)
    Description copied from class: `ResourceCalculator`
    Determine if a resource is not suitable for use as a divisor (will result in divide by 0, etc)
    Specified by:
      `isInvalidDivisor` in class `ResourceCalculator`
    Parameters:
      `r` - resource
    Returns:
      true if divisor is invalid (should not be used), false else

  - ### isAllInvalidDivisor

    public boolean isAllInvalidDivisor(org.apache.hadoop.yarn.api.records.Resource r)
    Description copied from class: `ResourceCalculator`
    Determine if all resources are zero.
    Specified by:
      `isAllInvalidDivisor` in class `ResourceCalculator`
    Parameters:
      `r` - resource
    Returns:
      true if all divisors are invalid (should not be used), false else

  - ### ratio

    public float ratio(org.apache.hadoop.yarn.api.records.Resource a, org.apache.hadoop.yarn.api.records.Resource b)
    Description copied from class: `ResourceCalculator`
    Ratio of resource `a` to resource `b`.
    Specified by:
      `ratio` in class `ResourceCalculator`
    Parameters:
      `a` - resource
      `b` - resource
    Returns:
      ratio of resource `a` to resource `b`

  - ### minRatio

    @Unstable public float minRatio(org.apache.hadoop.yarn.api.records.Resource a, org.apache.hadoop.yarn.api.records.Resource b)
    Computes the ratio of resource a over resource b. However, different from ratio(Resource, Resource), this returns the min-share of the resources. For example, ratio(Resource(10, 50), Resource(100, 100)) would return 0.5, whereas minRatio(Resource(10, 50), Resource(100, 100)) would return 0.1.
    Parameters:
      `a` - the numerator resource.
      `b` - the denominator resource.
    Returns:
      the min-share ratio of the resources.

  - ### divideAndCeil

    public org.apache.hadoop.yarn.api.records.Resource divideAndCeil(org.apache.hadoop.yarn.api.records.Resource numerator, int denominator)
    Description copied from class: `ResourceCalculator`
    Divide-and-ceil `numerator` by `denominator`.
    Specified by:
      `divideAndCeil` in class `ResourceCalculator`
    Parameters:
      `numerator` - numerator resource
      `denominator` - denominator
    Returns:
      resultant resource

  - ### divideAndCeil

    public org.apache.hadoop.yarn.api.records.Resource divideAndCeil(org.apache.hadoop.yarn.api.records.Resource numerator, long denominator)

  - ### divideAndCeil

    public org.apache.hadoop.yarn.api.records.Resource divideAndCeil(org.apache.hadoop.yarn.api.records.Resource numerator, float denominator)
    Description copied from class: `ResourceCalculator`
    Divide-and-ceil `numerator` by `denominator`.
    Specified by:
      `divideAndCeil` in class `ResourceCalculator`
    Parameters:
      `numerator` - numerator resource
      `denominator` - denominator
    Returns:
      resultant resource

  - ### normalize

    public org.apache.hadoop.yarn.api.records.Resource normalize(org.apache.hadoop.yarn.api.records.Resource r, org.apache.hadoop.yarn.api.records.Resource minimumResource, org.apache.hadoop.yarn.api.records.Resource maximumResource, org.apache.hadoop.yarn.api.records.Resource stepFactor)
    Description copied from class: `ResourceCalculator`
    Normalize resource `r` given the base `minimumResource` and verify against max allowed `maximumResource` using a step factor for the normalization.
    Specified by:
      `normalize` in class `ResourceCalculator`
    Parameters:
      `r` - resource
      `minimumResource` - minimum value
      `maximumResource` - the upper bound of the resource to be allocated
      `stepFactor` - the increment for resources to be allocated
    Returns:
      normalized resource

  - ### roundUp

    public org.apache.hadoop.yarn.api.records.Resource roundUp(org.apache.hadoop.yarn.api.records.Resource r, org.apache.hadoop.yarn.api.records.Resource stepFactor)
    Description copied from class: `ResourceCalculator`
    Round-up resource `r` given factor `stepFactor`.
    Specified by:
      `roundUp` in class `ResourceCalculator`
    Parameters:
      `r` - resource
      `stepFactor` - step-factor
    Returns:
      rounded resource

  - ### roundDown

    public org.apache.hadoop.yarn.api.records.Resource roundDown(org.apache.hadoop.yarn.api.records.Resource r, org.apache.hadoop.yarn.api.records.Resource stepFactor)
    Description copied from class: `ResourceCalculator`
    Round-down resource `r` given factor `stepFactor`.
    Specified by:
      `roundDown` in class `ResourceCalculator`
    Parameters:
      `r` - resource
      `stepFactor` - step-factor
    Returns:
      rounded resource

  - ### multiplyAndNormalizeUp

    public org.apache.hadoop.yarn.api.records.Resource multiplyAndNormalizeUp(org.apache.hadoop.yarn.api.records.Resource r, double[] by, org.apache.hadoop.yarn.api.records.Resource stepFactor)
    Description copied from class: `ResourceCalculator`
    Multiply resource `r` by factor `by` and normalize up using step-factor `stepFactor`.
    Specified by:
      `multiplyAndNormalizeUp` in class `ResourceCalculator`
    Parameters:
      `r` - resource to be multiplied
      `by` - multiplier array for all resource types
      `stepFactor` - factor by which to normalize up
    Returns:
      resulting normalized resource

  - ### multiplyAndNormalizeUp

    public org.apache.hadoop.yarn.api.records.Resource multiplyAndNormalizeUp(org.apache.hadoop.yarn.api.records.Resource r, double by, org.apache.hadoop.yarn.api.records.Resource stepFactor)
    Description copied from class: `ResourceCalculator`
    Multiply resource `r` by factor `by` and normalize up using step-factor `stepFactor`.
    Specified by:
      `multiplyAndNormalizeUp` in class `ResourceCalculator`
    Parameters:
      `r` - resource to be multiplied
      `by` - multiplier
      `stepFactor` - factor by which to normalize up
    Returns:
      resulting normalized resource

  - ### multiplyAndNormalizeDown

    public org.apache.hadoop.yarn.api.records.Resource multiplyAndNormalizeDown(org.apache.hadoop.yarn.api.records.Resource r, double by, org.apache.hadoop.yarn.api.records.Resource stepFactor)
    Description copied from class: `ResourceCalculator`
    Multiply resource `r` by factor `by` and normalize down using step-factor `stepFactor`.
    Specified by:
      `multiplyAndNormalizeDown` in class `ResourceCalculator`
    Parameters:
      `r` - resource to be multiplied
      `by` - multiplier
      `stepFactor` - factor by which to normalize down
    Returns:
      resulting normalized resource

  - ### fitsIn

    public boolean fitsIn(org.apache.hadoop.yarn.api.records.Resource smaller, org.apache.hadoop.yarn.api.records.Resource bigger)
    Description copied from class: `ResourceCalculator`
    Check if a smaller resource can be contained by bigger resource.
    Specified by:
      `fitsIn` in class `ResourceCalculator`
    Parameters:
      `smaller` - smaller resource.
      `bigger` - bigger resource.
    Returns:
      if true, smaller resource can be contained by bigger resource; false otherwise.

  - ### normalizeDown

    public org.apache.hadoop.yarn.api.records.Resource normalizeDown(org.apache.hadoop.yarn.api.records.Resource r, org.apache.hadoop.yarn.api.records.Resource stepFactor)
    Description copied from class: `ResourceCalculator`
    Get resource `r`and normalize down using step-factor `stepFactor`.
    Specified by:
      `normalizeDown` in class `ResourceCalculator`
    Parameters:
      `r` - resource to be multiplied
      `stepFactor` - factor by which to normalize down
    Returns:
      resulting normalized resource

  - ### isAnyMajorResourceZeroOrNegative

    public boolean isAnyMajorResourceZeroOrNegative(org.apache.hadoop.yarn.api.records.Resource resource)
    Description copied from class: `ResourceCalculator`
    Check if resource has any major resource types (which are all NodeManagers included) a zero value or negative value.
    Specified by:
      `isAnyMajorResourceZeroOrNegative` in class `ResourceCalculator`
    Parameters:
      `resource` - resource
    Returns:
      returns true if any resource is zero.

  - ### isAnyMajorResourceAboveZero

    public boolean isAnyMajorResourceAboveZero(org.apache.hadoop.yarn.api.records.Resource resource)
    Description copied from class: `ResourceCalculator`
    Check if resource has any major resource types (which are all NodeManagers included) has a > 0 value.
    Specified by:
      `isAnyMajorResourceAboveZero` in class `ResourceCalculator`
    Parameters:
      `resource` - resource
    Returns:
      returns true if any resource is > 0

  - ### getInsufficientResourceNames

    public Set<String> getInsufficientResourceNames(org.apache.hadoop.yarn.api.records.Resource required, org.apache.hadoop.yarn.api.records.Resource available)
    Description copied from class: `ResourceCalculator`
    Get insufficient resource names via comparing required resource and capacity resource.
    Specified by:
      `getInsufficientResourceNames` in class `ResourceCalculator`
    Parameters:
      `required` - - required resource
      `available` - - available resource
    Returns:
      insufficient resource names