Apache Hadoop Amazon Web Services support – Upgrading S3A to AWS SDK V2

##### General

- Overview
- Single Node Setup
- Cluster Setup
- Commands Reference
- FileSystem Shell
- Compatibility Specification
- Downstream Developer's Guide
- Admin Compatibility Guide
- Interface Classification
- FileSystem Specification

##### Common

- CLI Mini Cluster
- Fair Call Queue
- Native Libraries
- Proxy User
- Rack Awareness
- Secure Mode
- Service Level Authorization
- HTTP Authentication
- Credential Provider API
- Hadoop KMS
- Tracing
- Unix Shell Guide
- Registry
- Async Profiler

##### HDFS

- Architecture
- User Guide
- Commands Reference
- NameNode HA With QJM
- NameNode HA With NFS
- Observer NameNode
- Federation
- ViewFs
- ViewFsOverloadScheme
- Snapshots
- Edits Viewer
- Image Viewer
- Permissions and HDFS
- Quotas and HDFS
- libhdfs (C API)
- WebHDFS (REST API)
- HttpFS
- Short Circuit Local Reads
- Centralized Cache Management
- NFS Gateway
- Rolling Upgrade
- Extended Attributes
- Transparent Encryption
- Multihoming
- Storage Policies
- Memory Storage Support
- Synthetic Load Generator
- Erasure Coding
- Disk Balancer
- Upgrade Domain
- DataNode Admin
- Router Federation
- Provided Storage

##### MapReduce

- Tutorial
- Commands Reference
- Encrypted Shuffle
- Pluggable Shuffle/Sort
- Distributed Cache Deploy
- Support for YARN Shared Cache

##### MapReduce REST APIs

- MR Application Master
- MR History Server

##### YARN

- Architecture
- Commands Reference
- Capacity Scheduler
- Fair Scheduler
- ResourceManager Restart
- ResourceManager HA
- Resource Model
- Node Labels
- Node Attributes
- Web Application Proxy
- Timeline Server
- Timeline Service V.2
- Writing YARN Applications
- YARN Application Security
- NodeManager
- Running Applications in Docker Containers
- Running Applications in runC Containers
- Using CGroups
- Secure Containers
- Reservation System
- Graceful Decommission
- Opportunistic Containers
- YARN Federation
- Shared Cache
- Using GPU
- Using FPGA
- Placement Constraints
- YARN UI2

##### YARN REST APIs

- Introduction
- Resource Manager
- Node Manager
- Timeline Server
- Timeline Service V.2

##### YARN Service

- Overview
- QuickStart
- Concepts
- Yarn Service API
- Service Discovery
- System Services

##### Hadoop Compatible File Systems

- Aliyun OSS
- Amazon S3
- Azure Blob Storage
- Azure Data Lake Storage
- Tencent COS
- Huaweicloud OBS
- VolcanoEngine TOS

##### Auth

- Overview
- Examples
- Configuration
- Building

##### Tools

- Hadoop Streaming
- Hadoop Archives
- Hadoop Archive Logs
- DistCp
- HDFS Federation Balance
- GridMix
- Rumen
- Resource Estimator Service
- Scheduler Load Simulator
- Hadoop Benchmarking
- Dynamometer

##### Reference

- Changelog and Release Notes
- Java API docs
- Unix Shell API
- Metrics

##### Configuration

- core-default.xml
- hdfs-default.xml
- hdfs-rbf-default.xml
- mapred-default.xml
- yarn-default.xml
- kms-default.xml
- httpfs-default.xml
- Deprecated Properties

# Upgrading S3A to AWS SDK V2

This document explains the upcoming work for upgrading S3A to AWS SDK V2. This work is tracked in HADOOP-18073.

## Why the upgrade?

- Moving to SDK V2 will provide performance benefits. For example, the transfer manager for SDK V2 is built using java bindings of the AWS Common Runtime S3 client (https://github.com/awslabs/aws-crt-java) (CRT). CRT is a set of packages written in C, designed for maximising performance when interacting with AWS services such as S3.
- The V1 SDK is essentially in maintenance mode.
- New features such as additional checksum algorithms which S3A will benefit from are not available in SDK V1.

## What’s changing?

The SDK V2 for S3 is very different from SDK V1, and brings breaking changes for S3A. A complete list of the changes can be found in the Changelog.

## Packaging: `aws-java-sdk-bundle-1.12.x.jar` becomes `bundle-2.x.y.jar`

As the module name is lost, in hadoop releases a large JAR file with the name “bundle” is now part of the distribution. This is the AWS V2 SDK shaded artifact.

The new and old SDKs can co-exist; the only place that the hadoop code may still use the original SDK is when a non-standard V1 AWS credential provider is declared.

Any deployment of the S3A connector must include this JAR or the subset of non-shaded aws- JARs needed for communication with S3 and any other services used. As before: the exact set of dependencies used by the S3A connector is neither defined nor comes with any commitments of stability or compatibility of dependent libraries.

## Credential Provider changes and migration

- Interface change: com.amazonaws.auth.AWSCredentialsProvider has been replaced by software.amazon.awssdk.auth.credentials.AwsCredentialsProvider.
- Credential provider class changes: the package and class names of credential providers have changed.

The change in interface will mean that custom credential providers will need to be updated to now implement `software.amazon.awssdk.auth.credentials.AwsCredentialsProvider` instead of `com.amazonaws.auth.AWSCredentialsProvider`.

HADOOP-18980 introduces extended version of the credential provider remapping. `fs.s3a.aws.credentials.provider.mapping` can be used to list comma-separated key-value pairs of mapped credential providers that are separated by equal operator (=). The key can be used by `fs.s3a.aws.credentials.provider` or `fs.s3a.assumed.role.credentials.provider` configs, and the key will be translated into the specified value of credential provider class based on the key-value pair provided by the config `fs.s3a.aws.credentials.provider.mapping`.

For example, if `fs.s3a.aws.credentials.provider.mapping` is set with value:

    <property>
      <name>fs.s3a.aws.credentials.provider.mapping</name>
      <vale>
        com.amazonaws.auth.AnonymousAWSCredentials=org.apache.hadoop.fs.s3a.AnonymousAWSCredentialsProvider,
        com.amazonaws.auth.EC2ContainerCredentialsProviderWrapper=org.apache.hadoop.fs.s3a.auth.IAMInstanceCredentialsProvider,
        com.amazonaws.auth.InstanceProfileCredentialsProvider=org.apache.hadoop.fs.s3a.auth.IAMInstanceCredentialsProvider
      </vale>
    </property>

and if `fs.s3a.aws.credentials.provider` is set with:

    <property>
      <name>fs.s3a.aws.credentials.provider</name>
      <vale>com.amazonaws.auth.AnonymousAWSCredentials</vale>
    </property>

`com.amazonaws.auth.AnonymousAWSCredentials` will be internally remapped to `org.apache.hadoop.fs.s3a.AnonymousAWSCredentialsProvider` by S3A while preparing the AWS credential provider list.

Similarly, if `fs.s3a.assumed.role.credentials.provider` is set with:

    <property>
      <name>fs.s3a.assumed.role.credentials.provider</name>
      <vale>com.amazonaws.auth.InstanceProfileCredentialsProvider</vale>
    </property>

`com.amazonaws.auth.InstanceProfileCredentialsProvider` will be internally remapped to `org.apache.hadoop.fs.s3a.auth.IAMInstanceCredentialsProvider` by S3A while preparing the assumed role AWS credential provider list.

### Original V1 `AWSCredentialsProvider` interface

Note how the interface begins with the capitalized “AWS” acronym. The V2 interface starts with “Aws”. This is a very subtle change for developers to spot. Compilers will detect and report the type mismatch.

    package com.amazonaws.auth;

    public interface AWSCredentialsProvider {

        public AWSCredentials getCredentials();

        public void refresh();

    }

The interface binding also supported a factory method, `AWSCredentialsProvider instance()` which, if available, would be invoked in preference to using any constructor.

If the interface implemented `Closeable` or `AutoCloseable`, these would be invoked when the provider chain was being shut down.

### New V2 `AwsCredentialsProvider` interface

    package software.amazon.awssdk.auth.credentials;

    public interface AwsCredentialsProvider {

      AwsCredentials resolveCredentials();

    }

1. There is no `refresh()` method any more.
2. `getCredentials()` has become `resolveCredentials()`.
3. There is now the expectation in the SDK that credential resolution/lookup etc will be performed in `resolveCredentials()`.
4. If the interface implements `Closeable` or `AutoCloseable`, these will be invoked when the provider chain is being shut down.
5. A static method `create()` which returns an `AwsCredentialsProvider` or subclass; this will be used in preference to a constructor

### S3A `AWSCredentialProviderList` is now a V2 credential provider

The class `org.apache.hadoop.fs.s3a.AWSCredentialProviderList` has moved from being a V1 to a V2 credential provider; even if an instance can be created with existing code, the V1 methods will not resolve:

    java.lang.NoSuchMethodError: org.apache.hadoop.fs.s3a.AWSCredentialProviderList.getCredentials()Lcom/amazonaws/auth/AWSCredentials;
      at org.apache.hadoop.fs.store.diag.S3ADiagnosticsInfo.validateFilesystem(S3ADiagnosticsInfo.java:903)

### Migration of Credential Providers listed in `fs.s3a.aws.credentials.provider`

Before: `fs.s3a.aws.credentials.provider` took a list of v1 credential providers, This took a list containing 1. V1 credential providers implemented in the `hadoop-aws` module. 2. V1 credential providers implemented in the `aws-sdk-bundle` library. 3. Custom V1 credential providers placed onto the classpath. 4. Custom subclasses of hadoop-aws credential providers.

And here is how they change 1. All `hadoop-aws` credential providers migrated to V2. 2. Well-known `aws-sdk-bundle` credential providers automatically remapped to their V2 equivalents. 3. Custom v1 providers supported if the original `aws-sdk-bundle` JAR is on the classpath. 4. Custom subclasses of hadoop-aws credential providers need manual migration.

Because of (1) and (2), As result, standard `fs.s3a.aws.credentials.provider` configurations should seamlessly upgrade. This also means that the same provider list, if restricted to those classes, will work across versions.

### `hadoop-aws` credential providers migration to V2

All the fs.s3a credential providers have the same name and functionality as before.

Hadoop module credential provider | Authentication Mechanism
--- | ---
`org.apache.hadoop.fs.s3a.TemporaryAWSCredentialsProvider` | Session Credentials in configuration
`org.apache.hadoop.fs.s3a.SimpleAWSCredentialsProvider` | Simple name/secret credentials in configuration
`org.apache.hadoop.fs.s3a.AnonymousAWSCredentialsProvider` | Anonymous Login
`org.apache.hadoop.fs.s3a.auth.AssumedRoleCredentialProvider` | Assumed Role credentials
`org.apache.hadoop.fs.s3a.auth.IAMInstanceCredentialsProvider` | EC2/k8s instance credentials

### Automatic `aws-sdk-bundle` credential provider remapping

The commonly-used set of V1 credential providers are automatically remapped to V2 equivalents.

V1 Credential Provider | Remapped V2 substitute
--- | ---
`com.amazonaws.auth.AnonymousAWSCredentials` | `org.apache.hadoop.fs.s3a.AnonymousAWSCredentialsProvider`
`com.amazonaws.auth.EnvironmentVariableCredentialsProvider` | `software.amazon.awssdk.auth.credentials.EnvironmentVariableCredentialsProvider`
`com.amazonaws.auth.EC2ContainerCredentialsProviderWrapper` | `org.apache.hadoop.fs.s3a.auth.IAMInstanceCredentialsProvider`
`com.amazonaws.auth.InstanceProfileCredentialsProvider` | `org.apache.hadoop.fs.s3a.auth.IAMInstanceCredentialsProvider`
`com.amazonaws.auth.profile.ProfileCredentialsProvider` | `software.amazon.awssdk.auth.credentials.ProfileCredentialsProvider`

There are still a number of troublespots here:

#### Less widely used`com.amazonaws.auth.` AWS providers

There should be equivalents in the new SDK, but as well as being renamed they are likely to have moved different factory/builder mechanisms. Identify the changed classes and use their names in the `fs.s3a.aws.credentials.provider` option.

If a V2 equivalent is not found; provided the V1 SDK is added to the classpath, it should still be possible to use the existing classes.

#### Private/third-party credential providers

Provided the V1 SDK is added to the classpath, it should still be possible to use the existing classes.

Adding a V2 equivalent is the recommended long-term solution.

#### Custom subclasses of the Hadoop credential providers

Because all the standard hadoop credential providers have been upgraded, any subclasses of these are not going to link or work.

These will need to be manually migrated to being V2 Credential providers.

## Source code/binary integration changes

The major changes and how this affects S3A are listed below.

### SDK API Package Change

- Package names have changed, all classes in SDK V2 are under `software.amazon.awssdk`, SDK V1 classes were under `com.amazonaws`.
- There is no interoperability between the old and new classes.
- All classnames are different, often in very subtle ways. It is possible to use both in the same class, as is done in the package `org.apache.hadoop.fs.s3a.adapter`.
- All the core message classes are now automatically generated from a JSON protocol description.
- All getter methods have been renamed.
- All classes are constructed via builder methods
- Message classes are no longer Java `Serializable`.

Most of these changes simply create what will feel to be gratuitous migration effort; the removable of the `Serializable` nature from all message response classes can potentially break applications -such as anything passing them between Spark workers. See AWS SDK V2 issue Simplify Modeled Message Marshalling #82, note that it was filed in 2017, then implement your own workaround pending that issue being resolved.

### Compilation/Linkage Errors

Any code making use of V1 sdk classes will fail if they * Expect the V1 sdk classes to be on the classpath when `hadoop-aws` is declared as a dependency * Use V1-SDK-compatible methods previously exported by the `S3AFileSystem` class and associated classes. * Try to pass s3a classes to V1 SDK classes (e.g. credential providers).

The sole solution to these problems is “move to the V2 SDK”.

Some `S3AUtils` methods are deleted

    cannot find symbol
    [ERROR]   symbol:   method createAwsConf(org.apache.hadoop.conf.Configuration,java.lang.String)
    [ERROR]   location: class org.apache.hadoop.fs.s3a.S3AUtils

The signature and superclass of `AWSCredentialProviderList` has changed, which can surface in different ways

Signature mismatch

     cannot find symbol
    [ERROR]   symbol:   method getCredentials()
    [ERROR]   location: variable credentials of type org.apache.hadoop.fs.s3a.AWSCredentialProviderList

It is no longer a V1 credential provider, cannot be used to pass credentials to a V1 SDK class

    incompatible types: org.apache.hadoop.fs.s3a.AWSCredentialProviderList cannot be converted to com.amazonaws.auth.AWSCredentialsProvider

### `AmazonS3` replaced by `S3Client`; factory and accessor changed.

The V1 s3 client class `com.amazonaws.services.s3.AmazonS3` has been superseded by `software.amazon.awssdk.services.s3.S3Client`

The `S3ClientFactory` interface has been replaced by one that creates a V2 `S3Client`. * Custom implementations will need to be updated. * The `InconsistentS3ClientFactory` class has been deleted.

### `S3AFileSystem` method changes: `S3AInternals`.

The low-level s3 operations/client accessors have been moved into a new interface, `org.apache.hadoop.fs.s3a.S3AInternals`, which must be accessed via the `S3AFileSystem.getS3AInternals()` method. They have also been updated to return V2 SDK classes.

    @InterfaceStability.Unstable
    @InterfaceAudience.LimitedPrivate("testing/diagnostics")
    public interface S3AInternals {
      S3Client getAmazonS3V2Client(String reason);

      S3AStore getStore();
      @Retries.RetryTranslated
      @AuditEntryPoint
      String getBucketLocation() throws IOException;

      @AuditEntryPoint
      @Retries.RetryTranslated
      String getBucketLocation(String bucketName) throws IOException;

      @AuditEntryPoint
      @Retries.RetryTranslated
      HeadObjectResponse getObjectMetadata(Path path) throws IOException;

      AWSCredentialProviderList shareCredentials(final String purpose);

      @AuditEntryPoint
      @Retries.RetryTranslated
      HeadBucketResponse getBucketMetadata() throws IOException;

      boolean isMultipartCopyEnabled();

      @AuditEntryPoint
      @Retries.RetryTranslated
      long abortMultipartUploads(Path path) throws IOException;
    }

#### `S3AFileSystem.getAmazonS3ClientForTesting(String)` moved and return type changed

The `S3AFileSystem.getAmazonS3ClientForTesting()` method has been been deleted.

Compilation

    cannot find symbol
    [ERROR]   symbol:   method getAmazonS3ClientForTesting(java.lang.String)
    [ERROR]   location: variable fs of type org.apache.hadoop.fs.s3a.S3AFileSystem

It has been replaced by an `S3AInternals` equivalent which returns the V2 `S3Client` of the filesystem instance.

    ((S3AFilesystem)fs).getAmazonS3ClientForTesting("testing")

    ((S3AFilesystem)fs).getS3AInternals().getAmazonS3Client("testing")

##### `S3AFileSystem.getObjectMetadata(Path path)` moved to `S3AInternals`; return type changed

The `getObjectMetadata(Path)` call has been moved to the `S3AInternals` interface and an instance of the `software.amazon.awssdk.services.s3.model.HeadObjectResponse` class returned. The original `S3AFileSystem` method has been deleted

Before:

    ((S3AFilesystem)fs).getObjectMetadata(path)

After:

    ((S3AFilesystem)fs).getS3AInternals().getObjectMetadata(path)

##### `AWSCredentialProviderList shareCredentials(String)` moved to `S3AInternals`

The operation to share a reference-counted access to the AWS credentials used by the S3A FS has been moved to `S3AInternals`.

This is very much an implementation method, used to allow extension modules to share an authentication chain into other AWS SDK client services (dynamoDB, etc.).

### Delegation Tokens

1. Custom credential providers used in delegation token binding classes will need to be updated
2. The return type from delegation token binding has changed to support more class instances being returned in the future.

`AWSCredentialProviderList` has been upgraded to the V2 API. * It still retains a `refresh()` method but this is now a deprecated no-op. * It is still `Closeable`; its `close()` method iterates through all entries in the list; if they are `Closeable` or `AutoCloseable` then their `close()` method is invoked. * Accordingly, providers may still perform background refreshes in separate threads; the S3A client will close its provider list when the filesystem itself is closed.

### Signers

Interface change: com.amazonaws.auth.Signer has been replaced by software.amazon.awssdk.core.signer.Signer.

The change in signers will mean the custom signers will need to be updated to implement the new interface.

There is no support to assist in this migration.

### S3A Auditing Extensions.

The callbacks from the SDK have all changed, as has the interface `org.apache.hadoop.fs.s3a.audit.AWSAuditEventCallbacks`

Examine the interface and associated implementations to see how to migrate.

The option `fs.s3a.audit.request.handlers` to declare a list of v1 SDK `com.amazonaws.handlers.RequestHandler2` implementations to include in the AWS request chain is no longer supported: a warning is printed and the value ignored.

The V2 SDK equivalent, classes implementing `software.amazon.awssdk.core.interceptor.ExecutionInterceptor` can be declared in the configuration option `fs.s3a.audit.execution.interceptors`.