Apache Hadoop Amazon Web Services support – S3 Select

##### General

- Overview
- Single Node Setup
- Cluster Setup
- Commands Reference
- FileSystem Shell
- Compatibility Specification
- Downstream Developer's Guide
- Admin Compatibility Guide
- Interface Classification
- FileSystem Specification

##### Common

- CLI Mini Cluster
- Fair Call Queue
- Native Libraries
- Proxy User
- Rack Awareness
- Secure Mode
- Service Level Authorization
- HTTP Authentication
- Credential Provider API
- Hadoop KMS
- Tracing
- Unix Shell Guide
- Registry
- Async Profiler

##### HDFS

- Architecture
- User Guide
- Commands Reference
- NameNode HA With QJM
- NameNode HA With NFS
- Observer NameNode
- Federation
- ViewFs
- ViewFsOverloadScheme
- Snapshots
- Edits Viewer
- Image Viewer
- Permissions and HDFS
- Quotas and HDFS
- libhdfs (C API)
- WebHDFS (REST API)
- HttpFS
- Short Circuit Local Reads
- Centralized Cache Management
- NFS Gateway
- Rolling Upgrade
- Extended Attributes
- Transparent Encryption
- Multihoming
- Storage Policies
- Memory Storage Support
- Synthetic Load Generator
- Erasure Coding
- Disk Balancer
- Upgrade Domain
- DataNode Admin
- Router Federation
- Provided Storage

##### MapReduce

- Tutorial
- Commands Reference
- Encrypted Shuffle
- Pluggable Shuffle/Sort
- Distributed Cache Deploy
- Support for YARN Shared Cache

##### MapReduce REST APIs

- MR Application Master
- MR History Server

##### YARN

- Architecture
- Commands Reference
- Capacity Scheduler
- Fair Scheduler
- ResourceManager Restart
- ResourceManager HA
- Resource Model
- Node Labels
- Node Attributes
- Web Application Proxy
- Timeline Server
- Timeline Service V.2
- Writing YARN Applications
- YARN Application Security
- NodeManager
- Running Applications in Docker Containers
- Running Applications in runC Containers
- Using CGroups
- Secure Containers
- Reservation System
- Graceful Decommission
- Opportunistic Containers
- YARN Federation
- Shared Cache
- Using GPU
- Using FPGA
- Placement Constraints
- YARN UI2

##### YARN REST APIs

- Introduction
- Resource Manager
- Node Manager
- Timeline Server
- Timeline Service V.2

##### YARN Service

- Overview
- QuickStart
- Concepts
- Yarn Service API
- Service Discovery
- System Services

##### Hadoop Compatible File Systems

- Aliyun OSS
- Amazon S3
- Azure Blob Storage
- Azure Data Lake Storage
- Tencent COS
- Huaweicloud OBS
- VolcanoEngine TOS

##### Auth

- Overview
- Examples
- Configuration
- Building

##### Tools

- Hadoop Streaming
- Hadoop Archives
- Hadoop Archive Logs
- DistCp
- HDFS Federation Balance
- GridMix
- Rumen
- Resource Estimator Service
- Scheduler Load Simulator
- Hadoop Benchmarking
- Dynamometer

##### Reference

- Changelog and Release Notes
- Java API docs
- Unix Shell API
- Metrics

##### Configuration

- core-default.xml
- hdfs-default.xml
- hdfs-rbf-default.xml
- mapred-default.xml
- yarn-default.xml
- kms-default.xml
- httpfs-default.xml
- Deprecated Properties

# S3 Select

S3 Select is a feature for Amazon S3 introduced in April 2018. It allows for SQL-like SELECT expressions to be applied to files in some structured formats, including CSV and JSON.

It is no longer supported in Hadoop releases.

Any Hadoop release built on the AWS V2 SDK will reject calls to open files using the select APIs.

If a build of Hadoop with S3 Select is desired, the relevant classes can be found in hadoop trunk commit `8bf72346a59c`.

## Consequences of the removal

The path capabilities probe `fs.s3a.capability.select.sql` returns “false” for any and all `s3a://` paths.

Any `openFile()` call where a SQL query is passed in as a `must()` clause SHALL raise `UnsupportedOperationException`:

    // fails
    openFile("s3a://bucket/path")
      .must("fs.s3a.select.sql", "SELECT ...")
      .get();

Any `openFile()` call to an S3A Path where a SQL query is passed in as a `may()` clause SHALL be logged at WARN level the first time it is invoked, then ignored.

    // ignores the option after printing a warning.
    openFile("s3a://bucket/path")
      .may("fs.s3a.select.sql", "SELECT ...")
      .get();

The `hadoop s3guard select` command is no longer supported.

Previously, the command would either generate an S3 select or a error (with exit code 42 being the one for not enough arguments):

    hadoop s3guard select
    select [OPTIONS] [-limit rows] [-header (use|none|ignore)] [-out path] [-expected rows]
      [-compression (gzip|bzip2|none)] [-inputformat csv] [-outputformat csv] <PATH> <SELECT QUERY>
            make an S3 Select call

    [main] INFO  util.ExitUtil (ExitUtil.java:terminate(241)) - Exiting with status 42:
           42: Too few arguments

Now it will fail with exit code 55 always:

    hadoop s3guard select

    [main] INFO  util.ExitUtil (ExitUtil.java:terminate(241)) - Exiting with status 55:
           55: S3 Select is no longer supported