Apache Hadoop 3.5.0 – Apache Hadoop 3.0.0 Release Notes

##### General

- Overview
- Single Node Setup
- Cluster Setup
- Commands Reference
- FileSystem Shell
- Compatibility Specification
- Downstream Developer's Guide
- Admin Compatibility Guide
- Interface Classification
- FileSystem Specification

##### Common

- CLI Mini Cluster
- Fair Call Queue
- Native Libraries
- Proxy User
- Rack Awareness
- Secure Mode
- Service Level Authorization
- HTTP Authentication
- Credential Provider API
- Hadoop KMS
- Tracing
- Unix Shell Guide
- Registry
- Async Profiler

##### HDFS

- Architecture
- User Guide
- Commands Reference
- NameNode HA With QJM
- NameNode HA With NFS
- Observer NameNode
- Federation
- ViewFs
- ViewFsOverloadScheme
- Snapshots
- Edits Viewer
- Image Viewer
- Permissions and HDFS
- Quotas and HDFS
- libhdfs (C API)
- WebHDFS (REST API)
- HttpFS
- Short Circuit Local Reads
- Centralized Cache Management
- NFS Gateway
- Rolling Upgrade
- Extended Attributes
- Transparent Encryption
- Multihoming
- Storage Policies
- Memory Storage Support
- Synthetic Load Generator
- Erasure Coding
- Disk Balancer
- Upgrade Domain
- DataNode Admin
- Router Federation
- Provided Storage

##### MapReduce

- Tutorial
- Commands Reference
- Encrypted Shuffle
- Pluggable Shuffle/Sort
- Distributed Cache Deploy
- Support for YARN Shared Cache

##### MapReduce REST APIs

- MR Application Master
- MR History Server

##### YARN

- Architecture
- Commands Reference
- Capacity Scheduler
- Fair Scheduler
- ResourceManager Restart
- ResourceManager HA
- Resource Model
- Node Labels
- Node Attributes
- Web Application Proxy
- Timeline Server
- Timeline Service V.2
- Writing YARN Applications
- YARN Application Security
- NodeManager
- Running Applications in Docker Containers
- Running Applications in runC Containers
- Using CGroups
- Secure Containers
- Reservation System
- Graceful Decommission
- Opportunistic Containers
- YARN Federation
- Shared Cache
- Using GPU
- Using FPGA
- Placement Constraints
- YARN UI2

##### YARN REST APIs

- Introduction
- Resource Manager
- Node Manager
- Timeline Server
- Timeline Service V.2

##### YARN Service

- Overview
- QuickStart
- Concepts
- Yarn Service API
- Service Discovery
- System Services

##### Hadoop Compatible File Systems

- Aliyun OSS
- Amazon S3
- Azure Blob Storage
- Azure Data Lake Storage
- Tencent COS
- Huaweicloud OBS
- VolcanoEngine TOS

##### Auth

- Overview
- Examples
- Configuration
- Building

##### Tools

- Hadoop Streaming
- Hadoop Archives
- Hadoop Archive Logs
- DistCp
- HDFS Federation Balance
- GridMix
- Rumen
- Resource Estimator Service
- Scheduler Load Simulator
- Hadoop Benchmarking
- Dynamometer

##### Reference

- Changelog and Release Notes
- Java API docs
- Unix Shell API
- Metrics

##### Configuration

- core-default.xml
- hdfs-default.xml
- hdfs-rbf-default.xml
- mapred-default.xml
- yarn-default.xml
- kms-default.xml
- httpfs-default.xml
- Deprecated Properties

# Apache Hadoop 3.0.0 Release Notes

These release notes cover new developer and user-facing incompatibilities, important issues, features, and major improvements.

- YARN-7219 | Critical | Make AllocateRequestProto compatible with branch-2/branch-2.8

Change update_requests field id to 7. This matches the current field id in branch-2 and branch-2.8.

- YARN-1492 | Major | truly shared cache for jars (jobjar/libjar)

The YARN Shared Cache provides the facility to upload and manage shared application resources to HDFS in a safe and scalable manner. YARN applications can leverage resources uploaded by other applications or previous runs of the same application without having to re-upload and localize identical files multiple times. This will save network resources and reduce YARN application startup time.

- HDFS-10467 | Major | Router-based HDFS federation

HDFS Router-based Federation adds a RPC routing layer that provides a federated view of multiple HDFS namespaces. This is similar to the existing ViewFS and HDFS federation functionality, except the mount table is managed on the server-side by the routing layer rather than on the client. This simplifies access to a federated cluster for existing HDFS clients.

See HDFS-10467 and the HDFS Router-based Federation documentation for more details.

- YARN-5734 | Major | OrgQueue for easy CapacityScheduler queue configuration management

The OrgQueue extension to the capacity scheduler provides a programmatic way to change configurations by providing a REST API that users can call to modify queue configurations. This enables automation of queue configuration management by administrators in the queue’s `administer_queue` ACL.

- MAPREDUCE-5951 | Major | Add support for the YARN Shared Cache

MapReduce support for the YARN shared cache allows MapReduce jobs to take advantage of additional resource caching. This saves network bandwidth between the job submission client as well as within the YARN cluster itself. This will reduce job submission time and overall job runtime.

- YARN-6623 | Blocker | Add support to turn off launching privileged containers in the container-executor

A change in configuration for launching Docker containers under YARN. Docker container capabilities, mounts, networks and allowing privileged container have to specified in the container-executor.cfg. By default, all of the above are turned off. This change will break existing setups launching Docker containers under YARN. Please refer to the Docker containers under YARN documentation for more information.

- HADOOP-14816 | Major | Update Dockerfile to use Xenial

This patch changes the default build and test environment in the following ways:

* Switch from Ubuntu “Trusty” 14.04 to Ubuntu “Xenial” 16.04 * Switch from Oracle JDK 8 to OpenJDK 8 * Adds OpenJDK 9 to the build environment

- HADOOP-14957 | Major | ReconfigurationTaskStatus is exposing guava Optional in its public api

ReconfigurationTaskStatus’ API scope is reduced to LimitedPrivate, and its dependency on com.google.com.base.Optional is replaced by java.util.Optional.

- HADOOP-14840 | Major | Tool to estimate resource requirements of an application pipeline based on prior executions

The first version of Resource Estimator service, a tool that captures the historical resource usage of an app and predicts its future resource requirement.

- MAPREDUCE-6983 | Major | Moving logging APIs over to slf4j in hadoop-mapreduce-client-core

In hadoop-mapreduce-client-core module, the type of some public LOG variables were changed from org.apache.commons.logging.Log to org.slf4j.Logger. In the public methods that accepts logger, the logger was changed from org.apache.commons.logging.Log to org.slf4j.Logger.

- HDFS-12682 | Blocker | ECAdmin -listPolicies will always show SystemErasureCodingPolicies state as DISABLED

WARNING: No release note provided for this change.

- YARN-5085 | Major | Add support for change of container ExecutionType

This allows the Application Master to ask the Scheduler to change the ExecutionType of a running/allocated container.

- HDFS-12840 | Blocker | Creating a file with non-default EC policy in a EC zone is not correctly serialized in the editlog

Add ErasureCodingPolicyId to each OP_ADD edit log op.

- HADOOP-15059 | Blocker | 3.0 deployment cannot work with old version MR tar ball which breaks rolling upgrade

This change reverses the default delegation token format implemented by HADOOP-12563, but preserves the capability to read the new delegation token format. When the new format becomes default, then MR deployment jobs runs will be compatible with releases that contain this change.