Apache Hadoop 3.5.0 – Apache Hadoop 2.5.0 Release Notes

##### General

- Overview
- Single Node Setup
- Cluster Setup
- Commands Reference
- FileSystem Shell
- Compatibility Specification
- Downstream Developer's Guide
- Admin Compatibility Guide
- Interface Classification
- FileSystem Specification

##### Common

- CLI Mini Cluster
- Fair Call Queue
- Native Libraries
- Proxy User
- Rack Awareness
- Secure Mode
- Service Level Authorization
- HTTP Authentication
- Credential Provider API
- Hadoop KMS
- Tracing
- Unix Shell Guide
- Registry
- Async Profiler

##### HDFS

- Architecture
- User Guide
- Commands Reference
- NameNode HA With QJM
- NameNode HA With NFS
- Observer NameNode
- Federation
- ViewFs
- ViewFsOverloadScheme
- Snapshots
- Edits Viewer
- Image Viewer
- Permissions and HDFS
- Quotas and HDFS
- libhdfs (C API)
- WebHDFS (REST API)
- HttpFS
- Short Circuit Local Reads
- Centralized Cache Management
- NFS Gateway
- Rolling Upgrade
- Extended Attributes
- Transparent Encryption
- Multihoming
- Storage Policies
- Memory Storage Support
- Synthetic Load Generator
- Erasure Coding
- Disk Balancer
- Upgrade Domain
- DataNode Admin
- Router Federation
- Provided Storage

##### MapReduce

- Tutorial
- Commands Reference
- Encrypted Shuffle
- Pluggable Shuffle/Sort
- Distributed Cache Deploy
- Support for YARN Shared Cache

##### MapReduce REST APIs

- MR Application Master
- MR History Server

##### YARN

- Architecture
- Commands Reference
- Capacity Scheduler
- Fair Scheduler
- ResourceManager Restart
- ResourceManager HA
- Resource Model
- Node Labels
- Node Attributes
- Web Application Proxy
- Timeline Server
- Timeline Service V.2
- Writing YARN Applications
- YARN Application Security
- NodeManager
- Running Applications in Docker Containers
- Running Applications in runC Containers
- Using CGroups
- Secure Containers
- Reservation System
- Graceful Decommission
- Opportunistic Containers
- YARN Federation
- Shared Cache
- Using GPU
- Using FPGA
- Placement Constraints
- YARN UI2

##### YARN REST APIs

- Introduction
- Resource Manager
- Node Manager
- Timeline Server
- Timeline Service V.2

##### YARN Service

- Overview
- QuickStart
- Concepts
- Yarn Service API
- Service Discovery
- System Services

##### Hadoop Compatible File Systems

- Aliyun OSS
- Amazon S3
- Azure Blob Storage
- Azure Data Lake Storage
- Tencent COS
- Huaweicloud OBS
- VolcanoEngine TOS

##### Auth

- Overview
- Examples
- Configuration
- Building

##### Tools

- Hadoop Streaming
- Hadoop Archives
- Hadoop Archive Logs
- DistCp
- HDFS Federation Balance
- GridMix
- Rumen
- Resource Estimator Service
- Scheduler Load Simulator
- Hadoop Benchmarking
- Dynamometer

##### Reference

- Changelog and Release Notes
- Java API docs
- Unix Shell API
- Metrics

##### Configuration

- core-default.xml
- hdfs-default.xml
- hdfs-rbf-default.xml
- mapred-default.xml
- yarn-default.xml
- kms-default.xml
- httpfs-default.xml
- Deprecated Properties

# Apache Hadoop 2.5.0 Release Notes

These release notes cover new developer and user-facing incompatibilities, important issues, features, and major improvements.

- HADOOP-10342 | Major | Extend UserGroupInformation to return a UGI given a preauthenticated kerberos Subject

Add getUGIFromSubject to leverage an external kerberos authentication

- HDFS-6164 | Major | Remove lsr in OfflineImageViewer

The offlineimageviewer no longer generates lsr-style outputs. The functionality has been superseded by a tool that takes the fsimage and exposes WebHDFS-like API for user queries.

- HDFS-6168 | Major | Remove deprecated methods in DistributedFileSystem

WARNING: No release note provided for this change.

- HADOOP-10451 | Trivial | Remove unused field and imports from SaslRpcServer

SaslRpcServer.SASL_PROPS is removed. Any use of this variable should be replaced with the following code: SaslPropertiesResolver saslPropsResolver = SaslPropertiesResolver.getInstance(conf); Map<String, String> sasl_props = saslPropsResolver.getDefaultProperties();

- HDFS-6153 | Minor | Document “fileId” and “childrenNum” fields in the FileStatus Json schema

WARNING: No release note provided for this change.

- HADOOP-9919 | Major | Update hadoop-metrics2.properties examples to Yarn

Remove MRv1 settings from hadoop-metrics2.properties, add YARN settings instead.

- HDFS-6273 | Major | Config options to allow wildcard endpoints for namenode HTTP and HTTPS servers

HDFS-6273 introduces two new HDFS configuration keys: - dfs.namenode.http-bind-host - dfs.namenode.https-bind-host

The most common use case for these keys is to have the NameNode HTTP (or HTTPS) endpoints listen on all interfaces on multi-homed systems by setting the keys to 0.0.0.0 i.e. INADDR_ANY.

For the systems background on this usage of INADDR_ANY please refer to ip(7) in the Linux Programmer’s Manual (web link: http://man7.org/linux/man-pages/man7/ip.7.html).

These keys complement the existing NameNode options: - dfs.namenode.rpc-bind-host - dfs.namenode.servicerpc-bind-host

- HADOOP-10568 | Major | Add s3 server-side encryption

s3 server-side encryption is now supported.

To enable this feature, specify the following in your client-side configuration:

name: fs.s3n.server-side-encryption-algorithm value: AES256

- HDFS-6293 | Blocker | Issues with OIV processing PB-based fsimages

Set “dfs.namenode.legacy-oiv-image.dir” to an appropriate directory to make standby name node or secondary name node save its file system state in the old fsimage format during checkpointing. This image can be used for offline analysis using the OfflineImageViewer. Use the “hdfs oiv_legacy” command to process the old fsimage format.

- HDFS-6110 | Major | adding more slow action log in critical write path

Log slow i/o. Set log thresholds in dfsclient and datanode via the below new configs:

dfs.client.slow.io.warning.threshold.ms (Default 30 seconds) dfs.datanode.slow.io.warning.threshold.ms (Default 300ms)

- YARN-2107 | Major | Refactor timeline classes into server.timeline package

WARNING: No release note provided for this change.

- MAPREDUCE-5777 | Major | Support utf-8 text with BOM (byte order marker)

WARNING: No release note provided for this change.