Apache Hadoop 3.5.0 – Apache Hadoop 2.6.0 Release Notes

##### General

- Overview
- Single Node Setup
- Cluster Setup
- Commands Reference
- FileSystem Shell
- Compatibility Specification
- Downstream Developer's Guide
- Admin Compatibility Guide
- Interface Classification
- FileSystem Specification

##### Common

- CLI Mini Cluster
- Fair Call Queue
- Native Libraries
- Proxy User
- Rack Awareness
- Secure Mode
- Service Level Authorization
- HTTP Authentication
- Credential Provider API
- Hadoop KMS
- Tracing
- Unix Shell Guide
- Registry
- Async Profiler

##### HDFS

- Architecture
- User Guide
- Commands Reference
- NameNode HA With QJM
- NameNode HA With NFS
- Observer NameNode
- Federation
- ViewFs
- ViewFsOverloadScheme
- Snapshots
- Edits Viewer
- Image Viewer
- Permissions and HDFS
- Quotas and HDFS
- libhdfs (C API)
- WebHDFS (REST API)
- HttpFS
- Short Circuit Local Reads
- Centralized Cache Management
- NFS Gateway
- Rolling Upgrade
- Extended Attributes
- Transparent Encryption
- Multihoming
- Storage Policies
- Memory Storage Support
- Synthetic Load Generator
- Erasure Coding
- Disk Balancer
- Upgrade Domain
- DataNode Admin
- Router Federation
- Provided Storage

##### MapReduce

- Tutorial
- Commands Reference
- Encrypted Shuffle
- Pluggable Shuffle/Sort
- Distributed Cache Deploy
- Support for YARN Shared Cache

##### MapReduce REST APIs

- MR Application Master
- MR History Server

##### YARN

- Architecture
- Commands Reference
- Capacity Scheduler
- Fair Scheduler
- ResourceManager Restart
- ResourceManager HA
- Resource Model
- Node Labels
- Node Attributes
- Web Application Proxy
- Timeline Server
- Timeline Service V.2
- Writing YARN Applications
- YARN Application Security
- NodeManager
- Running Applications in Docker Containers
- Running Applications in runC Containers
- Using CGroups
- Secure Containers
- Reservation System
- Graceful Decommission
- Opportunistic Containers
- YARN Federation
- Shared Cache
- Using GPU
- Using FPGA
- Placement Constraints
- YARN UI2

##### YARN REST APIs

- Introduction
- Resource Manager
- Node Manager
- Timeline Server
- Timeline Service V.2

##### YARN Service

- Overview
- QuickStart
- Concepts
- Yarn Service API
- Service Discovery
- System Services

##### Hadoop Compatible File Systems

- Aliyun OSS
- Amazon S3
- Azure Blob Storage
- Azure Data Lake Storage
- Tencent COS
- Huaweicloud OBS
- VolcanoEngine TOS

##### Auth

- Overview
- Examples
- Configuration
- Building

##### Tools

- Hadoop Streaming
- Hadoop Archives
- Hadoop Archive Logs
- DistCp
- HDFS Federation Balance
- GridMix
- Rumen
- Resource Estimator Service
- Scheduler Load Simulator
- Hadoop Benchmarking
- Dynamometer

##### Reference

- Changelog and Release Notes
- Java API docs
- Unix Shell API
- Metrics

##### Configuration

- core-default.xml
- hdfs-default.xml
- hdfs-rbf-default.xml
- mapred-default.xml
- yarn-default.xml
- kms-default.xml
- httpfs-default.xml
- Deprecated Properties

# Apache Hadoop 2.6.0 Release Notes

These release notes cover new developer and user-facing incompatibilities, important issues, features, and major improvements.

- HADOOP-10244 | Major | TestKeyShell improperly tests the results of a Delete

Fix of inappropriate test of delete functionality.

- HADOOP-10583 | Minor | bin/hadoop key throws NPE with no args and assorted other fixups

bin/hadoop key with no args would throw an NPE.

- HDFS-2856 | Major | Fix block protocol so that Datanodes don’t require root or jsvc

SASL now can be used to secure the DataTransferProtocol, which transfers file block content between HDFS clients and DataNodes. In this configuration, it is no longer required for secured clusters to start the DataNode as root and bind to privileged ports.

- HADOOP-10839 | Major | Add unregisterSource() to MetricsSystem API

The MetricsSystem abstract class has added a new abstract method, unregisterSource, for unregistering a previously registered metrics source. Custom subclasses of MetricsSystem must be updated to provide an implementation of this method.

- HADOOP-8069 | Major | Enable TCP_NODELAY by default for IPC

This change enables the TCP_NODELAY flag for all Hadoop IPC connections, hence bypassing TCP Nagling. Nagling interacts poorly with TCP delayed ACKs especially for request-response protocols.

- HDFS-6482 | Major | Use block ID-based block layout on datanodes

The directory structure for finalized replicas on DNs has been changed. Now, the directory that a finalized replica goes in is determined uniquely by its ID. Specifically, we use a two-level directory structure, with the 24th through 17th bits identifying the correct directory at the first level and the 16th through 8th bits identifying the correct directory at the second level.

- HADOOP-10903 | Major | Enhance hadoop classpath command to expand wildcards or write classpath into jar manifest.

The “hadoop classpath” command has been enhanced to support options for automatic expansion of wildcards in classpath elements and writing the classpath to a jar file manifest. These options make it easier to construct a correct classpath for libhdfs applications.

- HDFS-573 | Major | Porting libhdfs to Windows

The libhdfs C API is now supported on Windows.

- HADOOP-8944 | Trivial | Shell command fs -count should include human readable option

Implements -h option for fs -count to show file sizes in human readable format. Additionally, ContentSummary.getHeader() now returns a different string that is incompatible with previous releases.

- HDFS-6376 | Major | Distcp data between two HA clusters requires another configuration

Allow distcp to copy data between HA clusters. Users can use a new configuration property “dfs.internal.nameservices” to explicitly specify the name services belonging to the local cluster, while continue using the configuration property “dfs.nameservices” to specify all the name services in the local and remote clusters.

- YARN-668 | Blocker | TokenIdentifier serialization should consider Unknown fields

WARNING: No release note provided for this change.

- YARN-1051 | Major | YARN Admission Control/Planner: enhancing the resource allocation model with time.

WARNING: No release note provided for this change.

- HADOOP-10681 | Major | Remove synchronized blocks from SnappyCodec and ZlibCodec buffering inner loop

Remove unnecessary synchronized blocks from Snappy/Zlib codecs.

- YARN-2615 | Blocker | ClientToAMTokenIdentifier and DelegationTokenIdentifier should allow extended fields

WARNING: No release note provided for this change.

- HDFS-6606 | Major | Optimize HDFS Encrypted Transport performance

HDFS now supports the option to configure AES encryption for block data transfer. AES offers improved cryptographic strength and performance over the prior options of 3DES and RC4.

- HDFS-7276 | Major | Limit the number of byte arrays used by DFSOutputStream

The following configuration properties are added.

- dfs.client.write.byte-array-manager.enabled: for enabling/disabling byte array manger. Default is false.

- dfs.client.write.byte-array-manager.count-threshold: The count threshold for each array length so that a manager is created only after the allocation count exceeds the threshold. In other words, the particular array length is not managed until the allocation count exceeds the threshold. Default is 128.

- dfs.client.write.byte-array-manager.count-limit: The maximum number of arrays allowed for each array length. Default is 2048.

- dfs.client.write.byte-array-manager.count-reset-time-period-ms: The time period in milliseconds that the allocation count for each array length is reset to zero if there is no increment. Default is 10,000ms, i.e. 10 seconds.