Apache Hadoop 3.5.0 – Apache Hadoop 2.0.0-alpha Release Notes

##### General

- Overview
- Single Node Setup
- Cluster Setup
- Commands Reference
- FileSystem Shell
- Compatibility Specification
- Downstream Developer's Guide
- Admin Compatibility Guide
- Interface Classification
- FileSystem Specification

##### Common

- CLI Mini Cluster
- Fair Call Queue
- Native Libraries
- Proxy User
- Rack Awareness
- Secure Mode
- Service Level Authorization
- HTTP Authentication
- Credential Provider API
- Hadoop KMS
- Tracing
- Unix Shell Guide
- Registry
- Async Profiler

##### HDFS

- Architecture
- User Guide
- Commands Reference
- NameNode HA With QJM
- NameNode HA With NFS
- Observer NameNode
- Federation
- ViewFs
- ViewFsOverloadScheme
- Snapshots
- Edits Viewer
- Image Viewer
- Permissions and HDFS
- Quotas and HDFS
- libhdfs (C API)
- WebHDFS (REST API)
- HttpFS
- Short Circuit Local Reads
- Centralized Cache Management
- NFS Gateway
- Rolling Upgrade
- Extended Attributes
- Transparent Encryption
- Multihoming
- Storage Policies
- Memory Storage Support
- Synthetic Load Generator
- Erasure Coding
- Disk Balancer
- Upgrade Domain
- DataNode Admin
- Router Federation
- Provided Storage

##### MapReduce

- Tutorial
- Commands Reference
- Encrypted Shuffle
- Pluggable Shuffle/Sort
- Distributed Cache Deploy
- Support for YARN Shared Cache

##### MapReduce REST APIs

- MR Application Master
- MR History Server

##### YARN

- Architecture
- Commands Reference
- Capacity Scheduler
- Fair Scheduler
- ResourceManager Restart
- ResourceManager HA
- Resource Model
- Node Labels
- Node Attributes
- Web Application Proxy
- Timeline Server
- Timeline Service V.2
- Writing YARN Applications
- YARN Application Security
- NodeManager
- Running Applications in Docker Containers
- Running Applications in runC Containers
- Using CGroups
- Secure Containers
- Reservation System
- Graceful Decommission
- Opportunistic Containers
- YARN Federation
- Shared Cache
- Using GPU
- Using FPGA
- Placement Constraints
- YARN UI2

##### YARN REST APIs

- Introduction
- Resource Manager
- Node Manager
- Timeline Server
- Timeline Service V.2

##### YARN Service

- Overview
- QuickStart
- Concepts
- Yarn Service API
- Service Discovery
- System Services

##### Hadoop Compatible File Systems

- Aliyun OSS
- Amazon S3
- Azure Blob Storage
- Azure Data Lake Storage
- Tencent COS
- Huaweicloud OBS
- VolcanoEngine TOS

##### Auth

- Overview
- Examples
- Configuration
- Building

##### Tools

- Hadoop Streaming
- Hadoop Archives
- Hadoop Archive Logs
- DistCp
- HDFS Federation Balance
- GridMix
- Rumen
- Resource Estimator Service
- Scheduler Load Simulator
- Hadoop Benchmarking
- Dynamometer

##### Reference

- Changelog and Release Notes
- Java API docs
- Unix Shell API
- Metrics

##### Configuration

- core-default.xml
- hdfs-default.xml
- hdfs-rbf-default.xml
- mapred-default.xml
- yarn-default.xml
- kms-default.xml
- httpfs-default.xml
- Deprecated Properties

# Apache Hadoop 2.0.0-alpha Release Notes

These release notes cover new developer and user-facing incompatibilities, important issues, features, and major improvements.

- HDFS-395 | Major | DFS Scalability: Incremental block reports

WARNING: No release note provided for this change.

- HADOOP-7524 | Major | Change RPC to allow multiple protocols including multiple versions of the same protocol

WARNING: No release note provided for this change.

- HADOOP-7704 | Minor | JsonFactory can be created only once and used for every next request to create JsonGenerator inside JMXJsonServlet

Reduce number of object created by JMXJsonServlet. (Devaraj K via Eric Yang)

- MAPREDUCE-3818 | Blocker | Trunk MRV1 compilation is broken.

Fixed broken compilation in TestSubmitJob after the patch for HDFS-2895.

- HDFS-2731 | Major | HA: Autopopulate standby name dirs if they’re empty

The HA NameNode may now be started with the “-bootstrapStandby” flag. This causes it to copy the namespace information and most recent checkpoint from its HA pair, and save it to local storage, allowing an HA setup to be bootstrapped without use of rsync or external tools.

- HDFS-2303 | Major | Unbundle jsvc

To run secure Datanodes users must install jsvc for their platform and set JSVC_HOME to point to the location of jsvc in their environment.

- HDFS-3044 | Major | fsck move should be non-destructive by default

The fsck “move” option is no longer destructive. It copies the accessible blocks of corrupt files to lost and found as before, but no longer deletes the corrupt files after copying the blocks. The original, destructive behavior can be enabled by specifying both the “move” and “delete” options.

- HADOOP-8154 | Major | DNS#getIPs shouldn’t silently return the local host IP for bogus interface names

WARNING: No release note provided for this change.

- HADOOP-8184 | Major | ProtoBuf RPC engine does not need it own reply packet - it can use the IPC layer reply packet.

This change will affect the output of errors for some Hadoop CLI commands. Specifically, the name of the exception class will no longer appear, and instead only the text of the exception message will appear.

- HADOOP-8149 | Major | cap space usage of default log4j rolling policy

Hadoop log files are now rolled by size instead of date (daily) by default. Tools that depend on the log file name format will need to be updated. Users who would like to maintain the previous settings of hadoop.root.logger and hadoop.security.logger can use their current log4j.properties files and update the HADOOP_ROOT_LOGGER and HADOOP_SECURITY_LOGGER environment variables to use DRFA and DRFAS respectively.

- HDFS-3137 | Major | Bump LAST_UPGRADABLE_LAYOUT_VERSION to -16

Upgrade from Hadoop versions earlier than 0.18 is not supported as of 2.0. To upgrade from an earlier release, first upgrade to 0.18, and then upgrade again from there.

- HDFS-3138 | Major | Move DatanodeInfo#ipcPort to DatanodeID

This change modifies DatanodeID, which is part of the client to server protocol, therefore clients must be upgraded with servers.

- HDFS-3164 | Major | Move DatanodeInfo#hostName to DatanodeID

This change modifies DatanodeID, which is part of the client to server protocol, therefore clients must be upgraded with servers.

- HDFS-3144 | Major | Refactor DatanodeID#getName by use

This change modifies DatanodeID, which is part of the client to server protocol, therefore clients must be upgraded with servers.

- HDFS-3004 | Major | Implement Recovery Mode

This is a new feature. It is documented in hdfs_user_guide.xml.

- HADOOP-8270 | Minor | hadoop-daemon.sh stop action should return 0 for an already stopped service

The daemon stop action no longer returns failure when stopping an already stopped service.

- HDFS-3094 | Major | add -nonInteractive and -force option to namenode -format command

The ‘namenode -format’ command now supports the flags ‘-nonInteractive’ and ‘-force’ to improve usefulness without user input.

- HADOOP-8314 | Major | HttpServer#hasAdminAccess should return false if authorization is enabled but user is not authenticated

WARNING: No release note provided for this change.

- HDFS-3286 | Major | When the threshold value for balancer is 0(zero) ,unexpected output is displayed

WARNING: No release note provided for this change.