Apache Hadoop 3.5.0 – Apache Hadoop Changelog

##### General

- Overview
- Single Node Setup
- Cluster Setup
- Commands Reference
- FileSystem Shell
- Compatibility Specification
- Downstream Developer's Guide
- Admin Compatibility Guide
- Interface Classification
- FileSystem Specification

##### Common

- CLI Mini Cluster
- Fair Call Queue
- Native Libraries
- Proxy User
- Rack Awareness
- Secure Mode
- Service Level Authorization
- HTTP Authentication
- Credential Provider API
- Hadoop KMS
- Tracing
- Unix Shell Guide
- Registry
- Async Profiler

##### HDFS

- Architecture
- User Guide
- Commands Reference
- NameNode HA With QJM
- NameNode HA With NFS
- Observer NameNode
- Federation
- ViewFs
- ViewFsOverloadScheme
- Snapshots
- Edits Viewer
- Image Viewer
- Permissions and HDFS
- Quotas and HDFS
- libhdfs (C API)
- WebHDFS (REST API)
- HttpFS
- Short Circuit Local Reads
- Centralized Cache Management
- NFS Gateway
- Rolling Upgrade
- Extended Attributes
- Transparent Encryption
- Multihoming
- Storage Policies
- Memory Storage Support
- Synthetic Load Generator
- Erasure Coding
- Disk Balancer
- Upgrade Domain
- DataNode Admin
- Router Federation
- Provided Storage

##### MapReduce

- Tutorial
- Commands Reference
- Encrypted Shuffle
- Pluggable Shuffle/Sort
- Distributed Cache Deploy
- Support for YARN Shared Cache

##### MapReduce REST APIs

- MR Application Master
- MR History Server

##### YARN

- Architecture
- Commands Reference
- Capacity Scheduler
- Fair Scheduler
- ResourceManager Restart
- ResourceManager HA
- Resource Model
- Node Labels
- Node Attributes
- Web Application Proxy
- Timeline Server
- Timeline Service V.2
- Writing YARN Applications
- YARN Application Security
- NodeManager
- Running Applications in Docker Containers
- Running Applications in runC Containers
- Using CGroups
- Secure Containers
- Reservation System
- Graceful Decommission
- Opportunistic Containers
- YARN Federation
- Shared Cache
- Using GPU
- Using FPGA
- Placement Constraints
- YARN UI2

##### YARN REST APIs

- Introduction
- Resource Manager
- Node Manager
- Timeline Server
- Timeline Service V.2

##### YARN Service

- Overview
- QuickStart
- Concepts
- Yarn Service API
- Service Discovery
- System Services

##### Hadoop Compatible File Systems

- Aliyun OSS
- Amazon S3
- Azure Blob Storage
- Azure Data Lake Storage
- Tencent COS
- Huaweicloud OBS
- VolcanoEngine TOS

##### Auth

- Overview
- Examples
- Configuration
- Building

##### Tools

- Hadoop Streaming
- Hadoop Archives
- Hadoop Archive Logs
- DistCp
- HDFS Federation Balance
- GridMix
- Rumen
- Resource Estimator Service
- Scheduler Load Simulator
- Hadoop Benchmarking
- Dynamometer

##### Reference

- Changelog and Release Notes
- Java API docs
- Unix Shell API
- Metrics

##### Configuration

- core-default.xml
- hdfs-default.xml
- hdfs-rbf-default.xml
- mapred-default.xml
- yarn-default.xml
- kms-default.xml
- httpfs-default.xml
- Deprecated Properties

# Apache Hadoop Changelog

## Release 0.22.1 - Unreleased (as of 2018-09-01)

### INCOMPATIBLE CHANGES:

JIRA | Summary | Priority | Component | Reporter | Contributor
--- | --- | --- | --- | --- | ---
HADOOP-6453 | Hadoop wrapper script shouldn’t ignore an existing JAVA_LIBRARY_PATH | Minor | scripts | Chad Metcalf |

### NEW FEATURES:

JIRA | Summary | Priority | Component | Reporter | Contributor
--- | --- | --- | --- | --- | ---
HADOOP-7119 | add Kerberos HTTP SPNEGO authentication support to Hadoop JT/NN/DN/TT web-consoles | Major | security | Alejandro Abdelnur | Alejandro Abdelnur
HADOOP-7937 | Forward port SequenceFile#syncFs and friends from Hadoop 1.x | Major | io | Eli Collins | Tom White
MAPREDUCE-3837 | Job tracker is not able to recover job in case of crash and after that no user can submit job. | Major | . | Mayank Bansal | Mayank Bansal

### IMPROVEMENTS:

JIRA | Summary | Priority | Component | Reporter | Contributor
--- | --- | --- | --- | --- | ---
HADOOP-6995 | Allow wildcards to be used in ProxyUsers configurations | Minor | security | Todd Lipcon | Todd Lipcon
HDFS-1601 | Pipeline ACKs are sent as lots of tiny TCP packets | Major | datanode | Todd Lipcon | Todd Lipcon
HADOOP-7272 | Remove unnecessary security related info logs | Major | ipc, security | Suresh Srinivas | Suresh Srinivas
HDFS-2246 | Shortcut a local client reads to a Datanodes files directly | Major | . | Sanjay Radia | Jitendra Nath Pandey
HADOOP-7338 | LocalDirAllocator improvements for MR-2178 | Major | . | Todd Lipcon | Benoy Antony
MAPREDUCE-4403 | Adding test case for resubmission of jobs in TestRecoveryManager | Minor | jobtracker | Mayank Bansal | Mayank Bansal
MAPREDUCE-4405 | Adding test case for HierarchicalQueue in TestJobQueueClient | Minor | client | Mayank Bansal | Mayank Bansal
MAPREDUCE-4349 | Distributed Cache gives inconsistent result if cache Archive files get deleted from task tracker | Minor | . | Mayank Bansal | Mayank Bansal
MAPREDUCE-2353 | Make the MR changes to reflect the API changes in SecureIO library | Major | security, task, tasktracker | Devaraj Das | Benoy Antony
MAPREDUCE-1521 | Protection against incorrectly configured reduces | Major | jobtracker | Arun C Murthy | Mahadev konar

### BUG FIXES:

JIRA | Summary | Priority | Component | Reporter | Contributor
--- | --- | --- | --- | --- | ---
MAPREDUCE-2420 | JobTracker should be able to renew delegation token over HTTP | Major | . | Boris Shkolnik | Boris Shkolnik
MAPREDUCE-2452 | Delegation token cancellation shouldn’t hold global JobTracker lock | Major | jobtracker | Devaraj Das | Devaraj Das
HADOOP-7621 | alfredo config should be in a file not readable by users | Critical | security | Alejandro Abdelnur | Aaron T. Myers
HDFS-2698 | BackupNode is downloading image from NameNode for every checkpoint | Major | namenode | Konstantin Shvachko | Konstantin Shvachko
HDFS-1910 | when dfs.name.dir and dfs.name.edits.dir are same fsimage will be saved twice every time | Minor | namenode | Gokul |
MAPREDUCE-3593 | MAPREDUCE Impersonation is not working in 22 | Major | job submission | Mayank Bansal | Mayank Bansal
MAPREDUCE-3725 | Hadoop 22 hadoop job -list returns user name as NULL | Major | client | Mayank Bansal | Mayank Bansal
HDFS-2718 | Optimize OP_ADD in edits loading | Major | namenode | Konstantin Shvachko | Konstantin Shvachko
HDFS-2877 | If locking of a storage dir fails, it will remove the other NN’s lock file on exit | Major | namenode | Todd Lipcon | Todd Lipcon
HADOOP-7680 | TestHardLink fails on Mac OS X, when gnu stat is in path | Major | . | Milind Bhandarkar | Milind Bhandarkar
HDFS-2991 | failure to load edits: ClassCastException | Blocker | namenode | Todd Lipcon | Todd Lipcon
MAPREDUCE-4164 | Hadoop 22 Exception thrown after task completion causes its reexecution | Major | tasktracker | Mayank Bansal | Mayank Bansal
MAPREDUCE-3863 | 0.22 branch mvn deploy is not publishing hadoop-streaming JAR | Critical | build | Alejandro Abdelnur | Benoy Antony
HDFS-3368 | Missing blocks due to bad DataNodes coming up and down. | Major | namenode | Konstantin Shvachko | Konstantin Shvachko
MAPREDUCE-2178 | Race condition in LinuxTaskController permissions handling | Major | security, task-controller | Todd Lipcon | Benoy Antony
MAPREDUCE-4314 | Synchronization in JvmManager for 0.22 branch | Major | tasktracker | Konstantin Shvachko | Benoy Antony
MAPREDUCE-2377 | task-controller fails to parse configuration if it doesn’t end in \n | Major | task-controller | Todd Lipcon | Benoy Antony
MAPREDUCE-4318 | TestRecoveryManager should not use raw and deprecated configuration parameters. | Major | test | Konstantin Shvachko | Benoy Antony
MAPREDUCE-4360 | Capacity Scheduler Hierarchical leaf queue does not honor the max capacity of container queue | Major | . | Mayank Bansal | Mayank Bansal
HDFS-1584 | Need to check TGT and renew if needed when fetching delegation tokens using HFTP | Major | security | Kan Zhang | Benoy Antony
HDFS-3402 | Fix hdfs scripts for secure datanodes | Minor | scripts, security | Benoy Antony | Benoy Antony
HADOOP-7115 | Add a cache for getpwuid_r and getpwgid_r calls | Major | . | Arun C Murthy | Alejandro Abdelnur
MAPREDUCE-4404 | Adding Test case for TestMRJobClient to verify the user name | Minor | client | Mayank Bansal | Mayank Bansal
MAPREDUCE-5706 | toBeDeleted parent directories aren’t being cleaned up | Major | security | Robert Kanter | Robert Kanter

### SUB-TASKS:

JIRA | Summary | Priority | Component | Reporter | Contributor
--- | --- | --- | --- | --- | ---
HDFS-2886 | CreateEditLogs should generate a realistic edit log. | Major | namenode | Konstantin Shvachko | Konstantin Shvachko
HADOOP-8381 | Substitute _HOST with hostname for HTTP principals | Minor | security | Benoy Antony | Benoy Antony
HADOOP-8383 | TestKerberosAuthenticator fails | Minor | security | Benoy Antony | Benoy Antony

### OTHER:

JIRA | Summary | Priority | Component | Reporter | Contributor
--- | --- | --- | --- | --- | ---
MAPREDUCE-4240 | Revert MAPREDUCE-2767 | Minor | security | Benoy Antony | Benoy Antony
MAPREDUCE-4243 | Modify mapreduce build to include task-controller | Minor | build | Benoy Antony | Benoy Antony
MAPREDUCE-4244 | Fix an issue related to do with setting of correct groups for tasks | Minor | security | Benoy Antony | Benoy Antony
MAPREDUCE-4246 | Failure in deleting user directories in Secure hadoop | Major | security | Benoy Antony | Benoy Antony
MAPREDUCE-4247 | TestTaskTrackerLocalization fails | Minor | security | Benoy Antony | Benoy Antony
MAPREDUCE-4248 | TestRecoveryManager fails | Minor | security | Benoy Antony | Benoy Antony
MAPREDUCE-4249 | Fix failures in streaming test TestFileArgs | Minor | security | Benoy Antony | Benoy Antony
HADOOP-8357 | Restore security in Hadoop 0.22 branch | Major | security | Konstantin Shvachko | Benoy Antony