Apache Hadoop 3.5.0 – Apache Hadoop Changelog

##### General

- Overview
- Single Node Setup
- Cluster Setup
- Commands Reference
- FileSystem Shell
- Compatibility Specification
- Downstream Developer's Guide
- Admin Compatibility Guide
- Interface Classification
- FileSystem Specification

##### Common

- CLI Mini Cluster
- Fair Call Queue
- Native Libraries
- Proxy User
- Rack Awareness
- Secure Mode
- Service Level Authorization
- HTTP Authentication
- Credential Provider API
- Hadoop KMS
- Tracing
- Unix Shell Guide
- Registry
- Async Profiler

##### HDFS

- Architecture
- User Guide
- Commands Reference
- NameNode HA With QJM
- NameNode HA With NFS
- Observer NameNode
- Federation
- ViewFs
- ViewFsOverloadScheme
- Snapshots
- Edits Viewer
- Image Viewer
- Permissions and HDFS
- Quotas and HDFS
- libhdfs (C API)
- WebHDFS (REST API)
- HttpFS
- Short Circuit Local Reads
- Centralized Cache Management
- NFS Gateway
- Rolling Upgrade
- Extended Attributes
- Transparent Encryption
- Multihoming
- Storage Policies
- Memory Storage Support
- Synthetic Load Generator
- Erasure Coding
- Disk Balancer
- Upgrade Domain
- DataNode Admin
- Router Federation
- Provided Storage

##### MapReduce

- Tutorial
- Commands Reference
- Encrypted Shuffle
- Pluggable Shuffle/Sort
- Distributed Cache Deploy
- Support for YARN Shared Cache

##### MapReduce REST APIs

- MR Application Master
- MR History Server

##### YARN

- Architecture
- Commands Reference
- Capacity Scheduler
- Fair Scheduler
- ResourceManager Restart
- ResourceManager HA
- Resource Model
- Node Labels
- Node Attributes
- Web Application Proxy
- Timeline Server
- Timeline Service V.2
- Writing YARN Applications
- YARN Application Security
- NodeManager
- Running Applications in Docker Containers
- Running Applications in runC Containers
- Using CGroups
- Secure Containers
- Reservation System
- Graceful Decommission
- Opportunistic Containers
- YARN Federation
- Shared Cache
- Using GPU
- Using FPGA
- Placement Constraints
- YARN UI2

##### YARN REST APIs

- Introduction
- Resource Manager
- Node Manager
- Timeline Server
- Timeline Service V.2

##### YARN Service

- Overview
- QuickStart
- Concepts
- Yarn Service API
- Service Discovery
- System Services

##### Hadoop Compatible File Systems

- Aliyun OSS
- Amazon S3
- Azure Blob Storage
- Azure Data Lake Storage
- Tencent COS
- Huaweicloud OBS
- VolcanoEngine TOS

##### Auth

- Overview
- Examples
- Configuration
- Building

##### Tools

- Hadoop Streaming
- Hadoop Archives
- Hadoop Archive Logs
- DistCp
- HDFS Federation Balance
- GridMix
- Rumen
- Resource Estimator Service
- Scheduler Load Simulator
- Hadoop Benchmarking
- Dynamometer

##### Reference

- Changelog and Release Notes
- Java API docs
- Unix Shell API
- Metrics

##### Configuration

- core-default.xml
- hdfs-default.xml
- hdfs-rbf-default.xml
- mapred-default.xml
- yarn-default.xml
- kms-default.xml
- httpfs-default.xml
- Deprecated Properties

# Apache Hadoop Changelog

## Release 0.21.1 - Unreleased (as of 2018-09-01)

### INCOMPATIBLE CHANGES:

JIRA | Summary | Priority | Component | Reporter | Contributor
--- | --- | --- | --- | --- | ---
MAPREDUCE-1905 | Context.setStatus() and progress() api are ignored | Blocker | task | Amareshwari Sriramadasu | Amareshwari Sriramadasu

### NEW FEATURES:

JIRA | Summary | Priority | Component | Reporter | Contributor
--- | --- | --- | --- | --- | ---
MAPREDUCE-2040 | Forrest Documentation for Dynamic Priority Scheduler | Minor | contrib/dynamic-scheduler | Thomas Sandholm | Thomas Sandholm

### IMPROVEMENTS:

JIRA | Summary | Priority | Component | Reporter | Contributor
--- | --- | --- | --- | --- | ---
MAPREDUCE-1501 | FileInputFormat to support multi-level/recursive directory listing | Major | . | Zheng Shao | Zheng Shao
HADOOP-6786 | test-patch needs to verify Herriot integrity | Major | build | Konstantin Boudnik | Konstantin Boudnik
HDFS-1343 | Instrumented build should be concentrated in one build area | Minor | build | Konstantin Boudnik | Konstantin Boudnik
MAPREDUCE-2140 | Re-generate fair scheduler design doc PDF | Trivial | . | Matei Zaharia | Matei Zaharia
HADOOP-7177 | CodecPool should report which compressor it is using | Trivial | native | Allen Wittenauer | Allen Wittenauer
HDFS-1596 | Move secondary namenode checkpoint configs from core-default.xml to hdfs-default.xml | Major | documentation, namenode | Patrick Angeles | Harsh J
HADOOP-7117 | Move secondary namenode checkpoint configs from core-default.xml to hdfs-default.xml | Major | conf | Patrick Angeles | Harsh J
HADOOP-7193 | Help message is wrong for touchz command. | Minor | fs | Uma Maheswara Rao G | Uma Maheswara Rao G

### BUG FIXES:

JIRA | Summary | Priority | Component | Reporter | Contributor
--- | --- | --- | --- | --- | ---
HADOOP-6925 | BZip2Codec incorrectly implements read() | Critical | io | Todd Lipcon | Todd Lipcon
HADOOP-6833 | IPC leaks call parameters when exceptions thrown | Blocker | . | Todd Lipcon | Todd Lipcon
MAPREDUCE-1897 | trunk build broken on compile-mapred-test | Major | test | Greg Roelofs | Konstantin Boudnik
MAPREDUCE-2032 | TestJobOutputCommitter fails in ant test run | Major | task | Amareshwari Sriramadasu | Dick King
HDFS-1363 | startFileInternal should return the last block of the file opened for append as an under-construction block | Major | . | Konstantin Shvachko | Konstantin Shvachko
HDFS-1411 | The startup command of the Backup Node is “bin/hdfs namenode -backup” | Minor | documentation | Ching-Shen Chen | Ching-Shen Chen
MAPREDUCE-1280 | Eclipse Plugin does not work with Eclipse Ganymede (3.4) | Major | . | Aaron Kimball | Alex Kozlov
HDFS-1413 | Broken links to HDFS Wiki in hdfs site and documentation. | Major | documentation | Konstantin Shvachko | Konstantin Shvachko
MAPREDUCE-1984 | herriot TestCluster fails because exclusion is not there | Major | . | Balaji Rajagopalan | Balaji Rajagopalan
HADOOP-6971 | Clover build doesn’t generate per-test coverage | Major | build, test | Konstantin Boudnik | Konstantin Boudnik
HDFS-1420 | Clover build doesn’t generate per-test coverage | Major | build, test | Konstantin Boudnik | Konstantin Boudnik
MAPREDUCE-2090 | Clover build doesn’t generate per-test coverage | Major | build, test | Konstantin Boudnik | Konstantin Boudnik
HADOOP-6969 | CHANGES.txt does not reflect the release of version 0.21.0. | Major | . | Konstantin Shvachko | Tom White
HDFS-1416 | CHANGES.txt does not reflect the release version 0.21.0. | Major | . | Konstantin Shvachko | Tom White
MAPREDUCE-2086 | CHANGES.txt does not reflect the release of version 0.21.0. | Major | . | Konstantin Shvachko | Tom White
HADOOP-6993 | Broken link on cluster setup page of docs | Major | documentation | Aaron T. Myers | Eli Collins
HDFS-1444 | Test related code of build.xml is error-prone and needs to be re-aligned. | Minor | build | Konstantin Boudnik | Konstantin Boudnik
HDFS-1452 | ant compile-contrib is broken | Major | contrib/hdfsproxy | Jakob Homan | Konstantin Boudnik
MAPREDUCE-2134 | ant binary-system is broken in mapreduce project. | Major | build | Vinay Kumar Thota | Konstantin Boudnik
HDFS-1474 | ant binary-system is broken | Major | build | Konstantin Boudnik | Konstantin Boudnik
HADOOP-6954 | Sources JARs are not correctly published to the Maven repository | Major | build | Tom White | Tom White
HADOOP-7052 | misspelling of threshold in conf/log4j.properties | Major | conf | Jingguo Yao | Jingguo Yao
HADOOP-7053 | wrong FSNamesystem Audit logging setting in conf/log4j.properties | Minor | conf | Jingguo Yao | Jingguo Yao
HDFS-1377 | Quota bug for partial blocks allows quotas to be violated | Blocker | namenode | Eli Collins | Eli Collins
HDFS-1206 | TestFiHFlush fails intermittently | Major | test | Tsz Wo Nicholas Sze | Konstantin Boudnik
MAPREDUCE-2223 | TestMRCLI might fail on Ubuntu with default /etc/hosts | Minor | test | Konstantin Boudnik | Konstantin Boudnik
HDFS-1548 | Fault-injection tests are executed multiple times if invoked with run-test-hdfs-fault-inject target | Major | build, test | Konstantin Boudnik | Konstantin Boudnik
HDFS-1552 | Remove java5 dependencies from build | Major | build | Konstantin Boudnik | Konstantin Boudnik
MAPREDUCE-2228 | Remove java5 dependencies from build | Major | build | Konstantin Boudnik | Konstantin Boudnik
MAPREDUCE-1929 | Allow artifacts to be published to the staging Apache Nexus Maven Repository | Blocker | build | Tom White | Tom White
HDFS-996 | JUnit tests should never depend on anything in conf | Blocker | test | Konstantin Boudnik | Konstantin Boudnik
HADOOP-7120 | 200 new Findbugs warnings | Major | test | Tsz Wo Nicholas Sze | Tsz Wo Nicholas Sze
HDFS-1598 | ListPathsServlet excludes .*.crc files | Major | namenode | Tsz Wo Nicholas Sze | Tsz Wo Nicholas Sze
HDFS-1612 | HDFS Design Documentation is outdated | Minor | documentation | Joe Crobak | Joe Crobak
HADOOP-7162 | FsShell: call srcFs.listStatus(src) twice | Minor | fs | Alexey Diomin | Alexey Diomin
HDFS-1665 | Balancer sleeps inadequately | Minor | balancer & mover | Tsz Wo Nicholas Sze | Tsz Wo Nicholas Sze
HDFS-1728 | SecondaryNameNode.checkpointSize is in byte but not MB. | Minor | namenode | Tsz Wo Nicholas Sze | Tsz Wo Nicholas Sze
HDFS-1750 | fs -ls hftp://file not working | Major | . | Tsz Wo Nicholas Sze | Tsz Wo Nicholas Sze
HADOOP-7174 | null is displayed in the console,if the src path is invalid while doing copyToLocal operation from commandLine | Minor | fs | Uma Maheswara Rao G | Uma Maheswara Rao G
HADOOP-7194 | Potential Resource leak in IOUtils.java | Major | io | Devaraj K | Devaraj K
HADOOP-7019 | Refactor build targets to enable faster cross project dev cycles. | Major | build | Owen O’Malley | Luke Lu
HDFS-1189 | Quota counts missed between clear quota and set quota | Major | namenode | Kang Xiao | John George
MAPREDUCE-2317 | HadoopArchives throwing NullPointerException while creating hadoop archives (.har files) | Minor | harchive | Devaraj K | Devaraj K
HADOOP-7183 | WritableComparator.get should not cache comparator objects | Blocker | . | Todd Lipcon | Tom White
HADOOP-7215 | RPC clients must connect over a network interface corresponding to the host name in the client’s kerberos principal key | Blocker | security | Suresh Srinivas | Suresh Srinivas

### TESTS:

JIRA | Summary | Priority | Component | Reporter | Contributor
--- | --- | --- | --- | --- | ---
HADOOP-6934 | test for ByteWritable comparator | Major | record | Johannes Zillmann | Johannes Zillmann

### SUB-TASKS:

JIRA | Summary | Priority | Component | Reporter | Contributor
--- | --- | --- | --- | --- | ---
HADOOP-7072 | Remove java5 dependencies from build | Major | build | Konstantin Boudnik | Konstantin Boudnik

### OTHER:

JIRA | Summary | Priority | Component | Reporter | Contributor
--- | --- | --- | --- | --- | ---
HADOOP-6944 | [Herriot] Implement a functionality for getting proxy users definitions like groups and hosts. | Major | test | Vinay Kumar Thota | Vinay Kumar Thota
MAPREDUCE-1809 | Ant build changes for Streaming system tests in contrib projects. | Major | build | Vinay Kumar Thota | Vinay Kumar Thota