Apache Hadoop 3.5.0 – Apache Hadoop Changelog

##### General

- Overview
- Single Node Setup
- Cluster Setup
- Commands Reference
- FileSystem Shell
- Compatibility Specification
- Downstream Developer's Guide
- Admin Compatibility Guide
- Interface Classification
- FileSystem Specification

##### Common

- CLI Mini Cluster
- Fair Call Queue
- Native Libraries
- Proxy User
- Rack Awareness
- Secure Mode
- Service Level Authorization
- HTTP Authentication
- Credential Provider API
- Hadoop KMS
- Tracing
- Unix Shell Guide
- Registry
- Async Profiler

##### HDFS

- Architecture
- User Guide
- Commands Reference
- NameNode HA With QJM
- NameNode HA With NFS
- Observer NameNode
- Federation
- ViewFs
- ViewFsOverloadScheme
- Snapshots
- Edits Viewer
- Image Viewer
- Permissions and HDFS
- Quotas and HDFS
- libhdfs (C API)
- WebHDFS (REST API)
- HttpFS
- Short Circuit Local Reads
- Centralized Cache Management
- NFS Gateway
- Rolling Upgrade
- Extended Attributes
- Transparent Encryption
- Multihoming
- Storage Policies
- Memory Storage Support
- Synthetic Load Generator
- Erasure Coding
- Disk Balancer
- Upgrade Domain
- DataNode Admin
- Router Federation
- Provided Storage

##### MapReduce

- Tutorial
- Commands Reference
- Encrypted Shuffle
- Pluggable Shuffle/Sort
- Distributed Cache Deploy
- Support for YARN Shared Cache

##### MapReduce REST APIs

- MR Application Master
- MR History Server

##### YARN

- Architecture
- Commands Reference
- Capacity Scheduler
- Fair Scheduler
- ResourceManager Restart
- ResourceManager HA
- Resource Model
- Node Labels
- Node Attributes
- Web Application Proxy
- Timeline Server
- Timeline Service V.2
- Writing YARN Applications
- YARN Application Security
- NodeManager
- Running Applications in Docker Containers
- Running Applications in runC Containers
- Using CGroups
- Secure Containers
- Reservation System
- Graceful Decommission
- Opportunistic Containers
- YARN Federation
- Shared Cache
- Using GPU
- Using FPGA
- Placement Constraints
- YARN UI2

##### YARN REST APIs

- Introduction
- Resource Manager
- Node Manager
- Timeline Server
- Timeline Service V.2

##### YARN Service

- Overview
- QuickStart
- Concepts
- Yarn Service API
- Service Discovery
- System Services

##### Hadoop Compatible File Systems

- Aliyun OSS
- Amazon S3
- Azure Blob Storage
- Azure Data Lake Storage
- Tencent COS
- Huaweicloud OBS
- VolcanoEngine TOS

##### Auth

- Overview
- Examples
- Configuration
- Building

##### Tools

- Hadoop Streaming
- Hadoop Archives
- Hadoop Archive Logs
- DistCp
- HDFS Federation Balance
- GridMix
- Rumen
- Resource Estimator Service
- Scheduler Load Simulator
- Hadoop Benchmarking
- Dynamometer

##### Reference

- Changelog and Release Notes
- Java API docs
- Unix Shell API
- Metrics

##### Configuration

- core-default.xml
- hdfs-default.xml
- hdfs-rbf-default.xml
- mapred-default.xml
- yarn-default.xml
- kms-default.xml
- httpfs-default.xml
- Deprecated Properties

# Apache Hadoop Changelog

## Release 2.4.1 - 2014-06-30

### IMPROVEMENTS:

JIRA | Summary | Priority | Component | Reporter | Contributor
--- | --- | --- | --- | --- | ---
HDFS-4052 | BlockManager#invalidateWork should print logs outside the lock | Minor | . | Jing Zhao | Jing Zhao
HADOOP-10466 | Lower the log level in UserGroupInformation | Minor | security | Nicolas Liochon | Nicolas Liochon
YARN-1892 | Excessive logging in RM | Minor | scheduler | Siddharth Seth | Jian He

### BUG FIXES:

JIRA | Summary | Priority | Component | Reporter | Contributor
--- | --- | --- | --- | --- | ---
YARN-1883 | TestRMAdminService fails due to inconsistent entries in UserGroups | Major | . | Mit Desai | Mit Desai
HADOOP-10455 | When there is an exception, ipc.Server should first check whether it is an terse exception | Major | ipc | Tsz Wo Nicholas Sze | Tsz Wo Nicholas Sze
MAPREDUCE-5818 | hsadmin cmd is missing in mapred.cmd | Major | . | Jian He | Jian He
HADOOP-10456 | Bug in Configuration.java exposed by Spark (ConcurrentModificationException) | Major | conf | Nishkam Ravi | Nishkam Ravi
YARN-1837 | TestMoveApplication.testMoveRejectedByScheduler randomly fails | Major | . | Tsuyoshi Ozawa | Hong Zhiguo
HDFS-6197 | Rolling upgrade rollback on Windows can fail attempting to rename edit log segment files to a destination that already exists. | Minor | namenode | Chris Nauroth | Chris Nauroth
HDFS-6198 | DataNode rolling upgrade does not correctly identify current block pool directory and replace with trash on Windows. | Major | datanode | Chris Nauroth | Chris Nauroth
YARN-1908 | Distributed shell with custom script has permission error. | Major | applications/distributed-shell | Tassapol Athiapinya | Vinod Kumar Vavilapalli
YARN-1907 | TestRMApplicationHistoryWriter#testRMWritingMassiveHistory runs slow and intermittently fails | Major | . | Mit Desai | Mit Desai
HDFS-6215 | Wrong error message for upgrade | Minor | . | Kihwal Lee | Kihwal Lee
HADOOP-10473 | TestCallQueueManager is still flaky | Minor | test | Tsz Wo Nicholas Sze | Tsz Wo Nicholas Sze
HDFS-6206 | DFSUtil.substituteForWildcardAddress may throw NPE | Major | . | Tsz Wo Nicholas Sze | Tsz Wo Nicholas Sze
HDFS-6204 | TestRBWBlockInvalidation may fail | Minor | test | Tsz Wo Nicholas Sze | Tsz Wo Nicholas Sze
HDFS-6209 | Fix flaky test TestValidateConfigurationSettings.testThatDifferentRPCandHttpPortsAreOK | Minor | test | Arpit Agarwal | Arpit Agarwal
HDFS-6208 | DataNode caching can leak file descriptors. | Major | datanode | Chris Nauroth | Chris Nauroth
YARN-1910 | TestAMRMTokens fails on windows | Major | . | Xuan Gong | Xuan Gong
MAPREDUCE-5824 | TestPipesNonJavaInputFormat.testFormat fails in windows | Major | . | Xuan Gong | Xuan Gong
YARN-1920 | TestFileSystemApplicationHistoryStore.testMissingApplicationAttemptHistoryData fails in windows | Major | . | Vinod Kumar Vavilapalli | Vinod Kumar Vavilapalli
YARN-1914 | Test TestFSDownload.testDownloadPublicWithStatCache fails on Windows | Major | . | Varun Vasudev | Varun Vasudev
YARN-1903 | Killing Container on NEW and LOCALIZING will result in exitCode and diagnostics not set | Major | . | Zhijie Shen | Zhijie Shen
YARN-1924 | STATE_STORE_OP_FAILED happens when ZKRMStateStore tries to update app(attempt) before storing it | Critical | . | Arpit Gupta | Jian He
MAPREDUCE-5815 | Fix NPE in TestMRAppMaster | Blocker | client, mrv2 | Gera Shegalov | Akira Ajisaka
YARN-1926 | DistributedShell unit tests fail on Windows | Major | . | Varun Vasudev | Varun Vasudev
HDFS-6231 | DFSClient hangs infinitely if using hedged reads and all eligible datanodes die. | Major | hdfs-client | Chris Nauroth | Chris Nauroth
MAPREDUCE-5826 | TestHistoryServerFileSystemStateStoreService.testTokenStore fails in windows | Major | . | Varun Vasudev | Varun Vasudev
HADOOP-10490 | TestMapFile and TestBloomMapFile leak file descriptors. | Minor | test | Chris Nauroth | Chris Nauroth
HDFS-6234 | TestDatanodeConfig#testMemlockLimit fails on Windows due to invalid file path. | Trivial | datanode, test | Chris Nauroth | Chris Nauroth
HADOOP-8826 | Docs still refer to 0.20.205 as stable line | Minor | . | Robert Joseph Evans | Mit Desai
HDFS-6235 | TestFileJournalManager can fail on Windows due to file locking if tests run out of order. | Trivial | namenode, test | Chris Nauroth | Chris Nauroth
HDFS-6229 | Race condition in failover can cause RetryCache fail to work | Major | ha | Jing Zhao | Jing Zhao
HDFS-6232 | OfflineEditsViewer throws a NPE on edits containing ACL modifications | Major | tools | Stephen Chu | Akira Ajisaka
HDFS-6236 | ImageServlet should use Time#monotonicNow to measure latency. | Minor | namenode | Chris Nauroth | Chris Nauroth
MAPREDUCE-5828 | TestMapReduceJobControl fails on JDK 7 + Windows | Major | . | Vinod Kumar Vavilapalli | Vinod Kumar Vavilapalli
YARN-1933 | TestAMRestart and TestNodeHealthService failing sometimes on Windows | Major | . | Jian He | Jian He
YARN-1928 | TestAMRMRPCNodeUpdates fails ocassionally | Major | . | Zhijie Shen | Zhijie Shen
YARN-1934 | Potential NPE in ZKRMStateStore caused by handling Disconnected event from ZK. | Blocker | resourcemanager | Rohith Sharma K S | Karthik Kambatla
MAPREDUCE-3191 | docs for map output compression incorrectly reference SequenceFile | Trivial | . | Todd Lipcon | Chen He
YARN-1931 | Private API change in YARN-1824 in 2.4 broke compatibility with previous releases | Blocker | applications | Thomas Graves | Sandy Ryza
YARN-1932 | Javascript injection on the job status page | Blocker | . | Mit Desai | Mit Desai
HDFS-6245 | datanode fails to start with a bad disk even when failed volumes is set | Major | . | Arpit Gupta | Arpit Agarwal
HADOOP-10522 | JniBasedUnixGroupMapping mishandles errors | Critical | . | Kihwal Lee | Kihwal Lee
MAPREDUCE-5827 | TestSpeculativeExecutionWithMRApp fails | Major | . | Zhijie Shen | Zhijie Shen
MAPREDUCE-5832 | Few tests in TestJobClient fail on Windows | Major | . | Jian He | Vinod Kumar Vavilapalli
HADOOP-10527 | Fix incorrect return code and allow more retries on EINTR | Major | . | Kihwal Lee | Kihwal Lee
MAPREDUCE-5841 | uber job doesn’t terminate on getting mapred job kill | Major | mrv2 | Sangjin Lee | Sangjin Lee
YARN-1975 | Used resources shows escaped html in CapacityScheduler and FairScheduler page | Major | resourcemanager | Nathan Roberts | Mit Desai
MAPREDUCE-5835 | Killing Task might cause the job to go to ERROR state | Critical | . | Ming Ma | Ming Ma
MAPREDUCE-5714 | TestMRAppComponentDependencies causes surefire to exit without saying proper goodbye | Major | test | Jinghui Wang | Jinghui Wang
HDFS-2882 | DN continues to start up, even if block pool fails to initialize | Major | datanode | Todd Lipcon | Vinayakumar B
YARN-1929 | DeadLock in RM when automatic failover is enabled. | Blocker | resourcemanager | Rohith Sharma K S | Karthik Kambatla
YARN-1201 | TestAMAuthorization fails with local hostname cannot be resolved | Minor | resourcemanager | Nemon Lou | Wangda Tan
HADOOP-10562 | Namenode exits on exception without printing stack trace in AbstractDelegationTokenSecretManager | Critical | . | Suresh Srinivas | Suresh Srinivas
HDFS-6340 | DN can’t finalize upgrade | Blocker | datanode | Rahul Singhal | Rahul Singhal
HDFS-6329 | WebHdfs does not work if HA is enabled on NN but logical URI is not configured. | Blocker | . | Kihwal Lee | Kihwal Lee
HDFS-6313 | WebHdfs may use the wrong NN when configured for multiple HA NNs | Blocker | webhdfs | Daryn Sharp | Kihwal Lee
YARN-1976 | Tracking url missing http protocol for FAILED application | Major | . | Yesha Vora | Junping Du
YARN-2016 | Yarn getApplicationRequest start time range is not honored | Major | resourcemanager | Venkat Ranganathan | Junping Du
YARN-1986 | In Fifo Scheduler, node heartbeat in between creating app and attempt causes NPE | Critical | . | Jon Bringhurst | Hong Zhiguo
MAPREDUCE-5821 | IFile merge allocates new byte array for every value | Major | performance, task | Todd Lipcon | Todd Lipcon
HDFS-6326 | WebHdfs ACL compatibility is broken | Blocker | webhdfs | Daryn Sharp | Chris Nauroth
HDFS-6361 | TestIdUserGroup.testUserUpdateSetting failed due to out of range nfsnobody Id | Major | nfs | Yongjun Zhang | Yongjun Zhang
HDFS-6362 | InvalidateBlocks is inconsistent in usage of DatanodeUuid and StorageID | Blocker | namenode | Arpit Agarwal | Arpit Agarwal
HDFS-6402 | Suppress findbugs warning for failure to override equals and hashCode in FsAclPermission. | Trivial | namenode | Chris Nauroth | Chris Nauroth
YARN-2066 | Wrong field is referenced in GetApplicationsRequestPBImpl#mergeLocalToBuilder() | Minor | . | Ted Yu | Hong Zhiguo
HDFS-6325 | Append should fail if the last block has insufficient number of replicas | Major | namenode | Konstantin Shvachko | Keith Pak
HDFS-6397 | NN shows inconsistent value in deadnode count | Critical | . | Mohammad Kamrul Islam | Mohammad Kamrul Islam
HADOOP-10612 | NFS failed to refresh the user group id mapping table | Major | nfs | Brandon Li | Brandon Li
YARN-2081 | TestDistributedShell fails after YARN-1962 | Minor | applications/distributed-shell | Hong Zhiguo | Hong Zhiguo
HDFS-6411 | nfs-hdfs-gateway mount raises I/O error and hangs when a unauthorized user attempts to access it | Major | nfs | Zhongyi Xie | Brandon Li
MAPREDUCE-5830 | HostUtil.getTaskLogUrl is not backwards binary compatible with 2.3 | Blocker | . | Jason Lowe | Akira Ajisaka
HADOOP-11273 | TestMiniKdc failure: login options not compatible with IBM JDK | Major | test | Gao Zhong Liang | Gao Zhong Liang

### TESTS:

JIRA | Summary | Priority | Component | Reporter | Contributor
--- | --- | --- | --- | --- | ---
HDFS-6189 | Multiple HDFS tests fail on Windows attempting to use a test root path containing a colon. | Major | test | Chris Nauroth | Chris Nauroth
YARN-1905 | TestProcfsBasedProcessTree must only run on Linux. | Trivial | nodemanager | Chris Nauroth | Chris Nauroth
YARN-1750 | TestNodeStatusUpdater#testNMRegistration is incorrect in test case | Major | nodemanager | Ming Ma | Wangda Tan
YARN-1947 | TestRMDelegationTokens#testRMDTMasterKeyStateOnRollingMasterKey is failing intermittently | Major | . | Jian He | Jian He
YARN-1281 | TestZKRMStateStoreZKClientConnections fails intermittently | Major | resourcemanager | Karthik Kambatla | Tsuyoshi Ozawa
MAPREDUCE-5833 | TestRMContainerAllocator fails ocassionally | Major | . | Zhijie Shen | Zhijie Shen
MAPREDUCE-5843 | TestMRKeyValueTextInputFormat failing on Windows | Major | . | Varun Vasudev | Varun Vasudev

### SUB-TASKS:

JIRA | Summary | Priority | Component | Reporter | Contributor
--- | --- | --- | --- | --- | ---
YARN-1696 | Document RM HA | Blocker | resourcemanager | Karthik Kambatla | Tsuyoshi Ozawa
YARN-1701 | Improve default paths of timeline store and generic history store | Major | . | Gera Shegalov | Tsuyoshi Ozawa
YARN-1962 | Timeline server is enabled by default | Major | . | Mohammad Kamrul Islam | Mohammad Kamrul Islam
YARN-1861 | Both RM stuck in standby mode when automatic failover is enabled | Blocker | resourcemanager | Arpit Gupta | Karthik Kambatla
YARN-1957 | ProportionalCapacitPreemptionPolicy handling of corner cases… | Major | resourcemanager | Carlo Curino | Carlo Curino
YARN-2053 | Slider AM fails to restart: NPE in RegisterApplicationMasterResponseProto$Builder.addAllNmTokensFromPreviousAttempts | Major | resourcemanager | Sumit Mohanty | Wangda Tan
YARN-1898 | Standby RM’s conf, stacks, logLevel, metrics, jmx and logs links are redirecting to Active RM | Major | resourcemanager | Yesha Vora | Xuan Gong