Apache Hadoop 3.5.0 – Apache Hadoop Changelog

##### General

- Overview
- Single Node Setup
- Cluster Setup
- Commands Reference
- FileSystem Shell
- Compatibility Specification
- Downstream Developer's Guide
- Admin Compatibility Guide
- Interface Classification
- FileSystem Specification

##### Common

- CLI Mini Cluster
- Fair Call Queue
- Native Libraries
- Proxy User
- Rack Awareness
- Secure Mode
- Service Level Authorization
- HTTP Authentication
- Credential Provider API
- Hadoop KMS
- Tracing
- Unix Shell Guide
- Registry
- Async Profiler

##### HDFS

- Architecture
- User Guide
- Commands Reference
- NameNode HA With QJM
- NameNode HA With NFS
- Observer NameNode
- Federation
- ViewFs
- ViewFsOverloadScheme
- Snapshots
- Edits Viewer
- Image Viewer
- Permissions and HDFS
- Quotas and HDFS
- libhdfs (C API)
- WebHDFS (REST API)
- HttpFS
- Short Circuit Local Reads
- Centralized Cache Management
- NFS Gateway
- Rolling Upgrade
- Extended Attributes
- Transparent Encryption
- Multihoming
- Storage Policies
- Memory Storage Support
- Synthetic Load Generator
- Erasure Coding
- Disk Balancer
- Upgrade Domain
- DataNode Admin
- Router Federation
- Provided Storage

##### MapReduce

- Tutorial
- Commands Reference
- Encrypted Shuffle
- Pluggable Shuffle/Sort
- Distributed Cache Deploy
- Support for YARN Shared Cache

##### MapReduce REST APIs

- MR Application Master
- MR History Server

##### YARN

- Architecture
- Commands Reference
- Capacity Scheduler
- Fair Scheduler
- ResourceManager Restart
- ResourceManager HA
- Resource Model
- Node Labels
- Node Attributes
- Web Application Proxy
- Timeline Server
- Timeline Service V.2
- Writing YARN Applications
- YARN Application Security
- NodeManager
- Running Applications in Docker Containers
- Running Applications in runC Containers
- Using CGroups
- Secure Containers
- Reservation System
- Graceful Decommission
- Opportunistic Containers
- YARN Federation
- Shared Cache
- Using GPU
- Using FPGA
- Placement Constraints
- YARN UI2

##### YARN REST APIs

- Introduction
- Resource Manager
- Node Manager
- Timeline Server
- Timeline Service V.2

##### YARN Service

- Overview
- QuickStart
- Concepts
- Yarn Service API
- Service Discovery
- System Services

##### Hadoop Compatible File Systems

- Aliyun OSS
- Amazon S3
- Azure Blob Storage
- Azure Data Lake Storage
- Tencent COS
- Huaweicloud OBS
- VolcanoEngine TOS

##### Auth

- Overview
- Examples
- Configuration
- Building

##### Tools

- Hadoop Streaming
- Hadoop Archives
- Hadoop Archive Logs
- DistCp
- HDFS Federation Balance
- GridMix
- Rumen
- Resource Estimator Service
- Scheduler Load Simulator
- Hadoop Benchmarking
- Dynamometer

##### Reference

- Changelog and Release Notes
- Java API docs
- Unix Shell API
- Metrics

##### Configuration

- core-default.xml
- hdfs-default.xml
- hdfs-rbf-default.xml
- mapred-default.xml
- yarn-default.xml
- kms-default.xml
- httpfs-default.xml
- Deprecated Properties

# Apache Hadoop Changelog

## Release 2.7.1 - 2015-07-06

### INCOMPATIBLE CHANGES:

JIRA | Summary | Priority | Component | Reporter | Contributor
--- | --- | --- | --- | --- | ---
HDFS-8226 | Non-HA rollback compatibility broken | Blocker | . | J.Andreina | J.Andreina

### IMPROVEMENTS:

JIRA | Summary | Priority | Component | Reporter | Contributor
--- | --- | --- | --- | --- | ---
HDFS-7546 | Document, and set an accepting default for dfs.namenode.kerberos.principal.pattern | Minor | security | Harsh J | Harsh J
YARN-3469 | ZKRMStateStore: Avoid setting watches that are not required | Minor | . | Jun Gong | Jun Gong
HDFS-8204 | Mover/Balancer should not schedule two replicas to the same DN | Minor | balancer & mover | Walter Su | Walter Su
HDFS-7770 | Need document for storage type label of data node storage locations under dfs.datanode.data.dir | Major | documentation | Xiaoyu Yao | Xiaoyu Yao
YARN-3539 | Compatibility doc to state that ATS v1 is a stable REST API | Major | documentation | Steve Loughran | Steve Loughran
YARN-3489 | RMServerUtils.validateResourceRequests should only obtain queue info once | Major | resourcemanager | Jason Lowe | Varun Saxena
HDFS-8521 | Add @VisibleForTesting annotation to {{BlockPoolSlice#selectReplicaToDelete}} | Trivial | . | Colin P. McCabe | Colin P. McCabe
HDFS-8361 | Choose SSD over DISK in block placement | Major | namenode | Tsz Wo Nicholas Sze | Tsz Wo Nicholas Sze

### BUG FIXES:

JIRA | Summary | Priority | Component | Reporter | Contributor
--- | --- | --- | --- | --- | ---
YARN-3243 | CapacityScheduler should pass headroom from parent to children to make sure ParentQueue obey its capacity limits. | Major | capacityscheduler, resourcemanager | Wangda Tan | Wangda Tan
YARN-3351 | AppMaster tracking URL is broken in HA | Major | webapp | Anubhav Dhoot | Anubhav Dhoot
HDFS-5215 | dfs.datanode.du.reserved is not considered while computing available space | Major | datanode | Brahma Reddy Battula | Brahma Reddy Battula
YARN-3457 | NPE when NodeManager.serviceInit fails and stopRecoveryStore called | Minor | nodemanager | Bibin A Chundatt | Bibin A Chundatt
YARN-3465 | Use LinkedHashMap to preserve order of resource requests | Major | nodemanager | zhihai xu | zhihai xu
HDFS-8091 | ACLStatus and XAttributes not properly presented to INodeAttributesProvider before returning to client | Major | namenode | Arun Suresh | Arun Suresh
HDFS-8081 | Split getAdditionalBlock() into two methods. | Major | . | Konstantin Shvachko | Konstantin Shvachko
YARN-3466 | Fix RM nodes web page to sort by node HTTP-address, #containers and node-label column | Major | resourcemanager, webapp | Jason Lowe | Jason Lowe
HDFS-7931 | DistributedFIleSystem should not look for keyProvider in cache if Encryption is disabled | Minor | hdfs-client | Arun Suresh | Arun Suresh
YARN-3382 | Some of UserMetricsInfo metrics are incorrectly set to root queue metrics | Major | webapp | Rohit Agarwal | Rohit Agarwal
YARN-3472 | Possible leak in DelegationTokenRenewer#allTokens | Major | . | Jian He | Rohith Sharma K S
HDFS-8127 | NameNode Failover during HA upgrade can cause DataNode to finalize upgrade | Blocker | ha | Jing Zhao | Jing Zhao
YARN-3462 | Patches applied for YARN-2424 are inconsistent between trunk and branch-2 | Major | . | Sidharta Seethana | Naganarasimha G R
HDFS-8151 | Always use snapshot path as source when invalid snapshot names are used for diff based distcp | Minor | distcp | Sushmitha Sreenivasan | Jing Zhao
HDFS-7934 | Update RollingUpgrade rollback documentation: should use bootstrapstandby for standby NN | Critical | documentation | J.Andreina | J.Andreina
HDFS-8149 | The footer of the Web UI “Hadoop, 2014” is old | Major | . | Akira Ajisaka | Brahma Reddy Battula
MAPREDUCE-6300 | Task list sort by task id broken | Minor | . | Siqi Li | Siqi Li
HDFS-8153 | Error Message points to wrong parent directory in case of path component name length error | Major | namenode | Anu Engineer | Anu Engineer
YARN-3493 | RM fails to come up with error “Failed to load/recover state” when mem settings are changed | Critical | yarn | Sumana Sathish | Jian He
YARN-3497 | ContainerManagementProtocolProxy modifies IPC timeout conf without making a copy | Major | client | Jason Lowe | Jason Lowe
MAPREDUCE-6238 | MR2 can’t run local jobs with -libjars command options which is a regression from MR1 | Critical | mrv2 | zhihai xu | zhihai xu
HDFS-8179 | DFSClient#getServerDefaults returns null within 1 hour of system start | Blocker | . | Xiaoyu Yao | Xiaoyu Yao
HDFS-8163 | Using monotonicNow for block report scheduling causes test failures on recently restarted systems | Blocker | datanode | Arpit Agarwal | Arpit Agarwal
HADOOP-11868 | Invalid user logins trigger large backtraces in server log | Major | . | Chang Li | Chang Li
YARN-3522 | DistributedShell uses the wrong user to put timeline data | Blocker | timelineserver | Zhijie Shen | Zhijie Shen
HDFS-8147 | Mover should not schedule two replicas to the same DN storage | Major | balancer & mover | Surendra Singh Lilhore | Surendra Singh Lilhore
HADOOP-11872 | “hadoop dfs” command prints message about using “yarn jar” on Windows(branch-2 only) | Minor | scripts | Varun Vasudev | Varun Vasudev
HADOOP-11730 | Regression: s3n read failure recovery broken | Major | fs/s3 | Takenori Sato | Takenori Sato
YARN-3516 | killing ContainerLocalizer action doesn’t take effect when private localizer receives FETCH_FAILURE status. | Minor | nodemanager | zhihai xu | zhihai xu
HADOOP-11802 | DomainSocketWatcher thread terminates sometimes after there is an I/O error during requestShortCircuitShm | Major | . | Eric Payne | Colin P. McCabe
HDFS-8070 | Pre-HDFS-7915 DFSClient cannot use short circuit on post-HDFS-7915 DataNode | Blocker | caching | Gopal V | Colin P. McCabe
YARN-3537 | NPE when NodeManager.serviceInit fails and stopRecoveryStore invoked | Major | nodemanager | Brahma Reddy Battula | Brahma Reddy Battula
YARN-3464 | Race condition in LocalizerRunner kills localizer before localizing all resources | Critical | nodemanager | zhihai xu | zhihai xu
MAPREDUCE-6252 | JobHistoryServer should not fail when encountering a missing directory | Major | jobhistoryserver | Craig Welch | Craig Welch
MAPREDUCE-6324 | Uber jobs fail to update AMRM token when it rolls over | Blocker | mr-am | Jason Lowe | Jason Lowe
MAPREDUCE-6334 | Fetcher#copyMapOutput is leaking usedMemory upon IOException during InMemoryMapOutput shuffle handler | Blocker | . | Eric Payne | Eric Payne
HDFS-8273 | FSNamesystem#Delete() should not call logSync() when holding the lock | Blocker | namenode | Jing Zhao | Haohui Mai
YARN-3485 | FairScheduler headroom calculation doesn’t consider maxResources for Fifo and FairShare policies | Critical | fairscheduler | Karthik Kambatla | Karthik Kambatla
HDFS-8269 | getBlockLocations() does not resolve the .reserved path and generates incorrect edit logs when updating the atime | Blocker | . | Yesha Vora | Haohui Mai
MAPREDUCE-6339 | Job history file is not flushed correctly because isTimerActive flag is not set true when flushTimerTask is scheduled. | Critical | mrv2 | zhihai xu | zhihai xu
HADOOP-11891 | OsSecureRandom should lazily fill its reservoir | Major | security | Arun Suresh | Arun Suresh
HDFS-8213 | DFSClient should use hdfs.client.htrace HTrace configuration prefix rather than hadoop.htrace | Critical | . | Billie Rinaldi | Colin P. McCabe
HADOOP-9658 | SnappyCodec#checkNativeCodeLoaded may unexpectedly fail when native code is not loaded | Major | . | Zhijie Shen | Zhijie Shen
MAPREDUCE-6259 | IllegalArgumentException due to missing job submit time | Major | jobhistoryserver | zhihai xu | zhihai xu
HDFS-8305 | HDFS INotify: the destination field of RenameOp should always end with the file name | Major | . | Colin P. McCabe | Colin P. McCabe
YARN-3385 | Race condition: KeeperException$NoNodeException will cause RM shutdown during ZK node deletion. | Critical | resourcemanager | zhihai xu | zhihai xu
HDFS-7980 | Incremental BlockReport will dramatically slow down the startup of a namenode | Major | . | Hui Zheng | Walter Su
YARN-3358 | Audit log not present while refreshing Service ACLs | Minor | resourcemanager | Varun Saxena | Varun Saxena
YARN-3554 | Default value for maximum nodemanager connect wait time is too high | Major | . | Jason Lowe | Naganarasimha G R
HDFS-7894 | Rolling upgrade readiness is not updated in jmx until query command is issued. | Critical | . | Kihwal Lee | Brahma Reddy Battula
HDFS-8245 | Standby namenode doesn’t process DELETED_BLOCK if the add block request is in edit log. | Major | . | Rushabh S Shah | Rushabh S Shah
YARN-3476 | Nodemanager can fail to delete local logs if log aggregation fails | Major | log-aggregation, nodemanager | Jason Lowe | Rohith Sharma K S
HADOOP-11663 | Remove description about Java 6 from docs | Minor | documentation | Masatake Iwasaki | Masatake Iwasaki
HDFS-7916 | ‘reportBadBlocks’ from datanodes to standby Node BPServiceActor goes for infinite loop | Critical | datanode | Vinayakumar B | Rushabh S Shah
YARN-3434 | Interaction between reservations and userlimit can result in significant ULF violation | Major | capacityscheduler | Thomas Graves | Thomas Graves
MAPREDUCE-6361 | NPE issue in shuffle caused by concurrent issue between copySucceeded() in one thread and copyFailed() in another thread on the same host | Critical | . | Junping Du | Junping Du
MAPREDUCE-6251 | JobClient needs additional retries at a higher level to address not-immediately-consistent dfs corner cases | Major | jobhistoryserver, mrv2 | Craig Welch | Craig Welch
HDFS-6300 | Prevent multiple balancers from running simultaneously | Critical | balancer & mover | Rakesh R | Rakesh R
HADOOP-11966 | Variable cygwin is undefined in hadoop-config.sh when executed through hadoop-daemon.sh. | Critical | scripts | Chris Nauroth | Chris Nauroth
YARN-3526 | ApplicationMaster tracking URL is incorrectly redirected on a QJM cluster | Major | resourcemanager, webapp | Weiwei Yang | Weiwei Yang
HDFS-8405 | Fix a typo in NamenodeFsck | Minor | namenode | Tsz Wo Nicholas Sze | Takanobu Asanuma
YARN-3601 | Fix UT TestRMFailover.testRMWebAppRedirect | Critical | resourcemanager, webapp | Weiwei Yang | Weiwei Yang
HDFS-8404 | Pending block replication can get stuck using older genstamp | Major | namenode | Nathan Roberts | Nathan Roberts
HADOOP-11973 | Ensure ZkDelegationTokenSecretManager namespace znodes get created with ACLs | Major | security | Gregory Chanan | Gregory Chanan
YARN-3677 | Fix findbugs warnings in yarn-server-resourcemanager | Minor | resourcemanager | Akira Ajisaka | Vinod Kumar Vavilapalli
YARN-3681 | yarn cmd says “could not find main class ‘queue’” in windows | Blocker | yarn | Sumana Sathish | Varun Saxena
YARN-3694 | Fix dead link for TimelineServer REST API | Minor | documentation | Akira Ajisaka | Jagadesh Kiran N
YARN-3646 | Applications are getting stuck some times in case of retry policy forever | Major | client | Raju Bairishetti | Raju Bairishetti
HDFS-8451 | DFSClient probe for encryption testing interprets empty URI property for “enabled” | Blocker | encryption | Steve Loughran | Steve Loughran
YARN-3675 | FairScheduler: RM quits when node removal races with continousscheduling on the same node | Critical | fairscheduler | Anubhav Dhoot | Anubhav Dhoot
YARN-3701 | Isolating the error of generating a single app report when getting all apps from generic history service | Blocker | timelineserver | Zhijie Shen | Zhijie Shen
YARN-2238 | filtering on UI sticks even if I move away from the page | Major | webapp | Sangjin Lee | Jian He
YARN-3626 | On Windows localized resources are not moved to the front of the classpath when they should be | Major | yarn | Craig Welch | Craig Welch
YARN-3723 | Need to clearly document primaryFilter and otherInfo value type | Critical | timelineserver | Zhijie Shen | Zhijie Shen
HADOOP-11934 | Use of JavaKeyStoreProvider in LdapGroupsMapping causes infinite loop | Blocker | security | Mike Yoder | Larry McCay
YARN-3725 | App submission via REST API is broken in secure mode due to Timeline DT service address is empty | Blocker | resourcemanager, timelineserver | Zhijie Shen | Zhijie Shen
YARN-3753 | RM failed to come up with “java.io.IOException: Wait for ZKClient creation timed out” | Critical | yarn | Sumana Sathish | Jian He
HDFS-8486 | DN startup may cause severe data loss | Blocker | datanode | Daryn Sharp | Daryn Sharp
HDFS-8270 | create() always retried with hardcoded timeout when file already exists with open lease | Major | hdfs-client | Andrey Stepachev | J.Andreina
HDFS-8523 | Remove usage information on unsupported operation “fsck -showprogress” from branch-2 | Major | documentation | J.Andreina | J.Andreina
YARN-3585 | NodeManager cannot exit on SHUTDOWN event triggered and NM recovery is enabled | Critical | . | Peng Zhang | Rohith Sharma K S
HADOOP-12058 | Fix dead links to DistCp and Hadoop Archives pages. | Minor | documentation, site | Kazuho Fujii | Kazuho Fujii
YARN-3764 | CapacityScheduler should forbid moving LeafQueue from one parent to another | Blocker | . | Wangda Tan | Wangda Tan
YARN-3733 | Fix DominantRC#compare() does not work as expected if cluster resource is empty | Blocker | resourcemanager | Bibin A Chundatt | Rohith Sharma K S
MAPREDUCE-6387 | Serialize the recently added Task#encryptedSpillKey field at the end | Minor | . | Arun Suresh | Arun Suresh
HDFS-8522 | Change heavily recorded NN logs from INFO to DEBUG level | Major | namenode | Xiaoyu Yao | Xiaoyu Yao
HDFS-8566 | HDFS documentation about debug commands wrongly identifies them as “hdfs dfs” commands | Major | documentation | Surendra Singh Lilhore | Surendra Singh Lilhore
HDFS-8583 | Document that NFS gateway does not work with rpcbind on SLES 11 | Major | documentation | Arpit Agarwal | Arpit Agarwal
HDFS-8572 | DN always uses HTTP/localhost@REALM principals in SPNEGO | Blocker | . | Haohui Mai | Haohui Mai
HDFS-8596 | TestDistributedFileSystem et al tests are broken in branch-2 due to incorrect setting of “datanode” attribute | Blocker | datanode | Yongjun Zhang | Yongjun Zhang
HDFS-8595 | TestCommitBlockSynchronization fails in branch-2.7 | Major | test | Arpit Agarwal | Arpit Agarwal
HDFS-8600 | TestWebHdfsFileSystemContract.testGetFileBlockLocations fails in branch-2.7 | Major | webhdfs | Arpit Agarwal | Arpit Agarwal
HADOOP-12078 | The default retry policy does not handle RetriableException correctly | Critical | ipc | Arpit Agarwal | Arpit Agarwal
HDFS-8576 | Lease recovery should return true if the lease can be released and the file can be closed | Major | namenode | J.Andreina | J.Andreina
HDFS-8544 | Incorrect port specified in HFTP Guide document in branch-2 | Major | documentation | Brahma Reddy Battula | Brahma Reddy Battula
HDFS-4660 | Block corruption can happen during pipeline recovery | Blocker | datanode | Peng Zhang | Kihwal Lee
YARN-3804 | Both RM are on standBy state when kerberos user not in yarn.admin.acl | Critical | resourcemanager | Bibin A Chundatt | Varun Saxena
HADOOP-12100 | ImmutableFsPermission should not override applyUmask since that method doesn’t modify the FsPermission | Major | . | Robert Kanter | Bibin A Chundatt
HDFS-8633 | Fix setting of dfs.datanode.readahead.bytes in hdfs-default.xml to match DFSConfigKeys | Minor | datanode | Ray Chiang | Ray Chiang
HDFS-8626 | Reserved RBW space is not released if creation of RBW File fails | Blocker | . | Kanaka Kumar Avvaru | Kanaka Kumar Avvaru
HADOOP-12103 | Small refactoring of DelegationTokenAuthenticationFilter to allow code sharing | Minor | security | Yongjun Zhang | Yongjun Zhang
HDFS-8480 | Fix performance and timeout issues in HDFS-7929 by using hard-links to preserve old edit logs instead of copying them | Critical | . | Zhe Zhang | Zhe Zhang
YARN-3842 | NMProxy should retry on NMNotYetReadyException | Critical | . | Karthik Kambatla | Robert Kanter
MAPREDUCE-6410 | Aggregated Logs Deletion doesnt work after refreshing Log Retention Settings in secure cluster | Critical | . | Zhang Wei | Varun Saxena
YARN-3809 | Failed to launch new attempts because ApplicationMasterLauncher’s threads all hang | Major | resourcemanager | Jun Gong | Jun Gong
YARN-3832 | Resource Localization fails on a cluster due to existing cache directories | Critical | nodemanager | Ranga Swamy | Brahma Reddy Battula
YARN-3850 | NM fails to read files from full disks which can lead to container logs being lost and other issues | Blocker | log-aggregation, nodemanager | Varun Saxena | Varun Saxena
HDFS-8681 | BlockScanner is incorrectly disabled by default | Blocker | datanode | Andrew Wang | Arpit Agarwal

### SUB-TASKS:

JIRA | Summary | Priority | Component | Reporter | Contributor
--- | --- | --- | --- | --- | ---
YARN-3487 | CapacityScheduler scheduler lock obtained unnecessarily when calling getQueue | Critical | capacityscheduler | Jason Lowe | Jason Lowe
YARN-2605 | [RM HA] Rest api endpoints doing redirect incorrectly | Major | resourcemanager | bc Wong | Xuan Gong
YARN-3544 | AM logs link missing in the RM UI for a completed app | Blocker | . | Hitesh Shah | Xuan Gong
YARN-3006 | Improve the error message when attempting manual failover with auto-failover enabled | Minor | . | Akira Ajisaka | Akira Ajisaka
YARN-3301 | Fix the format issue of the new RM web UI and AHS web UI after YARN-3272 / YARN-3262 | Major | resourcemanager | Xuan Gong | Xuan Gong
YARN-2918 | Don’t fail RM if queue’s configured labels are not existed in cluster-node-labels | Major | resourcemanager | Rohith Sharma K S | Wangda Tan
YARN-3609 | Move load labels from storage from serviceInit to serviceStart to make it works with RM HA case. | Major | resourcemanager | Wangda Tan | Wangda Tan
YARN-3686 | CapacityScheduler should trim default_node_label_expression | Critical | api, client, resourcemanager | Wangda Tan | Sunil Govindan
YARN-2900 | Application (Attempt and Container) Not Found in AHS results in Internal Server Error (500) | Major | timelineserver | Jonathan Eagles | Mit Desai
YARN-3711 | Documentation of ResourceManager HA should explain configurations about listen addresses | Minor | documentation | Masatake Iwasaki | Masatake Iwasaki
HDFS-8597 | Fix TestFSImage#testZeroBlockSize on Windows | Major | datanode, test | Xiaoyu Yao | Xiaoyu Yao
HDFS-7164 | Feature documentation for HDFS-6581 | Major | documentation | Arpit Agarwal | Arpit Agarwal