Apache Hadoop 3.5.0 – Apache Hadoop Changelog

##### General

- Overview
- Single Node Setup
- Cluster Setup
- Commands Reference
- FileSystem Shell
- Compatibility Specification
- Downstream Developer's Guide
- Admin Compatibility Guide
- Interface Classification
- FileSystem Specification

##### Common

- CLI Mini Cluster
- Fair Call Queue
- Native Libraries
- Proxy User
- Rack Awareness
- Secure Mode
- Service Level Authorization
- HTTP Authentication
- Credential Provider API
- Hadoop KMS
- Tracing
- Unix Shell Guide
- Registry
- Async Profiler

##### HDFS

- Architecture
- User Guide
- Commands Reference
- NameNode HA With QJM
- NameNode HA With NFS
- Observer NameNode
- Federation
- ViewFs
- ViewFsOverloadScheme
- Snapshots
- Edits Viewer
- Image Viewer
- Permissions and HDFS
- Quotas and HDFS
- libhdfs (C API)
- WebHDFS (REST API)
- HttpFS
- Short Circuit Local Reads
- Centralized Cache Management
- NFS Gateway
- Rolling Upgrade
- Extended Attributes
- Transparent Encryption
- Multihoming
- Storage Policies
- Memory Storage Support
- Synthetic Load Generator
- Erasure Coding
- Disk Balancer
- Upgrade Domain
- DataNode Admin
- Router Federation
- Provided Storage

##### MapReduce

- Tutorial
- Commands Reference
- Encrypted Shuffle
- Pluggable Shuffle/Sort
- Distributed Cache Deploy
- Support for YARN Shared Cache

##### MapReduce REST APIs

- MR Application Master
- MR History Server

##### YARN

- Architecture
- Commands Reference
- Capacity Scheduler
- Fair Scheduler
- ResourceManager Restart
- ResourceManager HA
- Resource Model
- Node Labels
- Node Attributes
- Web Application Proxy
- Timeline Server
- Timeline Service V.2
- Writing YARN Applications
- YARN Application Security
- NodeManager
- Running Applications in Docker Containers
- Running Applications in runC Containers
- Using CGroups
- Secure Containers
- Reservation System
- Graceful Decommission
- Opportunistic Containers
- YARN Federation
- Shared Cache
- Using GPU
- Using FPGA
- Placement Constraints
- YARN UI2

##### YARN REST APIs

- Introduction
- Resource Manager
- Node Manager
- Timeline Server
- Timeline Service V.2

##### YARN Service

- Overview
- QuickStart
- Concepts
- Yarn Service API
- Service Discovery
- System Services

##### Hadoop Compatible File Systems

- Aliyun OSS
- Amazon S3
- Azure Blob Storage
- Azure Data Lake Storage
- Tencent COS
- Huaweicloud OBS
- VolcanoEngine TOS

##### Auth

- Overview
- Examples
- Configuration
- Building

##### Tools

- Hadoop Streaming
- Hadoop Archives
- Hadoop Archive Logs
- DistCp
- HDFS Federation Balance
- GridMix
- Rumen
- Resource Estimator Service
- Scheduler Load Simulator
- Hadoop Benchmarking
- Dynamometer

##### Reference

- Changelog and Release Notes
- Java API docs
- Unix Shell API
- Metrics

##### Configuration

- core-default.xml
- hdfs-default.xml
- hdfs-rbf-default.xml
- mapred-default.xml
- yarn-default.xml
- kms-default.xml
- httpfs-default.xml
- Deprecated Properties

# Apache Hadoop Changelog

## Release 3.1.4 - 2020-07-21

### NEW FEATURES:

JIRA | Summary | Priority | Component | Reporter | Contributor
--- | --- | --- | --- | --- | ---
YARN-9761 | Allow overriding application submissions based on server side configs | Major | . | Jonathan Hung | pralabhkumar
YARN-9760 | Support configuring application priorities on a workflow level | Major | . | Jonathan Hung | Varun Saxena
HDFS-14745 | Backport HDFS persistent memory read cache support to branch-3.1 | Major | . | Feilong He | Feilong He
HDFS-12943 | Consistent Reads from Standby Node | Major | hdfs | Konstantin Shvachko | Konstantin Shvachko

### IMPROVEMENTS:

JIRA | Summary | Priority | Component | Reporter | Contributor
--- | --- | --- | --- | --- | ---
YARN-8361 | Change App Name Placement Rule to use App Name instead of App Id for configuration | Major | yarn | Zian Chen | Zian Chen
YARN-8750 | Refactor TestQueueMetrics | Minor | resourcemanager | Szilard Nemeth | Szilard Nemeth
HADOOP-15849 | Upgrade netty version to 3.10.6 | Major | . | Xiao Chen | Xiao Chen
HDFS-14064 | WEBHDFS: Support Enable/Disable EC Policy | Major | erasure-coding, webhdfs | Ayush Saxena | Ayush Saxena
HDFS-14113 | EC : Add Configuration to restrict UserDefined Policies | Major | erasure-coding | Ayush Saxena | Ayush Saxena
HDFS-14124 | EC : Support EC Commands (set/get/unset EcPolicy) via WebHdfs | Major | erasure-coding, httpfs, webhdfs | Souryakanta Dwivedy | Ayush Saxena
HDFS-14006 | Refactor name node to allow different token verification implementations | Major | . | CR Hota | CR Hota
HADOOP-15909 | KeyProvider class should implement Closeable | Major | kms | Kuhu Shukla | Kuhu Shukla
HDFS-14187 | Make warning message more clear when there are not enough data nodes for EC write | Major | erasure-coding | Kitti Nanasi | Kitti Nanasi
HDFS-14235 | Handle ArrayIndexOutOfBoundsException in DataNodeDiskMetrics#slowDiskDetectionDaemon | Major | . | Surendra Singh Lilhore | Ranith Sardar
HADOOP-16126 | ipc.Client.stop() may sleep too long to wait for all connections | Major | ipc | Tsz-wo Sze | Tsz-wo Sze
HADOOP-16140 | hadoop fs expunge to add -immediate option to purge trash immediately | Major | fs | Stephen O’Donnell | Stephen O’Donnell
HADOOP-15014 | KMS should log the IP address of the clients | Major | kms | Zsombor Gegesy | Zsombor Gegesy
HDFS-14460 | DFSUtil#getNamenodeWebAddr should return HTTPS address based on policy configured | Major | . | CR Hota | CR Hota
HDFS-14624 | When decommissioning a node, log remaining blocks to replicate periodically | Major | namenode | Stephen O’Donnell | Stephen O’Donnell
HDFS-13693 | Remove unnecessary search in INodeDirectory.addChild during image loading | Major | namenode | zhouyingchao | Lisheng Sun
HDFS-14313 | Get hdfs used space from FsDatasetImpl#volumeMap#ReplicaInfo in memory instead of df/du | Major | datanode, performance | Lisheng Sun | Lisheng Sun
HDFS-14678 | Allow triggerBlockReport to a specific namenode | Major | datanode | Leon Gao | Leon Gao
HDFS-14523 | Remove excess read lock for NetworkToplogy | Major | . | Wu Weiwei | Wu Weiwei
HDFS-14497 | Write lock held by metasave impact following RPC processing | Major | namenode | Xiaoqiao He | Xiaoqiao He
YARN-9756 | Create metric that sums total memory/vcores preempted per round | Major | capacity scheduler | Eric Payne | Manikandan R
YARN-9810 | Add queue capacity/maxcapacity percentage metrics | Major | . | Jonathan Hung | Shubham Gupta
HADOOP-16531 | Log more detail for slow RPC | Major | . | Chen Zhang | Chen Zhang
YARN-9763 | Print application tags in application summary | Major | . | Jonathan Hung | Manoj Kumar
YARN-9764 | Print application submission context label in application summary | Major | . | Jonathan Hung | Manoj Kumar
YARN-9824 | Fall back to configured queue ordering policy class name | Major | . | Jonathan Hung | Jonathan Hung
HADOOP-16069 | Support configure ZK_DTSM_ZK_KERBEROS_PRINCIPAL in ZKDelegationTokenSecretManager using principal with Schema /_HOST | Minor | common | luhuachao | luhuachao
YARN-9762 | Add submission context label to audit logs | Major | . | Jonathan Hung | Manoj Kumar
HDFS-14850 | Optimize FileSystemAccessService#getFileSystemConfiguration | Major | httpfs, performance | Lisheng Sun | Lisheng Sun
HDFS-14192 | Track missing DFS operations in Statistics and StorageStatistics | Major | . | Ayush Saxena | Ayush Saxena
HADOOP-16625 | Backport HADOOP-14624 to branch-3.1 | Major | . | Wei-Chiu Chuang | Wei-Chiu Chuang
YARN-9356 | Add more tests to ratio method in TestResourceCalculator | Major | . | Szilard Nemeth | Zoltan Siegl
HDFS-14915 | Move Superuser Check Before Taking Lock For Encryption API | Major | . | Ayush Saxena | Ayush Saxena
HDFS-14921 | Remove SuperUser Check in Setting Storage Policy in FileStatus During Listing | Major | . | Ayush Saxena | Ayush Saxena
HDFS-14923 | Remove dead code from HealthMonitor | Minor | . | Fei Hui | Fei Hui
YARN-9914 | Use separate configs for free disk space checking for full and not-full disks | Minor | yarn | Jim Brennan | Jim Brennan
MAPREDUCE-7208 | Tuning TaskRuntimeEstimator | Minor | . | Ahmed Hussein | Ahmed Hussein
HDFS-14942 | Change Log Level to debug in JournalNodeSyncer#syncWithJournalAtIndex | Minor | . | Lisheng Sun | Lisheng Sun
HDFS-14979 | [Observer Node] Balancer should submit getBlocks to Observer Node when possible | Major | balancer & mover, hdfs | Erik Krogen | Erik Krogen
HADOOP-16705 | MBeanInfoBuilder puts unnecessary memory pressure on the system with a debug log | Major | metrics | Lukas Majercak | Lukas Majercak
HADOOP-16712 | Config ha.failover-controller.active-standby-elector.zk.op.retries is not in core-default.xml | Trivial | . | Wei-Chiu Chuang | Xieming Li
HADOOP-16703 | Backport HADOOP-16152 to branch-3.1 | Major | . | Siyao Meng | Siyao Meng
HDFS-14952 | Skip safemode if blockTotal is 0 in new NN | Trivial | namenode | Rajesh Balamohan | Xiaoqiao He
YARN-8842 | Expose metrics for custom resource types in QueueMetrics | Major | . | Szilard Nemeth | Szilard Nemeth
YARN-9966 | Code duplication in UserGroupMappingPlacementRule | Major | . | Szilard Nemeth | Kevin Su
YARN-9937 | Add missing queue configs in RMWebService#CapacitySchedulerQueueInfo | Major | capacity scheduler | Prabhu Joseph | Prabhu Joseph
HADOOP-16718 | Allow disabling Server Name Indication (SNI) for Jetty | Major | . | Siyao Meng | Aravindan Vijayan
HADOOP-16735 | Make it clearer in config default that EnvironmentVariableCredentialsProvider supports AWS_SESSION_TOKEN | Minor | documentation, fs/s3 | Mingliang Liu | Mingliang Liu
YARN-10012 | Guaranteed and max capacity queue metrics for custom resources | Major | . | Jonathan Hung | Manikandan R
HDFS-15050 | Optimize log information when DFSInputStream meet CannotObtainBlockLengthException | Major | dfsclient | Xiaoqiao He | Xiaoqiao He
YARN-10033 | TestProportionalCapacityPreemptionPolicy not initializing vcores for effective max resources | Major | capacity scheduler, test | Eric Payne | Eric Payne
YARN-10039 | Allow disabling app submission from REST endpoints | Major | . | Jonathan Hung | Jonathan Hung
YARN-9894 | CapacitySchedulerPerf test for measuring hundreds of apps in a large number of queues. | Major | capacity scheduler, test | Eric Payne | Eric Payne
HADOOP-16771 | Update checkstyle to 8.26 and maven-checkstyle-plugin to 3.1.0 | Major | build | Andras Bokor | Andras Bokor
YARN-10009 | In Capacity Scheduler, DRC can treat minimum user limit percent as a max when custom resource is defined | Critical | capacity scheduler | Eric Payne | Eric Payne
HDFS-12999 | When reach the end of the block group, it may not need to flush all the data packets(flushAllInternals) twice. | Major | erasure-coding, hdfs-client | lufei | lufei
HDFS-15074 | DataNode.DataTransfer thread should catch all the expception and log it. | Major | datanode | Surendra Singh Lilhore | Hemanth Boyina
HDFS-14740 | Recover data blocks from persistent memory read cache during datanode restarts | Major | caching, datanode | Feilong He | Feilong He
HDFS-15097 | Purge log in KMS and HttpFS | Minor | httpfs, kms | Doris Gu | Doris Gu
HDFS-14968 | Add ability to know datanode staleness | Minor | datanode, logging, namenode | Ahmed Hussein | Ahmed Hussein
YARN-7913 | Improve error handling when application recovery fails with exception | Major | resourcemanager | Gergo Repas | Wilfred Spiegelenburg
HDFS-15119 | Allow expiration of cached locations in DFSInputStream | Minor | dfsclient | Ahmed Hussein | Ahmed Hussein
MAPREDUCE-7262 | MRApp helpers block for long intervals (500ms) | Minor | mr-am | Ahmed Hussein | Ahmed Hussein
YARN-10084 | Allow inheritance of max app lifetime / default app lifetime | Major | capacity scheduler | Eric Payne | Eric Payne
HDFS-12491 | Support wildcard in CLASSPATH for libhdfs | Major | libhdfs | John Zhuge | Muhammad Samir Khan
YARN-10116 | Expose diagnostics in RMAppManager summary | Major | . | Jonathan Hung | Jonathan Hung
HDFS-14758 | Decrease lease hard limit | Minor | . | Eric Payne | Hemanth Boyina
HDFS-15086 | Block scheduled counter never get decremet if the block got deleted before replication. | Major | 3.1.1 | Surendra Singh Lilhore | Hemanth Boyina
HDFS-15174 | Optimize ReplicaCachingGetSpaceUsed by reducing unnecessary io operations | Major | . | Lisheng Sun | Lisheng Sun
YARN-9018 | Add functionality to AuxiliaryLocalPathHandler to return all locations to read for a given path | Major | . | Kuhu Shukla | Kuhu Shukla
HDFS-14861 | Reset LowRedundancyBlocks Iterator periodically | Major | namenode | Stephen O’Donnell | Stephen O’Donnell
HADOOP-16882 | Update jackson-databind to 2.9.10.2 in branch-3.1, branch-2.10 | Blocker | . | Wei-Chiu Chuang | Lisheng Sun
HADOOP-16776 | backport HADOOP-16775: distcp copies to s3 are randomly corrupted | Blocker | tools/distcp | Amir Shenavandeh | Amir Shenavandeh
HDFS-15197 | [SBN read] Change ObserverRetryOnActiveException log to debug | Minor | hdfs | Chen Liang | Chen Liang
YARN-10200 | Add number of containers to RMAppManager summary | Major | . | Jonathan Hung | Jonathan Hung
HADOOP-16952 | Add .diff to gitignore | Minor | . | Ayush Saxena | Ayush Saxena
YARN-10212 | Create separate configuration for max global AM attempts | Major | . | Jonathan Hung | Bilwa S T
YARN-9954 | Configurable max application tags and max tag length | Major | . | Jonathan Hung | Bilwa S T
HADOOP-17001 | The suffix name of the unified compression class | Major | io | bianqi | bianqi
HDFS-15295 | AvailableSpaceBlockPlacementPolicy should use chooseRandomWithStorageTypeTwoTrial() for better performance. | Minor | . | Jinglun | Jinglun
HADOOP-17127 | Use RpcMetrics.TIMEUNIT to initialize rpc queueTime and processingTime | Minor | common | Jim Brennan | Jim Brennan

### BUG FIXES:

JIRA | Summary | Priority | Component | Reporter | Contributor
--- | --- | --- | --- | --- | ---
HDFS-13806 | EC: No error message for unsetting EC policy of the directory inherits the erasure coding policy from an ancestor directory | Minor | erasure-coding | Souryakanta Dwivedy | Ayush Saxena
HDFS-12459 | Fix revert: Add new op GETFILEBLOCKLOCATIONS to WebHDFS REST API | Major | webhdfs | Weiwei Yang | Weiwei Yang
HADOOP-15418 | Hadoop KMSAuthenticationFilter needs to use getPropsByPrefix instead of iterator to avoid ConcurrentModificationException | Major | common | Suma Shivaprasad | Suma Shivaprasad
HDFS-14004 | TestLeaseRecovery2#testCloseWhileRecoverLease fails intermittently in trunk | Major | . | Ayush Saxena | Ayush Saxena
HDFS-13959 | TestUpgradeDomainBlockPlacementPolicy is flaky | Major | . | Ayush Saxena | Ayush Saxena
YARN-8948 | PlacementRule interface should be for all YarnSchedulers | Major | . | Bibin Chundatt | Bibin Chundatt
HADOOP-16013 | DecayRpcScheduler decay thread should run as a daemon | Major | ipc | Erik Krogen | Erik Krogen
HDFS-14175 | EC: Native XOR decoder should reset the output buffer before using it. | Major | ec, hdfs | Surendra Singh Lilhore | Ayush Saxena
HDFS-14202 | “dfs.disk.balancer.max.disk.throughputInMBperSec” property is not working as per set value. | Major | diskbalancer | Ranith Sardar | Ranith Sardar
HADOOP-16032 | Distcp It should clear sub directory ACL before applying new ACL on it. | Major | tools/distcp | Ranith Sardar | Ranith Sardar
HADOOP-16127 | In ipc.Client, put a new connection could happen after stop | Major | ipc | Tsz-wo Sze | Tsz-wo Sze
YARN-4901 | QueueMetrics needs to be cleared before MockRM is initialized | Major | scheduler | Daniel Templeton | Peter Bacsko
HADOOP-16161 | NetworkTopology#getWeightUsingNetworkLocation return unexpected result | Major | net | Xiaoqiao He | Xiaoqiao He
HDFS-14434 | webhdfs that connect secure hdfs should not use user.name parameter | Minor | webhdfs | KWON BYUNGCHANG | KWON BYUNGCHANG
HDFS-14527 | Stop all DataNodes may result in NN terminate | Major | namenode | Xiaoqiao He | Xiaoqiao He
HDFS-14494 | Move Server logging of StatedId inside receiveRequestState() | Major | . | Konstantin Shvachko | Shweta
HDFS-14618 | Incorrect synchronization of ArrayList field (ArrayList is thread-unsafe). | Critical | . | Paul Ward | Paul Ward
HDFS-14610 | HashMap is not thread safe. Field storageMap is typically synchronized by storageMap. However, in one place, field storageMap is not protected with synchronized. | Critical | . | Paul Ward | Paul Ward
HDFS-14499 | Misleading REM_QUOTA value with snapshot and trash feature enabled for a directory | Major | snapshots | Shashikant Banerjee | Shashikant Banerjee
HDFS-14647 | NPE during secure namenode startup | Major | hdfs | Fengnan Li | Fengnan Li
HADOOP-16461 | Regression: FileSystem cache lock parses XML within the lock | Major | fs | Gopal Vijayaraghavan | Gopal Vijayaraghavan
HDFS-14660 | [SBN Read] ObserverNameNode should throw StandbyException for requests not from ObserverProxyProvider | Major | . | Chao Sun | Chao Sun
HDFS-14569 | Result of crypto -listZones is not formatted properly | Major | . | Hemanth Boyina | Hemanth Boyina
HADOOP-12282 | Connection thread’s name should be updated after address changing is detected | Major | ipc | zhouyingchao | Lisheng Sun
HDFS-14686 | HttpFS: HttpFSFileSystem#getErasureCodingPolicy always returns null | Major | httpfs | Siyao Meng | Siyao Meng
HADOOP-15865 | ConcurrentModificationException in Configuration.overlay() method | Major | . | Oleksandr Shevchenko | Oleksandr Shevchenko
HDFS-14759 | HDFS cat logs an info message | Major | . | Eric Badger | Eric Badger
HDFS-2470 | NN should automatically set permissions on dfs.namenode.*.dir | Major | namenode | Aaron Myers | Siddharth Wagle
YARN-9718 | Yarn REST API, services endpoint remote command ejection | Major | . | Eric Yang | Eric Yang
YARN-9813 | RM does not start on JDK11 when UIv2 is enabled | Critical | resourcemanager, yarn | Adam Antal | Adam Antal
YARN-9820 | RM logs InvalidStateTransitionException when app is submitted | Critical | . | Rohith Sharma K S | Prabhu Joseph
HDFS-14838 | RBF: Display RPC (instead of HTTP) Port Number in RBF web UI | Minor | rbf, ui | Xieming Li | Xieming Li
HDFS-14699 | Erasure Coding: Storage not considered in live replica when replication streams hard limit reached to threshold | Critical | ec | Zhao Yi Ming | Zhao Yi Ming
YARN-9833 | Race condition when DirectoryCollection.checkDirs() runs during container launch | Major | . | Peter Bacsko | Peter Bacsko
YARN-9837 | YARN Service fails to fetch status for Stopped apps with bigger spec files | Major | yarn-native-services | Tarun Parimi | Tarun Parimi
YARN-2255 | YARN Audit logging not added to log4j.properties | Major | . | Varun Saxena | Aihua Xu
HDFS-14836 | FileIoProvider should not increase FileIoErrors metric in datanode volume metric | Minor | . | Aiphago | Aiphago
HADOOP-16581 | ValueQueue does not trigger an async refill when number of values falls below watermark | Major | common, kms | Yuval Degani | Yuval Degani
HDFS-14853 | NPE in DFSNetworkTopology#chooseRandomWithStorageType() when the excludedNode is not present | Major | . | Ranith Sardar | Ranith Sardar
HDFS-13660 | DistCp job fails when new data is appended in the file while the distCp copy job is running | Critical | distcp | Mukund Thakur | Mukund Thakur
HDFS-14808 | EC: Improper size values for corrupt ec block in LOG | Major | ec | Harshakiran Reddy | Ayush Saxena
HDFS-14849 | Erasure Coding: the internal block is replicated many times when datanode is decommissioning | Major | ec, erasure-coding | HuangTao | HuangTao
YARN-9858 | Optimize RMContext getExclusiveEnforcedPartitions | Major | . | Jonathan Hung | Jonathan Hung
HDFS-14492 | Snapshot memory leak | Major | snapshots | Wei-Chiu Chuang | Wei-Chiu Chuang
HDFS-14418 | Remove redundant super user priveledge checks from namenode. | Major | . | Ayush Saxena | Ayush Saxena
HDFS-14216 | NullPointerException happens in NamenodeWebHdfs | Critical | . | lujie | lujie
HDFS-14637 | Namenode may not replicate blocks to meet the policy after enabling upgradeDomain | Major | namenode | Stephen O’Donnell | Stephen O’Donnell
HDFS-14879 | Header was wrong in Snapshot web UI | Major | . | Hemanth Boyina | Hemanth Boyina
HDFS-14655 | [SBN Read] Namenode crashes if one of The JN is down | Critical | . | Harshakiran Reddy | Ayush Saxena
HDFS-14859 | Prevent unnecessary evaluation of costly operation getNumLiveDataNodes when dfs.namenode.safemode.min.datanodes is not zero | Major | hdfs | Srinivasu Majeti | Srinivasu Majeti
YARN-6715 | Fix documentation about NodeHealthScriptRunner | Major | documentation, nodemanager | Peter Bacsko | Peter Bacsko
YARN-9552 | FairScheduler: NODE_UPDATE can cause NoSuchElementException | Major | fairscheduler | Peter Bacsko | Peter Bacsko
HDFS-14754 | Erasure Coding : The number of Under-Replicated Blocks never reduced | Critical | ec | Hemanth Boyina | Hemanth Boyina
HDFS-14245 | Class cast error in GetGroups with ObserverReadProxyProvider | Major | . | Shen Yinjie | Erik Krogen
HDFS-14373 | EC : Decoding is failing when block group last incomplete cell fall in to AlignedStripe | Critical | ec, hdfs-client | Surendra Singh Lilhore | Surendra Singh Lilhore
HDFS-14509 | DN throws InvalidToken due to inequality of password when upgrade NN 2.x to 3.x | Blocker | . | Yuxuan Wang | Yuxuan Wang
HDFS-14886 | In NameNode Web UI’s Startup Progress page, Loading edits always shows 0 sec | Major | . | Hemanth Boyina | Hemanth Boyina
YARN-8453 | Additional Unit tests to verify queue limit and max-limit with multiple resource types | Major | capacity scheduler | Sunil G | Adam Antal
HDFS-14890 | Setting permissions on name directory fails on non posix compliant filesystems | Blocker | . | hirik | Siddharth Wagle
HADOOP-16580 | Disable retry of FailoverOnNetworkExceptionRetry in case of AccessControlException | Major | common | Adam Antal | Adam Antal
HDFS-14909 | DFSNetworkTopology#chooseRandomWithStorageType() should not decrease storage count for excluded node which is already part of excluded scope | Major | namenode | Surendra Singh Lilhore | Surendra Singh Lilhore
HADOOP-16600 | StagingTestBase uses methods not available in Mockito 1.8.5 in branch-3.1 | Major | . | Lisheng Sun | Duo Zhang
MAPREDUCE-6441 | Improve temporary directory name generation in LocalDistributedCacheManager for concurrent processes | Major | . | William Watson | Haibo Chen
HADOOP-16662 | Remove unnecessary InnerNode check in NetworkTopology#add() | Minor | . | Lisheng Sun | Lisheng Sun
HDFS-14847 | Erasure Coding: Blocks are over-replicated while EC decommissioning | Critical | ec | Fei Hui | Fei Hui
HDFS-14913 | Correct the value of available count in DFSNetworkTopology#chooseRandomWithStorageType() | Major | . | Ayush Saxena | Ayush Saxena
YARN-9915 | Fix FindBug issue in QueueMetrics | Minor | . | Prabhu Joseph | Prabhu Joseph
HDFS-12749 | DN may not send block report to NN after NN restart | Major | datanode | TanYuxin | Xiaoqiao He
HDFS-13901 | INode access time is ignored because of race between open and rename | Major | . | Jinglun | Jinglun
YARN-9921 | Issue in PlacementConstraint when YARN Service AM retries allocation on component failure. | Major | . | Tarun Parimi | Tarun Parimi
HDFS-14910 | Rename Snapshot with Pre Descendants Fail With IllegalArgumentException. | Blocker | . | Íñigo Goiri | Wei-Chiu Chuang
HDFS-14308 | DFSStripedInputStream curStripeBuf is not freed by unbuffer() | Major | ec | Joe McDonnell | Zhao Yi Ming
HDFS-14931 | hdfs crypto commands limit column width | Major | . | Eric Badger | Eric Badger
HADOOP-16669 | TestRawLocalFileSystemContract.testPermission fails if no native library | Minor | common, test | Steve Loughran | Steve Loughran
HDFS-14920 | Erasure Coding: Decommission may hang If one or more datanodes are out of service during decommission | Major | ec | Fei Hui | Fei Hui
HDFS-13736 | BlockPlacementPolicyDefault can not choose favored nodes when ‘dfs.namenode.block-placement-policy.default.prefer-local-node’ set to false | Major | . | hu xiaodong | hu xiaodong
HDFS-14925 | rename operation should check nest snapshot | Major | namenode | Junwang Zhao | Junwang Zhao
YARN-9949 | Add missing queue configs for root queue in RMWebService#CapacitySchedulerInfo | Minor | capacity scheduler | Prabhu Joseph | Prabhu Joseph
HDFS-14945 | Revise PacketResponder’s log. | Minor | datanode | Xudong Cao | Xudong Cao
HDFS-14946 | Erasure Coding: Block recovery failed during decommissioning | Major | . | Fei Hui | Fei Hui
HDFS-14384 | When lastLocatedBlock token expire, it will take 1~3s second to refetch it. | Major | hdfs-client | Surendra Singh Lilhore | Surendra Singh Lilhore
HDFS-14806 | Bootstrap standby may fail if used in-progress tailing | Major | namenode | Chen Liang | Chen Liang
HDFS-14958 | TestBalancerWithNodeGroup is not using NetworkTopologyWithNodeGroup | Minor | hdfs | Jim Brennan | Jim Brennan
HDFS-14720 | DataNode shouldn’t report block as bad block if the block length is Long.MAX_VALUE. | Major | datanode | Surendra Singh Lilhore | Hemanth Boyina
HADOOP-16677 | Recalculate the remaining timeout millis correctly while throwing an InterupptedException in SocketIOWithTimeout. | Minor | common | Xudong Cao | Xudong Cao
HDFS-14884 | Add sanity check that zone key equals feinfo key while setting Xattrs | Major | encryption, hdfs | Mukul Kumar Singh | Mukul Kumar Singh
HADOOP-15097 | AbstractContractDeleteTest::testDeleteNonEmptyDirRecursive with misleading path | Minor | fs, test | zhoutai.zt | Xieming Li
YARN-9984 | FSPreemptionThread can cause NullPointerException while app is unregistered with containers running on a node | Major | fairscheduler | Wilfred Spiegelenburg | Wilfred Spiegelenburg
YARN-9983 | Typo in YARN Service overview documentation | Trivial | documentation | Denes Gerencser | Denes Gerencser
HADOOP-16719 | Remove the disallowed element config within maven-checkstyle-plugin | Major | . | Wanqiang Ji | Wanqiang Ji
HADOOP-16700 | RpcQueueTime may be negative when the response has to be sent later | Minor | . | xuzq | xuzq
HADOOP-15686 | Supress bogus AbstractWadlGeneratorGrammarGenerator in KMS stderr | Major | kms | Wei-Chiu Chuang | Wei-Chiu Chuang
HDFS-14940 | HDFS Balancer : Do not allow to set balancer maximum network bandwidth more than 1TB | Minor | balancer & mover | Souryakanta Dwivedy | Hemanth Boyina
YARN-9838 | Fix resource inconsistency for queues when moving app with reserved container to another queue | Critical | capacity scheduler | jiulongzhu | jiulongzhu
YARN-9968 | Public Localizer is exiting in NodeManager due to NullPointerException | Major | nodemanager | Tarun Parimi | Tarun Parimi
YARN-9011 | Race condition during decommissioning | Major | nodemanager | Peter Bacsko | Peter Bacsko
HDFS-14973 | Balancer getBlocks RPC dispersal does not function properly | Major | balancer & mover | Erik Krogen | Erik Krogen
MAPREDUCE-7240 | Exception ’ Invalid event: TA_TOO_MANY_FETCH_FAILURE at SUCCESS_FINISHING_CONTAINER’ cause job error | Critical | . | luhuachao | luhuachao
HDFS-14986 | ReplicaCachingGetSpaceUsed throws ConcurrentModificationException | Major | datanode, performance | Ryan Wu | Aiphago
MAPREDUCE-7249 | Invalid event TA_TOO_MANY_FETCH_FAILURE at SUCCESS_CONTAINER_CLEANUP causes job failure | Critical | applicationmaster, mrv2 | Wilfred Spiegelenburg | Wilfred Spiegelenburg
HDFS-15010 | BlockPoolSlice#addReplicaThreadPool static pool should be initialized by static method | Major | datanode | Surendra Singh Lilhore | Surendra Singh Lilhore
HADOOP-16744 | Fix building instruction to enable zstd | Minor | documentation | Masatake Iwasaki | Masatake Iwasaki
HDFS-14869 | Data loss in case of distcp using snapshot diff. Replication should include rename records if file was skipped in the previous iteration | Major | distcp | Aasha Medhi | Aasha Medhi
HADOOP-16754 | Fix docker failed to build yetus/hadoop | Blocker | build | Kevin Su | Kevin Su
HDFS-15032 | Balancer crashes when it fails to contact an unavailable NN via ObserverReadProxyProvider | Major | balancer & mover | Erik Krogen | Erik Krogen
HDFS-15036 | Active NameNode should not silently fail the image transfer | Major | namenode | Konstantin Shvachko | Chen Liang
HDFS-14519 | NameQuota is not update after concat operation, so namequota is wrong | Major | . | Ranith Sardar | Ranith Sardar
YARN-10055 | bower install fails | Blocker | build, yarn-ui-v2 | Akira Ajisaka | Akira Ajisaka
HDFS-15076 | Fix tests that hold FSDirectory lock, without holding FSNamesystem lock. | Major | test | Konstantin Shvachko | Konstantin Shvachko
HDFS-15073 | Replace curator-shaded guava import with the standard one | Minor | hdfs-client | Akira Ajisaka | Chandra Sanivarapu
HADOOP-16042 | Update the link to HadoopJavaVersion | Minor | documentation | Akira Ajisaka | Chandra Sanivarapu
HDFS-14934 | [SBN Read] Standby NN throws many InterruptedExceptions when dfs.ha.tail-edits.period is 0 | Major | . | Takanobu Asanuma | Ayush Saxena
YARN-10053 | Placement rules do not use correct group service init | Major | yarn | Wilfred Spiegelenburg | Wilfred Spiegelenburg
HDFS-15068 | DataNode could meet deadlock if invoke refreshVolumes when register | Major | datanode | Xiaoqiao He | Aiphago
MAPREDUCE-7255 | Fix typo in MapReduce documentaion example | Trivial | documentation | Sergey Pogorelov | Sergey Pogorelov
HDFS-15072 | HDFS MiniCluster fails to start when run in directory path with a % | Minor | . | Geoffrey Jacoby | Masatake Iwasaki
HDFS-15077 | Fix intermittent failure of TestDFSClientRetries#testLeaseRenewSocketTimeout | Minor | test | Masatake Iwasaki | Masatake Iwasaki
HDFS-15080 | Fix the issue in reading persistent memory cached data with an offset | Major | caching, datanode | Feilong He | Feilong He
YARN-7387 | org.apache.hadoop.yarn.server.resourcemanager.scheduler.capacity.TestIncreaseAllocationExpirer fails intermittently | Major | . | Miklos Szegedi | Jim Brennan
YARN-8672 | TestContainerManager#testLocalingResourceWhileContainerRunning occasionally times out | Major | nodemanager | Jason Darrell Lowe | Chandni Singh
HDFS-14957 | INodeReference Space Consumed was not same in QuotaUsage and ContentSummary | Major | namenode | Hemanth Boyina | Hemanth Boyina
MAPREDUCE-7252 | Handling 0 progress in SimpleExponential task runtime estimator | Minor | . | Ahmed Hussein | Ahmed Hussein
HADOOP-16749 | Configuration parsing of CDATA values are blank | Major | conf | Jonathan Turner Eagles | Daryn Sharp
HDFS-15095 | Fix accidental comment in flaky test TestDecommissioningStatus | Major | hdfs | Ahmed Hussein | Ahmed Hussein
HDFS-15099 | [SBN Read] checkOperation(WRITE) should throw ObserverRetryOnActiveException on ObserverNode | Major | namenode | Konstantin Shvachko | Chen Liang
HDFS-14578 | AvailableSpaceBlockPlacementPolicy always prefers local node | Major | block placement | Wei-Chiu Chuang | Ayush Saxena
HADOOP-16683 | Disable retry of FailoverOnNetworkExceptionRetry in case of wrapped AccessControlException | Major | common | Adam Antal | Adam Antal
MAPREDUCE-7256 | Fix javadoc error in SimpleExponentialSmoothing | Minor | documentation | Masatake Iwasaki | Masatake Iwasaki
YARN-8373 | RM Received RMFatalEvent of type CRITICAL_THREAD_CRASH | Major | fairscheduler, resourcemanager | Girish Bhat | Wilfred Spiegelenburg
MAPREDUCE-7247 | Modify HistoryServerRest.html content,change The job attempt id‘s datatype from string to int | Major | documentation | zhaoshengjie |
YARN-8148 | Update decimal values for queue capacities shown on queue status CLI | Major | client | Prabhu Joseph | Prabhu Joseph
HADOOP-16808 | Use forkCount and reuseForks parameters instead of forkMode in the config of maven surefire plugin | Minor | build | Akira Ajisaka | Xieming Li
HADOOP-16793 | Remove WARN log when ipc connection interrupted in Client#handleSaslConnectionFailure() | Minor | . | Lisheng Sun | Lisheng Sun
YARN-9790 | Failed to set default-application-lifetime if maximum-application-lifetime is less than or equal to zero | Major | . | kyungwan nam | kyungwan nam
HDFS-14993 | checkDiskError doesn’t work during datanode startup | Major | datanode | Yang Yun | Yang Yun
HDFS-13179 | TestLazyPersistReplicaRecovery#testDnRestartWithSavedReplicas fails intermittently | Critical | fs | Gabor Bota | Ahmed Hussein
MAPREDUCE-7259 | testSpeculateSuccessfulWithUpdateEvents fails Intermittently | Minor | . | Ahmed Hussein | Ahmed Hussein
MAPREDUCE-7079 | JobHistory#ServiceStop implementation is incorrect | Major | . | Jason Darrell Lowe | Ahmed Hussein
HDFS-15118 | [SBN Read] Slow clients when Observer reads are enabled but there are no Observers on the cluster. | Major | hdfs-client | Konstantin Shvachko | Chen Liang
HDFS-7175 | Client-side SocketTimeoutException during Fsck | Major | namenode | Carl Steinbach | Stephen O’Donnell
HDFS-15148 | dfs.namenode.send.qop.enabled should not apply to primary NN port | Major | . | Chen Liang | Chen Liang
HDFS-15115 | Namenode crash caused by NPE in BlockPlacementPolicyDefault when dynamically change logger to debug | Major | . | wangzhixiang | wangzhixiang
HDFS-15157 | Fix compilation failure for branch-3.1 | Major | . | Ayush Saxena | Ayush Saxena
HDFS-15158 | The number of failed volumes mismatch with volumeFailures of Datanode metrics | Minor | datanode | Yang Yun | Yang Yun
HADOOP-16849 | start-build-env.sh behaves incorrectly when username is numeric only | Minor | build | Jihyun Cho | Jihyun Cho
HDFS-15161 | When evictableMmapped or evictable size is zero, do not throw NoSuchElementException in ShortCircuitCache#close() | Major | . | Lisheng Sun | Lisheng Sun
HDFS-15164 | Fix TestDelegationTokensWithHA | Major | . | Ayush Saxena | Ayush Saxena
HADOOP-16868 | ipc.Server readAndProcess threw NullPointerException | Major | rpc-server | Tsz-wo Sze | Tsz-wo Sze
HADOOP-16869 | Upgrade findbugs-maven-plugin to 3.0.5 to fix mvn findbugs:findbugs failure | Major | build | Akira Ajisaka | Akira Ajisaka
HDFS-15052 | WebHDFS getTrashRoot leads to OOM due to FileSystem object creation | Major | webhdfs | Wei-Chiu Chuang | Masatake Iwasaki
HDFS-15185 | StartupProgress reports edits segments until the entire startup completes | Major | namenode | Konstantin Shvachko | Konstantin Shvachko
HDFS-15166 | Remove redundant field fStream in ByteStringLog | Major | . | Konstantin Shvachko | Xieming Li
HADOOP-16841 | The description of hadoop.http.authentication.signature.secret.file contains outdated information | Minor | documentation | Akira Ajisaka | Xieming Li
YARN-10156 | Fix typo ‘complaint’ which means quite different in Federation.md | Minor | documentation, federation | Sungpeo Kook | Sungpeo Kook
HDFS-15147 | LazyPersistTestCase wait logic is error-prone | Minor | . | Ahmed Hussein | Ahmed Hussein
HDFS-14668 | Support Fuse with Users from multiple Security Realms | Critical | fuse-dfs | Sailesh Patel | Istvan Fajth
HDFS-15111 | stopStandbyServices() should log which service state it is transitioning from. | Major | hdfs, logging | Konstantin Shvachko | Xieming Li
HDFS-15199 | NPE in BlockSender | Major | . | Ayush Saxena | Ayush Saxena
HADOOP-16891 | Upgrade jackson-databind to 2.9.10.3 | Blocker | . | Siyao Meng | Siyao Meng
HADOOP-16840 | AliyunOSS: getFileStatus throws FileNotFoundException in versioning bucket | Major | fs/oss | wujinhu | wujinhu
YARN-9427 | TestContainerSchedulerQueuing.testKillOnlyRequiredOpportunisticContainers fails sporadically | Major | scheduler, test | Prabhu Joseph | Ahmed Hussein
HDFS-15216 | Wrong Use Case of -showprogress in fsck | Major | . | Ravuri Sushma sree | Ravuri Sushma sree
HDFS-15211 | EC: File write hangs during close in case of Exception during updatePipeline | Critical | . | Ayush Saxena | Ayush Saxena
HDFS-15208 | Suppress bogus AbstractWadlGeneratorGrammarGenerator in KMS stderr in hdfs | Trivial | . | Wei-Chiu Chuang | Wei-Chiu Chuang
HDFS-15219 | DFS Client will stuck when ResponseProcessor.run throw Error | Major | hdfs-client | zhengchenyu | zhengchenyu
HADOOP-16949 | pylint fails in the build environment | Critical | build | Akira Ajisaka | Akira Ajisaka
HADOOP-14836 | Upgrade maven-clean-plugin to 3.1.0 | Major | build | Allen Wittenauer | Akira Ajisaka
MAPREDUCE-7272 | TaskAttemptListenerImpl excessive log messages | Major | . | Ahmed Hussein | Ahmed Hussein
HDFS-15283 | Cache pool MAXTTL is not persisted and restored on cluster restart | Major | namenode | Stephen O’Donnell | Stephen O’Donnell
HADOOP-16944 | Use Yetus 0.12.0 in GitHub PR | Major | build | Akira Ajisaka | Akira Ajisaka
HDFS-15276 | Concat on INodeRefernce fails with illegal state exception | Critical | . | Hemanth Boyina | Hemanth Boyina
HDFS-15281 | ZKFC ignores dfs.namenode.rpc-bind-host and uses dfs.namenode.rpc-address to bind to host address | Major | ha, namenode | Dhiraj Hegde | Dhiraj Hegde
HDFS-15297 | TestNNHandlesBlockReportPerStorage::blockReport_02 fails intermittently in trunk | Major | datanode, test | Mingliang Liu | Ayush Saxena
HADOOP-17014 | Upgrade jackson-databind to 2.9.10.4 | Blocker | . | Siyao Meng | Siyao Meng
HDFS-15286 | Concat on a same files deleting the file | Critical | . | Hemanth Boyina | Hemanth Boyina
HADOOP-15565 | ViewFileSystem.close doesn’t close child filesystems and causes FileSystem objects leak. | Major | . | Jinglun | Jinglun
HADOOP-17052 | NetUtils.connect() throws unchecked exception (UnresolvedAddressException) causing clients to abort | Major | net | Dhiraj Hegde | Dhiraj Hegde
YARN-10347 | Fix double locking in CapacityScheduler#reinitialize in branch-3.1 | Critical | capacity scheduler | Masatake Iwasaki | Masatake Iwasaki

### TESTS:

JIRA | Summary | Priority | Component | Reporter | Contributor
--- | --- | --- | --- | --- | ---
YARN-10072 | TestCSAllocateCustomResource failures | Major | yarn | Jim Brennan | Jim Brennan
YARN-10161 | TestRouterWebServicesREST is corrupting STDOUT | Minor | yarn | Jim Brennan | Jim Brennan
HADOOP-14206 | TestSFTPFileSystem#testFileExists failure: Invalid encoding for signature | Major | fs, test | John Zhuge | Jim Brennan

### SUB-TASKS:

JIRA | Summary | Priority | Component | Reporter | Contributor
--- | --- | --- | --- | --- | ---
HADOOP-15636 | Add ITestDynamoDBMetadataStore | Minor | fs/s3, test | Sean Mackrory | Gabor Bota
HADOOP-15787 | [JDK11] TestIPC.testRTEDuringConnectionSetup fails | Major | . | Akira Ajisaka | Zsolt Venczel
HDFS-14262 | [SBN read] Unclear Log.WARN message in GlobalStateIdContext | Major | hdfs | Shweta | Shweta
HDFS-14590 | [SBN Read] Add the document link to the top page | Major | documentation | Takanobu Asanuma | Takanobu Asanuma
YARN-9791 | Queue Mutation API does not allow to remove a config | Major | capacity scheduler | Prabhu Joseph | Prabhu Joseph
HDFS-14822 | [SBN read] Revisit GlobalStateIdContext locking when getting server state id | Major | hdfs | Chen Liang | Chen Liang
HDFS-14785 | [SBN read] Change client logging to be less aggressive | Major | hdfs | Chen Liang | Chen Liang
YARN-9864 | Format CS Configuration present in Configuration Store | Major | capacity scheduler | Prabhu Joseph | Prabhu Joseph
YARN-9801 | SchedConfCli does not work with https mode | Major | . | Prabhu Joseph | Prabhu Joseph
HDFS-14858 | [SBN read] Allow configurably enable/disable AlignmentContext on NameNode | Major | hdfs | Chen Liang | Chen Liang
HDFS-12979 | StandbyNode should upload FsImage to ObserverNode after checkpointing. | Major | hdfs | Konstantin Shvachko | Chen Liang
YARN-9873 | Mutation API Config Change need to update Version Number | Major | . | Prabhu Joseph | Prabhu Joseph
HDFS-14162 | Balancer should work with ObserverNode | Major | . | Konstantin Shvachko | Erik Krogen
YARN-9773 | Add QueueMetrics for Custom Resources | Major | . | Manikandan R | Manikandan R
HADOOP-16598 | Backport “HADOOP-16558 [COMMON+HDFS] use protobuf-maven-plugin to generate protobuf classes” to all active branches | Major | common | Duo Zhang | Duo Zhang
YARN-9950 | Unset Ordering Policy of Leaf/Parent queue converted from Parent/Leaf queue respectively | Major | capacity scheduler | Prabhu Joseph | Prabhu Joseph
YARN-9900 | Revert to previous state when Invalid Config is applied and Refresh Support in SchedulerConfig Format | Major | capacity scheduler | Prabhu Joseph | Prabhu Joseph
HADOOP-16610 | Upgrade to yetus 0.11.1 and use emoji vote on github pre commit | Major | build | Duo Zhang | Duo Zhang
YARN-9909 | Offline format of YarnConfigurationStore | Major | capacity scheduler | Prabhu Joseph | Prabhu Joseph
YARN-9836 | General usability improvements in showSimulationTrace.html | Minor | scheduler-load-simulator | Adam Antal | Adam Antal
HADOOP-16758 | Refine testing.md to tell user better how to use auth-keys.xml | Minor | fs/s3 | Mingliang Liu | Mingliang Liu
HADOOP-16609 | Add Jenkinsfile for all active branches | Major | build | Duo Zhang | Akira Ajisaka
YARN-10022 | Create RM Rest API to validate a CapacityScheduler Configuration | Major | . | Kinga Marton | Kinga Marton
HDFS-15173 | RBF: Delete repeated configuration ‘dfs.federation.router.metrics.enable’ | Minor | documentation, rbf | panlijie | panlijie
YARN-10139 | ValidateAndGetSchedulerConfiguration API fails when cluster max allocation > default 8GB | Major | . | Prabhu Joseph | Prabhu Joseph
HDFS-14731 | [FGL] Remove redundant locking on NameNode. | Major | namenode | Konstantin Shvachko | Konstantin Shvachko
HADOOP-14918 | Remove the Local Dynamo DB test option | Major | fs/s3 | Steve Loughran | Gabor Bota
HDFS-15305 | Extend ViewFS and provide ViewFSOverloadScheme implementation with scheme configurable. | Major | fs, hadoop-client, hdfs-client, viewfs | Uma Maheswara Rao G | Uma Maheswara Rao G

### OTHER:

JIRA | Summary | Priority | Component | Reporter | Contributor
--- | --- | --- | --- | --- | ---
YARN-8016 | Refine PlacementRule interface and add a app-name queue mapping rule as an example | Major | . | Zian Chen | Zian Chen
HADOOP-16491 | Upgrade jetty version to 9.3.27 | Major | . | Hrishikesh Gadre | Hrishikesh Gadre
HADOOP-16542 | Update commons-beanutils version to 1.9.4 | Major | . | Wei-Chiu Chuang | Kevin Su
HADOOP-16551 | The changelog*.md seems not generated when create-release | Blocker | . | Zhankun Tang |
YARN-9730 | Support forcing configured partitions to be exclusive based on app node label | Major | . | Jonathan Hung | Jonathan Hung
HDFS-14959 | [SBNN read] access time should be turned off | Major | documentation | Wei-Chiu Chuang | Chao Sun
HADOOP-16784 | Update the year to 2020 | Major | . | Ayush Saxena | Ayush Saxena
HADOOP-16871 | Upgrade Netty version to 4.1.45.Final to handle CVE-2019-20444,CVE-2019-16869 | Major | . | Aray Chenchu Sukesh | Aray Chenchu Sukesh
HADOOP-16982 | Update Netty to 4.1.48.Final | Blocker | . | Wei-Chiu Chuang | Lisheng Sun