Apache Hadoop 3.5.0 – Apache Hadoop Changelog

##### General

- Overview
- Single Node Setup
- Cluster Setup
- Commands Reference
- FileSystem Shell
- Compatibility Specification
- Downstream Developer's Guide
- Admin Compatibility Guide
- Interface Classification
- FileSystem Specification

##### Common

- CLI Mini Cluster
- Fair Call Queue
- Native Libraries
- Proxy User
- Rack Awareness
- Secure Mode
- Service Level Authorization
- HTTP Authentication
- Credential Provider API
- Hadoop KMS
- Tracing
- Unix Shell Guide
- Registry
- Async Profiler

##### HDFS

- Architecture
- User Guide
- Commands Reference
- NameNode HA With QJM
- NameNode HA With NFS
- Observer NameNode
- Federation
- ViewFs
- ViewFsOverloadScheme
- Snapshots
- Edits Viewer
- Image Viewer
- Permissions and HDFS
- Quotas and HDFS
- libhdfs (C API)
- WebHDFS (REST API)
- HttpFS
- Short Circuit Local Reads
- Centralized Cache Management
- NFS Gateway
- Rolling Upgrade
- Extended Attributes
- Transparent Encryption
- Multihoming
- Storage Policies
- Memory Storage Support
- Synthetic Load Generator
- Erasure Coding
- Disk Balancer
- Upgrade Domain
- DataNode Admin
- Router Federation
- Provided Storage

##### MapReduce

- Tutorial
- Commands Reference
- Encrypted Shuffle
- Pluggable Shuffle/Sort
- Distributed Cache Deploy
- Support for YARN Shared Cache

##### MapReduce REST APIs

- MR Application Master
- MR History Server

##### YARN

- Architecture
- Commands Reference
- Capacity Scheduler
- Fair Scheduler
- ResourceManager Restart
- ResourceManager HA
- Resource Model
- Node Labels
- Node Attributes
- Web Application Proxy
- Timeline Server
- Timeline Service V.2
- Writing YARN Applications
- YARN Application Security
- NodeManager
- Running Applications in Docker Containers
- Running Applications in runC Containers
- Using CGroups
- Secure Containers
- Reservation System
- Graceful Decommission
- Opportunistic Containers
- YARN Federation
- Shared Cache
- Using GPU
- Using FPGA
- Placement Constraints
- YARN UI2

##### YARN REST APIs

- Introduction
- Resource Manager
- Node Manager
- Timeline Server
- Timeline Service V.2

##### YARN Service

- Overview
- QuickStart
- Concepts
- Yarn Service API
- Service Discovery
- System Services

##### Hadoop Compatible File Systems

- Aliyun OSS
- Amazon S3
- Azure Blob Storage
- Azure Data Lake Storage
- Tencent COS
- Huaweicloud OBS
- VolcanoEngine TOS

##### Auth

- Overview
- Examples
- Configuration
- Building

##### Tools

- Hadoop Streaming
- Hadoop Archives
- Hadoop Archive Logs
- DistCp
- HDFS Federation Balance
- GridMix
- Rumen
- Resource Estimator Service
- Scheduler Load Simulator
- Hadoop Benchmarking
- Dynamometer

##### Reference

- Changelog and Release Notes
- Java API docs
- Unix Shell API
- Metrics

##### Configuration

- core-default.xml
- hdfs-default.xml
- hdfs-rbf-default.xml
- mapred-default.xml
- yarn-default.xml
- kms-default.xml
- httpfs-default.xml
- Deprecated Properties

# Apache Hadoop Changelog

## Release 3.0.3 - 2018-05-31

### INCOMPATIBLE CHANGES:

JIRA | Summary | Priority | Component | Reporter | Contributor
--- | --- | --- | --- | --- | ---
YARN-7190 | Ensure only NM classpath in 2.x gets TSv2 related hbase jars, not the user classpath | Major | timelineclient, timelinereader, timelineserver | Vrushali C | Varun Saxena
HDFS-13099 | RBF: Use the ZooKeeper as the default State Store | Minor | documentation | Yiqun Lin | Yiqun Lin

### NEW FEATURES:

JIRA | Summary | Priority | Component | Reporter | Contributor
--- | --- | --- | --- | --- | ---
HDFS-13283 | Percentage based Reserved Space Calculation for DataNode | Major | datanode, hdfs | Lukas Majercak | Lukas Majercak

### IMPROVEMENTS:

JIRA | Summary | Priority | Component | Reporter | Contributor
--- | --- | --- | --- | --- | ---
HDFS-12455 | WebHDFS - Adding “snapshot enabled” status to ListStatus query result. | Major | snapshots, webhdfs | Ajay Kumar | Ajay Kumar
HDFS-13062 | Provide support for JN to use separate journal disk per namespace | Major | federation, journal-node | Bharat Viswanadham | Bharat Viswanadham
HDFS-12933 | Improve logging when DFSStripedOutputStream failed to write some blocks | Minor | erasure-coding | Xiao Chen | chencan
HADOOP-13972 | ADLS to support per-store configuration | Major | fs/adl | John Zhuge | Sharad Sonker
YARN-7813 | Capacity Scheduler Intra-queue Preemption should be configurable for each queue | Major | capacity scheduler, scheduler preemption | Eric Payne | Eric Payne
HDFS-13175 | Add more information for checking argument in DiskBalancerVolume | Minor | diskbalancer | Lei (Eddy) Xu | Lei (Eddy) Xu
HDFS-11187 | Optimize disk access for last partial chunk checksum of Finalized replica | Major | datanode | Wei-Chiu Chuang | Gabor Bota
MAPREDUCE-7061 | SingleCluster setup document needs to be updated | Major | . | Bharat Viswanadham | Bharat Viswanadham
HADOOP-15263 | hadoop cloud-storage module to mark hadoop-common as provided; add azure-datalake | Minor | build | Steve Loughran | Steve Loughran
MAPREDUCE-7060 | Cherry Pick PathOutputCommitter class/factory to branch-3.0 | Minor | . | Steve Loughran | Steve Loughran
HADOOP-15279 | increase maven heap size recommendations | Minor | build, documentation, test | Allen Wittenauer | Allen Wittenauer
HDFS-13170 | Port webhdfs unmaskedpermission parameter to HTTPFS | Major | . | Stephen O’Donnell | Stephen O’Donnell
HDFS-13225 | StripeReader#checkMissingBlocks() ’s IOException info is incomplete | Major | erasure-coding, hdfs-client | lufei | lufei
HDFS-11394 | Support for getting erasure coding policy through WebHDFS#FileStatus | Major | erasure-coding, namenode | Kai Sasaki | Kai Sasaki
HADOOP-15311 | HttpServer2 needs a way to configure the acceptor/selector count | Major | common | Erik Krogen | Erik Krogen
HDFS-11600 | Refactor TestDFSStripedOutputStreamWithFailure test classes | Minor | erasure-coding, test | Andrew Wang | Sammi Chen
HDFS-12884 | BlockUnderConstructionFeature.truncateBlock should be of type BlockInfo | Major | namenode | Konstantin Shvachko | chencan
HADOOP-15334 | Upgrade Maven surefire plugin | Major | build | Arpit Agarwal | Arpit Agarwal
HADOOP-15312 | Undocumented KeyProvider configuration keys | Major | . | Wei-Chiu Chuang | LiXin Ge
YARN-7623 | Fix the CapacityScheduler Queue configuration documentation | Major | . | Arun Suresh | Jonathan Hung
HDFS-13314 | NameNode should optionally exit if it detects FsImage corruption | Major | namenode | Arpit Agarwal | Arpit Agarwal
HADOOP-15342 | Update ADLS connector to use the current SDK version (2.2.7) | Major | fs/adl | Atul Sikaria | Atul Sikaria
HDFS-13462 | Add BIND_HOST configuration for JournalNode’s HTTP and RPC Servers | Major | hdfs, journal-node | Lukas Majercak | Lukas Majercak
HADOOP-14841 | Kms client should disconnect if unable to get output stream from connection. | Major | kms | Xiao Chen | Rushabh S Shah
HDFS-12981 | renameSnapshot a Non-Existent snapshot to itself should throw error | Minor | hdfs | Sailesh Patel | Kitti Nanasi
YARN-8201 | Skip stacktrace of few exception from ClientRMService | Minor | . | Bibin A Chundatt | Bilwa S T
HADOOP-15441 | Log kms url and token service at debug level. | Minor | . | Wei-Chiu Chuang | Gabor Bota
HDFS-13544 | Improve logging for JournalNode in federated cluster | Major | federation, hdfs | Hanisha Koneru | Hanisha Koneru
HADOOP-15486 | Make NetworkTopology#netLock fair | Major | net | Nanda kumar | Nanda kumar
HDFS-13493 | Reduce the HttpServer2 thread count on DataNodes | Major | datanode | Erik Krogen | Erik Krogen
HADOOP-15449 | Increase default timeout of ZK session to avoid frequent NameNode failover | Critical | common | Karthik Palanisamy | Karthik Palanisamy

### BUG FIXES:

JIRA | Summary | Priority | Component | Reporter | Contributor
--- | --- | --- | --- | --- | ---
HDFS-11968 | ViewFS: StoragePolicies commands fail with HDFS federation | Major | hdfs | Mukul Kumar Singh | Mukul Kumar Singh
HDFS-12813 | RequestHedgingProxyProvider can hide Exception thrown from the Namenode for proxy size of 1 | Major | ha | Mukul Kumar Singh | Mukul Kumar Singh
HDFS-13048 | LowRedundancyReplicatedBlocks metric can be negative | Major | metrics | Akira Ajisaka | Akira Ajisaka
HDFS-13115 | In getNumUnderConstructionBlocks(), ignore the inodeIds for which the inodes have been deleted | Major | . | Yongjun Zhang | Yongjun Zhang
HDFS-10453 | ReplicationMonitor thread could stuck for long time due to the race between replication and delete of same file in a large cluster. | Major | namenode | He Xiaoqiao | He Xiaoqiao
MAPREDUCE-7053 | Timed out tasks can fail to produce thread dump | Major | . | Jason Lowe | Jason Lowe
HADOOP-15206 | BZip2 drops and duplicates records when input split size is small | Major | . | Aki Tanaka | Aki Tanaka
YARN-7937 | Fix http method name in Cluster Application Timeout Update API example request | Minor | docs, documentation | Charan Hebri | Charan Hebri
YARN-7947 | Capacity Scheduler intra-queue preemption can NPE for non-schedulable apps | Major | capacity scheduler, scheduler preemption | Eric Payne | Eric Payne
HADOOP-10571 | Use Log.*(Object, Throwable) overload to log exceptions | Major | . | Arpit Agarwal | Andras Bokor
HDFS-12781 | After Datanode down, In Namenode UI Datanode tab is throwing warning message. | Major | datanode | Harshakiran Reddy | Brahma Reddy Battula
HDFS-12070 | Failed block recovery leaves files open indefinitely and at risk for data loss | Major | . | Daryn Sharp | Kihwal Lee
HDFS-13145 | SBN crash when transition to ANN with in-progress edit tailing enabled | Major | ha, namenode | Chao Sun | Chao Sun
HDFS-13114 | CryptoAdmin#ReencryptZoneCommand should resolve Namespace info from path | Major | encryption, hdfs | Hanisha Koneru | Hanisha Koneru
HDFS-13081 | Datanode#checkSecureConfig should allow SASL and privileged HTTP | Major | datanode, security | Xiaoyu Yao | Ajay Kumar
MAPREDUCE-7059 | Downward Compatibility issue: MR job fails because of unknown setErasureCodingPolicy method from 3.x client to HDFS 2.x cluster | Critical | job submission | Jiandan Yang | Jiandan Yang
HADOOP-15275 | Incorrect javadoc for return type of RetryPolicy#shouldRetry | Minor | documentation | Nanda kumar | Nanda kumar
YARN-7511 | NPE in ContainerLocalizer when localization failed for running container | Major | nodemanager | Tao Yang | Tao Yang
MAPREDUCE-7023 | TestHadoopArchiveLogs.testCheckFilesAndSeedApps fails on rerun | Minor | test | Gergely Novák | Gergely Novák
HDFS-13040 | Kerberized inotify client fails despite kinit properly | Major | namenode | Wei-Chiu Chuang | Xiao Chen
YARN-7736 | Fix itemization in YARN federation document | Minor | documentation | Akira Ajisaka | Sen Zhao
HDFS-13164 | File not closed if streamer fail with DSQuotaExceededException | Major | hdfs-client | Xiao Chen | Xiao Chen
HADOOP-15289 | FileStatus.readFields() assertion incorrect | Critical | . | Steve Loughran | Steve Loughran
HDFS-13109 | Support fully qualified hdfs path in EZ commands | Major | hdfs | Hanisha Koneru | Hanisha Koneru
HADOOP-15296 | Fix a wrong link for RBF in the top page | Minor | documentation | Takanobu Asanuma | Takanobu Asanuma
HADOOP-15273 | distcp can’t handle remote stores with different checksum algorithms | Critical | tools/distcp | Steve Loughran | Steve Loughran
MAPREDUCE-6930 | mapreduce.map.cpu.vcores and mapreduce.reduce.cpu.vcores are both present twice in mapred-default.xml | Major | mrv2 | Daniel Templeton | Sen Zhao
HDFS-13190 | Document WebHDFS support for snapshot diff | Major | documentation, webhdfs | Xiaoyu Yao | Lokesh Jain
HDFS-13244 | Add stack, conf, metrics links to utilities dropdown in NN webUI | Major | . | Bharat Viswanadham | Bharat Viswanadham
HDFS-12156 | TestFSImage fails without -Pnative | Major | test | Akira Ajisaka | Akira Ajisaka
HDFS-13239 | Fix non-empty dir warning message when setting default EC policy | Minor | . | Hanisha Koneru | Bharat Viswanadham
YARN-8022 | ResourceManager UI cluster/app/<app-id> page fails to render | Blocker | webapp | Tarun Parimi | Tarun Parimi
MAPREDUCE-7064 | Flaky test TestTaskAttempt#testReducerCustomResourceTypes | Major | client, test | Peter Bacsko | Peter Bacsko
HDFS-12723 | TestReadStripedFileWithMissingBlocks#testReadFileWithMissingBlocks failing consistently. | Major | . | Rushabh S Shah | Ajay Kumar
YARN-7636 | Re-reservation count may overflow when cluster resource exhausted for a long time | Major | capacityscheduler | Tao Yang | Tao Yang
HDFS-12886 | Ignore minReplication for block recovery | Major | hdfs, namenode | Lukas Majercak | Lukas Majercak
HDFS-13296 | GenericTestUtils generates paths with drive letter in Windows and fail webhdfs related test cases | Major | . | Xiao Liang | Xiao Liang
HDFS-13268 | TestWebHdfsFileContextMainOperations fails on Windows | Major | . | Íñigo Goiri | Xiao Liang
YARN-8054 | Improve robustness of the LocalDirsHandlerService MonitoringTimerTask thread | Major | . | Jonathan Eagles | Jonathan Eagles
HDFS-13195 | DataNode conf page cannot display the current value after reconfig | Minor | datanode | maobaolong | maobaolong
YARN-8063 | DistributedShellTimelinePlugin wrongly check for entityId instead of entityType | Major | . | Rohith Sharma K S | Rohith Sharma K S
YARN-8062 | yarn rmadmin -getGroups returns group from which the user has been removed | Critical | . | Sumana Sathish | Sunil Govindan
YARN-8068 | Application Priority field causes NPE in app timeline publish when Hadoop 2.7 based clients to 2.8+ | Blocker | yarn | Sunil Govindan | Sunil Govindan
YARN-7734 | YARN-5418 breaks TestContainerLogsPage.testContainerLogPageAccess | Major | . | Miklos Szegedi | Tao Yang
HDFS-13087 | Snapshotted encryption zone information should be immutable | Major | encryption | LiXin Ge | LiXin Ge
HADOOP-12862 | LDAP Group Mapping over SSL can not specify trust store | Major | . | Wei-Chiu Chuang | Wei-Chiu Chuang
HDFS-13349 | Unresolved merge conflict in ViewFs.md | Blocker | documentation | Gera Shegalov | Yiqun Lin
HADOOP-15317 | Improve NetworkTopology chooseRandom’s loop | Major | . | Xiao Chen | Xiao Chen
HADOOP-15355 | TestCommonConfigurationFields is broken by HADOOP-15312 | Major | test | Konstantin Shvachko | LiXin Ge
HDFS-13350 | Negative legacy block ID will confuse Erasure Coding to be considered as striped block | Major | erasure-coding | Lei (Eddy) Xu | Lei (Eddy) Xu
YARN-7905 | Parent directory permission incorrect during public localization | Critical | . | Bibin A Chundatt | Bilwa S T
HDFS-13420 | License header is displayed in ArchivalStorage/MemoryStorage html pages | Minor | documentation | Akira Ajisaka | Akira Ajisaka
HDFS-13328 | Abstract ReencryptionHandler recursive logic in separate class. | Major | namenode | Surendra Singh Lilhore | Surendra Singh Lilhore
HADOOP-15357 | Configuration.getPropsWithPrefix no longer does variable substitution | Major | . | Jim Brennan | Jim Brennan
MAPREDUCE-7062 | Update mapreduce.job.tags description for making use for ATSv2 purpose. | Major | . | Charan Hebri | Charan Hebri
YARN-8073 | TimelineClientImpl doesn’t honor yarn.timeline-service.versions configuration | Major | . | Rohith Sharma K S | Rohith Sharma K S
HDFS-13427 | Fix the section titles of transparent encryption document | Minor | documentation | Akira Ajisaka | Akira Ajisaka
YARN-7527 | Over-allocate node resource in async-scheduling mode of CapacityScheduler | Major | capacityscheduler | Tao Yang | Tao Yang
YARN-8120 | JVM can crash with SIGSEGV when exiting due to custom leveldb logger | Major | nodemanager, resourcemanager | Jason Lowe | Jason Lowe
YARN-8147 | TestClientRMService#testGetApplications sporadically fails | Major | test | Jason Lowe | Jason Lowe
HDFS-13436 | Fix javadoc of package-info.java | Major | documentation | Akira Ajisaka | Akira Ajisaka
HADOOP-14970 | MiniHadoopClusterManager doesn’t respect lack of format option | Minor | . | Erik Krogen | Erik Krogen
HDFS-13330 | ShortCircuitCache#fetchOrCreate never retries | Major | . | Wei-Chiu Chuang | Gabor Bota
YARN-8156 | Increase the default value of yarn.timeline-service.app-collector.linger-period.ms | Major | . | Rohith Sharma K S | Charan Hebri
YARN-8165 | Incorrect queue name logging in AbstractContainerAllocator | Trivial | capacityscheduler | Weiwei Yang | Weiwei Yang
HDFS-12828 | OIV ReverseXML Processor fails with escaped characters | Critical | hdfs | Erik Krogen | Erik Krogen
HADOOP-15396 | Some java source files are executable | Minor | . | Akira Ajisaka | Shashikant Banerjee
YARN-6827 | [ATS1/1.5] NPE exception while publishing recovering applications into ATS during RM restart. | Major | resourcemanager | Rohith Sharma K S | Rohith Sharma K S
YARN-7786 | NullPointerException while launching ApplicationMaster | Major | . | lujie | lujie
HDFS-10183 | Prevent race condition during class initialization | Minor | fs | Pavel Avgustinov | Pavel Avgustinov
HDFS-13388 | RequestHedgingProxyProvider calls multiple configured NNs all the time | Major | hdfs-client | Jinglun | Jinglun
HDFS-13433 | webhdfs requests can be routed incorrectly in federated cluster | Critical | webhdfs | Arpit Agarwal | Arpit Agarwal
HDFS-13408 | MiniDFSCluster to support being built on randomized base directory | Major | test | Xiao Liang | Xiao Liang
HADOOP-15390 | Yarn RM logs flooded by DelegationTokenRenewer trying to renew KMS tokens | Critical | . | Xiao Chen | Xiao Chen
HDFS-13336 | Test cases of TestWriteToReplica failed in windows | Major | . | Xiao Liang | Xiao Liang
YARN-7598 | Document how to use classpath isolation for aux-services in YARN | Major | . | Xuan Gong | Xuan Gong
YARN-8183 | Fix ConcurrentModificationException inside RMAppAttemptMetrics#convertAtomicLongMaptoLongMap | Critical | yarn | Sumana Sathish | Suma Shivaprasad
HADOOP-15411 | AuthenticationFilter should use Configuration.getPropsWithPrefix instead of iterator | Critical | . | Suma Shivaprasad | Suma Shivaprasad
MAPREDUCE-7042 | Killed MR job data does not move to mapreduce.jobhistory.done-dir when ATS v2 is enabled | Major | . | Yesha Vora | Xuan Gong
YARN-8205 | Application State is not updated to ATS if AM launching is delayed. | Critical | . | Sumana Sathish | Rohith Sharma K S
YARN-8004 | Add unit tests for inter queue preemption for dominant resource calculator | Critical | yarn | Sumana Sathish | Zian Chen
YARN-8221 | RMWebServices also need to honor yarn.resourcemanager.display.per-user-apps | Major | webapp | Sunil Govindan | Sunil Govindan
YARN-8210 | AMRMClient logging on every heartbeat to track updation of AM RM token causes too many log lines to be generated in AM logs | Major | yarn | Suma Shivaprasad | Suma Shivaprasad
HDFS-13509 | Bug fix for breakHardlinks() of ReplicaInfo/LocalReplica, and fix TestFileAppend failures on Windows | Major | . | Xiao Liang | Xiao Liang
MAPREDUCE-7073 | Optimize TokenCache#obtainTokensForNamenodesInternal | Major | . | Bibin A Chundatt | Bibin A Chundatt
HADOOP-15406 | hadoop-nfs dependencies for mockito and junit are not test scope | Major | nfs | Jason Lowe | Jason Lowe
YARN-6385 | Fix checkstyle warnings in TestFileSystemApplicationHistoryStore | Minor | . | Yiqun Lin | Yiqun Lin
YARN-8222 | Fix potential NPE when gets RMApp from RM context | Critical | . | Tao Yang | Tao Yang
HDFS-13481 | TestRollingFileSystemSinkWithHdfs#testFlushThread: test failed intermittently | Major | hdfs | Gabor Bota | Gabor Bota
YARN-8217 | RmAuthenticationFilterInitializer /TimelineAuthenticationFilterInitializer should use Configuration.getPropsWithPrefix instead of iterator | Major | . | Suma Shivaprasad | Suma Shivaprasad
YARN-8025 | UsersManangers#getComputedResourceLimitForActiveUsers throws NPE due to preComputedActiveUserLimit is empty | Major | yarn | Jiandan Yang | Tao Yang
YARN-8232 | RMContainer lost queue name when RM HA happens | Major | resourcemanager | Hu Ziqian | Hu Ziqian
HDFS-13136 | Avoid taking FSN lock while doing group member lookup for FSD permission check | Major | namenode | Xiaoyu Yao | Xiaoyu Yao
HDFS-13537 | TestHdfsHelper does not generate jceks path properly for relative path in Windows | Major | . | Xiao Liang | Xiao Liang
YARN-7003 | DRAINING state of queues is not recovered after RM restart | Major | capacityscheduler | Tao Yang | Tao Yang
YARN-8244 | TestContainerSchedulerQueuing.testStartMultipleContainers failed | Major | . | Miklos Szegedi | Jim Brennan
YARN-8288 | Fix wrong number of table columns in Resource Model doc | Major | . | Weiwei Yang | Weiwei Yang
HDFS-13539 | DFSStripedInputStream NPE when reportCheckSumFailure | Major | . | Xiao Chen | Xiao Chen
YARN-8278 | DistributedScheduling is not working in HA | Blocker | . | Bibin A Chundatt | Bibin A Chundatt
HDFS-13581 | DN UI logs link is broken when https is enabled | Minor | datanode | Namit Maheshwari | Shashikant Banerjee
HDFS-13586 | Fsync fails on directories on Windows | Critical | datanode, hdfs | Lukas Majercak | Lukas Majercak
YARN-8179 | Preemption does not happen due to natural_termination_factor when DRF is used | Major | . | kyungwan nam | kyungwan nam
HADOOP-15450 | Avoid fsync storm triggered by DiskChecker and handle disk full situation | Blocker | . | Kihwal Lee | Arpit Agarwal
HDFS-13601 | Optimize ByteString conversions in PBHelper | Major | . | Andrew Wang | Andrew Wang
HDFS-13540 | DFSStripedInputStream should only allocate new buffers when reading | Major | . | Xiao Chen | Xiao Chen
HDFS-13588 | Fix TestFsDatasetImpl test failures on Windows | Major | . | Xiao Liang | Xiao Liang
YARN-8310 | Handle old NMTokenIdentifier, AMRMTokenIdentifier, and ContainerTokenIdentifier formats | Major | . | Robert Kanter | Robert Kanter
YARN-8344 | Missing nm.stop() in TestNodeManagerResync to fix testKillContainersOnResync | Major | . | Giovanni Matteo Fumarola | Giovanni Matteo Fumarola
YARN-8327 | Fix TestAggregatedLogFormat#testReadAcontainerLogs1 on Windows | Major | log-aggregation | Giovanni Matteo Fumarola | Giovanni Matteo Fumarola
YARN-8346 | Upgrading to 3.1 kills running containers with error “Opportunistic container queue is full” | Blocker | . | Rohith Sharma K S | Jason Lowe
HDFS-13611 | Unsafe use of Text as a ConcurrentHashMap key in PBHelperClient | Major | . | Andrew Wang | Andrew Wang
HDFS-13618 | Fix TestDataNodeFaultInjector test failures on Windows | Major | test | Xiao Liang | Xiao Liang
HADOOP-15473 | Configure serialFilter in KeyProvider to avoid UnrecoverableKeyException caused by JDK-8189997 | Critical | kms | Gabor Bota | Gabor Bota
YARN-8338 | TimelineService V1.5 doesn’t come up after HADOOP-15406 | Critical | . | Vinod Kumar Vavilapalli | Vinod Kumar Vavilapalli
HADOOP-9747 | Reduce unnecessary UGI synchronization | Critical | security | Daryn Sharp | Daryn Sharp
YARN-8518 | test-container-executor test_is_empty() is broken | Major | . | Jim Brennan | Jim Brennan

### TESTS:

JIRA | Summary | Priority | Component | Reporter | Contributor
--- | --- | --- | --- | --- | ---
HADOOP-15313 | TestKMS should close providers | Major | kms, test | Xiao Chen | Xiao Chen
HDFS-13503 | Fix TestFsck test failures on Windows | Major | hdfs | Xiao Liang | Xiao Liang
HDFS-13542 | TestBlockManager#testNeededReplicationWhileAppending fails due to improper cluster shutdown in TestBlockManager#testBlockManagerMachinesArray on Windows | Minor | . | Anbang Hu | Anbang Hu
HDFS-13551 | TestMiniDFSCluster#testClusterSetStorageCapacity does not shut down cluster | Minor | . | Anbang Hu | Anbang Hu
HDFS-11700 | TestHDFSServerPorts#testBackupNodePorts doesn’t pass on Windows | Minor | . | Anbang Hu | Anbang Hu
HDFS-13548 | TestResolveHdfsSymlink#testFcResolveAfs fails on Windows | Minor | . | Anbang Hu | Anbang Hu
HDFS-13567 | TestNameNodeMetrics#testGenerateEDEKTime,TestNameNodeMetrics#testResourceCheck should use a different cluster basedir | Minor | . | Anbang Hu | Anbang Hu
HDFS-13557 | TestDFSAdmin#testListOpenFiles fails on Windows | Minor | . | Anbang Hu | Anbang Hu
HDFS-13550 | TestDebugAdmin#testComputeMetaCommand fails on Windows | Minor | . | Anbang Hu | Anbang Hu
HDFS-13559 | TestBlockScanner does not close TestContext properly | Minor | . | Anbang Hu | Anbang Hu
HDFS-13570 | TestQuotaByStorageType,TestQuota,TestDFSOutputStream fail on Windows | Minor | . | Anbang Hu | Anbang Hu
HDFS-13558 | TestDatanodeHttpXFrame does not shut down cluster | Minor | . | Anbang Hu | Anbang Hu
HDFS-13554 | TestDatanodeRegistration#testForcedRegistration does not shut down cluster | Minor | . | Anbang Hu | Anbang Hu
HDFS-13556 | TestNestedEncryptionZones does not shut down cluster | Minor | . | Anbang Hu | Anbang Hu
HDFS-13560 | Insufficient system resources exist to complete the requested service for some tests on Windows | Minor | . | Anbang Hu | Anbang Hu
HDFS-13592 | TestNameNodePrunesMissingStorages#testNameNodePrunesUnreportedStorages does not shut down cluster properly | Minor | . | Anbang Hu | Anbang Hu
HDFS-13593 | TestBlockReaderLocalLegacy#testBlockReaderLocalLegacyWithAppend fails on Windows | Minor | test | Anbang Hu | Anbang Hu
HDFS-13587 | TestQuorumJournalManager fails on Windows | Minor | . | Anbang Hu | Anbang Hu
HDFS-13619 | TestAuditLoggerWithCommands fails on Windows | Minor | test | Anbang Hu | Anbang Hu
HDFS-13620 | Randomize the test directory path for TestHDFSFileSystemContract | Minor | . | Anbang Hu | Anbang Hu
HDFS-13591 | TestDFSShell#testSetrepLow fails on Windows | Minor | . | Anbang Hu | Anbang Hu

### SUB-TASKS:

JIRA | Summary | Priority | Component | Reporter | Contributor
--- | --- | --- | --- | --- | ---
HDFS-13068 | RBF: Add router admin option to manage safe mode | Major | . | Íñigo Goiri | Yiqun Lin
HADOOP-15040 | Upgrade AWS SDK to 1.11.271: NPE bug spams logs w/ Yarn Log Aggregation | Blocker | fs/s3 | Aaron Fabbri | Aaron Fabbri
HDFS-13119 | RBF: Manage unavailable clusters | Major | . | Íñigo Goiri | Yiqun Lin
HADOOP-15247 | Move commons-net up to 3.6 | Minor | fs | Steve Loughran | Steve Loughran
HDFS-13187 | RBF: Fix Routers information shown in the web UI | Minor | . | Wei Yan | Wei Yan
HDFS-13184 | RBF: Improve the unit test TestRouterRPCClientRetries | Minor | test | Yiqun Lin | Yiqun Lin
HDFS-13199 | RBF: Fix the hdfs router page missing label icon issue | Major | federation, hdfs | maobaolong | maobaolong
HADOOP-15264 | AWS “shaded” SDK 1.11.271 is pulling in netty 4.1.17 | Blocker | fs/s3 | Steve Loughran | Steve Loughran
HADOOP-15090 | Add ADL troubleshooting doc | Major | documentation, fs/adl | Steve Loughran | Steve Loughran
HDFS-13214 | RBF: Complete document of Router configuration | Major | . | Tao Jie | Yiqun Lin
HADOOP-15267 | S3A multipart upload fails when SSE-C encryption is enabled | Critical | fs/s3 | Anis Elleuch | Anis Elleuch
HDFS-13230 | RBF: ConnectionManager’s cleanup task will compare each pool’s own active conns with its total conns | Minor | . | Wei Yan | Chao Sun
HDFS-13233 | RBF: MountTableResolver doesn’t return the correct mount point of the given path | Major | hdfs | wangzhiyuan | wangzhiyuan
HADOOP-15277 | remove .FluentPropertyBeanIntrospector from CLI operation log output | Minor | conf | Steve Loughran | Steve Loughran
HDFS-13212 | RBF: Fix router location cache issue | Major | federation, hdfs | Weiwei Wu | Weiwei Wu
HDFS-13232 | RBF: ConnectionPool should return first usable connection | Minor | . | Wei Yan | Ekanth Sethuramalingam
HDFS-13240 | RBF: Update some inaccurate document descriptions | Minor | . | Yiqun Lin | Yiqun Lin
HDFS-11399 | Many tests fails in Windows due to injecting disk failures | Major | . | Yiqun Lin | Yiqun Lin
HDFS-13241 | RBF: TestRouterSafemode failed if the port 8888 is in use | Major | hdfs, test | maobaolong | maobaolong
HDFS-13253 | RBF: Quota management incorrect parent-child relationship judgement | Major | . | Yiqun Lin | Yiqun Lin
HDFS-13226 | RBF: Throw the exception if mount table entry validated failed | Major | hdfs | maobaolong | maobaolong
HDFS-12505 | Extend TestFileStatusWithECPolicy with a random EC policy | Major | erasure-coding, test | Takanobu Asanuma | Takanobu Asanuma
HDFS-12587 | Use Parameterized tests in TestBlockInfoStriped and TestLowRedundancyBlockQueues to test all EC policies | Major | erasure-coding, test | Takanobu Asanuma | Takanobu Asanuma
HADOOP-15308 | TestConfiguration fails on Windows because of paths | Major | test | Íñigo Goiri | Xiao Liang
HDFS-12773 | RBF: Improve State Store FS implementation | Major | . | Íñigo Goiri | Íñigo Goiri
HDFS-13198 | RBF: RouterHeartbeatService throws out CachedStateStore related exceptions when starting router | Minor | . | Wei Yan | Wei Yan
HDFS-13224 | RBF: Resolvers to support mount points across multiple subclusters | Major | . | Íñigo Goiri | Íñigo Goiri
HADOOP-15262 | AliyunOSS: move files under a directory in parallel when rename a directory | Major | fs/oss | wujinhu | wujinhu
HDFS-13215 | RBF: Move Router to its own module | Major | . | Íñigo Goiri | Wei Yan
HDFS-13250 | RBF: Router to manage requests across multiple subclusters | Major | . | Íñigo Goiri | Íñigo Goiri
HDFS-13318 | RBF: Fix FindBugs in hadoop-hdfs-rbf | Minor | . | Íñigo Goiri | Ekanth Sethuramalingam
HDFS-12792 | RBF: Test Router-based federation using HDFSContract | Major | . | Íñigo Goiri | Íñigo Goiri
YARN-7986 | ATSv2 REST API queries do not return results for uppercase application tags | Critical | . | Charan Hebri | Charan Hebri
HDFS-12512 | RBF: Add WebHDFS | Major | fs | Íñigo Goiri | Wei Yan
HDFS-13291 | RBF: Implement available space based OrderResolver | Major | . | Yiqun Lin | Yiqun Lin
HDFS-13204 | RBF: Optimize name service safe mode icon | Minor | . | liuhongtong | liuhongtong
HDFS-13352 | RBF: Add xsl stylesheet for hdfs-rbf-default.xml | Major | documentation | Takanobu Asanuma | Takanobu Asanuma
HDFS-13347 | RBF: Cache datanode reports | Minor | . | Íñigo Goiri | Íñigo Goiri
HDFS-13289 | RBF: TestConnectionManager#testCleanup() test case need correction | Minor | . | Dibyendu Karmakar | Dibyendu Karmakar
HDFS-13364 | RBF: Support NamenodeProtocol in the Router | Major | . | Íñigo Goiri | Íñigo Goiri
HADOOP-14651 | Update okhttp version to 2.7.5 | Major | fs/adl | Ray Chiang | Ray Chiang
YARN-6936 | [Atsv2] Retrospect storing entities into sub application table from client perspective | Major | . | Rohith Sharma K S | Rohith Sharma K S
HDFS-13353 | RBF: TestRouterWebHDFSContractCreate failed | Major | test | Takanobu Asanuma | Takanobu Asanuma
YARN-8107 | Give an informative message when incorrect format is used in ATSv2 filter attributes | Major | ATSv2 | Charan Hebri | Rohith Sharma K S
HDFS-13402 | RBF: Fix java doc for StateStoreFileSystemImpl | Minor | hdfs | Yiran Wu | Yiran Wu
HDFS-13410 | RBF: Support federation with no subclusters | Minor | . | Íñigo Goiri | Íñigo Goiri
HDFS-13384 | RBF: Improve timeout RPC call mechanism | Minor | . | Íñigo Goiri | Íñigo Goiri
HDFS-13045 | RBF: Improve error message returned from subcluster | Minor | . | Wei Yan | Íñigo Goiri
HDFS-13428 | RBF: Remove LinkedList From StateStoreFileImpl.java | Trivial | federation | BELUGA BEHR | BELUGA BEHR
HDFS-13386 | RBF: Wrong date information in list file(-ls) result | Minor | . | Dibyendu Karmakar | Dibyendu Karmakar
YARN-8027 | Setting hostname of docker container breaks for –net=host in docker 1.13 | Major | yarn | Jim Brennan | Jim Brennan
HADOOP-14999 | AliyunOSS: provide one asynchronous multi-part based uploading mechanism | Major | fs/oss | Genmao Yu | Genmao Yu
YARN-7810 | TestDockerContainerRuntime test failures due to UID lookup of a non-existent user | Major | . | Shane Kumpf | Shane Kumpf
HDFS-13435 | RBF: Improve the error loggings for printing the stack trace | Major | . | Yiqun Lin | Yiqun Lin
YARN-7189 | Container-executor doesn’t remove Docker containers that error out early | Major | yarn | Eric Badger | Eric Badger
HDFS-13466 | RBF: Add more router-related information to the UI | Minor | . | Wei Yan | Wei Yan
HDFS-13453 | RBF: getMountPointDates should fetch latest subdir time/date when parent dir is not present but /parent/child dirs are present in mount table | Major | . | Dibyendu Karmakar | Dibyendu Karmakar
HDFS-13478 | RBF: Disabled Nameservice store API | Major | . | Íñigo Goiri | Íñigo Goiri
HDFS-13490 | RBF: Fix setSafeMode in the Router | Major | . | Íñigo Goiri | Íñigo Goiri
HDFS-13484 | RBF: Disable Nameservices from the federation | Major | . | Íñigo Goiri | Íñigo Goiri
HDFS-13326 | RBF: Improve the interfaces to modify and view mount tables | Minor | . | Wei Yan | Gang Li
HDFS-13499 | RBF: Show disabled name services in the UI | Minor | . | Íñigo Goiri | Íñigo Goiri
YARN-8215 | ATS v2 returns invalid YARN_CONTAINER_ALLOCATED_HOST_HTTP_ADDRESS from NM | Critical | ATSv2 | Yesha Vora | Rohith Sharma K S
HDFS-13508 | RBF: Normalize paths (automatically) when adding, updating, removing or listing mount table entries | Minor | . | Ekanth Sethuramalingam | Ekanth Sethuramalingam
HDFS-13434 | RBF: Fix dead links in RBF document | Major | documentation | Akira Ajisaka | Chetna Chaudhari
YARN-8212 | Pending backlog for async allocation threads should be configurable | Major | . | Weiwei Yang | Tao Yang
HDFS-13488 | RBF: Reject requests when a Router is overloaded | Major | . | Íñigo Goiri | Íñigo Goiri
HDFS-13525 | RBF: Add unit test TestStateStoreDisabledNameservice | Major | . | Yiqun Lin | Yiqun Lin
YARN-8253 | HTTPS Ats v2 api call fails with “bad HTTP parsed” | Critical | ATSv2 | Yesha Vora | Charan Hebri
HADOOP-15454 | TestRollingFileSystemSinkWithLocal fails on Windows | Major | test | Xiao Liang | Xiao Liang
YARN-8247 | Incorrect HTTP status code returned by ATSv2 for non-whitelisted users | Critical | ATSv2 | Charan Hebri | Rohith Sharma K S
YARN-8130 | Race condition when container events are published for KILLED applications | Major | ATSv2 | Charan Hebri | Rohith Sharma K S
HADOOP-15498 | TestHadoopArchiveLogs (#testGenerateScript, #testPrepareWorkingDir) fails on Windows | Minor | . | Anbang Hu | Anbang Hu
HADOOP-15497 | TestTrash should use proper test path to avoid failing on Windows | Minor | . | Anbang Hu | Anbang Hu
YARN-4781 | Support intra-queue preemption for fairness ordering policy. | Major | scheduler | Wangda Tan | Eric Payne

### OTHER:

JIRA | Summary | Priority | Component | Reporter | Contributor
--- | --- | --- | --- | --- | ---
HDFS-13052 | WebHDFS: Add support for snasphot diff | Major | . | Lokesh Jain | Lokesh Jain
HADOOP-14742 | Document multi-URI replication Inode for ViewFS | Major | documentation, viewfs | Chris Douglas | Gera Shegalov