Apache Hadoop 3.5.0 – Apache Hadoop Changelog

##### General

- Overview
- Single Node Setup
- Cluster Setup
- Commands Reference
- FileSystem Shell
- Compatibility Specification
- Downstream Developer's Guide
- Admin Compatibility Guide
- Interface Classification
- FileSystem Specification

##### Common

- CLI Mini Cluster
- Fair Call Queue
- Native Libraries
- Proxy User
- Rack Awareness
- Secure Mode
- Service Level Authorization
- HTTP Authentication
- Credential Provider API
- Hadoop KMS
- Tracing
- Unix Shell Guide
- Registry
- Async Profiler

##### HDFS

- Architecture
- User Guide
- Commands Reference
- NameNode HA With QJM
- NameNode HA With NFS
- Observer NameNode
- Federation
- ViewFs
- ViewFsOverloadScheme
- Snapshots
- Edits Viewer
- Image Viewer
- Permissions and HDFS
- Quotas and HDFS
- libhdfs (C API)
- WebHDFS (REST API)
- HttpFS
- Short Circuit Local Reads
- Centralized Cache Management
- NFS Gateway
- Rolling Upgrade
- Extended Attributes
- Transparent Encryption
- Multihoming
- Storage Policies
- Memory Storage Support
- Synthetic Load Generator
- Erasure Coding
- Disk Balancer
- Upgrade Domain
- DataNode Admin
- Router Federation
- Provided Storage

##### MapReduce

- Tutorial
- Commands Reference
- Encrypted Shuffle
- Pluggable Shuffle/Sort
- Distributed Cache Deploy
- Support for YARN Shared Cache

##### MapReduce REST APIs

- MR Application Master
- MR History Server

##### YARN

- Architecture
- Commands Reference
- Capacity Scheduler
- Fair Scheduler
- ResourceManager Restart
- ResourceManager HA
- Resource Model
- Node Labels
- Node Attributes
- Web Application Proxy
- Timeline Server
- Timeline Service V.2
- Writing YARN Applications
- YARN Application Security
- NodeManager
- Running Applications in Docker Containers
- Running Applications in runC Containers
- Using CGroups
- Secure Containers
- Reservation System
- Graceful Decommission
- Opportunistic Containers
- YARN Federation
- Shared Cache
- Using GPU
- Using FPGA
- Placement Constraints
- YARN UI2

##### YARN REST APIs

- Introduction
- Resource Manager
- Node Manager
- Timeline Server
- Timeline Service V.2

##### YARN Service

- Overview
- QuickStart
- Concepts
- Yarn Service API
- Service Discovery
- System Services

##### Hadoop Compatible File Systems

- Aliyun OSS
- Amazon S3
- Azure Blob Storage
- Azure Data Lake Storage
- Tencent COS
- Huaweicloud OBS
- VolcanoEngine TOS

##### Auth

- Overview
- Examples
- Configuration
- Building

##### Tools

- Hadoop Streaming
- Hadoop Archives
- Hadoop Archive Logs
- DistCp
- HDFS Federation Balance
- GridMix
- Rumen
- Resource Estimator Service
- Scheduler Load Simulator
- Hadoop Benchmarking
- Dynamometer

##### Reference

- Changelog and Release Notes
- Java API docs
- Unix Shell API
- Metrics

##### Configuration

- core-default.xml
- hdfs-default.xml
- hdfs-rbf-default.xml
- mapred-default.xml
- yarn-default.xml
- kms-default.xml
- httpfs-default.xml
- Deprecated Properties

# Apache Hadoop Changelog

## Release 2.8.2 - 2017-10-24

### INCOMPATIBLE CHANGES:

JIRA | Summary | Priority | Component | Reporter | Contributor
--- | --- | --- | --- | --- | ---
HADOOP-13119 | Add ability to secure log servlet using proxy users | Major | . | Jeffrey E Rodriguez | Yuanbo Liu
HADOOP-14174 | Set default ADLS access token provider type to ClientCredential | Major | fs/adl | John Zhuge | John Zhuge

### IMPROVEMENTS:

JIRA | Summary | Priority | Component | Reporter | Contributor
--- | --- | --- | --- | --- | ---
HDFS-7764 | DirectoryScanner shouldn’t abort the scan if one directory had an error | Major | datanode | Rakesh R | Rakesh R
HDFS-7541 | Upgrade Domains in HDFS | Major | . | Ming Ma | Ming Ma
HDFS-10683 | Make class Token$PrivateToken private | Minor | . | John Zhuge | John Zhuge
HADOOP-13688 | Stop bundling HTML source code in javadoc JARs | Major | build | Andrew Wang | Andrew Wang
HADOOP-13737 | Cleanup DiskChecker interface | Major | util | Arpit Agarwal | Arpit Agarwal
HADOOP-13496 | Include file lengths in Mismatch in length error for distcp | Minor | . | Ted Yu | Ted Yu
HDFS-10534 | NameNode WebUI should display DataNode usage histogram | Major | namenode, ui | Zhe Zhang | Kai Sasaki
HADOOP-14050 | Add process name to kms process | Minor | kms, scripts | Rushabh S Shah | Rushabh S Shah
HDFS-11390 | Add process name to httpfs process | Minor | httpfs, scripts | John Zhuge | Weiwei Yang
HDFS-11333 | Print a user friendly error message when plugins are not found | Minor | namenode | Wei-Chiu Chuang | Wei-Chiu Chuang
HDFS-11466 | Change dfs.namenode.write-lock-reporting-threshold-ms default from 1000ms to 5000ms | Major | namenode | Andrew Wang | Andrew Wang
HDFS-11432 | Federation : Support fully qualified path for Quota/Snapshot/cacheadmin/cryptoadmin commands | Major | federation | Brahma Reddy Battula | Brahma Reddy Battula
HADOOP-14169 | Implement listStatusIterator, listLocatedStatus for ViewFs | Minor | viewfs | Erik Krogen | Erik Krogen
HADOOP-14233 | Delay construction of PreCondition.check failure message in Configuration#set | Major | . | Jonathan Eagles | Jonathan Eagles
HADOOP-14240 | Configuration#get return value optimization | Major | . | Jonathan Eagles | Jonathan Eagles
YARN-6339 | Improve performance for createAndGetApplicationReport | Major | . | yunjiong zhao | yunjiong zhao
HDFS-9705 | Refine the behaviour of getFileChecksum when length = 0 | Minor | . | Kai Zheng | Sammi Chen
HDFS-11628 | Clarify the behavior of HDFS Mover in documentation | Major | documentation | Xiaobing Zhou | Xiaobing Zhou
HADOOP-14104 | Client should always ask namenode for kms provider path. | Major | kms | Rushabh S Shah | Rushabh S Shah
HADOOP-14276 | Add a nanosecond API to Time/Timer/FakeTimer | Minor | util | Erik Krogen | Erik Krogen
HDFS-11558 | BPServiceActor thread name is too long | Minor | datanode | Tsz Wo Nicholas Sze | Xiaobing Zhou
HDFS-11648 | Lazy construct the IIP pathname | Major | . | Daryn Sharp | Daryn Sharp
HDFS-11634 | Optimize BlockIterator when iterating starts in the middle. | Major | . | Konstantin Shvachko | Konstantin Shvachko
HDFS-11384 | Add option for balancer to disperse getBlocks calls to avoid NameNode’s rpc.CallQueueLength spike | Major | balancer & mover | yunjiong zhao | Konstantin Shvachko
YARN-6457 | Allow custom SSL configuration to be supplied in WebApps | Major | webapp, yarn | Sanjay M Pujare | Sanjay M Pujare
HDFS-11641 | Reduce cost of audit logging by using FileStatus instead of HdfsFileStatus | Major | hdfs | Daryn Sharp | Daryn Sharp
YARN-6493 | Print requested node partition in assignContainer logs | Major | . | Jonathan Hung | Jonathan Hung
HDFS-11891 | DU#refresh should print the path of the directory when an exception is caught | Minor | . | Chen Liang | Chen Liang
HADOOP-14440 | Add metrics for connections dropped | Major | . | Eric Badger | Eric Badger
HDFS-11861 | ipc.Client.Connection#sendRpcRequest should log request name | Trivial | ipc | John Zhuge | John Zhuge
HDFS-11345 | Document the configuration key for FSNamesystem lock fairness | Minor | documentation, namenode | Zhe Zhang | Erik Krogen
YARN-6738 | LevelDBCacheTimelineStore should reuse ObjectMapper instances | Major | timelineserver | Zoltan Haindrich | Zoltan Haindrich
HADOOP-14515 | Specifically configure zookeeper-related log levels in KMS log4j | Major | kms | Xiao Chen | Xiao Chen
HDFS-11881 | NameNode consumes a lot of memory for snapshot diff report generation | Major | hdfs, snapshots | Manoj Govindassamy | Manoj Govindassamy
HDFS-12042 | Lazy initialize AbstractINodeDiffList#diffs for snapshots to reduce memory consumption | Major | . | Misha Dmitriev | Misha Dmitriev
HDFS-12078 | Add time unit to the description of property dfs.namenode.stale.datanode.interval in hdfs-default.xml | Minor | documentation, hdfs | Weiwei Yang | Weiwei Yang
HADOOP-14629 | Improve exception checking in FileContext related JUnit tests | Major | fs, test | Andras Bokor | Andras Bokor
HDFS-12137 | DN dataset lock should be fair | Critical | datanode | Daryn Sharp | Daryn Sharp
HADOOP-14659 | UGI getShortUserName does not need to search the Subject | Major | common | Daryn Sharp | Daryn Sharp
YARN-6768 | Improve performance of yarn api record toString and fromString | Major | . | Jonathan Eagles | Jonathan Eagles
YARN-5892 | Support user-specific minimum user limit percentage in Capacity Scheduler | Major | capacityscheduler | Eric Payne | Eric Payne
YARN-6917 | Queue path is recomputed from scratch on every allocation | Minor | capacityscheduler | Jason Lowe | Eric Payne
HDFS-12301 | NN File Browser UI: Navigate to a path when enter is pressed | Trivial | ui | Ravi Prakash | Ravi Prakash
HADOOP-14251 | Credential provider should handle property key deprecation | Critical | security | John Zhuge | John Zhuge
MAPREDUCE-6937 | Backport MAPREDUCE-6870 to branch-2 while preserving compatibility | Major | . | Zhe Zhang | Peter Bacsko
YARN-5547 | NMLeveldbStateStore should be more tolerant of unknown keys | Major | nodemanager | Jason Lowe | Ajith S
YARN-6930 | Admins should be able to explicitly enable specific LinuxContainerRuntime in the NodeManager | Major | nodemanager | Vinod Kumar Vavilapalli | Shane Kumpf

### BUG FIXES:

JIRA | Summary | Priority | Component | Reporter | Contributor
--- | --- | --- | --- | --- | ---
YARN-524 | TestYarnVersionInfo failing if generated properties doesn’t include an SVN URL | Minor | api | Steve Loughran | Steve Loughran
YARN-1471 | The SLS simulator is not running the preemption policy for CapacityScheduler | Minor | . | Carlo Curino | Carlo Curino
HADOOP-11703 | git should ignore .DS_Store files on Mac OS X | Major | . | Abin Shahab | Abin Shahab
YARN-4612 | Fix rumen and scheduler load simulator handle killed tasks properly | Major | . | Ming Ma | Ming Ma
YARN-4594 | container-executor fails to remove directory tree when chmod required | Major | nodemanager | Colin P. McCabe | Colin P. McCabe
YARN-4731 | container-executor should not follow symlinks in recursive_unlink_children | Blocker | . | Bibin A Chundatt | Colin P. McCabe
YARN-4812 | TestFairScheduler#testContinuousScheduling fails intermittently | Major | fairscheduler | Karthik Kambatla | Karthik Kambatla
YARN-4927 | TestRMHA#testTransitionedToActiveRefreshFail fails with FairScheduler | Major | test | Karthik Kambatla | Bibin A Chundatt
YARN-4562 | YARN WebApp ignores the configuration passed to it for keystore settings | Major | . | Sergey Shelukhin | Sergey Shelukhin
HDFS-9276 | Failed to Update HDFS Delegation Token for long running application in HA mode | Major | fs, ha, security | Liangliang Gu | Liangliang Gu
YARN-5333 | Some recovered apps are put into default queue when RM HA | Major | . | Jun Gong | Jun Gong
HADOOP-13437 | KMS should reload whitelist and default key ACLs when hot-reloading | Major | kms | Xiao Chen | Xiao Chen
HDFS-8312 | Trash does not descent into child directories to check for permissions | Critical | fs, security | Eric Yang | Weiwei Yang
YARN-5920 | Fix deadlock in TestRMHA.testTransitionedToStandbyShouldNotHang | Major | test | Rohith Sharma K S | Varun Saxena
HADOOP-13867 | FilterFileSystem should override rename(.., options) to take effect of Rename options called via FilterFileSystem implementations | Major | . | Vinayakumar B | Vinayakumar B
HADOOP-13508 | FsPermission string constructor does not recognize sticky bit | Major | . | Atul Sikaria | Atul Sikaria
YARN-5988 | RM unable to start in secure setup | Blocker | . | Ajith S | Ajith S
YARN-6054 | TimelineServer fails to start when some LevelDb state files are missing. | Critical | . | Ravi Prakash | Ravi Prakash
HADOOP-13928 | TestAdlFileContextMainOperationsLive.testGetFileContext1 runtime error | Major | fs/adl, test | John Zhuge | John Zhuge
HADOOP-13976 | Path globbing does not match newlines | Major | . | Eric Badger | Eric Badger
HDFS-11316 | TestDataNodeVolumeFailure#testUnderReplicationAfterVolFailure fails in trunk | Minor | . | Yiqun Lin | Yiqun Lin
HADOOP-14044 | Synchronization issue in delegation token cancel functionality | Major | . | Hrishikesh Gadre | Hrishikesh Gadre
HDFS-11377 | Balancer hung due to no available mover threads | Major | balancer & mover | yunjiong zhao | yunjiong zhao
YARN-6031 | Application recovery has failed when node label feature is turned off during RM recovery | Minor | scheduler | Ying Zhang | Ying Zhang
YARN-6137 | Yarn client implicitly invoke ATS client which accesses HDFS | Major | . | Yesha Vora | Li Lu
HADOOP-13433 | Race in UGI.reloginFromKeytab | Major | security | Duo Zhang | Duo Zhang
HADOOP-14058 | Fix NativeS3FileSystemContractBaseTest#testDirWithDifferentMarkersWorks | Minor | fs/s3, test | Akira Ajisaka | Yiqun Lin
HDFS-11084 | Add a regression test for sticky bit support of OIV ReverseXML processor | Major | tools | Wei-Chiu Chuang | Wei-Chiu Chuang
HDFS-11391 | Numeric usernames do no work with WebHDFS FS (write access) | Major | webhdfs | Pierre Villard | Pierre Villard
HDFS-11177 | ‘storagepolicies -getStoragePolicy’ command should accept URI based path. | Major | shell | Surendra Singh Lilhore | Surendra Singh Lilhore
HDFS-11404 | Increase timeout on TestShortCircuitLocalRead.testDeprecatedGetBlockLocalPathInfoRpc | Major | . | Eric Badger | Eric Badger
HADOOP-14114 | S3A can no longer handle unencoded + in URIs | Minor | fs/s3 | Sean Mackrory | Sean Mackrory
MAPREDUCE-6841 | Fix dead link in MapReduce tutorial document | Minor | documentation | Akira Ajisaka | Victor Nee
MAPREDUCE-6753 | Variable in byte printed directly in mapreduce client | Major | client | Nemo Chen | Kai Sasaki
YARN-6263 | NMTokenSecretManagerInRM.createAndGetNMToken is not thread safe | Major | yarn | Haibo Chen | Haibo Chen
HADOOP-14026 | start-build-env.sh: invalid docker image name | Major | build | Gergő Pásztor | Gergő Pásztor
HDFS-11441 | Add escaping to error message in KMS web UI | Minor | security | Aaron T. Myers | Aaron T. Myers
MAPREDUCE-6855 | Specify charset when create String in CredentialsTestJob | Minor | . | Akira Ajisaka | Kai Sasaki
YARN-6165 | Intra-queue preemption occurs even when preemption is turned off for a specific queue. | Major | capacity scheduler, scheduler preemption | Eric Payne | Eric Payne
YARN-6310 | OutputStreams in AggregatedLogFormat.LogWriter can be left open upon exceptions | Major | yarn | Haibo Chen | Haibo Chen
YARN-6321 | TestResources test timeouts are too aggressive | Major | test | Jason Lowe | Eric Badger
HDFS-11512 | Increase timeout on TestShortCircuitLocalRead#testSkipWithVerifyChecksum | Minor | . | Eric Badger | Eric Badger
HDFS-11499 | Decommissioning stuck because of failing recovery | Major | hdfs, namenode | Lukas Majercak | Lukas Majercak
HDFS-11395 | RequestHedgingProxyProvider#RequestHedgingInvocationHandler hides the Exception thrown from NameNode | Major | ha | Nanda kumar | Nanda kumar
YARN-4051 | ContainerKillEvent lost when container is still recovering and application finishes | Critical | nodemanager | sandflee | sandflee
YARN-6217 | TestLocalCacheDirectoryManager test timeout is too aggressive | Major | test | Jason Lowe | Miklos Szegedi
HDFS-11132 | Allow AccessControlException in contract tests when getFileStatus on subdirectory of existing files | Major | fs/adl, test | Vishwajeet Dusane | Vishwajeet Dusane
HADOOP-14204 | S3A multipart commit failing, “UnsupportedOperationException at java.util.Collections$UnmodifiableList.sort” | Critical | fs/s3 | Steve Loughran | Steve Loughran
HADOOP-14205 | No FileSystem for scheme: adl | Major | fs/adl | John Zhuge | John Zhuge
HDFS-11561 | HttpFS doc errors | Trivial | documentation, httpfs, test | Yuanbo Liu | Yuanbo Liu
HADOOP-9631 | ViewFs should use underlying FileSystem’s server side defaults | Major | fs, viewfs | Lohit Vijayarenu | Erik Krogen
HADOOP-14214 | DomainSocketWatcher::add()/delete() should not self interrupt while looping await() | Critical | hdfs-client | Mingliang Liu | Mingliang Liu
HADOOP-14195 | CredentialProviderFactory$getProviders is not thread-safe | Major | security | Vihang Karajgaonkar | Vihang Karajgaonkar
HADOOP-14211 | FilterFs and ChRootedFs are too aggressive about enforcing “authorityNeeded” | Major | viewfs | Erik Krogen | Erik Krogen
MAPREDUCE-6866 | Fix getNumMapTasks() documentation in JobConf | Minor | documentation | Joe Mészáros | Joe Mészáros
MAPREDUCE-6868 | License check for jdiff output files should be ignored | Major | build | Akira Ajisaka | Akira Ajisaka
HDFS-10506 | OIV’s ReverseXML processor cannot reconstruct some snapshot details | Major | tools | Colin P. McCabe | Akira Ajisaka
HDFS-11486 | Client close() should not fail fast if the last block is being decommissioned | Major | . | Wei-Chiu Chuang | Wei-Chiu Chuang
YARN-6359 | TestRM#testApplicationKillAtAcceptedState fails rarely due to race condition | Major | test | Robert Kanter | Robert Kanter
YARN-5368 | Memory leak in timeline server | Critical | timelineserver | Wataru Yukawa | Jonathan Eagles
HADOOP-14247 | FileContextMainOperationsBaseTest should clean up test root path | Minor | fs, test | Mingliang Liu | Mingliang Liu
YARN-6352 | Header injections are possible in application proxy servlet | Major | resourcemanager, security | Naganarasimha G R | Naganarasimha G R
MAPREDUCE-6873 | MR Job Submission Fails if MR framework application path not on defaultFS | Minor | mrv2 | Erik Krogen | Erik Krogen
HADOOP-14256 | [S3A DOC] Correct the format for “Seoul” example | Minor | documentation, fs/s3 | Brahma Reddy Battula | Brahma Reddy Battula
MAPREDUCE-6850 | Shuffle Handler keep-alive connections are closed from the server side | Major | . | Jonathan Eagles | Jonathan Eagles
HDFS-11592 | Closing a file has a wasteful preconditions in NameNode | Major | namenode | Eric Badger | Eric Badger
YARN-6354 | LeveldbRMStateStore can parse invalid keys when recovering reservations | Major | resourcemanager | Jason Lowe | Jason Lowe
HADOOP-14268 | Fix markdown itemization in hadoop-aws documents | Minor | documentation, fs/s3 | Akira Ajisaka | Akira Ajisaka
YARN-6436 | TestSchedulingPolicy#testParseSchedulingPolicy timeout is too low | Major | test | Jason Lowe | Eric Badger
YARN-6420 | RM startup failure due to wrong order in nodelabel editlog | Critical | . | Bibin A Chundatt | Bibin A Chundatt
YARN-6403 | Invalid local resource request can raise NPE and make NM exit | Major | nodemanager | Tao Yang | Tao Yang
HDFS-11538 | Move ClientProtocol HA proxies into hadoop-hdfs-client | Blocker | hdfs-client | Andrew Wang | Huafeng Wang
YARN-6437 | TestSignalContainer#testSignalRequestDeliveryToNM fails intermittently | Major | test | Jason Lowe | Jason Lowe
HDFS-11629 | Revert HDFS-11431 hadoop-hdfs-client JAR does not include ConfiguredFailoverProxyProvider. | Major | . | Andrew Wang | Andrew Wang
HADOOP-13996 | Fix some release build issues | Blocker | build | Andrew Wang | Andrew Wang
HDFS-11608 | HDFS write crashed with block size greater than 2 GB | Critical | hdfs-client | Xiaobing Zhou | Xiaobing Zhou
YARN-6288 | Exceptions during aggregated log writes are mishandled | Critical | log-aggregation | Akira Ajisaka | Akira Ajisaka
HADOOP-14066 | VersionInfo should be marked as public API | Critical | common | Thejas M Nair | Akira Ajisaka
HADOOP-14293 | Initialize FakeTimer with a less trivial value | Major | test | Andrew Wang | Andrew Wang
YARN-6461 | TestRMAdminCLI has very low test timeouts | Major | test | Jason Lowe | Eric Badger
HDFS-11163 | Mover should move the file blocks to default storage once policy is unset | Major | balancer & mover | Surendra Singh Lilhore | Surendra Singh Lilhore
YARN-6450 | TestContainerManagerWithLCE requires override for each new test added to ContainerManagerTest | Major | test | Jason Lowe | Jason Lowe
YARN-3760 | FSDataOutputStream leak in AggregatedLogFormat.LogWriter.close() | Critical | nodemanager | Daryn Sharp | Haibo Chen
YARN-5994 | TestCapacityScheduler.testAMLimitUsage fails intermittently | Major | . | Eric Badger | Eric Badger
YARN-6480 | Timeout is too aggressive for TestAMRestart.testPreemptedAMRestartOnRMRestart | Major | . | Eric Badger | Eric Badger
HADOOP-14311 | Add python2.7-dev to Dockerfile | Major | . | Allen Wittenauer | Allen Wittenauer
YARN-6304 | Skip rm.transitionToActive call to RM if RM is already active. | Major | resourcemanager | Rohith Sharma K S | Rohith Sharma K S
HDFS-11615 | FSNamesystemLock metrics can be inaccurate due to millisecond precision | Major | hdfs | Erik Krogen | Erik Krogen
HADOOP-14318 | Remove non-existent setfattr command option from FileSystemShell.md | Minor | documentation | Doris Gu | Doris Gu
HADOOP-14315 | Python example in the rack awareness document doesn’t work due to bad indentation | Minor | documentation | Kengo Seki | Kengo Seki
HDFS-11689 | New exception thrown by DFSClient#isHDFSEncryptionEnabled broke hacky hive code | Major | . | Yongjun Zhang | Yongjun Zhang
HDFS-11691 | Add a proper scheme to the datanode links in NN web UI | Major | . | Kihwal Lee | Kihwal Lee
HADOOP-14341 | Support multi-line value for ssl.server.exclude.cipher.list | Major | . | John Zhuge | John Zhuge
YARN-5617 | AMs only intended to run one attempt can be run more than once | Major | resourcemanager | Jason Lowe | Jason Lowe
HADOOP-14346 | CryptoOutputStream throws IOException on flush() if stream is closed | Major | . | Pierre Lacave | Pierre Lacave
HDFS-11709 | StandbyCheckpointer should handle an non-existing legacyOivImageDir gracefully | Critical | ha, namenode | Zhe Zhang | Erik Krogen
YARN-5894 | fixed license warning caused by de.ruedigermoeller:fst:jar:2.24 | Blocker | yarn | Haibo Chen | Haibo Chen
HADOOP-14320 | TestIPC.testIpcWithReaderQueuing fails intermittently | Major | . | Eric Badger | Eric Badger
YARN-6536 | TestAMRMClient.testAMRMClientWithSaslEncryption fails intermittently | Major | . | Eric Badger | Jason Lowe
HDFS-11609 | Some blocks can be permanently lost if nodes are decommissioned while dead | Blocker | namenode | Kihwal Lee | Kihwal Lee
HDFS-8498 | Blocks can be committed with wrong size | Critical | hdfs-client | Daryn Sharp | Jing Zhao
HDFS-11714 | Newly added NN storage directory won’t get initialized and cause space exhaustion | Critical | . | Kihwal Lee | Kihwal Lee
HADOOP-14371 | License error in TestLoadBalancingKMSClientProvider.java | Major | . | hu xiaodong | hu xiaodong
HADOOP-14369 | NetworkTopology calls expensive toString() when logging | Major | . | Íñigo Goiri | Íñigo Goiri
HADOOP-14306 | TestLocalFileSystem tests have very low timeouts | Major | . | Eric Badger | Eric Badger
HADOOP-14372 | TestSymlinkLocalFS timeouts are too low | Major | . | Eric Badger | Eric Badger
HADOOP-14207 | “dfsadmin -refreshCallQueue” fails with DecayRpcScheduler | Blocker | rpc-server | Surendra Singh Lilhore | Surendra Singh Lilhore
HDFS-11702 | Remove indefinite caching of key provider uri in DFSClient | Major | hdfs-client | Rushabh S Shah | Rushabh S Shah
HADOOP-14374 | License error in GridmixTestUtils.java | Major | . | lixinglong | lixinglong
HADOOP-14100 | Upgrade Jsch jar to latest version to fix vulnerability in old versions | Critical | . | Vinayakumar B | Vinayakumar B
HADOOP-14377 | Increase Common test timeouts from 1 second to 10 seconds | Major | . | Eric Badger | Eric Badger
HADOOP-14373 | License error In org.apache.hadoop.metrics2.util.Servers | Major | . | hu xiaodong | hu xiaodong
YARN-6552 | Increase YARN test timeouts from 1 second to 10 seconds | Major | . | Eric Badger | Eric Badger
MAPREDUCE-6882 | Increase MapReduce test timeouts from 1 second to 10 seconds | Major | . | Eric Badger | Eric Badger
HADOOP-14405 | Fix performance regression due to incorrect use of DataChecksum | Major | native, performance | LiXin Ge | LiXin Ge
HDFS-11745 | Increase HDFS test timeouts from 1 second to 10 seconds | Major | . | Eric Badger | Eric Badger
HDFS-11755 | Underconstruction blocks can be considered missing | Major | . | Nathan Roberts | Nathan Roberts
YARN-5543 | ResourceManager SchedulingMonitor could potentially terminate the preemption checker thread | Major | capacityscheduler, resourcemanager | Min Shen | Min Shen
HDFS-11674 | reserveSpaceForReplicas is not released if append request failed due to mirror down and replica recovered | Critical | datanode | Vinayakumar B | Vinayakumar B
HADOOP-14376 | Memory leak when reading a compressed file using the native library | Major | common, io | Eli Acherkan | Eli Acherkan
HDFS-11818 | TestBlockManager.testSufficientlyReplBlocksUsesNewRack fails intermittently | Major | . | Eric Badger | Nathan Roberts
YARN-6598 | History server getApplicationReport NPE when fetching report for pre-2.8 job | Blocker | timelineserver | Jason Lowe | Jason Lowe
YARN-6603 | NPE in RMAppsBlock | Major | resourcemanager | Jason Lowe | Jason Lowe
HDFS-11833 | HDFS architecture documentation describes outdated placement policy | Minor | . | dud | Chen Liang
HADOOP-14412 | HostsFileReader#getHostDetails is very expensive on large clusters | Major | util | Jason Lowe | Jason Lowe
YARN-6577 | Remove unused ContainerLocalization classes | Minor | nodemanager | ZhangBing Lin | ZhangBing Lin
YARN-6618 | TestNMLeveldbStateStoreService#testCompactionCycle can fail if compaction occurs more than once | Minor | test | Jason Lowe | Jason Lowe
HDFS-11849 | JournalNode startup failure exception should be logged in log file | Major | journal-node | Surendra Singh Lilhore | Surendra Singh Lilhore
HDFS-11864 | Document Metrics to track usage of memory for writes | Major | documentation | Brahma Reddy Battula | Yiqun Lin
YARN-6615 | AmIpFilter drops query parameters on redirect | Major | . | Wilfred Spiegelenburg | Wilfred Spiegelenburg
HADOOP-14449 | The ASF Header in ComparableVersion.java and SSLHostnameVerifier.java is not correct | Minor | common, documentation | ZhangBing Lin | ZhangBing Lin
HADOOP-14166 | Reset the DecayRpcScheduler AvgResponseTime metric to zero when queue is not used | Major | common | Surendra Singh Lilhore | Surendra Singh Lilhore
HDFS-11661 | GetContentSummary uses excessive amounts of memory | Blocker | namenode | Nathan Roberts | Wei-Chiu Chuang
YARN-6141 | ppc64le on Linux doesn’t trigger __linux get_executable codepath | Major | nodemanager | Sonia Garudi | Ayappan
HDFS-11445 | FSCK shows overall health status as corrupt even one replica is corrupt | Critical | . | Brahma Reddy Battula | Brahma Reddy Battula
YARN-6643 | TestRMFailover fails rarely due to port conflict | Major | test | Robert Kanter | Robert Kanter
HDFS-11817 | A faulty node can cause a lease leak and NPE on accessing data | Critical | . | Kihwal Lee | Kihwal Lee
YARN-6641 | Non-public resource localization on a bad disk causes subsequent containers failure | Major | . | Kuhu Shukla | Kuhu Shukla
HDFS-11078 | Fix NPE in LazyPersistFileScrubber | Major | . | Íñigo Goiri | Íñigo Goiri
HADOOP-14464 | hadoop-aws doc header warning #5 line wrapped | Trivial | documentation, fs/s3 | John Zhuge | John Zhuge
HDFS-5042 | Completed files lost after power failure | Critical | . | Dave Latham | Vinayakumar B
YARN-6649 | RollingLevelDBTimelineServer throws RuntimeException if object decoding ever fails runtime exception | Critical | . | Jonathan Eagles | Jonathan Eagles
HDFS-11893 | Fix TestDFSShell.testMoveWithTargetPortEmpty failure. | Major | test | Konstantin Shvachko | Brahma Reddy Battula
HDFS-11741 | Long running balancer may fail due to expired DataEncryptionKey | Major | balancer & mover | Wei-Chiu Chuang | Wei-Chiu Chuang
HDFS-11856 | Ability to re-add Upgrading Nodes (remote) to pipeline for future pipeline updates | Major | hdfs-client, rolling upgrades | Vinayakumar B | Vinayakumar B
HADOOP-14474 | Use OpenJDK 7 instead of Oracle JDK 7 to avoid oracle-java7-installer failures | Major | build | Akira Ajisaka | Akira Ajisaka
HDFS-10816 | TestComputeInvalidateWork#testDatanodeReRegistration fails due to race between test and replication monitor | Major | . | Eric Badger | Eric Badger
HDFS-11932 | BPServiceActor thread name is not correctly set | Major | hdfs | Chen Liang | Chen Liang
HDFS-11708 | Positional read will fail if replicas moved to different DNs after stream is opened | Critical | hdfs-client | Vinayakumar B | Vinayakumar B
HDFS-11711 | DN should not delete the block On “Too many open files” Exception | Critical | datanode | Brahma Reddy Battula | Brahma Reddy Battula
MAPREDUCE-6676 | NNBench should Throw IOException when rename and delete fails | Major | . | Brahma Reddy Battula | Brahma Reddy Battula
HADOOP-14208 | Fix typo in the top page in branch-2.8 | Trivial | common, documentation | Akira Ajisaka | Wenxin He
HDFS-11945 | Internal lease recovery may not be retried for a long time | Major | namenode | Kihwal Lee | Kihwal Lee
HADOOP-14511 | WritableRpcEngine.Invocation#toString NPE on null parameters | Minor | ipc | John Zhuge | John Zhuge
HADOOP-14512 | WASB atomic rename should not throw exception if the file is neither in src nor in dst when doing the rename | Major | fs/azure | Duo Xu | Duo Xu
YARN-6585 | RM fails to start when upgrading from 2.7 to 2.8 for clusters with node labels. | Blocker | . | Eric Payne | Sunil Govindan
HDFS-11967 | TestJMXGet fails occasionally | Major | . | Arpit Agarwal | Arpit Agarwal
YARN-6719 | Fix findbugs warnings in SLSCapacityScheduler.java | Major | . | Akira Ajisaka | Akira Ajisaka
HADOOP-14540 | Replace MRv1 specific terms in HostsFileReader | Minor | documentation | Akira Ajisaka | hu xiaodong
HDFS-11995 | HDFS Architecture documentation incorrectly describes writing to a local temporary file. | Minor | documentation | Chris Nauroth | Nanda kumar
HDFS-11736 | OIV tests should not write outside ‘target’ directory. | Major | . | Konstantin Shvachko | Yiqun Lin
YARN-6713 | Fix dead link in the Javadoc of FairSchedulerEventLog.java | Minor | documentation | Akira Ajisaka | Weiwei Yang
HADOOP-14533 | Size of args cannot be less than zero in TraceAdmin#run as its linkedlist | Trivial | common, tracing | Weisen Han | Weisen Han
HDFS-11960 | Successfully closed files can stay under-replicated. | Critical | . | Kihwal Lee | Kihwal Lee
HADOOP-14146 | KerberosAuthenticationHandler should authenticate with SPN in AP-REQ | Major | security | Daryn Sharp | Daryn Sharp
YARN-6467 | CSQueueMetrics needs to update the current metrics for default partition only | Major | capacity scheduler | Naganarasimha G R | Manikandan R
HADOOP-14024 | KMS JMX endpoint throws ClassNotFoundException | Critical | kms | Andrew Wang | John Zhuge
YARN-6749 | TestAppSchedulingInfo.testPriorityAccounting fails consistently | Major | . | Eric Badger | Naganarasimha G R
MAPREDUCE-6905 | Fix meaningless operations in TestDFSIO in some situation. | Major | tools/rumen | LiXin Ge | LiXin Ge
MAPREDUCE-6909 | LocalJobRunner fails when run on a node from multiple users | Blocker | client | Jason Lowe | Jason Lowe
HADOOP-13414 | Hide Jetty Server version header in HTTP responses | Major | security | Vinayakumar B | Surendra Singh Lilhore
HDFS-12089 | Fix ambiguous NN retry log message | Major | webhdfs | Eric Badger | Eric Badger
MAPREDUCE-6911 | TestMapreduceConfigFields.testCompareXmlAgainstConfigurationClass fails consistently | Major | . | Eric Badger | Eric Badger
YARN-6708 | Nodemanager container crash after ext3 folder limit | Critical | . | Bibin A Chundatt | Bibin A Chundatt
HADOOP-14563 | LoadBalancingKMSClientProvider#warmUpEncryptedKeys swallows IOException | Major | . | Rushabh S Shah | Rushabh S Shah
MAPREDUCE-6246 | DBOutputFormat.java appending extra semicolon to query which is incompatible with DB2 | Major | mrv1, mrv2 | ramtin | Gergely Novák
YARN-6428 | Queue AM limit is not honored in CS always | Major | . | Bibin A Chundatt | Bibin A Chundatt
YARN-6770 | [Docs] A small mistake in the example of TimelineClient | Trivial | docs | Jinjiang Ling | Jinjiang Ling
HADOOP-10829 | Iteration on CredentialProviderFactory.serviceLoader is thread-unsafe | Major | security | Benoy Antony | Benoy Antony
YARN-6809 | Fix typo in ResourceManagerHA.md | Trivial | documentation | Akira Ajisaka | Yeliang Cang
MAPREDUCE-5621 | mr-jobhistory-daemon.sh doesn’t have to execute mkdir and chown all the time | Minor | jobhistoryserver | Shinichi Yamashita | Shinichi Yamashita
YARN-6797 | TimelineWriter does not fully consume the POST response | Major | timelineclient | Jason Lowe | Jason Lowe
HDFS-11502 | Datanode UI should display hostname based on JMX bean instead of window.location.hostname | Major | hdfs | Jeffrey E Rodriguez | Jeffrey E Rodriguez
HADOOP-14646 | FileContextMainOperationsBaseTest#testListStatusFilterWithSomeMatches never runs | Minor | test | Andras Bokor | Andras Bokor
MAPREDUCE-6697 | Concurrent task limits should only be applied when necessary | Major | mrv2 | Jason Lowe | Nathan Roberts
YARN-6654 | RollingLevelDBTimelineStore backwards incompatible after fst upgrade | Blocker | . | Jonathan Eagles | Jonathan Eagles
YARN-6805 | NPE in LinuxContainerExecutor due to null PrivilegedOperationException exit code | Major | nodemanager | Jason Lowe | Jason Lowe
YARN-3260 | AM attempt fail to register before RM processes launch event | Critical | resourcemanager | Jason Lowe | Bibin A Chundatt
HDFS-12140 | Remove BPOfferService lock contention to get block pool id | Critical | datanode | Daryn Sharp | Daryn Sharp
HDFS-12112 | TestBlockManager#testBlockManagerMachinesArray sometimes fails with NPE | Minor | . | Wei-Chiu Chuang | Wei-Chiu Chuang
MAPREDUCE-6910 | MapReduceTrackingUriPlugin can not return the right URI of history server with HTTPS | Major | jobhistoryserver | Lantao Jin | Lantao Jin
HDFS-12158 | Secondary Namenode’s web interface lack configs for X-FRAME-OPTIONS protection | Major | namenode | Mukul Kumar Singh | Mukul Kumar Singh
YARN-6837 | Null LocalResource visibility or resource type can crash the nodemanager | Major | . | Jinjiang Ling | Jinjiang Ling
HDFS-11472 | Fix inconsistent replica size after a data pipeline failure | Critical | datanode | Wei-Chiu Chuang | Erik Krogen
HDFS-12177 | NameNode exits due to setting BlockPlacementPolicy loglevel to Debug | Major | block placement | Jiandan Yang | Jiandan Yang
HDFS-11742 | Improve balancer usability after HDFS-8818 | Blocker | . | Kihwal Lee | Kihwal Lee
HDFS-11896 | Non-dfsUsed will be doubled on dead node re-registration | Blocker | . | Brahma Reddy Battula | Brahma Reddy Battula
YARN-5728 | TestMiniYarnClusterNodeUtilization.testUpdateNodeUtilization timeout | Major | test | Akira Ajisaka | Akira Ajisaka
YARN-6628 | Unexpected jackson-core-2.2.3 dependency introduced | Blocker | timelineserver | Jason Lowe | Jonathan Eagles
YARN-5731 | Preemption calculation is not accurate when reserved containers are present in queue. | Major | capacity scheduler | Sunil Govindan | Wangda Tan
HADOOP-14683 | FileStatus.compareTo binary compatible issue | Blocker | . | Sergey Shelukhin | Akira Ajisaka
YARN-6872 | Ensure apps could run given NodeLabels are disabled post RM switchover/restart | Major | resourcemanager | Sunil Govindan | Sunil Govindan
YARN-6846 | Nodemanager can fail to fully delete application local directories when applications are killed | Critical | nodemanager | Jason Lowe | Jason Lowe
MAPREDUCE-6927 | MR job should only set tracking url if history was successfully written | Major | . | Eric Badger | Eric Badger
YARN-6890 | If UI is not secured, we allow user to kill other users’ job even yarn cluster is secured. | Critical | . | Sumana Sathish | Junping Du
HDFS-10326 | Disable setting tcp socket send/receive buffers for write pipelines | Major | datanode, hdfs | Daryn Sharp | Daryn Sharp
HDFS-12157 | Do fsyncDirectory(..) outside of FSDataset lock | Critical | datanode | Vinayakumar B | Vinayakumar B
HDFS-12278 | LeaseManager operations are inefficient in 2.8. | Blocker | namenode | Rushabh S Shah | Rushabh S Shah
YARN-6987 | Log app attempt during InvalidStateTransition | Major | . | Jonathan Eagles | Jonathan Eagles
YARN-7020 | TestAMRMProxy#testAMRMProxyTokenRenewal is flakey | Major | . | Robert Kanter | Robert Kanter
YARN-2416 | InvalidStateTransitonException in ResourceManager if AMLauncher does not receive response for startContainers() call in time | Critical | resourcemanager | Jian Fang | Jonathan Eagles
YARN-7048 | Fix tests faking kerberos to explicitly set ugi auth type | Major | yarn | Daryn Sharp | Daryn Sharp
HADOOP-14687 | AuthenticatedURL will reuse bad/expired session cookies | Critical | common | Daryn Sharp | Daryn Sharp
YARN-6640 | AM heartbeat stuck when responseId overflows MAX_INT | Blocker | . | Botong Huang | Botong Huang
HDFS-12319 | DirectoryScanner will throw IllegalStateException when Multiple BP’s are present | Blocker | . | Brahma Reddy Battula | Brahma Reddy Battula
HDFS-12299 | Race Between update pipeline and DN Re-Registration | Critical | . | Brahma Reddy Battula | Brahma Reddy Battula
YARN-7052 | RM SchedulingMonitor gives no indication why the spawned thread crashed. | Critical | yarn | Eric Payne | Eric Payne
YARN-7087 | NM failed to perform log aggregation due to absent container | Blocker | log-aggregation | Jason Lowe | Jason Lowe
YARN-7051 | Avoid concurrent modification exception in FifoIntraQueuePreemptionPlugin | Critical | capacity scheduler, scheduler preemption, yarn | Eric Payne | Eric Payne
HDFS-12364 | [branch-2.8.2] Fix the Compile Error after HDFS-12299 | Blocker | hdfs | Jiandan Yang | Jiandan Yang
YARN-7112 | TestAMRMProxy is failing with invalid request | Major | . | Jason Lowe | Jason Lowe
YARN-7076 | yarn application -list -appTypes <appType> is not working | Blocker | . | Jian He | Jian He
MAPREDUCE-6641 | TestTaskAttempt fails in trunk | Major | test | Tsuyoshi Ozawa | Haibo Chen
YARN-7083 | Log aggregation deletes/renames while file is open | Critical | nodemanager | Daryn Sharp | Jason Lowe
HADOOP-14814 | Fix incompatible API change on FsServerDefaults to HADOOP-14104 | Blocker | . | Junping Du | Junping Du
YARN-7023 | Incorrect ReservationId.compareTo() implementation | Minor | reservation system | Oleg Danilov | Oleg Danilov
HADOOP-14842 | Hadoop 2.8.2 release build process get stuck due to java issue | Blocker | build | Junping Du | Junping Du
YARN-5195 | RM intermittently crashed with NPE while handling APP_ATTEMPT_REMOVED event when async-scheduling enabled in CapacityScheduler | Major | resourcemanager | Karam Singh | sandflee
YARN-7034 | DefaultLinuxContainerRuntime and DockerLinuxContainerRuntime sends client environment variables to container-executor | Blocker | nodemanager | Miklos Szegedi | Miklos Szegedi
YARN-7325 | Remove unused container variable in DockerLinuxContainerRuntime | Minor | nodemanager | Shane Kumpf | Shane Kumpf
YARN-7246 | Fix the default docker binary path | Blocker | nodemanager | Shane Kumpf | Shane Kumpf
YARN-7333 | container-executor fails to remove entries from a directory that is not writable or executable | Critical | . | Jason Lowe | Jason Lowe
YARN-7230 | Document DockerContainerRuntime for branch-2.8 with proper scope and claim as an experimental feature | Blocker | documentation | Junping Du | Shane Kumpf
HADOOP-14958 | CLONE - Fix source-level compatibility after HADOOP-11252 | Blocker | . | Junping Du | Junping Du
YARN-7647 | NM print inappropriate error log when node-labels is enabled | Minor | . | Yang Wang | Yang Wang
YARN-7249 | Fix CapacityScheduler NPE issue when a container preempted while the node is being removed | Blocker | . | Wangda Tan | Wangda Tan

### TESTS:

JIRA | Summary | Priority | Component | Reporter | Contributor
--- | --- | --- | --- | --- | ---
HDFS-9300 | TestDirectoryScanner.testThrottle() is still a little flakey | Major | balancer & mover, test | Daniel Templeton | Daniel Templeton
HADOOP-13178 | TestShellBasedIdMapping.testStaticMapUpdate doesn’t work on OS X | Major | test | Allen Wittenauer | Kai Sasaki
YARN-5343 | TestContinuousScheduling#testSortedNodes fails intermittently | Minor | . | sandflee | Yufei Gu
MAPREDUCE-6831 | Flaky test TestJobImpl.testKilledDuringKillAbort | Major | mrv2 | Peter Bacsko | Peter Bacsko
HDFS-11290 | TestFSNameSystemMBean should wait until JMX cache is cleared | Major | test | Akira Ajisaka | Erik Krogen
YARN-5349 | TestWorkPreservingRMRestart#testUAMRecoveryOnRMWorkPreservingRestart fail intermittently | Minor | . | sandflee | Jason Lowe

### SUB-TASKS:

JIRA | Summary | Priority | Component | Reporter | Contributor
--- | --- | --- | --- | --- | ---
HADOOP-14032 | Reduce fair call queue priority inversion | Major | ipc | Daryn Sharp | Daryn Sharp
HADOOP-14034 | Allow ipc layer exceptions to selectively close connections | Major | ipc | Daryn Sharp | Daryn Sharp
HADOOP-14033 | Reduce fair call queue lock contention | Major | ipc | Daryn Sharp | Daryn Sharp
HADOOP-14135 | Remove URI parameter in AWSCredentialProvider constructors | Major | fs/s3 | Mingliang Liu | Mingliang Liu
HADOOP-14196 | Azure Data Lake doc is missing required config entry | Major | fs/adl | Atul Sikaria | Atul Sikaria
HADOOP-14197 | Fix ADLS doc for credential provider | Major | documentation, fs/adl | John Zhuge | John Zhuge
HADOOP-14230 | TestAdlFileSystemContractLive fails to clean up | Minor | fs/adl, test | John Zhuge | John Zhuge
HADOOP-14038 | Rename ADLS credential properties | Minor | fs/adl | John Zhuge | John Zhuge
HADOOP-14321 | Explicitly exclude S3A root dir ITests from parallel runs | Minor | fs/s3, test | Steve Loughran | Steve Loughran
HADOOP-14241 | Add ADLS sensitive config keys to default list | Minor | fs, fs/adl, security | John Zhuge | John Zhuge
HADOOP-14349 | Rename ADLS CONTRACT_ENABLE_KEY | Minor | fs/adl | Mingliang Liu | Mingliang Liu
HDFS-9005 | Provide configuration support for upgrade domain | Major | . | Ming Ma | Ming Ma
HDFS-9016 | Display upgrade domain information in fsck | Major | . | Ming Ma | Ming Ma
HDFS-9922 | Upgrade Domain placement policy status marks a good block in violation when there are decommissioned nodes | Minor | . | Chris Trezzo | Chris Trezzo
HADOOP-14035 | Reduce fair call queue backoff’s impact on clients | Major | ipc | Daryn Sharp | Daryn Sharp
YARN-6682 | Improve performance of AssignmentInformation datastructures | Major | . | Daryn Sharp | Daryn Sharp
YARN-6680 | Avoid locking overhead for NO_LABEL lookups | Major | resourcemanager | Daryn Sharp | Daryn Sharp
YARN-6681 | Eliminate double-copy of child queues in canAssignToThisQueue | Major | resourcemanager | Daryn Sharp | Daryn Sharp
YARN-2113 | Add cross-user preemption within CapacityScheduler’s leaf-queue | Major | capacity scheduler | Vinod Kumar Vavilapalli | Sunil Govindan
YARN-6775 | CapacityScheduler: Improvements to assignContainers, avoid unnecessary canAssignToUser/Queue calls | Major | capacityscheduler | Nathan Roberts | Nathan Roberts
YARN-6988 | container-executor fails for docker when command length > 4096 B | Major | yarn | Eric Badger | Eric Badger
HDFS-12473 | Change hosts JSON file format | Major | . | Ming Ma | Ming Ma
HDFS-9754 | Avoid unnecessary getBlockCollection calls in BlockManager | Major | namenode | Jing Zhao | Jing Zhao

### OTHER:

JIRA | Summary | Priority | Component | Reporter | Contributor
--- | --- | --- | --- | --- | ---
HADOOP-14344 | Revert HADOOP-13606 swift FS to add a service load metadata file | Major | . | John Zhuge | John Zhuge
HDFS-11717 | Add unit test for HDFS-11709 StandbyCheckpointer should handle non-existing legacyOivImageDir gracefully | Minor | ha, namenode | Erik Krogen | Erik Krogen