Apache Hadoop 3.5.0 – Apache Hadoop Changelog

##### General

- Overview
- Single Node Setup
- Cluster Setup
- Commands Reference
- FileSystem Shell
- Compatibility Specification
- Downstream Developer's Guide
- Admin Compatibility Guide
- Interface Classification
- FileSystem Specification

##### Common

- CLI Mini Cluster
- Fair Call Queue
- Native Libraries
- Proxy User
- Rack Awareness
- Secure Mode
- Service Level Authorization
- HTTP Authentication
- Credential Provider API
- Hadoop KMS
- Tracing
- Unix Shell Guide
- Registry
- Async Profiler

##### HDFS

- Architecture
- User Guide
- Commands Reference
- NameNode HA With QJM
- NameNode HA With NFS
- Observer NameNode
- Federation
- ViewFs
- ViewFsOverloadScheme
- Snapshots
- Edits Viewer
- Image Viewer
- Permissions and HDFS
- Quotas and HDFS
- libhdfs (C API)
- WebHDFS (REST API)
- HttpFS
- Short Circuit Local Reads
- Centralized Cache Management
- NFS Gateway
- Rolling Upgrade
- Extended Attributes
- Transparent Encryption
- Multihoming
- Storage Policies
- Memory Storage Support
- Synthetic Load Generator
- Erasure Coding
- Disk Balancer
- Upgrade Domain
- DataNode Admin
- Router Federation
- Provided Storage

##### MapReduce

- Tutorial
- Commands Reference
- Encrypted Shuffle
- Pluggable Shuffle/Sort
- Distributed Cache Deploy
- Support for YARN Shared Cache

##### MapReduce REST APIs

- MR Application Master
- MR History Server

##### YARN

- Architecture
- Commands Reference
- Capacity Scheduler
- Fair Scheduler
- ResourceManager Restart
- ResourceManager HA
- Resource Model
- Node Labels
- Node Attributes
- Web Application Proxy
- Timeline Server
- Timeline Service V.2
- Writing YARN Applications
- YARN Application Security
- NodeManager
- Running Applications in Docker Containers
- Running Applications in runC Containers
- Using CGroups
- Secure Containers
- Reservation System
- Graceful Decommission
- Opportunistic Containers
- YARN Federation
- Shared Cache
- Using GPU
- Using FPGA
- Placement Constraints
- YARN UI2

##### YARN REST APIs

- Introduction
- Resource Manager
- Node Manager
- Timeline Server
- Timeline Service V.2

##### YARN Service

- Overview
- QuickStart
- Concepts
- Yarn Service API
- Service Discovery
- System Services

##### Hadoop Compatible File Systems

- Aliyun OSS
- Amazon S3
- Azure Blob Storage
- Azure Data Lake Storage
- Tencent COS
- Huaweicloud OBS
- VolcanoEngine TOS

##### Auth

- Overview
- Examples
- Configuration
- Building

##### Tools

- Hadoop Streaming
- Hadoop Archives
- Hadoop Archive Logs
- DistCp
- HDFS Federation Balance
- GridMix
- Rumen
- Resource Estimator Service
- Scheduler Load Simulator
- Hadoop Benchmarking
- Dynamometer

##### Reference

- Changelog and Release Notes
- Java API docs
- Unix Shell API
- Metrics

##### Configuration

- core-default.xml
- hdfs-default.xml
- hdfs-rbf-default.xml
- mapred-default.xml
- yarn-default.xml
- kms-default.xml
- httpfs-default.xml
- Deprecated Properties

# Apache Hadoop Changelog

## Release 3.0.4 - Unreleased (as of 2018-09-02)

### NEW FEATURES:

JIRA | Summary | Priority | Component | Reporter | Contributor
--- | --- | --- | --- | --- | ---
HDFS-13448 | HDFS Block Placement - Ignore Locality for First Block Replica | Minor | block placement, hdfs-client | BELUGA BEHR | BELUGA BEHR

### IMPROVEMENTS:

JIRA | Summary | Priority | Component | Reporter | Contributor
--- | --- | --- | --- | --- | ---
HADOOP-15252 | Checkstyle version is not compatible with IDEA’s checkstyle plugin | Major | . | Andras Bokor | Andras Bokor
HDFS-13602 | Add checkOperation(WRITE) checks in FSNamesystem | Major | ha, namenode | Erik Krogen | Chao Sun
HDFS-13155 | BlockPlacementPolicyDefault.chooseTargetInOrder Not Checking Return Value for NULL | Minor | namenode | BELUGA BEHR | Zsolt Venczel
HDFS-13659 | Add more test coverage for contentSummary for snapshottable path | Major | namenode, test | Wei-Chiu Chuang | Wei-Chiu Chuang
HADOOP-15499 | Performance severe drop when running RawErasureCoderBenchmark with NativeRSRawErasureCoder | Major | . | Sammi Chen | Sammi Chen
HDFS-13653 | Make dfs.client.failover.random.order a per nameservice configuration | Major | federation | Ekanth Sethuramalingam | Ekanth Sethuramalingam
YARN-8394 | Improve data locality documentation for Capacity Scheduler | Major | . | Weiwei Yang | Weiwei Yang
HDFS-13641 | Add metrics for edit log tailing | Major | metrics | Chao Sun | Chao Sun
HDFS-13686 | Add overall metrics for FSNamesystemLock | Major | hdfs, namenode | Lukas Majercak | Lukas Majercak
HDFS-13692 | StorageInfoDefragmenter floods log when compacting StorageInfo TreeSet | Minor | . | Yiqun Lin | Bharat Viswanadham
HDFS-13703 | Avoid allocation of CorruptedBlocks hashmap when no corrupted blocks are hit | Major | performance | Todd Lipcon | Todd Lipcon
HADOOP-15554 | Improve JIT performance for Configuration parsing | Minor | conf, performance | Todd Lipcon | Todd Lipcon
HDFS-13714 | Fix TestNameNodePrunesMissingStorages test failures on Windows | Major | hdfs, namenode, test | Lukas Majercak | Lukas Majercak
HDFS-13712 | BlockReaderRemote.read() logging improvement | Minor | hdfs-client | Gergo Repas | Gergo Repas
HDFS-13719 | Docs around dfs.image.transfer.timeout are misleading | Major | . | Kitti Nanasi | Kitti Nanasi
HADOOP-15598 | DataChecksum calculate checksum is contented on hashtable synchronization | Major | common | Prasanth Jayachandran | Prasanth Jayachandran
HADOOP-15609 | Retry KMS calls when SSLHandshakeException occurs | Major | common, kms | Kitti Nanasi | Kitti Nanasi
HADOOP-15612 | Improve exception when tfile fails to load LzoCodec | Major | . | Gera Shegalov | Gera Shegalov
HDFS-11060 | make DEFAULT_MAX_CORRUPT_FILEBLOCKS_RETURNED configurable | Minor | hdfs | Lantao Jin | Lantao Jin
HDFS-13727 | Log full stack trace if DiskBalancer exits with an unhandled exception | Minor | diskbalancer | Stephen O’Donnell | Gabor Bota
YARN-8155 | Improve ATSv2 client logging in RM and NM publisher | Major | . | Rohith Sharma K S | Abhishek Modi
HDFS-13728 | Disk Balancer should not fail if volume usage is greater than capacity | Minor | diskbalancer | Stephen O’Donnell | Stephen O’Donnell
YARN-8568 | Replace the deprecated zk-address property in the HA config example in ResourceManagerHA.md | Minor | yarn | Antal Bálint Steinbach | Antal Bálint Steinbach
HDFS-13814 | Remove super user privilege requirement for NameNode.getServiceStatus | Minor | namenode | Chao Sun | Chao Sun
YARN-8559 | Expose mutable-conf scheduler’s configuration in RM /scheduler-conf endpoint | Major | resourcemanager | Anna Savarin | Weiwei Yang
HDFS-13813 | Exit NameNode if dangling child inode is detected when saving FsImage | Major | hdfs, namenode | Siyao Meng | Siyao Meng
HDFS-13821 | RBF: Add dfs.federation.router.mount-table.cache.enable so that users can disable cache | Major | hdfs | Fei Hui | Fei Hui
HDFS-13831 | Make block increment deletion number configurable | Major | . | Yiqun Lin | Ryan Wu
YARN-8051 | TestRMEmbeddedElector#testCallbackSynchronization is flakey | Major | test | Robert Kanter | Robert Kanter

### BUG FIXES:

JIRA | Summary | Priority | Component | Reporter | Contributor
--- | --- | --- | --- | --- | ---
HDFS-12857 | StoragePolicyAdmin should support schema based path | Major | namenode | Surendra Singh Lilhore | Surendra Singh Lilhore
YARN-7835 | [Atsv2] Race condition in NM while publishing events if second attempt is launched on the same node | Critical | . | Rohith Sharma K S | Rohith Sharma K S
YARN-7773 | YARN Federation used Mysql as state store throw exception, Unknown column ‘homeSubCluster’ in ‘field list’ | Blocker | federation | Yiran Wu | Yiran Wu
HDFS-13636 | Cross-Site Scripting vulnerability in HttpServer2 | Major | . | Haibo Yan | Haibo Yan
HDFS-13339 | Volume reference can’t be released and may lead to deadlock when DataXceiver does a check volume | Critical | datanode | liaoyuxiangqin | Zsolt Venczel
YARN-8382 | cgroup file leak in NM | Major | nodemanager | Hu Ziqian | Hu Ziqian
HDFS-13545 | “guarded” is misspelled as “gaurded” in FSPermissionChecker.java | Trivial | documentation | Jianchao Jia | Jianchao Jia
MAPREDUCE-7103 | Fix TestHistoryViewerPrinter on windows due to a mismatch line separator | Minor | . | Giovanni Matteo Fumarola | Giovanni Matteo Fumarola
HADOOP-15217 | FsUrlConnection does not handle paths with spaces | Major | fs | Joseph Fourny | Zsolt Venczel
HDFS-12950 | [oiv] ls will fail in secure cluster | Major | . | Brahma Reddy Battula | Wei-Chiu Chuang
YARN-8359 | Exclude containermanager.linux test classes on Windows | Major | . | Giovanni Matteo Fumarola | Jason Lowe
HDFS-13664 | Refactor ConfiguredFailoverProxyProvider to make inheritance easier | Minor | hdfs-client | Chao Sun | Chao Sun
HDFS-12670 | can’t renew HDFS tokens with only the hdfs client jar | Critical | . | Thomas Graves | Arpit Agarwal
HDFS-13667 | Typo: Marking all “datandoes” as stale | Trivial | namenode | Wei-Chiu Chuang | Nanda kumar
YARN-8405 | RM zk-state-store.parent-path ACLs has been changed since HADOOP-14773 | Major | . | Rohith Sharma K S | Íñigo Goiri
YARN-8404 | Timeline event publish need to be async to avoid Dispatcher thread leak in case ATS is down | Blocker | . | Rohith Sharma K S | Rohith Sharma K S
HDFS-13673 | TestNameNodeMetrics fails on Windows | Minor | test | Zuoming Zhang | Zuoming Zhang
HDFS-13676 | TestEditLogRace fails on Windows | Minor | test | Zuoming Zhang | Zuoming Zhang
HDFS-13174 | hdfs mover -p /path times out after 20 min | Major | balancer & mover | Istvan Fajth | Istvan Fajth
HADOOP-15523 | Shell command timeout given is in seconds whereas it is taken as millisec while scheduling | Major | . | Bilwa S T | Bilwa S T
HDFS-13682 | Cannot create encryption zone after KMS auth token expires | Critical | encryption, kms, namenode | Xiao Chen | Xiao Chen
YARN-8444 | NodeResourceMonitor crashes on bad swapFree value | Major | . | Jim Brennan | Jim Brennan
YARN-8443 | Total #VCores in cluster metrics is wrong when CapacityScheduler reserved some containers | Major | webapp | Tao Yang | Tao Yang
YARN-8457 | Compilation is broken with -Pyarn-ui | Major | webapp | Sunil Govindan | Sunil Govindan
YARN-8401 | [UI2] new ui is not accessible with out internet connection | Blocker | . | Bibin A Chundatt | Bibin A Chundatt
YARN-8451 | Multiple NM heartbeat thread created when a slow NM resync with RM | Major | nodemanager | Botong Huang | Botong Huang
HADOOP-15548 | Randomize local dirs | Minor | . | Jim Brennan | Jim Brennan
HDFS-13702 | Remove HTrace hooks from DFSClient to reduce CPU usage | Major | performance | Todd Lipcon | Todd Lipcon
HDFS-13635 | Incorrect message when block is not found | Major | datanode | Wei-Chiu Chuang | Gabor Bota
HADOOP-15571 | Multiple FileContexts created with the same configuration object should be allowed to have different umask | Critical | . | Vinod Kumar Vavilapalli | Vinod Kumar Vavilapalli
HDFS-13121 | NPE when request file descriptors when SC read | Minor | hdfs-client | Gang Xie | Zsolt Venczel
YARN-6265 | yarn.resourcemanager.fail-fast is used inconsistently | Major | resourcemanager | Daniel Templeton | Yuanbo Liu
YARN-8473 | Containers being launched as app tears down can leave containers in NEW state | Major | nodemanager | Jason Lowe | Jason Lowe
HDFS-13723 | Occasional “Should be different group” error in TestRefreshUserMappings#testGroupMappingRefresh | Major | security, test | Siyao Meng | Siyao Meng
HDFS-12837 | Intermittent failure in TestReencryptionWithKMS | Major | encryption, test | Surendra Singh Lilhore | Xiao Chen
HDFS-13729 | Fix broken links to RBF documentation | Minor | documentation | jwhitter | Gabor Bota
YARN-8515 | container-executor can crash with SIGPIPE after nodemanager restart | Major | . | Jim Brennan | Jim Brennan
YARN-8421 | when moving app, activeUsers is increased, even though app does not have outstanding request | Major | . | kyungwan nam |
HDFS-13524 | Occasional “All datanodes are bad” error in TestLargeBlock#testLargeBlockSize | Major | . | Wei-Chiu Chuang | Siyao Meng
HADOOP-15610 | Hadoop Docker Image Pip Install Fails | Critical | . | Jack Bearden | Jack Bearden
HADOOP-15614 | TestGroupsCaching.testExceptionOnBackgroundRefreshHandled reliably fails | Major | . | Kihwal Lee | Weiwei Yang
YARN-8548 | AllocationRespose proto setNMToken initBuilder not done | Major | . | Bibin A Chundatt | Bilwa S T
YARN-7748 | TestContainerResizing.testIncreaseContainerUnreservedWhenApplicationCompleted fails due to multiple container fail events | Major | capacityscheduler | Haibo Chen | Weiwei Yang
YARN-8577 | Fix the broken anchor in SLS site-doc | Minor | documentation | Weiwei Yang | Weiwei Yang
YARN-4606 | CapacityScheduler: applications could get starved because computation of #activeUsers considers pending apps | Critical | capacity scheduler, capacityscheduler | Karam Singh | Manikandan R
HDFS-13765 | Fix javadoc for FSDirMkdirOp#createParentDirectories | Minor | documentation | Lokesh Jain | Lokesh Jain
YARN-8434 | Update federation documentation of Nodemanager configurations | Minor | . | Bibin A Chundatt | Bibin A Chundatt
YARN-8558 | NM recovery level db not cleaned up properly on container finish | Critical | . | Bibin A Chundatt | Bibin A Chundatt
HADOOP-15637 | LocalFs#listLocatedStatus does not filter out hidden .crc files | Minor | fs | Erik Krogen | Erik Krogen
YARN-8397 | Potential thread leak in ActivitiesManager | Major | . | Rohith Sharma K S | Rohith Sharma K S
YARN-6966 | NodeManager metrics may return wrong negative values when NM restart | Major | . | Yang Wang | Szilard Nemeth
HDFS-13786 | EC: Display erasure coding policy for sub-directories is not working | Major | erasure-coding | Souryakanta Dwivedy | Ayush Saxena
YARN-8331 | Race condition in NM container launched after done | Major | . | Yang Wang | Pradeep Ambati
HADOOP-15638 | KMS Accept Queue Size default changed from 500 to 128 in Hadoop 3.x | Major | kms | Wei-Chiu Chuang | Wei-Chiu Chuang
HDFS-13738 | fsck -list-corruptfileblocks has infinite loop if user is not privileged. | Major | tools | Wei-Chiu Chuang | Yuen-Kuei Hsueh
HDFS-13758 | DatanodeManager should throw exception if it has BlockRecoveryCommand but the block is not under construction | Major | namenode | Wei-Chiu Chuang | chencan
YARN-8614 | Fix few annotation typos in YarnConfiguration | Trivial | . | Sen Zhao | Sen Zhao
HDFS-13746 | Still occasional “Should be different group” failure in TestRefreshUserMappings#testGroupMappingRefresh | Major | . | Siyao Meng | Siyao Meng
HDFS-10240 | Race between close/recoverLease leads to missing block | Major | . | zhouyingchao | Jinglun
YARN-8612 | Fix NM Collector Service Port issue in YarnConfiguration | Major | ATSv2 | Prabha Manepalli | Prabha Manepalli
HDFS-13747 | Statistic for list_located_status is incremented incorrectly by listStatusIterator | Minor | hdfs-client | Todd Lipcon | Antal Mihalyi
HADOOP-15674 | Test failure TestSSLHttpServer.testExcludedCiphers with TLS_ECDHE_RSA_WITH_AES_128_CBC_SHA256 cipher suite | Major | common | Gabor Bota | Szilard Nemeth
YARN-8640 | Restore previous state in container-executor after failure | Major | . | Jim Brennan | Jim Brennan
YARN-8679 | [ATSv2] If HBase cluster is down for long time, high chances that NM ContainerManager dispatcher get blocked | Major | . | Rohith Sharma K S | Wangda Tan
HADOOP-14314 | The OpenSolaris taxonomy link is dead in InterfaceClassification.md | Major | documentation | Daniel Templeton | Rui Gao
YARN-8649 | NPE in localizer hearbeat processing if a container is killed while localizing | Major | . | lujie | lujie
YARN-8719 | Typo correction for yarn configuration in OpportunisticContainers(federation) docs | Major | documentation, federation | Y. SREENIVASULU REDDY | Y. SREENIVASULU REDDY
HDFS-13731 | ReencryptionUpdater fails with ConcurrentModificationException during processCheckpoints | Major | encryption | Xiao Chen | Zsolt Venczel
HADOOP-15705 | Typo in the definition of “stable” in the interface classification | Minor | . | Daniel Templeton | Daniel Templeton
HDFS-13863 | FsDatasetImpl should log DiskOutOfSpaceException | Major | hdfs | Fei Hui | Fei Hui
HADOOP-15698 | KMS log4j is not initialized properly at startup | Major | kms | Kitti Nanasi | Kitti Nanasi
HADOOP-15706 | Typo in compatibility doc: SHOUD -> SHOULD | Trivial | . | Daniel Templeton | Laszlo Kollar
HDFS-13027 | Handle possible NPEs due to deleted blocks in race condition | Major | namenode | Vinayakumar B | Vinayakumar B

### TESTS:

JIRA | Summary | Priority | Component | Reporter | Contributor
--- | --- | --- | --- | --- | ---
HDFS-13632 | Randomize baseDir for MiniJournalCluster in MiniQJMHACluster for TestDFSAdminWithHA | Minor | . | Anbang Hu | Anbang Hu
HDFS-13651 | TestReencryptionHandler fails on Windows | Minor | . | Anbang Hu | Anbang Hu
MAPREDUCE-7102 | Fix TestJavaSerialization for Windows due a mismatch line separator | Minor | . | Giovanni Matteo Fumarola | Giovanni Matteo Fumarola
MAPREDUCE-7105 | Fix TestNativeCollectorOnlyHandler.testOnCall on Windows because of the path format | Minor | . | Giovanni Matteo Fumarola | Giovanni Matteo Fumarola
HDFS-13652 | Randomize baseDir for MiniDFSCluster in TestBlockScanner | Minor | . | Anbang Hu | Anbang Hu
HDFS-13649 | Randomize baseDir for MiniDFSCluster in TestReconstructStripedFile and TestReconstructStripedFileWithRandomECPolicy | Minor | . | Anbang Hu | Anbang Hu
HDFS-13650 | Randomize baseDir for MiniDFSCluster in TestDFSStripedInputStream and TestDFSStripedInputStreamWithRandomECPolicy | Minor | . | Anbang Hu | Anbang Hu
YARN-8370 | Some Node Manager tests fail on Windows due to improper path/file separator | Minor | . | Anbang Hu | Anbang Hu
YARN-8422 | TestAMSimulator failing with NPE | Minor | . | Giovanni Matteo Fumarola | Giovanni Matteo Fumarola
HADOOP-15532 | TestBasicDiskValidator fails with NoSuchFileException | Minor | . | Íñigo Goiri | Giovanni Matteo Fumarola
HDFS-13563 | TestDFSAdminWithHA times out on Windows | Minor | . | Anbang Hu | Lukas Majercak
HDFS-13681 | Fix TestStartup.testNNFailToStartOnReadOnlyNNDir test failure on Windows | Major | test | Xiao Liang | Xiao Liang

### SUB-TASKS:

JIRA | Summary | Priority | Component | Reporter | Contributor
--- | --- | --- | --- | --- | ---
HDFS-12978 | Fine-grained locking while consuming journal stream. | Major | namenode | Konstantin Shvachko | Konstantin Shvachko
HDFS-13637 | RBF: Router fails when threadIndex (in ConnectionPool) wraps around Integer.MIN_VALUE | Critical | federation | CR Hota | CR Hota
HDFS-13281 | Namenode#createFile should be /.reserved/raw/ aware. | Critical | encryption | Rushabh S Shah | Rushabh S Shah
YARN-4677 | RMNodeResourceUpdateEvent update from scheduler can lead to race condition | Major | graceful, resourcemanager, scheduler | Brook Zhou | Wilfred Spiegelenburg
HADOOP-15506 | Upgrade Azure Storage Sdk version to 7.0.0 and update corresponding code blocks | Minor | fs/azure | Esfandiar Manii | Esfandiar Manii
HADOOP-15529 | ContainerLaunch#testInvalidEnvVariableSubstitutionType is not supported in Windows | Minor | . | Giovanni Matteo Fumarola | Giovanni Matteo Fumarola
HADOOP-15533 | Make WASB listStatus messages consistent | Trivial | fs/azure | Esfandiar Manii | Esfandiar Manii
HADOOP-15458 | TestLocalFileSystem#testFSOutputStreamBuilder fails on Windows | Minor | test | Xiao Liang | Xiao Liang
HDFS-13726 | RBF: Fix RBF configuration links | Minor | documentation | Takanobu Asanuma | Takanobu Asanuma
HDFS-13475 | RBF: Admin cannot enforce Router enter SafeMode | Major | . | Wei Yan | Chao Sun
HDFS-13733 | RBF: Add Web UI configurations and descriptions to RBF document | Minor | documentation | Takanobu Asanuma | Takanobu Asanuma
HDFS-13743 | RBF: Router throws NullPointerException due to the invalid initialization of MountTableResolver | Major | . | Takanobu Asanuma | Takanobu Asanuma
HDFS-13750 | RBF: Router ID in RouterRpcClient is always null | Major | . | Takanobu Asanuma | Takanobu Asanuma
YARN-8129 | Improve error message for invalid value in fields attribute | Minor | ATSv2 | Charan Hebri | Abhishek Modi
HDFS-13848 | Refactor NameNode failover proxy providers | Major | ha, hdfs-client | Konstantin Shvachko | Konstantin Shvachko

### OTHER:

JIRA | Summary | Priority | Component | Reporter | Contributor
--- | --- | --- | --- | --- | ---
HDFS-13788 | Update EC documentation about rack fault tolerance | Major | documentation, erasure-coding | Xiao Chen | Kitti Nanasi