Apache Hadoop 3.5.0 – Apache Hadoop Changelog

##### General

- Overview
- Single Node Setup
- Cluster Setup
- Commands Reference
- FileSystem Shell
- Compatibility Specification
- Downstream Developer's Guide
- Admin Compatibility Guide
- Interface Classification
- FileSystem Specification

##### Common

- CLI Mini Cluster
- Fair Call Queue
- Native Libraries
- Proxy User
- Rack Awareness
- Secure Mode
- Service Level Authorization
- HTTP Authentication
- Credential Provider API
- Hadoop KMS
- Tracing
- Unix Shell Guide
- Registry
- Async Profiler

##### HDFS

- Architecture
- User Guide
- Commands Reference
- NameNode HA With QJM
- NameNode HA With NFS
- Observer NameNode
- Federation
- ViewFs
- ViewFsOverloadScheme
- Snapshots
- Edits Viewer
- Image Viewer
- Permissions and HDFS
- Quotas and HDFS
- libhdfs (C API)
- WebHDFS (REST API)
- HttpFS
- Short Circuit Local Reads
- Centralized Cache Management
- NFS Gateway
- Rolling Upgrade
- Extended Attributes
- Transparent Encryption
- Multihoming
- Storage Policies
- Memory Storage Support
- Synthetic Load Generator
- Erasure Coding
- Disk Balancer
- Upgrade Domain
- DataNode Admin
- Router Federation
- Provided Storage

##### MapReduce

- Tutorial
- Commands Reference
- Encrypted Shuffle
- Pluggable Shuffle/Sort
- Distributed Cache Deploy
- Support for YARN Shared Cache

##### MapReduce REST APIs

- MR Application Master
- MR History Server

##### YARN

- Architecture
- Commands Reference
- Capacity Scheduler
- Fair Scheduler
- ResourceManager Restart
- ResourceManager HA
- Resource Model
- Node Labels
- Node Attributes
- Web Application Proxy
- Timeline Server
- Timeline Service V.2
- Writing YARN Applications
- YARN Application Security
- NodeManager
- Running Applications in Docker Containers
- Running Applications in runC Containers
- Using CGroups
- Secure Containers
- Reservation System
- Graceful Decommission
- Opportunistic Containers
- YARN Federation
- Shared Cache
- Using GPU
- Using FPGA
- Placement Constraints
- YARN UI2

##### YARN REST APIs

- Introduction
- Resource Manager
- Node Manager
- Timeline Server
- Timeline Service V.2

##### YARN Service

- Overview
- QuickStart
- Concepts
- Yarn Service API
- Service Discovery
- System Services

##### Hadoop Compatible File Systems

- Aliyun OSS
- Amazon S3
- Azure Blob Storage
- Azure Data Lake Storage
- Tencent COS
- Huaweicloud OBS
- VolcanoEngine TOS

##### Auth

- Overview
- Examples
- Configuration
- Building

##### Tools

- Hadoop Streaming
- Hadoop Archives
- Hadoop Archive Logs
- DistCp
- HDFS Federation Balance
- GridMix
- Rumen
- Resource Estimator Service
- Scheduler Load Simulator
- Hadoop Benchmarking
- Dynamometer

##### Reference

- Changelog and Release Notes
- Java API docs
- Unix Shell API
- Metrics

##### Configuration

- core-default.xml
- hdfs-default.xml
- hdfs-rbf-default.xml
- mapred-default.xml
- yarn-default.xml
- kms-default.xml
- httpfs-default.xml
- Deprecated Properties

# Apache Hadoop Changelog

## Release 2.8.4 - 2018-05-15

### IMPROVEMENTS:

JIRA | Summary | Priority | Component | Reporter | Contributor
--- | --- | --- | --- | --- | ---
HADOOP-12817 | Enable TLS v1.1 and 1.2 | Major | security | Robert Kanter | Robert Kanter
HDFS-11409 | DatanodeInfo getNetworkLocation and setNetworkLocation shoud use volatile instead of synchronized | Minor | performance | Chen Liang | Chen Liang
HADOOP-14246 | Authentication Tokens should use SecureRandom instead of Random and 256 bit secrets | Major | security | Robert Kanter | Robert Kanter
HADOOP-14920 | KMSClientProvider won’t work with KMS delegation token retrieved from non-Java client. | Major | kms | Xiaoyu Yao | Xiaoyu Yao
HADOOP-14987 | Improve KMSClientProvider log around delegation token checking | Major | . | Xiaoyu Yao | Xiaoyu Yao
YARN-7495 | Improve robustness of the AggregatedLogDeletionService | Major | log-aggregation | Jonathan Eagles | Jonathan Eagles
YARN-7678 | Ability to enable logging of container memory stats | Major | nodemanager | Jim Brennan | Jim Brennan
YARN-7590 | Improve container-executor validation check | Major | security, yarn | Eric Yang | Eric Yang
HADOOP-15212 | Add independent secret manager method for logging expired tokens | Major | security | Daryn Sharp | Daryn Sharp
YARN-7728 | Expose container preemptions related information in Capacity Scheduler queue metrics | Major | . | Eric Payne | Eric Payne
MAPREDUCE-7048 | Uber AM can crash due to unknown task in statusUpdate | Major | mr-am | Peter Bacsko | Peter Bacsko
HADOOP-13972 | ADLS to support per-store configuration | Major | fs/adl | John Zhuge | Sharad Sonker
YARN-7813 | Capacity Scheduler Intra-queue Preemption should be configurable for each queue | Major | capacity scheduler, scheduler preemption | Eric Payne | Eric Payne
HDFS-11187 | Optimize disk access for last partial chunk checksum of Finalized replica | Major | datanode | Wei-Chiu Chuang | Gabor Bota
HADOOP-15279 | increase maven heap size recommendations | Minor | build, documentation, test | Allen Wittenauer | Allen Wittenauer
HDFS-12884 | BlockUnderConstructionFeature.truncateBlock should be of type BlockInfo | Major | namenode | Konstantin Shvachko | chencan
HADOOP-14841 | Kms client should disconnect if unable to get output stream from connection. | Major | kms | Xiao Chen | Rushabh S Shah

### BUG FIXES:

JIRA | Summary | Priority | Component | Reporter | Contributor
--- | --- | --- | --- | --- | ---
HADOOP-13988 | KMSClientProvider does not work with WebHDFS and Apache Knox w/ProxyUser | Major | common, kms | Greg Senia | Xiaoyu Yao
HADOOP-14029 | Fix KMSClientProvider for non-secure proxyuser use case | Major | kms | Xiaoyu Yao | Xiaoyu Yao
HDFS-12614 | FSPermissionChecker#getINodeAttrs() throws NPE when INodeAttributesProvider configured | Major | . | Manoj Govindassamy | Manoj Govindassamy
YARN-7489 | ConcurrentModificationException in RMAppImpl#getRMAppMetrics | Major | capacityscheduler | Tao Yang | Tao Yang
YARN-7525 | Incorrect query parameters in cluster nodes REST API document | Minor | documentation | Tao Yang | Tao Yang
HDFS-12754 | Lease renewal can hit a deadlock | Major | . | Kuhu Shukla | Kuhu Shukla
HDFS-11754 | Make FsServerDefaults cache configurable. | Minor | . | Rushabh S Shah | Mikhail Erofeev
MAPREDUCE-5124 | AM lacks flow control for task events | Major | mr-am | Jason Lowe | Peter Bacsko
HDFS-12833 | Distcp : Update the usage of delete option for dependency with update and overwrite option | Minor | distcp, hdfs | Harshakiran Reddy | usharani
HDFS-12907 | Allow read-only access to reserved raw for non-superusers | Major | namenode | Daryn Sharp | Rushabh S Shah
HDFS-12881 | Output streams closed with IOUtils suppressing write errors | Major | . | Jason Lowe | Ajay Kumar
YARN-7595 | Container launching code suppresses close exceptions after writes | Major | nodemanager | Jason Lowe | Jim Brennan
YARN-7661 | NodeManager metrics return wrong value after update node resource | Major | . | Yang Wang | Yang Wang
HDFS-12347 | TestBalancerRPCDelay#testBalancerRPCDelay fails very frequently | Critical | test | Xiao Chen | Bharat Viswanadham
HADOOP-15143 | NPE due to Invalid KerberosTicket in UGI | Major | . | Jitendra Nath Pandey | Mukul Kumar Singh
MAPREDUCE-7028 | Concurrent task progress updates causing NPE in Application Master | Blocker | mr-am | Gergo Repas | Gergo Repas
YARN-7619 | Max AM Resource value in Capacity Scheduler UI has to be refreshed for every user | Major | capacity scheduler, yarn | Eric Payne | Eric Payne
YARN-7663 | RMAppImpl:Invalid event: START at KILLED | Major | resourcemanager | lujie | lujie
YARN-6948 | Invalid event: ATTEMPT_ADDED at FINAL_SAVING | Major | yarn | lujie | lujie
HDFS-6804 | Add test for race condition between transferring block and appending block causes “Unexpected checksum mismatch exception” | Major | datanode | Gordon Wang | Brahma Reddy Battula
HDFS-9049 | Make Datanode Netty reverse proxy port to be configurable | Major | datanode | Vinayakumar B | Vinayakumar B
HADOOP-15150 | in FsShell, UGI params should be overidden through env vars(-D arg) | Major | . | Brahma Reddy Battula | Brahma Reddy Battula
HADOOP-15181 | Typo in SecureMode.md | Trivial | documentation | Masahiro Tanaka | Masahiro Tanaka
YARN-7102 | NM heartbeat stuck when responseId overflows MAX_INT | Critical | . | Botong Huang | Botong Huang
MAPREDUCE-7020 | Task timeout in uber mode can crash AM | Major | mr-am | Akira Ajisaka | Peter Bacsko
HDFS-13100 | Handle IllegalArgumentException when GETSERVERDEFAULTS is not implemented in webhdfs. | Critical | hdfs, webhdfs | Yongjun Zhang | Yongjun Zhang
YARN-7849 | TestMiniYarnClusterNodeUtilization#testUpdateNodeUtilization fails due to heartbeat sync error | Major | test | Jason Lowe | Botong Huang
HDFS-13120 | Snapshot diff could be corrupted after concat | Major | namenode, snapshots | Xiaoyu Yao | Xiaoyu Yao
HDFS-10453 | ReplicationMonitor thread could stuck for long time due to the race between replication and delete of same file in a large cluster. | Major | namenode | He Xiaoqiao | He Xiaoqiao
HDFS-8693 | refreshNamenodes does not support adding a new standby to a running DN | Critical | datanode, ha | Jian Fang | Ajith S
MAPREDUCE-7052 | TestFixedLengthInputFormat#testFormatCompressedIn is flaky | Major | client, test | Peter Bacsko | Peter Bacsko
HDFS-13112 | Token expiration edits may cause log corruption or deadlock | Critical | namenode | Daryn Sharp | Daryn Sharp
MAPREDUCE-7053 | Timed out tasks can fail to produce thread dump | Major | . | Jason Lowe | Jason Lowe
HADOOP-15206 | BZip2 drops and duplicates records when input split size is small | Major | . | Aki Tanaka | Aki Tanaka
YARN-7947 | Capacity Scheduler intra-queue preemption can NPE for non-schedulable apps | Major | capacity scheduler, scheduler preemption | Eric Payne | Eric Payne
HADOOP-14903 | Add json-smart explicitly to pom.xml | Major | common | Ray Chiang | Ray Chiang
HDFS-12070 | Failed block recovery leaves files open indefinitely and at risk for data loss | Major | . | Daryn Sharp | Kihwal Lee
HADOOP-15283 | Upgrade from findbugs 3.0.1 to spotbugs 3.1.2 in branch-2 to fix docker image build | Major | . | Xiao Chen | Akira Ajisaka
HDFS-13164 | File not closed if streamer fail with DSQuotaExceededException | Major | hdfs-client | Xiao Chen | Xiao Chen
HDFS-13109 | Support fully qualified hdfs path in EZ commands | Major | hdfs | Hanisha Koneru | Hanisha Koneru
MAPREDUCE-6930 | mapreduce.map.cpu.vcores and mapreduce.reduce.cpu.vcores are both present twice in mapred-default.xml | Major | mrv2 | Daniel Templeton | Sen Zhao
HDFS-12156 | TestFSImage fails without -Pnative | Major | test | Akira Ajisaka | Akira Ajisaka
YARN-8054 | Improve robustness of the LocalDirsHandlerService MonitoringTimerTask thread | Major | . | Jonathan Eagles | Jonathan Eagles
HDFS-13195 | DataNode conf page cannot display the current value after reconfig | Minor | datanode | maobaolong | maobaolong
HADOOP-12862 | LDAP Group Mapping over SSL can not specify trust store | Major | . | Wei-Chiu Chuang | Wei-Chiu Chuang
HADOOP-15317 | Improve NetworkTopology chooseRandom’s loop | Major | . | Xiao Chen | Xiao Chen
HDFS-13427 | Fix the section titles of transparent encryption document | Minor | documentation | Akira Ajisaka | Akira Ajisaka
YARN-8120 | JVM can crash with SIGSEGV when exiting due to custom leveldb logger | Major | nodemanager, resourcemanager | Jason Lowe | Jason Lowe
YARN-8147 | TestClientRMService#testGetApplications sporadically fails | Major | test | Jason Lowe | Jason Lowe
HADOOP-14970 | MiniHadoopClusterManager doesn’t respect lack of format option | Minor | . | Erik Krogen | Erik Krogen
HADOOP-15180 | branch-2 : daemon processes’ sysout overwrites ‘ulimit -a’ in daemon’s out file | Minor | scripts | Ranith Sardar | Ranith Sardar
HDFS-10183 | Prevent race condition during class initialization | Minor | fs | Pavel Avgustinov | Pavel Avgustinov
HADOOP-15390 | Yarn RM logs flooded by DelegationTokenRenewer trying to renew KMS tokens | Critical | . | Xiao Chen | Xiao Chen
HADOOP-15385 | Many tests are failing in hadoop-distcp project in branch-2 | Critical | tools/distcp | Rushabh S Shah | Jason Lowe
MAPREDUCE-7073 | Optimize TokenCache#obtainTokensForNamenodesInternal | Major | . | Bibin A Chundatt | Bibin A Chundatt

### SUB-TASKS:

JIRA | Summary | Priority | Component | Reporter | Contributor
--- | --- | --- | --- | --- | ---
HADOOP-13174 | Add more debug logs for delegation tokens and authentication | Minor | security | Xiao Chen | Xiao Chen
HADOOP-14799 | Update nimbus-jose-jwt to 4.41.1 | Major | . | Ray Chiang | Ray Chiang
HDFS-12396 | Webhdfs file system should get delegation token from kms provider. | Major | encryption, kms, webhdfs | Rushabh S Shah | Rushabh S Shah
HDFS-12574 | Add CryptoInputStream to WebHdfsFileSystem read call. | Major | encryption, kms, webhdfs | Rushabh S Shah | Rushabh S Shah

### OTHER:

JIRA | Summary | Priority | Component | Reporter | Contributor
--- | --- | --- | --- | --- | ---
HADOOP-15177 | Update the release year to 2018 | Blocker | build | Akira Ajisaka | Bharat Viswanadham