Apache Hadoop 3.5.0 – Apache Hadoop Changelog

##### General

- Overview
- Single Node Setup
- Cluster Setup
- Commands Reference
- FileSystem Shell
- Compatibility Specification
- Downstream Developer's Guide
- Admin Compatibility Guide
- Interface Classification
- FileSystem Specification

##### Common

- CLI Mini Cluster
- Fair Call Queue
- Native Libraries
- Proxy User
- Rack Awareness
- Secure Mode
- Service Level Authorization
- HTTP Authentication
- Credential Provider API
- Hadoop KMS
- Tracing
- Unix Shell Guide
- Registry
- Async Profiler

##### HDFS

- Architecture
- User Guide
- Commands Reference
- NameNode HA With QJM
- NameNode HA With NFS
- Observer NameNode
- Federation
- ViewFs
- ViewFsOverloadScheme
- Snapshots
- Edits Viewer
- Image Viewer
- Permissions and HDFS
- Quotas and HDFS
- libhdfs (C API)
- WebHDFS (REST API)
- HttpFS
- Short Circuit Local Reads
- Centralized Cache Management
- NFS Gateway
- Rolling Upgrade
- Extended Attributes
- Transparent Encryption
- Multihoming
- Storage Policies
- Memory Storage Support
- Synthetic Load Generator
- Erasure Coding
- Disk Balancer
- Upgrade Domain
- DataNode Admin
- Router Federation
- Provided Storage

##### MapReduce

- Tutorial
- Commands Reference
- Encrypted Shuffle
- Pluggable Shuffle/Sort
- Distributed Cache Deploy
- Support for YARN Shared Cache

##### MapReduce REST APIs

- MR Application Master
- MR History Server

##### YARN

- Architecture
- Commands Reference
- Capacity Scheduler
- Fair Scheduler
- ResourceManager Restart
- ResourceManager HA
- Resource Model
- Node Labels
- Node Attributes
- Web Application Proxy
- Timeline Server
- Timeline Service V.2
- Writing YARN Applications
- YARN Application Security
- NodeManager
- Running Applications in Docker Containers
- Running Applications in runC Containers
- Using CGroups
- Secure Containers
- Reservation System
- Graceful Decommission
- Opportunistic Containers
- YARN Federation
- Shared Cache
- Using GPU
- Using FPGA
- Placement Constraints
- YARN UI2

##### YARN REST APIs

- Introduction
- Resource Manager
- Node Manager
- Timeline Server
- Timeline Service V.2

##### YARN Service

- Overview
- QuickStart
- Concepts
- Yarn Service API
- Service Discovery
- System Services

##### Hadoop Compatible File Systems

- Aliyun OSS
- Amazon S3
- Azure Blob Storage
- Azure Data Lake Storage
- Tencent COS
- Huaweicloud OBS
- VolcanoEngine TOS

##### Auth

- Overview
- Examples
- Configuration
- Building

##### Tools

- Hadoop Streaming
- Hadoop Archives
- Hadoop Archive Logs
- DistCp
- HDFS Federation Balance
- GridMix
- Rumen
- Resource Estimator Service
- Scheduler Load Simulator
- Hadoop Benchmarking
- Dynamometer

##### Reference

- Changelog and Release Notes
- Java API docs
- Unix Shell API
- Metrics

##### Configuration

- core-default.xml
- hdfs-default.xml
- hdfs-rbf-default.xml
- mapred-default.xml
- yarn-default.xml
- kms-default.xml
- httpfs-default.xml
- Deprecated Properties

# Apache Hadoop Changelog

## Release 2.6.4 - 2016-02-11

### INCOMPATIBLE CHANGES:

JIRA | Summary | Priority | Component | Reporter | Contributor
--- | --- | --- | --- | --- | ---
YARN-3154 | Should not upload partial logs for MR jobs or other "short-running’ applications | Blocker | nodemanager, resourcemanager | Xuan Gong | Xuan Gong
HADOOP-11252 | RPC client does not time out by default | Critical | ipc | Wilfred Spiegelenburg | Masatake Iwasaki

### IMPROVEMENTS:

JIRA | Summary | Priority | Component | Reporter | Contributor
--- | --- | --- | --- | --- | ---
HDFS-7694 | FSDataInputStream should support “unbuffer” | Major | . | Colin P. McCabe | Colin P. McCabe
HDFS-8722 | Optimize datanode writes for small writes and flushes | Critical | . | Kihwal Lee | Kihwal Lee
HDFS-8647 | Abstract BlockManager’s rack policy into BlockPlacementPolicy | Major | . | Ming Ma | Brahma Reddy Battula
HDFS-9314 | Improve BlockPlacementPolicyDefault’s picking of excess replicas | Major | . | Ming Ma | Xiao Chen
MAPREDUCE-6436 | JobHistory cache issue | Blocker | . | Ryu Kobayashi | Kai Sasaki
HDFS-9415 | Document dfs.cluster.administrators and dfs.permissions.superusergroup | Major | documentation | Arpit Agarwal | Xiaobing Zhou

### BUG FIXES:

JIRA | Summary | Priority | Component | Reporter | Contributor
--- | --- | --- | --- | --- | ---
YARN-2975 | FSLeafQueue app lists are accessed without required locks | Blocker | . | Karthik Kambatla | Karthik Kambatla
HDFS-6945 | BlockManager should remove a block from excessReplicateMap and decrement ExcessBlocks metric when the block is removed | Critical | namenode | Akira Ajisaka | Akira Ajisaka
HDFS-4660 | Block corruption can happen during pipeline recovery | Blocker | datanode | Peng Zhang | Kihwal Lee
YARN-3842 | NMProxy should retry on NMNotYetReadyException | Critical | . | Karthik Kambatla | Robert Kanter
YARN-3695 | ServerProxy (NMProxy, etc.) shouldn’t retry forever for non network exception. | Major | . | Junping Du | Raju Bairishetti
HADOOP-12107 | long running apps may have a huge number of StatisticsData instances under FileSystem | Critical | fs | Sangjin Lee | Sangjin Lee
YARN-3849 | Too much of preemption activity causing continuos killing of containers across queues | Critical | capacityscheduler | Sunil Govindan | Sunil Govindan
HDFS-8767 | RawLocalFileSystem.listStatus() returns null for UNIX pipefile | Critical | . | Haohui Mai | Kanaka Kumar Avvaru
YARN-3535 | Scheduler must re-request container resources when RMContainer transitions from ALLOCATED to KILLED | Critical | capacityscheduler, fairscheduler, resourcemanager | Peng Zhang | Peng Zhang
YARN-3857 | Memory leak in ResourceManager with SIMPLE mode | Critical | resourcemanager | mujunchao | mujunchao
MAPREDUCE-5982 | Task attempts that fail from the ASSIGNED state can disappear | Major | mr-am | Jason Lowe | Chang Li
YARN-3697 | FairScheduler: ContinuousSchedulingThread can fail to shutdown | Critical | fairscheduler | zhihai xu | zhihai xu
MAPREDUCE-6492 | AsyncDispatcher exit with NPE on TaskAttemptImpl#sendJHStartEventForAssignedFailTask | Critical | . | Bibin A Chundatt | Bibin A Chundatt
YARN-4180 | AMLauncher does not retry on failures when talking to NM | Critical | resourcemanager | Anubhav Dhoot | Anubhav Dhoot
HDFS-9178 | Slow datanode I/O can cause a wrong node to be marked bad | Critical | . | Kihwal Lee | Kihwal Lee
HDFS-9220 | Reading small file (< 512 bytes) that is open for append fails due to incorrect checksum | Blocker | . | Bogdan Raducanu | Jing Zhao
HDFS-9313 | Possible NullPointerException in BlockManager if no excess replica can be chosen | Major | . | Ming Ma | Ming Ma
YARN-4354 | Public resource localization fails with NPE | Blocker | nodemanager | Jason Lowe | Jason Lowe
YARN-4380 | TestResourceLocalizationService.testDownloadingResourcesOnContainerKill fails intermittently | Major | test | Tsuyoshi Ozawa | Varun Saxena
HDFS-9294 | DFSClient deadlock when close file and failed to renew lease | Blocker | hdfs-client | DENG FEI | Brahma Reddy Battula
YARN-4452 | NPE when submit Unmanaged application | Critical | . | Naganarasimha G R | Naganarasimha G R
HDFS-9445 | Datanode may deadlock while handling a bad volume | Blocker | . | Kihwal Lee | Walter Su
MAPREDUCE-6577 | MR AM unable to load native library without MR_AM_ADMIN_USER_ENV set | Critical | mr-am | Sangjin Lee | Sangjin Lee
YARN-4546 | ResourceManager crash due to scheduling opportunity overflow | Critical | resourcemanager | Jason Lowe | Jason Lowe
HDFS-9600 | do not check replication if the block is under construction | Critical | . | Phil Yang | Phil Yang
HDFS-9574 | Reduce client failures during datanode restart | Major | . | Kihwal Lee | Kihwal Lee
YARN-4414 | Nodemanager connection errors are retried at multiple levels | Major | nodemanager | Jason Lowe | Chang Li
HADOOP-12706 | TestLocalFsFCStatistics#testStatisticsThreadLocalDataCleanUp times out occasionally | Major | test | Jason Lowe | Sangjin Lee
YARN-4581 | AHS writer thread leak makes RM crash while RM is recovering | Major | resourcemanager | sandflee | sandflee
MAPREDUCE-6554 | MRAppMaster servicestart failing with NPE in MRAppMaster#parsePreviousJobHistory | Critical | . | Bibin A Chundatt | Bibin A Chundatt
YARN-4598 | Invalid event: RESOURCE_FAILED at CONTAINER_CLEANEDUP_AFTER_KILL | Major | nodemanager | tangshangwen | tangshangwen
MAPREDUCE-6619 | HADOOP_CLASSPATH is overwritten in MR container | Major | mrv2 | shanyu zhao | Junping Du
MAPREDUCE-6618 | YarnClientProtocolProvider leaking the YarnClient thread. | Major | . | Xuan Gong | Xuan Gong
MAPREDUCE-6621 | Memory Leak in JobClient#submitJobInternal() | Major | . | Xuan Gong | Xuan Gong
MAPREDUCE-6363 | [NNBench] Lease mismatch error when running with multiple mappers | Critical | benchmarks | Brahma Reddy Battula | Bibin A Chundatt

### TESTS:

JIRA | Summary | Priority | Component | Reporter | Contributor
--- | --- | --- | --- | --- | ---
HADOOP-12736 | TestTimedOutTestsListener#testThreadDumpAndDeadlocks sometimes times out | Major | . | Xiao Chen | Xiao Chen

### SUB-TASKS:

JIRA | Summary | Priority | Component | Reporter | Contributor
--- | --- | --- | --- | --- | ---
YARN-3893 | Both RM in active state when Admin#transitionToActive failure from refeshAll() | Critical | resourcemanager | Bibin A Chundatt | Bibin A Chundatt
YARN-2902 | Killing a container that is localizing can orphan resources in the DOWNLOADING state | Major | nodemanager | Jason Lowe | Varun Saxena