Apache Hadoop 3.5.0 – Apache Hadoop Changelog

##### General

- Overview
- Single Node Setup
- Cluster Setup
- Commands Reference
- FileSystem Shell
- Compatibility Specification
- Downstream Developer's Guide
- Admin Compatibility Guide
- Interface Classification
- FileSystem Specification

##### Common

- CLI Mini Cluster
- Fair Call Queue
- Native Libraries
- Proxy User
- Rack Awareness
- Secure Mode
- Service Level Authorization
- HTTP Authentication
- Credential Provider API
- Hadoop KMS
- Tracing
- Unix Shell Guide
- Registry
- Async Profiler

##### HDFS

- Architecture
- User Guide
- Commands Reference
- NameNode HA With QJM
- NameNode HA With NFS
- Observer NameNode
- Federation
- ViewFs
- ViewFsOverloadScheme
- Snapshots
- Edits Viewer
- Image Viewer
- Permissions and HDFS
- Quotas and HDFS
- libhdfs (C API)
- WebHDFS (REST API)
- HttpFS
- Short Circuit Local Reads
- Centralized Cache Management
- NFS Gateway
- Rolling Upgrade
- Extended Attributes
- Transparent Encryption
- Multihoming
- Storage Policies
- Memory Storage Support
- Synthetic Load Generator
- Erasure Coding
- Disk Balancer
- Upgrade Domain
- DataNode Admin
- Router Federation
- Provided Storage

##### MapReduce

- Tutorial
- Commands Reference
- Encrypted Shuffle
- Pluggable Shuffle/Sort
- Distributed Cache Deploy
- Support for YARN Shared Cache

##### MapReduce REST APIs

- MR Application Master
- MR History Server

##### YARN

- Architecture
- Commands Reference
- Capacity Scheduler
- Fair Scheduler
- ResourceManager Restart
- ResourceManager HA
- Resource Model
- Node Labels
- Node Attributes
- Web Application Proxy
- Timeline Server
- Timeline Service V.2
- Writing YARN Applications
- YARN Application Security
- NodeManager
- Running Applications in Docker Containers
- Running Applications in runC Containers
- Using CGroups
- Secure Containers
- Reservation System
- Graceful Decommission
- Opportunistic Containers
- YARN Federation
- Shared Cache
- Using GPU
- Using FPGA
- Placement Constraints
- YARN UI2

##### YARN REST APIs

- Introduction
- Resource Manager
- Node Manager
- Timeline Server
- Timeline Service V.2

##### YARN Service

- Overview
- QuickStart
- Concepts
- Yarn Service API
- Service Discovery
- System Services

##### Hadoop Compatible File Systems

- Aliyun OSS
- Amazon S3
- Azure Blob Storage
- Azure Data Lake Storage
- Tencent COS
- Huaweicloud OBS
- VolcanoEngine TOS

##### Auth

- Overview
- Examples
- Configuration
- Building

##### Tools

- Hadoop Streaming
- Hadoop Archives
- Hadoop Archive Logs
- DistCp
- HDFS Federation Balance
- GridMix
- Rumen
- Resource Estimator Service
- Scheduler Load Simulator
- Hadoop Benchmarking
- Dynamometer

##### Reference

- Changelog and Release Notes
- Java API docs
- Unix Shell API
- Metrics

##### Configuration

- core-default.xml
- hdfs-default.xml
- hdfs-rbf-default.xml
- mapred-default.xml
- yarn-default.xml
- kms-default.xml
- httpfs-default.xml
- Deprecated Properties

# Apache Hadoop Changelog

## Release 2.6.5 - 2016-10-08

### IMPORTANT ISSUES:

JIRA | Summary | Priority | Component | Reporter | Contributor
--- | --- | --- | --- | --- | ---
YARN-4785 | inconsistent value type of the “type” field for LeafQueueInfo in response of RM REST API - cluster/scheduler | Major | webapp | Jayesh | Varun Vasudev

### IMPROVEMENTS:

JIRA | Summary | Priority | Component | Reporter | Contributor
--- | --- | --- | --- | --- | ---
HDFS-2580 | NameNode#main(…) can make use of GenericOptionsParser. | Minor | namenode | Harsh J | Harsh J
HADOOP-11301 | [optionally] update jmx cache to drop old metrics | Major | . | Maysam Yabandeh | Maysam Yabandeh
HDFS-9669 | TcpPeerServer should respect ipc.server.listen.queue.size | Major | . | Elliott Clark | Elliott Clark
HDFS-9629 | Update the footer of Web UI to show year 2016 | Major | . | Xiao Chen | Xiao Chen
HADOOP-12805 | Annotate CanUnbuffer with @InterfaceAudience.Public | Major | . | Ted Yu | Ted Yu
YARN-4690 | Skip object allocation in FSAppAttempt#getResourceUsage when possible | Major | . | Ming Ma | Ming Ma
HADOOP-12800 | Copy docker directory from 2.8 to 2.7/2.6 repos to enable pre-commit Jenkins runs | Major | build, yetus | Zhe Zhang | Zhe Zhang
MAPREDUCE-6622 | Add capability to set JHS job cache to a task-based limit | Critical | jobhistoryserver | Ray Chiang | Ray Chiang
HADOOP-12789 | log classpath of ApplicationClassLoader at INFO level | Minor | util | Sangjin Lee | Sangjin Lee
HDFS-10264 | Logging improvements in FSImageFormatProtobuf.Saver | Major | namenode | Konstantin Shvachko | Xiaobing Zhou
HDFS-10377 | CacheReplicationMonitor shutdown log message should use INFO level. | Major | logging, namenode | Konstantin Shvachko | Yiqun Lin
HADOOP-13290 | Appropriate use of generics in FairCallQueue | Major | ipc | Konstantin Shvachko | Jonathan Hung
HADOOP-13298 | Fix the leftover L&N files in hadoop-build-tools/src/main/resources/META-INF/ | Minor | . | Xiao Chen | Tsuyoshi Ozawa
YARN-5483 | Optimize RMAppAttempt#pullJustFinishedContainers | Major | . | sandflee | sandflee

### BUG FIXES:

JIRA | Summary | Priority | Component | Reporter | Contributor
--- | --- | --- | --- | --- | ---
HDFS-7258 | CacheReplicationMonitor rescan schedule log should use DEBUG level instead of INFO level | Minor | namenode | Xiaoyu Yao | Xiaoyu Yao
HDFS-7530 | Allow renaming of encryption zone roots | Minor | namenode | Charles Lamb | Charles Lamb
HADOOP-7817 | RawLocalFileSystem.append() should give FSDataOutputStream with accurate .getPos() | Major | fs | Kristofer Tomasette | Kanaka Kumar Avvaru
MAPREDUCE-6413 | TestLocalJobSubmission is failing with unknown host | Major | test | Jason Lowe | zhihai xu
MAPREDUCE-5817 | Mappers get rescheduled on node transition even after all reducers are completed | Major | applicationmaster | Sangjin Lee | Sangjin Lee
HDFS-8845 | DiskChecker should not traverse the entire tree | Major | . | Chang Li | Chang Li
HDFS-8581 | ContentSummary on / skips further counts on yielding lock | Minor | namenode | tongshiquan | J.Andreina
HADOOP-12348 | MetricsSystemImpl creates MetricsSourceAdapter with wrong time unit parameter. | Major | metrics | zhihai xu | zhihai xu
MAPREDUCE-6302 | Preempt reducers after a configurable timeout irrespective of headroom | Critical | . | mai shurong | Karthik Kambatla
HADOOP-12482 | Race condition in JMX cache update | Major | . | Tony Wu | Tony Wu
HDFS-9347 | Invariant assumption in TestQuorumJournalManager.shutdown() is wrong | Major | test | Wei-Chiu Chuang | Wei-Chiu Chuang
HADOOP-12559 | KMS connection failures should trigger TGT renewal | Major | security | Zhe Zhang | Zhe Zhang
HADOOP-12682 | Fix TestKMS#testKMSRestart* failure | Major | . | Wei-Chiu Chuang | Wei-Chiu Chuang
HADOOP-12773 | HBase classes fail to load with client/job classloader enabled | Major | util | Sangjin Lee | Sangjin Lee
HDFS-9752 | Permanent write failures may happen to slow writers during datanode rolling upgrades | Critical | . | Kihwal Lee | Walter Su
HADOOP-12589 | Fix intermittent test failure of TestCopyPreserveFlag | Major | test | Tsuyoshi Ozawa | Masatake Iwasaki
HADOOP-12810 | FileSystem#listLocatedStatus causes unnecessary RPC calls | Major | fs, fs/s3 | Ryan Blue | Ryan Blue
MAPREDUCE-6637 | Testcase Failure : TestFileInputFormat.testSplitLocationInfo | Major | test | Brahma Reddy Battula | Brahma Reddy Battula
MAPREDUCE-6635 | Unsafe long to int conversion in UncompressedSplitLineReader and IndexOutOfBoundsException | Critical | . | Sergey Shelukhin | Junping Du
YARN-2046 | Out of band heartbeats are sent only on container kill and possibly too early | Major | nodemanager | Jason Lowe | Ming Ma
YARN-4722 | AsyncDispatcher logs redundant event queue sizes | Major | . | Jason Lowe | Jason Lowe
YARN-4761 | NMs reconnecting with changed capabilities can lead to wrong cluster resource calculations on fair scheduler | Major | fairscheduler | Sangjin Lee | Sangjin Lee
MAPREDUCE-6656 | [NNBench] OP_DELETE operation isn’t working after MAPREDUCE-6363 | Blocker | . | J.Andreina | J.Andreina
HADOOP-12958 | PhantomReference for filesystem statistics can trigger OOM | Major | . | Jason Lowe | Sangjin Lee
HDFS-10182 | Hedged read might overwrite user’s buf | Major | . | zhouyingchao | zhouyingchao
YARN-4773 | Log aggregation performs extraneous filesystem operations when rolling log aggregation is disabled | Minor | nodemanager | Jason Lowe | Jun Gong
HDFS-10178 | Permanent write failures can happen if pipeline recoveries occur for the first packet | Critical | . | Kihwal Lee | Kihwal Lee
HDFS-10271 | Extra bytes are getting released from reservedSpace for append | Critical | . | Brahma Reddy Battula | Brahma Reddy Battula
YARN-4794 | Deadlock in NMClientImpl | Critical | . | Sumana Sathish | Jian He
HADOOP-13042 | Restore lost leveldbjni LICENSE and NOTICE changes | Major | . | Andrew Wang | Andrew Wang
HADOOP-13043 | Add LICENSE.txt entries for bundled javascript dependencies | Major | . | Andrew Wang | Andrew Wang
HADOOP-13052 | ChecksumFileSystem mishandles crc file permissions | Major | fs | Daryn Sharp | Daryn Sharp
YARN-5009 | NMLeveldbStateStoreService database can grow substantially leading to longer recovery times | Major | nodemanager | Jason Lowe | Jason Lowe
MAPREDUCE-6514 | Job hangs as ask is not updated after ramping down of all reducers | Blocker | applicationmaster | Varun Saxena | Varun Saxena
MAPREDUCE-6689 | MapReduce job can infinitely increase number of reducer resource requests | Blocker | . | Wangda Tan | Wangda Tan
MAPREDUCE-6558 | multibyte delimiters with compressed input files generate duplicate records | Major | mrv1, mrv2 | Wilfred Spiegelenburg | Wilfred Spiegelenburg
HDFS-9365 | Balancer does not work with the HDFS-6376 HA setup | Major | balancer & mover | Tsz Wo Nicholas Sze | Tsz Wo Nicholas Sze
YARN-4459 | container-executor should only kill process groups | Major | nodemanager | Jun Gong | Jun Gong
HDFS-10458 | getFileEncryptionInfo should return quickly for non-encrypted cluster | Major | encryption, namenode | Zhe Zhang | Zhe Zhang
YARN-5206 | RegistrySecurity includes id:pass in exception text if considered invalid | Minor | client, security | Steve Loughran | Steve Loughran
HADOOP-13189 | FairCallQueue makes callQueue larger than the configured capacity. | Major | ipc | Konstantin Shvachko | Vinitha Reddy Gankidi
HADOOP-13255 | KMSClientProvider should check and renew tgt when doing delegation token operations. | Major | kms | Xiao Chen | Xiao Chen
HADOOP-13192 | org.apache.hadoop.util.LineReader cannot handle multibyte delimiters correctly | Critical | util | binde | binde
YARN-5197 | RM leaks containers if running container disappears from node update | Critical | resourcemanager | Jason Lowe | Jason Lowe
YARN-5262 | Optimize sending RMNodeFinishedContainersPulledByAMEvent for every AM heartbeat | Major | resourcemanager | Rohith Sharma K S | Rohith Sharma K S
HADOOP-13350 | Additional fix to LICENSE and NOTICE | Blocker | build | Xiao Chen | Xiao Chen
HADOOP-12893 | Verify LICENSE.txt and NOTICE.txt | Blocker | build | Allen Wittenauer | Xiao Chen
HADOOP-13297 | Add missing dependency in setting maven-remote-resource-plugin to fix builds | Major | build | Akira Ajisaka | Sean Busbey
YARN-5353 | ResourceManager can leak delegation tokens when they are shared across apps | Critical | resourcemanager | Jason Lowe | Jason Lowe
HADOOP-11361 | Fix a race condition in MetricsSourceAdapter.updateJmxCache | Major | . | Brahma Reddy Battula | Brahma Reddy Battula
HDFS-10544 | Balancer doesn’t work with IPFailoverProxyProvider | Major | balancer & mover, ha | Zhe Zhang | Zhe Zhang
YARN-5462 | TestNodeStatusUpdater.testNodeStatusUpdaterRetryAndNMShutdown fails intermittently | Major | . | Eric Badger | Eric Badger
HADOOP-13434 | Add quoting to Shell class | Major | . | Owen O’Malley | Owen O’Malley
HDFS-9696 | Garbage snapshot records lingering forever | Critical | . | Kihwal Lee | Kihwal Lee
HADOOP-13494 | ReconfigurableBase can log sensitive information | Major | security | Sean Mackrory | Sean Mackrory
HDFS-9530 | ReservedSpace is not cleared for abandoned Blocks | Critical | datanode | Fei Hui | Brahma Reddy Battula
HDFS-10763 | Open files can leak permanently due to inconsistent lease update | Critical | . | Kihwal Lee | Kihwal Lee
HADOOP-13579 | Fix source-level compatibility after HADOOP-11252 | Blocker | . | Akira Ajisaka | Tsuyoshi Ozawa
HDFS-10870 | Wrong dfs.namenode.acls.enabled default in HdfsPermissionsGuide.apt.vm | Trivial | documentation | John Zhuge | John Zhuge
YARN-5694 | ZKRMStateStore can prevent the transition to standby in branch-2.7 if the ZK node is unreachable | Critical | resourcemanager | Daniel Templeton | Daniel Templeton

### TESTS:

JIRA | Summary | Priority | Component | Reporter | Contributor
--- | --- | --- | --- | --- | ---
MAPREDUCE-6191 | TestJavaSerialization fails with getting incorrect MR job result | Minor | test | sam liu | sam liu
HDFS-9688 | Test the effect of nested encryption zones in HDFS downgrade | Major | encryption, test | Zhe Zhang | Zhe Zhang

### SUB-TASKS:

JIRA | Summary | Priority | Component | Reporter | Contributor
--- | --- | --- | --- | --- | ---
YARN-4393 | TestResourceLocalizationService#testFailedDirsResourceRelease fails intermittently | Major | test | Varun Saxena | Varun Saxena
YARN-4573 | TestRMAppTransitions.testAppRunningKill and testAppKilledKilled fail on trunk | Major | resourcemanager, test | Takashi Ohnishi | Takashi Ohnishi
HDFS-10653 | Optimize conversion from path string to components | Major | hdfs | Daryn Sharp | Daryn Sharp