Apache Hadoop 3.5.0 – Apache Hadoop Changelog

##### General

- Overview
- Single Node Setup
- Cluster Setup
- Commands Reference
- FileSystem Shell
- Compatibility Specification
- Downstream Developer's Guide
- Admin Compatibility Guide
- Interface Classification
- FileSystem Specification

##### Common

- CLI Mini Cluster
- Fair Call Queue
- Native Libraries
- Proxy User
- Rack Awareness
- Secure Mode
- Service Level Authorization
- HTTP Authentication
- Credential Provider API
- Hadoop KMS
- Tracing
- Unix Shell Guide
- Registry
- Async Profiler

##### HDFS

- Architecture
- User Guide
- Commands Reference
- NameNode HA With QJM
- NameNode HA With NFS
- Observer NameNode
- Federation
- ViewFs
- ViewFsOverloadScheme
- Snapshots
- Edits Viewer
- Image Viewer
- Permissions and HDFS
- Quotas and HDFS
- libhdfs (C API)
- WebHDFS (REST API)
- HttpFS
- Short Circuit Local Reads
- Centralized Cache Management
- NFS Gateway
- Rolling Upgrade
- Extended Attributes
- Transparent Encryption
- Multihoming
- Storage Policies
- Memory Storage Support
- Synthetic Load Generator
- Erasure Coding
- Disk Balancer
- Upgrade Domain
- DataNode Admin
- Router Federation
- Provided Storage

##### MapReduce

- Tutorial
- Commands Reference
- Encrypted Shuffle
- Pluggable Shuffle/Sort
- Distributed Cache Deploy
- Support for YARN Shared Cache

##### MapReduce REST APIs

- MR Application Master
- MR History Server

##### YARN

- Architecture
- Commands Reference
- Capacity Scheduler
- Fair Scheduler
- ResourceManager Restart
- ResourceManager HA
- Resource Model
- Node Labels
- Node Attributes
- Web Application Proxy
- Timeline Server
- Timeline Service V.2
- Writing YARN Applications
- YARN Application Security
- NodeManager
- Running Applications in Docker Containers
- Running Applications in runC Containers
- Using CGroups
- Secure Containers
- Reservation System
- Graceful Decommission
- Opportunistic Containers
- YARN Federation
- Shared Cache
- Using GPU
- Using FPGA
- Placement Constraints
- YARN UI2

##### YARN REST APIs

- Introduction
- Resource Manager
- Node Manager
- Timeline Server
- Timeline Service V.2

##### YARN Service

- Overview
- QuickStart
- Concepts
- Yarn Service API
- Service Discovery
- System Services

##### Hadoop Compatible File Systems

- Aliyun OSS
- Amazon S3
- Azure Blob Storage
- Azure Data Lake Storage
- Tencent COS
- Huaweicloud OBS
- VolcanoEngine TOS

##### Auth

- Overview
- Examples
- Configuration
- Building

##### Tools

- Hadoop Streaming
- Hadoop Archives
- Hadoop Archive Logs
- DistCp
- HDFS Federation Balance
- GridMix
- Rumen
- Resource Estimator Service
- Scheduler Load Simulator
- Hadoop Benchmarking
- Dynamometer

##### Reference

- Changelog and Release Notes
- Java API docs
- Unix Shell API
- Metrics

##### Configuration

- core-default.xml
- hdfs-default.xml
- hdfs-rbf-default.xml
- mapred-default.xml
- yarn-default.xml
- kms-default.xml
- httpfs-default.xml
- Deprecated Properties

# Apache Hadoop Changelog

## Release 0.23.7 - 2013-04-18

### INCOMPATIBLE CHANGES:

JIRA | Summary | Priority | Component | Reporter | Contributor
--- | --- | --- | --- | --- | ---
HDFS-395 | DFS Scalability: Incremental block reports | Major | datanode, namenode | dhruba borthakur | Tomasz Nykiel

### NEW FEATURES:

JIRA | Summary | Priority | Component | Reporter | Contributor
--- | --- | --- | --- | --- | ---
HADOOP-9209 | Add shell command to dump file checksums | Major | fs, tools | Todd Lipcon | Todd Lipcon

### IMPROVEMENTS:

JIRA | Summary | Priority | Component | Reporter | Contributor
--- | --- | --- | --- | --- | ---
HADOOP-7358 | Improve log levels when exceptions caught in RPC handler | Minor | ipc | Todd Lipcon | Todd Lipcon
HADOOP-7886 | Add toString to FileStatus | Minor | . | Jakob Homan | SreeHari
HADOOP-8214 | make hadoop script recognize a full set of deprecated commands | Major | scripts | Roman Shaposhnik | Roman Shaposhnik
HADOOP-8711 | provide an option for IPC server users to avoid printing stack information for certain exceptions | Major | ipc | Brandon Li | Brandon Li
HDFS-3817 | avoid printing stack information for SafeModeException | Major | namenode | Brandon Li | Brandon Li
HADOOP-8075 | Lower native-hadoop library log from info to debug | Major | native | Eli Collins | Hızır Sefa İrken
HADOOP-9147 | Add missing fields to FIleStatus.toString | Trivial | . | Jonathan Allen | Jonathan Allen
MAPREDUCE-4907 | TrackerDistributedCacheManager issues too many getFileStatus calls | Major | mrv1, tasktracker | Sandy Ryza | Sandy Ryza
HADOOP-9216 | CompressionCodecFactory#getCodecClasses should trim the result of parsing by Configuration. | Major | io | Tsuyoshi Ozawa | Tsuyoshi Ozawa
HADOOP-8849 | FileUtil#fullyDelete should grant the target directories +rwx permissions before trying to delete them | Minor | . | Ivan A. Veselovsky | Ivan A. Veselovsky
MAPREDUCE-4458 | Warn if java.library.path is used for AM or Task | Major | mrv2 | Robert Joseph Evans | Robert Parker
MAPREDUCE-4949 | Enable multiple pi jobs to run in parallel | Minor | examples | Sandy Ryza | Sandy Ryza
HADOOP-9247 | parametrize Clover “generateXxx” properties to make them re-definable via -D in mvn calls | Minor | . | Ivan A. Veselovsky | Ivan A. Veselovsky
MAPREDUCE-4822 | Unnecessary conversions in History Events | Trivial | jobhistoryserver | Robert Joseph Evans | Chu Tong
MAPREDUCE-4989 | JSONify DataTables input data for Attempts page | Major | jobhistoryserver, mr-am | Ravi Prakash | Ravi Prakash
YARN-249 | Capacity Scheduler web page should show list of active users per queue like it used to (in 1.x) | Major | capacityscheduler | Ravi Prakash | Ravi Prakash
HADOOP-9336 | Allow UGI of current connection to be queried | Critical | ipc | Daryn Sharp | Daryn Sharp
HADOOP-9352 | Expose UGI.setLoginUser for tests | Major | security | Daryn Sharp | Daryn Sharp
HADOOP-8462 | Native-code implementation of bzip2 codec | Major | io | Govind Kamat | Govind Kamat
HADOOP-9374 | Add tokens from -tokenCacheFile into UGI | Major | security | Daryn Sharp | Daryn Sharp
YARN-443 | allow OS scheduling priority of NM to be different than the containers it launches | Major | nodemanager | Thomas Graves | Thomas Graves
HADOOP-9379 | capture the ulimit info after printing the log to the console | Trivial | . | Arpit Gupta | Arpit Gupta
MAPREDUCE-4990 | Construct debug strings conditionally in ShuffleHandler.Shuffle#sendMapOutput() | Trivial | . | Karthik Kambatla | Karthik Kambatla
HADOOP-9253 | Capture ulimit info in the logs at service start time | Major | . | Arpit Gupta | Arpit Gupta
YARN-525 | make CS node-locality-delay refreshable | Major | capacityscheduler | Thomas Graves | Thomas Graves
MAPREDUCE-5079 | Recovery should restore task state from job history info directly | Critical | mr-am | Jason Lowe | Jason Lowe

### BUG FIXES:

JIRA | Summary | Priority | Component | Reporter | Contributor
--- | --- | --- | --- | --- | ---
MAPREDUCE-3952 | In MR2, when Total input paths to process == 1, CombinefileInputFormat.getSplits() returns 0 split. | Major | mrv2 | Zhenxiao Luo | Bhallamudi Venkata Siva Kamesh
HDFS-1765 | Block Replication should respect under-replication block priority | Major | namenode | Hairong Kuang | Uma Maheswara Rao G
MAPREDUCE-4007 | JobClient getJob(JobID) should return NULL if the job does not exist (for backwards compatibility) | Major | mrv2 | Alejandro Abdelnur | Alejandro Abdelnur
HADOOP-6941 | Support non-SUN JREs in UserGroupInformation | Major | . | Stephen Watt | Devaraj Das
HADOOP-8251 | SecurityUtil.fetchServiceTicket broken after HADOOP-6941 | Blocker | security | Todd Lipcon | Todd Lipcon
HDFS-3119 | Overreplicated block is not deleted even after the replication factor is reduced after sync follwed by closing that file | Minor | namenode | J.Andreina | Ashish Singhi
HDFS-3256 | HDFS considers blocks under-replicated if topology script is configured with only 1 rack | Major | . | Aaron T. Myers | Aaron T. Myers
HADOOP-8346 | Changes to support Kerberos with non Sun JVM (HADOOP-6941) broke SPNEGO | Blocker | security | Alejandro Abdelnur | Devaraj Das
MAPREDUCE-4470 | Fix TestCombineFileInputFormat.testForEmptyFile | Major | test | Kihwal Lee | Ilya Katsov
YARN-83 | Change package of YarnClient to include apache | Major | client | Bikas Saha | Bikas Saha
YARN-40 | Provide support for missing yarn commands | Major | client | Devaraj K | Devaraj K
YARN-150 | AppRejectedTransition does not unregister app from master service and scheduler | Major | . | Bikas Saha | Bikas Saha
HADOOP-8878 | uppercase namenode hostname causes hadoop dfs calls with webhdfs filesystem and fsck to fail when security is on | Major | . | Arpit Gupta | Arpit Gupta
HDFS-4072 | On file deletion remove corresponding blocks pending replication | Minor | namenode | Jing Zhao | Jing Zhao
HDFS-2434 | TestNameNodeMetrics.testCorruptBlock fails intermittently | Major | test | Uma Maheswara Rao G | Jing Zhao
MAPREDUCE-4637 | Killing an unassigned task attempt causes the job to fail | Major | mrv2 | Tom White | Mayank Bansal
MAPREDUCE-4278 | cannot run two local jobs in parallel from the same gateway. | Major | . | Araceli Henley | Sandy Ryza
HADOOP-9155 | FsPermission should have different default value, 777 for directory and 666 for file | Minor | . | Binglin Chang | Binglin Chang
HADOOP-8816 | HTTP Error 413 full HEAD if using kerberos authentication | Major | net | Moritz Moeller | Moritz Moeller
HADOOP-9212 | Potential deadlock in FileSystem.Cache/IPC/UGI | Major | fs | Tom White | Tom White
HADOOP-9193 | hadoop script can inadvertently expand wildcard arguments when delegating to hdfs script | Minor | scripts | Jason Lowe | Andy Isaacson
MAPREDUCE-4946 | Type conversion of map completion events leads to performance problems with large jobs | Critical | mr-am | Jason Lowe | Jason Lowe
HADOOP-9231 | Parametrize staging URL for the uniformity of distributionManagement | Major | build | Konstantin Boudnik | Konstantin Boudnik
HDFS-4426 | Secondary namenode shuts down immediately after startup | Blocker | namenode | Jason Lowe | Arpit Agarwal
YARN-354 | WebAppProxyServer exits immediately after startup | Blocker | . | Liang Xie | Liang Xie
HADOOP-8857 | hadoop.http.authentication.signature.secret.file docs should not state that secret is randomly generated | Minor | security | Eli Collins | Alejandro Abdelnur
HDFS-4444 | Add space between total transaction time and number of transactions in FSEditLog#printStatistics | Trivial | . | Stephen Chu | Stephen Chu
YARN-133 | update web services docs for RM clusterMetrics | Major | resourcemanager | Thomas Graves | Ravi Prakash
HADOOP-9190 | packaging docs is broken | Major | documentation | Thomas Graves | Andy Isaacson
HDFS-4288 | NN accepts incremental BR as IBR in safemode | Critical | namenode | Daryn Sharp | Daryn Sharp
HADOOP-9221 | Convert remaining xdocs to APT | Major | . | Andy Isaacson | Andy Isaacson
YARN-343 | Capacity Scheduler maximum-capacity value -1 is invalid | Major | capacityscheduler | Thomas Graves | Xuan Gong
MAPREDUCE-4893 | MR AppMaster can do sub-optimal assignment of containers to map tasks leading to poor node locality | Major | applicationmaster | Bikas Saha | Bikas Saha
MAPREDUCE-4969 | TestKeyValueTextInputFormat test fails with Open JDK 7 | Major | test | Arpit Agarwal | Arpit Agarwal
YARN-364 | AggregatedLogDeletionService can take too long to delete logs | Major | . | Jason Lowe | Jason Lowe
HDFS-4462 | 2NN will fail to checkpoint after an HDFS upgrade from a pre-federation version of HDFS | Major | namenode | Aaron T. Myers | Aaron T. Myers
YARN-360 | Allow apps to concurrently register tokens for renewal | Critical | . | Daryn Sharp | Daryn Sharp
MAPREDUCE-4953 | HadoopPipes misuses fprintf | Major | pipes | Andy Isaacson | Andy Isaacson
HADOOP-9278 | HarFileSystem may leak file handle | Major | fs | Chris Nauroth | Chris Nauroth
YARN-357 | App submission should not be synchronized | Major | resourcemanager | Daryn Sharp | Daryn Sharp
YARN-355 | RM app submission jams under load | Blocker | resourcemanager | Daryn Sharp | Daryn Sharp
HADOOP-9289 | FsShell rm -f fails for non-matching globs | Blocker | fs | Daryn Sharp | Daryn Sharp
MAPREDUCE-4671 | AM does not tell the RM about container requests that are no longer needed | Major | . | Bikas Saha | Bikas Saha
HADOOP-9124 | SortedMapWritable violates contract of Map interface for equals() and hashCode() | Minor | io | Patrick Hunt | Surenkumar Nihalani
YARN-362 | Unexpected extra results when using webUI table search | Minor | . | Jason Lowe | Ravi Prakash
HADOOP-9302 | HDFS docs not linked from top level | Major | documentation | Thomas Graves | Andy Isaacson
HADOOP-9303 | command manual dfsadmin missing entry for restoreFailedStorage option | Major | . | Thomas Graves | Andy Isaacson
MAPREDUCE-5000 | TaskImpl.getCounters() can return the counters for the wrong task attempt when task is speculating | Critical | mr-am | Jason Lowe | Jason Lowe
HDFS-4495 | Allow client-side lease renewal to be retried beyond soft-limit | Major | hdfs-client | Kihwal Lee | Kihwal Lee
HADOOP-9154 | SortedMapWritable#putAll() doesn’t add key/value classes to the map | Major | io | Karthik Kambatla | Karthik Kambatla
MAPREDUCE-5009 | Killing the Task Attempt slated for commit does not clear the value from the Task commitAttempt member | Critical | mrv1 | Robert Parker | Robert Parker
YARN-400 | RM can return null application resource usage report leading to NPE in client | Critical | resourcemanager | Jason Lowe | Jason Lowe
YARN-236 | RM should point tracking URL to RM web page when app fails to start | Major | resourcemanager | Jason Lowe | Jason Lowe
HDFS-4222 | NN is unresponsive and loses heartbeats of DNs when Hadoop is configured to use LDAP and LDAP has issues | Minor | namenode | Xiaobo Peng | Xiaobo Peng
YARN-426 | Failure to download a public resource on a node prevents further downloads of the resource from that node | Critical | nodemanager | Jason Lowe | Jason Lowe
HADOOP-9339 | IPC.Server incorrectly sets UGI auth type | Major | ipc | Daryn Sharp | Daryn Sharp
MAPREDUCE-4871 | AM uses mapreduce.jobtracker.split.metainfo.maxsize but mapred-default has mapreduce.job.split.metainfo.maxsize | Major | mrv2 | Jason Lowe | Jason Lowe
YARN-269 | Resource Manager not logging the health_check_script result when taking it out | Major | resourcemanager | Thomas Graves | Jason Lowe
YARN-376 | Apps that have completed can appear as RUNNING on the NM UI | Blocker | resourcemanager | Jason Lowe | Jason Lowe
YARN-410 | New lines in diagnostics for a failed app on the per-application page make it hard to read | Major | . | Vinod Kumar Vavilapalli | Omkar Vinit Joshi
MAPREDUCE-4794 | DefaultSpeculator generates error messages on normal shutdown | Major | applicationmaster | Jason Lowe | Jason Lowe
HDFS-4128 | 2NN gets stuck in inconsistent state if edit log replay fails in the middle | Major | namenode | Todd Lipcon | Kihwal Lee
HDFS-4532 | RPC call queue may fill due to current user lookup | Critical | namenode | Daryn Sharp | Daryn Sharp
YARN-448 | Remove unnecessary hflush from log aggregation | Major | nodemanager | Kihwal Lee | Kihwal Lee
MAPREDUCE-5043 | Fetch failure processing can cause AM event queue to backup and eventually OOM | Blocker | mr-am | Jason Lowe | Jason Lowe
YARN-345 | Many InvalidStateTransitonException errors for ApplicationImpl in Node Manager | Critical | nodemanager | Devaraj K | Robert Parker
HDFS-4553 | Webhdfs will NPE on some unexpected response codes | Major | webhdfs | Daryn Sharp | Daryn Sharp
YARN-227 | Application expiration difficult to debug for end-users | Major | resourcemanager | Jason Lowe | Jason Lowe
MAPREDUCE-5027 | Shuffle does not limit number of outstanding connections | Major | . | Jason Lowe | Robert Parker
MAPREDUCE-3685 | There are some bugs in implementation of MergeManager | Critical | mrv2 | anty.rao | anty
HDFS-4544 | Error in deleting blocks should not do check disk, for all types of errors | Major | . | Amareshwari Sriramadasu | Arpit Agarwal
MAPREDUCE-5023 | History Server Web Services missing Job Counters | Critical | jobhistoryserver, webapps | Kendall Thrapp | Ravi Prakash
MAPREDUCE-5060 | Fetch failures that time out only count against the first map task | Critical | . | Robert Joseph Evans | Robert Joseph Evans
HDFS-3344 | Unreliable corrupt blocks counting in TestProcessCorruptBlocks | Major | namenode | Tsz Wo Nicholas Sze | Kihwal Lee
HADOOP-9406 | hadoop-client leaks dependency on JDK tools jar | Major | build | Alejandro Abdelnur | Alejandro Abdelnur
MAPREDUCE-5042 | Reducer unable to fetch for a map task that was recovered | Blocker | mr-am, security | Jason Lowe | Jason Lowe
MAPREDUCE-5053 | java.lang.InternalError from decompression codec cause reducer to fail | Major | . | Robert Parker | Robert Parker
MAPREDUCE-5075 | DistCp leaks input file handles | Major | distcp | Chris Nauroth | Chris Nauroth
YARN-109 | .tmp file is not deleted for localized archives | Major | nodemanager | Jason Lowe | Mayank Bansal
HDFS-4581 | DataNode#checkDiskError should not be called on network errors | Major | datanode | Rohit Kochar | Rohit Kochar
YARN-460 | CS user left in list of active users for the queue even when application finished | Blocker | capacityscheduler | Thomas Graves | Thomas Graves
MAPREDUCE-4992 | AM hangs in RecoveryService when recovering tasks with speculative attempts | Critical | mr-am | Robert Parker | Robert Parker
HDFS-4649 | Webhdfs cannot list large directories | Blocker | namenode, security, webhdfs | Daryn Sharp | Daryn Sharp
MAPREDUCE-5137 | AM web UI: clicking on Map Task results in 500 error | Major | applicationmaster | Thomas Graves | Thomas Graves

### TESTS:

JIRA | Summary | Priority | Component | Reporter | Contributor
--- | --- | --- | --- | --- | ---
HADOOP-8157 | TestRPCCallBenchmark#testBenchmarkWithWritable fails with RTE | Major | . | Eli Collins | Todd Lipcon
HADOOP-9067 | provide test for method org.apache.hadoop.fs.LocalFileSystem.reportChecksumFailure(Path, FSDataInputStream, long, FSDataInputStream, long) | Minor | . | Ivan A. Veselovsky | Ivan A. Veselovsky
MAPREDUCE-4905 | test org.apache.hadoop.mapred.pipes | Major | . | Aleksey Gorshkov | Aleksey Gorshkov
MAPREDUCE-4972 | Coverage fixing for org.apache.hadoop.mapreduce.jobhistory | Major | . | Aleksey Gorshkov | Aleksey Gorshkov
MAPREDUCE-4875 | coverage fixing for org.apache.hadoop.mapred | Major | test | Aleksey Gorshkov | Aleksey Gorshkov
MAPREDUCE-4991 | coverage for gridmix | Major | . | Aleksey Gorshkov | Aleksey Gorshkov
MAPREDUCE-5007 | fix coverage org.apache.hadoop.mapreduce.v2.hs | Major | . | Aleksey Gorshkov | Aleksey Gorshkov

### SUB-TASKS:

JIRA | Summary | Priority | Component | Reporter | Contributor
--- | --- | --- | --- | --- | ---
HDFS-2477 | Optimize computing the diff between a block report and the namenode state. | Major | namenode | Tomasz Nykiel | Tomasz Nykiel
HDFS-2495 | Increase granularity of write operations in ReplicationMonitor thus reducing contention for write lock | Major | namenode | Tomasz Nykiel | Tomasz Nykiel
HDFS-2476 | More CPU efficient data structure for under-replicated/over-replicated/invalidate blocks | Major | namenode | Tomasz Nykiel | Tomasz Nykiel
YARN-29 | Add a yarn-client module | Major | client | Vinod Kumar Vavilapalli | Vinod Kumar Vavilapalli
HDFS-4542 | Webhdfs doesn’t support secure proxy users | Blocker | webhdfs | Daryn Sharp | Daryn Sharp
HDFS-4560 | Webhdfs cannot use tokens obtained by another user | Major | webhdfs | Daryn Sharp | Daryn Sharp
HDFS-4566 | Webdhfs token cancelation should use authentication | Major | webhdfs | Daryn Sharp | Daryn Sharp
HDFS-4567 | Webhdfs does not need a token for token operations | Major | webhdfs | Daryn Sharp | Daryn Sharp
HDFS-4577 | Webhdfs operations should declare if authentication is required | Major | webhdfs | Daryn Sharp | Daryn Sharp
YARN-468 | coverage fix for org.apache.hadoop.yarn.server.webproxy.amfilter | Major | . | Aleksey Gorshkov | Aleksey Gorshkov
YARN-200 | yarn log does not output all needed information, and is in a binary format | Major | . | Robert Joseph Evans | Ravi Prakash
HDFS-4548 | Webhdfs doesn’t renegotiate SPNEGO token | Blocker | . | Daryn Sharp | Daryn Sharp