Apache Hadoop 3.5.0 – Apache Hadoop Changelog

##### General

- Overview
- Single Node Setup
- Cluster Setup
- Commands Reference
- FileSystem Shell
- Compatibility Specification
- Downstream Developer's Guide
- Admin Compatibility Guide
- Interface Classification
- FileSystem Specification

##### Common

- CLI Mini Cluster
- Fair Call Queue
- Native Libraries
- Proxy User
- Rack Awareness
- Secure Mode
- Service Level Authorization
- HTTP Authentication
- Credential Provider API
- Hadoop KMS
- Tracing
- Unix Shell Guide
- Registry
- Async Profiler

##### HDFS

- Architecture
- User Guide
- Commands Reference
- NameNode HA With QJM
- NameNode HA With NFS
- Observer NameNode
- Federation
- ViewFs
- ViewFsOverloadScheme
- Snapshots
- Edits Viewer
- Image Viewer
- Permissions and HDFS
- Quotas and HDFS
- libhdfs (C API)
- WebHDFS (REST API)
- HttpFS
- Short Circuit Local Reads
- Centralized Cache Management
- NFS Gateway
- Rolling Upgrade
- Extended Attributes
- Transparent Encryption
- Multihoming
- Storage Policies
- Memory Storage Support
- Synthetic Load Generator
- Erasure Coding
- Disk Balancer
- Upgrade Domain
- DataNode Admin
- Router Federation
- Provided Storage

##### MapReduce

- Tutorial
- Commands Reference
- Encrypted Shuffle
- Pluggable Shuffle/Sort
- Distributed Cache Deploy
- Support for YARN Shared Cache

##### MapReduce REST APIs

- MR Application Master
- MR History Server

##### YARN

- Architecture
- Commands Reference
- Capacity Scheduler
- Fair Scheduler
- ResourceManager Restart
- ResourceManager HA
- Resource Model
- Node Labels
- Node Attributes
- Web Application Proxy
- Timeline Server
- Timeline Service V.2
- Writing YARN Applications
- YARN Application Security
- NodeManager
- Running Applications in Docker Containers
- Running Applications in runC Containers
- Using CGroups
- Secure Containers
- Reservation System
- Graceful Decommission
- Opportunistic Containers
- YARN Federation
- Shared Cache
- Using GPU
- Using FPGA
- Placement Constraints
- YARN UI2

##### YARN REST APIs

- Introduction
- Resource Manager
- Node Manager
- Timeline Server
- Timeline Service V.2

##### YARN Service

- Overview
- QuickStart
- Concepts
- Yarn Service API
- Service Discovery
- System Services

##### Hadoop Compatible File Systems

- Aliyun OSS
- Amazon S3
- Azure Blob Storage
- Azure Data Lake Storage
- Tencent COS
- Huaweicloud OBS
- VolcanoEngine TOS

##### Auth

- Overview
- Examples
- Configuration
- Building

##### Tools

- Hadoop Streaming
- Hadoop Archives
- Hadoop Archive Logs
- DistCp
- HDFS Federation Balance
- GridMix
- Rumen
- Resource Estimator Service
- Scheduler Load Simulator
- Hadoop Benchmarking
- Dynamometer

##### Reference

- Changelog and Release Notes
- Java API docs
- Unix Shell API
- Metrics

##### Configuration

- core-default.xml
- hdfs-default.xml
- hdfs-rbf-default.xml
- mapred-default.xml
- yarn-default.xml
- kms-default.xml
- httpfs-default.xml
- Deprecated Properties

# Apache Hadoop Changelog

## Release 2.2.0 - 2013-10-15

### INCOMPATIBLE CHANGES:

JIRA | Summary | Priority | Component | Reporter | Contributor
--- | --- | --- | --- | --- | ---
YARN-1229 | Define constraints on Auxiliary Service names. Change ShuffleHandler service name from mapreduce.shuffle to mapreduce_shuffle. | Blocker | nodemanager | Tassapol Athiapinya | Xuan Gong
YARN-1228 | Clean up Fair Scheduler configuration loading | Major | scheduler | Sandy Ryza | Sandy Ryza
HADOOP-10020 | disable symlinks temporarily | Blocker | fs | Colin P. McCabe | Sanjay Radia

### IMPROVEMENTS:

JIRA | Summary | Priority | Component | Reporter | Contributor
--- | --- | --- | --- | --- | ---
HDFS-4817 | make HDFS advisory caching configurable on a per-file basis | Minor | hdfs-client | Colin P. McCabe | Colin P. McCabe
HADOOP-9758 | Provide configuration option for FileSystem/FileContext symlink resolution | Major | . | Andrew Wang | Andrew Wang
HDFS-5139 | Remove redundant -R option from setrep | Major | tools | Arpit Agarwal | Arpit Agarwal
YARN-1246 | Log application status in the rm log when app is done running | Minor | . | Arpit Gupta | Arpit Gupta
HDFS-5256 | Use guava LoadingCache to implement DFSClientCache | Major | nfs | Haohui Mai | Haohui Mai
HADOOP-8315 | Support SASL-authenticated ZooKeeper in ActiveStandbyElector | Major | auto-failover, ha | Todd Lipcon | Todd Lipcon
YARN-1213 | Restore config to ban submitting to undeclared pools in the Fair Scheduler | Major | scheduler | Sandy Ryza | Sandy Ryza
HDFS-5308 | Replace HttpConfig#getSchemePrefix with implicit schemes in HDFS JSP | Major | . | Haohui Mai | Haohui Mai

### BUG FIXES:

JIRA | Summary | Priority | Component | Reporter | Contributor
--- | --- | --- | --- | --- | ---
HDFS-5031 | BlockScanner scans the block multiple times and on restart scans everything | Blocker | datanode | Vinayakumar B | Vinayakumar B
MAPREDUCE-5488 | Job recovery fails after killing all the running containers for the app | Major | . | Arpit Gupta | Jian He
MAPREDUCE-5515 | Application Manager UI does not appear with Https enabled | Major | . | Omkar Vinit Joshi | Omkar Vinit Joshi
HDFS-5251 | Race between the initialization of NameNode and the http server | Major | . | Haohui Mai | Haohui Mai
HDFS-5228 | The RemoteIterator returned by DistributedFileSystem.listFiles(..) may throw NPE | Blocker | hdfs-client | Tsz Wo Nicholas Sze | Tsz Wo Nicholas Sze
MAPREDUCE-5523 | Need to add https port related property in Job history server | Major | . | Omkar Vinit Joshi | Omkar Vinit Joshi
YARN-1128 | FifoPolicy.computeShares throws NPE on empty list of Schedulables | Major | scheduler | Sandy Ryza | Karthik Kambatla
HADOOP-9776 | HarFileSystem.listStatus() returns invalid authority if port number is empty | Major | fs | shanyu zhao | shanyu zhao
HADOOP-9761 | ViewFileSystem#rename fails when using DistributedFileSystem | Blocker | viewfs | Andrew Wang | Andrew Wang
MAPREDUCE-5503 | TestMRJobClient.testJobClient is failing | Blocker | mrv2 | Jason Lowe | Jian He
YARN-1157 | ResourceManager UI has invalid tracking URL link for distributed shell application | Major | resourcemanager | Tassapol Athiapinya | Xuan Gong
MAPREDUCE-5170 | incorrect exception message if min node size > min rack size | Trivial | mrv2 | Sangjin Lee | Sangjin Lee
HDFS-5258 | Skip tests in TestHDFSCLI that are not applicable on Windows. | Minor | test | Chris Nauroth | Chuan Liu
HADOOP-9976 | Different versions of avro and avro-maven-plugin | Major | . | Karthik Kambatla | Karthik Kambatla
MAPREDUCE-5513 | ConcurrentModificationException in JobControl | Major | . | Jason Lowe | Robert Parker
MAPREDUCE-5545 | org.apache.hadoop.mapred.TestTaskAttemptListenerImpl.testCommitWindow times out | Major | . | Robert Kanter | Robert Kanter
HDFS-5268 | NFS write commit verifier is not set in a few places | Major | nfs | Brandon Li | Brandon Li
HDFS-5265 | Namenode fails to start when dfs.https.port is unspecified | Major | . | Haohui Mai | Haohui Mai
YARN-1221 | With Fair Scheduler, reserved MB reported in RM web UI increases indefinitely | Major | resourcemanager, scheduler | Sandy Ryza | Siqi Li
YARN-1247 | test-container-executor has gotten out of sync with the changes to container-executor | Major | nodemanager | Roman Shaposhnik | Roman Shaposhnik
MAPREDUCE-5544 | JobClient#getJob loads job conf twice | Major | . | Sandy Ryza | Sandy Ryza
YARN-1215 | Yarn URL should include userinfo | Major | api | Chuan Liu | Chuan Liu
YARN-1262 | TestApplicationCleanup relies on all containers assigned in a single heartbeat | Major | . | Sandy Ryza | Karthik Kambatla
HADOOP-10003 | HarFileSystem.listLocatedStatus() fails | Major | fs | Jason Dere |
HDFS-5255 | Distcp job fails with hsftp when https is enabled in insecure cluster | Major | . | Yesha Vora | Arpit Agarwal
MAPREDUCE-5536 | mapreduce.jobhistory.webapp.https.address property is not respected | Blocker | . | Yesha Vora | Omkar Vinit Joshi
HADOOP-10012 | Secure Oozie jobs fail with delegation token renewal exception in Namenode HA setup | Blocker | ha | Arpit Gupta | Suresh Srinivas
HDFS-5279 | Guard against NullPointerException in NameNode JSP pages before initialization of FSNamesystem. | Major | namenode | Chris Nauroth | Chris Nauroth
MAPREDUCE-5459 | Update the doc of running MRv1 examples jar on YARN | Major | . | Zhijie Shen | Zhijie Shen
YARN-1141 | Updating resource requests should be decoupled with updating blacklist | Major | . | Zhijie Shen | Zhijie Shen
MAPREDUCE-5554 | hdfs-site.xml included in hadoop-mapreduce-client-jobclient tests jar is breaking tests for downstream components | Minor | test | Robert Kanter | Robert Kanter
YARN-876 | Node resource is added twice when node comes back from unhealthy to healthy | Major | resourcemanager | Peng Zhang | Peng Zhang
HDFS-5289 | Race condition in TestRetryCacheWithHA#testCreateSymlink causes spurious test failure | Major | test | Aaron T. Myers | Aaron T. Myers
MAPREDUCE-5489 | MR jobs hangs as it does not use the node-blacklisting feature in RM requests | Critical | . | Yesha Vora | Zhijie Shen
YARN-890 | The roundup for memory values on resource manager UI is misleading | Major | resourcemanager | Trupti Dhavle | Xuan Gong
YARN-1236 | FairScheduler setting queue name in RMApp is not working | Major | resourcemanager | Sandy Ryza | Sandy Ryza
YARN-1271 | “Text file busy” errors launching containers again | Major | nodemanager | Sandy Ryza | Sandy Ryza
YARN-1149 | NM throws InvalidStateTransitonException: Invalid event: APPLICATION_LOG_HANDLING_FINISHED at RUNNING | Major | . | Ramya Sunil | Xuan Gong
MAPREDUCE-5442 | $HADOOP_MAPRED_HOME/$HADOOP_CONF_DIR setting not working on Windows | Major | client | Yingda Chen | Yingda Chen
YARN-1219 | FSDownload changes file suffix making FileUtil.unTar() throw exception | Major | nodemanager | shanyu zhao | shanyu zhao
MAPREDUCE-5533 | Speculative execution does not function for reduce | Major | applicationmaster | Tassapol Athiapinya | Xuan Gong
HDFS-5300 | FSNameSystem#deleteSnapshot() should not check owner in case of permissions disabled | Major | namenode | Vinayakumar B | Vinayakumar B
YARN-1251 | TestDistributedShell#TestDSShell failed with timeout | Major | applications/distributed-shell | Junping Du | Xuan Gong
YARN-1167 | Submitted distributed shell application shows appMasterHost = empty | Major | applications/distributed-shell | Tassapol Athiapinya | Xuan Gong
YARN-1273 | Distributed shell does not account for start container failures reported asynchronously. | Major | . | Hitesh Shah | Hitesh Shah
YARN-1032 | NPE in RackResolve | Critical | . | Lohit Vijayarenu | Lohit Vijayarenu
YARN-1090 | Job does not get into Pending State | Major | . | Yesha Vora | Jian He
YARN-1274 | LCE fails to run containers that don’t have resources to localize | Blocker | nodemanager | Alejandro Abdelnur | Siddharth Seth
YARN-1278 | New AM does not start after rm restart | Blocker | . | Yesha Vora | Hitesh Shah
HDFS-5299 | DFS client hangs in updatePipeline RPC when failover happened | Blocker | namenode | Vinayakumar B | Vinayakumar B

### TESTS:

JIRA | Summary | Priority | Component | Reporter | Contributor
--- | --- | --- | --- | --- | ---
HADOOP-9948 | Add a config value to CLITestHelper to skip tests on Windows | Minor | test | Chuan Liu | Chuan Liu
HDFS-5186 | TestFileJournalManager fails on Windows due to file handle leaks | Minor | namenode, test | Chuan Liu | Chuan Liu
MAPREDUCE-5525 | Increase timeout of TestDFSIO.testAppend and TestMRJobsWithHistoryService.testJobHistoryData | Minor | mrv2, test | Chuan Liu | Chuan Liu

### SUB-TASKS:

JIRA | Summary | Priority | Component | Reporter | Contributor
--- | --- | --- | --- | --- | ---
YARN-1203 | Application Manager UI does not appear with Https enabled | Major | . | Yesha Vora | Omkar Vinit Joshi
YARN-1204 | Need to add https port related property in Yarn | Major | . | Yesha Vora | Omkar Vinit Joshi
MAPREDUCE-5505 | Clients should be notified job finished only after job successfully unregistered | Critical | . | Jian He | Zhijie Shen
YARN-1214 | Register ClientToken MasterKey in SecretManager after it is saved | Critical | resourcemanager | Jian He | Jian He
HDFS-5246 | Make Hadoop nfs server port and mount daemon port configurable | Major | nfs | Jinghui Wang | Jinghui Wang
YARN-49 | Improve distributed shell application to work on a secure cluster | Major | applications/distributed-shell | Hitesh Shah | Vinod Kumar Vavilapalli
MAPREDUCE-5531 | Binary and source incompatibility in mapreduce.TaskID and mapreduce.TaskAttemptID between branch-1 and branch-2 | Blocker | mrv1, mrv2 | Robert Kanter | Robert Kanter
MAPREDUCE-5529 | Binary incompatibilities in mapred.lib.TotalOrderPartitioner between branch-1 and branch-2 | Blocker | mrv1, mrv2 | Robert Kanter | Robert Kanter
YARN-899 | Get queue administration ACLs working | Major | scheduler | Sandy Ryza | Xuan Gong
MAPREDUCE-5538 | MRAppMaster#shutDownJob shouldn’t send job end notification before checking isLastRetry | Blocker | . | Zhijie Shen | Zhijie Shen
HDFS-5230 | Introduce RpcInfo to decouple XDR classes from the RPC API | Major | nfs | Haohui Mai | Haohui Mai
YARN-1070 | ContainerImpl State Machine: Invalid event: CONTAINER_KILLED_ON_REQUEST at CONTAINER_CLEANEDUP_AFTER_KILL | Major | nodemanager | Hitesh Shah | Zhijie Shen
MAPREDUCE-5551 | Binary Incompatibility of O.A.H.U.mapred.SequenceFileAsBinaryOutputFormat.WritableValueBytes | Blocker | . | Zhijie Shen | Zhijie Shen
YARN-1260 | RM_HOME link breaks when webapp.https.address related properties are not specified | Major | . | Yesha Vora | Omkar Vinit Joshi
MAPREDUCE-5530 | Binary and source incompatibility in mapred.lib.CombineFileInputFormat between branch-1 and branch-2 | Blocker | mrv1, mrv2 | Robert Kanter | Robert Kanter
YARN-621 | RM triggers web auth failure before first job | Critical | resourcemanager | Allen Wittenauer | Omkar Vinit Joshi
YARN-1256 | NM silently ignores non-existent service in StartContainerRequest | Critical | . | Bikas Saha | Xuan Gong
YARN-1131 | $yarn logs command should return an appropriate error message if YARN application is still running | Minor | client | Tassapol Athiapinya | Siddharth Seth
YARN-1254 | NM is polluting container’s credentials | Major | . | Vinod Kumar Vavilapalli | Omkar Vinit Joshi
HDFS-5306 | Datanode https port is not available at the namenode | Major | datanode, namenode | Suresh Srinivas | Suresh Srinivas
HADOOP-10017 | Fix NPE in DFSClient#getDelegationToken when doing Distcp from a secured cluster to an insecured cluster | Major | . | Jing Zhao | Haohui Mai
YARN-1277 | Add http policy support for YARN daemons | Major | . | Suresh Srinivas | Omkar Vinit Joshi
MAPREDUCE-5562 | MR AM should exit when unregister() throws exception | Major | . | Zhijie Shen | Zhijie Shen
HDFS-5259 | Support client which combines appended data with old data before sends it to NFS server | Major | nfs | Yesha Vora | Brandon Li