Apache Hadoop 3.5.0 – Apache Hadoop Changelog

##### General

- Overview
- Single Node Setup
- Cluster Setup
- Commands Reference
- FileSystem Shell
- Compatibility Specification
- Downstream Developer's Guide
- Admin Compatibility Guide
- Interface Classification
- FileSystem Specification

##### Common

- CLI Mini Cluster
- Fair Call Queue
- Native Libraries
- Proxy User
- Rack Awareness
- Secure Mode
- Service Level Authorization
- HTTP Authentication
- Credential Provider API
- Hadoop KMS
- Tracing
- Unix Shell Guide
- Registry
- Async Profiler

##### HDFS

- Architecture
- User Guide
- Commands Reference
- NameNode HA With QJM
- NameNode HA With NFS
- Observer NameNode
- Federation
- ViewFs
- ViewFsOverloadScheme
- Snapshots
- Edits Viewer
- Image Viewer
- Permissions and HDFS
- Quotas and HDFS
- libhdfs (C API)
- WebHDFS (REST API)
- HttpFS
- Short Circuit Local Reads
- Centralized Cache Management
- NFS Gateway
- Rolling Upgrade
- Extended Attributes
- Transparent Encryption
- Multihoming
- Storage Policies
- Memory Storage Support
- Synthetic Load Generator
- Erasure Coding
- Disk Balancer
- Upgrade Domain
- DataNode Admin
- Router Federation
- Provided Storage

##### MapReduce

- Tutorial
- Commands Reference
- Encrypted Shuffle
- Pluggable Shuffle/Sort
- Distributed Cache Deploy
- Support for YARN Shared Cache

##### MapReduce REST APIs

- MR Application Master
- MR History Server

##### YARN

- Architecture
- Commands Reference
- Capacity Scheduler
- Fair Scheduler
- ResourceManager Restart
- ResourceManager HA
- Resource Model
- Node Labels
- Node Attributes
- Web Application Proxy
- Timeline Server
- Timeline Service V.2
- Writing YARN Applications
- YARN Application Security
- NodeManager
- Running Applications in Docker Containers
- Running Applications in runC Containers
- Using CGroups
- Secure Containers
- Reservation System
- Graceful Decommission
- Opportunistic Containers
- YARN Federation
- Shared Cache
- Using GPU
- Using FPGA
- Placement Constraints
- YARN UI2

##### YARN REST APIs

- Introduction
- Resource Manager
- Node Manager
- Timeline Server
- Timeline Service V.2

##### YARN Service

- Overview
- QuickStart
- Concepts
- Yarn Service API
- Service Discovery
- System Services

##### Hadoop Compatible File Systems

- Aliyun OSS
- Amazon S3
- Azure Blob Storage
- Azure Data Lake Storage
- Tencent COS
- Huaweicloud OBS
- VolcanoEngine TOS

##### Auth

- Overview
- Examples
- Configuration
- Building

##### Tools

- Hadoop Streaming
- Hadoop Archives
- Hadoop Archive Logs
- DistCp
- HDFS Federation Balance
- GridMix
- Rumen
- Resource Estimator Service
- Scheduler Load Simulator
- Hadoop Benchmarking
- Dynamometer

##### Reference

- Changelog and Release Notes
- Java API docs
- Unix Shell API
- Metrics

##### Configuration

- core-default.xml
- hdfs-default.xml
- hdfs-rbf-default.xml
- mapred-default.xml
- yarn-default.xml
- kms-default.xml
- httpfs-default.xml
- Deprecated Properties

# Apache Hadoop Changelog

## Release 0.23.6 - 2013-02-06

### IMPROVEMENTS:

JIRA | Summary | Priority | Component | Reporter | Contributor
--- | --- | --- | --- | --- | ---
YARN-80 | Support delay scheduling for node locality in MR2’s capacity scheduler | Major | capacityscheduler | Todd Lipcon | Arun C Murthy
HADOOP-8931 | Add Java version to startup message | Trivial | . | Eli Collins | Eli Collins
MAPREDUCE-4811 | JobHistoryServer should show when it was started in WebUI About page | Minor | jobhistoryserver, mrv2 | Ravi Prakash | Ravi Prakash
MAPREDUCE-4764 | repair test org.apache.hadoop.mapreduce.security.TestBinaryTokenFile | Major | . | Ivan A. Veselovsky |
HADOOP-8992 | Enhance unit-test coverage of class HarFileSystem | Minor | . | Ivan A. Veselovsky | Ivan A. Veselovsky
HADOOP-8561 | Introduce HADOOP_PROXY_USER for secure impersonation in child hadoop client processes | Major | security | Luke Lu | Yu Gao
MAPREDUCE-4845 | ClusterStatus.getMaxMemory() and getUsedMemory() exist in MR1 but not MR2 | Major | client | Sandy Ryza | Sandy Ryza
YARN-285 | RM should be able to provide a tracking link for apps that have already been purged | Major | . | Derek Dagit | Derek Dagit
MAPREDUCE-4899 | Provide a plugin to the Yarn Web App Proxy to generate tracking links for M/R appllications given the ID | Major | . | Derek Dagit | Derek Dagit
MAPREDUCE-4810 | Add admin command options for ApplicationMaster | Minor | applicationmaster | Jason Lowe | Haifeng Chen

### BUG FIXES:

JIRA | Summary | Priority | Component | Reporter | Contributor
--- | --- | --- | --- | --- | ---
HADOOP-7868 | Hadoop native fails to compile when default linker option is -Wl,–as-needed | Major | native | James Page | Trevor Robinson
HDFS-2757 | Cannot read a local block that’s being written to when using the local read short circuit | Major | . | Jean-Daniel Cryans | Jean-Daniel Cryans
HDFS-4186 | logSync() is called with the write lock held while releasing lease | Critical | namenode | Kihwal Lee | Kihwal Lee
HADOOP-9072 | Hadoop-Common-0.23-Build Fails to build in Jenkins | Major | . | Robert Parker | Robert Parker
MAPREDUCE-4825 | JobImpl.finished doesn’t expect ERROR as a final job state | Major | mr-am | Jason Lowe | Jason Lowe
MAPREDUCE-4817 | Hardcoded task ping timeout kills tasks localizing large amounts of data | Critical | applicationmaster, mr-am | Jason Lowe | Thomas Graves
HDFS-4242 | Map.Entry is incorrectly used in LeaseManager | Major | namenode | Tsz Wo Nicholas Sze | Tsz Wo Nicholas Sze
HADOOP-9108 | Add a method to clear terminateCalled to ExitUtil for test cases | Major | util | Kihwal Lee | Kihwal Lee
HDFS-4233 | NN keeps serving even after no journals started while rolling edit | Blocker | namenode | Kihwal Lee | Kihwal Lee
YARN-251 | Proxy URI generation fails for blank tracking URIs | Major | resourcemanager | Tom White | Tom White
MAPREDUCE-4836 | Elapsed time for running tasks on AM web UI tasks page is 0 | Major | . | Ravi Prakash | Ravi Prakash
HDFS-4254 | testAllEditsDirsFailOnFlush makes subsequent test cases fail (0.23.6 only) | Major | test | Kihwal Lee | Kihwal Lee
YARN-72 | NM should handle cleaning up containers when it shuts down | Major | nodemanager | Hitesh Shah | Sandy Ryza
YARN-258 | RM web page UI shows Invalid Date for start and finish times | Major | resourcemanager | Ravi Prakash | Ravi Prakash
YARN-266 | RM and JHS Web UIs are blank because AppsBlock is not escaping string properly | Critical | resourcemanager | Ravi Prakash | Ravi Prakash
MAPREDUCE-4861 | Cleanup: Remove unused mapreduce.security.token.DelegationTokenRenewal | Major | . | Karthik Kambatla | Karthik Kambatla
MAPREDUCE-4856 | TestJobOutputCommitter uses same directory as TestJobCleanup | Major | test | Sandy Ryza | Sandy Ryza
HADOOP-9135 | JniBasedUnixGroupsMappingWithFallback should log at debug rather than info during fallback | Trivial | security | Colin P. McCabe | Colin P. McCabe
HDFS-4315 | DNs with multiple BPs can have BPOfferServices fail to start due to unsynchronized map access | Major | datanode | Aaron T. Myers | Aaron T. Myers
HDFS-4232 | NN fails to write a fsimage with stale leases | Blocker | namenode | Daryn Sharp | Daryn Sharp
HADOOP-9152 | HDFS can report negative DFS Used on clusters with very small amounts of data | Minor | fs | Brock Noland | Brock Noland
YARN-223 | Change processTree interface to work better with native code | Critical | . | Radim Kolar | Radim Kolar
MAPREDUCE-4895 | Fix compilation failure of org.apache.hadoop.mapred.gridmix.TestResourceUsageEmulators | Major | . | Dennis Y | Dennis Y
HADOOP-9105 | FsShell -moveFromLocal erroneously fails | Major | fs | Daryn Sharp | Daryn Sharp
MAPREDUCE-4842 | Shuffle race can hang reducer | Blocker | mrv2 | Jason Lowe | Mariappan Asokan
MAPREDUCE-4833 | Task can get stuck in FAIL_CONTAINER_CLEANUP | Critical | applicationmaster, mrv2 | Robert Joseph Evans | Robert Parker
MAPREDUCE-4890 | Invalid TaskImpl state transitions when task fails while speculating | Critical | mr-am | Jason Lowe | Jason Lowe
HADOOP-9169 | Bring branch-0.23 ExitUtil up to same level as branch-2 | Major | . | Robert Joseph Evans | Robert Joseph Evans
MAPREDUCE-4902 | Fix typo “receievd” should be “received” in log output | Trivial | . | Albert Chu | Albert Chu
YARN-225 | Proxy Link in RM UI thows NPE in Secure mode | Critical | resourcemanager | Devaraj K | Devaraj K
MAPREDUCE-4813 | AM timing out during job commit | Critical | applicationmaster | Jason Lowe | Jason Lowe
YARN-293 | Node Manager leaks LocalizerRunner object for every Container | Critical | nodemanager | Devaraj K | Robert Joseph Evans
MAPREDUCE-4279 | getClusterStatus() fails with null pointer exception when running jobs in local mode | Major | jobtracker | Rahul Jain | Devaraj K
HDFS-4270 | Replications of the highest priority should be allowed to choose a source datanode that has reached its max replication limit | Minor | namenode | Derek Dagit | Derek Dagit
MAPREDUCE-4832 | MR AM can get in a split brain situation | Critical | applicationmaster | Robert Joseph Evans | Jason Lowe
MAPREDUCE-4894 | Renewal / cancellation of JobHistory tokens | Blocker | jobhistoryserver, mrv2 | Siddharth Seth | Siddharth Seth
MAPREDUCE-4819 | AM can rerun job after reporting final job status to the client | Blocker | mr-am | Jason Lowe | Bikas Saha
MAPREDUCE-4913 | TestMRAppMaster#testMRAppMasterMissingStaging occasionally exits | Major | mr-am | Jason Lowe | Jason Lowe
YARN-170 | NodeManager stop() gets called twice on shutdown | Major | nodemanager | Sandy Ryza | Sandy Ryza
HADOOP-9181 | Set daemon flag for HttpServer’s QueuedThreadPool | Major | . | Liang Xie | Liang Xie
YARN-320 | RM should always be able to renew its own tokens | Blocker | resourcemanager | Daryn Sharp | Daryn Sharp
YARN-325 | RM CapacityScheduler can deadlock when getQueueInfo() is called and a container is completing | Blocker | capacityscheduler | Jason Lowe | Arun C Murthy
MAPREDUCE-4848 | TaskAttemptContext cast error during AM recovery | Major | mr-am | Jason Lowe | Haifeng Chen
HADOOP-9097 | Maven RAT plugin is not checking all source files | Critical | build | Tom White | Thomas Graves
HDFS-4385 | Maven RAT plugin is not checking all source files | Critical | build | Thomas Graves | Thomas Graves
MAPREDUCE-4934 | Maven RAT plugin is not checking all source files | Critical | build | Thomas Graves | Thomas Graves
YARN-334 | Maven RAT plugin is not checking all source files | Critical | . | Thomas Graves | Thomas Graves
MAPREDUCE-4678 | Running the Pentomino example with defaults throws java.lang.NegativeArraySizeException | Minor | examples | Chris McConnell | Chris McConnell
MAPREDUCE-4925 | The pentomino option parser may be buggy | Major | examples | Karthik Kambatla | Karthik Kambatla
MAPREDUCE-4921 | JobClient should acquire HS token with RM principal | Blocker | client | Daryn Sharp | Daryn Sharp
HDFS-4426 | Secondary namenode shuts down immediately after startup | Blocker | namenode | Jason Lowe | Arpit Agarwal
YARN-354 | WebAppProxyServer exits immediately after startup | Blocker | . | Liang Xie | Liang Xie
HADOOP-9242 | Duplicate surefire plugin config in hadoop-common | Major | test | Andrey Klochkov | Andrey Klochkov
HADOOP-9255 | relnotes.py missing last jira | Critical | scripts | Thomas Graves | Thomas Graves

### TESTS:

JIRA | Summary | Priority | Component | Reporter | Contributor
--- | --- | --- | --- | --- | ---
HADOOP-9038 | provide unit-test coverage of class org.apache.hadoop.fs.LocalDirAllocator.AllocatorPerContext.PathIterator | Minor | . | Ivan A. Veselovsky | Ivan A. Veselovsky
HADOOP-9217 | Print thread dumps when hadoop-common tests fail | Major | test | Andrey Klochkov | Andrey Klochkov

### SUB-TASKS:

JIRA | Summary | Priority | Component | Reporter | Contributor
--- | --- | --- | --- | --- | ---
HADOOP-9020 | Add a SASL PLAIN server | Major | ipc, security | Daryn Sharp | Daryn Sharp
YARN-204 | test coverage for org.apache.hadoop.tools | Major | applications | Aleksey Gorshkov | Aleksey Gorshkov
HDFS-4247 | saveNamespace should be tolerant of dangling lease | Blocker | namenode | Daryn Sharp | Daryn Sharp
HDFS-4248 | Renames may remove file leases | Blocker | namenode | Daryn Sharp | Daryn Sharp
YARN-280 | RM does not reject app submission with invalid tokens | Major | resourcemanager | Daryn Sharp | Daryn Sharp
YARN-50 | Implement renewal / cancellation of Delegation Tokens | Blocker | . | Siddharth Seth | Siddharth Seth

### OTHER:

JIRA | Summary | Priority | Component | Reporter | Contributor
--- | --- | --- | --- | --- | ---
HADOOP-8427 | Convert Forrest docs to APT, incremental | Major | documentation | Eli Collins | Andy Isaacson