Apache Hadoop 3.5.0 – Apache Hadoop Changelog

##### General

- Overview
- Single Node Setup
- Cluster Setup
- Commands Reference
- FileSystem Shell
- Compatibility Specification
- Downstream Developer's Guide
- Admin Compatibility Guide
- Interface Classification
- FileSystem Specification

##### Common

- CLI Mini Cluster
- Fair Call Queue
- Native Libraries
- Proxy User
- Rack Awareness
- Secure Mode
- Service Level Authorization
- HTTP Authentication
- Credential Provider API
- Hadoop KMS
- Tracing
- Unix Shell Guide
- Registry
- Async Profiler

##### HDFS

- Architecture
- User Guide
- Commands Reference
- NameNode HA With QJM
- NameNode HA With NFS
- Observer NameNode
- Federation
- ViewFs
- ViewFsOverloadScheme
- Snapshots
- Edits Viewer
- Image Viewer
- Permissions and HDFS
- Quotas and HDFS
- libhdfs (C API)
- WebHDFS (REST API)
- HttpFS
- Short Circuit Local Reads
- Centralized Cache Management
- NFS Gateway
- Rolling Upgrade
- Extended Attributes
- Transparent Encryption
- Multihoming
- Storage Policies
- Memory Storage Support
- Synthetic Load Generator
- Erasure Coding
- Disk Balancer
- Upgrade Domain
- DataNode Admin
- Router Federation
- Provided Storage

##### MapReduce

- Tutorial
- Commands Reference
- Encrypted Shuffle
- Pluggable Shuffle/Sort
- Distributed Cache Deploy
- Support for YARN Shared Cache

##### MapReduce REST APIs

- MR Application Master
- MR History Server

##### YARN

- Architecture
- Commands Reference
- Capacity Scheduler
- Fair Scheduler
- ResourceManager Restart
- ResourceManager HA
- Resource Model
- Node Labels
- Node Attributes
- Web Application Proxy
- Timeline Server
- Timeline Service V.2
- Writing YARN Applications
- YARN Application Security
- NodeManager
- Running Applications in Docker Containers
- Running Applications in runC Containers
- Using CGroups
- Secure Containers
- Reservation System
- Graceful Decommission
- Opportunistic Containers
- YARN Federation
- Shared Cache
- Using GPU
- Using FPGA
- Placement Constraints
- YARN UI2

##### YARN REST APIs

- Introduction
- Resource Manager
- Node Manager
- Timeline Server
- Timeline Service V.2

##### YARN Service

- Overview
- QuickStart
- Concepts
- Yarn Service API
- Service Discovery
- System Services

##### Hadoop Compatible File Systems

- Aliyun OSS
- Amazon S3
- Azure Blob Storage
- Azure Data Lake Storage
- Tencent COS
- Huaweicloud OBS
- VolcanoEngine TOS

##### Auth

- Overview
- Examples
- Configuration
- Building

##### Tools

- Hadoop Streaming
- Hadoop Archives
- Hadoop Archive Logs
- DistCp
- HDFS Federation Balance
- GridMix
- Rumen
- Resource Estimator Service
- Scheduler Load Simulator
- Hadoop Benchmarking
- Dynamometer

##### Reference

- Changelog and Release Notes
- Java API docs
- Unix Shell API
- Metrics

##### Configuration

- core-default.xml
- hdfs-default.xml
- hdfs-rbf-default.xml
- mapred-default.xml
- yarn-default.xml
- kms-default.xml
- httpfs-default.xml
- Deprecated Properties

# Apache Hadoop Changelog

## Release 0.23.5 - 2012-11-29

### INCOMPATIBLE CHANGES:

JIRA | Summary | Priority | Component | Reporter | Contributor
--- | --- | --- | --- | --- | ---
HDFS-4080 | Add a separate logger for block state change logs to enable turning off those logs | Major | namenode | Kihwal Lee | Kihwal Lee

### IMPROVEMENTS:

JIRA | Summary | Priority | Component | Reporter | Contributor
--- | --- | --- | --- | --- | ---
HADOOP-8789 | Tests setLevel(Level.OFF) should be Level.ERROR | Minor | test | Andy Isaacson | Andy Isaacson
HADOOP-8755 | Print thread dump when tests fail due to timeout | Major | test | Andrey Klochkov | Andrey Klochkov
HADOOP-8851 | Use -XX:+HeapDumpOnOutOfMemoryError JVM option in the forked tests | Minor | test | Ivan A. Veselovsky | Ivan A. Veselovsky
HDFS-3483 | Better error message when hdfs fsck is run against a ViewFS config | Major | . | Stephen Chu | Stephen Fritz
HADOOP-8889 | Upgrade to Surefire 2.12.3 | Major | build, test | Todd Lipcon | Todd Lipcon
HADOOP-8932 | JNI-based user-group mapping modules can be too chatty on lookup failures | Major | security | Kihwal Lee | Kihwal Lee
HADOOP-8926 | hadoop.util.PureJavaCrc32 cache hit-ratio is low for static data | Major | util | Gopal V | Gopal V
HADOOP-8930 | Cumulative code coverage calculation | Major | test | Andrey Klochkov | Andrey Klochkov
MAPREDUCE-4229 | Counter names’ memory usage can be decreased by interning | Major | jobtracker | Todd Lipcon | Miomir Boljanovic
MAPREDUCE-4752 | Reduce MR AM memory usage through String Interning | Major | mrv2 | Robert Joseph Evans | Robert Joseph Evans
YARN-165 | RM should point tracking URL to RM web page for app when AM fails | Blocker | resourcemanager | Jason Lowe | Jason Lowe
MAPREDUCE-4763 | repair test org.apache.hadoop.mapreduce.security.TestUmbilicalProtocolWithJobToken | Minor | . | Ivan A. Veselovsky |
MAPREDUCE-4666 | JVM metrics for history server | Minor | jobhistoryserver | Jason Lowe | Jason Lowe
MAPREDUCE-4517 | Too many INFO messages written out during AM to RM heartbeat | Minor | applicationmaster | James Kinley | Jason Lowe
YARN-216 | Remove jquery theming support | Major | . | Todd Lipcon | Robert Joseph Evans
MAPREDUCE-4802 | Takes a long time to load the task list on the AM for large jobs | Major | mr-am, mrv2, webapps | Ravi Prakash | Ravi Prakash

### BUG FIXES:

JIRA | Summary | Priority | Component | Reporter | Contributor
--- | --- | --- | --- | --- | ---
HADOOP-8775 | MR2 distcp permits non-positive value to -bandwidth option which causes job never to complete | Major | . | Sandy Ryza | Sandy Ryza
HDFS-3919 | MiniDFSCluster:waitClusterUp can hang forever | Minor | test | Andy Isaacson | Andy Isaacson
HADOOP-8819 | Should use && instead of & in a few places in FTPFileSystem,FTPInputStream,S3InputStream,ViewFileSystem,ViewFs | Major | fs | Brandon Li | Brandon Li
YARN-28 | TestCompositeService fails on jdk7 | Major | . | Thomas Graves | Thomas Graves
MAPREDUCE-4674 | Hadoop examples secondarysort has a typo “secondarysrot” in the usage | Minor | . | Robert Justice | Robert Justice
HADOOP-8791 | rm “Only deletes non empty directory and files.” | Major | documentation | Bertrand Dechoux | Jing Zhao
HADOOP-8386 | hadoop script doesn’t work if ‘cd’ prints to stdout (default behavior in Ubuntu) | Major | scripts | Christopher Berner | Christopher Berner
YARN-116 | RM is missing ability to add include/exclude files without a restart | Major | resourcemanager | xieguiming | xieguiming
YARN-131 | Incorrect ACL properties in capacity scheduler documentation | Major | capacityscheduler | Ahmed Radwan | Ahmed Radwan
HDFS-3905 | Secure cluster cannot use hftp to an insecure cluster | Critical | hdfs-client, security | Daryn Sharp | Daryn Sharp
YARN-102 | Move the apache licence header to the top of the file in MemStore.java | Trivial | resourcemanager | Devaraj K | Devaraj K
HDFS-3996 | Add debug log removed in HDFS-3873 back | Minor | . | Eli Collins | Eli Collins
HDFS-3829 | TestHftpURLTimeouts fails intermittently with JDK7 | Major | test | Trevor Robinson | Trevor Robinson
MAPREDUCE-4554 | Job Credentials are not transmitted if security is turned off | Major | job submission, security | Benoy Antony | Benoy Antony
MAPREDUCE-4705 | Historyserver links expire before the history data does | Critical | jobhistoryserver, mrv2 | Jason Lowe | Jason Lowe
YARN-30 | TestNMWebServicesApps, TestRMWebServicesApps and TestRMWebServicesNodes fail on jdk7 | Major | . | Thomas Graves | Thomas Graves
HDFS-3824 | TestHftpDelegationToken fails intermittently with JDK7 | Major | test | Trevor Robinson | Trevor Robinson
HDFS-3224 | Bug in check for DN re-registration with different storage ID | Minor | . | Eli Collins | Jason Lowe
HADOOP-8906 | paths with multiple globs are unreliable | Critical | fs | Daryn Sharp | Daryn Sharp
HDFS-4016 | back-port HDFS-3582 to branch-0.23 | Minor | . | Ivan A. Veselovsky | Ivan A. Veselovsky
MAPREDUCE-4521 | mapreduce.user.classpath.first incompatibility with 0.20/1.x | Major | mrv2 | Jason Lowe | Ravi Prakash
YARN-161 | Yarn Common has multiple compiler warnings for unchecked operations | Major | api | Chris Nauroth | Chris Nauroth
YARN-43 | TestResourceTrackerService fail intermittently on jdk7 | Major | . | Thomas Graves | Thomas Graves
MAPREDUCE-4721 | Task startup time in JHS is same as job startup time. | Major | jobhistoryserver | Ravi Prakash | Ravi Prakash
YARN-32 | TestApplicationTokens fails intermintently on jdk7 | Major | . | Thomas Graves | Vinod Kumar Vavilapalli
YARN-163 | Retrieving container log via NM webapp can hang with multibyte characters in log | Major | nodemanager | Jason Lowe | Jason Lowe
MAPREDUCE-4479 | Fix parameter order in assertEquals() in TestCombineInputFileFormat.java | Major | test | Mariappan Asokan | Mariappan Asokan
MAPREDUCE-4733 | Reducer can fail to make progress during shuffle if too many reducers complete consecutively | Major | applicationmaster, mrv2 | Jason Lowe | Jason Lowe
MAPREDUCE-4740 | only .jars can be added to the Distributed Cache classpath | Blocker | mrv2 | Robert Joseph Evans | Robert Joseph Evans
YARN-174 | TestNodeStatusUpdater is failing in trunk | Major | nodemanager | Robert Joseph Evans | Vinod Kumar Vavilapalli
HDFS-4090 | getFileChecksum() result incompatible when called against zero-byte files. | Critical | hdfs-client | Kihwal Lee | Kihwal Lee
HADOOP-8962 | RawLocalFileSystem.listStatus fails when a child filename contains a colon | Critical | fs | Jason Lowe | Jason Lowe
YARN-177 | CapacityScheduler - adding a queue while the RM is running has wacky results | Critical | capacityscheduler | Thomas Graves | Arun C Murthy
YARN-178 | Fix custom ProcessTree instance creation | Critical | . | Radim Kolar | Radim Kolar
YARN-180 | Capacity scheduler - containers that get reserved create container token to early | Critical | capacityscheduler | Thomas Graves | Arun C Murthy
YARN-139 | Interrupted Exception within AsyncDispatcher leads to user confusion | Major | api | Nathan Roberts | Vinod Kumar Vavilapalli
MAPREDUCE-4741 | WARN and ERROR messages logged during normal AM shutdown | Minor | applicationmaster, mrv2 | Jason Lowe | Vinod Kumar Vavilapalli
MAPREDUCE-4730 | AM crashes due to OOM while serving up map task completion events | Blocker | applicationmaster, mrv2 | Jason Lowe | Jason Lowe
MAPREDUCE-4748 | Invalid event: T_ATTEMPT_SUCCEEDED at SUCCEEDED | Blocker | mrv2 | Robert Joseph Evans | Jason Lowe
MAPREDUCE-1806 | CombineFileInputFormat does not work with paths not on default FS | Major | harchive | Paul Yang | Gera Shegalov
HADOOP-8986 | Server$Call object is never released after it is sent | Critical | ipc | Robert Joseph Evans | Robert Joseph Evans
YARN-159 | RM web ui applications page should be sorted to display last app first | Major | resourcemanager | Thomas Graves | Thomas Graves
YARN-166 | capacity scheduler doesn’t allow capacity < 1.0 | Major | capacityscheduler | Thomas Graves | Thomas Graves
MAPREDUCE-4724 | job history web ui applications page should be sorted to display last app first | Major | jobhistoryserver | Thomas Graves | Thomas Graves
YARN-189 | deadlock in RM - AMResponse object | Blocker | resourcemanager | Thomas Graves | Thomas Graves
MAPREDUCE-4746 | The MR Application Master does not have a config to set environment variables | Major | applicationmaster | Robert Parker | Robert Parker
MAPREDUCE-4729 | job history UI not showing all job attempts | Major | jobhistoryserver | Thomas Graves | Vinod Kumar Vavilapalli
MAPREDUCE-4771 | KeyFieldBasedPartitioner not partitioning properly when configured | Major | mrv2 | Jason Lowe | Jason Lowe
HDFS-4075 | Reduce recommissioning overhead | Critical | namenode | Kihwal Lee | Kihwal Lee
YARN-201 | CapacityScheduler can take a very long time to schedule containers if requests are off cluster | Critical | capacityscheduler | Jason Lowe | Jason Lowe
MAPREDUCE-4772 | Fetch failures can take way too long for a map to be restarted | Critical | mrv2 | Robert Joseph Evans | Robert Joseph Evans
HDFS-3990 | NN’s health report has severe performance problems | Critical | namenode | Daryn Sharp | Daryn Sharp
MAPREDUCE-4782 | NLineInputFormat skips first line of last InputSplit | Blocker | client | Mark Fuhs | Mark Fuhs
HDFS-4162 | Some malformed and unquoted HTML strings are returned from datanode web ui | Minor | datanode | Derek Dagit | Derek Dagit
MAPREDUCE-4774 | JobImpl does not handle asynchronous task events in FAILED state | Major | applicationmaster, mrv2 | Ivan A. Veselovsky | Jason Lowe
YARN-206 | TestApplicationCleanup.testContainerCleanup occasionally fails | Major | resourcemanager | Jason Lowe | Jason Lowe
MAPREDUCE-4751 | AM stuck in KILL_WAIT for days | Major | . | Ravi Prakash | Vinod Kumar Vavilapalli
MAPREDUCE-4787 | TestJobMonitorAndPrint is broken | Major | test | Ravi Prakash | Robert Parker
HDFS-4172 | namenode does not URI-encode parameters when building URI for datanode request | Minor | namenode | Derek Dagit | Derek Dagit
MAPREDUCE-4425 | Speculation + Fetch failures can lead to a hung job | Critical | mrv2 | Siddharth Seth | Jason Lowe
MAPREDUCE-4786 | Job End Notification retry interval is 5 milliseconds by default | Major | mrv2 | Ravi Prakash | Ravi Prakash
HADOOP-9022 | Hadoop distcp tool fails to copy file if -m 0 specified | Major | . | Haiyang Jiang | Jonathan Eagles
HADOOP-9025 | org.apache.hadoop.tools.TestCopyListing failing | Major | . | Robert Joseph Evans | Jonathan Eagles
HDFS-4181 | LeaseManager tries to double remove and prints extra messages | Critical | namenode | Kihwal Lee | Kihwal Lee
YARN-212 | NM state machine ignores an APPLICATION_CONTAINER_FINISHED event when it shouldn’t | Blocker | nodemanager | Nathan Roberts | Nathan Roberts
YARN-144 | MiniMRYarnCluster launches RM and JHS on default ports | Major | . | Robert Parker | Robert Parker
HDFS-4182 | SecondaryNameNode leaks NameCache entries | Critical | namenode | Todd Lipcon | Robert Joseph Evans
MAPREDUCE-4797 | LocalContainerAllocator can loop forever trying to contact the RM | Major | applicationmaster | Jason Lowe | Jason Lowe
MAPREDUCE-4720 | Browser thinks History Server main page JS is taking too long | Major | . | Robert Joseph Evans | Ravi Prakash
HDFS-4186 | logSync() is called with the write lock held while releasing lease | Critical | namenode | Kihwal Lee | Kihwal Lee
MAPREDUCE-4801 | ShuffleHandler can generate large logs due to prematurely closed channels | Critical | . | Jason Lowe | Jason Lowe
YARN-214 | RMContainerImpl does not handle event EXPIRE at state RUNNING | Major | resourcemanager | Jason Lowe | Jonathan Eagles
YARN-151 | Browser thinks RM main page JS is taking too long | Major | . | Robert Joseph Evans | Ravi Prakash
MAPREDUCE-4549 | Distributed cache conflicts breaks backwards compatability | Blocker | mrv2 | Robert Joseph Evans | Robert Joseph Evans

### SUB-TASKS:

JIRA | Summary | Priority | Component | Reporter | Contributor
--- | --- | --- | --- | --- | ---
MAPREDUCE-4107 | Fix tests in org.apache.hadoop.ipc.TestSocketFactory | Major | mrv2 | Devaraj K | Devaraj K
YARN-202 | Log Aggregation generates a storm of fsync() for namenode | Critical | . | Kihwal Lee | Kihwal Lee
YARN-186 | Coverage fixing LinuxContainerExecutor | Major | resourcemanager, scheduler | Aleksey Gorshkov | Aleksey Gorshkov
YARN-188 | Coverage fixing for CapacityScheduler | Major | capacityscheduler | Aleksey Gorshkov | Aleksey Gorshkov
YARN-219 | NM should aggregate logs when application finishes. | Critical | nodemanager | Robert Joseph Evans | Robert Joseph Evans

### OTHER:

JIRA | Summary | Priority | Component | Reporter | Contributor
--- | --- | --- | --- | --- | ---
MAPREDUCE-4596 | Split StateMachine state from states seen by MRClientProtocol (for Job, Task, TaskAttempt) | Major | applicationmaster, mrv2 | Siddharth Seth | Siddharth Seth
YARN-154 | Create Yarn trunk and commit jobs | Major | . | Eli Collins | Robert Joseph Evans
MAPREDUCE-4266 | remove Ant remnants from MR | Major | build | Alejandro Abdelnur | Alejandro Abdelnur
HADOOP-8860 | Split MapReduce and YARN sections in documentation navigation | Major | documentation | Tom White | Tom White