Apache Hadoop 3.5.0 – Apache Hadoop Changelog

##### General

- Overview
- Single Node Setup
- Cluster Setup
- Commands Reference
- FileSystem Shell
- Compatibility Specification
- Downstream Developer's Guide
- Admin Compatibility Guide
- Interface Classification
- FileSystem Specification

##### Common

- CLI Mini Cluster
- Fair Call Queue
- Native Libraries
- Proxy User
- Rack Awareness
- Secure Mode
- Service Level Authorization
- HTTP Authentication
- Credential Provider API
- Hadoop KMS
- Tracing
- Unix Shell Guide
- Registry
- Async Profiler

##### HDFS

- Architecture
- User Guide
- Commands Reference
- NameNode HA With QJM
- NameNode HA With NFS
- Observer NameNode
- Federation
- ViewFs
- ViewFsOverloadScheme
- Snapshots
- Edits Viewer
- Image Viewer
- Permissions and HDFS
- Quotas and HDFS
- libhdfs (C API)
- WebHDFS (REST API)
- HttpFS
- Short Circuit Local Reads
- Centralized Cache Management
- NFS Gateway
- Rolling Upgrade
- Extended Attributes
- Transparent Encryption
- Multihoming
- Storage Policies
- Memory Storage Support
- Synthetic Load Generator
- Erasure Coding
- Disk Balancer
- Upgrade Domain
- DataNode Admin
- Router Federation
- Provided Storage

##### MapReduce

- Tutorial
- Commands Reference
- Encrypted Shuffle
- Pluggable Shuffle/Sort
- Distributed Cache Deploy
- Support for YARN Shared Cache

##### MapReduce REST APIs

- MR Application Master
- MR History Server

##### YARN

- Architecture
- Commands Reference
- Capacity Scheduler
- Fair Scheduler
- ResourceManager Restart
- ResourceManager HA
- Resource Model
- Node Labels
- Node Attributes
- Web Application Proxy
- Timeline Server
- Timeline Service V.2
- Writing YARN Applications
- YARN Application Security
- NodeManager
- Running Applications in Docker Containers
- Running Applications in runC Containers
- Using CGroups
- Secure Containers
- Reservation System
- Graceful Decommission
- Opportunistic Containers
- YARN Federation
- Shared Cache
- Using GPU
- Using FPGA
- Placement Constraints
- YARN UI2

##### YARN REST APIs

- Introduction
- Resource Manager
- Node Manager
- Timeline Server
- Timeline Service V.2

##### YARN Service

- Overview
- QuickStart
- Concepts
- Yarn Service API
- Service Discovery
- System Services

##### Hadoop Compatible File Systems

- Aliyun OSS
- Amazon S3
- Azure Blob Storage
- Azure Data Lake Storage
- Tencent COS
- Huaweicloud OBS
- VolcanoEngine TOS

##### Auth

- Overview
- Examples
- Configuration
- Building

##### Tools

- Hadoop Streaming
- Hadoop Archives
- Hadoop Archive Logs
- DistCp
- HDFS Federation Balance
- GridMix
- Rumen
- Resource Estimator Service
- Scheduler Load Simulator
- Hadoop Benchmarking
- Dynamometer

##### Reference

- Changelog and Release Notes
- Java API docs
- Unix Shell API
- Metrics

##### Configuration

- core-default.xml
- hdfs-default.xml
- hdfs-rbf-default.xml
- mapred-default.xml
- yarn-default.xml
- kms-default.xml
- httpfs-default.xml
- Deprecated Properties

# Apache Hadoop Changelog

## Release 0.23.10 - 2013-12-09

### INCOMPATIBLE CHANGES:

JIRA | Summary | Priority | Component | Reporter | Contributor
--- | --- | --- | --- | --- | ---
YARN-707 | Add user info in the YARN ClientToken | Blocker | . | Bikas Saha | Jason Lowe

### IMPROVEMENTS:

JIRA | Summary | Priority | Component | Reporter | Contributor
--- | --- | --- | --- | --- | ---
HDFS-5010 | Reduce the frequency of getCurrentUser() calls from namenode | Major | namenode, performance | Kihwal Lee | Kihwal Lee
MAPREDUCE-1981 | Improve getSplits performance by using listLocatedStatus | Major | job submission | Hairong Kuang | Hairong Kuang
YARN-985 | Nodemanager should log where a resource was localized | Major | nodemanager | Ravi Prakash | Ravi Prakash
HADOOP-9686 | Easy access to final parameters in Configuration | Major | conf | Jason Lowe | Jason Lowe
HADOOP-8704 | add request logging to jetty/httpserver | Major | . | Thomas Graves | Jonathan Eagles
MAPREDUCE-5640 | Rename TestLineRecordReader in jobclient module | Trivial | test | Jason Lowe | Jason Lowe

### BUG FIXES:

JIRA | Summary | Priority | Component | Reporter | Contributor
--- | --- | --- | --- | --- | ---
HDFS-3020 | Auto-logSync based on edit log buffer size broken | Critical | namenode | Todd Lipcon | Todd Lipcon
MAPREDUCE-4579 | TestTaskAttempt fails jdk7 | Major | . | Thomas Graves | Thomas Graves
HDFS-3970 | BlockPoolSliceStorage#doRollback(..) should use BlockPoolSliceStorage instead of DataStorage to read prev version file. | Major | datanode | Vinayakumar B | Vinayakumar B
HDFS-4482 | ReplicationMonitor thread can exit with NPE due to the race between delete and replication of same file. | Blocker | namenode | Uma Maheswara Rao G | Uma Maheswara Rao G
MAPREDUCE-4571 | TestHsWebServicesJobs fails on jdk7 | Major | webapps | Thomas Graves | Thomas Graves
MAPREDUCE-4716 | TestHsWebServicesJobsQuery.testJobsQueryStateInvalid fails with jdk7 | Major | jobhistoryserver | Thomas Graves | Thomas Graves
HDFS-4943 | WebHdfsFileSystem does not work when original file path has encoded chars | Minor | webhdfs | Jerry He | Jerry He
MAPREDUCE-3193 | FileInputFormat doesn’t read files recursively in the input path dir | Major | mrv1, mrv2 | Ramgopal N | Devaraj K
HDFS-4984 | Incorrect Quota counting in INodeFile | Major | namenode | Kihwal Lee | Jing Zhao
HDFS-4972 | [branch-0.23] permission check and operation are done in a separate lock for getBlockLocations() | Major | namenode | Kihwal Lee | Kihwal Lee
MAPREDUCE-5380 | Invalid mapred command should return non-zero exit code | Major | . | Stephen Chu | Stephen Chu
HDFS-4998 | TestUnderReplicatedBlocks fails intermittently | Major | test | Kihwal Lee | Kihwal Lee
MAPREDUCE-5317 | Stale files left behind for failed jobs | Major | mrv2 | Ravi Prakash | Ravi Prakash
YARN-949 | Failed log aggregation can leave a file open. | Major | nodemanager | Kihwal Lee | Kihwal Lee
MAPREDUCE-5251 | Reducer should not implicate map attempt if it has insufficient space to fetch map output | Major | mrv2 | Jason Lowe | Ashwin Shankar
MAPREDUCE-5419 | TestSlive is getting FileNotFound Exception | Major | mrv2 | Robert Parker | Robert Parker
HDFS-5028 | LeaseRenewer throw java.util.ConcurrentModificationException when timeout | Major | . | yunjiong zhao | yunjiong zhao
MAPREDUCE-5440 | TestCopyCommitter Fails on JDK7 | Major | mrv2 | Robert Parker | Robert Parker
HADOOP-9476 | Some test cases in TestUserGroupInformation fail if ran after testSetLoginUser. | Major | security, test | Kihwal Lee | Robert Parker
MAPREDUCE-5425 | Junit in TestJobHistoryServer failing in jdk 7 | Major | jobhistoryserver | Ashwin Shankar | Robert Parker
HDFS-5047 | Supress logging of full stack trace of quota and lease exceptions | Major | namenode | Kihwal Lee | Robert Parker
HDFS-4993 | fsck can fail if a file is renamed or deleted | Major | . | Kihwal Lee | Robert Parker
HADOOP-9757 | Har metadata cache can grow without limit | Major | fs | Jason Lowe | Cristina L. Abad
YARN-1036 | Distributed Cache gives inconsistent result if cache files get deleted from task tracker | Major | nodemanager | Ravi Prakash | Ravi Prakash
YARN-337 | RM handles killed application tracking URL poorly | Major | resourcemanager | Jason Lowe | Jason Lowe
MAPREDUCE-5001 | LocalJobRunner has race condition resulting in job failures | Major | . | Brock Noland | Sandy Ryza
HDFS-4994 | Audit log getContentSummary() calls | Minor | namenode | Kihwal Lee | Robert Parker
MAPREDUCE-5468 | AM recovery does not work for map only jobs | Blocker | . | Yesha Vora | Vinod Kumar Vavilapalli
HDFS-4329 | DFSShell issues with directories with spaces in name | Major | hdfs-client | Andy Isaacson | Cristina L. Abad
YARN-1101 | Active nodes can be decremented below 0 | Major | resourcemanager | Robert Parker | Robert Parker
MAPREDUCE-5475 | MRClientService does not verify ACLs properly | Blocker | mr-am, mrv2 | Jason Lowe | Jason Lowe
MAPREDUCE-5414 | TestTaskAttempt fails jdk7 with NullPointerException | Major | test | Nemon Lou | Nemon Lou
YARN-1176 | RM web services ClusterMetricsInfo total nodes doesn’t include unhealthy nodes | Critical | resourcemanager | Thomas Graves | Jonathan Eagles
MAPREDUCE-5504 | mapred queue -info inconsistent with types | Major | client | Thomas Graves | Kousuke Saruta
YARN-1243 | ResourceManager: Error in handling event type NODE_UPDATE to the scheduler - NPE at SchedulerApp.java:411 | Major | capacityscheduler | Sanjay Upadhyay | Jason Lowe
MAPREDUCE-5543 | In-memory map outputs can be leaked after shuffle completes in 0.23 | Blocker | mrv2 | Jason Lowe | Jason Lowe
MAPREDUCE-5513 | ConcurrentModificationException in JobControl | Major | . | Jason Lowe | Robert Parker
MAPREDUCE-5569 | FloatSplitter is not generating correct splits | Major | . | Nathan Roberts | Nathan Roberts
YARN-155 | TestAppManager intermittently fails with jdk7 | Major | . | Thomas Graves | Thomas Graves
MAPREDUCE-5585 | TestCopyCommitter#testNoCommitAction Fails on JDK7 | Major | . | Jonathan Eagles | Jonathan Eagles
MAPREDUCE-5586 | TestCopyMapper#testCopyFailOnBlockSizeDifference fails when run from hadoop-tools/hadoop-distcp directory | Major | . | Jonathan Eagles | Jonathan Eagles
HDFS-5346 | Avoid unnecessary call to getNumLiveDataNodes() for each block during IBR processing | Major | namenode, performance | Kihwal Lee | Ravi Prakash
MAPREDUCE-5587 | TestTextOutputFormat fails on JDK7 | Major | . | Jonathan Eagles | Jonathan Eagles
MAPREDUCE-5561 | org.apache.hadoop.mapreduce.v2.app.job.impl.TestJobImpl testcase failing on trunk | Critical | . | Cindy Li | Karthik Kambatla
YARN-1031 | JQuery UI components reference external css in branch-23 | Major | . | Jonathan Eagles | Jonathan Eagles
YARN-1386 | NodeManager mistakenly loses resources and relocalizes them | Critical | nodemanager | Jason Lowe | Jason Lowe
HADOOP-10091 | Job with a har archive as input fails on 0.23 | Blocker | fs | Jason Lowe | Jason Lowe
HDFS-4995 | Make getContentSummary() less expensive | Major | namenode | Kihwal Lee | Kihwal Lee
HDFS-5438 | Flaws in block report processing can cause data loss | Critical | namenode | Kihwal Lee | Kihwal Lee
MAPREDUCE-5373 | TestFetchFailure.testFetchFailureMultipleReduces could fail intermittently | Major | . | Chuan Liu | Jonathan Eagles
YARN-1419 | TestFifoScheduler.testAppAttemptMetrics fails intermittently under jdk7 | Minor | scheduler | Jonathan Eagles | Jonathan Eagles
HDFS-5557 | Write pipeline recovery for the last packet in the block may cause rejection of valid replicas | Critical | . | Kihwal Lee | Kihwal Lee
HDFS-5526 | Datanode cannot roll back to previous layout version | Blocker | datanode | Tsz Wo Nicholas Sze | Kihwal Lee
HDFS-5558 | LeaseManager monitor thread can crash if the last block is complete but another block is not. | Major | . | Kihwal Lee | Kihwal Lee
HDFS-5357 | TestFileSystemAccessService failures in JDK7 | Major | . | Robert Parker | Robert Parker

### TESTS:

JIRA | Summary | Priority | Component | Reporter | Contributor
--- | --- | --- | --- | --- | ---
YARN-1119 | Add ClusterMetrics checks to tho TestRMNodeTransitions tests | Major | resourcemanager | Robert Parker | Mit Desai
MAPREDUCE-5610 | TestSleepJob fails in jdk7 | Major | . | Jonathan Eagles | Jonathan Eagles

### SUB-TASKS:

JIRA | Summary | Priority | Component | Reporter | Contributor
--- | --- | --- | --- | --- | ---
HADOOP-9748 | Reduce blocking on UGI.ensureInitialized | Critical | security | Daryn Sharp | Daryn Sharp
YARN-573 | Shared data structures in Public Localizer and Private Localizer are not Thread safe. | Critical | . | Omkar Vinit Joshi | Omkar Vinit Joshi
HDFS-5239 | Allow FSNamesystem lock fairness to be configurable | Major | namenode | Daryn Sharp | Daryn Sharp
HADOOP-9956 | RPC listener inefficiently assigns connections to readers | Major | ipc | Daryn Sharp | Daryn Sharp
HADOOP-9955 | RPC idle connection closing is extremely inefficient | Major | ipc | Daryn Sharp | Daryn Sharp