Apache Hadoop 3.5.0 – Apache Hadoop Changelog

##### General

- Overview
- Single Node Setup
- Cluster Setup
- Commands Reference
- FileSystem Shell
- Compatibility Specification
- Downstream Developer's Guide
- Admin Compatibility Guide
- Interface Classification
- FileSystem Specification

##### Common

- CLI Mini Cluster
- Fair Call Queue
- Native Libraries
- Proxy User
- Rack Awareness
- Secure Mode
- Service Level Authorization
- HTTP Authentication
- Credential Provider API
- Hadoop KMS
- Tracing
- Unix Shell Guide
- Registry
- Async Profiler

##### HDFS

- Architecture
- User Guide
- Commands Reference
- NameNode HA With QJM
- NameNode HA With NFS
- Observer NameNode
- Federation
- ViewFs
- ViewFsOverloadScheme
- Snapshots
- Edits Viewer
- Image Viewer
- Permissions and HDFS
- Quotas and HDFS
- libhdfs (C API)
- WebHDFS (REST API)
- HttpFS
- Short Circuit Local Reads
- Centralized Cache Management
- NFS Gateway
- Rolling Upgrade
- Extended Attributes
- Transparent Encryption
- Multihoming
- Storage Policies
- Memory Storage Support
- Synthetic Load Generator
- Erasure Coding
- Disk Balancer
- Upgrade Domain
- DataNode Admin
- Router Federation
- Provided Storage

##### MapReduce

- Tutorial
- Commands Reference
- Encrypted Shuffle
- Pluggable Shuffle/Sort
- Distributed Cache Deploy
- Support for YARN Shared Cache

##### MapReduce REST APIs

- MR Application Master
- MR History Server

##### YARN

- Architecture
- Commands Reference
- Capacity Scheduler
- Fair Scheduler
- ResourceManager Restart
- ResourceManager HA
- Resource Model
- Node Labels
- Node Attributes
- Web Application Proxy
- Timeline Server
- Timeline Service V.2
- Writing YARN Applications
- YARN Application Security
- NodeManager
- Running Applications in Docker Containers
- Running Applications in runC Containers
- Using CGroups
- Secure Containers
- Reservation System
- Graceful Decommission
- Opportunistic Containers
- YARN Federation
- Shared Cache
- Using GPU
- Using FPGA
- Placement Constraints
- YARN UI2

##### YARN REST APIs

- Introduction
- Resource Manager
- Node Manager
- Timeline Server
- Timeline Service V.2

##### YARN Service

- Overview
- QuickStart
- Concepts
- Yarn Service API
- Service Discovery
- System Services

##### Hadoop Compatible File Systems

- Aliyun OSS
- Amazon S3
- Azure Blob Storage
- Azure Data Lake Storage
- Tencent COS
- Huaweicloud OBS
- VolcanoEngine TOS

##### Auth

- Overview
- Examples
- Configuration
- Building

##### Tools

- Hadoop Streaming
- Hadoop Archives
- Hadoop Archive Logs
- DistCp
- HDFS Federation Balance
- GridMix
- Rumen
- Resource Estimator Service
- Scheduler Load Simulator
- Hadoop Benchmarking
- Dynamometer

##### Reference

- Changelog and Release Notes
- Java API docs
- Unix Shell API
- Metrics

##### Configuration

- core-default.xml
- hdfs-default.xml
- hdfs-rbf-default.xml
- mapred-default.xml
- yarn-default.xml
- kms-default.xml
- httpfs-default.xml
- Deprecated Properties

# Apache Hadoop Changelog

## Release 0.20.204.0 - 2011-09-02

### INCOMPATIBLE CHANGES:

JIRA | Summary | Priority | Component | Reporter | Contributor
--- | --- | --- | --- | --- | ---
HADOOP-6255 | Create an rpm integration project | Major | . | Owen O’Malley | Eric Yang

### NEW FEATURES:

JIRA | Summary | Priority | Component | Reporter | Contributor
--- | --- | --- | --- | --- | ---
HADOOP-7144 | Expose JMX with something like JMXProxyServlet | Major | . | Luke Lu | Robert Joseph Evans
MAPREDUCE-2558 | Add queue-level metrics 0.20-security branch | Major | jobtracker | Jeffrey Naisbitt | Jeffrey Naisbitt
HADOOP-7398 | create a mechanism to suppress the HADOOP_HOME deprecated warning | Major | . | Owen O’Malley | Owen O’Malley
MAPREDUCE-1938 | Ability for having user’s classes take precedence over the system classes for tasks’ classpath | Blocker | job submission, task, tasktracker | Devaraj Das | Krishna Ramachandran

### IMPROVEMENTS:

JIRA | Summary | Priority | Component | Reporter | Contributor
--- | --- | --- | --- | --- | ---
HDFS-1773 | Remove a datanode from cluster if include list is not empty and this datanode is removed from both include and exclude lists | Minor | namenode | Tanping Wang | Tanping Wang
MAPREDUCE-2456 | Show the reducer taskid and map/reduce tasktrackers for “Failed fetch notification #_ for task attempt…” log messages | Trivial | jobtracker | Jeffrey Naisbitt | Jeffrey Naisbitt
HADOOP-7248 | Have a way to automatically update Eclipse .classpath file when new libs are added to the classpath through Ivy for 0.20-* based sources | Minor | build | Konstantin Boudnik | Thomas Graves
HADOOP-7277 | Add Eclipse launch tasks for the 0.20-security branch | Minor | build | Jeffrey Naisbitt | Jeffrey Naisbitt
MAPREDUCE-2479 | Backport MAPREDUCE-1568 to hadoop security branch | Major | tasktracker | Robert Joseph Evans | Robert Joseph Evans
MAPREDUCE-2490 | Log blacklist debug count | Trivial | jobtracker | Jonathan Eagles | Jonathan Eagles
MAPREDUCE-2495 | The distributed cache cleanup thread has no monitoring to check to see if it has died for some reason | Minor | distributed-cache | Robert Joseph Evans | Robert Joseph Evans
MAPREDUCE-2524 | Backport trunk heuristics for failing maps when we get fetch failures retrieving map output during shuffle | Minor | tasktracker | Thomas Graves | Thomas Graves

### BUG FIXES:

JIRA | Summary | Priority | Component | Reporter | Contributor
--- | --- | --- | --- | --- | ---
MAPREDUCE-118 | Job.getJobID() will always return null | Blocker | client | Amar Kamat | Amareshwari Sriramadasu
HDFS-1258 | Clearing namespace quota on “/” corrupts FS image | Blocker | namenode | Aaron T. Myers | Aaron T. Myers
HDFS-1377 | Quota bug for partial blocks allows quotas to be violated | Blocker | namenode | Eli Collins | Eli Collins
HDFS-1750 | fs -ls hftp://file not working | Major | . | Tsz Wo Nicholas Sze | Tsz Wo Nicholas Sze
HDFS-1758 | Web UI JSP pages thread safety issue | Minor | tools | Tanping Wang | Tanping Wang
MAPREDUCE-2411 | When you submit a job to a queue with no ACLs you get an inscrutible NPE | Minor | . | Dick King | Dick King
HDFS-1189 | Quota counts missed between clear quota and set quota | Major | namenode | Kang Xiao | John George
MAPREDUCE-2443 | Fix FI build - broken after MR-2429 | Minor | test | Siddharth Seth | Siddharth Seth
HADOOP-7232 | Fix javadoc warnings | Blocker | documentation | Owen O’Malley | Owen O’Malley
HDFS-1822 | Editlog opcodes overlap between 20 security and later releases | Blocker | namenode | Suresh Srinivas | Suresh Srinivas
MAPREDUCE-2451 | Log the reason string of healthcheck script | Trivial | jobtracker | Thomas Graves | Thomas Graves
HADOOP-7274 | CLONE - IOUtils.readFully and IOUtils.skipFully have typo in exception creation’s message | Minor | util | Jonathan Eagles | Jonathan Eagles
MAPREDUCE-2514 | ReinitTrackerAction class name misspelled RenitTrackerAction in task tracker log | Trivial | tasktracker | Jonathan Eagles | Jonathan Eagles
HDFS-1592 | Datanode startup doesn’t honor volumes.tolerated | Major | . | Bharath Mundlapudi | Bharath Mundlapudi
HADOOP-7330 | The metrics source mbean implementation should return the attribute value instead of the object | Major | metrics | Luke Lu | Luke Lu
HDFS-2023 | Backport of NPE for File.list and File.listFiles | Major | datanode | Bharath Mundlapudi | Bharath Mundlapudi
MAPREDUCE-2529 | Recognize Jetty bug 1342 and handle it | Major | tasktracker | Thomas Graves | Thomas Graves
HADOOP-7364 | TestMiniMRDFSCaching fails if test.build.dir is set to something other than build/test | Major | test | Thomas Graves | Thomas Graves
HDFS-1878 | TestHDFSServerPorts unit test failure - race condition in FSNamesystem.close() causes NullPointerException without serious consequence | Minor | namenode | Matt Foley | Matt Foley
HDFS-2057 | Wait time to terminate the threads causing unit tests to take longer time | Major | datanode | Bharath Mundlapudi | Bharath Mundlapudi
HADOOP-7373 | Tarball deployment doesn’t work with {start,stop}-{dfs,mapred} | Major | . | Owen O’Malley | Owen O’Malley
HDFS-1692 | In secure mode, Datanode process doesn’t exit when disks fail. | Major | datanode | Bharath Mundlapudi | Bharath Mundlapudi
HADOOP-7324 | Ganglia plugins for metrics v2 | Blocker | metrics | Luke Lu | Priyo Mustafi
MAPREDUCE-2409 | Distributed Cache does not differentiate between file /archive for files with the same path | Major | distributed-cache | Siddharth Seth | Siddharth Seth
HADOOP-7475 | hadoop-setup-single-node.sh is broken | Blocker | . | Eric Yang | Eric Yang
MAPREDUCE-2447 | Set JvmContext sooner for a task - MR2429 | Minor | . | Siddharth Seth | Siddharth Seth
MAPREDUCE-2621 | TestCapacityScheduler fails with “Queue “q1” does not exist” | Minor | . | Sherry Chen | Sherry Chen
MAPREDUCE-2651 | Race condition in Linux Task Controller for job log directory creation | Major | task-controller | Bharath Mundlapudi | Bharath Mundlapudi
HADOOP-7356 | RPM packages broke bin/hadoop script for hadoop 0.20.205 | Blocker | . | Eric Yang | Eric Yang
MAPREDUCE-2846 | a small % of all tasks fail with DefaultTaskController | Blocker | task, task-controller, tasktracker | Allen Wittenauer | Owen O’Malley
MAPREDUCE-2804 | “Creation of symlink to attempt log dir failed.” message is not useful | Blocker | . | Allen Wittenauer | Owen O’Malley
MAPREDUCE-2358 | MapReduce assumes HDFS as the default filesystem | Major | . | Owen O’Malley | Krishna Ramachandran
MAPREDUCE-517 | The capacity-scheduler should assign multiple tasks per heartbeat | Critical | . | Arun C Murthy | Arun C Murthy
MAPREDUCE-2357 | When extending inputsplit (non-FileSplit), all exceptions are ignored | Major | task | Owen O’Malley | Luke Lu
MAPREDUCE-2366 | TaskTracker can’t retrieve stdout and stderr from web UI | Major | tasktracker | Owen O’Malley | Dick King
MAPREDUCE-2362 | Unit test failures: TestBadRecords and TestTaskTrackerMemoryManager | Major | test | Owen O’Malley | Greg Roelofs
MAPREDUCE-2356 | A task succeeded even though there were errors on all attempts. | Major | . | Owen O’Malley | Luke Lu
MAPREDUCE-2555 | JvmInvalidate errors in the gridmix TT logs | Minor | tasktracker | Thomas Graves | Thomas Graves
MAPREDUCE-2360 | Pig fails when using non-default FileSystem | Major | client | Owen O’Malley |
MAPREDUCE-2359 | Distributed cache doesn’t use non-default FileSystems correctly | Major | . | Owen O’Malley | Krishna Ramachandran
MAPREDUCE-2418 | Errors not shown in the JobHistory servlet (specifically Counter Limit Exceeded) | Minor | . | Siddharth Seth | Siddharth Seth
MAPREDUCE-2429 | Check jvmid during task status report | Major | tasktracker | Arun C Murthy | Siddharth Seth
MAPREDUCE-2364 | Shouldn’t hold lock on rjob while localizing resources. | Major | tasktracker | Owen O’Malley | Devaraj Das

### TESTS:

JIRA | Summary | Priority | Component | Reporter | Contributor
--- | --- | --- | --- | --- | ---
HDFS-2044 | TestQueueProcessingStatistics failing automatic test due to timing issues | Major | test | Matt Foley | Matt Foley
HDFS-2218 | Disable TestHdfsProxy.testHdfsProxyInterface in 0.20-security and branch-1 until HDFS-2217 is fixed | Blocker | contrib/hdfsproxy, test | Matt Foley | Matt Foley

### SUB-TASKS:

JIRA | Summary | Priority | Component | Reporter | Contributor
--- | --- | --- | --- | --- | ---
HDFS-1445 | Batch the calls in DataStorage to FileUtil.createHardLink(), so we call it once per directory instead of once per file | Major | datanode | Matt Foley | Matt Foley
HDFS-1541 | Not marking datanodes dead When namenode in safemode | Major | namenode | Hairong Kuang | Hairong Kuang
HDFS-1767 | Namenode should ignore non-initial block reports from datanodes when in safemode during startup | Major | datanode | Matt Foley | Matt Foley
MAPREDUCE-2413 | TaskTracker should handle disk failures at both startup and runtime | Major | task-controller, tasktracker | Bharath Mundlapudi | Ravi Gummadi
MAPREDUCE-2415 | Distribute TaskTracker userlogs onto multiple disks | Major | task-controller, tasktracker | Bharath Mundlapudi | Bharath Mundlapudi