Apache Hadoop 3.5.0 – Apache Hadoop Changelog

##### General

- Overview
- Single Node Setup
- Cluster Setup
- Commands Reference
- FileSystem Shell
- Compatibility Specification
- Downstream Developer's Guide
- Admin Compatibility Guide
- Interface Classification
- FileSystem Specification

##### Common

- CLI Mini Cluster
- Fair Call Queue
- Native Libraries
- Proxy User
- Rack Awareness
- Secure Mode
- Service Level Authorization
- HTTP Authentication
- Credential Provider API
- Hadoop KMS
- Tracing
- Unix Shell Guide
- Registry
- Async Profiler

##### HDFS

- Architecture
- User Guide
- Commands Reference
- NameNode HA With QJM
- NameNode HA With NFS
- Observer NameNode
- Federation
- ViewFs
- ViewFsOverloadScheme
- Snapshots
- Edits Viewer
- Image Viewer
- Permissions and HDFS
- Quotas and HDFS
- libhdfs (C API)
- WebHDFS (REST API)
- HttpFS
- Short Circuit Local Reads
- Centralized Cache Management
- NFS Gateway
- Rolling Upgrade
- Extended Attributes
- Transparent Encryption
- Multihoming
- Storage Policies
- Memory Storage Support
- Synthetic Load Generator
- Erasure Coding
- Disk Balancer
- Upgrade Domain
- DataNode Admin
- Router Federation
- Provided Storage

##### MapReduce

- Tutorial
- Commands Reference
- Encrypted Shuffle
- Pluggable Shuffle/Sort
- Distributed Cache Deploy
- Support for YARN Shared Cache

##### MapReduce REST APIs

- MR Application Master
- MR History Server

##### YARN

- Architecture
- Commands Reference
- Capacity Scheduler
- Fair Scheduler
- ResourceManager Restart
- ResourceManager HA
- Resource Model
- Node Labels
- Node Attributes
- Web Application Proxy
- Timeline Server
- Timeline Service V.2
- Writing YARN Applications
- YARN Application Security
- NodeManager
- Running Applications in Docker Containers
- Running Applications in runC Containers
- Using CGroups
- Secure Containers
- Reservation System
- Graceful Decommission
- Opportunistic Containers
- YARN Federation
- Shared Cache
- Using GPU
- Using FPGA
- Placement Constraints
- YARN UI2

##### YARN REST APIs

- Introduction
- Resource Manager
- Node Manager
- Timeline Server
- Timeline Service V.2

##### YARN Service

- Overview
- QuickStart
- Concepts
- Yarn Service API
- Service Discovery
- System Services

##### Hadoop Compatible File Systems

- Aliyun OSS
- Amazon S3
- Azure Blob Storage
- Azure Data Lake Storage
- Tencent COS
- Huaweicloud OBS
- VolcanoEngine TOS

##### Auth

- Overview
- Examples
- Configuration
- Building

##### Tools

- Hadoop Streaming
- Hadoop Archives
- Hadoop Archive Logs
- DistCp
- HDFS Federation Balance
- GridMix
- Rumen
- Resource Estimator Service
- Scheduler Load Simulator
- Hadoop Benchmarking
- Dynamometer

##### Reference

- Changelog and Release Notes
- Java API docs
- Unix Shell API
- Metrics

##### Configuration

- core-default.xml
- hdfs-default.xml
- hdfs-rbf-default.xml
- mapred-default.xml
- yarn-default.xml
- kms-default.xml
- httpfs-default.xml
- Deprecated Properties

# Apache Hadoop Changelog

## Release 0.15.0 - 2007-10-19

### NEW FEATURES:

JIRA | Summary | Priority | Component | Reporter | Contributor
--- | --- | --- | --- | --- | ---
HADOOP-1727 | Make …hbase.io.MapWritable more generic so that it can be included in …hadoop.io | Minor | io | Jim Kellerman | Jim Kellerman
HADOOP-1351 | Want to kill a particular task or attempt | Major | . | Owen O’Malley | Enis Soztutar
HADOOP-1880 | SleepJob | Major | . | Enis Soztutar | Enis Soztutar
HADOOP-1809 | Add link to irc channel #hadoop | Major | . | Enis Soztutar | Enis Soztutar
HADOOP-1894 | Add fancy graphs for mapred task statuses | Major | . | Enis Soztutar | Enis Soztutar
HADOOP-1914 | HDFS should have a NamenodeProtocol to allow secondary namenodes and rebalancing processes to communicate with a primary namenode | Major | . | Hairong Kuang | Hairong Kuang
HADOOP-1851 | Map output compression codec cannot be set independently of job output compression codec | Major | . | Riccardo Boscolo | Arun C Murthy
HADOOP-1963 | Code contribution of Kosmos Filesystem implementation of Hadoop Filesystem interface | Major | fs | Sriram Rao | Sriram Rao
HADOOP-1822 | Allow SOCKS proxy configuration to remotely access the DFS and submit Jobs | Minor | ipc | Christophe Taton | Christophe Taton
HADOOP-789 | DFS shell should return a list of nodes for a file saying that where the blocks for these files are located. | Minor | . | Mahadev konar | Mahadev konar

### IMPROVEMENTS:

JIRA | Summary | Priority | Component | Reporter | Contributor
--- | --- | --- | --- | --- | ---
HADOOP-1667 | organize CHANGES.txt messages into sections for future releases | Major | documentation | Doug Cutting | Doug Cutting
HADOOP-1610 | Add metrics for failed tasks | Major | . | Devaraj Das | Devaraj Das
HADOOP-1621 | Make FileStatus a concrete class | Major | fs | Chris Douglas | Chris Douglas
HADOOP-1595 | Add an option to setReplication method to wait for completion of replication | Major | . | Christian Kunz | Tsz Wo Nicholas Sze
HADOOP-1693 | Remove LOG members from PendingReplicationBlocks and ReplicationTargetChooser. | Major | . | Konstantin Shvachko | Konstantin Shvachko
HADOOP-1231 | Add generics to Mapper and Reducer interfaces | Major | . | Owen O’Malley | Tom White
HADOOP-1436 | Redesign Tool and ToolBase API and releted functionality | Major | util | Enis Soztutar | Enis Soztutar
HADOOP-1703 | Small cleanup of DistributedFileSystem and DFSClient | Trivial | . | Christophe Taton | Christophe Taton
HADOOP-1743 | INode refactoring | Major | . | Konstantin Shvachko | Konstantin Shvachko
HADOOP-1731 | contrib jar file names should include hadoop version number | Major | . | Doug Cutting | Doug Cutting
HADOOP-1756 | Add toString() methods to some Writable types | Major | io | Andrzej Bialecki | Andrzej Bialecki
HADOOP-1744 | Small cleanup of DistributedFileSystem and DFSClient (next) | Trivial | . | Christophe Taton | Christophe Taton
HADOOP-1654 | IOUtils class | Major | io | Enis Soztutar | Enis Soztutar
HADOOP-1759 | File name should be represented by a byte array instead of a String | Major | . | Konstantin Shvachko | Konstantin Shvachko
HADOOP-1766 | Merging Block and BlockInfo classes on name-node. | Major | . | Konstantin Shvachko | Konstantin Shvachko
HADOOP-1158 | JobTracker should collect statistics of failed map output fetches, and take decisions to reexecute map tasks and/or restart the (possibly faulty) Jetty server on the TaskTracker | Major | . | Devaraj Das | Arun C Murthy
HADOOP-1777 | Typo issue in the job details JSP page | Trivial | . | Thomas Friol | Thomas Friol
HADOOP-1767 | JobClient CLI cleanup and improvement | Minor | . | Christophe Taton | Christophe Taton
HADOOP-1750 | We should log better if something goes wrong with the process fork | Major | . | Owen O’Malley | Owen O’Malley
HADOOP-1803 | Generalize making contrib bin content executable in ant package target | Minor | build | stack | stack
HADOOP-1779 | Small INodeDirectory enhancement to get all existing INodes components on a path | Trivial | . | Christophe Taton | Christophe Taton
HADOOP-1425 | Rework the various programs in ‘examples’ to extend ToolBase | Minor | . | Arun C Murthy | Enis Soztutar
HADOOP-1687 | Name-node memory size estimates and optimization proposal. | Major | . | Konstantin Shvachko | Konstantin Shvachko
HADOOP-1266 | Remove DatanodeDescriptor dependency from NetworkTopology | Major | . | Konstantin Shvachko | Hairong Kuang
HADOOP-785 | Divide the server and client configurations | Major | conf | Owen O’Malley | Arun C Murthy
HADOOP-1718 | Test coverage target in build files using clover | Major | build | Nigel Daley | Nigel Daley
HADOOP-1774 | Remove use of INode.parent in Block CRC upgrade | Major | . | Raghu Angadi | Raghu Angadi
HADOOP-1592 | Print the diagnostic error messages for FAILED task-attempts to the user console via TaskCompletionEvents | Major | . | Arun C Murthy | Amar Kamat
HADOOP-1878 | Change priority feature in the job details JSP page misses spaces between each priority link | Trivial | . | Thomas Friol | Thomas Friol
HADOOP-1762 | Namenode does not need to store storageID and datanodeID persistently | Major | . | Raghu Angadi | Raghu Angadi
HADOOP-1500 | typo’s in dfs webui | Trivial | . | Nigel Daley | Nigel Daley
HADOOP-1921 | Save the configuration of completed/failed jobs and make them available via the web-ui. | Major | . | Arun C Murthy | Amar Kamat
HADOOP-1908 | Restructure data node code so that block sending/receiving is seperated from data transfer header handling | Major | . | Hairong Kuang | Hairong Kuang
HADOOP-1933 | Consider include/exclude files while listing datanodes. | Major | . | Raghu Angadi | Raghu Angadi
HADOOP-1926 | Design/implement a set of compression benchmarks for the map-reduce framework | Major | . | Arun C Murthy | Arun C Murthy
HADOOP-1626 | DFSAdmin. Help messages are missing for -finalizeUpgrade and -metasave. | Blocker | . | Konstantin Shvachko | Lohit Vijayarenu
HADOOP-1968 | Wildcard input syntax (glob) should support {} | Major | fs | eric baldeschwieler | Hairong Kuang
HADOOP-1906 | JobConf should warn about the existance of obsolete mapred-default.xml. | Major | conf | Owen O’Malley | Arun C Murthy
HADOOP-1971 | Constructing a JobConf without a class leads to a very misleading error message. | Minor | . | Ted Dunning | Enis Soztutar
HADOOP-1942 | Increase the concurrency of transaction logging to edits log | Blocker | . | dhruba borthakur | dhruba borthakur
HADOOP-2046 | Documentation: improve mapred javadocs | Blocker | documentation | Arun C Murthy | Arun C Murthy
HADOOP-1881 | Update documentation for hadoop’s configuration post HADOOP-785 | Major | documentation | Arun C Murthy | Arun C Murthy

### BUG FIXES:

JIRA | Summary | Priority | Component | Reporter | Contributor
--- | --- | --- | --- | --- | ---
HADOOP-1636 | constant should be user-configurable: MAX_COMPLETE_USER_JOBS_IN_MEMORY | Major | . | Michael Bieniosek | Michael Bieniosek
HADOOP-1463 | dfs.datanode.du.reserved semantics being violated | Blocker | . | Hairong Kuang | Hairong Kuang
HADOOP-999 | DFS Client should create file when the user creates the file | Major | . | Owen O’Malley | Tsz Wo Nicholas Sze
HADOOP-1565 | DFSScalability: reduce memory usage of namenode | Major | . | dhruba borthakur | dhruba borthakur
HADOOP-1651 | Some improvements in progress reporting | Major | . | Devaraj Das | Devaraj Das
HADOOP-932 | File locking interface and implementation should be remvoed. | Minor | fs | Raghu Angadi | Raghu Angadi
HADOOP-1692 | DfsTask cache interferes with operation | Minor | util | Chris Douglas | Chris Douglas
HADOOP-1689 | .sh scripts do not work on Solaris | Minor | scripts | David Biesack | Doug Cutting
HADOOP-1772 | Hadoop does not run in Cygwin in Windows | Critical | scripts | Tsz Wo Nicholas Sze | Tsz Wo Nicholas Sze
HADOOP-1749 | TestDFSUpgrade some times fails with an assert | Major | . | Raghu Angadi | Enis Soztutar
HADOOP-1601 | GenericWritable should use ReflectionUtils.newInstance to avoid problems with classloaders | Major | io | Owen O’Malley | Enis Soztutar
HADOOP-1748 | Task Trackers fail to launch tasks when they have relative log directories configured | Major | . | Owen O’Malley | Owen O’Malley
HADOOP-1775 | MapWritable and SortedMapWritable - Writable problems | Major | io | Jim Kellerman | Jim Kellerman
HADOOP-1656 | HDFS does not record the blocksize for a file | Major | . | Sameer Paranjpye | dhruba borthakur
HADOOP-1739 | ConnectException in TaskTracker Child | Major | . | Srikanth Kakani | Doug Cutting
HADOOP-1795 | Task.moveTaskOutputs is escaping special characters in output filenames | Critical | . | Frédéric Bertin | Frédéric Bertin
HADOOP-1812 | TestIPC and TestRPC should use dynamically allocated ports | Major | ipc | Doug Cutting | Doug Cutting
HADOOP-1810 | Incorrect Value type in MRBench (SmallJobs) | Blocker | . | Devaraj Das | Devaraj Das
HADOOP-1832 | listTables() returns duplicate tables | Major | . | Andrew Hitchcock | Jim Kellerman
HADOOP-1806 | DfsTask no longer compiles | Major | build | Chris Douglas | Chris Douglas
HADOOP-1758 | processing escapes in a jute record is quadratic | Blocker | record | Dick King | Vivek Ratan
HADOOP-1569 | distcp should use the Path -> FileSystem interface like the rest of Hadoop | Major | util | Owen O’Malley | Chris Douglas
HADOOP-1817 | MultiFileSplit does not write and read the total length | Major | . | Thomas Friol | Thomas Friol
HADOOP-1838 | Files created with an pre-0.15 gets blocksize as zero, causing performance degradation | Blocker | . | dhruba borthakur | dhruba borthakur
HADOOP-1018 | Single lost heartbeat leads to a “Lost task tracker” | Major | . | Andrzej Bialecki | Arun C Murthy
HADOOP-1818 | MutliFileInputFormat returns “empty” MultiFileSplit when number of paths < number of splits | Major | . | Thomas Friol | Thomas Friol
HADOOP-1840 | Task’s diagnostic messages are lost sometimes | Critical | . | Arun C Murthy | Arun C Murthy
HADOOP-1846 | DatanodeReport should distinguish live datanodes from dead datanodes | Major | . | Hairong Kuang | Hairong Kuang
HADOOP-1885 | Race condition in MiniDFSCluster shutdown | Major | test | Chris Douglas | Chris Douglas
HADOOP-1708 | make files visible in the namespace as soon as they are created | Major | . | dhruba borthakur | dhruba borthakur
HADOOP-1819 | The JobTracker should ensure that it is running on the right host. | Major | . | Owen O’Malley | Owen O’Malley
HADOOP-1890 | Revert a debug patch. | Trivial | . | Raghu Angadi | Raghu Angadi
HADOOP-1889 | Fix path in EC2 scripts for building your own AMI | Major | contrib/cloud | Tom White | Tom White
HADOOP-1892 | In the Job UI, some links don’t work | Major | . | Devaraj Das | Amar Kamat
HADOOP-120 | Reading an ArrayWriter does not work because valueClass does not get initialized | Major | io | Dick King | Cameron Pope
HADOOP-1907 | JobClient.runJob kills the job for failed tasks with no diagnostics | Major | . | Christian Kunz | Christian Kunz
HADOOP-1904 | ArrayIndexOutOfBoundException in BlocksMap | Blocker | . | Konstantin Shvachko | Konstantin Shvachko
HADOOP-1887 | ArrayIndexOutOfBoundsException with trunk | Major | . | Raghu Angadi | dhruba borthakur
HADOOP-89 | files are not visible until they are closed | Critical | . | Yoram Arnon | dhruba borthakur
HADOOP-1882 | Remove extra ’*’s from FsShell.limitDecimal() | Minor | . | Raghu Angadi | Raghu Angadi
HADOOP-1573 | Support for 0 reducers in PIPES | Major | . | Christian Kunz | Owen O’Malley
HADOOP-1783 | keyToPath in Jets3tFileSystemStore needs to return absolute path | Major | fs/s3 | Ahad Rana | Tom White
HADOOP-1925 | Hadoop Pipes doesn’t compile on solaris | Major | . | Owen O’Malley | Owen O’Malley
HADOOP-1076 | Periodic checkpointing cannot resume if the secondary name-node fails. | Major | . | Konstantin Shvachko | dhruba borthakur
HADOOP-1932 | Test dfs.TestFileCreation.testFileCreation failed on Windows | Blocker | test | Mukund Madhugiri | dhruba borthakur
HADOOP-1940 | TestDFSUpgradeFromImage doesn’t shut down its MiniDFSCluster | Major | test | Chris Douglas | Chris Douglas
HADOOP-1930 | Too many fetch-failures issue | Blocker | . | Christian Kunz | Arun C Murthy
HADOOP-1910 | Extra checks in DFS.create() are not necessary. | Minor | . | Raghu Angadi | Raghu Angadi
HADOOP-1897 | about.html page is there but not linked. | Major | . | Enis Soztutar | Enis Soztutar
HADOOP-1953 | the job tracker should wait beteween calls to try and delete the system directory | Blocker | . | Owen O’Malley | Owen O’Malley
HADOOP-1935 | NullPointerException in internalReleaseCreate | Blocker | . | Konstantin Shvachko | dhruba borthakur
HADOOP-1946 | du should be not called on every heartbeat | Blocker | . | Konstantin Shvachko | Hairong Kuang
HADOOP-1948 | Spurious error message during block crc upgrade. | Blocker | . | Raghu Angadi | Raghu Angadi
HADOOP-1934 | the os.name string on Mac OS contains spaces, which causes the c++ compilation to fail | Major | . | Owen O’Malley | Owen O’Malley
HADOOP-1959 | Use of File.separator in StatusHttpServer prevents running Junit tests inside eclipse on Windows | Minor | . | Jim Kellerman | Jim Kellerman
HADOOP-1978 | Name-node should remove edits.new during startup rather than renaming it to edits. | Blocker | . | Konstantin Shvachko | Konstantin Shvachko
HADOOP-1695 | Secondary Namenode halt when SocketTimeoutException at startup | Blocker | . | Koji Noguchi | dhruba borthakur
HADOOP-1955 | Corrupted block replication retries for ever | Blocker | . | Koji Noguchi | Raghu Angadi
HADOOP-1961 | -get, -copyToLocal fail when single filename is passed | Blocker | . | Koji Noguchi | Raghu Angadi
HADOOP-1997 | TestCheckpoint fails on Windows | Blocker | . | Konstantin Shvachko | Konstantin Shvachko
HADOOP-1983 | jobs using pipes interface with tasks not using java output format have a good chance of not updating progress and timing out | Major | . | Christian Kunz | Owen O’Malley
HADOOP-1875 | multiple dfs.client.buffer.dir directories are not treated as alternatives | Blocker | fs | Christian Kunz | Hairong Kuang
HADOOP-1992 | Sort validation is taking considerably longer than before | Blocker | . | Mukund Madhugiri | Arun C Murthy
HADOOP-1874 | lost task trackers – jobs hang | Blocker | . | Christian Kunz | Devaraj Das
HADOOP-1944 | Maps which ran on trackers declared ‘lost’ are being marked as FAILED rather than KILLED | Blocker | . | Arun C Murthy | Devaraj Das
HADOOP-2026 | Namenode prints out too many log lines for “Number of transactions” | Blocker | . | dhruba borthakur | dhruba borthakur
HADOOP-2022 | Task times are not saved correctly (bug in hadoop-1874) | Blocker | . | Devaraj Das | Amar Kamat
HADOOP-2031 | Lost tasktracker not handled properly leading to tips wrongly being kept as completed, and hence not rescheduled | Blocker | . | Devaraj Das | Devaraj Das
HADOOP-2018 | Broken pipe SocketException in DataNode$DataXceiver | Blocker | . | Konstantin Shvachko | Hairong Kuang
HADOOP-2023 | TestLocalDirAllocator fails on Windows | Blocker | fs | Mukund Madhugiri | Hairong Kuang
HADOOP-2016 | Race condition in removing a KILLED task from tasktracker | Blocker | . | Devaraj Das | Arun C Murthy
HADOOP-1771 | streaming hang when IOException in MROutputThread. (NPE) | Blocker | . | Koji Noguchi | Lohit Vijayarenu
HADOOP-2028 | distcp fails if log dir not specified and destination not present | Blocker | util | Chris Douglas | Chris Douglas
HADOOP-1788 | Increase the buffer size of pipes from 1k to 128k | Blocker | . | Owen O’Malley | Amareshwari Sriramadasu
HADOOP-2051 | JobTracker’s TaskCommitQueue is vulnerable to non-IOExceptions | Blocker | . | Arun C Murthy | Arun C Murthy
HADOOP-1973 | NPE at JobTracker startup.. | Blocker | . | Gautam Kowshik | Amareshwari Sriramadasu
HADOOP-2044 | Namenode encounters ClassCastException exceptions for INodeFileUnderConstruction | Blocker | . | dhruba borthakur | dhruba borthakur
HADOOP-2033 | In SequenceFile sync doesn’t work unless the file is compressed (block or record) | Blocker | . | Owen O’Malley | Owen O’Malley
HADOOP-2072 | RawLocalFileStatus is causing Path problems | Major | fs | Dennis Kubes |
HADOOP-2070 | Test org.apache.hadoop.mapred.pipes.TestPipes.unknown failed | Blocker | . | Mukund Madhugiri | Owen O’Malley
HADOOP-2080 | ChecksumFileSystem checksum file size incorrect. | Blocker | fs | Richard Lee | Owen O’Malley
HADOOP-2048 | DISTCP mapper should report progress more often | Blocker | . | Runping Qi | Chris Douglas
HADOOP-2073 | Datanode corruption if machine dies while writing VERSION file | Blocker | . | Michael Bieniosek | Konstantin Shvachko
HADOOP-1825 | hadoop-daemon.sh script fails if HADOOP_PID_DIR doesn’t exist | Minor | scripts | Michael Bieniosek | Michael Bieniosek
HADOOP-2103 | HADOOP-2046 caused some javadoc anomalies | Major | documentation | Nigel Daley | Nigel Daley
HADOOP-2102 | ToolBase doesn’t keep configuration | Blocker | util | Dennis Kubes | Dennis Kubes
HADOOP-795 | hdfs -cp /a/b/c /x/y acts like hdfs -cp /a/b/c/* /x/y | Minor | . | arkady borkovsky | Mahadev konar
HADOOP-1316 | “Go to parent directory” does not work on windows. | Minor | . | Konstantin Shvachko | Mahadev konar
HADOOP-1792 | df command doesn’t exist under windows | Major | fs | Benjamin Francisoud | Mahadev konar

### OTHER:

JIRA | Summary | Priority | Component | Reporter | Contributor
--- | --- | --- | --- | --- | ---
HADOOP-1879 | Warnings With JDK1.6.0_02 | Minor | . | Nilay Vaish | Nilay Vaish