Apache Hadoop 3.5.0 – Apache Hadoop 0.20.2 Release Notes

##### General

- Overview
- Single Node Setup
- Cluster Setup
- Commands Reference
- FileSystem Shell
- Compatibility Specification
- Downstream Developer's Guide
- Admin Compatibility Guide
- Interface Classification
- FileSystem Specification

##### Common

- CLI Mini Cluster
- Fair Call Queue
- Native Libraries
- Proxy User
- Rack Awareness
- Secure Mode
- Service Level Authorization
- HTTP Authentication
- Credential Provider API
- Hadoop KMS
- Tracing
- Unix Shell Guide
- Registry
- Async Profiler

##### HDFS

- Architecture
- User Guide
- Commands Reference
- NameNode HA With QJM
- NameNode HA With NFS
- Observer NameNode
- Federation
- ViewFs
- ViewFsOverloadScheme
- Snapshots
- Edits Viewer
- Image Viewer
- Permissions and HDFS
- Quotas and HDFS
- libhdfs (C API)
- WebHDFS (REST API)
- HttpFS
- Short Circuit Local Reads
- Centralized Cache Management
- NFS Gateway
- Rolling Upgrade
- Extended Attributes
- Transparent Encryption
- Multihoming
- Storage Policies
- Memory Storage Support
- Synthetic Load Generator
- Erasure Coding
- Disk Balancer
- Upgrade Domain
- DataNode Admin
- Router Federation
- Provided Storage

##### MapReduce

- Tutorial
- Commands Reference
- Encrypted Shuffle
- Pluggable Shuffle/Sort
- Distributed Cache Deploy
- Support for YARN Shared Cache

##### MapReduce REST APIs

- MR Application Master
- MR History Server

##### YARN

- Architecture
- Commands Reference
- Capacity Scheduler
- Fair Scheduler
- ResourceManager Restart
- ResourceManager HA
- Resource Model
- Node Labels
- Node Attributes
- Web Application Proxy
- Timeline Server
- Timeline Service V.2
- Writing YARN Applications
- YARN Application Security
- NodeManager
- Running Applications in Docker Containers
- Running Applications in runC Containers
- Using CGroups
- Secure Containers
- Reservation System
- Graceful Decommission
- Opportunistic Containers
- YARN Federation
- Shared Cache
- Using GPU
- Using FPGA
- Placement Constraints
- YARN UI2

##### YARN REST APIs

- Introduction
- Resource Manager
- Node Manager
- Timeline Server
- Timeline Service V.2

##### YARN Service

- Overview
- QuickStart
- Concepts
- Yarn Service API
- Service Discovery
- System Services

##### Hadoop Compatible File Systems

- Aliyun OSS
- Amazon S3
- Azure Blob Storage
- Azure Data Lake Storage
- Tencent COS
- Huaweicloud OBS
- VolcanoEngine TOS

##### Auth

- Overview
- Examples
- Configuration
- Building

##### Tools

- Hadoop Streaming
- Hadoop Archives
- Hadoop Archive Logs
- DistCp
- HDFS Federation Balance
- GridMix
- Rumen
- Resource Estimator Service
- Scheduler Load Simulator
- Hadoop Benchmarking
- Dynamometer

##### Reference

- Changelog and Release Notes
- Java API docs
- Unix Shell API
- Metrics

##### Configuration

- core-default.xml
- hdfs-default.xml
- hdfs-rbf-default.xml
- mapred-default.xml
- yarn-default.xml
- kms-default.xml
- httpfs-default.xml
- Deprecated Properties

# Apache Hadoop 0.20.2 Release Notes

These release notes cover new developer and user-facing incompatibilities, important issues, features, and major improvements.

- MAPREDUCE-826 | Trivial | harchive doesn’t use ToolRunner / harchive returns 0 even if the job fails with exception

Use ToolRunner for archives job and return non zero error code on failure.

- MAPREDUCE-112 | Blocker | Reduce Input Records and Reduce Output Records counters are not being set when using the new Mapreduce reducer API

Updates of counters for reduce input and output records were added in the new API so they are available for jobs using the new API.

- HADOOP-6231 | Major | Allow caching of filesystem instances to be disabled on a per-instance basis

Allow a general mechanism to disable the cache on a per filesystem basis by using property fs.<schemename>.impl.disable.cache. eg. fs.har.impl.disable.cache in core-default.xml

- MAPREDUCE-979 | Blocker | JobConf.getMemoryFor{Map|Reduce}Task doesn’t fallback to newer config knobs when mapred.taskmaxvmem is set to DISABLED_MEMORY_LIMIT of -1

Added support to fallback to new task memory configuration when deprecated memory configuration values are set to disabled.

- HDFS-677 | Blocker | Rename failure due to quota results in deletion of src directory

Rename properly considers the case where both source and destination are over quota; operation will fail with error indication.

- HADOOP-6097 | Major | Multiple bugs w/ Hadoop archives

Bugs fixed for Hadoop archives: character escaping in paths, LineReader and file system caching.

- HDFS-761 | Major | Failure to process rename operation from edits log due to quota verification

Corrected an error when checking quota policy that resulted in a failure to read the edits log, stopping the primary/secondary name node.

- MAPREDUCE-1068 | Major | In hadoop-0.20.0 streaming job do not throw proper verbose error message if file is not present

Fix streaming job to show proper message if file is is not present, for -file option.

- HDFS-596 | Blocker | Memory leak in libhdfs: hdfsFreeFileInfo() in libhdfs does not free memory for mOwner and mGroup

Memory leak in function hdfsFreeFileInfo in libhdfs. This bug affects fuse-dfs severely.

- MAPREDUCE-1147 | Blocker | Map output records counter missing for map-only jobs in new API

Adds a counter to track the number of records emitted by map writing directly to HDFS i.e map tasks of job with 0 reducers.

- MAPREDUCE-1182 | Blocker | Reducers fail with OutOfMemoryError while copying Map outputs

Modifies shuffle related memory parameters to use ‘long’ from ‘int’ so that sizes greater than maximum integer size are handled correctly

- HDFS-781 | Blocker | Metrics PendingDeletionBlocks is not decremented

Correct PendingDeletionBlocks metric to properly decrement counts.

- HDFS-793 | Blocker | DataNode should first receive the whole packet ack message before it constructs and sends its own ack message for the packet

WARNING: No release note provided for this change.

- HADOOP-6428 | Major | HttpServer sleeps with negative values

Corrected arithmetic error that made sleep times less than zero.

- HADOOP-6460 | Blocker | Namenode runs of out of memory due to memory leak in ipc Server

If an IPC server response buffer has grown to than 1MB, it is replaced by a smaller buffer to free up the Java heap that was used. This will improve the longevity of the name service.

- MAPREDUCE-433 | Major | TestReduceFetch failed.

Resolves the test failure by modifying the test to base it on spill counters rather than on bytes read/written. It also introduces a new configuration parameter “mapred.job.shuffle.input.buffer.percent” to provide finer grained control on the memory limit to be used during shuffle.

- HADOOP-6498 | Blocker | IPC client bug may cause rpc call hang

Correct synchronization error in IPC where handler thread could hang if request reader got an error.

- MAPREDUCE-623 | Major | Resolve javac warnings in mapred

Removes javac warnings by either resolving them or suppressing them (wherever resolution is not possible)