Apache Hadoop 3.5.0 – Apache Hadoop 0.23.2 Release Notes

##### General

- Overview
- Single Node Setup
- Cluster Setup
- Commands Reference
- FileSystem Shell
- Compatibility Specification
- Downstream Developer's Guide
- Admin Compatibility Guide
- Interface Classification
- FileSystem Specification

##### Common

- CLI Mini Cluster
- Fair Call Queue
- Native Libraries
- Proxy User
- Rack Awareness
- Secure Mode
- Service Level Authorization
- HTTP Authentication
- Credential Provider API
- Hadoop KMS
- Tracing
- Unix Shell Guide
- Registry
- Async Profiler

##### HDFS

- Architecture
- User Guide
- Commands Reference
- NameNode HA With QJM
- NameNode HA With NFS
- Observer NameNode
- Federation
- ViewFs
- ViewFsOverloadScheme
- Snapshots
- Edits Viewer
- Image Viewer
- Permissions and HDFS
- Quotas and HDFS
- libhdfs (C API)
- WebHDFS (REST API)
- HttpFS
- Short Circuit Local Reads
- Centralized Cache Management
- NFS Gateway
- Rolling Upgrade
- Extended Attributes
- Transparent Encryption
- Multihoming
- Storage Policies
- Memory Storage Support
- Synthetic Load Generator
- Erasure Coding
- Disk Balancer
- Upgrade Domain
- DataNode Admin
- Router Federation
- Provided Storage

##### MapReduce

- Tutorial
- Commands Reference
- Encrypted Shuffle
- Pluggable Shuffle/Sort
- Distributed Cache Deploy
- Support for YARN Shared Cache

##### MapReduce REST APIs

- MR Application Master
- MR History Server

##### YARN

- Architecture
- Commands Reference
- Capacity Scheduler
- Fair Scheduler
- ResourceManager Restart
- ResourceManager HA
- Resource Model
- Node Labels
- Node Attributes
- Web Application Proxy
- Timeline Server
- Timeline Service V.2
- Writing YARN Applications
- YARN Application Security
- NodeManager
- Running Applications in Docker Containers
- Running Applications in runC Containers
- Using CGroups
- Secure Containers
- Reservation System
- Graceful Decommission
- Opportunistic Containers
- YARN Federation
- Shared Cache
- Using GPU
- Using FPGA
- Placement Constraints
- YARN UI2

##### YARN REST APIs

- Introduction
- Resource Manager
- Node Manager
- Timeline Server
- Timeline Service V.2

##### YARN Service

- Overview
- QuickStart
- Concepts
- Yarn Service API
- Service Discovery
- System Services

##### Hadoop Compatible File Systems

- Aliyun OSS
- Amazon S3
- Azure Blob Storage
- Azure Data Lake Storage
- Tencent COS
- Huaweicloud OBS
- VolcanoEngine TOS

##### Auth

- Overview
- Examples
- Configuration
- Building

##### Tools

- Hadoop Streaming
- Hadoop Archives
- Hadoop Archive Logs
- DistCp
- HDFS Federation Balance
- GridMix
- Rumen
- Resource Estimator Service
- Scheduler Load Simulator
- Hadoop Benchmarking
- Dynamometer

##### Reference

- Changelog and Release Notes
- Java API docs
- Unix Shell API
- Metrics

##### Configuration

- core-default.xml
- hdfs-default.xml
- hdfs-rbf-default.xml
- mapred-default.xml
- yarn-default.xml
- kms-default.xml
- httpfs-default.xml
- Deprecated Properties

# Apache Hadoop 0.23.2 Release Notes

These release notes cover new developer and user-facing incompatibilities, important issues, features, and major improvements.

- HDFS-2887 | Major | Define a FSVolume interface

FSVolume, is a part of FSDatasetInterface implementation, should not be referred outside FSDataset. A new FSVolumeInterface is defined. The BlockVolumeChoosingPolicy.chooseVolume(..) method signature is also updated.

- HDFS-2950 | Minor | Secondary NN HTTPS address should be listed as a NAMESERVICE_SPECIFIC_KEY

The configuration dfs.secondary.https.port has been renamed to dfs.namenode.secondary.https-port for consistency. The old configuration is still supported via a deprecation path.

- MAPREDUCE-3634 | Major | All daemons should crash instead of hanging around when their EventHandlers get exceptions

Fixed all daemons to crash instead of hanging around when their EventHandlers get exceptions.

- MAPREDUCE-3798 | Major | TestJobCleanup testCustomCleanup is failing

Fixed failing TestJobCleanup.testCusomCleanup() and moved it to the maven build.

- HDFS-2907 | Minor | Make FSDataset in Datanode Pluggable

Add a private conf property dfs.datanode.fsdataset.factory to make FSDataset in Datanode pluggable.

- MAPREDUCE-3738 | Critical | NM can hang during shutdown if AppLogAggregatorImpl thread dies unexpectedly

Committed to trunk and branch-0.23. Thanks Jason.

- MAPREDUCE-3866 | Minor | bin/yarn prints the command line unnecessarily

Fixed the bin/yarn script to not print the command line unnecessarily.

- MAPREDUCE-3730 | Minor | Allow restarted NM to rejoin cluster before RM expires it

Modified RM to allow restarted NMs to be able to join the cluster without waiting for expiry.

- MAPREDUCE-2793 | Critical | [MR-279] Maintain consistency in naming appIDs, jobIDs and attemptIDs

Corrected AppIDs, JobIDs, TaskAttemptIDs to be of correct format on the web pages.

- MAPREDUCE-3910 | Blocker | user not allowed to submit jobs even though queue -showacls shows it allows

Fixed a bug in CapacityScheduler LeafQueue which was causing app-submission to fail.

- MAPREDUCE-3686 | Critical | history server web ui - job counter values for map/reduce not shown properly

Fixed two bugs in Counters because of which web app displays zero counter values for framework counters.

- MAPREDUCE-3901 | Major | lazy load JobHistory Task and TaskAttempt details

Modified JobHistory records in YARN to lazily load job and task reports so as to improve UI response times.

- MAPREDUCE-2855 | Major | ResourceBundle lookup during counter name resolution takes a lot of time

Passing a cached class-loader to ResourceBundle creator to minimize counter names lookup time.

- MAPREDUCE-3922 | Minor | Fix the potential problem compiling 32 bit binaries on a x86_64 host.

Fixed build to not compile 32bit container-executor binary by default on all platforms.

- MAPREDUCE-3931 | Major | MR tasks failing due to changing timestamps on Resources to download

Changed PB implementation of LocalResource to take locks so that race conditions don’t fail tasks by inadvertantly changing the timestamps.

- MAPREDUCE-3920 | Major | Revise yarn default port number selection

port number changes for resourcemanager and nodemanager

- HADOOP-8131 | Critical | FsShell put doesn’t correctly handle a non-existent dir

WARNING: No release note provided for this change.

- MAPREDUCE-3792 | Critical | job -list displays only the jobs submitted by a particular user

Fix “bin/mapred job -list” to display all jobs instead of only the jobs owned by the user.

- MAPREDUCE-3614 | Major | ** finalState UNDEFINED if AM is killed by hand**

Fixed MR AM to close history file quickly and send a correct final state to the RM when it is killed.

- MAPREDUCE-3009 | Major | RM UI -> Applications -> Application(Job History) -> Map Tasks -> Task ID -> Node link is not working

Fixed node link on JobHistory webapp.

- MAPREDUCE-3954 | Blocker | Clean up passing HEAPSIZE to yarn and mapred commands.

Added new envs to separate heap size for different daemons started via bin scripts.

- MAPREDUCE-3975 | Blocker | Default value not set for Configuration parameter mapreduce.job.local.dir

Exporting mapreduce.job.local.dir for mapreduce tasks to use as job-level shared scratch space.

- MAPREDUCE-3982 | Critical | TestEmptyJob fails with FileNotFound

Fixed FileOutputCommitter to not err out for an ‘empty-job’ whose tasks don’t write any outputs.

- HADOOP-8164 | Major | Handle paths using back slash as path separator for windows only

This jira only allows providing paths using back slash as separator on Windows. The back slash on *nix system will be used as escape character. The support for paths using back slash as path separator will be removed in HADOOP-8139 in release 23.3.

- HADOOP-8175 | Major | Add mkdir -p flag

FsShell mkdir now accepts a -p flag. Like unix, mkdir -p will not fail if the directory already exists. Unlike unix, intermediate directories are always created, regardless of the flag, to avoid incompatibilities at this time.