Apache Hadoop 3.5.0 – Apache Hadoop Changelog

##### General

- Overview
- Single Node Setup
- Cluster Setup
- Commands Reference
- FileSystem Shell
- Compatibility Specification
- Downstream Developer's Guide
- Admin Compatibility Guide
- Interface Classification
- FileSystem Specification

##### Common

- CLI Mini Cluster
- Fair Call Queue
- Native Libraries
- Proxy User
- Rack Awareness
- Secure Mode
- Service Level Authorization
- HTTP Authentication
- Credential Provider API
- Hadoop KMS
- Tracing
- Unix Shell Guide
- Registry
- Async Profiler

##### HDFS

- Architecture
- User Guide
- Commands Reference
- NameNode HA With QJM
- NameNode HA With NFS
- Observer NameNode
- Federation
- ViewFs
- ViewFsOverloadScheme
- Snapshots
- Edits Viewer
- Image Viewer
- Permissions and HDFS
- Quotas and HDFS
- libhdfs (C API)
- WebHDFS (REST API)
- HttpFS
- Short Circuit Local Reads
- Centralized Cache Management
- NFS Gateway
- Rolling Upgrade
- Extended Attributes
- Transparent Encryption
- Multihoming
- Storage Policies
- Memory Storage Support
- Synthetic Load Generator
- Erasure Coding
- Disk Balancer
- Upgrade Domain
- DataNode Admin
- Router Federation
- Provided Storage

##### MapReduce

- Tutorial
- Commands Reference
- Encrypted Shuffle
- Pluggable Shuffle/Sort
- Distributed Cache Deploy
- Support for YARN Shared Cache

##### MapReduce REST APIs

- MR Application Master
- MR History Server

##### YARN

- Architecture
- Commands Reference
- Capacity Scheduler
- Fair Scheduler
- ResourceManager Restart
- ResourceManager HA
- Resource Model
- Node Labels
- Node Attributes
- Web Application Proxy
- Timeline Server
- Timeline Service V.2
- Writing YARN Applications
- YARN Application Security
- NodeManager
- Running Applications in Docker Containers
- Running Applications in runC Containers
- Using CGroups
- Secure Containers
- Reservation System
- Graceful Decommission
- Opportunistic Containers
- YARN Federation
- Shared Cache
- Using GPU
- Using FPGA
- Placement Constraints
- YARN UI2

##### YARN REST APIs

- Introduction
- Resource Manager
- Node Manager
- Timeline Server
- Timeline Service V.2

##### YARN Service

- Overview
- QuickStart
- Concepts
- Yarn Service API
- Service Discovery
- System Services

##### Hadoop Compatible File Systems

- Aliyun OSS
- Amazon S3
- Azure Blob Storage
- Azure Data Lake Storage
- Tencent COS
- Huaweicloud OBS
- VolcanoEngine TOS

##### Auth

- Overview
- Examples
- Configuration
- Building

##### Tools

- Hadoop Streaming
- Hadoop Archives
- Hadoop Archive Logs
- DistCp
- HDFS Federation Balance
- GridMix
- Rumen
- Resource Estimator Service
- Scheduler Load Simulator
- Hadoop Benchmarking
- Dynamometer

##### Reference

- Changelog and Release Notes
- Java API docs
- Unix Shell API
- Metrics

##### Configuration

- core-default.xml
- hdfs-default.xml
- hdfs-rbf-default.xml
- mapred-default.xml
- yarn-default.xml
- kms-default.xml
- httpfs-default.xml
- Deprecated Properties

# Apache Hadoop Changelog

## Release 2.7.5 - 2017-12-14

### IMPORTANT ISSUES:

JIRA | Summary | Priority | Component | Reporter | Contributor
--- | --- | --- | --- | --- | ---
YARN-6959 | RM may allocate wrong AM Container for new attempt | Major | capacity scheduler, fairscheduler, scheduler | Yuqi Wang | Yuqi Wang

### IMPROVEMENTS:

JIRA | Summary | Priority | Component | Reporter | Contributor
--- | --- | --- | --- | --- | ---
HDFS-8829 | Make SO_RCVBUF and SO_SNDBUF size configurable for DataTransferProtocol sockets and allow configuring auto-tuning | Major | datanode | He Tianyi | He Tianyi
HADOOP-13442 | Optimize UGI group lookups | Major | . | Daryn Sharp | Daryn Sharp
MAPREDUCE-6937 | Backport MAPREDUCE-6870 to branch-2 while preserving compatibility | Major | . | Zhe Zhang | Peter Bacsko
HADOOP-14827 | Allow StopWatch to accept a Timer parameter for tests | Minor | common, test | Erik Krogen | Erik Krogen
HDFS-12131 | Add some of the FSNamesystem JMX values as metrics | Minor | hdfs, namenode | Erik Krogen | Erik Krogen
HDFS-8865 | Improve quota initialization performance | Major | . | Kihwal Lee | Kihwal Lee
HDFS-12420 | Add an option to disallow ‘namenode format -force’ | Major | . | Ajay Kumar | Ajay Kumar
MAPREDUCE-6975 | Logging task counters | Major | task | Prabhu Joseph | Prabhu Joseph
HDFS-12823 | Backport HDFS-9259 “Make SO_SNDBUF size configurable at DFSClient” to branch-2.7 | Major | hdfs, hdfs-client | Erik Krogen | Erik Krogen

### BUG FIXES:

JIRA | Summary | Priority | Component | Reporter | Contributor
--- | --- | --- | --- | --- | ---
MAPREDUCE-6165 | [JDK8] TestCombineFileInputFormat failed on JDK8 | Minor | . | Wei Yan | Akira Ajisaka
HDFS-8797 | WebHdfsFileSystem creates too many connections for pread | Major | webhdfs | Jing Zhao | Jing Zhao
HDFS-9003 | ForkJoin thread pool leaks | Major | . | Kihwal Lee | Kihwal Lee
HDFS-9107 | Prevent NN’s unrecoverable death spiral after full GC | Critical | namenode | Daryn Sharp | Daryn Sharp
MAPREDUCE-6750 | TestHSAdminServer.testRefreshSuperUserGroups is failing | Minor | test | Kihwal Lee | Kihwal Lee
HDFS-10738 | Fix TestRefreshUserMappings.testRefreshSuperUserGroupsConfiguration test failure | Major | test | Rakesh R | Rakesh R
HADOOP-14702 | Fix formatting issue and regression caused by conversion from APT to Markdown | Minor | documentation | Doris Gu | Doris Gu
HDFS-12157 | Do fsyncDirectory(..) outside of FSDataset lock | Critical | datanode | Vinayakumar B | Vinayakumar B
MAPREDUCE-6931 | Remove TestDFSIO “Total Throughput” calculation | Critical | benchmarks, test | Dennis Huo | Dennis Huo
HADOOP-14867 | Update HDFS Federation setup document, for incorrect property name for secondary name node http address | Major | . | Bharat Viswanadham | Bharat Viswanadham
MAPREDUCE-6957 | shuffle hangs after a node manager connection timeout | Major | mrv2 | Jooseong Kim | Jooseong Kim
HDFS-12323 | NameNode terminates after full GC thinking QJM unresponsive if full GC is much longer than timeout | Major | namenode, qjm | Erik Krogen | Erik Krogen
YARN-5195 | RM intermittently crashed with NPE while handling APP_ATTEMPT_REMOVED event when async-scheduling enabled in CapacityScheduler | Major | resourcemanager | Karam Singh | sandflee
HADOOP-14902 | LoadGenerator#genFile write close timing is incorrectly calculated | Major | fs | Jason Lowe | Hanisha Koneru
YARN-7084 | TestSchedulingMonitor#testRMStarts fails sporadically | Major | . | Jason Lowe | Jason Lowe
HDFS-12578 | TestDeadDatanode#testNonDFSUsedONDeadNodeReReg failing in branch-2.7 | Blocker | test | Xiao Chen | Ajay Kumar
HADOOP-14919 | BZip2 drops records when reading data in splits | Critical | . | Aki Tanaka | Jason Lowe
HDFS-12832 | INode.getFullPathName may throw ArrayIndexOutOfBoundsException lead to NameNode exit | Critical | namenode | DENG FEI | Konstantin Shvachko
HDFS-12638 | Delete copy-on-truncate block along with the original block, when deleting a file being truncated | Blocker | hdfs | Jiandan Yang | Konstantin Shvachko

### TESTS:

JIRA | Summary | Priority | Component | Reporter | Contributor
--- | --- | --- | --- | --- | ---
HDFS-9153 | Pretty-format the output for DFSIO | Major | . | Kai Zheng | Kai Zheng
HDFS-12596 | Add TestFsck#testFsckCorruptWhenOneReplicaIsCorrupt back to branch-2.7 | Major | test | Xiao Chen | Xiao Chen

### SUB-TASKS:

JIRA | Summary | Priority | Component | Reporter | Contributor
--- | --- | --- | --- | --- | ---
HDFS-10711 | Optimize FSPermissionChecker group membership check | Major | hdfs | Daryn Sharp | Daryn Sharp
HADOOP-14881 | LoadGenerator should use Time.monotonicNow() to measure durations | Major | . | Chetna Chaudhari | Bharat Viswanadham
YARN-5402 | Fix NoSuchMethodError in ClusterMetricsInfo | Major | webapp | Weiwei Yang | Weiwei Yang

### OTHER:

JIRA | Summary | Priority | Component | Reporter | Contributor
--- | --- | --- | --- | --- | ---
HDFS-10984 | Expose nntop output as metrics | Major | namenode | Siddharth Wagle | Siddharth Wagle