Apache Hadoop 3.5.0 – Apache Hadoop Changelog

##### General

- Overview
- Single Node Setup
- Cluster Setup
- Commands Reference
- FileSystem Shell
- Compatibility Specification
- Downstream Developer's Guide
- Admin Compatibility Guide
- Interface Classification
- FileSystem Specification

##### Common

- CLI Mini Cluster
- Fair Call Queue
- Native Libraries
- Proxy User
- Rack Awareness
- Secure Mode
- Service Level Authorization
- HTTP Authentication
- Credential Provider API
- Hadoop KMS
- Tracing
- Unix Shell Guide
- Registry
- Async Profiler

##### HDFS

- Architecture
- User Guide
- Commands Reference
- NameNode HA With QJM
- NameNode HA With NFS
- Observer NameNode
- Federation
- ViewFs
- ViewFsOverloadScheme
- Snapshots
- Edits Viewer
- Image Viewer
- Permissions and HDFS
- Quotas and HDFS
- libhdfs (C API)
- WebHDFS (REST API)
- HttpFS
- Short Circuit Local Reads
- Centralized Cache Management
- NFS Gateway
- Rolling Upgrade
- Extended Attributes
- Transparent Encryption
- Multihoming
- Storage Policies
- Memory Storage Support
- Synthetic Load Generator
- Erasure Coding
- Disk Balancer
- Upgrade Domain
- DataNode Admin
- Router Federation
- Provided Storage

##### MapReduce

- Tutorial
- Commands Reference
- Encrypted Shuffle
- Pluggable Shuffle/Sort
- Distributed Cache Deploy
- Support for YARN Shared Cache

##### MapReduce REST APIs

- MR Application Master
- MR History Server

##### YARN

- Architecture
- Commands Reference
- Capacity Scheduler
- Fair Scheduler
- ResourceManager Restart
- ResourceManager HA
- Resource Model
- Node Labels
- Node Attributes
- Web Application Proxy
- Timeline Server
- Timeline Service V.2
- Writing YARN Applications
- YARN Application Security
- NodeManager
- Running Applications in Docker Containers
- Running Applications in runC Containers
- Using CGroups
- Secure Containers
- Reservation System
- Graceful Decommission
- Opportunistic Containers
- YARN Federation
- Shared Cache
- Using GPU
- Using FPGA
- Placement Constraints
- YARN UI2

##### YARN REST APIs

- Introduction
- Resource Manager
- Node Manager
- Timeline Server
- Timeline Service V.2

##### YARN Service

- Overview
- QuickStart
- Concepts
- Yarn Service API
- Service Discovery
- System Services

##### Hadoop Compatible File Systems

- Aliyun OSS
- Amazon S3
- Azure Blob Storage
- Azure Data Lake Storage
- Tencent COS
- Huaweicloud OBS
- VolcanoEngine TOS

##### Auth

- Overview
- Examples
- Configuration
- Building

##### Tools

- Hadoop Streaming
- Hadoop Archives
- Hadoop Archive Logs
- DistCp
- HDFS Federation Balance
- GridMix
- Rumen
- Resource Estimator Service
- Scheduler Load Simulator
- Hadoop Benchmarking
- Dynamometer

##### Reference

- Changelog and Release Notes
- Java API docs
- Unix Shell API
- Metrics

##### Configuration

- core-default.xml
- hdfs-default.xml
- hdfs-rbf-default.xml
- mapred-default.xml
- yarn-default.xml
- kms-default.xml
- httpfs-default.xml
- Deprecated Properties

# Apache Hadoop Changelog

## Release 2.8.3 - 2017-12-12

### INCOMPATIBLE CHANGES:

JIRA | Summary | Priority | Component | Reporter | Contributor
--- | --- | --- | --- | --- | ---
HADOOP-13588 | ConfServlet should respect Accept request header | Major | conf | Weiwei Yang | Weiwei Yang
HADOOP-14260 | Configuration.dumpConfiguration should redact sensitive information | Major | conf, security | Vihang Karajgaonkar | John Zhuge

### NEW FEATURES:

JIRA | Summary | Priority | Component | Reporter | Contributor
--- | --- | --- | --- | --- | ---
HADOOP-13933 | Add haadmin -getAllServiceState option to get the HA state of all the NameNodes/ResourceManagers | Major | tools | Surendra Singh Lilhore | Surendra Singh Lilhore
HDFS-10480 | Add an admin command to list currently open files | Major | . | Kihwal Lee | Manoj Govindassamy

### IMPROVEMENTS:

JIRA | Summary | Priority | Component | Reporter | Contributor
--- | --- | --- | --- | --- | ---
HADOOP-13628 | Support to retrieve specific property from configuration via REST API | Major | conf | Weiwei Yang | Weiwei Yang
HDFS-12143 | Improve performance of getting and removing inode features | Major | namenode | Daryn Sharp | Daryn Sharp
HDFS-12171 | Reduce IIP object allocations for inode lookup | Major | namenode | Daryn Sharp | Daryn Sharp
HADOOP-14455 | ViewFileSystem#rename should support be supported within same nameservice with different mountpoints | Major | viewfs | Brahma Reddy Battula | Brahma Reddy Battula
HADOOP-14627 | Support MSI and DeviceCode token provider in ADLS | Major | fs/adl | Atul Sikaria | Atul Sikaria
YARN-7037 | Optimize data transfer with zero-copy approach for containerlogs REST API in NMWebServices | Major | nodemanager | Tao Yang | Tao Yang
HADOOP-14827 | Allow StopWatch to accept a Timer parameter for tests | Minor | common, test | Erik Krogen | Erik Krogen
HDFS-12131 | Add some of the FSNamesystem JMX values as metrics | Minor | hdfs, namenode | Erik Krogen | Erik Krogen
HADOOP-14844 | Remove requirement to specify TenantGuid for MSI Token Provider | Major | fs/adl | Atul Sikaria | Atul Sikaria
HADOOP-14864 | FSDataInputStream#unbuffer UOE should include stream class name | Minor | fs | John Zhuge | Bharat Viswanadham
HDFS-12441 | Suppress UnresolvedPathException in namenode log | Minor | . | Kihwal Lee | Kihwal Lee
MAPREDUCE-6958 | Shuffle audit logger should log size of shuffle transfer | Minor | . | Jason Lowe | Jason Lowe
HDFS-12420 | Add an option to disallow ‘namenode format -force’ | Major | . | Ajay Kumar | Ajay Kumar
HADOOP-14521 | KMS client needs retry logic | Major | . | Rushabh S Shah | Rushabh S Shah
HDFS-12603 | Enable async edit logging by default | Major | namenode | Andrew Wang | Andrew Wang
HDFS-12642 | Log block and datanode details in BlockRecoveryWorker | Major | datanode | Xiao Chen | Xiao Chen
HADOOP-14880 | [KMS] Document&test missing KMS client side configs | Minor | . | Wei-Chiu Chuang | Gabor Bota
HDFS-12619 | Do not catch and throw unchecked exceptions if IBRs fail to process | Minor | namenode | Wei-Chiu Chuang | Wei-Chiu Chuang
YARN-4163 | Audit getQueueInfo and getApplications calls | Major | . | Chang Li | Chang Li
MAPREDUCE-6975 | Logging task counters | Major | task | Prabhu Joseph | Prabhu Joseph

### BUG FIXES:

JIRA | Summary | Priority | Component | Reporter | Contributor
--- | --- | --- | --- | --- | ---
YARN-5444 | Fix failing unit tests in TestLinuxContainerExecutorWithMocks | Major | nodemanager | Yufei Gu | Yufei Gu
MAPREDUCE-6808 | Log map attempts as part of shuffle handler audit log | Major | . | Jonathan Eagles | Gergő Pásztor
HADOOP-14578 | Bind IPC connections to kerberos UPN host for proxy users | Major | ipc | Daryn Sharp | Daryn Sharp
HADOOP-14677 | mvn clean compile fails | Major | build | Andras Bokor | Andras Bokor
HADOOP-14702 | Fix formatting issue and regression caused by conversion from APT to Markdown | Minor | documentation | Doris Gu | Doris Gu
YARN-6965 | Duplicate instantiation in FairSchedulerQueueInfo | Minor | fairscheduler | Masahiro Tanaka | Masahiro Tanaka
HDFS-11738 | Hedged pread takes more time when block moved from initial locations | Major | hdfs-client | Vinayakumar B | Vinayakumar B
HDFS-12318 | Fix IOException condition for openInfo in DFSInputStream | Major | . | legend | legend
HDFS-12336 | Listing encryption zones still fails when deleted EZ is not a direct child of snapshottable directory | Minor | encryption, hdfs | Wellington Chevreuil | Wellington Chevreuil
MAPREDUCE-6931 | Remove TestDFSIO “Total Throughput” calculation | Critical | benchmarks, test | Dennis Huo | Dennis Huo
YARN-7116 | CapacityScheduler Web UI: Queue’s AM usage is always show on per-user’s AM usage. | Major | capacity scheduler, webapp | Wangda Tan | Wangda Tan
HADOOP-14824 | Update ADLS SDK to 2.2.2 for MSI fix | Major | fs/adl | Atul Sikaria | Atul Sikaria
YARN-7120 | CapacitySchedulerPage NPE in “Aggregate scheduler counts” section | Minor | . | Eric Payne | Eric Payne
YARN-7164 | TestAMRMClientOnRMRestart fails sporadically with bind address in use | Major | test | Jason Lowe | Jason Lowe
HDFS-12369 | Edit log corruption due to hard lease recovery of not-closed file which has snapshots | Major | namenode | Xiao Chen | Xiao Chen
HADOOP-14867 | Update HDFS Federation setup document, for incorrect property name for secondary name node http address | Major | . | Bharat Viswanadham | Bharat Viswanadham
YARN-4727 | Unable to override the $HADOOP_CONF_DIR env variable for container | Major | nodemanager | Terence Yim | Jason Lowe
MAPREDUCE-6957 | shuffle hangs after a node manager connection timeout | Major | mrv2 | Jooseong Kim | Jooseong Kim
HDFS-12424 | Datatable sorting on the Datanode Information page in the Namenode UI is broken | Major | . | Shawna Martell | Shawna Martell
HDFS-12323 | NameNode terminates after full GC thinking QJM unresponsive if full GC is much longer than timeout | Major | namenode, qjm | Erik Krogen | Erik Krogen
MAPREDUCE-6960 | Shuffle Handler prints disk error stack traces for every read failure. | Major | . | Kuhu Shukla | Kuhu Shukla
HDFS-11799 | Introduce a config to allow setting up write pipeline with fewer nodes than replication factor | Major | . | Yongjun Zhang | Brahma Reddy Battula
YARN-6771 | Use classloader inside configuration class to make new classes | Major | . | Jongyoul Lee | Jongyoul Lee
HDFS-12526 | FSDirectory should use Time.monotonicNow for durations | Minor | . | Chetna Chaudhari | Bharat Viswanadham
HDFS-12371 | “BlockVerificationFailures” and “BlocksVerified” show up as 0 in Datanode JMX | Major | metrics | Sai Nukavarapu | Hanisha Koneru
MAPREDUCE-6966 | DistSum should use Time.monotonicNow for measuring durations | Minor | . | Chetna Chaudhari | Chetna Chaudhari
HDFS-12531 | Fix conflict in the javadoc of UnderReplicatedBlocks.java in branch-2 | Minor | documentation | Akira Ajisaka | Bharat Viswanadham
HDFS-12495 | TestPendingInvalidateBlock#testPendingDeleteUnknownBlocks fails intermittently | Major | . | Eric Badger | Eric Badger
HADOOP-14891 | Remove references to Guava Objects.toStringHelper | Major | . | Jonathan Eagles | Jonathan Eagles
HADOOP-14902 | LoadGenerator#genFile write close timing is incorrectly calculated | Major | fs | Jason Lowe | Hanisha Koneru
YARN-7084 | TestSchedulingMonitor#testRMStarts fails sporadically | Major | . | Jason Lowe | Jason Lowe
YARN-7226 | Whitelisted variables do not support delayed variable expansion | Major | nodemanager | Jason Lowe | Jason Lowe
YARN-7285 | ContainerExecutor always launches with priorities due to yarn-default property | Minor | nodemanager | Jason Lowe | Jason Lowe
YARN-7245 | Max AM Resource column in Active Users Info section of Capacity Scheduler UI page should be updated per-user | Major | capacity scheduler, yarn | Eric Payne | Eric Payne
HADOOP-14912 | FairCallQueue may defer servicing calls | Major | ipc | Daryn Sharp | Daryn Sharp
HDFS-12659 | Update TestDeadDatanode#testNonDFSUsedONDeadNodeReReg to increase heartbeat recheck interval | Minor | . | Ajay Kumar | Ajay Kumar
HDFS-12485 | expunge may fail to remove trash from encryption zone | Major | . | Wei-Chiu Chuang | Wei-Chiu Chuang
YARN-7333 | container-executor fails to remove entries from a directory that is not writable or executable | Critical | . | Jason Lowe | Jason Lowe
HADOOP-14966 | Handle JDK-8071638 for hadoop-common | Blocker | . | Bibin A Chundatt | Bibin A Chundatt
HDFS-9914 | Fix configurable WebhDFS connect/read timeout | Blocker | hdfs-client, webhdfs | Xiaoyu Yao | Xiaoyu Yao
YARN-7244 | ShuffleHandler is not aware of disks that are added | Major | . | Kuhu Shukla | Kuhu Shukla
HADOOP-14990 | Clean up jdiff xml files added for 2.8.2 release | Blocker | . | Subru Krishnan | Junping Du
HADOOP-14919 | BZip2 drops records when reading data in splits | Critical | . | Aki Tanaka | Jason Lowe
YARN-7370 | Preemption properties should be refreshable | Major | capacity scheduler, scheduler preemption | Eric Payne | Gergely Novák
YARN-7361 | Improve the docker container runtime documentation | Major | . | Shane Kumpf | Shane Kumpf
YARN-7469 | Capacity Scheduler Intra-queue preemption: User can starve if newest app is exactly at user limit | Major | capacity scheduler, yarn | Eric Payne | Eric Payne
HADOOP-15047 | Python is required for -Preleasedoc but not documented in branch-2.8 | Major | build, documentation | Akira Ajisaka | Bharat Viswanadham
YARN-7496 | CS Intra-queue preemption user-limit calculations are not in line with LeafQueue user-limit calculations | Major | . | Eric Payne | Eric Payne
HDFS-12832 | INode.getFullPathName may throw ArrayIndexOutOfBoundsException lead to NameNode exit | Critical | namenode | DENG FEI | Konstantin Shvachko
HDFS-12638 | Delete copy-on-truncate block along with the original block, when deleting a file being truncated | Blocker | hdfs | Jiandan Yang | Konstantin Shvachko

### SUB-TASKS:

JIRA | Summary | Priority | Component | Reporter | Contributor
--- | --- | --- | --- | --- | ---
HADOOP-14678 | AdlFilesystem#initialize swallows exception when getting user name | Minor | fs/adl | John Zhuge | John Zhuge
HADOOP-14892 | MetricsSystemImpl should use Time.monotonicNow for measuring durations | Minor | . | Chetna Chaudhari | Chetna Chaudhari
HADOOP-14881 | LoadGenerator should use Time.monotonicNow() to measure durations | Major | . | Chetna Chaudhari | Bharat Viswanadham
HADOOP-14893 | WritableRpcEngine should use Time.monotonicNow | Minor | . | Chetna Chaudhari | Chetna Chaudhari
HDFS-12386 | Add fsserver defaults call to WebhdfsFileSystem. | Minor | webhdfs | Rushabh S Shah | Rushabh S Shah