Apache Hadoop 3.5.0 – Apache Hadoop Changelog

##### General

- Overview
- Single Node Setup
- Cluster Setup
- Commands Reference
- FileSystem Shell
- Compatibility Specification
- Downstream Developer's Guide
- Admin Compatibility Guide
- Interface Classification
- FileSystem Specification

##### Common

- CLI Mini Cluster
- Fair Call Queue
- Native Libraries
- Proxy User
- Rack Awareness
- Secure Mode
- Service Level Authorization
- HTTP Authentication
- Credential Provider API
- Hadoop KMS
- Tracing
- Unix Shell Guide
- Registry
- Async Profiler

##### HDFS

- Architecture
- User Guide
- Commands Reference
- NameNode HA With QJM
- NameNode HA With NFS
- Observer NameNode
- Federation
- ViewFs
- ViewFsOverloadScheme
- Snapshots
- Edits Viewer
- Image Viewer
- Permissions and HDFS
- Quotas and HDFS
- libhdfs (C API)
- WebHDFS (REST API)
- HttpFS
- Short Circuit Local Reads
- Centralized Cache Management
- NFS Gateway
- Rolling Upgrade
- Extended Attributes
- Transparent Encryption
- Multihoming
- Storage Policies
- Memory Storage Support
- Synthetic Load Generator
- Erasure Coding
- Disk Balancer
- Upgrade Domain
- DataNode Admin
- Router Federation
- Provided Storage

##### MapReduce

- Tutorial
- Commands Reference
- Encrypted Shuffle
- Pluggable Shuffle/Sort
- Distributed Cache Deploy
- Support for YARN Shared Cache

##### MapReduce REST APIs

- MR Application Master
- MR History Server

##### YARN

- Architecture
- Commands Reference
- Capacity Scheduler
- Fair Scheduler
- ResourceManager Restart
- ResourceManager HA
- Resource Model
- Node Labels
- Node Attributes
- Web Application Proxy
- Timeline Server
- Timeline Service V.2
- Writing YARN Applications
- YARN Application Security
- NodeManager
- Running Applications in Docker Containers
- Running Applications in runC Containers
- Using CGroups
- Secure Containers
- Reservation System
- Graceful Decommission
- Opportunistic Containers
- YARN Federation
- Shared Cache
- Using GPU
- Using FPGA
- Placement Constraints
- YARN UI2

##### YARN REST APIs

- Introduction
- Resource Manager
- Node Manager
- Timeline Server
- Timeline Service V.2

##### YARN Service

- Overview
- QuickStart
- Concepts
- Yarn Service API
- Service Discovery
- System Services

##### Hadoop Compatible File Systems

- Aliyun OSS
- Amazon S3
- Azure Blob Storage
- Azure Data Lake Storage
- Tencent COS
- Huaweicloud OBS
- VolcanoEngine TOS

##### Auth

- Overview
- Examples
- Configuration
- Building

##### Tools

- Hadoop Streaming
- Hadoop Archives
- Hadoop Archive Logs
- DistCp
- HDFS Federation Balance
- GridMix
- Rumen
- Resource Estimator Service
- Scheduler Load Simulator
- Hadoop Benchmarking
- Dynamometer

##### Reference

- Changelog and Release Notes
- Java API docs
- Unix Shell API
- Metrics

##### Configuration

- core-default.xml
- hdfs-default.xml
- hdfs-rbf-default.xml
- mapred-default.xml
- yarn-default.xml
- kms-default.xml
- httpfs-default.xml
- Deprecated Properties

# Apache Hadoop Changelog

## Release 3.3.6 - 2023-06-19

### NEW FEATURES:

JIRA | Summary | Priority | Component | Reporter | Contributor
--- | --- | --- | --- | --- | ---
HADOOP-18671 | Add recoverLease(), setSafeMode(), isFileClosed() APIs to FileSystem | Major | fs | Wei-Chiu Chuang | Tak-Lon (Stephen) Wu

### IMPROVEMENTS:

JIRA | Summary | Priority | Component | Reporter | Contributor
--- | --- | --- | --- | --- | ---
HDFS-15368 | TestBalancerWithHANameNodes#testBalancerWithObserver failed occasionally | Major | . | Xiaoqiao He | Xiaoqiao He
HDFS-15383 | RBF: Disable watch in ZKDelegationSecretManager for performance | Major | . | Fengnan Li | Fengnan Li
HDFS-15803 | EC: Remove unnecessary method (getWeight) in StripedReconstructionInfo | Trivial | . | Haiyang Hu | Haiyang Hu
HDFS-16024 | RBF: Rename data to the Trash should be based on src locations | Major | rbf | Xiangyi Zhu | Xiangyi Zhu
HDFS-16016 | BPServiceActor add a new thread to handle IBR | Minor | . | JiangHua Zhu | Viraj Jasani
HADOOP-17749 | Remove lock contention in SelectorPool of SocketIOWithTimeout | Major | common | Xuesen Liang | Xuesen Liang
HDFS-16480 | Fix typo: indicies -> indices | Minor | . | Jiale Qi | Jiale Qi
HADOOP-18466 | Limit the findbugs suppression IS2_INCONSISTENT_SYNC to S3AFileSystem field | Minor | fs/s3 | Viraj Jasani | Viraj Jasani
MAPREDUCE-7370 | Parallelize MultipleOutputs#close call | Major | . | Prabhu Joseph | Ashutosh Gupta
YARN-11360 | Add number of decommissioning/shutdown nodes to YARN cluster metrics. | Major | client, resourcemanager | Chris Nauroth | Chris Nauroth
HADOOP-18472 | Upgrade to snakeyaml 1.33 | Major | . | PJ Fanning | PJ Fanning
HDFS-16811 | Support DecommissionBackoffMonitor parameters reconfigurable | Major | . | Haiyang Hu | Haiyang Hu
HADOOP-18433 | Fix main thread name. | Major | common, ipc | zhengchenyu | zhengchenyu
HDFS-16851 | RBF: Add a utility to dump the StateStore | Major | rbf | Owen O’Malley | Owen O’Malley
HDFS-16839 | It should consider EC reconstruction work when we determine if a node is busy | Major | ec, erasure-coding | Kidd5368 | Kidd5368
HDFS-16887 | Log start and end of phase/step in startup progress | Minor | namenode | Viraj Jasani | Viraj Jasani
HDFS-16891 | Avoid the overhead of copy-on-write exception list while loading inodes sub sections in parallel | Major | namenode | Viraj Jasani | Viraj Jasani
HADOOP-18604 | Add compile platform in the hadoop version output | Major | . | Ayush Saxena | Ayush Saxena
HDFS-16888 | BlockManager#maxReplicationStreams, replicationStreamsHardLimit, blocksReplWorkMultiplier and PendingReconstructionBlocks#timeout should be volatile | Major | . | Haiyang Hu | Haiyang Hu
HADOOP-18592 | Sasl connection failure should log remote address | Major | . | Viraj Jasani | Viraj Jasani
HADOOP-18625 | Fix method name of RPC.Builder#setnumReaders | Minor | ipc | Haiyang Hu | Haiyang Hu
HDFS-16882 | RBF: Add cache hit rate metric in MountTableResolver#getDestinationForPath | Minor | rbf | farmmamba | farmmamba
HDFS-16907 | Add LastHeartbeatResponseTime for BP service actor | Major | . | Viraj Jasani | Viraj Jasani
HADOOP-18628 | Server connection should log host name before returning VersionMismatch error | Minor | ipc | Viraj Jasani | Viraj Jasani
HADOOP-18596 | Distcp -update between different cloud stores to use modification time while checking for file skip. | Major | tools/distcp | Mehakmeet Singh | Mehakmeet Singh
HADOOP-18215 | Enhance WritableName to be able to return aliases for classes that use serializers | Minor | . | Bryan Beaudreault | Bryan Beaudreault
HADOOP-18622 | Upgrade ant to 1.10.13 | Major | . | Aleksandr Nikolaev | Aleksandr Nikolaev
YARN-11394 | Fix hadoop-yarn-server-resourcemanager module Java Doc Errors. | Major | resourcemanager | Shilun Fan | Shilun Fan
HADOOP-18535 | Implement token storage solution based on MySQL | Major | . | Hector Sandoval Chaverri | Hector Sandoval Chaverri
HADOOP-18646 | Upgrade Netty to 4.1.89.Final | Major | build | Aleksandr Nikolaev | Aleksandr Nikolaev
HADOOP-18684 | S3A filesystem to support binding to other URI schemes | Major | . | Harshit Gupta | Harshit Gupta
HADOOP-18590 | Publish SBOM artifacts | Major | build | Dongjoon Hyun | Dongjoon Hyun
HADOOP-18597 | Simplify single node instructions for creating directories for Map Reduce | Trivial | documentation | Nikita Eshkeev | Nikita Eshkeev
HADOOP-18691 | Add a CallerContext getter on the Schedulable interface | Major | . | Christos Bisias | Christos Bisias
HADOOP-18689 | Bump jettison from 1.5.3 to 1.5.4 in /hadoop-project | Major | common | Ayush Saxena |
HDFS-16988 | Improve NameServices info at JournalNode web UI | Minor | . | Zhaohui Wang | Zhaohui Wang
HADOOP-18637 | S3A to support upload of files greater than 2 GB using DiskBlocks | Major | fs/s3 | Harshit Gupta | Harshit Gupta
HADOOP-18695 | S3A: reject multipart copy requests when disabled | Minor | fs/s3 | Steve Loughran | Steve Loughran

### BUG FIXES:

JIRA | Summary | Priority | Component | Reporter | Contributor
--- | --- | --- | --- | --- | ---
HDFS-16628 | RBF: Correct target directory when move to trash for kerberos login user. | Major | rbf | Xiping Zhang | Xiping Zhang
HDFS-16633 | Reserved Space For Replicas is not released on some cases | Major | hdfs | Prabhu Joseph | Ashutosh Gupta
HDFS-16566 | Erasure Coding: Recovery may cause excess replicas when busy DN exsits | Major | ec, erasure-coding | Kidd5368 | Kidd5368
HADOOP-18471 | An unhandled ArrayIndexOutOfBoundsException in DefaultStringifier.storeArray() if provided with an empty input | Minor | common, io | FuzzingTeam | FuzzingTeam
HDFS-16809 | EC striped block is not sufficient when doing in maintenance | Major | ec, erasure-coding | dingshun | dingshun
YARN-11390 | TestResourceTrackerService.testNodeRemovalNormally: Shutdown nodes should be 0 now expected: <1> but was: <0> | Major | yarn | Bence Kosztolnik | Bence Kosztolnik
HDFS-16852 | Register the shutdown hook only when not in shutdown for KeyProviderCache constructor | Minor | hdfs | Xing Lin | Xing Lin
HADOOP-18567 | LogThrottlingHelper: the dependent recorder is not triggered correctly | Major | . | Chengbing Liu |
YARN-11395 | Resource Manager UI, cluster/appattempt/*, can not present FINAL_SAVING state | Critical | yarn | Bence Kosztolnik | Bence Kosztolnik
YARN-11392 | ClientRMService implemented getCallerUgi and verifyUserAccessForRMApp methods but forget to use sometimes, caused audit log missing. | Major | yarn | Beibei Zhao | Beibei Zhao
HDFS-16872 | Fix log throttling by declaring LogThrottlingHelper as static members | Major | . | Chengbing Liu |
HADOOP-18591 | Fix a typo in Trash | Minor | documentation | xiaoping.huang | xiaoping.huang
HDFS-16764 | ObserverNamenode handles addBlock rpc and throws a FileNotFoundException | Critical | . | ZanderXu | ZanderXu
HADOOP-18584 | [NFS GW] Fix regression after netty4 migration | Major | . | Wei-Chiu Chuang | Wei-Chiu Chuang
HADOOP-18279 | Cancel fileMonitoringTimer even if trustManager isn’t defined | Major | common, test | Steve Vaughan | Steve Vaughan
HADOOP-18576 | Java 11 JavaDoc fails due to missing package comments | Major | build, common | Steve Loughran | Steve Vaughan
HADOOP-18612 | Avoid mixing canonical and non-canonical when performing comparisons | Minor | common, test | Steve Vaughan | Steve Vaughan
HDFS-16925 | Namenode audit log to only include IP address of client | Major | . | Viraj Jasani | Viraj Jasani
HADOOP-18633 | fix test AbstractContractDistCpTest#testDistCpUpdateCheckFileSkip | Major | tools/distcp | Mehakmeet Singh | Mehakmeet Singh
HADOOP-18582 | No need to clean tmp files in distcp direct mode | Major | tools/distcp | 10000kang | 10000kang
HADOOP-18636 | LocalDirAllocator cannot recover from directory tree deletion during the life of a filesystem client | Minor | fs, fs/azure, fs/s3 | Steve Loughran | Steve Loughran
HDFS-16935 | TestFsDatasetImpl.testReportBadBlocks brittle | Minor | test | Steve Loughran | Viraj Jasani
HDFS-16942 | Send error to datanode if FBR is rejected due to bad lease | Major | datanode, namenode | Stephen O’Donnell | Stephen O’Donnell
HADOOP-18662 | ListFiles with recursive fails with FNF | Major | . | Ayush Saxena | Ayush Saxena
HADOOP-18680 | Insufficient heap during full test runs in Docker container. | Minor | build | Chris Nauroth | Chris Nauroth
HADOOP-18714 | Wrong StringUtils.join() called in AbstractContractRootDirectoryTest | Trivial | test | Attila Doroszlai | Attila Doroszlai
HADOOP-18705 | ABFS should exclude incompatible credential providers | Major | fs/azure | Tamas Domok | Tamas Domok
HADOOP-18660 | Filesystem Spelling Mistake | Trivial | fs | Sebastian Baunsgaard | Sebastian Baunsgaard
MAPREDUCE-7437 | MR Fetcher class to use an AtomicInteger to generate IDs. | Major | build, client | Steve Loughran | Steve Loughran
HDFS-16672 | Fix lease interval comparison in BlockReportLeaseManager | Trivial | namenode | dzcxzl | dzcxzl
YARN-11482 | Fix bug of DRF comparison DominantResourceFairnessComparator2 in fair scheduler | Major | fairscheduler | Xiaoqiao He | Xiaoqiao He
HDFS-16897 | Fix abundant Broken pipe exception in BlockSender | Minor | hdfs | fanluo | fanluo
HADOOP-18715 | Add debug log for getting details of tokenKindMap | Minor | . | Pralabh Kumar | Pralabh Kumar
YARN-11312 | [UI2] Refresh buttons don’t work after EmberJS upgrade | Minor | yarn-ui-v2 | Brian Goerlitz | Susheel Gupta
HADOOP-18724 | Open file fails with NumberFormatException for S3AFileSystem | Critical | fs, fs/azure, fs/s3 | Ayush Saxena | Steve Loughran
HADOOP-18652 | Path.suffix raises NullPointerException | Minor | hdfs-client | Patrick Grandjean | Patrick Grandjean
HDFS-17022 | Fix the exception message to print the Identifier pattern | Minor | . | Nishtha Shah | Nishtha Shah
HDFS-17017 | Fix the issue of arguments number limit in report command in DFSAdmin. | Major | . | Haiyang Hu | Haiyang Hu
HADOOP-18755 | openFile builder new optLong() methods break hbase-filesystem | Major | fs | Steve Loughran | Steve Loughran
HDFS-17011 | Fix the metric of “HttpPort” at DataNodeInfo | Minor | . | Zhaohui Wang | Zhaohui Wang
HDFS-17003 | Erasure Coding: invalidate wrong block after reporting bad blocks from datanode | Critical | namenode | farmmamba | farmmamba
HADOOP-18718 | Fix several maven build warnings | Minor | build | Dongjoon Hyun | Dongjoon Hyun

### TESTS:

JIRA | Summary | Priority | Component | Reporter | Contributor
--- | --- | --- | --- | --- | ---
YARN-11388 | Prevent resource leaks in TestClientRMService. | Minor | test | Chris Nauroth | Chris Nauroth

### SUB-TASKS:

JIRA | Summary | Priority | Component | Reporter | Contributor
--- | --- | --- | --- | --- | ---
HDFS-15654 | TestBPOfferService#testMissBlocksWhenReregister fails intermittently | Major | datanode | Ahmed Hussein | Ahmed Hussein
HDFS-15674 | TestBPOfferService#testMissBlocksWhenReregister fails on trunk | Major | . | Ahmed Hussein | Masatake Iwasaki
HADOOP-18380 | fs.s3a.prefetch.block.size to be read through longBytesOption | Major | fs/s3 | Steve Loughran | Viraj Jasani
HADOOP-18186 | s3a prefetching to use SemaphoredDelegatingExecutor for submitting work | Major | fs/s3 | Steve Loughran | Viraj Jasani
HADOOP-18377 | hadoop-aws maven build to add a prefetch profile to run all tests with prefetching | Major | fs/s3, test | Steve Loughran | Viraj Jasani
HADOOP-18455 | s3a prefetching Executor should be closed | Major | fs/s3 | Viraj Jasani | Viraj Jasani
HADOOP-18378 | Implement readFully(long position, byte[] buffer, int offset, int length) | Minor | fs/s3 | Ahmar Suhail | Alessandro Passaro
HADOOP-18189 | S3PrefetchingInputStream to support status probes when closed | Minor | fs/s3 | Steve Loughran | Viraj Jasani
HADOOP-18156 | Address JavaDoc warnings in classes like MarkerTool, S3ObjectAttributes, etc. | Minor | fs/s3 | Mukund Thakur | Ankit Saurabh
HADOOP-18482 | ITestS3APrefetchingInputStream does not skip if no CSV test file available | Minor | fs/s3 | Daniel Carl Jones | Daniel Carl Jones
HADOOP-18531 | assertion failure in ITestS3APrefetchingInputStream | Major | fs/s3, test | Steve Loughran | Ashutosh Gupta
HADOOP-18620 | Avoid using grizzly-http-* APIs | Major | . | Viraj Jasani | Viraj Jasani
HADOOP-18351 | S3A prefetching: Error logging during reads | Minor | . | Ahmar Suhail | Ankit Saurabh
HADOOP-17746 | compatibility table in directory_markers.md doesn’t render right | Minor | fs/s3, site | Steve Loughran | Masatake Iwasaki
HADOOP-18606 | Add reason in in x-ms-client-request-id on a retry API call. | Minor | fs/azure | Pranav Saxena | Pranav Saxena
HADOOP-18146 | ABFS: Add changes for expect hundred continue header with append requests | Major | fs/azure | Anmol Asrani | Anmol Asrani
HADOOP-18647 | x-ms-client-request-id to have some way that identifies retry of an API. | Minor | fs/azure | Pranav Saxena | Pranav Saxena
HADOOP-18012 | ABFS: Enable config controlled ETag check for Rename idempotency | Major | fs/azure | Sneha Vijayarajan | Sree Bhattacharyya
HADOOP-18696 | S3A ITestS3ABucketExistence access point test failure | Major | fs/s3, test | Steve Loughran | Steve Loughran
HADOOP-18399 | S3A Prefetch - SingleFilePerBlockCache to use LocalDirAllocator | Major | fs/s3 | Steve Loughran | Viraj Jasani
HADOOP-18703 | Backport S3A prefetching stream to branch-3.3 | Major | fs/s3 | Steve Loughran | Steve Loughran
HADOOP-18697 | Fix transient failure of ITestS3APrefetchingInputStream#testRandomReadLargeFile | Major | fs/s3, test | Steve Loughran | Viraj Jasani
HADOOP-18688 | S3A audit header to include count of items in delete ops | Major | fs/s3 | Steve Loughran | Viraj Jasani
HADOOP-18740 | s3a prefetch cache blocks should be accessed by RW locks | Major | . | Viraj Jasani | Viraj Jasani
HADOOP-18763 | Upgrade aws-java-sdk to 1.12.367+ | Major | fs/s3 | Steve Loughran | Viraj Jasani

### OTHER:

JIRA | Summary | Priority | Component | Reporter | Contributor
--- | --- | --- | --- | --- | ---
HDFS-16822 | HostRestrictingAuthorizationFilter should pass through requests if they don’t access WebHDFS API | Major | . | Takanobu Asanuma | Takanobu Asanuma
HDFS-16886 | Fix documentation for StateStoreRecordOperations#get(Class …, Query …) | Trivial | . | Simbarashe Dzinamarira | Simbarashe Dzinamarira
HDFS-16902 | Add Namenode status to BPServiceActor metrics and improve logging in offerservice | Major | . | Viraj Jasani | Viraj Jasani
HDFS-16901 | RBF: Routers should propagate the real user in the UGI via the caller context | Major | . | Simbarashe Dzinamarira | Simbarashe Dzinamarira
HADOOP-18658 | snakeyaml dependency: upgrade to v2.0 | Major | . | PJ Fanning | PJ Fanning
HADOOP-18693 | Upgrade Apache Derby from 10.10.2.0 to 10.14.2.0 due to CVEs | Major | build, test | PJ Fanning |
HADOOP-18712 | Upgrade to jetty 9.4.51 due to cve | Major | common | PJ Fanning | PJ Fanning
HADOOP-18727 | Fix WriteOperations.listMultipartUploads function description | Trivial | fs/s3 | Dongjoon Hyun | Dongjoon Hyun
HADOOP-18761 | Remove mysql-connector-java | Blocker | . | Wei-Chiu Chuang | Wei-Chiu Chuang