Apache Hadoop 3.5.0 – Apache Hadoop Changelog

##### General

- Overview
- Single Node Setup
- Cluster Setup
- Commands Reference
- FileSystem Shell
- Compatibility Specification
- Downstream Developer's Guide
- Admin Compatibility Guide
- Interface Classification
- FileSystem Specification

##### Common

- CLI Mini Cluster
- Fair Call Queue
- Native Libraries
- Proxy User
- Rack Awareness
- Secure Mode
- Service Level Authorization
- HTTP Authentication
- Credential Provider API
- Hadoop KMS
- Tracing
- Unix Shell Guide
- Registry
- Async Profiler

##### HDFS

- Architecture
- User Guide
- Commands Reference
- NameNode HA With QJM
- NameNode HA With NFS
- Observer NameNode
- Federation
- ViewFs
- ViewFsOverloadScheme
- Snapshots
- Edits Viewer
- Image Viewer
- Permissions and HDFS
- Quotas and HDFS
- libhdfs (C API)
- WebHDFS (REST API)
- HttpFS
- Short Circuit Local Reads
- Centralized Cache Management
- NFS Gateway
- Rolling Upgrade
- Extended Attributes
- Transparent Encryption
- Multihoming
- Storage Policies
- Memory Storage Support
- Synthetic Load Generator
- Erasure Coding
- Disk Balancer
- Upgrade Domain
- DataNode Admin
- Router Federation
- Provided Storage

##### MapReduce

- Tutorial
- Commands Reference
- Encrypted Shuffle
- Pluggable Shuffle/Sort
- Distributed Cache Deploy
- Support for YARN Shared Cache

##### MapReduce REST APIs

- MR Application Master
- MR History Server

##### YARN

- Architecture
- Commands Reference
- Capacity Scheduler
- Fair Scheduler
- ResourceManager Restart
- ResourceManager HA
- Resource Model
- Node Labels
- Node Attributes
- Web Application Proxy
- Timeline Server
- Timeline Service V.2
- Writing YARN Applications
- YARN Application Security
- NodeManager
- Running Applications in Docker Containers
- Running Applications in runC Containers
- Using CGroups
- Secure Containers
- Reservation System
- Graceful Decommission
- Opportunistic Containers
- YARN Federation
- Shared Cache
- Using GPU
- Using FPGA
- Placement Constraints
- YARN UI2

##### YARN REST APIs

- Introduction
- Resource Manager
- Node Manager
- Timeline Server
- Timeline Service V.2

##### YARN Service

- Overview
- QuickStart
- Concepts
- Yarn Service API
- Service Discovery
- System Services

##### Hadoop Compatible File Systems

- Aliyun OSS
- Amazon S3
- Azure Blob Storage
- Azure Data Lake Storage
- Tencent COS
- Huaweicloud OBS
- VolcanoEngine TOS

##### Auth

- Overview
- Examples
- Configuration
- Building

##### Tools

- Hadoop Streaming
- Hadoop Archives
- Hadoop Archive Logs
- DistCp
- HDFS Federation Balance
- GridMix
- Rumen
- Resource Estimator Service
- Scheduler Load Simulator
- Hadoop Benchmarking
- Dynamometer

##### Reference

- Changelog and Release Notes
- Java API docs
- Unix Shell API
- Metrics

##### Configuration

- core-default.xml
- hdfs-default.xml
- hdfs-rbf-default.xml
- mapred-default.xml
- yarn-default.xml
- kms-default.xml
- httpfs-default.xml
- Deprecated Properties

# Apache Hadoop Changelog

## Release 3.3.5 - 2023-03-14

### IMPORTANT ISSUES:

JIRA | Summary | Priority | Component | Reporter | Contributor
--- | --- | --- | --- | --- | ---
HADOOP-17956 | Replace all default Charset usage with UTF-8 | Major | . | Viraj Jasani | Viraj Jasani
HADOOP-18621 | CryptoOutputStream::close leak when encrypted zones + quota exceptions | Critical | fs | Colm Dougan | Colm Dougan

### NEW FEATURES:

JIRA | Summary | Priority | Component | Reporter | Contributor
--- | --- | --- | --- | --- | ---
HADOOP-18003 | Add a method appendIfAbsent for CallerContext | Minor | . | Tao Li | Tao Li
HDFS-16331 | Make dfs.blockreport.intervalMsec reconfigurable | Major | . | Tao Li | Tao Li
HDFS-16371 | Exclude slow disks when choosing volume | Major | . | Tao Li | Tao Li
HDFS-16400 | Reconfig DataXceiver parameters for datanode | Major | . | Tao Li | Tao Li
HDFS-16399 | Reconfig cache report parameters for datanode | Major | . | Tao Li | Tao Li
HDFS-16398 | Reconfig block report parameters for datanode | Major | . | Tao Li | Tao Li
HDFS-16396 | Reconfig slow peer parameters for datanode | Major | . | Tao Li | Tao Li
HDFS-16397 | Reconfig slow disk parameters for datanode | Major | . | Tao Li | Tao Li
MAPREDUCE-7341 | Add a task-manifest output committer for Azure and GCS | Major | client | Steve Loughran | Steve Loughran
HADOOP-18163 | hadoop-azure support for the Manifest Committer of MAPREDUCE-7341 | Major | fs/azure | Steve Loughran | Steve Loughran
HDFS-16413 | Reconfig dfs usage parameters for datanode | Major | . | Tao Li | Tao Li
HDFS-16521 | DFS API to retrieve slow datanodes | Major | . | Viraj Jasani | Viraj Jasani
HDFS-16568 | dfsadmin -reconfig option to start/query reconfig on all live datanodes | Major | . | Viraj Jasani | Viraj Jasani
HDFS-16582 | Expose aggregate latency of slow node as perceived by the reporting node | Major | . | Viraj Jasani | Viraj Jasani
HDFS-16595 | Slow peer metrics - add median, mad and upper latency limits | Major | . | Viraj Jasani | Viraj Jasani
YARN-11241 | Add uncleaning option for local app log file with log-aggregation enabled | Major | log-aggregation | Ashutosh Gupta | Ashutosh Gupta
HADOOP-18103 | High performance vectored read API in Hadoop | Major | common, fs, fs/adl, fs/s3 | Mukund Thakur | Mukund Thakur

### IMPROVEMENTS:

JIRA | Summary | Priority | Component | Reporter | Contributor
--- | --- | --- | --- | --- | ---
HADOOP-17276 | Extend CallerContext to make it include many items | Major | . | Hui Fei | Hui Fei
HDFS-15745 | Make DataNodePeerMetrics#LOW_THRESHOLD_MS and MIN_OUTLIER_DETECTION_NODES configurable | Major | . | Haibin Huang | Haibin Huang
HDFS-16266 | Add remote port information to HDFS audit log | Major | . | Tao Li | Tao Li
YARN-10997 | Revisit allocation and reservation logging | Major | . | Andras Gyori | Andras Gyori
HDFS-16310 | RBF: Add client port to CallerContext for Router | Major | . | Tao Li | Tao Li
HDFS-16352 | return the real datanode numBlocks in #getDatanodeStorageReport | Major | . | qinyuren | qinyuren
HDFS-16426 | fix nextBlockReportTime when trigger full block report force | Major | . | qinyuren | qinyuren
HDFS-16430 | Validate maximum blocks in EC group when adding an EC policy | Minor | ec, erasure-coding | daimin | daimin
HDFS-16403 | Improve FUSE IO performance by supporting FUSE parameter max_background | Minor | fuse-dfs | daimin | daimin
HDFS-16262 | Async refresh of cached locations in DFSInputStream | Major | . | Bryan Beaudreault | Bryan Beaudreault
HADOOP-18093 | Better exception handling for testFileStatusOnMountLink() in ViewFsBaseTest.java | Trivial | . | Xing Lin | Xing Lin
HDFS-16423 | balancer should not get blocks on stale storages | Major | balancer & mover | qinyuren | qinyuren
HADOOP-18139 | Allow configuration of zookeeper server principal | Major | auth | Owen O’Malley | Owen O’Malley
YARN-11076 | Upgrade jQuery version in Yarn UI2 | Major | yarn-ui-v2 | Tamas Domok | Tamas Domok
HDFS-16495 | RBF should prepend the client ip rather than append it. | Major | . | Owen O’Malley | Owen O’Malley
HADOOP-18144 | getTrashRoot/s in ViewFileSystem should return viewFS path, not targetFS path | Major | common | Xing Lin | Xing Lin
HADOOP-18162 | hadoop-common enhancements for the Manifest Committer of MAPREDUCE-7341 | Major | fs | Steve Loughran | Steve Loughran
HDFS-16529 | Remove unnecessary setObserverRead in TestConsistentReadsObserver | Trivial | test | Zhaohui Wang | Zhaohui Wang
HDFS-16530 | setReplication debug log creates a new string even if debug is disabled | Major | namenode | Stephen O’Donnell | Stephen O’Donnell
HDFS-16457 | Make fs.getspaceused.classname reconfigurable | Major | namenode | yanbin.zhang | yanbin.zhang
HDFS-16427 | Add debug log for BlockManager#chooseExcessRedundancyStriped | Minor | erasure-coding | Tao Li | Tao Li
HDFS-16497 | EC: Add param comment for liveBusyBlockIndices with HDFS-14768 | Minor | erasure-coding, namanode | caozhiqiang | caozhiqiang
HDFS-16389 | Improve NNThroughputBenchmark test mkdirs | Major | benchmarks, namenode | JiangHua Zhu | JiangHua Zhu
HADOOP-17551 | Upgrade maven-site-plugin to 3.11.0 | Major | . | Akira Ajisaka | Ashutosh Gupta
HDFS-16519 | Add throttler to EC reconstruction | Minor | datanode, ec | daimin | daimin
HDFS-14478 | Add libhdfs APIs for openFile | Major | hdfs-client, libhdfs, native | Sahil Takiar | Sahil Takiar
HADOOP-16202 | Enhance openFile() for better read performance against object stores | Major | fs, fs/s3, tools/distcp | Steve Loughran | Steve Loughran
YARN-11116 | Migrate Times util from SimpleDateFormat to thread-safe DateTimeFormatter class | Minor | . | Jonathan Turner Eagles | Jonathan Turner Eagles
HDFS-16520 | Improve EC pread: avoid potential reading whole block | Major | dfsclient, ec, erasure-coding | daimin | daimin
HADOOP-18167 | Add metrics to track delegation token secret manager operations | Major | . | Hector Sandoval Chaverri | Hector Sandoval Chaverri
YARN-10080 | Support show app id on localizer thread pool | Major | nodemanager | zhoukang | Ashutosh Gupta
HADOOP-18172 | Change scope of getRootFallbackLink for InodeTree to make them accessible from outside package | Minor | . | Xing Lin | Xing Lin
HDFS-16588 | Backport HDFS-16584 to branch-3.3. | Major | balancer & mover, namenode | JiangHua Zhu | JiangHua Zhu
HADOOP-18240 | Upgrade Yetus to 0.14.0 | Major | build | Akira Ajisaka | Ashutosh Gupta
HDFS-16585 | Add @VisibleForTesting in Dispatcher.java after HDFS-16268 | Trivial | . | Wei-Chiu Chuang | Ashutosh Gupta
HADOOP-18244 | Fix Hadoop-Common JavaDoc Error on branch-3.3 | Major | common | Shilun Fan | Shilun Fan
HADOOP-18269 | Misleading method name in DistCpOptions | Minor | tools/distcp | guophilipse | guophilipse
HADOOP-18275 | update os-maven-plugin to 1.7.0 | Minor | build | Steve Loughran | Steve Loughran
HDFS-16610 | Make fsck read timeout configurable | Major | hdfs-client | Stephen O’Donnell | Stephen O’Donnell
HDFS-16576 | Remove unused imports in HDFS project | Minor | . | Ashutosh Gupta | Ashutosh Gupta
HDFS-16629 | [JDK 11] Fix javadoc warnings in hadoop-hdfs module | Minor | hdfs | Shilun Fan | Shilun Fan
YARN-11172 | Fix testDelegationToken | Major | test | zhengchenyu | zhengchenyu
HADOOP-17833 | Improve Magic Committer Performance | Minor | fs/s3 | Steve Loughran | Steve Loughran
HADOOP-18288 | Total requests and total requests per sec served by RPC servers | Major | . | Viraj Jasani | Viraj Jasani
HADOOP-18336 | tag FSDataInputStream.getWrappedStream() @Public/@Stable | Minor | fs | Steve Loughran | Ashutosh Gupta
HADOOP-13144 | Enhancing IPC client throughput via multiple connections per user | Minor | ipc | Jason Kace | Íñigo Goiri
HDFS-16712 | Fix incorrect placeholder in DataNode.java | Major | . | ZanderXu | ZanderXu
HDFS-16702 | MiniDFSCluster should report cause of exception in assertion error | Minor | hdfs | Steve Vaughan | Steve Vaughan
HADOOP-18365 | Updated addresses are still accessed using the old IP address | Major | common | Steve Vaughan | Steve Vaughan
HDFS-16687 | RouterFsckServlet replicates code from DfsServlet base class | Major | federation | Steve Vaughan | Steve Vaughan
HADOOP-18333 | hadoop-client-runtime impact by CVE-2022-2047 CVE-2022-2048 due to shaded jetty | Major | build | phoebe chen | Ashutosh Gupta
HADOOP-18406 | Adds alignment context to call path for creating RPC proxy with multiple connections per user. | Major | ipc | Simbarashe Dzinamarira | Simbarashe Dzinamarira
HDFS-16684 | Exclude self from JournalNodeSyncer when using a bind host | Major | journal-node | Steve Vaughan | Steve Vaughan
HDFS-16686 | GetJournalEditServlet fails to authorize valid Kerberos request | Major | journal-node | Steve Vaughan | Steve Vaughan
YARN-11303 | Upgrade jquery ui to 1.13.2 | Major | security | D M Murali Krishna Reddy | Ashutosh Gupta
HADOOP-16769 | LocalDirAllocator to provide diagnostics when file creation fails | Minor | util | Ramesh Kumar Thangarajan | Ashutosh Gupta
HADOOP-18341 | upgrade commons-configuration2 to 2.8.0 and commons-text to 1.9 | Major | . | PJ Fanning | PJ Fanning
HDFS-16776 | Erasure Coding: The length of targets should be checked when DN gets a reconstruction task | Major | . | Kidd5368 | Kidd5368
HADOOP-18469 | Add XMLUtils methods to centralise code that creates secure XML parsers | Major | . | PJ Fanning | PJ Fanning
HADOOP-18442 | Remove the hadoop-openstack module | Major | build, fs, fs/swift | Steve Loughran | Steve Loughran
HADOOP-18468 | upgrade jettison json jar due to fix CVE-2022-40149 | Major | build | PJ Fanning | PJ Fanning
HADOOP-17779 | Lock File System Creator Semaphore Uninterruptibly | Minor | fs | David Mollitor | David Mollitor
HADOOP-18360 | Update commons-csv from 1.0 to 1.9.0. | Minor | common | Shilun Fan | Shilun Fan
HADOOP-18493 | update jackson-databind 2.12.7.1 due to CVE fixes | Major | . | PJ Fanning | PJ Fanning
HADOOP-17563 | Update Bouncy Castle to 1.68 or later | Major | build | Takanobu Asanuma | PJ Fanning
HADOOP-18497 | Upgrade commons-text version to fix CVE-2022-42889 | Major | build | Xiaoqiao He | PJ Fanning
HDFS-16795 | Use secure XML parser utils in hdfs classes | Major | . | PJ Fanning | PJ Fanning
YARN-11330 | Use secure XML parser utils in YARN | Major | . | PJ Fanning | PJ Fanning
MAPREDUCE-7411 | Use secure XML parser utils in MapReduce | Major | . | PJ Fanning | PJ Fanning
HADOOP-18512 | upgrade woodstox-core to 5.4.0 for security fix | Major | common | phoebe chen | PJ Fanning
YARN-11363 | Remove unused TimelineVersionWatcher and TimelineVersion from hadoop-yarn-server-tests | Major | test, yarn | Ashutosh Gupta | Ashutosh Gupta
YARN-11364 | Docker Container to accept docker Image name with sha256 digest | Major | yarn | Ashutosh Gupta | Ashutosh Gupta
HADOOP-18517 | ABFS: Add fs.azure.enable.readahead option to disable readahead | Major | fs/azure | Steve Loughran | Steve Loughran
HADOOP-18484 | upgrade hsqldb to v2.7.1 due to CVE | Major | . | PJ Fanning | Ashutosh Gupta
HDFS-16844 | [RBF] The routers should be resiliant against exceptions from StateStore | Major | rbf | Owen O’Malley | Owen O’Malley
HADOOP-18573 | Improve error reporting on non-standard kerberos names | Blocker | security | Steve Loughran | Steve Loughran
HADOOP-18561 | CVE-2021-37533 on commons-net is included in hadoop common and hadoop-client-runtime | Blocker | build | phoebe chen | Steve Loughran
HADOOP-18067 | Über-jira: S3A Hadoop 3.3.5 features | Major | fs/s3 | Steve Loughran | Mukund Thakur
YARN-10444 | Node Manager to use openFile() with whole-file read policy for localizing files. | Minor | nodemanager | Steve Loughran | Steve Loughran
HADOOP-18661 | Fix bin/hadoop usage script terminology | Blocker | scripts | Steve Loughran | Steve Loughran

### BUG FIXES:

JIRA | Summary | Priority | Component | Reporter | Contributor
--- | --- | --- | --- | --- | ---
HADOOP-17116 | Skip Retry INFO logging on first failover from a proxy | Major | ha | Hanisha Koneru | Hanisha Koneru
YARN-10553 | Refactor TestDistributedShell | Major | distributed-shell, test | Ahmed Hussein | Ahmed Hussein
HDFS-15839 | RBF: Cannot get method setBalancerBandwidth on Router Client | Major | rbf | Yang Yun | Yang Yun
HADOOP-17588 | CryptoInputStream#close() should be synchronized | Major | . | Renukaprasad C | Renukaprasad C
HADOOP-17836 | Improve logging on ABFS error reporting | Minor | fs/azure | Steve Loughran | Steve Loughran
HADOOP-17989 | ITestAzureBlobFileSystemDelete failing “Operations has null HTTP response” | Major | fs/azure, test | Steve Loughran | Steve Loughran
YARN-11055 | In cgroups-operations.c some fprintf format strings don’t end with “\n” | Minor | nodemanager | Gera Shegalov | Gera Shegalov
YARN-11065 | Bump follow-redirects from 1.13.3 to 1.14.7 in hadoop-yarn-ui | Major | yarn-ui-v2 | Akira Ajisaka |
HDFS-16303 | Losing over 100 datanodes in state decommissioning results in full blockage of all datanode decommissioning | Major | . | Kevin Wikant | Kevin Wikant
HDFS-16443 | Fix edge case where DatanodeAdminDefaultMonitor doubly enqueues a DatanodeDescriptor on exception | Major | hdfs | Kevin Wikant | Kevin Wikant
HDFS-16449 | Fix hadoop web site release notes and changelog not available | Minor | documentation | guophilipse | guophilipse
YARN-10788 | TestCsiClient fails | Major | test | Akira Ajisaka | Akira Ajisaka
HADOOP-18126 | Update junit 5 version due to build issues | Major | bulid | PJ Fanning | PJ Fanning
YARN-11033 | isAbsoluteResource is not correct for dynamically created queues | Minor | yarn | Tamas Domok | Tamas Domok
YARN-10894 | Follow up YARN-10237: fix the new test case in TestRMWebServicesCapacitySched | Major | . | Tamas Domok | Tamas Domok
YARN-11022 | Fix the documentation for max-parallel-apps in CS | Major | capacity scheduler | Tamas Domok | Tamas Domok
HADOOP-18150 | Fix ITestAuditManagerDisabled after S3A audit logging was enabled in HADOOP-18091 | Major | fs/s3 | Mehakmeet Singh | Mehakmeet Singh
HADOOP-17976 | abfs etag extraction inconsistent between LIST and HEAD calls | Minor | fs/azure | Steve Loughran | Steve Loughran
HADOOP-18129 | Change URI[] in INodeLink to String[] to reduce memory footprint of ViewFileSystem | Major | . | Abhishek Das | Abhishek Das
HADOOP-18145 | Fileutil’s unzip method causes unzipped files to lose their original permissions | Major | common | jingxiong zhong | jingxiong zhong
HDFS-16518 | KeyProviderCache close cached KeyProvider with Hadoop ShutdownHookManager | Major | hdfs | Lei Yang | Lei Yang
HADOOP-18169 | getDelegationTokens in ViewFs should also fetch the token from the fallback FS | Major | . | Xing Lin | Xing Lin
HDFS-16479 | EC: NameNode should not send a reconstruction work when the source datanodes are insufficient | Critical | ec, erasure-coding | Yuanbo Liu | Takanobu Asanuma
HDFS-16509 | Fix decommission UnsupportedOperationException: Remove unsupported | Major | namenode | daimin | daimin
HDFS-16456 | EC: Decommission a rack with only on dn will fail when the rack number is equal with replication | Critical | ec, namenode | caozhiqiang | caozhiqiang
HADOOP-18201 | Remove base and bucket overrides for endpoint in ITestS3ARequesterPays.java | Major | fs/s3 | Mehakmeet Singh | Daniel Carl Jones
HDFS-16536 | TestOfflineImageViewer fails on branch-3.3 | Major | test | Akira Ajisaka | Ashutosh Gupta
HDFS-16538 | EC decoding failed due to not enough valid inputs | Major | erasure-coding | qinyuren | qinyuren
HDFS-16544 | EC decoding failed due to invalid buffer | Major | erasure-coding | qinyuren | qinyuren
HADOOP-17564 | Fix typo in UnixShellGuide.html | Trivial | . | Takanobu Asanuma | Ashutosh Gupta
HDFS-16552 | Fix NPE for TestBlockManager | Major | . | Tao Li | Tao Li
MAPREDUCE-7246 | In MapredAppMasterRest#Mapreduce_Application_Master_Info_API, the datatype of appId should be “string”. | Major | documentation | jenny | Ashutosh Gupta
YARN-10187 | Removing hadoop-yarn-project/hadoop-yarn/README as it is no longer maintained. | Minor | documentation | N Sanketh Reddy | Ashutosh Gupta
HADOOP-16515 | Update the link to compatibility guide | Minor | documentation | Akira Ajisaka | Ashutosh Gupta
HDFS-16185 | Fix comment in LowRedundancyBlocks.java | Minor | documentation | Akira Ajisaka | Ashutosh Gupta
HADOOP-17479 | Fix the examples of hadoop config prefix | Minor | documentation | Akira Ajisaka | Ashutosh Gupta
HADOOP-18222 | Prevent DelegationTokenSecretManagerMetrics from registering multiple times | Major | . | Hector Sandoval Chaverri | Hector Sandoval Chaverri
HDFS-16540 | Data locality is lost when DataNode pod restarts in kubernetes | Major | namenode | Huaxiang Sun | Huaxiang Sun
YARN-11133 | YarnClient gets the wrong EffectiveMinCapacity value | Major | api | Zilong Zhu | Zilong Zhu
YARN-10850 | TimelineService v2 lists containers for all attempts when filtering for one | Major | timelinereader | Benjamin Teke | Benjamin Teke
YARN-11141 | Capacity Scheduler does not support ambiguous queue names when moving application across queues | Major | capacity scheduler | András Győri | András Győri
HDFS-16586 | Purge FsDatasetAsyncDiskService threadgroup; it causes BPServiceActor$CommandProcessingThread IllegalThreadStateException ‘fatal exception and exit’ | Major | datanode | Michael Stack | Michael Stack
HADOOP-18251 | Fix failure of extracting JIRA id from commit message in git_jira_fix_version_check.py | Minor | build | Masatake Iwasaki | Masatake Iwasaki
YARN-11128 | Fix comments in TestProportionalCapacityPreemptionPolicy* | Minor | capacityscheduler, documentation | Ashutosh Gupta | Ashutosh Gupta
HADOOP-18234 | s3a access point xml examples are wrong | Minor | documentation, fs/s3 | Steve Loughran | Ashutosh Gupta
HADOOP-18238 | Fix reentrancy check in SFTPFileSystem.close() | Major | common | yi liu | Ashutosh Gupta
HDFS-16583 | DatanodeAdminDefaultMonitor can get stuck in an infinite loop | Major | . | Stephen O’Donnell | Stephen O’Donnell
HDFS-16608 | Fix the link in TestClientProtocolForPipelineRecovery | Minor | documentation | Samrat Deb | Samrat Deb
HDFS-16563 | Namenode WebUI prints sensitive information on Token Expiry | Major | namanode, security, webhdfs | Renukaprasad C | Renukaprasad C
HDFS-16623 | IllegalArgumentException in LifelineSender | Major | . | ZanderXu | ZanderXu
HDFS-16064 | Determine when to invalidate corrupt replicas based on number of usable replicas | Major | datanode, namenode | Kevin Wikant | Kevin Wikant
HADOOP-18255 | fsdatainputstreambuilder.md refers to hadoop 3.3.3, when it shouldn’t | Minor | documentation | Steve Loughran | Ashutosh Gupta
MAPREDUCE-7387 | Fix TestJHSSecurity#testDelegationToken AssertionError due to HDFS-16563 | Major | . | Shilun Fan | Shilun Fan
MAPREDUCE-7369 | MapReduce tasks timing out when spends more time on MultipleOutputs#close | Major | . | Prabhu Joseph | Ashutosh Gupta
MAPREDUCE-7391 | TestLocalDistributedCacheManager failing after HADOOP-16202 | Major | test | Steve Loughran | Steve Loughran
HDFS-16591 | StateStoreZooKeeper fails to initialize | Major | rbf | Hector Sandoval Chaverri | Hector Sandoval Chaverri
HADOOP-18321 | Fix when to read an additional record from a BZip2 text file split | Critical | io | Ashutosh Gupta | Ashutosh Gupta
HADOOP-18100 | Change scope of inner classes in InodeTree to make them accessible outside package | Major | . | Abhishek Das | Abhishek Das
HADOOP-18217 | shutdownhookmanager should not be multithreaded (deadlock possible) | Minor | util | Catherinot Remi |
MAPREDUCE-7372 | MapReduce set permission too late in copyJar method | Major | mrv2 | Zhang Dongsheng |
HADOOP-18330 | S3AFileSystem removes Path when calling createS3Client | Minor | fs/s3 | Ashutosh Pant | Ashutosh Pant
HADOOP-18390 | Fix out of sync import for HADOOP-18321 | Minor | . | Ashutosh Gupta | Ashutosh Gupta
HADOOP-18340 | deleteOnExit does not work with S3AFileSystem | Minor | fs/s3 | Huaxiang Sun | Huaxiang Sun
HADOOP-18383 | Codecs with @DoNotPool annotation are not closed causing memory leak | Major | common | Kevin Sewell | Kevin Sewell
HDFS-16729 | RBF: fix some unreasonably annotated docs | Major | documentation, rbf | JiangHua Zhu | JiangHua Zhu
HADOOP-18398 | Prevent AvroRecord*.class from being included non-test jar | Major | common | YUBI LEE | YUBI LEE
HDFS-4043 | Namenode Kerberos Login does not use proper hostname for host qualified hdfs principal name. | Major | security | Ahad Rana | Steve Vaughan
MAPREDUCE-7403 | Support spark dynamic partitioning in the Manifest Committer | Major | mrv2 | Steve Loughran | Steve Loughran
HDFS-16732 | [SBN READ] Avoid get location from observer when the block report is delayed. | Critical | hdfs | zhengchenyu | zhengchenyu
HADOOP-18375 | Fix failure of shelltest for hadoop_add_ldlibpath | Minor | test | Masatake Iwasaki | Masatake Iwasaki
HDFS-16755 | TestQJMWithFaults.testUnresolvableHostName() can fail due to unexpected host resolution | Minor | test | Steve Vaughan | Steve Vaughan
HADOOP-18400 | Fix file split duplicating records from a succeeding split when reading BZip2 text files | Critical | . | Ashutosh Gupta | Ashutosh Gupta
HADOOP-18242 | ABFS Rename Failure when tracking metadata is in incomplete state | Major | fs/azure | Mehakmeet Singh | Mehakmeet Singh
HADOOP-18456 | NullPointerException in ObjectListingIterator’s constructor | Blocker | fs/s3 | Quanlong Huang | Steve Loughran
HADOOP-18444 | Add Support for localized trash for ViewFileSystem in Trash.moveToAppropriateTrash | Major | . | Xing Lin | Xing Lin
HADOOP-18443 | Upgrade snakeyaml to 1.32 | Major | security | Ashutosh Gupta | Ashutosh Gupta
HDFS-16766 | hdfs ec command loads (administrator provided) erasure code policy files without disabling xml entity expansion | Major | security | Jing | Ashutosh Gupta
HDFS-13369 | FSCK Report broken with RequestHedgingProxyProvider | Major | hdfs | Harshakiran Reddy | Ranith Sardar
YARN-11039 | LogAggregationFileControllerFactory::getFileControllerForRead can leak threads | Blocker | log-aggregation | Rajesh Balamohan | Steve Loughran
HADOOP-18499 | S3A to support HTTPS web proxies | Major | fs/s3 | Mehakmeet Singh | Mehakmeet Singh
HADOOP-18233 | Possible race condition with TemporaryAWSCredentialsProvider | Major | auth, fs/s3 | Jason Sleight | Jimmy Wong
MAPREDUCE-7425 | Document Fix for yarn.app.mapreduce.client-am.ipc.max-retries | Major | yarn | teng wang | teng wang
HADOOP-18528 | Disable abfs prefetching by default | Major | fs/azure | Mehakmeet Singh | Mehakmeet Singh
HDFS-16836 | StandbyCheckpointer can still trigger rollback fs image after RU is finalized | Major | hdfs | Lei Yang | Lei Yang
HADOOP-18324 | Interrupting RPC Client calls can lead to thread exhaustion | Critical | ipc | Owen O’Malley | Owen O’Malley
HDFS-16832 | [SBN READ] Fix NPE when check the block location of empty directory | Major | . | zhengchenyu | zhengchenyu
HADOOP-18498 | [ABFS]: Error introduced when SAS Token containing ‘?’ prefix is passed | Minor | fs/azure | Sree Bhattacharyya | Sree Bhattacharyya
HDFS-16847 | RBF: StateStore writer should not commit tmp fail if there was an error in writing the file. | Critical | hdfs, rbf | Simbarashe Dzinamarira | Simbarashe Dzinamarira
HADOOP-18401 | No ARM binaries in branch-3.3.x releases | Minor | build | Ling Xu |
HADOOP-18408 | [ABFS]: ITestAbfsManifestCommitProtocol fails on nonHNS configuration | Minor | fs/azure, test | Pranav Saxena | Sree Bhattacharyya
HADOOP-18402 | S3A committer NPE in spark job abort | Blocker | fs/s3 | Steve Loughran | Steve Loughran
HADOOP-18569 | NFS Gateway may release buffer too early | Blocker | nfs | Attila Doroszlai | Attila Doroszlai
HADOOP-18574 | Changing log level of IOStatistics increment to make the DEBUG logs less noisy | Major | fs/s3 | Mehakmeet Singh | Mehakmeet Singh
HADOOP-18521 | ABFS ReadBufferManager buffer sharing across concurrent HTTP requests | Critical | fs/azure | Steve Loughran | Steve Loughran
MAPREDUCE-7375 | JobSubmissionFiles don’t set right permission after mkdirs | Major | mrv2 | Zhang Dongsheng |
HADOOP-17717 | Update wildfly openssl to 1.1.3.Final | Major | . | Wei-Chiu Chuang | Wei-Chiu Chuang
HADOOP-18598 | maven site generation doesn’t include javadocs | Blocker | site | Steve Loughran | Steve Loughran
HDFS-16895 | NamenodeHeartbeatService should use credentials of logged in user | Major | rbf | Hector Sandoval Chaverri | Hector Sandoval Chaverri
HDFS-16853 | The UT TestLeaseRecovery2#testHardLeaseRecoveryAfterNameNodeRestart failed because HADOOP-18324 | Blocker | . | ZanderXu | ZanderXu
HADOOP-18641 | cyclonedx maven plugin breaks builds on recent maven releases (3.9.0) | Major | build | Steve Loughran | Steve Loughran
HDFS-16923 | The getListing RPC will throw NPE if the path does not exist | Critical | . | ZanderXu | ZanderXu
HDFS-16896 | HDFS Client hedged read has increased failure rate than without hedged read | Major | hdfs-client | Tom McCormick | Tom McCormick
YARN-11383 | Workflow priority mappings is case sensitive | Major | yarn | Aparajita Choudhary | Aparajita Choudhary
HDFS-16939 | Fix the thread safety bug in LowRedundancyBlocks | Major | namanode | Shuyan Zhang | Shuyan Zhang
HDFS-16934 | org.apache.hadoop.hdfs.tools.TestDFSAdmin#testAllDatanodesReconfig regression | Minor | dfsadmin, test | Steve Loughran | Shilun Fan

### TESTS:

JIRA | Summary | Priority | Component | Reporter | Contributor
--- | --- | --- | --- | --- | ---
HDFS-16573 | Fix test TestDFSStripedInputStreamWithRandomECPolicy | Minor | test | daimin | daimin
HDFS-16637 | TestHDFSCLI#testAll consistently failing | Major | . | Viraj Jasani | Viraj Jasani
YARN-11248 | Add unit test for FINISHED_CONTAINERS_PULLED_BY_AM event on DECOMMISSIONING | Major | test | Ashutosh Gupta | Ashutosh Gupta
HDFS-16625 | Unit tests aren’t checking for PMDK availability | Major | test | Steve Vaughan | Steve Vaughan

### SUB-TASKS:

JIRA | Summary | Priority | Component | Reporter | Contributor
--- | --- | --- | --- | --- | ---
HDFS-13293 | RBF: The RouterRPCServer should transfer client IP via CallerContext to NamenodeRpcServer | Major | rbf | Baolong Mao | Hui Fei
HDFS-15630 | RBF: Fix wrong client IP info in CallerContext when requests mount points with multi-destinations. | Major | rbf | Chengwei Wang | Chengwei Wang
HADOOP-17152 | Implement wrapper for guava newArrayList and newLinkedList | Major | common | Ahmed Hussein | Viraj Jasani
HADOOP-17851 | S3A to support user-specified content encoding | Minor | fs/s3 | Holden Karau | Holden Karau
HADOOP-17492 | abfs listLocatedStatus to support incremental/async page fetching | Major | fs/azure | Steve Loughran | Steve Loughran
HADOOP-17409 | Remove S3Guard - no longer needed | Major | fs/s3 | Steve Loughran | Steve Loughran
HADOOP-18084 | ABFS: Add testfilePath while verifying test contents are read correctly | Minor | fs/azure, test | Anmol Asrani | Anmol Asrani
HDFS-16169 | Fix TestBlockTokenWithDFSStriped#testEnd2End failure | Major | test | Hui Fei | secfree
HADOOP-18091 | S3A auditing leaks memory through ThreadLocal references | Major | fs/s3 | Steve Loughran | Steve Loughran
HADOOP-18071 | ABFS: Set driver global timeout for ITestAzureBlobFileSystemBasics | Major | fs/azure | Sumangala Patki | Sumangala Patki
HADOOP-17765 | ABFS: Use Unique File Paths in Tests | Major | fs/azure | Sumangala Patki | Sumangala Patki
HADOOP-17862 | ABFS: Fix unchecked cast compiler warning for AbfsListStatusRemoteIterator | Major | fs/azure | Sumangala Patki | Sumangala Patki
HADOOP-18075 | ABFS: Fix failure caused by listFiles() in ITestAbfsRestOperationException | Major | fs/azure | Sumangala Patki | Sumangala Patki
HADOOP-18112 | Implement paging during S3 multi object delete. | Critical | fs/s3 | Mukund Thakur | Mukund Thakur
HADOOP-16204 | ABFS tests to include terasort | Minor | fs/azure, test | Steve Loughran | Steve Loughran
HDFS-13248 | RBF: Namenode need to choose block location for the client | Major | . | Wu Weiwei | Owen O’Malley
HADOOP-13704 | S3A getContentSummary() to move to listFiles(recursive) to count children; instrument use | Minor | fs/s3 | Steve Loughran | Ahmar Suhail
HADOOP-14661 | S3A to support Requester Pays Buckets | Minor | common, util | Mandus Momberg | Daniel Carl Jones
HDFS-16484 | [SPS]: Fix an infinite loop bug in SPSPathIdProcessor thread | Major | . | qinyuren | qinyuren
HADOOP-17682 | ABFS: Support FileStatus input to OpenFileWithOptions() via OpenFileParameters | Major | fs/azure | Sumangala Patki | Sumangala Patki
HADOOP-15983 | Use jersey-json that is built to use jackson2 | Major | build | Akira Ajisaka | PJ Fanning
HADOOP-18104 | Add configs to configure minSeekForVectorReads and maxReadSizeForVectorReads | Major | common, fs | Mukund Thakur | Mukund Thakur
HADOOP-18168 | ITestMarkerTool.testRunLimitedLandsatAudit failing due to most of bucket content purged | Minor | fs/s3, test | Steve Loughran | Daniel Carl Jones
HADOOP-12020 | Support configuration of different S3 storage classes | Major | fs/s3 | Yann Landrin-Schweitzer | Monthon Klongklaew
HADOOP-18105 | Implement a variant of ElasticByteBufferPool which uses weak references for garbage collection. | Major | common, fs | Mukund Thakur | Mukund Thakur
HADOOP-18107 | Vectored IO support for large S3 files. | Major | fs/s3 | Mukund Thakur | Mukund Thakur
HADOOP-18106 | Handle memory fragmentation in S3 Vectored IO implementation. | Major | fs/s3 | Mukund Thakur | Mukund Thakur
HADOOP-17461 | Add thread-level IOStatistics Context | Major | fs, fs/azure, fs/s3 | Steve Loughran | Mehakmeet Singh
HADOOP-18372 | ILoadTestS3ABulkDeleteThrottling failing | Minor | fs/s3, test | Steve Loughran | Ahmar Suhail
HADOOP-18368 | ITestCustomSigner fails when access point name has ‘-’ | Minor | . | Ahmar Suhail | Ahmar Suhail
HADOOP-15964 | Add S3A support for Async Scatter/Gather IO | Major | fs/s3 | Steve Loughran | Mukund Thakur
HADOOP-18366 | ITestS3Select.testSelectSeekFullLandsat is timing out | Minor | . | Ahmar Suhail | Ahmar Suhail
HADOOP-18373 | IOStatisticsContext tuning | Minor | fs/s3, test | Steve Loughran | Viraj Jasani
HADOOP-18227 | Add input stream IOstats for vectored IO api in S3A. | Major | fs/s3 | Mukund Thakur | Mukund Thakur
HADOOP-18392 | Propagate vectored s3a input stream stats to file system stats. | Major | fs/s3 | Mukund Thakur | Mukund Thakur
HADOOP-18355 | Update previous index properly while validating overlapping ranges. | Major | common, fs/s3 | Mukund Thakur | Mukund Thakur
HADOOP-18371 | s3a FS init logs at warn if fs.s3a.create.storage.class is unset | Blocker | fs/s3 | Steve Loughran | Viraj Jasani
HADOOP-18385 | ITestS3ACannedACLs failure; not in a span | Major | fs/s3, test | Steve Loughran | Ashutosh Gupta
HADOOP-18403 | Fix FileSystem leak in ITestS3AAWSCredentialsProvider | Minor | fs/s3 | Viraj Jasani | Viraj Jasani
HADOOP-17882 | distcp to use openFile() with sequential IO; ranges of reads | Major | tools/distcp | Steve Loughran | Steve Loughran
HADOOP-18391 | Improve VectoredReadUtils#readVectored() for direct buffers | Major | fs | Steve Loughran | Mukund Thakur
HADOOP-18407 | Improve vectored IO api spec. | Minor | fs, fs/s3 | Mukund Thakur | Mukund Thakur
HADOOP-18339 | S3A storage class option only picked up when buffering writes to disk | Major | fs/s3 | Steve Loughran | Monthon Klongklaew
HADOOP-18410 | S3AInputStream.unbuffer() async drain not releasing http connections | Blocker | fs/s3 | Steve Loughran | Steve Loughran
HADOOP-18439 | Fix VectoredIO for LocalFileSystem when checksum is enabled. | Major | common | Mukund Thakur | Mukund Thakur
HADOOP-18416 | ITestS3AIOStatisticsContext failure | Major | fs/s3, test | Steve Loughran | Mehakmeet Singh
HADOOP-18347 | Restrict vectoredIO threadpool to reduce memory pressure | Major | common, fs, fs/adl, fs/s3 | Rajesh Balamohan | Mukund Thakur
HADOOP-18463 | Add an integration test to process data asynchronously during vectored read. | Major | . | Mukund Thakur | Mukund Thakur
HADOOP-15460 | S3A FS to add “fs.s3a.create.performance” to the builder file creation option set | Major | fs/s3 | Steve Loughran | Steve Loughran
HADOOP-18382 | Upgrade AWS SDK to V2 - Prerequisites | Minor | . | Ahmar Suhail | Ahmar Suhail
HADOOP-18480 | upgrade AWS SDK to 1.12.316 | Major | build, fs/s3 | Steve Loughran | Steve Loughran
HADOOP-18460 | ITestS3AContractVectoredRead.testStopVectoredIoOperationsUnbuffer failing | Minor | fs/s3, test | Steve Loughran | Mukund Thakur
HADOOP-18488 | Cherrypick HADOOP-11245 to branch-3.3 | Major | . | Wei-Chiu Chuang | Ashutosh Gupta
HADOOP-18481 | AWS v2 SDK upgrade log to not warn of use standard AWS Credential Providers | Major | fs/s3 | Steve Loughran | Ahmar Suhail
HADOOP-18476 | Abfs and S3A FileContext bindings to close wrapped filesystems in finalizer | Blocker | fs/azure, fs/s3 | Steve Loughran | Steve Loughran
HADOOP-18304 | Improve S3A committers documentation clarity | Trivial | documentation | Daniel Carl Jones | Daniel Carl Jones
HADOOP-18465 | S3A server-side encryption tests fail before checking encryption tests should skip | Minor | fs/s3, test | Daniel Carl Jones | Daniel Carl Jones
HADOOP-18530 | ChecksumFileSystem::readVectored might return byte buffers not positioned at 0 | Blocker | fs | Harshit Gupta | Harshit Gupta
HADOOP-18457 | ABFS: Support for account level throttling | Major | . | Anmol Asrani | Anmol Asrani
HADOOP-18560 | AvroFSInput opens a stream twice and discards the second one without closing | Blocker | fs | Steve Loughran | Steve Loughran
HADOOP-18526 | Leak of S3AInstrumentation instances via hadoop Metrics references | Blocker | fs/s3 | Steve Loughran | Steve Loughran
HADOOP-18546 | disable purging list of in progress reads in abfs stream closed | Blocker | fs/azure | Steve Loughran | Pranav Saxena
HADOOP-18577 | ABFS: add probes of readahead fix | Major | fs/azure | Steve Loughran | Steve Loughran
HADOOP-11867 | Add a high-performance vectored read API. | Major | fs, fs/azure, fs/s3, hdfs-client | Gopal Vijayaraghavan | Mukund Thakur
HADOOP-18507 | VectorIO FileRange type to support a “reference” field | Major | fs | Steve Loughran | Steve Loughran
HADOOP-18627 | site intro docs to make clear Kerberos is mandatory for secure clusters | Major | site | Steve Loughran | Arnout Engelen
HADOOP-17584 | s3a magic committer may commit more data | Major | fs/s3 | yinan zhan | Steve Loughran
HADOOP-18642 | Cut excess dependencies from hadoop-azure, hadoop-aliyun transitive imports; fix LICENSE-binary | Blocker | build, fs/azure, fs/oss | Steve Loughran | Steve Loughran

### OTHER:

JIRA | Summary | Priority | Component | Reporter | Contributor
--- | --- | --- | --- | --- | ---
HDFS-15854 | Make some parameters configurable for SlowDiskTracker and SlowPeerTracker | Major | . | Tao Li | Tao Li
YARN-10747 | Bump YARN CSI protobuf version to 3.7.1 | Major | . | Siyao Meng | Siyao Meng
HDFS-16139 | Update BPServiceActor Scheduler’s nextBlockReportTime atomically | Major | . | Viraj Jasani | Viraj Jasani
HADOOP-18014 | CallerContext should not include some characters | Major | . | Takanobu Asanuma | Takanobu Asanuma
MAPREDUCE-7371 | DistributedCache alternative APIs should not use DistributedCache APIs internally | Major | . | Viraj Jasani | Viraj Jasani
HADOOP-18114 | Documentation Syntax Error Fix > AWS Assumed Roles | Trivial | documentation, fs/s3 | Joey Krabacher | Joey Krabacher
HDFS-16481 | Provide support to set Http and Rpc ports in MiniJournalCluster | Major | . | Viraj Jasani | Viraj Jasani
HDFS-16502 | Reconfigure Block Invalidate limit | Major | . | Viraj Jasani | Viraj Jasani
HDFS-16522 | Set Http and Ipc ports for Datanodes in MiniDFSCluster | Major | . | Viraj Jasani | Viraj Jasani
HADOOP-18191 | Log retry count while handling exceptions in RetryInvocationHandler | Minor | . | Viraj Jasani | Viraj Jasani
HDFS-16551 | Backport HADOOP-17588 to 3.3 and other active old branches. | Major | . | Renukaprasad C | Renukaprasad C
HDFS-16618 | sync_file_range error should include more volume and file info | Minor | . | Viraj Jasani | Viraj Jasani
HADOOP-18300 | Update google-gson to 2.9.0 | Minor | build | Igor Dvorzhak | Igor Dvorzhak
HADOOP-18397 | Shutdown AWSSecurityTokenService when its resources are no longer in use | Major | fs/s3 | Viraj Jasani | Viraj Jasani
HADOOP-18575 | Make XML transformer factory more lenient | Major | common | PJ Fanning | PJ Fanning
HADOOP-18586 | Update the year to 2023 | Major | . | Ayush Saxena | Ayush Saxena
HADOOP-18587 | upgrade to jettison 1.5.3 to fix CVE-2022-40150 | Major | common | PJ Fanning | PJ Fanning