Apache Hadoop Azure support – Deprecated WASB Driver

##### General

- Overview
- Single Node Setup
- Cluster Setup
- Commands Reference
- FileSystem Shell
- Compatibility Specification
- Downstream Developer's Guide
- Admin Compatibility Guide
- Interface Classification
- FileSystem Specification

##### Common

- CLI Mini Cluster
- Fair Call Queue
- Native Libraries
- Proxy User
- Rack Awareness
- Secure Mode
- Service Level Authorization
- HTTP Authentication
- Credential Provider API
- Hadoop KMS
- Tracing
- Unix Shell Guide
- Registry
- Async Profiler

##### HDFS

- Architecture
- User Guide
- Commands Reference
- NameNode HA With QJM
- NameNode HA With NFS
- Observer NameNode
- Federation
- ViewFs
- ViewFsOverloadScheme
- Snapshots
- Edits Viewer
- Image Viewer
- Permissions and HDFS
- Quotas and HDFS
- libhdfs (C API)
- WebHDFS (REST API)
- HttpFS
- Short Circuit Local Reads
- Centralized Cache Management
- NFS Gateway
- Rolling Upgrade
- Extended Attributes
- Transparent Encryption
- Multihoming
- Storage Policies
- Memory Storage Support
- Synthetic Load Generator
- Erasure Coding
- Disk Balancer
- Upgrade Domain
- DataNode Admin
- Router Federation
- Provided Storage

##### MapReduce

- Tutorial
- Commands Reference
- Encrypted Shuffle
- Pluggable Shuffle/Sort
- Distributed Cache Deploy
- Support for YARN Shared Cache

##### MapReduce REST APIs

- MR Application Master
- MR History Server

##### YARN

- Architecture
- Commands Reference
- Capacity Scheduler
- Fair Scheduler
- ResourceManager Restart
- ResourceManager HA
- Resource Model
- Node Labels
- Node Attributes
- Web Application Proxy
- Timeline Server
- Timeline Service V.2
- Writing YARN Applications
- YARN Application Security
- NodeManager
- Running Applications in Docker Containers
- Running Applications in runC Containers
- Using CGroups
- Secure Containers
- Reservation System
- Graceful Decommission
- Opportunistic Containers
- YARN Federation
- Shared Cache
- Using GPU
- Using FPGA
- Placement Constraints
- YARN UI2

##### YARN REST APIs

- Introduction
- Resource Manager
- Node Manager
- Timeline Server
- Timeline Service V.2

##### YARN Service

- Overview
- QuickStart
- Concepts
- Yarn Service API
- Service Discovery
- System Services

##### Hadoop Compatible File Systems

- Aliyun OSS
- Amazon S3
- Azure Blob Storage
- Azure Data Lake Storage
- Tencent COS
- Huaweicloud OBS
- VolcanoEngine TOS

##### Auth

- Overview
- Examples
- Configuration
- Building

##### Tools

- Hadoop Streaming
- Hadoop Archives
- Hadoop Archive Logs
- DistCp
- HDFS Federation Balance
- GridMix
- Rumen
- Resource Estimator Service
- Scheduler Load Simulator
- Hadoop Benchmarking
- Dynamometer

##### Reference

- Changelog and Release Notes
- Java API docs
- Unix Shell API
- Metrics

##### Configuration

- core-default.xml
- hdfs-default.xml
- hdfs-rbf-default.xml
- mapred-default.xml
- yarn-default.xml
- kms-default.xml
- httpfs-default.xml
- Deprecated Properties

# Deprecated WASB Driver

### Note: WASB Driver is removed and won’t be a part of official hadoop releases starting from hadoop-3.5.0

## Introduction

WASB Driver was a legacy Hadoop File System driver that was developed to support FNS(FlatNameSpace) Azure Storage accounts that do not honor File-Folder syntax. HDFS Folder operations hence were mimicked at client side by WASB driver and certain folder operations like Rename and Delete could lead to a lot of IOPs with client-side enumeration and orchestration of rename/delete operation blob by blob. It was not ideal for other APIs too as initial checks for path is a file or folder needed to be done over multiple metadata calls. These led to a degraded performance.

To provide better service to Analytics users, Microsoft released ADLS Gen2 which are HNS (Hierarchical Namespace) enabled, i.e. File-Folder aware storage accounts. ABFS driver was designed to overcome the inherent deficiencies of WASB and users were informed to migrate to ABFS driver.

### Challenges and limitations of WASB Driver

Users of the legacy WASB driver face a number of challenges and limitations: 1. They cannot leverage the optimizations and benefits of the latest ABFS driver. 2. They need to deal with the compatibility issues should the files and folders were modified with the legacy WASB driver and the ABFS driver concurrently in a phased transition situation. 3. There are differences for supported features for FNS and HNS over ABFS Driver. 4. In certain cases, they must perform a significant amount of re-work on their workloads to migrate to the ABFS driver, which is available only on HNS enabled accounts in a fully tested and supported scenario.

## Deprecation of WASB Driver

We have introduced a new feature that will enable the ABFS driver to support FNS accounts (over BlobEndpoint that WASB Driver uses) using the ABFS scheme. This feature will enable us to use the ABFS driver to interact with data stored in GPv2 (General Purpose v2) storage accounts.

With this feature, the WASB users are now required to migrate to the ABFS driver without much re-work on their workloads. They will however need to change the URIs from the WASB scheme to the ABFS scheme. Refer to Wasb To Abfs Migration Guide for more details.

With removal of WASB Driver our aim is to remove any ambiguity for new user onboards as there will be only one Microsoft driver for Azure Storage and migrating users will get SLA bound support for driver and service, which was not guaranteed over WASB.

We anticipate that this feature will serve as a stepping stone for users to move to HNS enabled accounts with the ABFS driver, which is our recommended stack for big data analytics on ADLS Gen2.

### Impact for existing ABFS users using ADLS Gen2 (HNS enabled account)

Removal of WASB Driver does not impact the existing users who are using ADLS Gen2 Accounts (HNS enabled account) with ABFS driver.

They do not need to make any changes to their workloads or configurations. They will still enjoy the benefits of HNS, such as atomic operations, fine-grained access control, scalability, and performance.

### Official recommendation

Microsoft continues to recommend all Big Data and Analytics users to use Azure Data Lake Gen2 (ADLS Gen2) using the ABFS driver and will continue to optimize this scenario in the future, we believe that this new option will help all those users to transition to a supported scenario immediately, while they plan to ultimately move to ADLS Gen2 (HNS enabled account).

### New Authentication Options for a migrating user

Below auth types that WASB provided will continue to work on the new FNS over ABFS Driver over configuration that accepts these SAS types (similar to WASB): 1. SharedKey 2. Account SAS 3. Service/Container SAS

Below authentication types that were not supported by WASB driver but supported by ABFS driver will continue to be available for new FNS over ABFS Driver 1. OAuth 2.0 Client Credentials 2. OAuth 2.0: Refresh Token 3. Azure Managed Identity 4. Custom OAuth 2.0 Token Provider

Refer to ABFS Authentication for more details.

### ABFS Features Not Available for migrating Users

Certain features of ABFS Driver will be available only to users using HNS accounts with ABFS driver. 1. ABFS Driver’s SAS Token Provider plugin for UserDelegation SAS and Fixed SAS. 2. Client Provided Encryption Key (CPK) support for Data ingress and egress.