[HBASE-14439] New/Improved Filesystem Abstractions

• Type: Improvement
• Status: Closed
• Priority: Critical
• Resolution: Implemented
• Assignee: Unassigned
• Reporter: Ben Lau
• Created: 15/Sep/15 22:01
• Updated: 23/Jun/22 20:29
• Resolved: 23/Jun/22 20:29

Description

Ticket for work in progress on new FileSystem abstractions. Previously, we (Yahoo) submitted a ticket that would add support for humongous (1 million region+) tables via a hierarchical layout (HBASE-13991). However open source is moving in a similar but not identical direction in the future and so the patch will not be merged into open source.

We will be working on a different patch now with folks from open source. It will create/add to 2 layers-- a path abstraction layer and a use-oriented abstraction layer. The path abstraction layer is epitomized by classes like FsUtils (and in the patch new classes like AFsLayout). The use oriented abstraction layer is epitomized by existing classes like MasterFileSystem/HRegionFileSystem (and possibly new classes later) that build on the path abstraction layer and focus on 'doing things' (eg creating regions) and less on the gritty details like the paths.

This work on abstracting and isolating the paths from the use cases will help Yahoo not diverge too much from open source with its internal 'Humongous' table hierarchical layout, while also helping open source move further towards the eventual goal of redoing the FS layout in a similar (but different) hierarchical layout later that focuses on data directory uniformity (unlike the humongous patch) and storing hierarchy in the meta table instead which enables new optimizations (see HBASE-14090.)

Attached to this ticket is some work we've done at Yahoo so far that will be put into an open source HBase branch for further collaboration. The patch is not meant to be complete yet and is a work in progress. (Please wait on patch comments/reviews.) It also includes some Yahoo-specific 'humongous' layout code that will be removed before submission in open source.

Issue Links

• Dependent: HBASE-6205 Support an option to keep data of dropped table for some time
• duplicates: HBASE-16250 Remove direct use of Hadoop Path/FileSysem 2/5
• duplicates: HBASE-16251 Remove direct use of Hadoop Path/FileSysem 3/5
• duplicates: HBASE-16248 Remove direct use of Hadoop Path/FileSysem
• is duplicated by: HBASE-16249 Remove direct use of Hadoop Path/FileSysem 1/5
• is duplicated by: HBASE-16252 Remove direct use of Hadoop Path/FileSysem 4/5
• is duplicated by: HBASE-16253 Remove direct use of Hadoop Path/FileSysem 5/5
• is related to: HBASE-14090 Redo FS layout; let go of tables/regions/stores directory hierarchy in DFS
• is related to: HBASE-14887 Use Orientated Abstractions

Sub-Tasks

• update MasterStorage / RegionStorage to have a exists-in-storage check and archive methods: Closed
• Remove directory layout/ filesystem references from CompactionTool: Closed
• comment out broken test-compile references: Closed
• Remove directory layout/ filesystem references from the code in master/procedure directory: Closed
• Remove directory layout/ filesystem references from Master: Closed
• Add ThreadPool in Legacy implementations of MasterStorage/ RegionStorage: Closed
• Remove directory layout / fs references from HBase IO package: Closed
• remove directory layout / fs references from TableSnapshotScanner: Closed
• remove direct layout/fs references from mapreduce utilities: Closed
• remove directory layout / fs references from MOB: Closed
• remove directory layout / fs references from compaction: Closed
• decouple Replication from backing files of WAL: Closed
• remove directory layout / fs references from bulkload code: Closed
• Remove directory layout/ filesystem references from hbck tool: Closed
• ensure WAL code no longer presumes colocation with region storage: Closed
• remove directory layout / fs references from snapshots: Closed
• fold FSUtil classes into fs integration package: Closed
• ensure split operation doesn't directly reference fs / legacy integrations: Closed
• ensure merge operations don't reference filesystem or legacy implementation directly: Closed
• TEST: update HBaseTestingUtility to avoid direct use of filesystem / legacy implementation: Closed
• TEST: update integration tests to use MasterStorage/RegionStorage: Closed
• TEST: Remove directory layout/ filesystem references form hbck unit tests: Closed
• TEST: Remove directory layout/ filesystem references form unit tests for master/procedure: Closed
• TEST: update ScanPerformanceEvaluation to use MasterStorage / RegionStorage: Closed
• TEST: Remove directory layout/ filesystem references form unit tests for master: Closed
• TEST: update snapshot related tests to rely on MasterStorage / RegionStorage: Closed
• TEST: update mapreduce tests to use masterstorage / regionstorage: Closed
• TEST: update MOB unit tests to user MasterStorage/ RegionStorage APIs: Closed
• TEST: update archiving tests to use masterstorage/regionstorage: Closed
• TEST: update split tests: Closed
• TEST: update merge tests: Closed
• TEST: update compaction tests: Closed
• TEST: update unit tests for io package: Closed
• TEST: update tests for misc regionserver tests: Closed
• TEST: cleanup misc tests to ensure no direct filesystem use: Closed
• Remove directory layout/ filesystem references from Cleaners and a few other modules in master: Closed
• Refactor ExportSnapshot, SnapshotInfo and remove FS references from it: Closed
• Move FileLink and HFileLink classes to fs.legacy package: Closed