[HUDI-1479] Replace FSUtils.getAllPartitionPaths() with HoodieTableMetadata#getAllPartitionPaths()

• Type: Task
• Status: Resolved
• Priority: Blocker
• Resolution: Fixed
• Fix Version/s: 0.7.0
• Component/s: code-quality
• Labels: pull-request-available
• Epic Link: Metadata Table for File Listing & Query Planning
• Assignee: Udit Mehrotra
• Reporter: Vinoth Chandar
• Created: 20/Dec/20 19:08
• Updated: 17/Jan/24 15:29
• Resolved: 11/Jan/21 06:25

Description

Change #1

public static List<String> getAllPartitionPaths(FileSystem fs, String basePathStr, boolean useFileListingFromMetadata, boolean verifyListings,
                                                  boolean assumeDatePartitioning) throws IOException {
    if (assumeDatePartitioning) {
      return getAllPartitionFoldersThreeLevelsDown(fs, basePathStr);
    } else {
      HoodieTableMetadata tableMetadata = HoodieTableMetadata.create(fs.getConf(), basePathStr, "/tmp/", useFileListingFromMetadata,
          verifyListings, false, false);
      return tableMetadata.getAllPartitionPaths();
    }
 }

is the current implementation, where `HoodieTableMetadata.create()` always creates `HoodieBackedTableMetadata`. Instead we should create `FileSystemBackedTableMetadata` if useFileListingFromMetadata==false anyways. This helps address https://github.com/apache/hudi/pull/2398/files#r550709687

Change #2

On master, we have the `HoodieEngineContext` abstraction, which allows for parallel execution. We should consider moving it to `hudi-common` (its doable) and then have `FileSystemBackedTableMetadata` redone such that it can do parallelized listings using the passed in engine. either HoodieSparkEngineContext or HoodieJavaEngineContext. HoodieBackedTableMetadata#getPartitionsToFilesMapping has some parallelized code. We should take one pass and see if that can be redone a bit as well.  Food for thought: https://github.com/apache/hudi/pull/2398#discussion_r550711216

 

Change #3

There are places, where we call fs.listStatus() directly. We should make them go through the HoodieTable.getMetadata()... route as well. Essentially, all listing should be concentrated to `FileSystemBackedTableMetadata`

Issue Links

• causes: HUDI-7307 Record index lookup fails for long running Spark jobs on secured Yarn clusters
• is a parent of: HUDI-1510 Move HoodieEngineContext to hudi-common module
• links to: GitHub Pull Request #2417