Uploaded image for project: 'Hive'
  1. Hive
  2. HIVE-14269 Performance optimizations for data on S3
  3. HIVE-15093

S3-to-S3 Renames: Files should be moved individually rather than at a directory level



    • Sub-task
    • Status: Resolved
    • Major
    • Resolution: Won't Fix
    • 2.1.0
    • None
    • Hive
    • None


      Hive's MoveTask uses the Hive.moveFile method to move data within a distributed filesystem as well as blobstore filesystems.

      If the move is done within the same filesystem:

      1: If the source path is a subdirectory of the destination path, files will be moved one by one using a threapool of workers

      2: If the source path is not a subdirectory of the destination path, a single rename operation is used to move the entire directory

      The second option may not work well on blobstores such as S3. Renames are not metadata operations and require copying all the data. Client connectors to blobstores may not efficiently rename directories. Worst case, the connector will copy each file one by one, sequentially rather than using a threadpool of workers to copy the data (e.g. HADOOP-13600).

      Hive already has code to rename files using a threadpool of workers, but this only occurs in case number 1.

      This JIRA aims to modify the code so that case 1 is triggered when copying within a blobstore. The focus is on copies within a blobstore because needToCopy will return true if the src and target filesystems are different, in which case a different code path is triggered.


        1. HIVE-15093.1.patch
          3 kB
          Sahil Takiar
        2. HIVE-15093.2.patch
          22 kB
          Sahil Takiar
        3. HIVE-15093.3.patch
          22 kB
          Sahil Takiar
        4. HIVE-15093.4.patch
          26 kB
          Sahil Takiar
        5. HIVE-15093.5.patch
          27 kB
          Sahil Takiar
        6. HIVE-15093.6.patch
          27 kB
          Sahil Takiar
        7. HIVE-15093.7.patch
          27 kB
          Sahil Takiar
        8. HIVE-15093.8.patch
          21 kB
          Sahil Takiar
        9. HIVE-15093.9.patch
          22 kB
          Sahil Takiar

        Issue Links



              stakiar Sahil Takiar
              stakiar Sahil Takiar
              0 Vote for this issue
              8 Start watching this issue