Uploaded image for project: 'Hive'
  1. Hive
  2. HIVE-22928

Allow hive.exec.stagingdir to be a fully qualified directory name

    XMLWordPrintableJSON

    Details

    • Type: Improvement
    • Status: Patch Available
    • Priority: Minor
    • Resolution: Unresolved
    • Affects Version/s: 3.1.2
    • Fix Version/s: None
    • Component/s: Configuration, Hive
    • Labels:
      None
    • Target Version/s:

      Description

      Currently, hive.exec.stagingdir can only be set as a relative directory name that, for operations like insert or insert overwrite, will be placed either under the table directory or the partition directory.

      For cases where an HDFS cluster is small but the data being inserted is very large (greater than the capacity of the HDFS cluster, as mentioned in a comment by Ashutosh Chauhan on HIVE-14270), the client may want to set their staging directory to be an explicit blobstore path (or any filesystem path), rather than relying on Hive to intelligently build the blobstore path based on an interpretation of the job. We may lose locality guarantees, but because renames are just as expensive on blobstores no matter what the prefix is, this isn't considered a terribly large loss (assuming only blobstore customers use this functionality).

      Note that hive.blobstore.use.blobstore.as.scratchdir doesn't actually suffice in this case, as the stagingdir is not the same.

      This commit enables Hive customers to set an absolute location for all staging directories. For instances where the configured stagingdir scheme is not the same as the scheme for the table location, the default stagingdir configuration is used. This avoids a cross-filesystem rename, which is impossible anyway.

        Attachments

        1. HIVE-22928.patch
          5 kB
          Thomas Poepping
        2. HIVE-22928.6.patch
          5 kB
          Thomas Poepping
        3. HIVE-22928.5.patch
          5 kB
          Thomas Poepping
        4. HIVE-22928.4.patch
          5 kB
          Thomas Poepping
        5. HIVE-22928.3.patch
          5 kB
          Thomas Poepping
        6. HIVE-22928.2.patch
          5 kB
          Thomas Poepping

          Activity

            People

            • Assignee:
              poeppt Thomas Poepping
              Reporter:
              poeppt Thomas Poepping
            • Votes:
              0 Vote for this issue
              Watchers:
              3 Start watching this issue

              Dates

              • Created:
                Updated: