Uploaded image for project: 'Hadoop Common'
  1. Hadoop Common
  2. HADOOP-13204 Über-jira: S3a phase III: scale and tuning
  3. HADOOP-14138

Remove S3A ref from META-INF service discovery, rely on existing core-default entry

    Details

    • Type: Sub-task
    • Status: Resolved
    • Priority: Critical
    • Resolution: Fixed
    • Affects Version/s: 2.9.0
    • Fix Version/s: 2.8.0, 2.7.4, 3.0.0-alpha4
    • Component/s: fs/s3
    • Labels:
      None
    • Target Version/s:
    • Hadoop Flags:
      Incompatible change
    • Release Note:
      Hide
      The classpath implementing the s3a filesystem is now defined in core-default.xml. Attempting to instantiate an S3A filesystem instance using a Configuration instance which has not included the default resorts will fail. Applications should not be doing this anyway, as it will lose other critical configuration options needed by the filesystem.
      Show
      The classpath implementing the s3a filesystem is now defined in core-default.xml. Attempting to instantiate an S3A filesystem instance using a Configuration instance which has not included the default resorts will fail. Applications should not be doing this anyway, as it will lose other critical configuration options needed by the filesystem.

      Description

      As discussed in HADOOP-14132, the shaded AWS library is killing performance starting all hadoop operations, due to classloading on FS service discovery.

      This is despite the fact that there is an entry for fs.s3a.impl in core-default.xml, we don't need service discovery here

      Proposed:

      1. cut the entry from /hadoop-aws/src/main/resources/META-INF/services/org.apache.hadoop.fs.FileSystem
      2. when HADOOP-14132 is in, move to that, including declaring an XML file exclusively for s3a entries

      I want this one in first as its a major performance regression, and one we coula actually backport to 2.7.x, just to improve load time slightly there too

        Attachments

        1. HADOOP-14138-branch-2-001.patch
          0.6 kB
          Steve Loughran
        2. HADOOP-14138.001.patch
          0.6 kB
          Jason Lowe

          Issue Links

            Activity

              People

              • Assignee:
                stevel@apache.org Steve Loughran
                Reporter:
                stevel@apache.org Steve Loughran
              • Votes:
                0 Vote for this issue
                Watchers:
                7 Start watching this issue

                Dates

                • Created:
                  Updated:
                  Resolved: