Uploaded image for project: 'Hadoop Common'
  1. Hadoop Common
  2. HADOOP-14238

[Umbrella] Rechecking Guava's object is not exposed to user-facing API

    Details

    • Type: Improvement
    • Status: Open
    • Priority: Blocker
    • Resolution: Unresolved
    • Affects Version/s: None
    • Fix Version/s: None
    • Component/s: None
    • Labels:
      None

      Description

      This is reported by Hitesh Shah on HADOOP-10101.
      At least, AMRMClient#waitFor takes Guava's Supplier instance as an instance.

        Issue Links

          Activity

          Hide
          ozawa Tsuyoshi Ozawa added a comment - - edited

          I'm checking with a following method for judging whether user-facing api use Guava's object, though it's not elegant one:

          1. Executing find . -name "*.java" | xargs grep "google.common" under libraries which seems user-facing api (hadoop-hdfs-project/hadoop-hdfs-client/, hadoop-common-project, , hadoop-yarn-project/hadoop-yarn/{hadoop-yarn-client, hadoop-yarn-client}, hadoop-mapreduce-project/hadoop-mapreduce-client, )
          2. Open code and whether it has @InterfaceAudience.Private. If so, we don't need to check the code.
          3. Check the usage of guava's objects.

          Show
          ozawa Tsuyoshi Ozawa added a comment - - edited I'm checking with a following method for judging whether user-facing api use Guava's object, though it's not elegant one: 1. Executing find . -name "*.java" | xargs grep "google.common" under libraries which seems user-facing api (hadoop-hdfs-project/hadoop-hdfs-client/, hadoop-common-project, , hadoop-yarn-project/hadoop-yarn/{hadoop-yarn-client, hadoop-yarn-client}, hadoop-mapreduce-project/hadoop-mapreduce-client, ) 2. Open code and whether it has @InterfaceAudience.Private. If so, we don't need to check the code. 3. Check the usage of guava's objects.
          Hide
          ozawa Tsuyoshi Ozawa added a comment -

          I checked hadoop-hdfs-project/hadoop-hdfs-client/, and there are no exposure of Guava's object.

          Show
          ozawa Tsuyoshi Ozawa added a comment - I checked hadoop-hdfs-project/hadoop-hdfs-client/, and there are no exposure of Guava's object.
          Hide
          andrew.wang Andrew Wang added a comment -

          Since HADOOP-10101 was committed, is the plan to finish this investigation, then shade Guava everywhere?

          A more sure way of finding API pollution would be writing a bytecode analysis tool. We'd filter for classes and methods annotated IA.Public and LimitedPrivate, then check those method signatures for Guava class references.

          Another possible quick thing you could try is using JACC or JDiff to compare current vs. shaded Guava.

          Show
          andrew.wang Andrew Wang added a comment - Since HADOOP-10101 was committed, is the plan to finish this investigation, then shade Guava everywhere? A more sure way of finding API pollution would be writing a bytecode analysis tool. We'd filter for classes and methods annotated IA.Public and LimitedPrivate, then check those method signatures for Guava class references. Another possible quick thing you could try is using JACC or JDiff to compare current vs. shaded Guava.
          Hide
          ozawa Tsuyoshi Ozawa added a comment -

          Thanks Andrew for the information. Annotation File Utilities, being included in Checker Framework, can extract Annotations included in classes. Hence, we can use it to check whether classes have IA.Public and LimitedPrivate. Let me try.

          https://checkerframework.org/annotation-file-utilities/#extract-annotations

          Show
          ozawa Tsuyoshi Ozawa added a comment - Thanks Andrew for the information. Annotation File Utilities, being included in Checker Framework, can extract Annotations included in classes. Hence, we can use it to check whether classes have IA.Public and LimitedPrivate. Let me try. https://checkerframework.org/annotation-file-utilities/#extract-annotations
          Hide
          andrew.wang Andrew Wang added a comment -

          I'm upgrading this to a blocker, since it's a prereq for shading. I'm also going to file a blocker JIRA for shading guava everywhere.

          Show
          andrew.wang Andrew Wang added a comment - I'm upgrading this to a blocker, since it's a prereq for shading. I'm also going to file a blocker JIRA for shading guava everywhere.
          Hide
          andrew.wang Andrew Wang added a comment -

          Hi Tsuyoshi Ozawa, do you have an update for this JIRA?

          Show
          andrew.wang Andrew Wang added a comment - Hi Tsuyoshi Ozawa , do you have an update for this JIRA?

            People

            • Assignee:
              ozawa Tsuyoshi Ozawa
              Reporter:
              ozawa Tsuyoshi Ozawa
            • Votes:
              0 Vote for this issue
              Watchers:
              5 Start watching this issue

              Dates

              • Created:
                Updated:

                Development