Hadoop Common
  1. Hadoop Common
  2. HADOOP-5064 Hadoop 1.0
  3. HADOOP-6668

Apply audience and stability annotations to classes in common

    Details

    • Type: Sub-task Sub-task
    • Status: Closed
    • Priority: Blocker Blocker
    • Resolution: Fixed
    • Affects Version/s: None
    • Fix Version/s: 0.21.0
    • Component/s: documentation
    • Labels:
      None
    • Hadoop Flags:
      Reviewed

      Description

      Mark private implementation classes with the InterfaceAudience.Private or InterfaceAudience.LimitedPrivate annotation to exclude them from user Javadoc and JDiff.

      1. HADOOP-6668-0.21.patch
        294 kB
        Tom White
      2. HADOOP-6668.patch
        88 kB
        Tom White
      3. HADOOP-6668.patch
        88 kB
        Tom White
      4. HADOOP-6668.patch
        123 kB
        Tom White
      5. HADOOP-6668.patch
        120 kB
        Tom White
      6. HADOOP-6668.patch
        150 kB
        Tom White
      7. HADOOP-6668.patch
        294 kB
        Tom White
      8. HADOOP-6668.patch
        295 kB
        Tom White
      9. HADOOP-6668.patch
        295 kB
        Tom White
      10. HADOOP-6668.patch
        297 kB
        Tom White
      11. common.interface-tw-annotated.txt
        22 kB
        Tom White
      12. ASF.LICENSE.NOT.GRANTED--HADOOP-6668.patch
        125 kB
        Tom White

        Issue Links

          Activity

          Hide
          Hudson added a comment -

          Integrated in Hadoop-Common-trunk #364 (See http://hudson.zones.apache.org/hudson/job/Hadoop-Common-trunk/364/)
          HADOOP-6668. Apply audience and stability annotations to classes in common.

          Show
          Hudson added a comment - Integrated in Hadoop-Common-trunk #364 (See http://hudson.zones.apache.org/hudson/job/Hadoop-Common-trunk/364/ ) HADOOP-6668 . Apply audience and stability annotations to classes in common.
          Hide
          Tom White added a comment -

          I've just committed this.

          Show
          Tom White added a comment - I've just committed this.
          Hide
          Tom White added a comment -

          An updated patch for trunk (the old one fell out of date), and one for the 0.21 branch.

          Show
          Tom White added a comment - An updated patch for trunk (the old one fell out of date), and one for the 0.21 branch.
          Hide
          Hadoop QA added a comment -

          +1 overall. Here are the results of testing the latest attachment
          http://issues.apache.org/jira/secure/attachment/12446156/HADOOP-6668.patch
          against trunk revision 950605.

          +1 @author. The patch does not contain any @author tags.

          +0 tests included. The patch appears to be a documentation patch that doesn't require tests.

          +1 javadoc. The javadoc tool did not generate any warning messages.

          +1 javac. The applied patch does not increase the total number of javac compiler warnings.

          +1 findbugs. The patch does not introduce any new Findbugs warnings.

          +1 release audit. The applied patch does not increase the total number of release audit warnings.

          +1 core tests. The patch passed core unit tests.

          +1 contrib tests. The patch passed contrib unit tests.

          Test results: http://hudson.zones.apache.org/hudson/job/Hadoop-Patch-h1.grid.sp2.yahoo.net/73/testReport/
          Findbugs warnings: http://hudson.zones.apache.org/hudson/job/Hadoop-Patch-h1.grid.sp2.yahoo.net/73/artifact/trunk/build/test/findbugs/newPatchFindbugsWarnings.html
          Checkstyle results: http://hudson.zones.apache.org/hudson/job/Hadoop-Patch-h1.grid.sp2.yahoo.net/73/artifact/trunk/build/test/checkstyle-errors.html
          Console output: http://hudson.zones.apache.org/hudson/job/Hadoop-Patch-h1.grid.sp2.yahoo.net/73/console

          This message is automatically generated.

          Show
          Hadoop QA added a comment - +1 overall. Here are the results of testing the latest attachment http://issues.apache.org/jira/secure/attachment/12446156/HADOOP-6668.patch against trunk revision 950605. +1 @author. The patch does not contain any @author tags. +0 tests included. The patch appears to be a documentation patch that doesn't require tests. +1 javadoc. The javadoc tool did not generate any warning messages. +1 javac. The applied patch does not increase the total number of javac compiler warnings. +1 findbugs. The patch does not introduce any new Findbugs warnings. +1 release audit. The applied patch does not increase the total number of release audit warnings. +1 core tests. The patch passed core unit tests. +1 contrib tests. The patch passed contrib unit tests. Test results: http://hudson.zones.apache.org/hudson/job/Hadoop-Patch-h1.grid.sp2.yahoo.net/73/testReport/ Findbugs warnings: http://hudson.zones.apache.org/hudson/job/Hadoop-Patch-h1.grid.sp2.yahoo.net/73/artifact/trunk/build/test/findbugs/newPatchFindbugsWarnings.html Checkstyle results: http://hudson.zones.apache.org/hudson/job/Hadoop-Patch-h1.grid.sp2.yahoo.net/73/artifact/trunk/build/test/checkstyle-errors.html Console output: http://hudson.zones.apache.org/hudson/job/Hadoop-Patch-h1.grid.sp2.yahoo.net/73/console This message is automatically generated.
          Hide
          Tom White added a comment -

          Here's a new patch that makes the three interfaces Public Evolving. I think this is ready to go now.

          Show
          Tom White added a comment - Here's a new patch that makes the three interfaces Public Evolving. I think this is ready to go now.
          Hide
          Sanjay Radia added a comment -

          > PositionedReadable, Seekable and Syncable:

          The definitions of the methods are in the PositionedReadable, Seekable and Syncable Java interfaces.
          The definitions of the methods are needed in the user javadoc; only APis's marked as audience public
          appear in the user javadoc.
          The user can reasonably expect to lookup an extended interfaces of an audience-public API.
          The only way to avoid this is to actually copy the method signatures but not extend the interfaces.
          Unless we are prepared to do that, I think we are stuck with making them audience public.
          Hairong is probably right that the originally these interfaces were targeted for the implementor.

          Show
          Sanjay Radia added a comment - > PositionedReadable, Seekable and Syncable: The definitions of the methods are in the PositionedReadable, Seekable and Syncable Java interfaces. The definitions of the methods are needed in the user javadoc; only APis's marked as audience public appear in the user javadoc. The user can reasonably expect to lookup an extended interfaces of an audience-public API. The only way to avoid this is to actually copy the method signatures but not extend the interfaces. Unless we are prepared to do that, I think we are stuck with making them audience public. Hairong is probably right that the originally these interfaces were targeted for the implementor.
          Hide
          Hairong Kuang added a comment -

          Hi Tom, yes, I see your use case, but this is not the intended use when we designed those interfaces. I think we made a mistake to let FsDataInputStream/FsDataOutputStream implement those interfaces.

          As we discussed in our meeting, shall we mark them as "Public evolving"?

          Show
          Hairong Kuang added a comment - Hi Tom, yes, I see your use case, but this is not the intended use when we designed those interfaces. I think we made a mistake to let FsDataInputStream/FsDataOutputStream implement those interfaces. As we discussed in our meeting, shall we mark them as "Public evolving"?
          Hide
          Tom White added a comment -

          Hi Hairong,

          Do you have a comment on my use case? It would be good to get this resolved soon. Thanks!

          Show
          Tom White added a comment - Hi Hairong, Do you have a comment on my use case? It would be good to get this resolved soon. Thanks!
          Hide
          Tom White added a comment -

          The use case I was thinking of was that Seekable is needed if you want to write code that handles CompressionInputStream and FSDataInputStream polymorphically. This would allow you to write a file utility for processing compressed or uncompressed data.

          Show
          Tom White added a comment - The use case I was thinking of was that Seekable is needed if you want to write code that handles CompressionInputStream and FSDataInputStream polymorphically. This would allow you to write a file utility for processing compressed or uncompressed data.
          Hide
          Hairong Kuang added a comment -

          > users can be expected to call the public methods on FSDataInputStream that are defined in these interfaces.
          Yes, users can call these methods. But they are not expected to use these interfaces. These interfaces are intended for the implementation of the input streams of all kinds of file systems.

          Show
          Hairong Kuang added a comment - > users can be expected to call the public methods on FSDataInputStream that are defined in these interfaces. Yes, users can call these methods. But they are not expected to use these interfaces. These interfaces are intended for the implementation of the input streams of all kinds of file systems.
          Hide
          Tom White added a comment -

          > Please mark Path#makeQualified(URI defaultUri, Path workingDir ) as limited private

          Done.

          > > PositionedReadable, Seekable and Syncable: I think these are user-facing, hence Public.
          > Unfortunately these interfaces are implemented as user facing. But they are indeed for the implementation of each file system's I/O streams. I prefer not to make them public.

          FSDataInputStream implements PositionedReadable and Seekable, and users can be expected to call the public methods on FSDataInputStream that are defined in these interfaces. So it seems reasonable to make them public, doesn't it? There's a similar argument for FSDataOutputStream and Syncable.

          Show
          Tom White added a comment - > Please mark Path#makeQualified(URI defaultUri, Path workingDir ) as limited private Done. > > PositionedReadable, Seekable and Syncable: I think these are user-facing, hence Public. > Unfortunately these interfaces are implemented as user facing. But they are indeed for the implementation of each file system's I/O streams. I prefer not to make them public. FSDataInputStream implements PositionedReadable and Seekable, and users can be expected to call the public methods on FSDataInputStream that are defined in these interfaces. So it seems reasonable to make them public, doesn't it? There's a similar argument for FSDataOutputStream and Syncable.
          Hide
          Hairong Kuang added a comment -

          > PositionedReadable, Seekable and Syncable: I think these are user-facing, hence Public.
          Unfortunately these interfaces are implemented as user facing. But they are indeed for the implementation of each file system's I/O streams. I prefer not to make them public.

          Show
          Hairong Kuang added a comment - > PositionedReadable, Seekable and Syncable: I think these are user-facing, hence Public. Unfortunately these interfaces are implemented as user facing. But they are indeed for the implementation of each file system's I/O streams. I prefer not to make them public.
          Hide
          Sanjay Radia added a comment -

          Please mark Path#makeQualified(URI defaultUri, Path workingDir ) as limited private

          Reason - it will not work for chrooted file systems - this should really be an operation on the file system
          Can anyone think of a better solution?

          Show
          Sanjay Radia added a comment - Please mark Path#makeQualified(URI defaultUri, Path workingDir ) as limited private Reason - it will not work for chrooted file systems - this should really be an operation on the file system Can anyone think of a better solution?
          Hide
          Tom White added a comment -

          Thanks for the detailed feedback, Sanjay. I have produced a new patch which addresses all your points (see below). I've also gone through and added Public Stable to those classes where I had left it implicit. As you said, this is getting close now.

          > * ContentSummary, FsServerDefaults Public, Evolving. But all the writable methods are Private.

          Agreed - done.

          > * PositionedReadable, Seekable and Syncable - there was an internal debate in my corridor on this - lets discuss this on the jira.

          I think these are user-facing, hence Public. Seekable#seekToNewSource should be private though, since it's an internal method.

          > * security classes limited private evolving

          OK - done. But do we need further changes given MAPREDUCE-1651?

          > * DF, DU used by HDFS and MR for checking localfs - hence limitedPrivate evolving

          Done

          > * FsShell - we are only exposing bin/hadoop fs; FsShell is an implementation of it. Hence private

          Done

          > * Trash - private - the public cmds delete and expunge cmds expose the functionality

          I think the intent is for users to be able to move files to trash, and expunge trash, from Java programs. I think Hive and Pig may use it too. So I think it's Public Stable (or Evolving perhaps).

          > * Metrics - see my detailed comment above

          Done

          > * The io.serializer classes are private but the class name is public since it can be specified in config (use the same solution as proposed for metric plugins.

          I made the serializer classes LimitedPrivate (HDFS, MapReduce), and left the Serialization implementations Public, but with Private methods.

          > * GenericOptionsParser, private but Tool, ToolRunner public.

          OK - done.

          > * ReflectionUtils, limited private.

          Client code may use ReflectionUtils.newInstance() - so I would suggest Public Evolving.

          Show
          Tom White added a comment - Thanks for the detailed feedback, Sanjay. I have produced a new patch which addresses all your points (see below). I've also gone through and added Public Stable to those classes where I had left it implicit. As you said, this is getting close now. > * ContentSummary, FsServerDefaults Public, Evolving. But all the writable methods are Private. Agreed - done. > * PositionedReadable, Seekable and Syncable - there was an internal debate in my corridor on this - lets discuss this on the jira. I think these are user-facing, hence Public. Seekable#seekToNewSource should be private though, since it's an internal method. > * security classes limited private evolving OK - done. But do we need further changes given MAPREDUCE-1651 ? > * DF, DU used by HDFS and MR for checking localfs - hence limitedPrivate evolving Done > * FsShell - we are only exposing bin/hadoop fs; FsShell is an implementation of it. Hence private Done > * Trash - private - the public cmds delete and expunge cmds expose the functionality I think the intent is for users to be able to move files to trash, and expunge trash, from Java programs. I think Hive and Pig may use it too. So I think it's Public Stable (or Evolving perhaps). > * Metrics - see my detailed comment above Done > * The io.serializer classes are private but the class name is public since it can be specified in config (use the same solution as proposed for metric plugins. I made the serializer classes LimitedPrivate (HDFS, MapReduce), and left the Serialization implementations Public, but with Private methods. > * GenericOptionsParser, private but Tool, ToolRunner public. OK - done. > * ReflectionUtils, limited private. Client code may use ReflectionUtils.newInstance() - so I would suggest Public Evolving.
          Hide
          Sanjay Radia added a comment -

          Tom, I have finished going though your annotations on Suresh's HADOOP-6289 proposal. Here are my comments on
          areas of disagreements that you highlighted. Looks like we are fairly close now:

          • ipc packge is LimitedPrivate, Evolving - agreed, we can wait to make it stable.
          • The classification classes Public, evolving - agreed
          • Configuration related classes - I would have prefered to make only Configuration public because I am not a big fan of Configured or Configurable - I can live with making all of them public but I do feel we are exposing more than we need to.
          • ContentSummary, FsServerDefaults Public, Evolving. But all the writable methods are Private.
          • PositionedReadable, Seekable and Syncable - there was an internal debate in my corridor on this - lets discuss this on the jira.
          • security classes limited private evolving
          • DF, DU used by HDFS and MR for checking localfs - hence limitedPrivate evolving
          • FsShell - we are only exposing bin/hadoop fs; FsShell is an implementation of it. Hence private
          • Trash - private - the public cmds delete and expunge cmds expose the functionality
          • BinaryComparable, RawComparator, WritableComparator - public, agreed
          • MapFile and SequenceFile - public, agreed
          • io.compress classes - public except for bzip2 and zlib which are private unstable- agreed
          • MD5Hash - public, agreed.
          • Metrics - see my detailed comment above
          • The io.serializer classes are private but the class name is public since it can be specified in config (use the same solution as proposed for metric plugins.
          • DNSToSwitchMapping - public, agreed
          • GenericOptionsParser, private but Tool, ToolRunner public.
          • Progressable - public, agreed
          • ReflectionUtils, limited private.
          • FTPException, S3Exception, S3FileSystemException, VersionMismatchException, MultipleIOException - public, agreed
          • BloomMapFile - public agreed
          • DefaultStringifier, SetFile, WritableFactories, WritableFactory, WritableName, WritableUtils public
          • TFile - public, Evolving.
          • io.retry classes are used in HDFS and MapReduce, - LimitedPrivate, Evolving, agreed
          • IOUtils - public, agreed
          • LogLevel CLI - public, agreed
          • AbstractFileSystem.java - leave as is Public evolving
          Show
          Sanjay Radia added a comment - Tom, I have finished going though your annotations on Suresh's HADOOP-6289 proposal. Here are my comments on areas of disagreements that you highlighted. Looks like we are fairly close now: ipc packge is LimitedPrivate, Evolving - agreed, we can wait to make it stable. The classification classes Public, evolving - agreed Configuration related classes - I would have prefered to make only Configuration public because I am not a big fan of Configured or Configurable - I can live with making all of them public but I do feel we are exposing more than we need to. ContentSummary, FsServerDefaults Public, Evolving. But all the writable methods are Private. PositionedReadable, Seekable and Syncable - there was an internal debate in my corridor on this - lets discuss this on the jira. security classes limited private evolving DF, DU used by HDFS and MR for checking localfs - hence limitedPrivate evolving FsShell - we are only exposing bin/hadoop fs; FsShell is an implementation of it. Hence private Trash - private - the public cmds delete and expunge cmds expose the functionality BinaryComparable, RawComparator, WritableComparator - public, agreed MapFile and SequenceFile - public, agreed io.compress classes - public except for bzip2 and zlib which are private unstable- agreed MD5Hash - public, agreed. Metrics - see my detailed comment above The io.serializer classes are private but the class name is public since it can be specified in config (use the same solution as proposed for metric plugins. DNSToSwitchMapping - public, agreed GenericOptionsParser, private but Tool, ToolRunner public. Progressable - public, agreed ReflectionUtils, limited private. FTPException, S3Exception, S3FileSystemException, VersionMismatchException, MultipleIOException - public, agreed BloomMapFile - public agreed DefaultStringifier, SetFile, WritableFactories, WritableFactory, WritableName, WritableUtils public TFile - public, Evolving. io.retry classes are used in HDFS and MapReduce, - LimitedPrivate, Evolving, agreed IOUtils - public, agreed LogLevel CLI - public, agreed AbstractFileSystem.java - leave as is Public evolving
          Hide
          Tom White added a comment -

          Sanjay, These changes seem reasonable. 5. is a good solution to the problem of exposing class names for use in config. I'll update the patch soon.

          Show
          Tom White added a comment - Sanjay, These changes seem reasonable. 5. is a good solution to the problem of exposing class names for use in config. I'll update the patch soon.
          Hide
          Sanjay Radia added a comment -

          Have gone through the Suresh's original proposal from HADOOP-6289 and Tom's patch and comments on it.
          There are several areas of disagreements - I will comment on each of them separately.
          Let me start with metrics - it is very instructive use case with some subtle issues.
          Suresh got it mostly right in his proposal. Here is slightly modified version to highlight the
          reasoning behind the classification choices for metrics:

          1. The actual metrics themsleves - many or most are audience-public; more comfortable tagging them evolving rather than stable.
            The actual metrics have to be documented separately, since making the implementation-class audience-pubic is not appropriate.
            • RPC metrics
            • JVM Metrics
            • HDFS metrics
            • MR Metrics
          2. Interfaces for those implementing metrics (Audience-LimitedPrivate MR, HDFS)
            Developers use these Interfaces for adding new metrics to the Hadoop system (For example classes in item 3. use these interfaces to implement metrics)
            • org/apache/hadoop/metrics/util/MetricsDynamicMBeanBase.java hdfs
            • org/apache/hadoop/metrics/util/MetricsIntValue.java hdfs
            • org/apache/hadoop/metrics/util/MetricsLongValue.java hdfs
            • org/apache/hadoop/metrics/util/MetricsRegistry.java hdfs
            • org/apache/hadoop/metrics/util/MetricsTimeVaryingInt.java hdfs
            • org/apache/hadoop/metrics/util/MetricsTimeVaryingLong.java hdfs
            • org/apache/hadoop/metrics/util/MetricsTimeVaryingRate.java hdfs
            • org/apache/hadoop/metrics/util/MBeanUtil.java
            • org/apache/hadoop/metrics/Updater.java hdfs mapreduce
            • org/apache/hadoop/metrics/MetricsUtil.java
          3. Actual implementations of metrics (Audience-Private)
            (Note the actual metrics can be public - see item 1)
            • org/apache/hadoop/ipc/metrics/RpcActivityMBean.java
            • org/apache/hadoop/ipc/metrics/RpcMetrics.java
            • org/apache/hadoop/ipc/metrics/RpcMgt.java
            • org/apache/hadoop/ipc/metrics/RpcMgtMBean.java
            • org/apache/hadoop/metrics/jvm/EventCounter.java
            • org/apache/hadoop/metrics/jvm/JvmMetrics.java
          4. Interfaces needed to implement the plugins (See the actual impls in item 5: Ganglia, File, etc) - Audience-public, evolving
            • org/apache/hadoop/metrics/spi/AbstractMetricsContext.java
            • org/apache/hadoop/metrics/spi/CompositeContext.java
            • org/apache/hadoop/metrics/spi/MetricsRecordImpl.java
            • org/apache/hadoop/metrics/spi/MetricValue.java
            • org/apache/hadoop/metrics/spi/OutputRecord.java
            • org/apache/hadoop/metrics/spi/Util.java
          5. The impelmentation of metrics plugins
            The actual impl is audience-private but the class name is public since the class name is used in the config file.
            So mark the class as audience-public(evolving), but mark all its public methods as audience-private.
            • org/apache/hadoop/metrics/ganglia/GangliaContext.java
            • org/apache/hadoop/metrics/file/FileContext.java hdfs
            • org/apache/hadoop/metrics/spi/NoEmitMetricsContext.java
            • org/apache/hadoop/metrics/spi/NullContext.java
            • org/apache/hadoop/metrics/spi/NullContextWithUpdateThread.java
          6. Metrics Framework Impl - audience-private
            • org/apache/hadoop/metrics/MetricsContext.java hdfs mapreduce
            • org/apache/hadoop/metrics/util/MetricsBase.java
            • org/apache/hadoop/metrics/MetricsRecord.java hdfs mapreduce
            • org/apache/hadoop/metrics/ContextFactory.java
            • org/apache/hadoop/metrics/MetricsException.java
            • org/apache/hadoop/metrics/MetricsServlet.java
          Show
          Sanjay Radia added a comment - Have gone through the Suresh's original proposal from HADOOP-6289 and Tom's patch and comments on it. There are several areas of disagreements - I will comment on each of them separately. Let me start with metrics - it is very instructive use case with some subtle issues. Suresh got it mostly right in his proposal. Here is slightly modified version to highlight the reasoning behind the classification choices for metrics: The actual metrics themsleves - many or most are audience-public ; more comfortable tagging them evolving rather than stable . The actual metrics have to be documented separately, since making the implementation-class audience-pubic is not appropriate. RPC metrics JVM Metrics HDFS metrics MR Metrics Interfaces for those implementing metrics ( Audience-LimitedPrivate MR, HDFS) Developers use these Interfaces for adding new metrics to the Hadoop system (For example classes in item 3. use these interfaces to implement metrics) org/apache/hadoop/metrics/util/MetricsDynamicMBeanBase.java hdfs org/apache/hadoop/metrics/util/MetricsIntValue.java hdfs org/apache/hadoop/metrics/util/MetricsLongValue.java hdfs org/apache/hadoop/metrics/util/MetricsRegistry.java hdfs org/apache/hadoop/metrics/util/MetricsTimeVaryingInt.java hdfs org/apache/hadoop/metrics/util/MetricsTimeVaryingLong.java hdfs org/apache/hadoop/metrics/util/MetricsTimeVaryingRate.java hdfs org/apache/hadoop/metrics/util/MBeanUtil.java org/apache/hadoop/metrics/Updater.java hdfs mapreduce org/apache/hadoop/metrics/MetricsUtil.java Actual implementations of metrics ( Audience-Private ) (Note the actual metrics can be public - see item 1) org/apache/hadoop/ipc/metrics/RpcActivityMBean.java org/apache/hadoop/ipc/metrics/RpcMetrics.java org/apache/hadoop/ipc/metrics/RpcMgt.java org/apache/hadoop/ipc/metrics/RpcMgtMBean.java org/apache/hadoop/metrics/jvm/EventCounter.java org/apache/hadoop/metrics/jvm/JvmMetrics.java Interfaces needed to implement the plugins (See the actual impls in item 5: Ganglia, File, etc) - Audience-public , evolving org/apache/hadoop/metrics/spi/AbstractMetricsContext.java org/apache/hadoop/metrics/spi/CompositeContext.java org/apache/hadoop/metrics/spi/MetricsRecordImpl.java org/apache/hadoop/metrics/spi/MetricValue.java org/apache/hadoop/metrics/spi/OutputRecord.java org/apache/hadoop/metrics/spi/Util.java The impelmentation of metrics plugins The actual impl is audience-private but the class name is public since the class name is used in the config file. So mark the class as audience-public (evolving), but mark all its public methods as audience-private . org/apache/hadoop/metrics/ganglia/GangliaContext.java org/apache/hadoop/metrics/file/FileContext.java hdfs org/apache/hadoop/metrics/spi/NoEmitMetricsContext.java org/apache/hadoop/metrics/spi/NullContext.java org/apache/hadoop/metrics/spi/NullContextWithUpdateThread.java Metrics Framework Impl - audience-private org/apache/hadoop/metrics/MetricsContext.java hdfs mapreduce org/apache/hadoop/metrics/util/MetricsBase.java org/apache/hadoop/metrics/MetricsRecord.java hdfs mapreduce org/apache/hadoop/metrics/ContextFactory.java org/apache/hadoop/metrics/MetricsException.java org/apache/hadoop/metrics/MetricsServlet.java
          Hide
          Sanjay Radia added a comment -

          >An experienced Java user who was new to Hadoop might be surprised to learn that they couldn't use an API that had public Java visibility.
          There are two separate issues - what is the default and whether or not audience-private interfaces need to be
          labelled. The default should be: if you don't label it as audience-public then it is not. Separately we can
          decided if ALL interfaces need to be labelled. If you want to label every interface then we need a way to enforce that for consistency.
          But which every way we go, only audience-public interfaces will appear in the user-javadoc - I don't believe there was any disagreement on this at all in this or the other jiras.

          Show
          Sanjay Radia added a comment - >An experienced Java user who was new to Hadoop might be surprised to learn that they couldn't use an API that had public Java visibility. There are two separate issues - what is the default and whether or not audience-private interfaces need to be labelled. The default should be: if you don't label it as audience-public then it is not . Separately we can decided if ALL interfaces need to be labelled. If you want to label every interface then we need a way to enforce that for consistency. But which every way we go, only audience-public interfaces will appear in the user-javadoc - I don't believe there was any disagreement on this at all in this or the other jiras.
          Hide
          Doug Cutting added a comment -

          > the assumption is that only interfaces marked as audience-public will appear in the user javadoc.

          That's not what's currently implemented. But Tom tells me it would be relatively easy to change that, so that only things explicitly marked @Public would be included in the javadoc.

          If we declare that the default is @Private for Java-declared public things then I think we'll need to add @Public not just to classes but also to methods too, no? Or would you use a different default for methods? That's possible, but seems confusing. The least confusing approach would be to have defaults match Java declarations, and for the annotations only to be used when things diverge from these.

          Show
          Doug Cutting added a comment - > the assumption is that only interfaces marked as audience-public will appear in the user javadoc. That's not what's currently implemented. But Tom tells me it would be relatively easy to change that, so that only things explicitly marked @Public would be included in the javadoc. If we declare that the default is @Private for Java-declared public things then I think we'll need to add @Public not just to classes but also to methods too, no? Or would you use a different default for methods? That's possible, but seems confusing. The least confusing approach would be to have defaults match Java declarations, and for the annotations only to be used when things diverge from these.
          Hide
          Tom White added a comment -

          Currently anything marked as LimitedPrivate or Private is excluded from javadoc (HADOOP-6658). If we changed this to only include @Public annotated members, wouldn't we have to mark public methods and fields as @Public too?

          Show
          Tom White added a comment - Currently anything marked as LimitedPrivate or Private is excluded from javadoc ( HADOOP-6658 ). If we changed this to only include @Public annotated members, wouldn't we have to mark public methods and fields as @Public too?
          Hide
          Sanjay Radia added a comment -

          >An experienced Java user who was new to Hadoop might be surprised to learn that they couldn't use an API that had public Java visibility.
          There are two separate issues - what is the default and whether or not audience-private interfaces need to be
          labelled. The default should be that if you don't label it as audience-public then it is not. Separately we can
          decided if ALL interfaces need to be labelled. If you want to label every interface then we need a way to enforce that for consistency.
          But which every way we go, only audience-public interfaces will appear in the user-javadoc - I don't believe there was any disagreement on this at all in this or the other jiras.

          Show
          Sanjay Radia added a comment - >An experienced Java user who was new to Hadoop might be surprised to learn that they couldn't use an API that had public Java visibility. There are two separate issues - what is the default and whether or not audience-private interfaces need to be labelled. The default should be that if you don't label it as audience-public then it is not . Separately we can decided if ALL interfaces need to be labelled. If you want to label every interface then we need a way to enforce that for consistency. But which every way we go, only audience-public interfaces will appear in the user-javadoc - I don't believe there was any disagreement on this at all in this or the other jiras.
          Hide
          Sanjay Radia added a comment -

          >> from the point of view of the user of an interface - unless is it declared as audience-public don't assume it is.

          >This is risky if they remain in the javadoc. Folks are accustomed to assuming that documented APIs without other qualifications are APIs that will be supported in the future.
          If you see the discussion in this and the other related Jiras, the assumption is that only interfaces marked as audience-public will appear in the user javadoc. Doesn;t this address your concern?

          Show
          Sanjay Radia added a comment - >> from the point of view of the user of an interface - unless is it declared as audience-public don't assume it is. >This is risky if they remain in the javadoc. Folks are accustomed to assuming that documented APIs without other qualifications are APIs that will be supported in the future. If you see the discussion in this and the other related Jiras, the assumption is that only interfaces marked as audience-public will appear in the user javadoc. Doesn;t this address your concern?
          Hide
          Tom White added a comment -

          Sanjay> Similarly, from the point of view of the user of an interface - unless is it declared as audience-public don't assume it is.

          An experienced Java user who was new to Hadoop might be surprised to learn that they couldn't use an API that had public Java visibility.

          Do you have any comments on the notes I made in the attachment on classifications for particular classes?

          Doug> I don't fundamentally care what the default is. I do care what ends up in javadoc and in releases.

          In MAPREDUCE-1623 we have marked all the classes as Public/LimitedPrivate/Private, which has the effect of controlling what appears in Javadoc, so should be sufficient for 0.21.

          Show
          Tom White added a comment - Sanjay> Similarly, from the point of view of the user of an interface - unless is it declared as audience-public don't assume it is. An experienced Java user who was new to Hadoop might be surprised to learn that they couldn't use an API that had public Java visibility. Do you have any comments on the notes I made in the attachment on classifications for particular classes? Doug> I don't fundamentally care what the default is. I do care what ends up in javadoc and in releases. In MAPREDUCE-1623 we have marked all the classes as Public/LimitedPrivate/Private, which has the effect of controlling what appears in Javadoc, so should be sufficient for 0.21.
          Hide
          Doug Cutting added a comment -

          > if you really intend to make an interface audience-public please do so explicitly.

          We need to make it clear to users what's stable and public: we don't want them to start using something that's private or unstable without being well aware that they're doing so. If an API appears in Javadoc as 'public class Foo' with no further information then users might assume that it will be supported in the future. So if we choose to make 'private' or 'evolving' the default for otherwise unmarked public classes, then we should remove such classes from the Javadoc.

          > from the point of view of the user of an interface - unless is it declared as audience-public don't assume it is.

          This is risky if they remain in the javadoc. Folks are accustomed to assuming that documented APIs without other qualifications are APIs that will be supported in the future.

          If folks unwittingly use an evolving or private API and are broken by an upgrade it doesn't help if we tell them they didn't read our fine print. If we can get an "Evolving:" at the front of the javadoc for every evolving feature (HADOOP-6675) and exclude all non-public APIs from javadoc then users would be well forewarned.

          We should use tools like JDiff to see what new APIs are introduced. We should review the list of new APIs in each release and check that those that we might not support long-term are clearly marked as evolving.

          I don't fundamentally care what the default is. I do care what ends up in javadoc and in releases. Getting javadoc and jdiff to exclude classes without an explicit scope declaration seems perhaps a lot of work when we'll still need to closely review what's in releases. If someone feels strongly that private or evolving should be the default then perhaps they should work on getting javadoc and jdiff to enforce that?

          Show
          Doug Cutting added a comment - > if you really intend to make an interface audience-public please do so explicitly. We need to make it clear to users what's stable and public: we don't want them to start using something that's private or unstable without being well aware that they're doing so. If an API appears in Javadoc as 'public class Foo' with no further information then users might assume that it will be supported in the future. So if we choose to make 'private' or 'evolving' the default for otherwise unmarked public classes, then we should remove such classes from the Javadoc. > from the point of view of the user of an interface - unless is it declared as audience-public don't assume it is. This is risky if they remain in the javadoc. Folks are accustomed to assuming that documented APIs without other qualifications are APIs that will be supported in the future. If folks unwittingly use an evolving or private API and are broken by an upgrade it doesn't help if we tell them they didn't read our fine print. If we can get an "Evolving:" at the front of the javadoc for every evolving feature ( HADOOP-6675 ) and exclude all non-public APIs from javadoc then users would be well forewarned. We should use tools like JDiff to see what new APIs are introduced. We should review the list of new APIs in each release and check that those that we might not support long-term are clearly marked as evolving. I don't fundamentally care what the default is. I do care what ends up in javadoc and in releases. Getting javadoc and jdiff to exclude classes without an explicit scope declaration seems perhaps a lot of work when we'll still need to closely review what's in releases. If someone feels strongly that private or evolving should be the default then perhaps they should work on getting javadoc and jdiff to enforce that?
          Hide
          Sanjay Radia added a comment -

          >> > * If an API is not annotated
          >> ... It should default to audience-private unstable.
          >I don't agree with this since it violates the principle of least surprise - if an otherwise unannotated class is public, then a user would reasonably expect it to be public.

          Making an interface public is a big deal - it is basically a contract to support that interface in a compatible fashion for a very long time.
          Hence if you really intend to make an interface audience-public please do so explicitly.
          Most language default a non-declared field or method to be private. Ours is the same.
          I don't understand your point of principle of least surprise. From the point of view of the provider of the interface, there is no surprise - unless it is declared explicitly as audience-public it is not. Similarly, from the point of view of the user of an interface - unless is it declared as audience-public don't assume it is.
          It is unfortunate that Java does not have sub-package private -this would have allowed us to significantly reduce the number of public elements.

          Thus we need to explicitly label every interface that is audience-public (regardless of stability), and all audience-private interfaces that are stable, or evolving.
          Labeling everything is not necessary; further it is not possible to maintainthis consistently over time. Will we reject every patch that had undeclared classification?
          Our record for enforcing Javadoc is pathetic.

          Show
          Sanjay Radia added a comment - >> > * If an API is not annotated >> ... It should default to audience-private unstable. >I don't agree with this since it violates the principle of least surprise - if an otherwise unannotated class is public, then a user would reasonably expect it to be public. Making an interface public is a big deal - it is basically a contract to support that interface in a compatible fashion for a very long time. Hence if you really intend to make an interface audience-public please do so explicitly . Most language default a non-declared field or method to be private. Ours is the same. I don't understand your point of principle of least surprise. From the point of view of the provider of the interface, there is no surprise - unless it is declared explicitly as audience-public it is not. Similarly, from the point of view of the user of an interface - unless is it declared as audience-public don't assume it is. It is unfortunate that Java does not have sub-package private -this would have allowed us to significantly reduce the number of public elements. Thus we need to explicitly label every interface that is audience-public (regardless of stability), and all audience-private interfaces that are stable, or evolving. Labeling everything is not necessary; further it is not possible to maintainthis consistently over time. Will we reject every patch that had undeclared classification? Our record for enforcing Javadoc is pathetic.
          Hide
          Tom White added a comment -

          >Limited private for use by which components or subprojects?
          Limited private to MR and HDFS.
          >> How would I write my own metrics context?
          Who is "I"? If MR or HDFS developer then he is part of the target audience.
          If "I" is an application developer then he runs a risk using a limited-private API - it may change in incompatible ways of the MR and HDFS projects are okay with it.

          Sorry, "I" refers to an application developer writing a new metrics context. There is a service provider interface in org.apache.hadoop.metrics.spi, which clearly needs to be available to such a developer. My question was really about what is the appropriate component for LimitedPrivate, since an application developer may not host it in MR or HDFS. The option you outline is probably the best - mark the SPI classes as LimitedPrivate (MR and HDFS), and then the developer would use the developer javadoc. The only thing that isn't quite right is that the application developer is using an interface that is private to MR and HDFS, but it would work.

          The metrics APIs are separate form the actual metrics.
          You can consider marking specific classes that implement metrics public or private - but if you mark them as public then you are asserting that the actual implementation classes for the specific metrics are public - ie the impl class API is public.
          If your intension is to make specific sets of metrics (say for NN) public then you have to document that in your documentation - the annotation mechanism cannot help here unless we extend it.

          I understand, but what we need to solve is the discoverability problem - if we mark all the metrics classes as LimitedPrivate (which is what I think you are suggesting), then how do users find the documentation to configure them?

          Show
          Tom White added a comment - >Limited private for use by which components or subprojects? Limited private to MR and HDFS. >> How would I write my own metrics context? Who is "I"? If MR or HDFS developer then he is part of the target audience. If "I" is an application developer then he runs a risk using a limited-private API - it may change in incompatible ways of the MR and HDFS projects are okay with it. Sorry, "I" refers to an application developer writing a new metrics context. There is a service provider interface in org.apache.hadoop.metrics.spi, which clearly needs to be available to such a developer. My question was really about what is the appropriate component for LimitedPrivate, since an application developer may not host it in MR or HDFS. The option you outline is probably the best - mark the SPI classes as LimitedPrivate (MR and HDFS), and then the developer would use the developer javadoc. The only thing that isn't quite right is that the application developer is using an interface that is private to MR and HDFS, but it would work. The metrics APIs are separate form the actual metrics. You can consider marking specific classes that implement metrics public or private - but if you mark them as public then you are asserting that the actual implementation classes for the specific metrics are public - ie the impl class API is public. If your intension is to make specific sets of metrics (say for NN) public then you have to document that in your documentation - the annotation mechanism cannot help here unless we extend it. I understand, but what we need to solve is the discoverability problem - if we mark all the metrics classes as LimitedPrivate (which is what I think you are suggesting), then how do users find the documentation to configure them?
          Hide
          Sanjay Radia added a comment -

          > BTW the metrics themselves are also interfaces and they would have their own classification (but we can't use
          the annotations).
          Sorry I was little hasty here in my explanation.
          The metrics APIs are separate form the actual metrics.
          You can consider marking specific classes that implement metrics public or private - but if you mark them as public then you are asserting that the actual implementation classes for the specific metrics are public - ie the impl class API is public.
          If your intension is to make specific sets of metrics (say for NN) public then you have to document that in your documentation - the annotation mechanism cannot help here unless we extend it.

          Show
          Sanjay Radia added a comment - > BTW the metrics themselves are also interfaces and they would have their own classification (but we can't use the annotations). Sorry I was little hasty here in my explanation. The metrics APIs are separate form the actual metrics. You can consider marking specific classes that implement metrics public or private - but if you mark them as public then you are asserting that the actual implementation classes for the specific metrics are public - ie the impl class API is public. If your intension is to make specific sets of metrics (say for NN) public then you have to document that in your documentation - the annotation mechanism cannot help here unless we extend it.
          Hide
          Sanjay Radia added a comment -

          >>I disagree with your metrics example: .... They should be limited private.
          >Limited private for use by which components or subprojects?
          Limited private to MR and HDFS.
          >> How would I write my own metrics context?
          Who is "I"? If MR or HDFS developer then he is part of the target audience.
          If "I" is an application developer then he runs a risk using a limited-private API - it may change in incompatible ways of the MR and HDFS projects are okay with it.

          BTW the metrics themselves are also interfaces and they would have their own classification (but we can't use
          the annotations).

          Show
          Sanjay Radia added a comment - >>I disagree with your metrics example: .... They should be limited private. >Limited private for use by which components or subprojects? Limited private to MR and HDFS. >> How would I write my own metrics context? Who is "I"? If MR or HDFS developer then he is part of the target audience. If "I" is an application developer then he runs a risk using a limited-private API - it may change in incompatible ways of the MR and HDFS projects are okay with it. BTW the metrics themselves are also interfaces and they would have their own classification (but we can't use the annotations).
          Hide
          Tom White added a comment -

          Java public does not imply audience public. An interface is audience public only if we expect applications to use it directly. This is probably the biggest difference between the approach taken in this jira and HADOOP-6289.
          I do not fully understand the distinction you are trying to make in " APIs that are exposed to the user but not coded against directly (e.g. the metrics API,)"

          Users configure the metrics system, and to do so they need to have a way of discovering which metrics contexts are available. By marking the metrics classes like org.apache.hadoop.metrics.file.FileContext Private or LimitedPrivate you make them no longer discoverable, since we do not publish javadoc for private elements. (Also its class javadoc has useful information in it.) The class is public in the sense that it is referenced in the metrics configuration, even though the user does not code against it directly.

          I disagree with your metrics example: we should not expose APIs like metrics as audience public. They should be limited private.

          Limited private for use by which components or subprojects? How would I write my own metrics context?

          In the majority cases I agree, except that
          some private interfaces need to be marked as stable: protocols like client protocol should be marked as
          limited-private and stable

          I agree with this. (Indeed I marked ClientProtocol in MAPREDUCE-1623 private stable.)

          > * If an API is not annotated then it defaults to the usual Java visibility and Apache HADOOP stability (http://wiki.apache.org/hadoop/Roadmap) rules.
          This is different than what was stated in HADOOP-5073. It should default to audience-private unstable.

          I don't agree with this since it violates the principle of least surprise - if an otherwise unannotated class is public, then a user would reasonably expect it to be public. I think the onus is on developers writing code for Hadoop to mark things as private. Longer term it may help to separate the client APIs from the implementation (MAPREDUCE-1638).

          Show
          Tom White added a comment - Java public does not imply audience public. An interface is audience public only if we expect applications to use it directly. This is probably the biggest difference between the approach taken in this jira and HADOOP-6289 . I do not fully understand the distinction you are trying to make in " APIs that are exposed to the user but not coded against directly (e.g. the metrics API,)" Users configure the metrics system, and to do so they need to have a way of discovering which metrics contexts are available. By marking the metrics classes like org.apache.hadoop.metrics.file.FileContext Private or LimitedPrivate you make them no longer discoverable, since we do not publish javadoc for private elements. (Also its class javadoc has useful information in it.) The class is public in the sense that it is referenced in the metrics configuration, even though the user does not code against it directly. I disagree with your metrics example: we should not expose APIs like metrics as audience public. They should be limited private. Limited private for use by which components or subprojects? How would I write my own metrics context? In the majority cases I agree, except that some private interfaces need to be marked as stable: protocols like client protocol should be marked as limited-private and stable I agree with this. (Indeed I marked ClientProtocol in MAPREDUCE-1623 private stable.) > * If an API is not annotated then it defaults to the usual Java visibility and Apache HADOOP stability ( http://wiki.apache.org/hadoop/Roadmap ) rules. This is different than what was stated in HADOOP-5073 . It should default to audience-private unstable. I don't agree with this since it violates the principle of least surprise - if an otherwise unannotated class is public, then a user would reasonably expect it to be public. I think the onus is on developers writing code for Hadoop to mark things as private. Longer term it may help to separate the client APIs from the implementation ( MAPREDUCE-1638 ).
          Hide
          Sanjay Radia added a comment -

          The propsoal in HADOOP-5073 and HADOOP-6289 have used different criteria than
          the one stated in this Jira (see TW's comment in the begining of this Jira)
          Let me clarify:
          > * Public user APIs should be marked as 'public stable'. These appear in public Javadoc.
          > * APIs that are exposed to the user but not coded against directly should be marked as 'public evolving'. ...
          > These appear in public Javadoc.

          Java public does not imply audience public. An interface is audience public only if we expect applications to use it directly. This is probably the biggest difference between the approach taken in this jira and HADOOP-6289.
          I do not fully understand the distinction you are trying to make in " APIs that are exposed to the user but not coded against directly (e.g. the metrics API,)"
          I disagree with your metrics example: we should not expose APIs like metrics as audience public. They should be limited private.
          Don't mark an interface audience-public unless it is absolutely necessary.

          > * Non-user facing APIs that are used within HADOOP subprojects should be marked as 'limited-private evolving'
          > (e.g. RPC) or 'limited-private unstable'. These appear in developer Javadoc.
          In the majority cases I agree, except that
          some private interfaces need to be marked as stable: protocols like client protocol should be marked as
          limited-private and stable. They impact interoperation and in the future they will impact rolling upgrades.
          Again see the HADOOP-5073's FAQ.

          > * Anything that is private implementation code should be marked as 'private unstable'. These appear in developer Javadoc.
          Agreed.

          > * If an API is not annotated then it defaults to the usual Java visibility and Apache HADOOP stability (http://wiki.apache.org/hadoop/Roadmap) rules.
          This is different than what was stated in HADOOP-5073. It should default to audience-private unstable.

          Show
          Sanjay Radia added a comment - The propsoal in HADOOP-5073 and HADOOP-6289 have used different criteria than the one stated in this Jira (see TW's comment in the begining of this Jira) Let me clarify: > * Public user APIs should be marked as 'public stable'. These appear in public Javadoc. > * APIs that are exposed to the user but not coded against directly should be marked as 'public evolving'. ... > These appear in public Javadoc. Java public does not imply audience public. An interface is audience public only if we expect applications to use it directly. This is probably the biggest difference between the approach taken in this jira and HADOOP-6289 . I do not fully understand the distinction you are trying to make in " APIs that are exposed to the user but not coded against directly (e.g. the metrics API,)" I disagree with your metrics example: we should not expose APIs like metrics as audience public. They should be limited private. Don't mark an interface audience-public unless it is absolutely necessary. > * Non-user facing APIs that are used within HADOOP subprojects should be marked as 'limited-private evolving' > (e.g. RPC) or 'limited-private unstable'. These appear in developer Javadoc. In the majority cases I agree, except that some private interfaces need to be marked as stable: protocols like client protocol should be marked as limited-private and stable . They impact interoperation and in the future they will impact rolling upgrades. Again see the HADOOP-5073 's FAQ. > * Anything that is private implementation code should be marked as 'private unstable'. These appear in developer Javadoc. Agreed. > * If an API is not annotated then it defaults to the usual Java visibility and Apache HADOOP stability ( http://wiki.apache.org/hadoop/Roadmap ) rules. This is different than what was stated in HADOOP-5073 . It should default to audience-private unstable.
          Hide
          Tom White added a comment -

          I've gone through the document in HADOOP-6289 and added annotations that describe the differences with the patch for this issue (I've updated the patch here in many cases to make the two agree). However, there are a number of places where we need further discussion. The lines I've added comments on begin [tw].

          Show
          Tom White added a comment - I've gone through the document in HADOOP-6289 and added annotations that describe the differences with the patch for this issue (I've updated the patch here in many cases to make the two agree). However, there are a number of places where we need further discussion. The lines I've added comments on begin [tw] .
          Hide
          Tom White added a comment -

          Sanjay, I did see your proposal in HADOOP-5073, which informed my choice of classifications in this issue, but I missed the linked issue (HADOOP-6289). I think we are broadly in agreement on the actual classifications - I think going through these concrete differences to reconcile them would be the best way to get this resolved. I'll post a list of differences soon.

          Show
          Tom White added a comment - Sanjay, I did see your proposal in HADOOP-5073 , which informed my choice of classifications in this issue, but I missed the linked issue ( HADOOP-6289 ). I think we are broadly in agreement on the actual classifications - I think going through these concrete differences to reconcile them would be the best way to get this resolved. I'll post a list of differences soon.
          Hide
          Sanjay Radia added a comment -

          Tom, based on my first glance at the list of criteria that you state above and the patch you have attached it appears that you have also ignored the original jira Hadoop-5073 and the very concrete proposal in that Jira:
          https://issues.apache.org/jira/browse/HADOOP-5073?focusedCommentId=12759723&page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel#action_12759723
          HADOOP-6289, which was linked to Hadoop-5073, used that concrete proposal as the starting point.
          (BTW HADOOP-6668 should have been linked to 5073 - if it had been, the duplication might have been detected.)

          I suggest that we first come to an agreement on the criteria.
          I will shortly post a my feedback on the criteria you have state above and contrast this against those used on the proposal in Hadoop-6289.

          Show
          Sanjay Radia added a comment - Tom, based on my first glance at the list of criteria that you state above and the patch you have attached it appears that you have also ignored the original jira Hadoop-5073 and the very concrete proposal in that Jira: https://issues.apache.org/jira/browse/HADOOP-5073?focusedCommentId=12759723&page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel#action_12759723 HADOOP-6289 , which was linked to Hadoop-5073, used that concrete proposal as the starting point. (BTW HADOOP-6668 should have been linked to 5073 - if it had been, the duplication might have been detected.) I suggest that we first come to an agreement on the criteria. I will shortly post a my feedback on the criteria you have state above and contrast this against those used on the proposal in Hadoop-6289.
          Hide
          Hadoop QA added a comment -

          -1 overall. Here are the results of testing the latest attachment
          http://issues.apache.org/jira/secure/attachment/12443216/HADOOP-6668.patch
          against trunk revision 939510.

          +1 @author. The patch does not contain any @author tags.

          +0 tests included. The patch appears to be a documentation patch that doesn't require tests.

          +1 javadoc. The javadoc tool did not generate any warning messages.

          +1 javac. The applied patch does not increase the total number of javac compiler warnings.

          +1 findbugs. The patch does not introduce any new Findbugs warnings.

          +1 release audit. The applied patch does not increase the total number of release audit warnings.

          -1 core tests. The patch failed core unit tests.

          +1 contrib tests. The patch passed contrib unit tests.

          Test results: http://hudson.zones.apache.org/hudson/job/Hadoop-Patch-h4.grid.sp2.yahoo.net/489/testReport/
          Findbugs warnings: http://hudson.zones.apache.org/hudson/job/Hadoop-Patch-h4.grid.sp2.yahoo.net/489/artifact/trunk/build/test/findbugs/newPatchFindbugsWarnings.html
          Checkstyle results: http://hudson.zones.apache.org/hudson/job/Hadoop-Patch-h4.grid.sp2.yahoo.net/489/artifact/trunk/build/test/checkstyle-errors.html
          Console output: http://hudson.zones.apache.org/hudson/job/Hadoop-Patch-h4.grid.sp2.yahoo.net/489/console

          This message is automatically generated.

          Show
          Hadoop QA added a comment - -1 overall. Here are the results of testing the latest attachment http://issues.apache.org/jira/secure/attachment/12443216/HADOOP-6668.patch against trunk revision 939510. +1 @author. The patch does not contain any @author tags. +0 tests included. The patch appears to be a documentation patch that doesn't require tests. +1 javadoc. The javadoc tool did not generate any warning messages. +1 javac. The applied patch does not increase the total number of javac compiler warnings. +1 findbugs. The patch does not introduce any new Findbugs warnings. +1 release audit. The applied patch does not increase the total number of release audit warnings. -1 core tests. The patch failed core unit tests. +1 contrib tests. The patch passed contrib unit tests. Test results: http://hudson.zones.apache.org/hudson/job/Hadoop-Patch-h4.grid.sp2.yahoo.net/489/testReport/ Findbugs warnings: http://hudson.zones.apache.org/hudson/job/Hadoop-Patch-h4.grid.sp2.yahoo.net/489/artifact/trunk/build/test/findbugs/newPatchFindbugsWarnings.html Checkstyle results: http://hudson.zones.apache.org/hudson/job/Hadoop-Patch-h4.grid.sp2.yahoo.net/489/artifact/trunk/build/test/checkstyle-errors.html Console output: http://hudson.zones.apache.org/hudson/job/Hadoop-Patch-h4.grid.sp2.yahoo.net/489/console This message is automatically generated.
          Hide
          Tom White added a comment -

          Thanks, Sanjay. Yes, it's unfortunate that we missed each others JIRAs! I'll have a look at the differences tomorrow, so we can get this into 0.21 in the coming days.

          Show
          Tom White added a comment - Thanks, Sanjay. Yes, it's unfortunate that we missed each others JIRAs! I'll have a look at the differences tomorrow, so we can get this into 0.21 in the coming days.
          Hide
          Sanjay Radia added a comment -

          Hmm I and also suresh had missed this Jira. There was duplicate working going on in hadoop-6289 which was filed
          in sept 09.
          There is a proposal attached in the other one - we should sync the two.
          Suresh and I spent a fair amount of time in the analysis of the common but were waiting for feedback before generating the patch.

          Show
          Sanjay Radia added a comment - Hmm I and also suresh had missed this Jira. There was duplicate working going on in hadoop-6289 which was filed in sept 09. There is a proposal attached in the other one - we should sync the two. Suresh and I spent a fair amount of time in the analysis of the common but were waiting for feedback before generating the patch.
          Hide
          Tom White added a comment -

          Here's the javadoc and JDiff with 0.20 generated with this patch applied.

          Show
          Tom White added a comment - Here's the javadoc and JDiff with 0.20 generated with this patch applied.
          Hide
          Tom White added a comment -

          The old patch went stale - here's a new one.

          Show
          Tom White added a comment - The old patch went stale - here's a new one.
          Hide
          Hadoop QA added a comment -

          +1 overall. Here are the results of testing the latest attachment
          http://issues.apache.org/jira/secure/attachment/12443010/HADOOP-6668.patch
          against trunk revision 938590.

          +1 @author. The patch does not contain any @author tags.

          +0 tests included. The patch appears to be a documentation patch that doesn't require tests.

          +1 javadoc. The javadoc tool did not generate any warning messages.

          +1 javac. The applied patch does not increase the total number of javac compiler warnings.

          +1 findbugs. The patch does not introduce any new Findbugs warnings.

          +1 release audit. The applied patch does not increase the total number of release audit warnings.

          +1 core tests. The patch passed core unit tests.

          +1 contrib tests. The patch passed contrib unit tests.

          Test results: http://hudson.zones.apache.org/hudson/job/Hadoop-Patch-h1.grid.sp2.yahoo.net/55/testReport/
          Findbugs warnings: http://hudson.zones.apache.org/hudson/job/Hadoop-Patch-h1.grid.sp2.yahoo.net/55/artifact/trunk/build/test/findbugs/newPatchFindbugsWarnings.html
          Checkstyle results: http://hudson.zones.apache.org/hudson/job/Hadoop-Patch-h1.grid.sp2.yahoo.net/55/artifact/trunk/build/test/checkstyle-errors.html
          Console output: http://hudson.zones.apache.org/hudson/job/Hadoop-Patch-h1.grid.sp2.yahoo.net/55/console

          This message is automatically generated.

          Show
          Hadoop QA added a comment - +1 overall. Here are the results of testing the latest attachment http://issues.apache.org/jira/secure/attachment/12443010/HADOOP-6668.patch against trunk revision 938590. +1 @author. The patch does not contain any @author tags. +0 tests included. The patch appears to be a documentation patch that doesn't require tests. +1 javadoc. The javadoc tool did not generate any warning messages. +1 javac. The applied patch does not increase the total number of javac compiler warnings. +1 findbugs. The patch does not introduce any new Findbugs warnings. +1 release audit. The applied patch does not increase the total number of release audit warnings. +1 core tests. The patch passed core unit tests. +1 contrib tests. The patch passed contrib unit tests. Test results: http://hudson.zones.apache.org/hudson/job/Hadoop-Patch-h1.grid.sp2.yahoo.net/55/testReport/ Findbugs warnings: http://hudson.zones.apache.org/hudson/job/Hadoop-Patch-h1.grid.sp2.yahoo.net/55/artifact/trunk/build/test/findbugs/newPatchFindbugsWarnings.html Checkstyle results: http://hudson.zones.apache.org/hudson/job/Hadoop-Patch-h1.grid.sp2.yahoo.net/55/artifact/trunk/build/test/checkstyle-errors.html Console output: http://hudson.zones.apache.org/hudson/job/Hadoop-Patch-h1.grid.sp2.yahoo.net/55/console This message is automatically generated.
          Hide
          Tom White added a comment -

          New patch with changes for HADOOP-6677.

          Using this patch I've generated javadoc and JDiff with 0.20.

          Show
          Tom White added a comment - New patch with changes for HADOOP-6677 . Using this patch I've generated javadoc and JDiff with 0.20 .
          Hide
          Tom White added a comment -

          To help evaluate this work, I've patched JDiff to add an -incompatible option, which only shows incompatible differences. (In fact it shows some false positives too. That is, changes that are in fact compatible.) Using this patched JDiff (patch is at https://sourceforge.net/tracker/?func=detail&aid=2990626&group_id=37160&atid=419055#), I have produced a diff between 0.20 and trunk with both the patch for this issue, and HADOOP-6709 applied (I backported the annotations from HADOOP-5073, and the patch for this issue to 0.20):

          http://people.apache.org/~tomwhite/HADOOP-6668/docs/jdiff/changes.html

          There are no significant differences here, so I believe that this patch is ready to be committed. (Note that the changes in org.apache.hadoop.io will disappear once HADOOP-6698 is in.)

          I'd appreciate it if someone could review this change. Thanks.

          Show
          Tom White added a comment - To help evaluate this work, I've patched JDiff to add an -incompatible option, which only shows incompatible differences. (In fact it shows some false positives too. That is, changes that are in fact compatible.) Using this patched JDiff (patch is at https://sourceforge.net/tracker/?func=detail&aid=2990626&group_id=37160&atid=419055# ), I have produced a diff between 0.20 and trunk with both the patch for this issue, and HADOOP-6709 applied (I backported the annotations from HADOOP-5073 , and the patch for this issue to 0.20): http://people.apache.org/~tomwhite/HADOOP-6668/docs/jdiff/changes.html There are no significant differences here, so I believe that this patch is ready to be committed. (Note that the changes in org.apache.hadoop.io will disappear once HADOOP-6698 is in.) I'd appreciate it if someone could review this change. Thanks.
          Hide
          Hadoop QA added a comment -

          +1 overall. Here are the results of testing the latest attachment
          http://issues.apache.org/jira/secure/attachment/12441908/HADOOP-6668.patch
          against trunk revision 934619.

          +1 @author. The patch does not contain any @author tags.

          +0 tests included. The patch appears to be a documentation patch that doesn't require tests.

          +1 javadoc. The javadoc tool did not generate any warning messages.

          +1 javac. The applied patch does not increase the total number of javac compiler warnings.

          +1 findbugs. The patch does not introduce any new Findbugs warnings.

          +1 release audit. The applied patch does not increase the total number of release audit warnings.

          +1 core tests. The patch passed core unit tests.

          +1 contrib tests. The patch passed contrib unit tests.

          Test results: http://hudson.zones.apache.org/hudson/job/Hadoop-Patch-h4.grid.sp2.yahoo.net/463/testReport/
          Findbugs warnings: http://hudson.zones.apache.org/hudson/job/Hadoop-Patch-h4.grid.sp2.yahoo.net/463/artifact/trunk/build/test/findbugs/newPatchFindbugsWarnings.html
          Checkstyle results: http://hudson.zones.apache.org/hudson/job/Hadoop-Patch-h4.grid.sp2.yahoo.net/463/artifact/trunk/build/test/checkstyle-errors.html
          Console output: http://hudson.zones.apache.org/hudson/job/Hadoop-Patch-h4.grid.sp2.yahoo.net/463/console

          This message is automatically generated.

          Show
          Hadoop QA added a comment - +1 overall. Here are the results of testing the latest attachment http://issues.apache.org/jira/secure/attachment/12441908/HADOOP-6668.patch against trunk revision 934619. +1 @author. The patch does not contain any @author tags. +0 tests included. The patch appears to be a documentation patch that doesn't require tests. +1 javadoc. The javadoc tool did not generate any warning messages. +1 javac. The applied patch does not increase the total number of javac compiler warnings. +1 findbugs. The patch does not introduce any new Findbugs warnings. +1 release audit. The applied patch does not increase the total number of release audit warnings. +1 core tests. The patch passed core unit tests. +1 contrib tests. The patch passed contrib unit tests. Test results: http://hudson.zones.apache.org/hudson/job/Hadoop-Patch-h4.grid.sp2.yahoo.net/463/testReport/ Findbugs warnings: http://hudson.zones.apache.org/hudson/job/Hadoop-Patch-h4.grid.sp2.yahoo.net/463/artifact/trunk/build/test/findbugs/newPatchFindbugsWarnings.html Checkstyle results: http://hudson.zones.apache.org/hudson/job/Hadoop-Patch-h4.grid.sp2.yahoo.net/463/artifact/trunk/build/test/checkstyle-errors.html Console output: http://hudson.zones.apache.org/hudson/job/Hadoop-Patch-h4.grid.sp2.yahoo.net/463/console This message is automatically generated.
          Hide
          Tom White added a comment -

          New patch which makes classes in io.compress Evolving (not Stable) since several classes made incompatible changes to constructors between 0.20 and trunk (to be 0.21) by throwing IOException (in 0.20 they don't throw an exception). Similarly util.GenericOptionsParser is now Evolving since its constructors now throw IOException.

          I've marked the security classes as Evolving too. It's likely some could be marked as private or limited-private, but I didn't know which ones.

          Show
          Tom White added a comment - New patch which makes classes in io.compress Evolving (not Stable) since several classes made incompatible changes to constructors between 0.20 and trunk (to be 0.21) by throwing IOException (in 0.20 they don't throw an exception). Similarly util.GenericOptionsParser is now Evolving since its constructors now throw IOException. I've marked the security classes as Evolving too. It's likely some could be marked as private or limited-private, but I didn't know which ones.
          Hide
          Hadoop QA added a comment -

          +1 overall. Here are the results of testing the latest attachment
          http://issues.apache.org/jira/secure/attachment/12441236/HADOOP-6668.patch
          against trunk revision 932115.

          +1 @author. The patch does not contain any @author tags.

          +0 tests included. The patch appears to be a documentation patch that doesn't require tests.

          +1 javadoc. The javadoc tool did not generate any warning messages.

          +1 javac. The applied patch does not increase the total number of javac compiler warnings.

          +1 findbugs. The patch does not introduce any new Findbugs warnings.

          +1 release audit. The applied patch does not increase the total number of release audit warnings.

          +1 core tests. The patch passed core unit tests.

          +1 contrib tests. The patch passed contrib unit tests.

          Test results: http://hudson.zones.apache.org/hudson/job/Hadoop-Patch-h4.grid.sp2.yahoo.net/451/testReport/
          Findbugs warnings: http://hudson.zones.apache.org/hudson/job/Hadoop-Patch-h4.grid.sp2.yahoo.net/451/artifact/trunk/build/test/findbugs/newPatchFindbugsWarnings.html
          Checkstyle results: http://hudson.zones.apache.org/hudson/job/Hadoop-Patch-h4.grid.sp2.yahoo.net/451/artifact/trunk/build/test/checkstyle-errors.html
          Console output: http://hudson.zones.apache.org/hudson/job/Hadoop-Patch-h4.grid.sp2.yahoo.net/451/console

          This message is automatically generated.

          Show
          Hadoop QA added a comment - +1 overall. Here are the results of testing the latest attachment http://issues.apache.org/jira/secure/attachment/12441236/HADOOP-6668.patch against trunk revision 932115. +1 @author. The patch does not contain any @author tags. +0 tests included. The patch appears to be a documentation patch that doesn't require tests. +1 javadoc. The javadoc tool did not generate any warning messages. +1 javac. The applied patch does not increase the total number of javac compiler warnings. +1 findbugs. The patch does not introduce any new Findbugs warnings. +1 release audit. The applied patch does not increase the total number of release audit warnings. +1 core tests. The patch passed core unit tests. +1 contrib tests. The patch passed contrib unit tests. Test results: http://hudson.zones.apache.org/hudson/job/Hadoop-Patch-h4.grid.sp2.yahoo.net/451/testReport/ Findbugs warnings: http://hudson.zones.apache.org/hudson/job/Hadoop-Patch-h4.grid.sp2.yahoo.net/451/artifact/trunk/build/test/findbugs/newPatchFindbugsWarnings.html Checkstyle results: http://hudson.zones.apache.org/hudson/job/Hadoop-Patch-h4.grid.sp2.yahoo.net/451/artifact/trunk/build/test/checkstyle-errors.html Console output: http://hudson.zones.apache.org/hudson/job/Hadoop-Patch-h4.grid.sp2.yahoo.net/451/console This message is automatically generated.
          Hide
          Tom White added a comment -

          New patch with a few more annotations.

          With this patch and a similar backport to the 0.20 branch, the differences with the 0.20 (generated by HADOOP-6658 and a modified JDiff that doesn't show compatible changes) look like this: http://people.apache.org/~tomwhite/HADOOP-6668/docs/jdiff/changes.html.

          These are a mixture of compatible changes (that look incompatible to JDiff, e.g. the ones in org.apache.hadoop.io.compress), or elements that were deprecated in 0.20 and have been removed in trunk, e.g. FileSystem#delete(Path). Do we need to restore these deprecated methods in trunk for compatibilities sake in the forthcoming 0.21 release?

          Show
          Tom White added a comment - New patch with a few more annotations. With this patch and a similar backport to the 0.20 branch, the differences with the 0.20 (generated by HADOOP-6658 and a modified JDiff that doesn't show compatible changes) look like this: http://people.apache.org/~tomwhite/HADOOP-6668/docs/jdiff/changes.html . These are a mixture of compatible changes (that look incompatible to JDiff, e.g. the ones in org.apache.hadoop.io.compress), or elements that were deprecated in 0.20 and have been removed in trunk, e.g. FileSystem#delete(Path). Do we need to restore these deprecated methods in trunk for compatibilities sake in the forthcoming 0.21 release?
          Hide
          Tom White added a comment -

          This is harder to achieve. The standard doclet is not every extensible, so we'd probably have to copy then modify it; see http://java.sun.com/j2se/javadoc/faq/index.html#sourcedoclet. I've created HADOOP-6675 to track this.

          Show
          Tom White added a comment - This is harder to achieve. The standard doclet is not every extensible, so we'd probably have to copy then modify it; see http://java.sun.com/j2se/javadoc/faq/index.html#sourcedoclet . I've created HADOOP-6675 to track this.
          Hide
          Doug Cutting added a comment -

          > The annotations do appear in javadoc [ ... ]

          I think they should be more prominent, like deprecation. A bolded "Evolving" as the first word of the javadoc comment, that appears on the package summary page, like "Deprecated" would be ideal. We want folks to be very aware that these APIs may change, so that, when they do, they won't scream at us for breaking things.

          Show
          Doug Cutting added a comment - > The annotations do appear in javadoc [ ... ] I think they should be more prominent, like deprecation. A bolded "Evolving" as the first word of the javadoc comment, that appears on the package summary page, like "Deprecated" would be ideal. We want folks to be very aware that these APIs may change, so that, when they do, they won't scream at us for breaking things.
          Hide
          Tom White added a comment -
          Show
          Tom White added a comment - The annotations do appear in javadoc, see http://people.apache.org/~tomwhite/HADOOP-6668/docs/api/org/apache/hadoop/io/serializer/SerializationBase.html , for example.
          Hide
          Doug Cutting added a comment -

          > I marked all of the serializer classes as public evolving in the patch.

          Great!

          It would be good if the evolving status was visible in javadoc, similar to deprecation. Should I file an issue for that? I'd be happy to help out with it, if you like.

          Show
          Doug Cutting added a comment - > I marked all of the serializer classes as public evolving in the patch. Great! It would be good if the evolving status was visible in javadoc, similar to deprecation. Should I file an issue for that? I'd be happy to help out with it, if you like.
          Hide
          Tom White added a comment -

          > The io.serializer.avro package should probably be public evolving. Perhaps all of io.serialization should be?

          I agree - I marked all of the serializer classes as public evolving in the patch.

          Show
          Tom White added a comment - > The io.serializer.avro package should probably be public evolving. Perhaps all of io.serialization should be? I agree - I marked all of the serializer classes as public evolving in the patch.
          Hide
          Hadoop QA added a comment -

          +1 overall. Here are the results of testing the latest attachment
          http://issues.apache.org/jira/secure/attachment/12440548/HADOOP-6668.patch
          against trunk revision 930096.

          +1 @author. The patch does not contain any @author tags.

          +0 tests included. The patch appears to be a documentation patch that doesn't require tests.

          +1 javadoc. The javadoc tool did not generate any warning messages.

          +1 javac. The applied patch does not increase the total number of javac compiler warnings.

          +1 findbugs. The patch does not introduce any new Findbugs warnings.

          +1 release audit. The applied patch does not increase the total number of release audit warnings.

          +1 core tests. The patch passed core unit tests.

          +1 contrib tests. The patch passed contrib unit tests.

          Test results: http://hudson.zones.apache.org/hudson/job/Hadoop-Patch-h4.grid.sp2.yahoo.net/439/testReport/
          Findbugs warnings: http://hudson.zones.apache.org/hudson/job/Hadoop-Patch-h4.grid.sp2.yahoo.net/439/artifact/trunk/build/test/findbugs/newPatchFindbugsWarnings.html
          Checkstyle results: http://hudson.zones.apache.org/hudson/job/Hadoop-Patch-h4.grid.sp2.yahoo.net/439/artifact/trunk/build/test/checkstyle-errors.html
          Console output: http://hudson.zones.apache.org/hudson/job/Hadoop-Patch-h4.grid.sp2.yahoo.net/439/console

          This message is automatically generated.

          Show
          Hadoop QA added a comment - +1 overall. Here are the results of testing the latest attachment http://issues.apache.org/jira/secure/attachment/12440548/HADOOP-6668.patch against trunk revision 930096. +1 @author. The patch does not contain any @author tags. +0 tests included. The patch appears to be a documentation patch that doesn't require tests. +1 javadoc. The javadoc tool did not generate any warning messages. +1 javac. The applied patch does not increase the total number of javac compiler warnings. +1 findbugs. The patch does not introduce any new Findbugs warnings. +1 release audit. The applied patch does not increase the total number of release audit warnings. +1 core tests. The patch passed core unit tests. +1 contrib tests. The patch passed contrib unit tests. Test results: http://hudson.zones.apache.org/hudson/job/Hadoop-Patch-h4.grid.sp2.yahoo.net/439/testReport/ Findbugs warnings: http://hudson.zones.apache.org/hudson/job/Hadoop-Patch-h4.grid.sp2.yahoo.net/439/artifact/trunk/build/test/findbugs/newPatchFindbugsWarnings.html Checkstyle results: http://hudson.zones.apache.org/hudson/job/Hadoop-Patch-h4.grid.sp2.yahoo.net/439/artifact/trunk/build/test/checkstyle-errors.html Console output: http://hudson.zones.apache.org/hudson/job/Hadoop-Patch-h4.grid.sp2.yahoo.net/439/console This message is automatically generated.
          Hide
          Doug Cutting added a comment -

          The io.serializer.avro package should probably be public evolving. Perhaps all of io.serialization should be?

          Show
          Doug Cutting added a comment - The io.serializer.avro package should probably be public evolving. Perhaps all of io.serialization should be?
          Hide
          Tom White added a comment -

          With this patch (and HADOOP-6658) the public Javadoc would look like this: http://people.apache.org/~tomwhite/HADOOP-6668/docs/api/

          Show
          Tom White added a comment - With this patch (and HADOOP-6658 ) the public Javadoc would look like this: http://people.apache.org/~tomwhite/HADOOP-6668/docs/api/
          Hide
          Tom White added a comment -

          I have had a first attempt at applying annotations to the common codebase. They are summarized in this table; consult the patch for finer-grained detail.

          Package org.apache.hadoop. Visibility & Stability Notes
          classification public stable Should we mark them evolving?
          conf public stable  
          fs public stable filesystem implementations are public stable, but any internal classes are private unstable
          fs.shell private unstable  
          http limited-private unstable  
          io public stable  
          io.compress public stable  
          io.compress.bzip2 private unstable  
          io.compress.zlib private unstable  
          io.file.tfile public evolving  
          io.retry limited-private evolving  
          io.serializer* public evolving  
          ipc* limited-private evolving  
          metrics* public evolving  
          net mixture of public evolving and limited-private unstable  
          record* deprecated  
          security* mixture of public/limited-private evolving?  
          util public evolving GenericOptionsParser, ReflectionUtils, Progressable, Tool, ToolRunner are public stable
          util.bloom public stable  
          util.hash public stable  

          I haven't annotated the security APIs yet - if anyone has suggestions for these, I'd be grateful.

          I'm wondering whether to mark deprecated elements as 'evolving' (if they were deprecated in 0.19 or earlier), so that they can be removed sooner than the next major release. We wouldn't necessarily do this, but it might be wise to have the option.

          Here are some guidelines that I used in applying the annotations. Selecting the annotations to apply is a judgement call, so I'd appreciate feedback both on the general rules and the particular annotations I have applied.

          • Public user APIs should be marked as 'public stable'. These appear in public Javadoc.
          • APIs that are exposed to the user but not coded against directly should be marked as 'public evolving' (e.g. the metrics API, which is used in configuration). These appear in public Javadoc.
          • Non-user facing APIs that are used within Hadoop subprojects should be marked as 'limited-private evolving' (e.g. RPC) or 'limited-private unstable'. These appear in developer Javadoc.
          • Anything that is private implementation code should be marked as 'private unstable'. These appear in developer Javadoc.
          • If an API is not annotated then it defaults to the usual Java visibility and Apache Hadoop stability (http://wiki.apache.org/hadoop/Roadmap) rules.
          Show
          Tom White added a comment - I have had a first attempt at applying annotations to the common codebase. They are summarized in this table; consult the patch for finer-grained detail. Package org.apache.hadoop. Visibility & Stability Notes classification public stable Should we mark them evolving? conf public stable   fs public stable filesystem implementations are public stable, but any internal classes are private unstable fs.shell private unstable   http limited-private unstable   io public stable   io.compress public stable   io.compress.bzip2 private unstable   io.compress.zlib private unstable   io.file.tfile public evolving   io.retry limited-private evolving   io.serializer* public evolving   ipc* limited-private evolving   metrics* public evolving   net mixture of public evolving and limited-private unstable   record* deprecated   security* mixture of public/limited-private evolving?   util public evolving GenericOptionsParser, ReflectionUtils, Progressable, Tool, ToolRunner are public stable util.bloom public stable   util.hash public stable   I haven't annotated the security APIs yet - if anyone has suggestions for these, I'd be grateful. I'm wondering whether to mark deprecated elements as 'evolving' (if they were deprecated in 0.19 or earlier), so that they can be removed sooner than the next major release. We wouldn't necessarily do this, but it might be wise to have the option. Here are some guidelines that I used in applying the annotations. Selecting the annotations to apply is a judgement call, so I'd appreciate feedback both on the general rules and the particular annotations I have applied. Public user APIs should be marked as 'public stable'. These appear in public Javadoc. APIs that are exposed to the user but not coded against directly should be marked as 'public evolving' (e.g. the metrics API, which is used in configuration). These appear in public Javadoc. Non-user facing APIs that are used within Hadoop subprojects should be marked as 'limited-private evolving' (e.g. RPC) or 'limited-private unstable'. These appear in developer Javadoc. Anything that is private implementation code should be marked as 'private unstable'. These appear in developer Javadoc. If an API is not annotated then it defaults to the usual Java visibility and Apache Hadoop stability ( http://wiki.apache.org/hadoop/Roadmap ) rules.

            People

            • Assignee:
              Tom White
              Reporter:
              Tom White
            • Votes:
              0 Vote for this issue
              Watchers:
              7 Start watching this issue

              Dates

              • Created:
                Updated:
                Resolved:

                Development