Hadoop YARN
  1. Hadoop YARN
  2. YARN-142

[Umbrella] Cleanup YARN APIs w.r.t exceptions

    Details

    • Type: Task Task
    • Status: Closed
    • Priority: Blocker Blocker
    • Resolution: Fixed
    • Affects Version/s: 0.23.3, 2.0.0-alpha
    • Fix Version/s: 2.1.0-beta
    • Component/s: None
    • Labels:
      None

      Description

      Ref: MAPREDUCE-4067

      All YARN APIs currently throw YarnRemoteException.
      1) This cannot be extended in it's current form.
      2) The RPC layer can throw IOExceptions. These end up showing up as UndeclaredThrowableExceptions.

      1. YARN-142.1.patch
        167 kB
        Xuan Gong
      2. YARN-142.2.patch
        435 kB
        Xuan Gong
      3. YARN-142.3.patch
        207 kB
        Xuan Gong
      4. YARN-142.4.patch
        220 kB
        Xuan Gong

        Issue Links

          Activity

          Siddharth Seth created issue -
          Siddharth Seth made changes -
          Field Original Value New Value
          Link This issue blocks MAPREDUCE-4067 [ MAPREDUCE-4067 ]
          Arun C Murthy made changes -
          Assignee Siddharth Seth [ sseth ] Xuan Gong [ xgong ]
          Hide
          Xuan Gong added a comment -

          sudo find . -name '*.java' -print0 | xargs -0 grep -l 'YarnRemoteException'
          -----All the java files which contains the YarnRemoteException and need to replace with IOException

          ./hadoop-yarn/hadoop-yarn-api/src/main/java/org/apache/hadoop/yarn/api/AMRMProtocol.java
          ./hadoop-yarn/hadoop-yarn-api/src/main/java/org/apache/hadoop/yarn/api/ClientRMProtocol.java
          ./hadoop-yarn/hadoop-yarn-api/src/main/java/org/apache/hadoop/yarn/api/ContainerManager.java
          ./hadoop-yarn/hadoop-yarn-api/src/main/java/org/apache/hadoop/yarn/api/RMAdminProtocol.java
          ./hadoop-yarn/hadoop-yarn-api/src/main/java/org/apache/hadoop/yarn/exceptions/impl/pb/YarnRemoteExceptionPBImpl.java
          ./hadoop-yarn/hadoop-yarn-api/src/main/java/org/apache/hadoop/yarn/exceptions/YarnRemoteException.java
          ./hadoop-yarn/hadoop-yarn-api/target/generated-sources/proto/org/apache/hadoop/yarn/proto/YarnProtos.java
          ./hadoop-yarn/hadoop-yarn-applications/hadoop-yarn-applications-distributedshell/src/main/java/org/apache/hadoop/yarn/applications/distributedshell/ApplicationMaster.java
          ./hadoop-yarn/hadoop-yarn-applications/hadoop-yarn-applications-distributedshell/src/main/java/org/apache/hadoop/yarn/applications/distributedshell/Client.java
          ./hadoop-yarn/hadoop-yarn-applications/hadoop-yarn-applications-unmanaged-am-launcher/src/main/java/org/apache/hadoop/yarn/applications/unmanagedamlauncher/UnmanagedAMLauncher.java
          ./hadoop-yarn/hadoop-yarn-client/src/main/java/org/apache/hadoop/yarn/client/cli/ApplicationCLI.java
          ./hadoop-yarn/hadoop-yarn-client/src/main/java/org/apache/hadoop/yarn/client/cli/NodeCLI.java
          ./hadoop-yarn/hadoop-yarn-client/src/main/java/org/apache/hadoop/yarn/client/YarnClient.java
          ./hadoop-yarn/hadoop-yarn-client/src/main/java/org/apache/hadoop/yarn/client/YarnClientImpl.java
          ./hadoop-yarn/hadoop-yarn-common/src/main/java/org/apache/hadoop/yarn/api/impl/pb/client/AMRMProtocolPBClientImpl.java
          ./hadoop-yarn/hadoop-yarn-common/src/main/java/org/apache/hadoop/yarn/api/impl/pb/client/ClientRMProtocolPBClientImpl.java
          ./hadoop-yarn/hadoop-yarn-common/src/main/java/org/apache/hadoop/yarn/api/impl/pb/client/ContainerManagerPBClientImpl.java
          ./hadoop-yarn/hadoop-yarn-common/src/main/java/org/apache/hadoop/yarn/api/impl/pb/client/RMAdminProtocolPBClientImpl.java
          ./hadoop-yarn/hadoop-yarn-common/src/main/java/org/apache/hadoop/yarn/api/impl/pb/service/AMRMProtocolPBServiceImpl.java
          ./hadoop-yarn/hadoop-yarn-common/src/main/java/org/apache/hadoop/yarn/api/impl/pb/service/ClientRMProtocolPBServiceImpl.java
          ./hadoop-yarn/hadoop-yarn-common/src/main/java/org/apache/hadoop/yarn/api/impl/pb/service/ContainerManagerPBServiceImpl.java
          ./hadoop-yarn/hadoop-yarn-common/src/main/java/org/apache/hadoop/yarn/api/impl/pb/service/RMAdminProtocolPBServiceImpl.java
          ./hadoop-yarn/hadoop-yarn-common/src/main/java/org/apache/hadoop/yarn/factories/impl/pb/YarnRemoteExceptionFactoryPBImpl.java
          ./hadoop-yarn/hadoop-yarn-common/src/main/java/org/apache/hadoop/yarn/factories/YarnRemoteExceptionFactory.java
          ./hadoop-yarn/hadoop-yarn-common/src/main/java/org/apache/hadoop/yarn/factory/providers/YarnRemoteExceptionFactoryProvider.java
          ./hadoop-yarn/hadoop-yarn-common/src/main/java/org/apache/hadoop/yarn/ipc/RPCUtil.java
          ./hadoop-yarn/hadoop-yarn-common/src/test/java/org/apache/hadoop/yarn/TestContainerLaunchRPC.java
          ./hadoop-yarn/hadoop-yarn-common/src/test/java/org/apache/hadoop/yarn/TestRPC.java
          ./hadoop-yarn/hadoop-yarn-common/src/test/java/org/apache/hadoop/yarn/TestRPCFactories.java
          ./hadoop-yarn/hadoop-yarn-common/target/generated-sources/proto/org/apache/hadoop/yarn/ipc/RpcProtos.java
          ./hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-common/src/main/java/org/apache/hadoop/yarn/server/api/impl/pb/client/ResourceTrackerPBClientImpl.java
          ./hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-common/src/main/java/org/apache/hadoop/yarn/server/api/impl/pb/service/ResourceTrackerPBServiceImpl.java
          ./hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-common/src/main/java/org/apache/hadoop/yarn/server/api/ResourceTracker.java
          ./hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-common/src/test/java/org/apache/hadoop/yarn/TestRPCFactories.java
          ./hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/main/java/org/apache/hadoop/yarn/server/nodemanager/api/impl/pb/client/LocalizationProtocolPBClientImpl.java
          ./hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/main/java/org/apache/hadoop/yarn/server/nodemanager/api/impl/pb/service/LocalizationProtocolPBServiceImpl.java
          ./hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/main/java/org/apache/hadoop/yarn/server/nodemanager/api/LocalizationProtocol.java
          ./hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/main/java/org/apache/hadoop/yarn/server/nodemanager/api/protocolrecords/impl/pb/LocalResourceStatusPBImpl.java
          ./hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/main/java/org/apache/hadoop/yarn/server/nodemanager/api/protocolrecords/LocalResourceStatus.java
          ./hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/main/java/org/apache/hadoop/yarn/server/nodemanager/containermanager/ContainerManagerImpl.java
          ./hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/main/java/org/apache/hadoop/yarn/server/nodemanager/containermanager/localizer/ContainerLocalizer.java
          ./hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/main/java/org/apache/hadoop/yarn/server/nodemanager/NodeStatusUpdaterImpl.java
          ./hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/test/java/org/apache/hadoop/yarn/server/nodemanager/containermanager/BaseContainerManagerTest.java
          ./hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/test/java/org/apache/hadoop/yarn/server/nodemanager/containermanager/localizer/MockLocalResourceStatus.java
          ./hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/test/java/org/apache/hadoop/yarn/server/nodemanager/containermanager/TestContainerManager.java
          ./hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/test/java/org/apache/hadoop/yarn/server/nodemanager/LocalRMInterface.java
          ./hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/test/java/org/apache/hadoop/yarn/server/nodemanager/MockNodeStatusUpdater.java
          ./hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/test/java/org/apache/hadoop/yarn/server/nodemanager/TestNodeStatusUpdater.java
          ./hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/target/generated-sources/proto/org/apache/hadoop/yarn/proto/YarnServerNodemanagerServiceProtos.java
          ./hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/AdminService.java
          ./hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/ApplicationMasterService.java
          ./hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/ClientRMService.java
          ./hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/ResourceTrackerService.java
          ./hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/test/java/org/apache/hadoop/yarn/server/resourcemanager/Application.java
          ./hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/test/java/org/apache/hadoop/yarn/server/resourcemanager/NodeManager.java
          ./hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/test/java/org/apache/hadoop/yarn/server/resourcemanager/security/TestClientTokens.java
          ./hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/test/java/org/apache/hadoop/yarn/server/resourcemanager/TestAMAuthorization.java
          ./hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/test/java/org/apache/hadoop/yarn/server/resourcemanager/TestApplicationACLs.java
          ./hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/test/java/org/apache/hadoop/yarn/server/resourcemanager/TestApplicationMasterLauncher.java
          ./hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/test/java/org/apache/hadoop/yarn/server/resourcemanager/TestClientRMService.java
          ./hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-tests/src/test/java/org/apache/hadoop/yarn/server/MiniYARNCluster.java
          ./hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-tests/src/test/java/org/apache/hadoop/yarn/server/TestContainerManagerSecurity.java
          ./hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-web-proxy/src/main/java/org/apache/hadoop/yarn/server/webproxy/AppReportFetcher.java

          Show
          Xuan Gong added a comment - sudo find . -name '*.java' -print0 | xargs -0 grep -l 'YarnRemoteException' -----All the java files which contains the YarnRemoteException and need to replace with IOException ./hadoop-yarn/hadoop-yarn-api/src/main/java/org/apache/hadoop/yarn/api/AMRMProtocol.java ./hadoop-yarn/hadoop-yarn-api/src/main/java/org/apache/hadoop/yarn/api/ClientRMProtocol.java ./hadoop-yarn/hadoop-yarn-api/src/main/java/org/apache/hadoop/yarn/api/ContainerManager.java ./hadoop-yarn/hadoop-yarn-api/src/main/java/org/apache/hadoop/yarn/api/RMAdminProtocol.java ./hadoop-yarn/hadoop-yarn-api/src/main/java/org/apache/hadoop/yarn/exceptions/impl/pb/YarnRemoteExceptionPBImpl.java ./hadoop-yarn/hadoop-yarn-api/src/main/java/org/apache/hadoop/yarn/exceptions/YarnRemoteException.java ./hadoop-yarn/hadoop-yarn-api/target/generated-sources/proto/org/apache/hadoop/yarn/proto/YarnProtos.java ./hadoop-yarn/hadoop-yarn-applications/hadoop-yarn-applications-distributedshell/src/main/java/org/apache/hadoop/yarn/applications/distributedshell/ApplicationMaster.java ./hadoop-yarn/hadoop-yarn-applications/hadoop-yarn-applications-distributedshell/src/main/java/org/apache/hadoop/yarn/applications/distributedshell/Client.java ./hadoop-yarn/hadoop-yarn-applications/hadoop-yarn-applications-unmanaged-am-launcher/src/main/java/org/apache/hadoop/yarn/applications/unmanagedamlauncher/UnmanagedAMLauncher.java ./hadoop-yarn/hadoop-yarn-client/src/main/java/org/apache/hadoop/yarn/client/cli/ApplicationCLI.java ./hadoop-yarn/hadoop-yarn-client/src/main/java/org/apache/hadoop/yarn/client/cli/NodeCLI.java ./hadoop-yarn/hadoop-yarn-client/src/main/java/org/apache/hadoop/yarn/client/YarnClient.java ./hadoop-yarn/hadoop-yarn-client/src/main/java/org/apache/hadoop/yarn/client/YarnClientImpl.java ./hadoop-yarn/hadoop-yarn-common/src/main/java/org/apache/hadoop/yarn/api/impl/pb/client/AMRMProtocolPBClientImpl.java ./hadoop-yarn/hadoop-yarn-common/src/main/java/org/apache/hadoop/yarn/api/impl/pb/client/ClientRMProtocolPBClientImpl.java ./hadoop-yarn/hadoop-yarn-common/src/main/java/org/apache/hadoop/yarn/api/impl/pb/client/ContainerManagerPBClientImpl.java ./hadoop-yarn/hadoop-yarn-common/src/main/java/org/apache/hadoop/yarn/api/impl/pb/client/RMAdminProtocolPBClientImpl.java ./hadoop-yarn/hadoop-yarn-common/src/main/java/org/apache/hadoop/yarn/api/impl/pb/service/AMRMProtocolPBServiceImpl.java ./hadoop-yarn/hadoop-yarn-common/src/main/java/org/apache/hadoop/yarn/api/impl/pb/service/ClientRMProtocolPBServiceImpl.java ./hadoop-yarn/hadoop-yarn-common/src/main/java/org/apache/hadoop/yarn/api/impl/pb/service/ContainerManagerPBServiceImpl.java ./hadoop-yarn/hadoop-yarn-common/src/main/java/org/apache/hadoop/yarn/api/impl/pb/service/RMAdminProtocolPBServiceImpl.java ./hadoop-yarn/hadoop-yarn-common/src/main/java/org/apache/hadoop/yarn/factories/impl/pb/YarnRemoteExceptionFactoryPBImpl.java ./hadoop-yarn/hadoop-yarn-common/src/main/java/org/apache/hadoop/yarn/factories/YarnRemoteExceptionFactory.java ./hadoop-yarn/hadoop-yarn-common/src/main/java/org/apache/hadoop/yarn/factory/providers/YarnRemoteExceptionFactoryProvider.java ./hadoop-yarn/hadoop-yarn-common/src/main/java/org/apache/hadoop/yarn/ipc/RPCUtil.java ./hadoop-yarn/hadoop-yarn-common/src/test/java/org/apache/hadoop/yarn/TestContainerLaunchRPC.java ./hadoop-yarn/hadoop-yarn-common/src/test/java/org/apache/hadoop/yarn/TestRPC.java ./hadoop-yarn/hadoop-yarn-common/src/test/java/org/apache/hadoop/yarn/TestRPCFactories.java ./hadoop-yarn/hadoop-yarn-common/target/generated-sources/proto/org/apache/hadoop/yarn/ipc/RpcProtos.java ./hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-common/src/main/java/org/apache/hadoop/yarn/server/api/impl/pb/client/ResourceTrackerPBClientImpl.java ./hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-common/src/main/java/org/apache/hadoop/yarn/server/api/impl/pb/service/ResourceTrackerPBServiceImpl.java ./hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-common/src/main/java/org/apache/hadoop/yarn/server/api/ResourceTracker.java ./hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-common/src/test/java/org/apache/hadoop/yarn/TestRPCFactories.java ./hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/main/java/org/apache/hadoop/yarn/server/nodemanager/api/impl/pb/client/LocalizationProtocolPBClientImpl.java ./hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/main/java/org/apache/hadoop/yarn/server/nodemanager/api/impl/pb/service/LocalizationProtocolPBServiceImpl.java ./hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/main/java/org/apache/hadoop/yarn/server/nodemanager/api/LocalizationProtocol.java ./hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/main/java/org/apache/hadoop/yarn/server/nodemanager/api/protocolrecords/impl/pb/LocalResourceStatusPBImpl.java ./hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/main/java/org/apache/hadoop/yarn/server/nodemanager/api/protocolrecords/LocalResourceStatus.java ./hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/main/java/org/apache/hadoop/yarn/server/nodemanager/containermanager/ContainerManagerImpl.java ./hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/main/java/org/apache/hadoop/yarn/server/nodemanager/containermanager/localizer/ContainerLocalizer.java ./hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/main/java/org/apache/hadoop/yarn/server/nodemanager/NodeStatusUpdaterImpl.java ./hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/test/java/org/apache/hadoop/yarn/server/nodemanager/containermanager/BaseContainerManagerTest.java ./hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/test/java/org/apache/hadoop/yarn/server/nodemanager/containermanager/localizer/MockLocalResourceStatus.java ./hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/test/java/org/apache/hadoop/yarn/server/nodemanager/containermanager/TestContainerManager.java ./hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/test/java/org/apache/hadoop/yarn/server/nodemanager/LocalRMInterface.java ./hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/test/java/org/apache/hadoop/yarn/server/nodemanager/MockNodeStatusUpdater.java ./hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/test/java/org/apache/hadoop/yarn/server/nodemanager/TestNodeStatusUpdater.java ./hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/target/generated-sources/proto/org/apache/hadoop/yarn/proto/YarnServerNodemanagerServiceProtos.java ./hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/AdminService.java ./hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/ApplicationMasterService.java ./hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/ClientRMService.java ./hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/ResourceTrackerService.java ./hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/test/java/org/apache/hadoop/yarn/server/resourcemanager/Application.java ./hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/test/java/org/apache/hadoop/yarn/server/resourcemanager/NodeManager.java ./hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/test/java/org/apache/hadoop/yarn/server/resourcemanager/security/TestClientTokens.java ./hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/test/java/org/apache/hadoop/yarn/server/resourcemanager/TestAMAuthorization.java ./hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/test/java/org/apache/hadoop/yarn/server/resourcemanager/TestApplicationACLs.java ./hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/test/java/org/apache/hadoop/yarn/server/resourcemanager/TestApplicationMasterLauncher.java ./hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/test/java/org/apache/hadoop/yarn/server/resourcemanager/TestClientRMService.java ./hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-tests/src/test/java/org/apache/hadoop/yarn/server/MiniYARNCluster.java ./hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-tests/src/test/java/org/apache/hadoop/yarn/server/TestContainerManagerSecurity.java ./hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-web-proxy/src/main/java/org/apache/hadoop/yarn/server/webproxy/AppReportFetcher.java
          Hide
          Xuan Gong added a comment -

          Use IOException to replace the YarnRemoteException

          Show
          Xuan Gong added a comment - Use IOException to replace the YarnRemoteException
          Xuan Gong made changes -
          Attachment YARN-142.1.patch [ 12562003 ]
          Hide
          Siddharth Seth added a comment -

          Xuan, thanks for looking into this.

          https://issues.apache.org/jira/browse/MAPREDUCE-4067?focusedCommentId=13269876&page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel#comment-13269876
          Comment on the original patch from MR-4067.

          I think it'd be useful to have the APIs throw IOException and YarnRemoteException. The IOException indicating errors from the RPC layer, YarnException indicating errors from Yarn itself.
          Rename YarnRemoteException to YarnException after the patch is reviewed.
          Adding additional exception types - such as UnknownAppId, UnknownContainer, AppAlreadyUnregistered, MisconfiguredQueueException, UnknownQueueException etc can be added later - if they're of any use - other than providing debug information, which can be inferred from reasonable error messages.

          Comments on the current patch.

          • It seems to have some spillover from another patch - NodeStatusUpdaterImpl refers to backOff and a couple other new fields.
          • LocalizationProtocol..LocalResourceStatus serializes an exception from the Localizer. This can't be changed to IOException - needs to be a PBSerializedException. In the previous patch, I had moved YarnRemoteExceptionPBImpl over to SerializedException for this particular case.
          Show
          Siddharth Seth added a comment - Xuan, thanks for looking into this. https://issues.apache.org/jira/browse/MAPREDUCE-4067?focusedCommentId=13269876&page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel#comment-13269876 Comment on the original patch from MR-4067. I think it'd be useful to have the APIs throw IOException and YarnRemoteException. The IOException indicating errors from the RPC layer, YarnException indicating errors from Yarn itself. Rename YarnRemoteException to YarnException after the patch is reviewed. Adding additional exception types - such as UnknownAppId, UnknownContainer, AppAlreadyUnregistered, MisconfiguredQueueException, UnknownQueueException etc can be added later - if they're of any use - other than providing debug information, which can be inferred from reasonable error messages. Comments on the current patch. It seems to have some spillover from another patch - NodeStatusUpdaterImpl refers to backOff and a couple other new fields. LocalizationProtocol..LocalResourceStatus serializes an exception from the Localizer. This can't be changed to IOException - needs to be a PBSerializedException. In the previous patch, I had moved YarnRemoteExceptionPBImpl over to SerializedException for this particular case.
          Xuan Gong made changes -
          Attachment YARN-142.2.patch [ 12562939 ]
          Xuan Gong made changes -
          Attachment YARN-142.2.patch [ 12562939 ]
          Xuan Gong made changes -
          Attachment YARN-142.2.patch [ 12562948 ]
          Xuan Gong made changes -
          Attachment YARN-142.2.patch [ 12562948 ]
          Xuan Gong made changes -
          Attachment YARN-142.2.patch [ 12562959 ]
          Xuan Gong made changes -
          Attachment YARN-142.3.patch [ 12563856 ]
          Hide
          Xuan Gong added a comment -

          Code change on the latest trunk version

          Show
          Xuan Gong added a comment - Code change on the latest trunk version
          Xuan Gong made changes -
          Attachment YARN-142.4.patch [ 12567750 ]
          Vinod Kumar Vavilapalli made changes -
          Parent YARN-386 [ 12631203 ]
          Issue Type Bug [ 1 ] Sub-task [ 7 ]
          Hide
          Tom White added a comment -

          > I think it'd be useful to have the APIs throw IOException and YarnRemoteException. The IOException indicating errors from the RPC layer, YarnException indicating errors from Yarn itself.

          I see the latest patch has

          throws UnknownApplicationException,YarnRemoteException,IOException

          even though UnknownApplicationException is a subclass of YarnRemoteException, and YarnRemoteException is a subclass of IOException. It would be simpler to make the method signature

          throws IOException

          and draw attention to the different subclasses in the javadoc if needed.

          Show
          Tom White added a comment - > I think it'd be useful to have the APIs throw IOException and YarnRemoteException. The IOException indicating errors from the RPC layer, YarnException indicating errors from Yarn itself. I see the latest patch has throws UnknownApplicationException,YarnRemoteException,IOException even though UnknownApplicationException is a subclass of YarnRemoteException, and YarnRemoteException is a subclass of IOException. It would be simpler to make the method signature throws IOException and draw attention to the different subclasses in the javadoc if needed.
          Hide
          Siddharth Seth added a comment -

          and draw attention to the different subclasses in the javadoc if needed.

          While YarnRemoteException is a subclass of IOException, explicitly calling it out in the API isn't super useful. Was hoping to keep exceptions generated by YARN separate from RPC errors.
          The RPC layer, however, doesn not seem to handle anything other than IOException and it's derivatives. Changing YarnRemoteException to be independent of IOException would be in incompatible change at a later point - so maybe we should consider fixing the RPC layer to allow additional exceptions right now.

          Show
          Siddharth Seth added a comment - and draw attention to the different subclasses in the javadoc if needed. While YarnRemoteException is a subclass of IOException, explicitly calling it out in the API isn't super useful. Was hoping to keep exceptions generated by YARN separate from RPC errors. The RPC layer, however, doesn not seem to handle anything other than IOException and it's derivatives. Changing YarnRemoteException to be independent of IOException would be in incompatible change at a later point - so maybe we should consider fixing the RPC layer to allow additional exceptions right now.
          Hitesh Shah made changes -
          Priority Critical [ 2 ] Blocker [ 1 ]
          Siddharth Seth made changes -
          Link This issue relates HADOOP-9343 [ HADOOP-9343 ]
          Hide
          Siddharth Seth added a comment -

          After HADOOP-9343, it should be possible for YarnException to not be rooted at IOException. So all methods can declare IOException and YarnException - and have the specializations of YarnException listed in the Javadoc.

          Show
          Siddharth Seth added a comment - After HADOOP-9343 , it should be possible for YarnException to not be rooted at IOException. So all methods can declare IOException and YarnException - and have the specializations of YarnException listed in the Javadoc.
          Siddharth Seth made changes -
          Link This issue is related too YARN-625 [ YARN-625 ]
          Siddharth Seth made changes -
          Link This issue is related too YARN-627 [ YARN-627 ]
          Siddharth Seth made changes -
          Link This issue is related too YARN-628 [ YARN-628 ]
          Hide
          Xuan Gong added a comment -

          Split it into several sub-tasks:
          1. Make YarnRemoteException not be rooted at IOException
          2. change AMRMProtocol api to throw IOException and YarnRemoteException
          3. change ClientRMProtocol api to throw IOException and YarnRemoteException
          4. change ContainerManager api to throw IOException and YarnRemoteException
          5. change RMAdminProtocol api to throw IOException and YarnRemoteException

          Show
          Xuan Gong added a comment - Split it into several sub-tasks: 1. Make YarnRemoteException not be rooted at IOException 2. change AMRMProtocol api to throw IOException and YarnRemoteException 3. change ClientRMProtocol api to throw IOException and YarnRemoteException 4. change ContainerManager api to throw IOException and YarnRemoteException 5. change RMAdminProtocol api to throw IOException and YarnRemoteException
          Xuan Gong made changes -
          Link This issue requires YARN-629 [ YARN-629 ]
          Xuan Gong made changes -
          Link This issue requires YARN-632 [ YARN-632 ]
          Xuan Gong made changes -
          Link This issue requires YARN-633 [ YARN-633 ]
          Xuan Gong made changes -
          Link This issue requires YARN-630 [ YARN-630 ]
          Xuan Gong made changes -
          Link This issue requires YARN-631 [ YARN-631 ]
          Siddharth Seth made changes -
          Link This issue is related too YARN-634 [ YARN-634 ]
          Hide
          Vinod Kumar Vavilapalli added a comment -

          Editing title to be more appropriate..

          Show
          Vinod Kumar Vavilapalli added a comment - Editing title to be more appropriate..
          Vinod Kumar Vavilapalli made changes -
          Summary Change YARN APIs to throw IOException [Umbrella] Cleanup YARN APIs w.r.t exceptions
          Target Version/s 2.0.5-beta [ 12324029 ]
          Xuan Gong made changes -
          Link This issue requires YARN-635 [ YARN-635 ]
          Gavin made changes -
          Link This issue relates to HADOOP-9343 [ HADOOP-9343 ]
          Gavin made changes -
          Link This issue relates to HADOOP-9343 [ HADOOP-9343 ]
          Gavin made changes -
          Link This issue is related to YARN-625 [ YARN-625 ]
          Gavin made changes -
          Link This issue is related to YARN-625 [ YARN-625 ]
          Gavin made changes -
          Link This issue is related to YARN-627 [ YARN-627 ]
          Gavin made changes -
          Link This issue is related to YARN-627 [ YARN-627 ]
          Gavin made changes -
          Link This issue is related to YARN-628 [ YARN-628 ]
          Gavin made changes -
          Link This issue is related to YARN-628 [ YARN-628 ]
          Gavin made changes -
          Link This issue is related to YARN-634 [ YARN-634 ]
          Gavin made changes -
          Link This issue is related to YARN-634 [ YARN-634 ]
          Vinod Kumar Vavilapalli made changes -
          Parent YARN-386 [ 12631203 ]
          Issue Type Sub-task [ 7 ] Task [ 3 ]
          Vinod Kumar Vavilapalli made changes -
          Link This issue is related to YARN-386 [ YARN-386 ]
          Vinod Kumar Vavilapalli made changes -
          Link This issue is related to YARN-628 [ YARN-628 ]
          Vinod Kumar Vavilapalli made changes -
          Link This issue requires YARN-631 [ YARN-631 ]
          Vinod Kumar Vavilapalli made changes -
          Link This issue is related to YARN-634 [ YARN-634 ]
          Vinod Kumar Vavilapalli made changes -
          Link This issue requires YARN-630 [ YARN-630 ]
          Vinod Kumar Vavilapalli made changes -
          Link This issue requires YARN-633 [ YARN-633 ]
          Vinod Kumar Vavilapalli made changes -
          Link This issue requires YARN-632 [ YARN-632 ]
          Vinod Kumar Vavilapalli made changes -
          Link This issue requires YARN-629 [ YARN-629 ]
          Vinod Kumar Vavilapalli made changes -
          Link This issue is related to YARN-625 [ YARN-625 ]
          Vinod Kumar Vavilapalli made changes -
          Link This issue requires YARN-635 [ YARN-635 ]
          Hide
          Siddharth Seth added a comment -

          Resolving since all the sub-tasks are done.

          Show
          Siddharth Seth added a comment - Resolving since all the sub-tasks are done.
          Siddharth Seth made changes -
          Status Open [ 1 ] Resolved [ 5 ]
          Assignee Xuan Gong [ xgong ]
          Fix Version/s 2.1.0-beta [ 12324029 ]
          Resolution Fixed [ 1 ]
          Hide
          Arun C Murthy added a comment -

          Nice! +1!

          Show
          Arun C Murthy added a comment - Nice! +1!
          Arun C Murthy made changes -
          Status Resolved [ 5 ] Closed [ 6 ]
          Transition Time In Source Status Execution Times Last Executer Last Execution Date
          Open Open Resolved Resolved
          251d 2h 4m 1 Siddharth Seth 11/Jun/13 23:37
          Resolved Resolved Closed Closed
          76d 23h 38m 1 Arun C Murthy 27/Aug/13 23:15

            People

            • Assignee:
              Unassigned
              Reporter:
              Siddharth Seth
            • Votes:
              0 Vote for this issue
              Watchers:
              14 Start watching this issue

              Dates

              • Created:
                Updated:
                Resolved:

                Development