Uploaded image for project: 'Hadoop HDFS'
  1. Hadoop HDFS
  2. HDFS-11333

Print a user friendly error message when plugins are not found

    Details

      Description

      If NameNode is unable to find plugins (specified in dfs.namenode.plugins), it terminates abruptly with the following stack trace:

      Failed to start namenode.
      java.lang.RuntimeException: java.lang.ClassNotFoundException: Class XXX not found
        at org.apache.hadoop.conf.Configuration.getClasses(Configuration.java:2178)
        at org.apache.hadoop.conf.Configuration.getInstances(Configuration.java:2250)
        at org.apache.hadoop.hdfs.server.namenode.NameNode.startCommonServices(NameNode.java:713)
        at org.apache.hadoop.hdfs.server.namenode.NameNode.initialize(NameNode.java:691)
        at org.apache.hadoop.hdfs.server.namenode.NameNode.<init>(NameNode.java:843)
        at org.apache.hadoop.hdfs.server.namenode.NameNode.<init>(NameNode.java:822)
        at org.apache.hadoop.hdfs.server.namenode.NameNode.createNameNode(NameNode.java:1543)
        at org.apache.hadoop.hdfs.server.namenode.NameNode.main(NameNode.java:1611)

      We should catch this exception, log a warning message and let it proceed, as missing the third party library does not affect the functionality of NameNode. We caught this bug during a CDH upgrade where a third party plugin was not in the lib directory of the newer version of CDH.

      1. HDFS-11333.001.patch
        3 kB
        Wei-Chiu Chuang
      2. HDFS-11333.002.patch
        2 kB
        Wei-Chiu Chuang
      3. HDFS-11333.002.patch
        2 kB
        Wei-Chiu Chuang
      4. HDFS-11333.003.patch
        2 kB
        Wei-Chiu Chuang

        Activity

        Hide
        jojochuang Wei-Chiu Chuang added a comment -

        Attach a patch with a simple unit test.

        Show
        jojochuang Wei-Chiu Chuang added a comment - Attach a patch with a simple unit test.
        Hide
        hadoopqa Hadoop QA added a comment -
        -1 overall



        Vote Subsystem Runtime Comment
        0 reexec 0m 22s Docker mode activated.
        +1 @author 0m 0s The patch does not contain any @author tags.
        +1 test4tests 0m 0s The patch appears to include 1 new or modified test files.
        +1 mvninstall 13m 31s trunk passed
        +1 compile 0m 46s trunk passed
        +1 checkstyle 0m 28s trunk passed
        +1 mvnsite 0m 55s trunk passed
        +1 mvneclipse 0m 12s trunk passed
        +1 findbugs 1m 48s trunk passed
        +1 javadoc 0m 40s trunk passed
        +1 mvninstall 0m 49s the patch passed
        +1 compile 0m 46s the patch passed
        +1 javac 0m 46s the patch passed
        -0 checkstyle 0m 27s hadoop-hdfs-project/hadoop-hdfs: The patch generated 2 new + 236 unchanged - 0 fixed = 238 total (was 236)
        +1 mvnsite 0m 53s the patch passed
        +1 mvneclipse 0m 10s the patch passed
        +1 whitespace 0m 0s The patch has no whitespace issues.
        +1 findbugs 1m 54s the patch passed
        +1 javadoc 0m 39s the patch passed
        -1 unit 70m 27s hadoop-hdfs in the patch failed.
        +1 asflicense 0m 18s The patch does not generate ASF License warnings.
        96m 19s



        Reason Tests
        Failed junit tests hadoop.hdfs.TestAclsEndToEnd
        Timed out junit tests org.apache.hadoop.hdfs.server.datanode.TestDataNodeVolumeFailureReporting



        Subsystem Report/Notes
        Docker Image:yetus/hadoop:a9ad5d6
        JIRA Issue HDFS-11333
        JIRA Patch URL https://issues.apache.org/jira/secure/attachment/12847086/HDFS-11333.001.patch
        Optional Tests asflicense compile javac javadoc mvninstall mvnsite unit findbugs checkstyle
        uname Linux 878dd00f4a08 3.13.0-106-generic #153-Ubuntu SMP Tue Dec 6 15:44:32 UTC 2016 x86_64 x86_64 x86_64 GNU/Linux
        Build tool maven
        Personality /testptch/hadoop/patchprocess/precommit/personality/provided.sh
        git revision trunk / e648b6e
        Default Java 1.8.0_111
        findbugs v3.0.0
        checkstyle https://builds.apache.org/job/PreCommit-HDFS-Build/18154/artifact/patchprocess/diff-checkstyle-hadoop-hdfs-project_hadoop-hdfs.txt
        unit https://builds.apache.org/job/PreCommit-HDFS-Build/18154/artifact/patchprocess/patch-unit-hadoop-hdfs-project_hadoop-hdfs.txt
        Test Results https://builds.apache.org/job/PreCommit-HDFS-Build/18154/testReport/
        modules C: hadoop-hdfs-project/hadoop-hdfs U: hadoop-hdfs-project/hadoop-hdfs
        Console output https://builds.apache.org/job/PreCommit-HDFS-Build/18154/console
        Powered by Apache Yetus 0.5.0-SNAPSHOT http://yetus.apache.org

        This message was automatically generated.

        Show
        hadoopqa Hadoop QA added a comment - -1 overall Vote Subsystem Runtime Comment 0 reexec 0m 22s Docker mode activated. +1 @author 0m 0s The patch does not contain any @author tags. +1 test4tests 0m 0s The patch appears to include 1 new or modified test files. +1 mvninstall 13m 31s trunk passed +1 compile 0m 46s trunk passed +1 checkstyle 0m 28s trunk passed +1 mvnsite 0m 55s trunk passed +1 mvneclipse 0m 12s trunk passed +1 findbugs 1m 48s trunk passed +1 javadoc 0m 40s trunk passed +1 mvninstall 0m 49s the patch passed +1 compile 0m 46s the patch passed +1 javac 0m 46s the patch passed -0 checkstyle 0m 27s hadoop-hdfs-project/hadoop-hdfs: The patch generated 2 new + 236 unchanged - 0 fixed = 238 total (was 236) +1 mvnsite 0m 53s the patch passed +1 mvneclipse 0m 10s the patch passed +1 whitespace 0m 0s The patch has no whitespace issues. +1 findbugs 1m 54s the patch passed +1 javadoc 0m 39s the patch passed -1 unit 70m 27s hadoop-hdfs in the patch failed. +1 asflicense 0m 18s The patch does not generate ASF License warnings. 96m 19s Reason Tests Failed junit tests hadoop.hdfs.TestAclsEndToEnd Timed out junit tests org.apache.hadoop.hdfs.server.datanode.TestDataNodeVolumeFailureReporting Subsystem Report/Notes Docker Image:yetus/hadoop:a9ad5d6 JIRA Issue HDFS-11333 JIRA Patch URL https://issues.apache.org/jira/secure/attachment/12847086/HDFS-11333.001.patch Optional Tests asflicense compile javac javadoc mvninstall mvnsite unit findbugs checkstyle uname Linux 878dd00f4a08 3.13.0-106-generic #153-Ubuntu SMP Tue Dec 6 15:44:32 UTC 2016 x86_64 x86_64 x86_64 GNU/Linux Build tool maven Personality /testptch/hadoop/patchprocess/precommit/personality/provided.sh git revision trunk / e648b6e Default Java 1.8.0_111 findbugs v3.0.0 checkstyle https://builds.apache.org/job/PreCommit-HDFS-Build/18154/artifact/patchprocess/diff-checkstyle-hadoop-hdfs-project_hadoop-hdfs.txt unit https://builds.apache.org/job/PreCommit-HDFS-Build/18154/artifact/patchprocess/patch-unit-hadoop-hdfs-project_hadoop-hdfs.txt Test Results https://builds.apache.org/job/PreCommit-HDFS-Build/18154/testReport/ modules C: hadoop-hdfs-project/hadoop-hdfs U: hadoop-hdfs-project/hadoop-hdfs Console output https://builds.apache.org/job/PreCommit-HDFS-Build/18154/console Powered by Apache Yetus 0.5.0-SNAPSHOT http://yetus.apache.org This message was automatically generated.
        Hide
        linyiqun Yiqun Lin added a comment -

        Good catch, Wei-Chiu Chuang. The patch LGTM. +1. Seems there is an unused import.

         import org.apache.hadoop.conf.Configuration;
        +import org.apache.hadoop.fs.CommonConfigurationKeysPublic;
         import org.apache.hadoop.fs.FileSystem;
        
        Show
        linyiqun Yiqun Lin added a comment - Good catch, Wei-Chiu Chuang . The patch LGTM. +1. Seems there is an unused import. import org.apache.hadoop.conf.Configuration; + import org.apache.hadoop.fs.CommonConfigurationKeysPublic; import org.apache.hadoop.fs.FileSystem;
        Hide
        jojochuang Wei-Chiu Chuang added a comment -

        Thanks for the review, Yiqun Lin!
        Incidentally, I also found the same plugin configuration for DataNode, and a DataNode can also fail to start for the same reason (configuration key: dfs.datanode.plugins).

        Do you think DataNode should also log a warning and proceed if plugins can't be loaded?

        Show
        jojochuang Wei-Chiu Chuang added a comment - Thanks for the review, Yiqun Lin ! Incidentally, I also found the same plugin configuration for DataNode, and a DataNode can also fail to start for the same reason (configuration key: dfs.datanode.plugins). Do you think DataNode should also log a warning and proceed if plugins can't be loaded?
        Hide
        aw Allen Wittenauer added a comment -

        Plugins are used for a variety of things, including security. Starting up the NN when a security plugin is missing will also lower the security of the NN. That would be a very bad thing. It seems like catching, logging, and then failing to start is the correct behavior.

        Show
        aw Allen Wittenauer added a comment - Plugins are used for a variety of things, including security. Starting up the NN when a security plugin is missing will also lower the security of the NN. That would be a very bad thing. It seems like catching, logging, and then failing to start is the correct behavior.
        Hide
        jojochuang Wei-Chiu Chuang added a comment -

        I see. Didn't realize there are security concerns in it. That makes total sense to me.

        Show
        jojochuang Wei-Chiu Chuang added a comment - I see. Didn't realize there are security concerns in it. That makes total sense to me.
        Hide
        linyiqun Yiqun Lin added a comment -

        The suggestion from Allen Wittenauer also makes sense to me. Based on v01 patch, I think it's suffient to just rethrow the exception here.

        +    try {
        +      plugins = conf.getInstances(DFS_NAMENODE_PLUGINS_KEY,
        +          ServicePlugin.class);
        +    } catch (RuntimeException e) {
        +      LOG.warn("Unable to load plugins", e);
               throw e;          <===
        +    }
        

        And this logic can be reused for datanode plugins loading.

        Show
        linyiqun Yiqun Lin added a comment - The suggestion from Allen Wittenauer also makes sense to me. Based on v01 patch, I think it's suffient to just rethrow the exception here. + try { + plugins = conf.getInstances(DFS_NAMENODE_PLUGINS_KEY, + ServicePlugin.class); + } catch (RuntimeException e) { + LOG.warn( "Unable to load plugins" , e); throw e; <=== + } And this logic can be reused for datanode plugins loading.
        Hide
        jojochuang Wei-Chiu Chuang added a comment -

        Thanks Yiqun Lin.
        Posted my v002 patch.

        Add an error message that explicitly says what the exception means. The exception is rethrown afterwards. Also, print the list of plugins to help troubleshooting.

        Show
        jojochuang Wei-Chiu Chuang added a comment - Thanks Yiqun Lin . Posted my v002 patch. Add an error message that explicitly says what the exception means. The exception is rethrown afterwards. Also, print the list of plugins to help troubleshooting.
        Hide
        hadoopqa Hadoop QA added a comment -
        -1 overall



        Vote Subsystem Runtime Comment
        0 reexec 0m 16s Docker mode activated.
        +1 @author 0m 0s The patch does not contain any @author tags.
        -1 test4tests 0m 0s The patch doesn't appear to include any new or modified tests. Please justify why no new tests are needed for this patch. Also please list what manual steps were performed to verify this patch.
        +1 mvninstall 12m 28s trunk passed
        +1 compile 0m 45s trunk passed
        +1 checkstyle 0m 30s trunk passed
        +1 mvnsite 0m 50s trunk passed
        +1 mvneclipse 0m 12s trunk passed
        +1 findbugs 1m 44s trunk passed
        +1 javadoc 0m 39s trunk passed
        +1 mvninstall 0m 45s the patch passed
        +1 compile 0m 43s the patch passed
        +1 javac 0m 43s the patch passed
        +1 checkstyle 0m 27s the patch passed
        +1 mvnsite 0m 48s the patch passed
        +1 mvneclipse 0m 10s the patch passed
        +1 whitespace 0m 0s The patch has no whitespace issues.
        +1 findbugs 1m 47s the patch passed
        +1 javadoc 0m 36s the patch passed
        -1 unit 91m 51s hadoop-hdfs in the patch failed.
        +1 asflicense 0m 18s The patch does not generate ASF License warnings.
        116m 5s



        Reason Tests
        Failed junit tests hadoop.hdfs.server.datanode.TestDataNodeVolumeFailure
          hadoop.hdfs.server.datanode.TestDataNodeVolumeFailureReporting



        Subsystem Report/Notes
        Docker Image:yetus/hadoop:a9ad5d6
        JIRA Issue HDFS-11333
        JIRA Patch URL https://issues.apache.org/jira/secure/attachment/12852421/HDFS-11333.002.patch
        Optional Tests asflicense compile javac javadoc mvninstall mvnsite unit findbugs checkstyle
        uname Linux fcf4653fe373 3.13.0-107-generic #154-Ubuntu SMP Tue Dec 20 09:57:27 UTC 2016 x86_64 x86_64 x86_64 GNU/Linux
        Build tool maven
        Personality /testptch/hadoop/patchprocess/precommit/personality/provided.sh
        git revision trunk / 4ed33e9
        Default Java 1.8.0_121
        findbugs v3.0.0
        unit https://builds.apache.org/job/PreCommit-HDFS-Build/18359/artifact/patchprocess/patch-unit-hadoop-hdfs-project_hadoop-hdfs.txt
        Test Results https://builds.apache.org/job/PreCommit-HDFS-Build/18359/testReport/
        modules C: hadoop-hdfs-project/hadoop-hdfs U: hadoop-hdfs-project/hadoop-hdfs
        Console output https://builds.apache.org/job/PreCommit-HDFS-Build/18359/console
        Powered by Apache Yetus 0.5.0-SNAPSHOT http://yetus.apache.org

        This message was automatically generated.

        Show
        hadoopqa Hadoop QA added a comment - -1 overall Vote Subsystem Runtime Comment 0 reexec 0m 16s Docker mode activated. +1 @author 0m 0s The patch does not contain any @author tags. -1 test4tests 0m 0s The patch doesn't appear to include any new or modified tests. Please justify why no new tests are needed for this patch. Also please list what manual steps were performed to verify this patch. +1 mvninstall 12m 28s trunk passed +1 compile 0m 45s trunk passed +1 checkstyle 0m 30s trunk passed +1 mvnsite 0m 50s trunk passed +1 mvneclipse 0m 12s trunk passed +1 findbugs 1m 44s trunk passed +1 javadoc 0m 39s trunk passed +1 mvninstall 0m 45s the patch passed +1 compile 0m 43s the patch passed +1 javac 0m 43s the patch passed +1 checkstyle 0m 27s the patch passed +1 mvnsite 0m 48s the patch passed +1 mvneclipse 0m 10s the patch passed +1 whitespace 0m 0s The patch has no whitespace issues. +1 findbugs 1m 47s the patch passed +1 javadoc 0m 36s the patch passed -1 unit 91m 51s hadoop-hdfs in the patch failed. +1 asflicense 0m 18s The patch does not generate ASF License warnings. 116m 5s Reason Tests Failed junit tests hadoop.hdfs.server.datanode.TestDataNodeVolumeFailure   hadoop.hdfs.server.datanode.TestDataNodeVolumeFailureReporting Subsystem Report/Notes Docker Image:yetus/hadoop:a9ad5d6 JIRA Issue HDFS-11333 JIRA Patch URL https://issues.apache.org/jira/secure/attachment/12852421/HDFS-11333.002.patch Optional Tests asflicense compile javac javadoc mvninstall mvnsite unit findbugs checkstyle uname Linux fcf4653fe373 3.13.0-107-generic #154-Ubuntu SMP Tue Dec 20 09:57:27 UTC 2016 x86_64 x86_64 x86_64 GNU/Linux Build tool maven Personality /testptch/hadoop/patchprocess/precommit/personality/provided.sh git revision trunk / 4ed33e9 Default Java 1.8.0_121 findbugs v3.0.0 unit https://builds.apache.org/job/PreCommit-HDFS-Build/18359/artifact/patchprocess/patch-unit-hadoop-hdfs-project_hadoop-hdfs.txt Test Results https://builds.apache.org/job/PreCommit-HDFS-Build/18359/testReport/ modules C: hadoop-hdfs-project/hadoop-hdfs U: hadoop-hdfs-project/hadoop-hdfs Console output https://builds.apache.org/job/PreCommit-HDFS-Build/18359/console Powered by Apache Yetus 0.5.0-SNAPSHOT http://yetus.apache.org This message was automatically generated.
        Hide
        linyiqun Yiqun Lin added a comment -

        LGTM, +1. Wei-Chiu Chuang, you may delete v02 patch with warn message printing. It makes me confused there are two same name patches.

        Show
        linyiqun Yiqun Lin added a comment - LGTM, +1. Wei-Chiu Chuang , you may delete v02 patch with warn message printing. It makes me confused there are two same name patches.
        Hide
        jojochuang Wei-Chiu Chuang added a comment -

        Thanks for catching that Yiqun Lin.
        Copied the second 002 patch to 003. I would prefer to keep the patch history.

        Show
        jojochuang Wei-Chiu Chuang added a comment - Thanks for catching that Yiqun Lin . Copied the second 002 patch to 003. I would prefer to keep the patch history.
        Hide
        hadoopqa Hadoop QA added a comment -
        -1 overall



        Vote Subsystem Runtime Comment
        0 reexec 0m 18s Docker mode activated.
        +1 @author 0m 0s The patch does not contain any @author tags.
        -1 test4tests 0m 0s The patch doesn't appear to include any new or modified tests. Please justify why no new tests are needed for this patch. Also please list what manual steps were performed to verify this patch.
        +1 mvninstall 12m 54s trunk passed
        +1 compile 0m 44s trunk passed
        +1 checkstyle 0m 30s trunk passed
        +1 mvnsite 0m 51s trunk passed
        +1 mvneclipse 0m 13s trunk passed
        +1 findbugs 1m 42s trunk passed
        +1 javadoc 0m 40s trunk passed
        +1 mvninstall 0m 45s the patch passed
        +1 compile 0m 42s the patch passed
        +1 javac 0m 42s the patch passed
        +1 checkstyle 0m 27s the patch passed
        +1 mvnsite 0m 48s the patch passed
        +1 mvneclipse 0m 10s the patch passed
        +1 whitespace 0m 0s The patch has no whitespace issues.
        +1 findbugs 1m 49s the patch passed
        +1 javadoc 0m 39s the patch passed
        +1 unit 63m 40s hadoop-hdfs in the patch passed.
        +1 asflicense 0m 19s The patch does not generate ASF License warnings.
        88m 32s



        Subsystem Report/Notes
        Docker Image:yetus/hadoop:a9ad5d6
        JIRA Issue HDFS-11333
        JIRA Patch URL https://issues.apache.org/jira/secure/attachment/12852542/HDFS-11333.003.patch
        Optional Tests asflicense compile javac javadoc mvninstall mvnsite unit findbugs checkstyle
        uname Linux b3c5f4210214 3.13.0-106-generic #153-Ubuntu SMP Tue Dec 6 15:44:32 UTC 2016 x86_64 x86_64 x86_64 GNU/Linux
        Build tool maven
        Personality /testptch/hadoop/patchprocess/precommit/personality/provided.sh
        git revision trunk / 719df99
        Default Java 1.8.0_121
        findbugs v3.0.0
        Test Results https://builds.apache.org/job/PreCommit-HDFS-Build/18369/testReport/
        modules C: hadoop-hdfs-project/hadoop-hdfs U: hadoop-hdfs-project/hadoop-hdfs
        Console output https://builds.apache.org/job/PreCommit-HDFS-Build/18369/console
        Powered by Apache Yetus 0.5.0-SNAPSHOT http://yetus.apache.org

        This message was automatically generated.

        Show
        hadoopqa Hadoop QA added a comment - -1 overall Vote Subsystem Runtime Comment 0 reexec 0m 18s Docker mode activated. +1 @author 0m 0s The patch does not contain any @author tags. -1 test4tests 0m 0s The patch doesn't appear to include any new or modified tests. Please justify why no new tests are needed for this patch. Also please list what manual steps were performed to verify this patch. +1 mvninstall 12m 54s trunk passed +1 compile 0m 44s trunk passed +1 checkstyle 0m 30s trunk passed +1 mvnsite 0m 51s trunk passed +1 mvneclipse 0m 13s trunk passed +1 findbugs 1m 42s trunk passed +1 javadoc 0m 40s trunk passed +1 mvninstall 0m 45s the patch passed +1 compile 0m 42s the patch passed +1 javac 0m 42s the patch passed +1 checkstyle 0m 27s the patch passed +1 mvnsite 0m 48s the patch passed +1 mvneclipse 0m 10s the patch passed +1 whitespace 0m 0s The patch has no whitespace issues. +1 findbugs 1m 49s the patch passed +1 javadoc 0m 39s the patch passed +1 unit 63m 40s hadoop-hdfs in the patch passed. +1 asflicense 0m 19s The patch does not generate ASF License warnings. 88m 32s Subsystem Report/Notes Docker Image:yetus/hadoop:a9ad5d6 JIRA Issue HDFS-11333 JIRA Patch URL https://issues.apache.org/jira/secure/attachment/12852542/HDFS-11333.003.patch Optional Tests asflicense compile javac javadoc mvninstall mvnsite unit findbugs checkstyle uname Linux b3c5f4210214 3.13.0-106-generic #153-Ubuntu SMP Tue Dec 6 15:44:32 UTC 2016 x86_64 x86_64 x86_64 GNU/Linux Build tool maven Personality /testptch/hadoop/patchprocess/precommit/personality/provided.sh git revision trunk / 719df99 Default Java 1.8.0_121 findbugs v3.0.0 Test Results https://builds.apache.org/job/PreCommit-HDFS-Build/18369/testReport/ modules C: hadoop-hdfs-project/hadoop-hdfs U: hadoop-hdfs-project/hadoop-hdfs Console output https://builds.apache.org/job/PreCommit-HDFS-Build/18369/console Powered by Apache Yetus 0.5.0-SNAPSHOT http://yetus.apache.org This message was automatically generated.
        Hide
        jojochuang Wei-Chiu Chuang added a comment -

        Committed the patch to trunk, branch-2, branch-2.8 and branch-2.7.
        Thanks very much for the review from Yiqun Lin and comments from Allen Wittenauer!

        Show
        jojochuang Wei-Chiu Chuang added a comment - Committed the patch to trunk, branch-2, branch-2.8 and branch-2.7. Thanks very much for the review from Yiqun Lin and comments from Allen Wittenauer !
        Hide
        hudson Hudson added a comment -

        SUCCESS: Integrated in Jenkins build Hadoop-trunk-Commit #11256 (See https://builds.apache.org/job/Hadoop-trunk-Commit/11256/)
        HDFS-11333. Print a user friendly error message when plugins are not (weichiu: rev 859bd159ae554174200334b5eb1d7e8dbef958ad)

        • (edit) hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/datanode/DataNode.java
        • (edit) hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/NameNode.java
        Show
        hudson Hudson added a comment - SUCCESS: Integrated in Jenkins build Hadoop-trunk-Commit #11256 (See https://builds.apache.org/job/Hadoop-trunk-Commit/11256/ ) HDFS-11333 . Print a user friendly error message when plugins are not (weichiu: rev 859bd159ae554174200334b5eb1d7e8dbef958ad) (edit) hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/datanode/DataNode.java (edit) hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/NameNode.java
        Hide
        vinodkv Vinod Kumar Vavilapalli added a comment -

        2.8.1 became a security release. Moving fix-version to 2.8.2 after the fact.

        Show
        vinodkv Vinod Kumar Vavilapalli added a comment - 2.8.1 became a security release. Moving fix-version to 2.8.2 after the fact.

          People

          • Assignee:
            jojochuang Wei-Chiu Chuang
            Reporter:
            jojochuang Wei-Chiu Chuang
          • Votes:
            0 Vote for this issue
            Watchers:
            6 Start watching this issue

            Dates

            • Created:
              Updated:
              Resolved:

              Development