Uploaded image for project: 'Hadoop HDFS'
  1. Hadoop HDFS
  2. HDFS-12412

Change ErasureCodingWorker.stripedReadPool to cached thread pool

    Details

    • Target Version/s:
    • Release Note:
      Changed {{stripedReadPool}} to unbounded cachedThreadPool. User should combine {{dfs.datanode.ec.reconstruction.stripedblock.threads}} and {{dfs.namenode.replication.max-streams}} to tune recovery performance.

      Description

      In ErasureCodingWorker, it uses stripedReconstructionPool to schedule the EC recovery tasks, while uses stripedReadPool for the reader threads in each recovery task. We only need one of them to throttle the speed of recovery process, because each EC recovery task has a fix number of source readers (i.e., 3 for RS(3,2)). And because of the findings in HDFS-12044, the speed of EC recovery can be throttled by strippedReconstructionPool with xmitsInProgress.

      Moreover, keeping stripedReadPool makes customer difficult to understand and calculate the right balance between dfs.datanode.ec.reconstruction.stripedread.threads, dfs.datanode.ec.reconstruction.stripedblock.threads.size and maxReplicationStreams. For example, a small stripread.threads (comparing to which reconstruction.threads.size implies), will unnecessarily limit the speed of recovery, which leads to larger MTTR.

      1. HDFS-12412.00.patch
        3 kB
        Lei (Eddy) Xu
      2. HDFS-12412.01.patch
        5 kB
        Lei (Eddy) Xu

        Activity

        Hide
        andrew.wang Andrew Wang added a comment -

        I've cherry-picked this to branch-3.0 for beta1.

        Show
        andrew.wang Andrew Wang added a comment - I've cherry-picked this to branch-3.0 for beta1.
        Hide
        hadoopqa Hadoop QA added a comment -
        -1 overall



        Vote Subsystem Runtime Comment
        0 reexec 0m 20s Docker mode activated.
              Prechecks
        +1 @author 0m 0s The patch does not contain any @author tags.
        -1 test4tests 0m 0s The patch doesn't appear to include any new or modified tests. Please justify why no new tests are needed for this patch. Also please list what manual steps were performed to verify this patch.
              trunk Compile Tests
        +1 mvninstall 16m 15s trunk passed
        +1 compile 1m 6s trunk passed
        +1 checkstyle 0m 44s trunk passed
        +1 mvnsite 1m 0s trunk passed
        +1 findbugs 1m 43s trunk passed
        +1 javadoc 0m 41s trunk passed
              Patch Compile Tests
        +1 mvninstall 0m 57s the patch passed
        +1 compile 0m 53s the patch passed
        +1 javac 0m 53s the patch passed
        +1 checkstyle 0m 42s hadoop-hdfs-project/hadoop-hdfs: The patch generated 0 new + 415 unchanged - 2 fixed = 415 total (was 417)
        +1 mvnsite 0m 55s the patch passed
        +1 whitespace 0m 0s The patch has no whitespace issues.
        +1 xml 0m 1s The patch has no ill-formed XML file.
        +1 findbugs 1m 55s the patch passed
        +1 javadoc 0m 40s the patch passed
              Other Tests
        -1 unit 108m 59s hadoop-hdfs in the patch failed.
        +1 asflicense 0m 21s The patch does not generate ASF License warnings.
        138m 36s



        Reason Tests
        Failed junit tests hadoop.hdfs.TestReconstructStripedFile
          hadoop.hdfs.TestSnapshotCommands
          hadoop.hdfs.TestClientProtocolForPipelineRecovery
          hadoop.hdfs.TestLeaseRecoveryStriped
          hadoop.hdfs.TestRollingUpgrade
          hadoop.hdfs.TestParallelShortCircuitReadUnCached
          hadoop.hdfs.server.blockmanagement.TestReconstructStripedBlocksWithRackAwareness
          hadoop.hdfs.security.TestDelegationToken
          hadoop.hdfs.server.namenode.TestDiskspaceQuotaUpdate
          hadoop.hdfs.TestEncryptedTransfer
        Timed out junit tests org.apache.hadoop.hdfs.TestWriteReadStripedFile



        Subsystem Report/Notes
        Docker Image:yetus/hadoop:71bbb86
        JIRA Issue HDFS-12412
        JIRA Patch URL https://issues.apache.org/jira/secure/attachment/12886754/HDFS-12412.01.patch
        Optional Tests asflicense compile javac javadoc mvninstall mvnsite unit findbugs checkstyle xml
        uname Linux c01dee58fa7f 3.13.0-123-generic #172-Ubuntu SMP Mon Jun 26 18:04:35 UTC 2017 x86_64 x86_64 x86_64 GNU/Linux
        Build tool maven
        Personality /testptch/hadoop/patchprocess/precommit/personality/provided.sh
        git revision trunk / 68282c8
        Default Java 1.8.0_144
        findbugs v3.1.0-RC1
        unit https://builds.apache.org/job/PreCommit-HDFS-Build/21107/artifact/patchprocess/patch-unit-hadoop-hdfs-project_hadoop-hdfs.txt
        Test Results https://builds.apache.org/job/PreCommit-HDFS-Build/21107/testReport/
        modules C: hadoop-hdfs-project/hadoop-hdfs U: hadoop-hdfs-project/hadoop-hdfs
        Console output https://builds.apache.org/job/PreCommit-HDFS-Build/21107/console
        Powered by Apache Yetus 0.6.0-SNAPSHOT http://yetus.apache.org

        This message was automatically generated.

        Show
        hadoopqa Hadoop QA added a comment - -1 overall Vote Subsystem Runtime Comment 0 reexec 0m 20s Docker mode activated.       Prechecks +1 @author 0m 0s The patch does not contain any @author tags. -1 test4tests 0m 0s The patch doesn't appear to include any new or modified tests. Please justify why no new tests are needed for this patch. Also please list what manual steps were performed to verify this patch.       trunk Compile Tests +1 mvninstall 16m 15s trunk passed +1 compile 1m 6s trunk passed +1 checkstyle 0m 44s trunk passed +1 mvnsite 1m 0s trunk passed +1 findbugs 1m 43s trunk passed +1 javadoc 0m 41s trunk passed       Patch Compile Tests +1 mvninstall 0m 57s the patch passed +1 compile 0m 53s the patch passed +1 javac 0m 53s the patch passed +1 checkstyle 0m 42s hadoop-hdfs-project/hadoop-hdfs: The patch generated 0 new + 415 unchanged - 2 fixed = 415 total (was 417) +1 mvnsite 0m 55s the patch passed +1 whitespace 0m 0s The patch has no whitespace issues. +1 xml 0m 1s The patch has no ill-formed XML file. +1 findbugs 1m 55s the patch passed +1 javadoc 0m 40s the patch passed       Other Tests -1 unit 108m 59s hadoop-hdfs in the patch failed. +1 asflicense 0m 21s The patch does not generate ASF License warnings. 138m 36s Reason Tests Failed junit tests hadoop.hdfs.TestReconstructStripedFile   hadoop.hdfs.TestSnapshotCommands   hadoop.hdfs.TestClientProtocolForPipelineRecovery   hadoop.hdfs.TestLeaseRecoveryStriped   hadoop.hdfs.TestRollingUpgrade   hadoop.hdfs.TestParallelShortCircuitReadUnCached   hadoop.hdfs.server.blockmanagement.TestReconstructStripedBlocksWithRackAwareness   hadoop.hdfs.security.TestDelegationToken   hadoop.hdfs.server.namenode.TestDiskspaceQuotaUpdate   hadoop.hdfs.TestEncryptedTransfer Timed out junit tests org.apache.hadoop.hdfs.TestWriteReadStripedFile Subsystem Report/Notes Docker Image:yetus/hadoop:71bbb86 JIRA Issue HDFS-12412 JIRA Patch URL https://issues.apache.org/jira/secure/attachment/12886754/HDFS-12412.01.patch Optional Tests asflicense compile javac javadoc mvninstall mvnsite unit findbugs checkstyle xml uname Linux c01dee58fa7f 3.13.0-123-generic #172-Ubuntu SMP Mon Jun 26 18:04:35 UTC 2017 x86_64 x86_64 x86_64 GNU/Linux Build tool maven Personality /testptch/hadoop/patchprocess/precommit/personality/provided.sh git revision trunk / 68282c8 Default Java 1.8.0_144 findbugs v3.1.0-RC1 unit https://builds.apache.org/job/PreCommit-HDFS-Build/21107/artifact/patchprocess/patch-unit-hadoop-hdfs-project_hadoop-hdfs.txt Test Results https://builds.apache.org/job/PreCommit-HDFS-Build/21107/testReport/ modules C: hadoop-hdfs-project/hadoop-hdfs U: hadoop-hdfs-project/hadoop-hdfs Console output https://builds.apache.org/job/PreCommit-HDFS-Build/21107/console Powered by Apache Yetus 0.6.0-SNAPSHOT http://yetus.apache.org This message was automatically generated.
        Hide
        hudson Hudson added a comment -

        SUCCESS: Integrated in Jenkins build Hadoop-trunk-Commit #12857 (See https://builds.apache.org/job/Hadoop-trunk-Commit/12857/)
        HDFS-12412. Change ErasureCodingWorker.stripedReadPool to cached thread (lei: rev 123342cd0759ff88801d4f5ab10987f6e3f344b0)

        • (edit) hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/DFSConfigKeys.java
        • (edit) hadoop-hdfs-project/hadoop-hdfs/src/site/markdown/HDFSErasureCoding.md
        • (edit) hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/datanode/erasurecode/ErasureCodingWorker.java
        • (edit) hadoop-hdfs-project/hadoop-hdfs/src/main/resources/hdfs-default.xml
        Show
        hudson Hudson added a comment - SUCCESS: Integrated in Jenkins build Hadoop-trunk-Commit #12857 (See https://builds.apache.org/job/Hadoop-trunk-Commit/12857/ ) HDFS-12412 . Change ErasureCodingWorker.stripedReadPool to cached thread (lei: rev 123342cd0759ff88801d4f5ab10987f6e3f344b0) (edit) hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/DFSConfigKeys.java (edit) hadoop-hdfs-project/hadoop-hdfs/src/site/markdown/HDFSErasureCoding.md (edit) hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/datanode/erasurecode/ErasureCodingWorker.java (edit) hadoop-hdfs-project/hadoop-hdfs/src/main/resources/hdfs-default.xml
        Hide
        eddyxu Lei (Eddy) Xu added a comment -

        Thanks for the reviews, Andrew Wang

        Ran TestReconstructStripedFile,TestReadStripedFileWithMissingBlocks locally and passes. The rest seem not relevant. Filed HDFS-12439 for TestReconstructStripedFile.

        Committed to trunk.

        Show
        eddyxu Lei (Eddy) Xu added a comment - Thanks for the reviews, Andrew Wang Ran TestReconstructStripedFile,TestReadStripedFileWithMissingBlocks locally and passes. The rest seem not relevant. Filed HDFS-12439 for TestReconstructStripedFile. Committed to trunk .
        Hide
        hadoopqa Hadoop QA added a comment -
        -1 overall



        Vote Subsystem Runtime Comment
        0 reexec 0m 14s Docker mode activated.
              Prechecks
        +1 @author 0m 0s The patch does not contain any @author tags.
        -1 test4tests 0m 0s The patch doesn't appear to include any new or modified tests. Please justify why no new tests are needed for this patch. Also please list what manual steps were performed to verify this patch.
              trunk Compile Tests
        +1 mvninstall 15m 13s trunk passed
        +1 compile 0m 54s trunk passed
        +1 checkstyle 0m 43s trunk passed
        +1 mvnsite 1m 3s trunk passed
        +1 findbugs 1m 49s trunk passed
        +1 javadoc 0m 43s trunk passed
              Patch Compile Tests
        +1 mvninstall 0m 58s the patch passed
        +1 compile 0m 55s the patch passed
        +1 javac 0m 55s the patch passed
        +1 checkstyle 0m 39s hadoop-hdfs-project/hadoop-hdfs: The patch generated 0 new + 415 unchanged - 2 fixed = 415 total (was 417)
        +1 mvnsite 0m 58s the patch passed
        +1 whitespace 0m 0s The patch has no whitespace issues.
        +1 findbugs 1m 54s the patch passed
        +1 javadoc 0m 40s the patch passed
              Other Tests
        -1 unit 83m 38s hadoop-hdfs in the patch failed.
        +1 asflicense 0m 22s The patch does not generate ASF License warnings.
        112m 5s



        Reason Tests
        Failed junit tests hadoop.hdfs.TestPread
          hadoop.hdfs.TestDFSStripedOutputStreamWithFailure100
          hadoop.hdfs.TestReadStripedFileWithMissingBlocks
          hadoop.hdfs.security.TestDelegationTokenForProxyUser
          hadoop.hdfs.tools.TestDFSAdminWithHA
          hadoop.hdfs.TestDFSStripedOutputStreamWithFailure180
          hadoop.hdfs.qjournal.TestNNWithQJM
          hadoop.hdfs.TestDFSStripedOutputStreamWithFailure190
          hadoop.hdfs.TestQuota
          hadoop.hdfs.TestDFSStripedOutputStreamWithFailure020
          hadoop.hdfs.TestLeaseRecoveryStriped
          hadoop.hdfs.TestDFSStripedOutputStreamWithFailure120
          hadoop.hdfs.server.datanode.TestDataNodeVolumeFailureReporting
          hadoop.hdfs.TestDFSStripedOutputStreamWithFailure150
          hadoop.hdfs.TestReconstructStripedFile
        Timed out junit tests org.apache.hadoop.hdfs.TestReadStripedFileWithDecodingCorruptData
          org.apache.hadoop.hdfs.TestWriteReadStripedFile
          org.apache.hadoop.hdfs.server.namenode.ha.TestStandbyInProgressTail
          org.apache.hadoop.hdfs.server.namenode.ha.TestPipelinesFailover
          org.apache.hadoop.hdfs.server.namenode.ha.TestXAttrsWithHA
          org.apache.hadoop.hdfs.server.namenode.ha.TestHASafeMode



        Subsystem Report/Notes
        Docker Image:yetus/hadoop:71bbb86
        JIRA Issue HDFS-12412
        JIRA Patch URL https://issues.apache.org/jira/secure/attachment/12886518/HDFS-12412.00.patch
        Optional Tests asflicense compile javac javadoc mvninstall mvnsite unit findbugs checkstyle
        uname Linux 5872c864f6a5 3.13.0-123-generic #172-Ubuntu SMP Mon Jun 26 18:04:35 UTC 2017 x86_64 x86_64 x86_64 GNU/Linux
        Build tool maven
        Personality /testptch/hadoop/patchprocess/precommit/personality/provided.sh
        git revision trunk / 86f4d1c
        Default Java 1.8.0_144
        findbugs v3.1.0-RC1
        unit https://builds.apache.org/job/PreCommit-HDFS-Build/21106/artifact/patchprocess/patch-unit-hadoop-hdfs-project_hadoop-hdfs.txt
        Test Results https://builds.apache.org/job/PreCommit-HDFS-Build/21106/testReport/
        modules C: hadoop-hdfs-project/hadoop-hdfs U: hadoop-hdfs-project/hadoop-hdfs
        Console output https://builds.apache.org/job/PreCommit-HDFS-Build/21106/console
        Powered by Apache Yetus 0.6.0-SNAPSHOT http://yetus.apache.org

        This message was automatically generated.

        Show
        hadoopqa Hadoop QA added a comment - -1 overall Vote Subsystem Runtime Comment 0 reexec 0m 14s Docker mode activated.       Prechecks +1 @author 0m 0s The patch does not contain any @author tags. -1 test4tests 0m 0s The patch doesn't appear to include any new or modified tests. Please justify why no new tests are needed for this patch. Also please list what manual steps were performed to verify this patch.       trunk Compile Tests +1 mvninstall 15m 13s trunk passed +1 compile 0m 54s trunk passed +1 checkstyle 0m 43s trunk passed +1 mvnsite 1m 3s trunk passed +1 findbugs 1m 49s trunk passed +1 javadoc 0m 43s trunk passed       Patch Compile Tests +1 mvninstall 0m 58s the patch passed +1 compile 0m 55s the patch passed +1 javac 0m 55s the patch passed +1 checkstyle 0m 39s hadoop-hdfs-project/hadoop-hdfs: The patch generated 0 new + 415 unchanged - 2 fixed = 415 total (was 417) +1 mvnsite 0m 58s the patch passed +1 whitespace 0m 0s The patch has no whitespace issues. +1 findbugs 1m 54s the patch passed +1 javadoc 0m 40s the patch passed       Other Tests -1 unit 83m 38s hadoop-hdfs in the patch failed. +1 asflicense 0m 22s The patch does not generate ASF License warnings. 112m 5s Reason Tests Failed junit tests hadoop.hdfs.TestPread   hadoop.hdfs.TestDFSStripedOutputStreamWithFailure100   hadoop.hdfs.TestReadStripedFileWithMissingBlocks   hadoop.hdfs.security.TestDelegationTokenForProxyUser   hadoop.hdfs.tools.TestDFSAdminWithHA   hadoop.hdfs.TestDFSStripedOutputStreamWithFailure180   hadoop.hdfs.qjournal.TestNNWithQJM   hadoop.hdfs.TestDFSStripedOutputStreamWithFailure190   hadoop.hdfs.TestQuota   hadoop.hdfs.TestDFSStripedOutputStreamWithFailure020   hadoop.hdfs.TestLeaseRecoveryStriped   hadoop.hdfs.TestDFSStripedOutputStreamWithFailure120   hadoop.hdfs.server.datanode.TestDataNodeVolumeFailureReporting   hadoop.hdfs.TestDFSStripedOutputStreamWithFailure150   hadoop.hdfs.TestReconstructStripedFile Timed out junit tests org.apache.hadoop.hdfs.TestReadStripedFileWithDecodingCorruptData   org.apache.hadoop.hdfs.TestWriteReadStripedFile   org.apache.hadoop.hdfs.server.namenode.ha.TestStandbyInProgressTail   org.apache.hadoop.hdfs.server.namenode.ha.TestPipelinesFailover   org.apache.hadoop.hdfs.server.namenode.ha.TestXAttrsWithHA   org.apache.hadoop.hdfs.server.namenode.ha.TestHASafeMode Subsystem Report/Notes Docker Image:yetus/hadoop:71bbb86 JIRA Issue HDFS-12412 JIRA Patch URL https://issues.apache.org/jira/secure/attachment/12886518/HDFS-12412.00.patch Optional Tests asflicense compile javac javadoc mvninstall mvnsite unit findbugs checkstyle uname Linux 5872c864f6a5 3.13.0-123-generic #172-Ubuntu SMP Mon Jun 26 18:04:35 UTC 2017 x86_64 x86_64 x86_64 GNU/Linux Build tool maven Personality /testptch/hadoop/patchprocess/precommit/personality/provided.sh git revision trunk / 86f4d1c Default Java 1.8.0_144 findbugs v3.1.0-RC1 unit https://builds.apache.org/job/PreCommit-HDFS-Build/21106/artifact/patchprocess/patch-unit-hadoop-hdfs-project_hadoop-hdfs.txt Test Results https://builds.apache.org/job/PreCommit-HDFS-Build/21106/testReport/ modules C: hadoop-hdfs-project/hadoop-hdfs U: hadoop-hdfs-project/hadoop-hdfs Console output https://builds.apache.org/job/PreCommit-HDFS-Build/21106/console Powered by Apache Yetus 0.6.0-SNAPSHOT http://yetus.apache.org This message was automatically generated.
        Hide
        andrew.wang Andrew Wang added a comment -

        I also filed and posted a patch on HDFS-12438 for the config renaming I mentioned.

        Show
        andrew.wang Andrew Wang added a comment - I also filed and posted a patch on HDFS-12438 for the config renaming I mentioned.
        Hide
        andrew.wang Andrew Wang added a comment -

        +1 LGTM pending precommit, thanks Eddy.

        Show
        andrew.wang Andrew Wang added a comment - +1 LGTM pending precommit, thanks Eddy.
        Hide
        eddyxu Lei (Eddy) Xu added a comment -

        Thanks for the inputs, Andrew Wang. Removed references of dfs.datanode.ec.reconstruction.stripedread.threads in the

        {01}

        patch.

        Show
        eddyxu Lei (Eddy) Xu added a comment - Thanks for the inputs, Andrew Wang . Removed references of dfs.datanode.ec.reconstruction.stripedread.threads in the {01} patch.
        Hide
        andrew.wang Andrew Wang added a comment -

        While I was looking at this a little more, I noticed that "dfs.datanode.ec.reconstruction.stripedblock.threads.size" is named poorly. Could you do another JIRA to rename this to drop the ".size" suffix, for consistency with other similar config keys that control the size of a thread pool?

        It'd also be good to add a release note to this JIRA to point users at what keys to use for tuning reconstruction performance. If you want to update the EC user docs as part of this JIRA too, that'd also be good.

        Show
        andrew.wang Andrew Wang added a comment - While I was looking at this a little more, I noticed that "dfs.datanode.ec.reconstruction.stripedblock.threads.size" is named poorly. Could you do another JIRA to rename this to drop the ".size" suffix, for consistency with other similar config keys that control the size of a thread pool? It'd also be good to add a release note to this JIRA to point users at what keys to use for tuning reconstruction performance. If you want to update the EC user docs as part of this JIRA too, that'd also be good.
        Hide
        andrew.wang Andrew Wang added a comment -

        Thanks for working on this Eddy. I did a grep for the removed config key:

        -> % ag "stripedread.threads"     
        hadoop-hdfs-project/hadoop-hdfs/src/site/markdown/HDFSErasureCoding.md
        140:  1. `dfs.datanode.ec.reconstruction.stripedread.threads` - Number of concurrent reader threads. Default value is 20 threads.
        
        hadoop-hdfs-project/hadoop-hdfs/src/main/resources/hdfs-default.xml
        3052:  <name>dfs.datanode.ec.reconstruction.stripedread.threads</name>
        

        We need to update these as well. Otherwise, +1 LGTM!

        Show
        andrew.wang Andrew Wang added a comment - Thanks for working on this Eddy. I did a grep for the removed config key: -> % ag "stripedread.threads" hadoop-hdfs-project/hadoop-hdfs/src/site/markdown/HDFSErasureCoding.md 140: 1. `dfs.datanode.ec.reconstruction.stripedread.threads` - Number of concurrent reader threads. Default value is 20 threads. hadoop-hdfs-project/hadoop-hdfs/src/main/resources/hdfs-default.xml 3052: <name>dfs.datanode.ec.reconstruction.stripedread.threads</name> We need to update these as well. Otherwise, +1 LGTM!
        Hide
        eddyxu Lei (Eddy) Xu added a comment -

        Change stripedReadPool to an unbound cacheThreadPool. Andrew Wang, Kai Zheng could you give a review?

        Thanks.

        Show
        eddyxu Lei (Eddy) Xu added a comment - Change stripedReadPool to an unbound cacheThreadPool . Andrew Wang , Kai Zheng could you give a review? Thanks.
        Hide
        andrew.wang Andrew Wang added a comment -

        Agree, nice idea here Eddy. We can always add more tunables later.

        Show
        andrew.wang Andrew Wang added a comment - Agree, nice idea here Eddy. We can always add more tunables later.
        Hide
        drankye Kai Zheng added a comment -

        Thanks Eddy for the ping.

        The idea to remove the striped read pool and reuse the same reconstruction pool sounds good to me, since given the later and the most often used erasure codec, we can roughly estimate the striped read threads need. We can also simplify the configuration and codes.

        So as you said, you probably have the idea how to reduce the recommended value or default value and validate the configuration value for the reconstruction pool size, assuming you know how many concurrent reconstruction tasks to be performed and so on.

        Less configuration with reasonable defaults would make the brand feature more easier to use. When needed, we can fine-tune and add more later.

        Show
        drankye Kai Zheng added a comment - Thanks Eddy for the ping. The idea to remove the striped read pool and reuse the same reconstruction pool sounds good to me, since given the later and the most often used erasure codec, we can roughly estimate the striped read threads need. We can also simplify the configuration and codes. So as you said, you probably have the idea how to reduce the recommended value or default value and validate the configuration value for the reconstruction pool size, assuming you know how many concurrent reconstruction tasks to be performed and so on. Less configuration with reasonable defaults would make the brand feature more easier to use. When needed, we can fine-tune and add more later.
        Hide
        eddyxu Lei (Eddy) Xu added a comment - - edited

        Ping Kai Zheng SammiChen Andrew Wang what would you think?

        Show
        eddyxu Lei (Eddy) Xu added a comment - - edited Ping Kai Zheng SammiChen Andrew Wang what would you think?

          People

          • Assignee:
            eddyxu Lei (Eddy) Xu
            Reporter:
            eddyxu Lei (Eddy) Xu
          • Votes:
            0 Vote for this issue
            Watchers:
            5 Start watching this issue

            Dates

            • Created:
              Updated:
              Resolved:

              Development