Details

    • Type: Sub-task
    • Status: Resolved
    • Priority: Major
    • Resolution: Fixed
    • Affects Version/s: YARN-2915
    • Fix Version/s: YARN-2915
    • Labels:
      None
    • Hadoop Flags:
      Reviewed

      Description

      This JIRA tracks a SQL-based implementation of the Federation State and Policy Store, which implements YARN-3662 APIs.

      1. YARN-3663-YARN-2915.v1.patch
        63 kB
        Giovanni Matteo Fumarola
      2. YARN-3663-YARN-2915.v2.patch
        109 kB
        Giovanni Matteo Fumarola
      3. YARN-3663-YARN-2915.v3.patch
        157 kB
        Giovanni Matteo Fumarola
      4. YARN-3663-YARN-2915.v4.patch
        157 kB
        Giovanni Matteo Fumarola

        Issue Links

          Activity

          Hide
          giovanni.fumarola Giovanni Matteo Fumarola added a comment - - edited

          Carlo Curino, Subru Krishnan Can you guys take a look of the patch?

          The new tests will fail since they are a DataSource dependency fixed by HADOOP-13545.

          Only for heads up, this code has been running on few thousands of nodes for 6 month.

          Show
          giovanni.fumarola Giovanni Matteo Fumarola added a comment - - edited Carlo Curino , Subru Krishnan Can you guys take a look of the patch? The new tests will fail since they are a DataSource dependency fixed by HADOOP-13545 . Only for heads up, this code has been running on few thousands of nodes for 6 month.
          Hide
          curino Carlo Curino added a comment -

          Now that HADOOP-13545 is committed I kicked yetus to run the usual QA.

          Show
          curino Carlo Curino added a comment - Now that HADOOP-13545 is committed I kicked yetus to run the usual QA.
          Hide
          curino Carlo Curino added a comment -

          Giovanni Matteo Fumarola, the current patch doesn't seem to contain the definition of store procedures except for the HSQLDB tests.

          I think we need to include at least the SQLServer version you are currently running in production and open a JIRA under the Federation V2 umbrella YARN-5597 for a MySQL or DB-agnostic translation of the store procedures---which will be less performant but convenient.
          I know you were hesitating due to the SQLServer JDBC dependency. I think this is not an issue anymore since MS open-sourced the JDBC driver for SQLServer: *https://blogs.msdn.microsoft.com/jdbcteam/2016/11/17/open-source-jdbc-maven/*

          Please check whether things compile and run well using that dependency (pom entry below).

          <dependency>
              <groupId>com.microsoft.sqlserver</groupId>
              <artifactId>mssql-jdbc</artifactId>
              <version>6.1.0.jre7</version>
          </dependency>
          
          Show
          curino Carlo Curino added a comment - Giovanni Matteo Fumarola , the current patch doesn't seem to contain the definition of store procedures except for the HSQLDB tests. I think we need to include at least the SQLServer version you are currently running in production and open a JIRA under the Federation V2 umbrella YARN-5597 for a MySQL or DB-agnostic translation of the store procedures---which will be less performant but convenient. I know you were hesitating due to the SQLServer JDBC dependency. I think this is not an issue anymore since MS open-sourced the JDBC driver for SQLServer: * https://blogs.msdn.microsoft.com/jdbcteam/2016/11/17/open-source-jdbc-maven/* Please check whether things compile and run well using that dependency (pom entry below). <dependency>     <groupId>com.microsoft.sqlserver</groupId>     <artifactId>mssql-jdbc</artifactId>     <version>6.1.0.jre7</version> </dependency>
          Hide
          hadoopqa Hadoop QA added a comment -
          -1 overall



          Vote Subsystem Runtime Comment
          0 reexec 0m 19s Docker mode activated.
          +1 @author 0m 0s The patch does not contain any @author tags.
          +1 test4tests 0m 0s The patch appears to include 4 new or modified test files.
          0 mvndep 2m 7s Maven dependency ordering for branch
          +1 mvninstall 16m 48s YARN-2915 passed
          +1 compile 19m 22s YARN-2915 passed
          +1 checkstyle 2m 23s YARN-2915 passed
          +1 mvnsite 1m 34s YARN-2915 passed
          +1 mvneclipse 1m 7s YARN-2915 passed
          0 findbugs 0m 0s Skipped patched modules with no Java source: hadoop-project
          +1 findbugs 2m 52s YARN-2915 passed
          +1 javadoc 1m 24s YARN-2915 passed
          0 mvndep 0m 22s Maven dependency ordering for patch
          +1 mvninstall 1m 12s the patch passed
          +1 compile 16m 51s the patch passed
          +1 javac 16m 51s the patch passed
          -0 checkstyle 2m 5s root: The patch generated 8 new + 206 unchanged - 0 fixed = 214 total (was 206)
          +1 mvnsite 1m 41s the patch passed
          +1 mvneclipse 1m 15s the patch passed
          +1 whitespace 0m 0s The patch has no whitespace issues.
          +1 xml 0m 3s The patch has no ill-formed XML file.
          0 findbugs 0m 0s Skipped patched modules with no Java source: hadoop-project
          -1 findbugs 1m 7s hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-common generated 3 new + 0 unchanged - 0 fixed = 3 total (was 0)
          -1 javadoc 0m 29s hadoop-yarn-server-common in the patch failed.
          +1 unit 0m 21s hadoop-project in the patch passed.
          -1 unit 0m 39s hadoop-yarn-api in the patch failed.
          -1 unit 1m 21s hadoop-yarn-server-common in the patch failed.
          +1 asflicense 0m 40s The patch does not generate ASF License warnings.
          103m 23s



          Reason Tests
          FindBugs module:hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-common
            org.apache.hadoop.yarn.server.federation.store.impl.SQLFederationStateStore.getApplicationsHomeSubCluster(GetApplicationsHomeSubClusterRequest) may fail to clean up java.sql.ResultSet Obligation to clean up resource created at SQLFederationStateStore.java:up java.sql.ResultSet Obligation to clean up resource created at SQLFederationStateStore.java:[line 689] is not discharged
            org.apache.hadoop.yarn.server.federation.store.impl.SQLFederationStateStore.getPoliciesConfigurations(GetSubClusterPoliciesConfigurationsRequest) may fail to clean up java.sql.ResultSet Obligation to clean up resource created at SQLFederationStateStore.java:up java.sql.ResultSet Obligation to clean up resource created at SQLFederationStateStore.java:[line 884] is not discharged
            org.apache.hadoop.yarn.server.federation.store.impl.SQLFederationStateStore.getSubClusters(GetSubClustersInfoRequest) may fail to clean up java.sql.ResultSet Obligation to clean up resource created at SQLFederationStateStore.java:up java.sql.ResultSet Obligation to clean up resource created at SQLFederationStateStore.java:[line 437] is not discharged
          Failed junit tests hadoop.yarn.conf.TestYarnConfigurationFields
            hadoop.yarn.server.federation.store.impl.TestSQLFederationStateStore



          Subsystem Report/Notes
          Docker Image:yetus/hadoop:a9ad5d6
          JIRA Issue YARN-3663
          JIRA Patch URL https://issues.apache.org/jira/secure/attachment/12862899/YARN-3663-YARN-2915.v1.patch
          Optional Tests asflicense compile javac javadoc mvninstall mvnsite unit xml findbugs checkstyle
          uname Linux 8a4fdb3d8cf5 3.13.0-106-generic #153-Ubuntu SMP Tue Dec 6 15:44:32 UTC 2016 x86_64 x86_64 x86_64 GNU/Linux
          Build tool maven
          Personality /testptch/hadoop/patchprocess/precommit/personality/provided.sh
          git revision YARN-2915 / a7585da
          Default Java 1.8.0_121
          findbugs v3.0.0
          checkstyle https://builds.apache.org/job/PreCommit-YARN-Build/15588/artifact/patchprocess/diff-checkstyle-root.txt
          findbugs https://builds.apache.org/job/PreCommit-YARN-Build/15588/artifact/patchprocess/new-findbugs-hadoop-yarn-project_hadoop-yarn_hadoop-yarn-server_hadoop-yarn-server-common.html
          javadoc https://builds.apache.org/job/PreCommit-YARN-Build/15588/artifact/patchprocess/patch-javadoc-hadoop-yarn-project_hadoop-yarn_hadoop-yarn-server_hadoop-yarn-server-common.txt
          unit https://builds.apache.org/job/PreCommit-YARN-Build/15588/artifact/patchprocess/patch-unit-hadoop-yarn-project_hadoop-yarn_hadoop-yarn-api.txt
          unit https://builds.apache.org/job/PreCommit-YARN-Build/15588/artifact/patchprocess/patch-unit-hadoop-yarn-project_hadoop-yarn_hadoop-yarn-server_hadoop-yarn-server-common.txt
          Test Results https://builds.apache.org/job/PreCommit-YARN-Build/15588/testReport/
          modules C: hadoop-project hadoop-yarn-project/hadoop-yarn/hadoop-yarn-api hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-common U: .
          Console output https://builds.apache.org/job/PreCommit-YARN-Build/15588/console
          Powered by Apache Yetus 0.5.0-SNAPSHOT http://yetus.apache.org

          This message was automatically generated.

          Show
          hadoopqa Hadoop QA added a comment - -1 overall Vote Subsystem Runtime Comment 0 reexec 0m 19s Docker mode activated. +1 @author 0m 0s The patch does not contain any @author tags. +1 test4tests 0m 0s The patch appears to include 4 new or modified test files. 0 mvndep 2m 7s Maven dependency ordering for branch +1 mvninstall 16m 48s YARN-2915 passed +1 compile 19m 22s YARN-2915 passed +1 checkstyle 2m 23s YARN-2915 passed +1 mvnsite 1m 34s YARN-2915 passed +1 mvneclipse 1m 7s YARN-2915 passed 0 findbugs 0m 0s Skipped patched modules with no Java source: hadoop-project +1 findbugs 2m 52s YARN-2915 passed +1 javadoc 1m 24s YARN-2915 passed 0 mvndep 0m 22s Maven dependency ordering for patch +1 mvninstall 1m 12s the patch passed +1 compile 16m 51s the patch passed +1 javac 16m 51s the patch passed -0 checkstyle 2m 5s root: The patch generated 8 new + 206 unchanged - 0 fixed = 214 total (was 206) +1 mvnsite 1m 41s the patch passed +1 mvneclipse 1m 15s the patch passed +1 whitespace 0m 0s The patch has no whitespace issues. +1 xml 0m 3s The patch has no ill-formed XML file. 0 findbugs 0m 0s Skipped patched modules with no Java source: hadoop-project -1 findbugs 1m 7s hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-common generated 3 new + 0 unchanged - 0 fixed = 3 total (was 0) -1 javadoc 0m 29s hadoop-yarn-server-common in the patch failed. +1 unit 0m 21s hadoop-project in the patch passed. -1 unit 0m 39s hadoop-yarn-api in the patch failed. -1 unit 1m 21s hadoop-yarn-server-common in the patch failed. +1 asflicense 0m 40s The patch does not generate ASF License warnings. 103m 23s Reason Tests FindBugs module:hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-common   org.apache.hadoop.yarn.server.federation.store.impl.SQLFederationStateStore.getApplicationsHomeSubCluster(GetApplicationsHomeSubClusterRequest) may fail to clean up java.sql.ResultSet Obligation to clean up resource created at SQLFederationStateStore.java:up java.sql.ResultSet Obligation to clean up resource created at SQLFederationStateStore.java: [line 689] is not discharged   org.apache.hadoop.yarn.server.federation.store.impl.SQLFederationStateStore.getPoliciesConfigurations(GetSubClusterPoliciesConfigurationsRequest) may fail to clean up java.sql.ResultSet Obligation to clean up resource created at SQLFederationStateStore.java:up java.sql.ResultSet Obligation to clean up resource created at SQLFederationStateStore.java: [line 884] is not discharged   org.apache.hadoop.yarn.server.federation.store.impl.SQLFederationStateStore.getSubClusters(GetSubClustersInfoRequest) may fail to clean up java.sql.ResultSet Obligation to clean up resource created at SQLFederationStateStore.java:up java.sql.ResultSet Obligation to clean up resource created at SQLFederationStateStore.java: [line 437] is not discharged Failed junit tests hadoop.yarn.conf.TestYarnConfigurationFields   hadoop.yarn.server.federation.store.impl.TestSQLFederationStateStore Subsystem Report/Notes Docker Image:yetus/hadoop:a9ad5d6 JIRA Issue YARN-3663 JIRA Patch URL https://issues.apache.org/jira/secure/attachment/12862899/YARN-3663-YARN-2915.v1.patch Optional Tests asflicense compile javac javadoc mvninstall mvnsite unit xml findbugs checkstyle uname Linux 8a4fdb3d8cf5 3.13.0-106-generic #153-Ubuntu SMP Tue Dec 6 15:44:32 UTC 2016 x86_64 x86_64 x86_64 GNU/Linux Build tool maven Personality /testptch/hadoop/patchprocess/precommit/personality/provided.sh git revision YARN-2915 / a7585da Default Java 1.8.0_121 findbugs v3.0.0 checkstyle https://builds.apache.org/job/PreCommit-YARN-Build/15588/artifact/patchprocess/diff-checkstyle-root.txt findbugs https://builds.apache.org/job/PreCommit-YARN-Build/15588/artifact/patchprocess/new-findbugs-hadoop-yarn-project_hadoop-yarn_hadoop-yarn-server_hadoop-yarn-server-common.html javadoc https://builds.apache.org/job/PreCommit-YARN-Build/15588/artifact/patchprocess/patch-javadoc-hadoop-yarn-project_hadoop-yarn_hadoop-yarn-server_hadoop-yarn-server-common.txt unit https://builds.apache.org/job/PreCommit-YARN-Build/15588/artifact/patchprocess/patch-unit-hadoop-yarn-project_hadoop-yarn_hadoop-yarn-api.txt unit https://builds.apache.org/job/PreCommit-YARN-Build/15588/artifact/patchprocess/patch-unit-hadoop-yarn-project_hadoop-yarn_hadoop-yarn-server_hadoop-yarn-server-common.txt Test Results https://builds.apache.org/job/PreCommit-YARN-Build/15588/testReport/ modules C: hadoop-project hadoop-yarn-project/hadoop-yarn/hadoop-yarn-api hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-common U: . Console output https://builds.apache.org/job/PreCommit-YARN-Build/15588/console Powered by Apache Yetus 0.5.0-SNAPSHOT http://yetus.apache.org This message was automatically generated.
          Hide
          giovanni.fumarola Giovanni Matteo Fumarola added a comment -

          Thanks Carlo Curino for your feedback.

          I added SQLServer store procs in the patch, and MSSQL jdbc runtime dependency in hadoop-yarn-server-common as you suggested. Since I am adding a new dependency I updated the license information as well. I fixed the findbugs errors and the failed tests.

          Show
          giovanni.fumarola Giovanni Matteo Fumarola added a comment - Thanks Carlo Curino for your feedback. I added SQLServer store procs in the patch, and MSSQL jdbc runtime dependency in hadoop-yarn-server-common as you suggested. Since I am adding a new dependency I updated the license information as well. I fixed the findbugs errors and the failed tests.
          Hide
          hadoopqa Hadoop QA added a comment -
          -1 overall



          Vote Subsystem Runtime Comment
          0 reexec 15m 53s Docker mode activated.
          +1 @author 0m 0s The patch does not contain any @author tags.
          +1 test4tests 0m 0s The patch appears to include 22 new or modified test files.
          0 mvndep 1m 28s Maven dependency ordering for branch
          +1 mvninstall 14m 39s YARN-2915 passed
          +1 compile 16m 52s YARN-2915 passed
          +1 checkstyle 2m 0s YARN-2915 passed
          +1 mvnsite 11m 12s YARN-2915 passed
          +1 mvneclipse 2m 53s YARN-2915 passed
          0 findbugs 0m 0s Skipped patched modules with no Java source: hadoop-project .
          +1 findbugs 3m 22s YARN-2915 passed
          +1 javadoc 4m 43s YARN-2915 passed
          0 mvndep 0m 18s Maven dependency ordering for patch
          +1 mvninstall 15m 30s the patch passed
          +1 compile 13m 59s the patch passed
          +1 javac 13m 59s the patch passed
          -0 checkstyle 2m 1s root: The patch generated 2 new + 212 unchanged - 0 fixed = 214 total (was 212)
          +1 mvnsite 9m 57s the patch passed
          +1 mvneclipse 1m 54s the patch passed
          +1 whitespace 0m 0s The patch has no whitespace issues.
          +1 xml 0m 3s The patch has no ill-formed XML file.
          0 findbugs 0m 0s Skipped patched modules with no Java source: hadoop-project .
          +1 findbugs 4m 3s the patch passed
          +1 javadoc 5m 38s the patch passed
          -1 unit 18m 26s root in the patch failed.
          -1 asflicense 0m 40s The patch generated 2 ASF License warnings.
          168m 44s



          Reason Tests
          Failed junit tests hadoop.ha.TestZKFailoverController



          Subsystem Report/Notes
          Docker Image:yetus/hadoop:612578f
          JIRA Issue YARN-3663
          JIRA Patch URL https://issues.apache.org/jira/secure/attachment/12863140/YARN-3663-YARN-2915.v2.patch
          Optional Tests asflicense compile javac javadoc mvninstall mvnsite unit xml findbugs checkstyle
          uname Linux 34e602597828 3.13.0-106-generic #153-Ubuntu SMP Tue Dec 6 15:44:32 UTC 2016 x86_64 x86_64 x86_64 GNU/Linux
          Build tool maven
          Personality /testptch/hadoop/patchprocess/precommit/personality/provided.sh
          git revision YARN-2915 / 5eacf0c
          Default Java 1.8.0_121
          findbugs v3.0.0
          checkstyle https://builds.apache.org/job/PreCommit-YARN-Build/15612/artifact/patchprocess/diff-checkstyle-root.txt
          unit https://builds.apache.org/job/PreCommit-YARN-Build/15612/artifact/patchprocess/patch-unit-root.txt
          Test Results https://builds.apache.org/job/PreCommit-YARN-Build/15612/testReport/
          asflicense https://builds.apache.org/job/PreCommit-YARN-Build/15612/artifact/patchprocess/patch-asflicense-problems.txt
          modules C: hadoop-project hadoop-yarn-project/hadoop-yarn/hadoop-yarn-api hadoop-yarn-project/hadoop-yarn/hadoop-yarn-common hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-common . U: .
          Console output https://builds.apache.org/job/PreCommit-YARN-Build/15612/console
          Powered by Apache Yetus 0.5.0-SNAPSHOT http://yetus.apache.org

          This message was automatically generated.

          Show
          hadoopqa Hadoop QA added a comment - -1 overall Vote Subsystem Runtime Comment 0 reexec 15m 53s Docker mode activated. +1 @author 0m 0s The patch does not contain any @author tags. +1 test4tests 0m 0s The patch appears to include 22 new or modified test files. 0 mvndep 1m 28s Maven dependency ordering for branch +1 mvninstall 14m 39s YARN-2915 passed +1 compile 16m 52s YARN-2915 passed +1 checkstyle 2m 0s YARN-2915 passed +1 mvnsite 11m 12s YARN-2915 passed +1 mvneclipse 2m 53s YARN-2915 passed 0 findbugs 0m 0s Skipped patched modules with no Java source: hadoop-project . +1 findbugs 3m 22s YARN-2915 passed +1 javadoc 4m 43s YARN-2915 passed 0 mvndep 0m 18s Maven dependency ordering for patch +1 mvninstall 15m 30s the patch passed +1 compile 13m 59s the patch passed +1 javac 13m 59s the patch passed -0 checkstyle 2m 1s root: The patch generated 2 new + 212 unchanged - 0 fixed = 214 total (was 212) +1 mvnsite 9m 57s the patch passed +1 mvneclipse 1m 54s the patch passed +1 whitespace 0m 0s The patch has no whitespace issues. +1 xml 0m 3s The patch has no ill-formed XML file. 0 findbugs 0m 0s Skipped patched modules with no Java source: hadoop-project . +1 findbugs 4m 3s the patch passed +1 javadoc 5m 38s the patch passed -1 unit 18m 26s root in the patch failed. -1 asflicense 0m 40s The patch generated 2 ASF License warnings. 168m 44s Reason Tests Failed junit tests hadoop.ha.TestZKFailoverController Subsystem Report/Notes Docker Image:yetus/hadoop:612578f JIRA Issue YARN-3663 JIRA Patch URL https://issues.apache.org/jira/secure/attachment/12863140/YARN-3663-YARN-2915.v2.patch Optional Tests asflicense compile javac javadoc mvninstall mvnsite unit xml findbugs checkstyle uname Linux 34e602597828 3.13.0-106-generic #153-Ubuntu SMP Tue Dec 6 15:44:32 UTC 2016 x86_64 x86_64 x86_64 GNU/Linux Build tool maven Personality /testptch/hadoop/patchprocess/precommit/personality/provided.sh git revision YARN-2915 / 5eacf0c Default Java 1.8.0_121 findbugs v3.0.0 checkstyle https://builds.apache.org/job/PreCommit-YARN-Build/15612/artifact/patchprocess/diff-checkstyle-root.txt unit https://builds.apache.org/job/PreCommit-YARN-Build/15612/artifact/patchprocess/patch-unit-root.txt Test Results https://builds.apache.org/job/PreCommit-YARN-Build/15612/testReport/ asflicense https://builds.apache.org/job/PreCommit-YARN-Build/15612/artifact/patchprocess/patch-asflicense-problems.txt modules C: hadoop-project hadoop-yarn-project/hadoop-yarn/hadoop-yarn-api hadoop-yarn-project/hadoop-yarn/hadoop-yarn-common hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-common . U: . Console output https://builds.apache.org/job/PreCommit-YARN-Build/15612/console Powered by Apache Yetus 0.5.0-SNAPSHOT http://yetus.apache.org This message was automatically generated.
          Hide
          giovanni.fumarola Giovanni Matteo Fumarola added a comment -

          The test failed and the asflicense failure are not related to the patch.
          About checkstyle, we had the same issue with those 2 functions also in YARN-5407.

          Show
          giovanni.fumarola Giovanni Matteo Fumarola added a comment - The test failed and the asflicense failure are not related to the patch. About checkstyle, we had the same issue with those 2 functions also in YARN-5407 .
          Hide
          curino Carlo Curino added a comment - - edited

          Thanks Giovanni Matteo Fumarola for the updated patch.

          Conf:

          1. You set YarnConfiguration.DEFAULT_FEDERATION_STATESTORE_SQL_MAXCONNECTIONS to 1. Isn't this too small? What values you commonly use? I would put a conservative but not overly tight value, otherwise users are forced to learn/modify more params.
          2. We should omit {{yarn.federation.state-store.sql.max-connections} from yarn-default.xml like you do for all other params in this patch.

          In SQLFederationStateStore

          1. For most fields except CALL_SP_GET_ALL_POLICY_CONFIGURATIONS you simply use the plural to differentiate the getter of one or all items. Be consistent (e.g., remove the ALL here, or add it everywhere else)
          2. Minor: userNameDB --> userName, passwordDB --> password
          3. When you throw the exceptions (e.g., subcluster registration), it might be nice in the message to include the sub-clusterID / ip or any other info one can use to debug.
          4. Can you comment on why we are using: FederationStateStoreErrorCode? They don't seem to be connected to SQL error codes, and they are not used anywhere else (we normally use named exception, which are easier to understand/track).
          5. at line 277-278: formatting
          6. We should try to remove redundance, e.g., you have lots of things that look like this:
                  try {
                  FederationMembershipStateStoreInputValidator
                      .validateGetSubClusterInfoRequest(subClusterRequest);
                } catch (FederationStateStoreInvalidInputException e) {
                  FederationStateStoreUtils.logAndThrowInvalidInputException(LOG,
                      "Unable to obtain the infomation about the SubCluster. "
                          + e.getMessage());
                }
              

            They could be factored out to FederationMembershipStateStoreInputValidator.validate(subClusterRequest) where the type of input param is used to differentiate the method, and the logAndThrowInvalidInputException is done on that side. Same goes for checkSubClusterInfo.

          7. Similarly to the above we should try to factor out the very repetitive code to create connection/statements, set params, run, and throw. I don't have a specific advise on this, but the code is mostly copy and paste, which we should avoid.
          8. Move the fromStringToSubClusterState to the SubclusterState class (and call it fromString().
          9. Why getSubCluster and getSubClusters use different mechanics for return values? (registered params vs ResultSet)? Might be worth to be consistent (probably using ResultSet).
          10. Line 540: Is this behavior (overwrite existing) consistent with general YARN? (I think so, but want to check)
          11. Some of the log are a bit vague {{LOG.debug("Got the information about the specified application }} say spefically what info where gotten
          12. if you use LOG.debug consider to prefix it with a check if we are in debug mode (save time/objects creations for the String that are then not used).
          13. You have several {{ if (cstmt.getInt(2) != 1) }} ROWCOUNT checks. This mix the no tuple where changed to multiple tuple where changed. Distinguishing the two cases, might help debug (do we have duplicates in DB, or the entry was not found). (Not mandatory, just somethign to consider)
          14. setPolicyConfiguration You are doing cstmt.setBytes(3, new byte[policyConf.getParams().remaining()]); which adds an empty byte[] instead of what is coming in input.
          15. getCurrentVersion and loadVersion throw a NotSupportedException or something of the sort, a silent return null is easy to confuse people. (I know the full version will be in V2, let's just have a clear breakage if someone try to use this methods).

          (.... to be continued ...)

          (.... continued...)
          In FederationStateStoreInputValidator:

          1. Please consider to rename the validate methods (ok to have separate JIRA for this).

          In FederationStateStoreUtils

          1. You log at info and debug level inconsistently for the set* you added. I would suggest debug for all.

          In HSQLDBFederationStateStore

          1. the empty constructor with super() is redundant, just omit the constructor alltogether.
          2. I think the code would be more readable if all the schema was in a well-formatted hsqldb-federation-schema.sql (or broken down like the SQLServer one) and the code was reading from file the statements and executing them.
          3. The use of SELECT * is kind of dangerous, because it hides field renames/moves and other schema evolution issues, that might lead to hard to debug, I would use explicit named fields always
          4. shouldn't SP_SUBCLUSTERHEARTBEAT update the lastHeartBeat field? More concerning, why do the tests miss this? If I am correct, you should fix this and more importantly add coverage for this in the tests.
          5. Please comment on line 136 {{HAVING COUNT = 0 }}, what are you doing there?
          6. in init I don't like the silent catching of exception... why do we have it? Since this is only used for testing I rather have a loud failure than a silent moving on.

          Store Procedures:

          1. Can you create only one or two files called sqlserver-federation-schema.sql and sqlserver-federation-storeproc.sql? Makes the reading easier, and git will do well handling row-level conflicts in most cases.
          2. applicationHomeSubcluster switch from locale to UTC time consistently with other timestamps.
          3. policies the params should be much bigger than 512. This might contain complex info about how each queue/user should be mapped across multiple sub-clusters. I would argue for a fairly large amount of space (we should have a small number of policies anyway). This is particularly important if/when we will use things like Hekaton (since schema changes are majorly painful).
          4. sp_deleteExpiredApplicationsHomeSubCluster the -10 should not be hard-coded. How was that chosen? What is that expressed in? I would argue this should be an input param, and the value should come up from somewhere in yarn conf.
          5. sp_deregisterSubCluster why is STATE configurable? Does it takes different values?
          6. sp_registerSubCluster is @capability VARCHAR(6000) big enough? Same question for the 256 char for the addresses.
          7. (NOT TO FIX NOW) I don't love the fact that if two subcluster are mis-configured to clash on subclusterID we silently override each other entries... I know this behavior is needed to support HA, but we should (in V2) improve and detect issues like this.
          8. sp_getApplicationsHomeSubCluster, sp_getPoliciesConfigurations, sp_getSubClusters use of SELECT * should be avoided.
          9. sp_getPolicyConfiguration, sp_setPolicyConfiguration size of param should be increased.
          10. sp_getSubCluster why using the outparams instead of ResultSet?

          TOP LEVEL CONCERNS:

          1. What is our schema-evolution story? Should we introduce views?
          2. A concern are SQL injections, for example what happens if in SP_DELETEAPPLICATIONHOMESUBCLUSTER someone sends in an applicationId = "(SELECT * FROM applications)" ? Do we remove all apps? (maybe bad example, but you see what I mean, possibly this is the FederationStateStoreInputValidator responsibility, let's double check it is done properly).
          3. In v2 it would be nice to have a MySQL version as well, consider abstracting some of the SQL stuff, are we architecturally ready to have multiple SQL dialects for store proc etc?
          4. Do we have enough test coverage? E.g., the HSQLDB issues on updating heartbeat is concerning. Please double check the tests cover enough of the possible issues (you saw some in production, replicate them in the tests).
          Show
          curino Carlo Curino added a comment - - edited Thanks Giovanni Matteo Fumarola for the updated patch. Conf: You set YarnConfiguration.DEFAULT_FEDERATION_STATESTORE_SQL_MAXCONNECTIONS to 1. Isn't this too small? What values you commonly use? I would put a conservative but not overly tight value, otherwise users are forced to learn/modify more params. We should omit {{yarn.federation.state-store.sql.max-connections} from yarn-default.xml like you do for all other params in this patch. In SQLFederationStateStore For most fields except CALL_SP_GET_ALL_POLICY_CONFIGURATIONS you simply use the plural to differentiate the getter of one or all items. Be consistent (e.g., remove the ALL here, or add it everywhere else) Minor: userNameDB --> userName, passwordDB --> password When you throw the exceptions (e.g., subcluster registration), it might be nice in the message to include the sub-clusterID / ip or any other info one can use to debug. Can you comment on why we are using: FederationStateStoreErrorCode ? They don't seem to be connected to SQL error codes, and they are not used anywhere else (we normally use named exception, which are easier to understand/track). at line 277-278: formatting We should try to remove redundance, e.g., you have lots of things that look like this: try { FederationMembershipStateStoreInputValidator .validateGetSubClusterInfoRequest(subClusterRequest); } catch (FederationStateStoreInvalidInputException e) { FederationStateStoreUtils.logAndThrowInvalidInputException(LOG, "Unable to obtain the infomation about the SubCluster. " + e.getMessage()); } They could be factored out to FederationMembershipStateStoreInputValidator.validate(subClusterRequest) where the type of input param is used to differentiate the method, and the logAndThrowInvalidInputException is done on that side. Same goes for checkSubClusterInfo . Similarly to the above we should try to factor out the very repetitive code to create connection/statements, set params, run, and throw. I don't have a specific advise on this, but the code is mostly copy and paste, which we should avoid. Move the fromStringToSubClusterState to the SubclusterState class (and call it fromString(). Why getSubCluster and getSubClusters use different mechanics for return values? (registered params vs ResultSet)? Might be worth to be consistent (probably using ResultSet). Line 540: Is this behavior (overwrite existing) consistent with general YARN? (I think so, but want to check) Some of the log are a bit vague {{LOG.debug("Got the information about the specified application }} say spefically what info where gotten if you use LOG.debug consider to prefix it with a check if we are in debug mode (save time/objects creations for the String that are then not used). You have several {{ if (cstmt.getInt(2) != 1) }} ROWCOUNT checks. This mix the no tuple where changed to multiple tuple where changed. Distinguishing the two cases, might help debug (do we have duplicates in DB, or the entry was not found). (Not mandatory, just somethign to consider) setPolicyConfiguration You are doing cstmt.setBytes(3, new byte [policyConf.getParams().remaining()] ); which adds an empty byte[] instead of what is coming in input. getCurrentVersion and loadVersion throw a NotSupportedException or something of the sort, a silent return null is easy to confuse people. (I know the full version will be in V2, let's just have a clear breakage if someone try to use this methods). (.... to be continued ...) (.... continued...) In FederationStateStoreInputValidator : Please consider to rename the validate methods (ok to have separate JIRA for this). In FederationStateStoreUtils You log at info and debug level inconsistently for the set* you added. I would suggest debug for all. In HSQLDBFederationStateStore the empty constructor with super() is redundant, just omit the constructor alltogether. I think the code would be more readable if all the schema was in a well-formatted hsqldb-federation-schema.sql (or broken down like the SQLServer one) and the code was reading from file the statements and executing them. The use of SELECT * is kind of dangerous, because it hides field renames/moves and other schema evolution issues, that might lead to hard to debug, I would use explicit named fields always shouldn't SP_SUBCLUSTERHEARTBEAT update the lastHeartBeat field? More concerning, why do the tests miss this? If I am correct, you should fix this and more importantly add coverage for this in the tests. Please comment on line 136 {{HAVING COUNT = 0 }}, what are you doing there? in init I don't like the silent catching of exception... why do we have it? Since this is only used for testing I rather have a loud failure than a silent moving on. Store Procedures: Can you create only one or two files called sqlserver-federation-schema.sql and sqlserver-federation-storeproc.sql? Makes the reading easier, and git will do well handling row-level conflicts in most cases. applicationHomeSubcluster switch from locale to UTC time consistently with other timestamps. policies the params should be much bigger than 512. This might contain complex info about how each queue/user should be mapped across multiple sub-clusters. I would argue for a fairly large amount of space (we should have a small number of policies anyway). This is particularly important if/when we will use things like Hekaton (since schema changes are majorly painful). sp_deleteExpiredApplicationsHomeSubCluster the -10 should not be hard-coded. How was that chosen? What is that expressed in? I would argue this should be an input param, and the value should come up from somewhere in yarn conf. sp_deregisterSubCluster why is STATE configurable? Does it takes different values? sp_registerSubCluster is @capability VARCHAR(6000) big enough? Same question for the 256 char for the addresses. (NOT TO FIX NOW) I don't love the fact that if two subcluster are mis-configured to clash on subclusterID we silently override each other entries... I know this behavior is needed to support HA, but we should (in V2) improve and detect issues like this. sp_getApplicationsHomeSubCluster, sp_getPoliciesConfigurations, sp_getSubClusters use of SELECT * should be avoided. sp_getPolicyConfiguration, sp_setPolicyConfiguration size of param should be increased. sp_getSubCluster why using the outparams instead of ResultSet? TOP LEVEL CONCERNS: What is our schema-evolution story? Should we introduce views? A concern are SQL injections, for example what happens if in SP_DELETEAPPLICATIONHOMESUBCLUSTER someone sends in an applicationId = "(SELECT * FROM applications)" ? Do we remove all apps? (maybe bad example, but you see what I mean, possibly this is the FederationStateStoreInputValidator responsibility, let's double check it is done properly). In v2 it would be nice to have a MySQL version as well, consider abstracting some of the SQL stuff, are we architecturally ready to have multiple SQL dialects for store proc etc? Do we have enough test coverage? E.g., the HSQLDB issues on updating heartbeat is concerning. Please double check the tests cover enough of the possible issues (you saw some in production, replicate them in the tests).
          Hide
          giovanni.fumarola Giovanni Matteo Fumarola added a comment - - edited

          Thanks Carlo Curino for the accurate review. Let me go in order.

          Conf
          1. Every process should have at most 1 connection open to the StateStore except for Router. It has 10. This is for performance and scalability reason of the Database. FederationInterceptor open one connection for each application running in the node. More than that can cause a possible DoS to StateStore.
          2. Done.

          SQLFederationStateStore
          1. Done.
          2. Done.
          3. Done.
          4. Done.
          5. When I do auto-format from eclipse by using hadoop-format.xml it gives me it in 2 lines.
          6. Done.
          7. We can improve/reduce the amount of code later under a different Jira. I am quite confident in this part and I would avoid to change it.
          8. Done.
          9. We can improve/reduce the amount of code later under a different Jira. I am quite confident in this part and I would avoid to change it.
          10. That is a safe check, since admin's cluster can setup their own database.
          11. Done.
          12. Done.
          13. Done.
          14. Done.
          15. Done.

          FederationStateStoreUtils
          1. Done

          HSQLDBFederationStateStore
          1. Done
          2. I tried to move in a file but I am adding a lot of complexity and I did not find a way to let it work. I tried to format in SQL standard inside the file itself but the autoformat messes it up.
          3. Done.
          4. Fixed it - good catch.
          5. In that way I only insert a new application. That code is like if (! app exists) add app.
          6. Done.

          Stored procedure
          1. Done.
          2. Done.
          3. Increased to 6000.
          4. My bad, it was a for a performance test - introduced in the patch by mistake.
          5. We can DEREGISTER it or put it in a LOST state.
          6. We did not run in any issues with those settings. I would keep like this.
          7. Let's open a JIRA about it.
          8. Done.
          9. Increase to 6000.
          10. As I said before, let's keep this one since we have one stored procedure for each function. We can reduce/improve later.

          Top level concerns:
          1. The current design was not edited over the last year. However in case we need to evolve the schema we will need to add them.
          2. This cannot happen because we have input validators.
          3. For that scenario, we have to add only the stored procedure for the specific database.
          4. I edited some tests to cover some scenario we did not check. However, the current state covers all the possible scenarios.

          Show
          giovanni.fumarola Giovanni Matteo Fumarola added a comment - - edited Thanks Carlo Curino for the accurate review. Let me go in order. Conf 1. Every process should have at most 1 connection open to the StateStore except for Router. It has 10. This is for performance and scalability reason of the Database. FederationInterceptor open one connection for each application running in the node. More than that can cause a possible DoS to StateStore. 2. Done. SQLFederationStateStore 1. Done. 2. Done. 3. Done. 4. Done. 5. When I do auto-format from eclipse by using hadoop-format.xml it gives me it in 2 lines. 6. Done. 7. We can improve/reduce the amount of code later under a different Jira. I am quite confident in this part and I would avoid to change it. 8. Done. 9. We can improve/reduce the amount of code later under a different Jira. I am quite confident in this part and I would avoid to change it. 10. That is a safe check, since admin's cluster can setup their own database. 11. Done. 12. Done. 13. Done. 14. Done. 15. Done. FederationStateStoreUtils 1. Done HSQLDBFederationStateStore 1. Done 2. I tried to move in a file but I am adding a lot of complexity and I did not find a way to let it work. I tried to format in SQL standard inside the file itself but the autoformat messes it up. 3. Done. 4. Fixed it - good catch. 5. In that way I only insert a new application. That code is like if (! app exists) add app. 6. Done. Stored procedure 1. Done. 2. Done. 3. Increased to 6000. 4. My bad, it was a for a performance test - introduced in the patch by mistake. 5. We can DEREGISTER it or put it in a LOST state. 6. We did not run in any issues with those settings. I would keep like this. 7. Let's open a JIRA about it. 8. Done. 9. Increase to 6000. 10. As I said before, let's keep this one since we have one stored procedure for each function. We can reduce/improve later. Top level concerns: 1. The current design was not edited over the last year. However in case we need to evolve the schema we will need to add them. 2. This cannot happen because we have input validators. 3. For that scenario, we have to add only the stored procedure for the specific database. 4. I edited some tests to cover some scenario we did not check. However, the current state covers all the possible scenarios.
          Hide
          hadoopqa Hadoop QA added a comment -
          -1 overall



          Vote Subsystem Runtime Comment
          0 reexec 0m 21s Docker mode activated.
          +1 @author 0m 0s The patch does not contain any @author tags.
          +1 test4tests 0m 0s The patch appears to include 9 new or modified test files.
          0 mvndep 1m 30s Maven dependency ordering for branch
          +1 mvninstall 15m 25s YARN-2915 passed
          +1 compile 16m 6s YARN-2915 passed
          +1 checkstyle 1m 58s YARN-2915 passed
          +1 mvnsite 10m 0s YARN-2915 passed
          +1 mvneclipse 3m 1s YARN-2915 passed
          0 findbugs 0m 0s Skipped patched modules with no Java source: hadoop-project .
          +1 findbugs 2m 1s YARN-2915 passed
          +1 javadoc 4m 33s YARN-2915 passed
          0 mvndep 0m 20s Maven dependency ordering for patch
          +1 mvninstall 15m 11s the patch passed
          +1 compile 15m 50s the patch passed
          +1 javac 15m 50s the patch passed
          -0 checkstyle 2m 9s root: The patch generated 4 new + 214 unchanged - 0 fixed = 218 total (was 214)
          +1 mvnsite 10m 25s the patch passed
          +1 mvneclipse 1m 49s the patch passed
          +1 whitespace 0m 0s The patch has no whitespace issues.
          +1 xml 0m 2s The patch has no ill-formed XML file.
          0 findbugs 0m 0s Skipped patched modules with no Java source: hadoop-project .
          +1 findbugs 2m 22s the patch passed
          +1 javadoc 4m 43s the patch passed
          -1 unit 18m 1s root in the patch failed.
          -1 asflicense 0m 40s The patch generated 2 ASF License warnings.
          149m 59s



          Reason Tests
          Failed junit tests hadoop.ha.TestZKFailoverController



          Subsystem Report/Notes
          Docker Image:yetus/hadoop:612578f
          JIRA Issue YARN-3663
          JIRA Patch URL https://issues.apache.org/jira/secure/attachment/12864083/YARN-3663-YARN-2915.v3.patch
          Optional Tests asflicense compile javac javadoc mvninstall mvnsite unit xml findbugs checkstyle
          uname Linux e96643d4403e 3.13.0-106-generic #153-Ubuntu SMP Tue Dec 6 15:44:32 UTC 2016 x86_64 x86_64 x86_64 GNU/Linux
          Build tool maven
          Personality /testptch/hadoop/patchprocess/precommit/personality/provided.sh
          git revision YARN-2915 / 47451e0
          Default Java 1.8.0_121
          findbugs v3.0.0
          checkstyle https://builds.apache.org/job/PreCommit-YARN-Build/15680/artifact/patchprocess/diff-checkstyle-root.txt
          unit https://builds.apache.org/job/PreCommit-YARN-Build/15680/artifact/patchprocess/patch-unit-root.txt
          Test Results https://builds.apache.org/job/PreCommit-YARN-Build/15680/testReport/
          asflicense https://builds.apache.org/job/PreCommit-YARN-Build/15680/artifact/patchprocess/patch-asflicense-problems.txt
          modules C: hadoop-project hadoop-yarn-project/hadoop-yarn/hadoop-yarn-api hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-common . U: .
          Console output https://builds.apache.org/job/PreCommit-YARN-Build/15680/console
          Powered by Apache Yetus 0.5.0-SNAPSHOT http://yetus.apache.org

          This message was automatically generated.

          Show
          hadoopqa Hadoop QA added a comment - -1 overall Vote Subsystem Runtime Comment 0 reexec 0m 21s Docker mode activated. +1 @author 0m 0s The patch does not contain any @author tags. +1 test4tests 0m 0s The patch appears to include 9 new or modified test files. 0 mvndep 1m 30s Maven dependency ordering for branch +1 mvninstall 15m 25s YARN-2915 passed +1 compile 16m 6s YARN-2915 passed +1 checkstyle 1m 58s YARN-2915 passed +1 mvnsite 10m 0s YARN-2915 passed +1 mvneclipse 3m 1s YARN-2915 passed 0 findbugs 0m 0s Skipped patched modules with no Java source: hadoop-project . +1 findbugs 2m 1s YARN-2915 passed +1 javadoc 4m 33s YARN-2915 passed 0 mvndep 0m 20s Maven dependency ordering for patch +1 mvninstall 15m 11s the patch passed +1 compile 15m 50s the patch passed +1 javac 15m 50s the patch passed -0 checkstyle 2m 9s root: The patch generated 4 new + 214 unchanged - 0 fixed = 218 total (was 214) +1 mvnsite 10m 25s the patch passed +1 mvneclipse 1m 49s the patch passed +1 whitespace 0m 0s The patch has no whitespace issues. +1 xml 0m 2s The patch has no ill-formed XML file. 0 findbugs 0m 0s Skipped patched modules with no Java source: hadoop-project . +1 findbugs 2m 22s the patch passed +1 javadoc 4m 43s the patch passed -1 unit 18m 1s root in the patch failed. -1 asflicense 0m 40s The patch generated 2 ASF License warnings. 149m 59s Reason Tests Failed junit tests hadoop.ha.TestZKFailoverController Subsystem Report/Notes Docker Image:yetus/hadoop:612578f JIRA Issue YARN-3663 JIRA Patch URL https://issues.apache.org/jira/secure/attachment/12864083/YARN-3663-YARN-2915.v3.patch Optional Tests asflicense compile javac javadoc mvninstall mvnsite unit xml findbugs checkstyle uname Linux e96643d4403e 3.13.0-106-generic #153-Ubuntu SMP Tue Dec 6 15:44:32 UTC 2016 x86_64 x86_64 x86_64 GNU/Linux Build tool maven Personality /testptch/hadoop/patchprocess/precommit/personality/provided.sh git revision YARN-2915 / 47451e0 Default Java 1.8.0_121 findbugs v3.0.0 checkstyle https://builds.apache.org/job/PreCommit-YARN-Build/15680/artifact/patchprocess/diff-checkstyle-root.txt unit https://builds.apache.org/job/PreCommit-YARN-Build/15680/artifact/patchprocess/patch-unit-root.txt Test Results https://builds.apache.org/job/PreCommit-YARN-Build/15680/testReport/ asflicense https://builds.apache.org/job/PreCommit-YARN-Build/15680/artifact/patchprocess/patch-asflicense-problems.txt modules C: hadoop-project hadoop-yarn-project/hadoop-yarn/hadoop-yarn-api hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-common . U: . Console output https://builds.apache.org/job/PreCommit-YARN-Build/15680/console Powered by Apache Yetus 0.5.0-SNAPSHOT http://yetus.apache.org This message was automatically generated.
          Hide
          giovanni.fumarola Giovanni Matteo Fumarola added a comment -

          Same result from the previous execution.
          The test failed and the asflicense failure are not related to the patch.
          About checkstyle, we had the same issue with those 2 functions also in YARN-5407.

          Show
          giovanni.fumarola Giovanni Matteo Fumarola added a comment - Same result from the previous execution. The test failed and the asflicense failure are not related to the patch. About checkstyle, we had the same issue with those 2 functions also in YARN-5407 .
          Hide
          curino Carlo Curino added a comment -

          Giovanni Matteo Fumarola thanks for addressing my comments. I understand all your answers, and I am ok to postpone some of the refactorings I asked (we should think a bit more about especially for SQLFederationStateStore point 7).

          Overall the patch is +1, but please:

          1. Open all the follow-up JIRAs we mentioned as sub-tasks of YARN-5597 (federation v2)
          2. Address the first 2 checkstyle (UTC_CALENDAR)... I think it should be private, and the naming could be a standard utcCalendar.

          Patch is good to commit once that is done. If you get it done tonight or tomorrow morning, I will commit it tomorrow late morning, otherwise I asked Subru Krishnan to take-over reviewing and commit,
          as I will be traveling with limited connectivity for a few days.

          Show
          curino Carlo Curino added a comment - Giovanni Matteo Fumarola thanks for addressing my comments. I understand all your answers, and I am ok to postpone some of the refactorings I asked (we should think a bit more about especially for SQLFederationStateStore point 7). Overall the patch is +1, but please: Open all the follow-up JIRAs we mentioned as sub-tasks of YARN-5597 (federation v2) Address the first 2 checkstyle ( UTC_CALENDAR )... I think it should be private , and the naming could be a standard utcCalendar . Patch is good to commit once that is done. If you get it done tonight or tomorrow morning, I will commit it tomorrow late morning, otherwise I asked Subru Krishnan to take-over reviewing and commit, as I will be traveling with limited connectivity for a few days.
          Hide
          giovanni.fumarola Giovanni Matteo Fumarola added a comment -

          Thanks Carlo Curino. I just uploaded the new patch (V4). I am going to open a new jira about the optimization and refactoring.

          Show
          giovanni.fumarola Giovanni Matteo Fumarola added a comment - Thanks Carlo Curino . I just uploaded the new patch (V4). I am going to open a new jira about the optimization and refactoring.
          Hide
          curino Carlo Curino added a comment -

          Thanks Giovanni Matteo Fumarola for the contribution and addressing the review, I committed the patch to branch YARN-2915.

          Show
          curino Carlo Curino added a comment - Thanks Giovanni Matteo Fumarola for the contribution and addressing the review, I committed the patch to branch YARN-2915 .
          Hide
          giovanni.fumarola Giovanni Matteo Fumarola added a comment -

          For tracking the refactoring: YARN-6526

          Show
          giovanni.fumarola Giovanni Matteo Fumarola added a comment - For tracking the refactoring: YARN-6526

            People

            • Assignee:
              giovanni.fumarola Giovanni Matteo Fumarola
              Reporter:
              giovanni.fumarola Giovanni Matteo Fumarola
            • Votes:
              0 Vote for this issue
              Watchers:
              7 Start watching this issue

              Dates

              • Created:
                Updated:
                Resolved:

                Development