Uploaded image for project: 'Falcon'
  1. Falcon
  2. FALCON-790

Falcon UI to enable entity/process/feed edits and management

    Details

    • Type: New Feature
    • Status: Resolved
    • Priority: Critical
    • Resolution: Fixed
    • Affects Version/s: 0.6
    • Fix Version/s: 0.6.1
    • Component/s: ui
    • Labels:

      Description

      Falcon UI is currently read-only. As a step towards enabling a full fledged cluster/feed/process management and monitoring UI, the following features will be added to the existing UI:
      For the first phase -
      Entity Creation:
      1. allow to create a new cluster from the UI
      2. allow to create a new feed from the UI
      3. allow to create a new process from the UI
      4. allow to browse and import an entity XML and then edit
      5. Clone an existing entity and edit.

      Provisioning and managing:
      1. Submit an entity
      2. Schedule an entity
      3. Suspend an entity
      4. Delete an entity

      For the second Phase -
      1. integrating with Ambari view for login.

      1. 4.4.zip
        1.08 MB
        kenneth ho
      2. create feed_1.png
        256 kB
        kenneth ho
      3. falcon.log
        289 kB
        Armando Reyna
      4. FALCON-790-additional.patch
        4 kB
        Srikanth Sundarrajan
      5. FALCON-790-V4.patch
        12 kB
        kenneth ho
      6. landing.png
        66 kB
        kenneth ho
      7. new cluster.png
        336 kB
        kenneth ho
      8. new feed_2.png
        274 kB
        kenneth ho
      9. new feed_3.png
        114 kB
        kenneth ho
      10. new feed_4.png
        304 kB
        kenneth ho
      11. patch_1.0.1.patch
        1.81 MB
        Armando Reyna
      12. patch.tar.gz
        547 kB
        kenneth ho
      13. rat.txt.gz
        6.02 MB
        Srikanth Sundarrajan
      14. README.md
        1 kB
        Armando Reyna

        Issue Links

          Activity

          Hide
          kho@hortonworks.com kenneth ho added a comment -

          Landing Page

          Show
          kho@hortonworks.com kenneth ho added a comment - Landing Page
          Hide
          kho@hortonworks.com kenneth ho added a comment -

          create new cluster

          Show
          kho@hortonworks.com kenneth ho added a comment - create new cluster
          Hide
          kho@hortonworks.com kenneth ho added a comment -

          create feed page 1

          Show
          kho@hortonworks.com kenneth ho added a comment - create feed page 1
          Hide
          kho@hortonworks.com kenneth ho added a comment -

          create feed page 2

          Show
          kho@hortonworks.com kenneth ho added a comment - create feed page 2
          Hide
          svenkat Venkatesh Seetharam added a comment -

          kenneth ho, this is a very large patch, 3 MB and will be hard to review. Can you break this into subtasks for each screen may be and then will be easier to review the code. What about tests for these? Can the tests go into falcon-regression? Thanks!

          Show
          svenkat Venkatesh Seetharam added a comment - kenneth ho , this is a very large patch, 3 MB and will be hard to review. Can you break this into subtasks for each screen may be and then will be easier to review the code. What about tests for these? Can the tests go into falcon-regression? Thanks!
          Hide
          sriksun Srikanth Sundarrajan added a comment -

          +1 to Venkatesh Seetharam's comment. It is going to be fairly challenging to review such a large patch in one go. kenneth ho, Can this be broken up into manageable chunks ?

          Show
          sriksun Srikanth Sundarrajan added a comment - +1 to Venkatesh Seetharam 's comment. It is going to be fairly challenging to review such a large patch in one go. kenneth ho , Can this be broken up into manageable chunks ?
          Hide
          ajayyadava Ajay Yadava added a comment -

          kenneth ho Will the existing UI (html/css/js) co-exist with this one? Reason I ask is because I didn't see any deletions in the patch. If that is the case then we should probably think of making the front end stack consistent. Old one used dust.js for templating and current one uses Angular js.

          Another question that I have is will the authentication work only with Ambari or can it be used even if someone doesn't use Ambari? If yes, then how different will the two processes be?

          +1 to breaking it in manageable chunks and uploading to review board will help to review and provide inline comments on the code.

          Show
          ajayyadava Ajay Yadava added a comment - kenneth ho Will the existing UI (html/css/js) co-exist with this one? Reason I ask is because I didn't see any deletions in the patch. If that is the case then we should probably think of making the front end stack consistent. Old one used dust.js for templating and current one uses Angular js. Another question that I have is will the authentication work only with Ambari or can it be used even if someone doesn't use Ambari? If yes, then how different will the two processes be? +1 to breaking it in manageable chunks and uploading to review board will help to review and provide inline comments on the code.
          Hide
          kho@hortonworks.com kenneth ho added a comment -

          @ajayyadava The new UI co-exist with the existing one,in the source code. In the meanwhile we keep both just in case we need to switch back to the old one, but in time, we will retire and deprecate the old one.
          We have split the patch into multiple chunks, and i am going to post it today.

          I will create a seperate jira for the authentication (login page).

          Show
          kho@hortonworks.com kenneth ho added a comment - @ajayyadava The new UI co-exist with the existing one,in the source code. In the meanwhile we keep both just in case we need to switch back to the old one, but in time, we will retire and deprecate the old one. We have split the patch into multiple chunks, and i am going to post it today. I will create a seperate jira for the authentication (login page).
          Hide
          kho@hortonworks.com kenneth ho added a comment -

          This patch contains all the infrastructure for the application and a read only view for cluster, feed and process. It doesn't include the code and modules to add/copy/run a cluster, feed or process.

          It has the following file structure:
          app: Contains all the resources for the applications they are detailed in the next part.
          dist: The generated folder when the build is performed.
          express-data: A folder to allocate all the mocks for local testing and development.
          node_modules: The generated folder for the integration with nodeJS.
          target: For maven.
          bower.json: Configuration file generated from nodeJS.
          falcon-ui.iml: Configuration file generated from maven.
          Gruntfile.js: Grunt configuration and task registrations for development, integration to sandbox and building.
          karma.conf.js: Karma configuration for testing.
          package.json: Node dependencies and configurations.
          pom.xml: Information about the project and configuration details used by Maven to build the project.
          server.js: Configuration for express server, development.

          app
          css: Contains all the styles, fonts, images and bootstrap libraries for the views.
          html: Contains all the web pages and templates.
          js: Contains all the buisness of the application, following the angular framework it englobes controllers, directives, services and other libraries used.
          test: All the jasmin specs for javascript testing and protractor specs for E2E testing.

          Show
          kho@hortonworks.com kenneth ho added a comment - This patch contains all the infrastructure for the application and a read only view for cluster, feed and process. It doesn't include the code and modules to add/copy/run a cluster, feed or process. It has the following file structure: app: Contains all the resources for the applications they are detailed in the next part. dist: The generated folder when the build is performed. express-data: A folder to allocate all the mocks for local testing and development. node_modules: The generated folder for the integration with nodeJS. target: For maven. bower.json: Configuration file generated from nodeJS. falcon-ui.iml: Configuration file generated from maven. Gruntfile.js: Grunt configuration and task registrations for development, integration to sandbox and building. karma.conf.js: Karma configuration for testing. package.json: Node dependencies and configurations. pom.xml: Information about the project and configuration details used by Maven to build the project. server.js: Configuration for express server, development. app css: Contains all the styles, fonts, images and bootstrap libraries for the views. html: Contains all the web pages and templates. js: Contains all the buisness of the application, following the angular framework it englobes controllers, directives, services and other libraries used. test: All the jasmin specs for javascript testing and protractor specs for E2E testing.
          Hide
          kho@hortonworks.com kenneth ho added a comment -

          2nd patch for cluster entity setup UI.

          Show
          kho@hortonworks.com kenneth ho added a comment - 2nd patch for cluster entity setup UI.
          Hide
          kho@hortonworks.com kenneth ho added a comment -

          add feed entity setup page

          Show
          kho@hortonworks.com kenneth ho added a comment - add feed entity setup page
          Hide
          kho@hortonworks.com kenneth ho added a comment -

          added process entity setup page.

          Show
          kho@hortonworks.com kenneth ho added a comment - added process entity setup page.
          Hide
          sriksun Srikanth Sundarrajan added a comment -

          Looking at the first patch...

          Some of the files *.less doesn't seem to have license entry on them.

          Show
          sriksun Srikanth Sundarrajan added a comment - Looking at the first patch ... Some of the files *.less doesn't seem to have license entry on them.
          Hide
          ajayyadava Ajay Yadava added a comment -

          I am also reviewing the first patch. My first observation is that this is a pretty elaborate UI stack as it uses lots of third party frameworks/libraries. Advantage of this is that we have lots of front end best practices, goodies and flexibility in place which will serve to any case. On the other hand, in a project like Apache Falcon it will significantly increase the entry-barrier for someone to contribute or even review the UI work. It's relatively easy and possible for someone working on Falcon java part to just use a plain jquery and html to probably fix an urgent issue but not other way around. Due to a totally disjoint set of people working on UI/backend we will need back and forth between back end and UI developers to debug an issue as none will be able to debug the other part of the code and pin point where exactly the issue is. I am not saying it's a wrong thing to do but just putting it out here so that we are conscious of the choices we are making.

          Other comments:

          • I think if we use a precompiled version of bootstrap we can significantly reduce the number of files that we have. Even though it's a vendor code it's still lot of code to checkout and compile.
          • We shouldn't commit the .iml files.
          • IMHO we should document the various frameworks/vendor libraries that we have used e.g. we should specify that we use jasmine for running tests, angular for templating. This is important to ensure that we have homogeneous use of frameworks. In the front end stack there are so many choices that we can end up with different frameworks for same thing in different parts of the code. This will also help us in cross checking the licensing terms and conditions.
          • We should also add documentation on how to contribute a patch detailing each step of development, like how to compile, get the local environment up, how to run tests etc.
          • Also, IMHO, we shouldn't have both angular and dust.js together. We should either remove old dust.js part(if we need to revert back then we can pull it back from previous version) or we should convert that part of the code also to angular. It's not a good practice to have two frameworks for same thing and we won't be able to prevent further accidental use of dust.js in some code. It also adds extra responsibility on the developers to know both dust and angular to maintain and support both of them.
          Show
          ajayyadava Ajay Yadava added a comment - I am also reviewing the first patch. My first observation is that this is a pretty elaborate UI stack as it uses lots of third party frameworks/libraries. Advantage of this is that we have lots of front end best practices, goodies and flexibility in place which will serve to any case. On the other hand, in a project like Apache Falcon it will significantly increase the entry-barrier for someone to contribute or even review the UI work. It's relatively easy and possible for someone working on Falcon java part to just use a plain jquery and html to probably fix an urgent issue but not other way around. Due to a totally disjoint set of people working on UI/backend we will need back and forth between back end and UI developers to debug an issue as none will be able to debug the other part of the code and pin point where exactly the issue is. I am not saying it's a wrong thing to do but just putting it out here so that we are conscious of the choices we are making. Other comments: I think if we use a precompiled version of bootstrap we can significantly reduce the number of files that we have. Even though it's a vendor code it's still lot of code to checkout and compile. We shouldn't commit the .iml files. IMHO we should document the various frameworks/vendor libraries that we have used e.g. we should specify that we use jasmine for running tests, angular for templating. This is important to ensure that we have homogeneous use of frameworks. In the front end stack there are so many choices that we can end up with different frameworks for same thing in different parts of the code. This will also help us in cross checking the licensing terms and conditions. We should also add documentation on how to contribute a patch detailing each step of development, like how to compile, get the local environment up, how to run tests etc. Also, IMHO, we shouldn't have both angular and dust.js together. We should either remove old dust.js part(if we need to revert back then we can pull it back from previous version) or we should convert that part of the code also to angular. It's not a good practice to have two frameworks for same thing and we won't be able to prevent further accidental use of dust.js in some code. It also adds extra responsibility on the developers to know both dust and angular to maintain and support both of them.
          Hide
          sriksun Srikanth Sundarrajan added a comment -

          kenneth ho, Am fairly new to JS, so trying to figure my way around this. Sorry it is taking longer.

          Show
          sriksun Srikanth Sundarrajan added a comment - kenneth ho , Am fairly new to JS, so trying to figure my way around this. Sorry it is taking longer.
          Hide
          sriksun Srikanth Sundarrajan added a comment -

          As per our discussion, let us try and push this in. I feel the the following things are needed before this can be pushed into trunk.

          1. Looks like building with 790 requires some additional binaries (ex. npm, grunt etc). These are used via maven-antrun. Builds are likely to fail without these being locally installed. This also means that the build on builds.apache.org would fail as well. We need a way to manage this without requiring this installation if possible.
          2. Some files are missing license headers
          3. Build leaves behind some files on the root, which fail rat:check on subsequent builds. It might be helpful to redirect these to target/
          4. IDE specific files to be removed from patch
          5. If builds are made seamless without requiring any additional setup (as outlined in #1 above), then few instructions on how to bring up the UI would be handy, but this can be handled in a separate JIRA. However if build requires additional steps, then they need to be documented before we push this in.

          kenneth ho, Apologies once again for holding up the review on this. Thanks for contributing to the UI efforts on Falcon and more importantly laying the foundation for the Falcon UI.

          Show
          sriksun Srikanth Sundarrajan added a comment - As per our discussion , let us try and push this in. I feel the the following things are needed before this can be pushed into trunk. 1. Looks like building with 790 requires some additional binaries (ex. npm, grunt etc). These are used via maven-antrun. Builds are likely to fail without these being locally installed. This also means that the build on builds.apache.org would fail as well. We need a way to manage this without requiring this installation if possible. 2. Some files are missing license headers 3. Build leaves behind some files on the root, which fail rat:check on subsequent builds. It might be helpful to redirect these to target/ 4. IDE specific files to be removed from patch 5. If builds are made seamless without requiring any additional setup (as outlined in #1 above), then few instructions on how to bring up the UI would be handy, but this can be handled in a separate JIRA. However if build requires additional steps, then they need to be documented before we push this in. kenneth ho , Apologies once again for holding up the review on this. Thanks for contributing to the UI efforts on Falcon and more importantly laying the foundation for the Falcon UI.
          Hide
          armando.reyna Armando Reyna added a comment - - edited

          Hi Srikanth Sundarrajan, I'm Armando nice to meet you, I built the patches please let me know about any concerns of the code or anything related to the falcon-ui.

          Can you please tell me of which patch are we talking about on this comment? [^patch01.patch]?

          Thanks.

          Show
          armando.reyna Armando Reyna added a comment - - edited Hi Srikanth Sundarrajan , I'm Armando nice to meet you, I built the patches please let me know about any concerns of the code or anything related to the falcon-ui. Can you please tell me of which patch are we talking about on this comment? [^patch01.patch] ? Thanks.
          Hide
          armando.reyna Armando Reyna added a comment - - edited

          falcon-ui installation ** Just to install it in local without sandbox **

          Show
          armando.reyna Armando Reyna added a comment - - edited falcon-ui installation ** Just to install it in local without sandbox **
          Hide
          svenkat Venkatesh Seetharam added a comment -

          Why is this hosted in nodejs? This is not what we have followed in Falcon.

          NodeJs , npm, Grunt must be installed in the local pc.

          This is not acceptable for users. Not sure why do you need NodeJS? If you still need, pull it using maven and we do not ask anything to be installed for building falcon apart from maven and jdk.

          IMHO, we should be using Bootstrap and AngularJS or backend and connect with REST APIs in Falcon.

          Can you please explain the architecture of this?

          Show
          svenkat Venkatesh Seetharam added a comment - Why is this hosted in nodejs? This is not what we have followed in Falcon. NodeJs , npm, Grunt must be installed in the local pc. This is not acceptable for users. Not sure why do you need NodeJS? If you still need, pull it using maven and we do not ask anything to be installed for building falcon apart from maven and jdk. IMHO, we should be using Bootstrap and AngularJS or backend and connect with REST APIs in Falcon. Can you please explain the architecture of this?
          Hide
          armando.reyna Armando Reyna added a comment - - edited

          Srikanth Sundarrajan regarding your comments:

          1/5. I attached README.md, please let me know if I should generate a path to add this file to the falcon-ui root. *** Just for local install, nothing else needs to be done for the sandbox ***

          2. I was looking at [^patch01.patch] and I didn't found any file without the header, can you please give me an example? All the content in app/css/bootstrap and app/js/lib is content of libraries, I think they shouldn't have the apache headers right?

          3. When I built the entire app, no rat errors where returned.

          4. Again can you give me an example? I dont see any files for IDE's.

          Thanks.

          Show
          armando.reyna Armando Reyna added a comment - - edited Srikanth Sundarrajan regarding your comments: 1/5. I attached README.md , please let me know if I should generate a path to add this file to the falcon-ui root. *** Just for local install, nothing else needs to be done for the sandbox *** 2. I was looking at [^patch01.patch] and I didn't found any file without the header, can you please give me an example? All the content in app/css/bootstrap and app/js/lib is content of libraries, I think they shouldn't have the apache headers right? 3. When I built the entire app, no rat errors where returned. 4. Again can you give me an example? I dont see any files for IDE's. Thanks.
          Hide
          armando.reyna Armando Reyna added a comment -

          Sure, NodeJS is used to run falcon-ui in local without using the sandbox/API's, if it is not needed for users I should remove those and generate a path just to build with maven.

          You're right on the sandbox version we use angularJS to connect to the REST API's in Falcon.

          Please let me build a new path in order to include just the necessary to build with maven.

          Thanks.

          Show
          armando.reyna Armando Reyna added a comment - Sure, NodeJS is used to run falcon-ui in local without using the sandbox/API's, if it is not needed for users I should remove those and generate a path just to build with maven. You're right on the sandbox version we use angularJS to connect to the REST API's in Falcon. Please let me build a new path in order to include just the necessary to build with maven. Thanks.
          Hide
          armando.reyna Armando Reyna added a comment -

          Log of the falcon project build. BUILD SUCCESS

          Show
          armando.reyna Armando Reyna added a comment - Log of the falcon project build. BUILD SUCCESS
          Hide
          armando.reyna Armando Reyna added a comment -

          Hi, I was testing the [^patch01.patch] and I was succesful in the build of the entire application including the falcon-ui falcon.log
          There's nothing else to do for the sandbox installation than copy the app inside the sandbox, all the instructions are on the falcon-ui/pom.xml
          I cloned a new repository on my local from https://github.com/apache/falcon then apply the patch [^patch01.patch] then build the application and it didn't throw any errors.

          Are you doing something different? If so, can you specify which are those steps so I can replicate the errors?

          Thanks.

          Show
          armando.reyna Armando Reyna added a comment - Hi, I was testing the [^patch01.patch] and I was succesful in the build of the entire application including the falcon-ui falcon.log There's nothing else to do for the sandbox installation than copy the app inside the sandbox, all the instructions are on the falcon-ui/pom.xml I cloned a new repository on my local from https://github.com/apache/falcon then apply the patch [^patch01.patch] then build the application and it didn't throw any errors. Are you doing something different? If so, can you specify which are those steps so I can replicate the errors? Thanks.
          Hide
          sriksun Srikanth Sundarrajan added a comment -

          Sorry Armando Reyna if my comment was terse. I will try and be a bit more specific. Will share steps to reproduce these shortly.

          Show
          sriksun Srikanth Sundarrajan added a comment - Sorry Armando Reyna if my comment was terse. I will try and be a bit more specific. Will share steps to reproduce these shortly.
          Hide
          sriksun Srikanth Sundarrajan added a comment -
          dataqa@hydra:~/sriksun$ git clone https://git-wip-us.apache.org/repos/asf/falcon.git
          Cloning into 'falcon'...
          remote: Counting objects: 39909, done.
          remote: Compressing objects: 100% (18113/18113), done.
          Receiving objects: 100% (39909/39909), 20.61 MiB | 410 KiB/s, done.
          remote: Total 39909 (delta 15671), reused 33332 (delta 11417)
          Resolving deltas: 100% (15671/15671), done.
          dataqa@hydra:~/sriksun$ cd falcon/
          dataqa@hydra:~/sriksun/falcon$ curl "https://issues.apache.org/jira/secure/attachment/12689541/12689541_patch01.patch" | git apply -v
            % Total    % Received % Xferd  Average Speed   Time    Time     Time  Current
                                           Dload  Upload   Total   Spent    Left  Speed
          100 1650k  100 1650k    0     0   119k      0  0:00:13  0:00:13 --:--:--  208k
          ...
          dataqa@hydra:~/sriksun/falcon$ export MAVEN_OPTS="-Xmx1024m -XX:MaxPermSize=256m" && mvn clean install -Ptest-patch,hadoop-2
          [INFO] Scanning for projects...
          [ERROR] The build could not read 1 project -> [Help 1]
          [ERROR]   
          [ERROR]   The project org.apache.falcon:falcon-ui:0.6-incubating-SNAPSHOT (/home/dataqa/sriksun/falcon/falcon-ui/pom.xml) has 1 error
          [ERROR]     Non-resolvable parent POM: Could not find artifact org.apache.falcon:falcon-main:pom:0.6-incubating-SNAPSHOT and 'parent.relativePath' points at wrong local POM @ line 21, column 11 -> [Help 2]
          [ERROR] 
          [ERROR] To see the full stack trace of the errors, re-run Maven with the -e switch.
          [ERROR] Re-run Maven using the -X switch to enable full debug logging.
          [ERROR] 
          [ERROR] For more information about the errors and possible solutions, please read the following articles:
          [ERROR] [Help 1] http://cwiki.apache.org/confluence/display/MAVEN/ProjectBuildingException
          [ERROR] [Help 2] http://cwiki.apache.org/confluence/display/MAVEN/UnresolvableModelException
          
          Show
          sriksun Srikanth Sundarrajan added a comment - dataqa@hydra:~/sriksun$ git clone https://git-wip-us.apache.org/repos/asf/falcon.git Cloning into 'falcon'... remote: Counting objects: 39909, done. remote: Compressing objects: 100% (18113/18113), done. Receiving objects: 100% (39909/39909), 20.61 MiB | 410 KiB/s, done. remote: Total 39909 (delta 15671), reused 33332 (delta 11417) Resolving deltas: 100% (15671/15671), done. dataqa@hydra:~/sriksun$ cd falcon/ dataqa@hydra:~/sriksun/falcon$ curl "https://issues.apache.org/jira/secure/attachment/12689541/12689541_patch01.patch" | git apply -v % Total % Received % Xferd Average Speed Time Time Time Current Dload Upload Total Spent Left Speed 100 1650k 100 1650k 0 0 119k 0 0:00:13 0:00:13 --:--:-- 208k ... dataqa@hydra:~/sriksun/falcon$ export MAVEN_OPTS="-Xmx1024m -XX:MaxPermSize=256m" && mvn clean install -Ptest-patch,hadoop-2 [INFO] Scanning for projects... [ERROR] The build could not read 1 project -> [Help 1] [ERROR] [ERROR] The project org.apache.falcon:falcon-ui:0.6-incubating-SNAPSHOT (/home/dataqa/sriksun/falcon/falcon-ui/pom.xml) has 1 error [ERROR] Non-resolvable parent POM: Could not find artifact org.apache.falcon:falcon-main:pom:0.6-incubating-SNAPSHOT and 'parent.relativePath' points at wrong local POM @ line 21, column 11 -> [Help 2] [ERROR] [ERROR] To see the full stack trace of the errors, re-run Maven with the -e switch. [ERROR] Re-run Maven using the -X switch to enable full debug logging. [ERROR] [ERROR] For more information about the errors and possible solutions, please read the following articles: [ERROR] [Help 1] http://cwiki.apache.org/confluence/display/MAVEN/ProjectBuildingException [ERROR] [Help 2] http://cwiki.apache.org/confluence/display/MAVEN/UnresolvableModelException
          Hide
          sriksun Srikanth Sundarrajan added a comment - - edited

          Modified falcon-ui/pom.xml with update parent pom version

            <parent>
              <groupId>org.apache.falcon</groupId>
              <artifactId>falcon-main</artifactId>
              <version>0.6-incubating-SNAPSHOT</version>
            </parent>
          

          to

            <parent>
              <groupId>org.apache.falcon</groupId>
              <artifactId>falcon-main</artifactId>
              <version>0.7-SNAPSHOT</version>
            </parent>
          
          Show
          sriksun Srikanth Sundarrajan added a comment - - edited Modified falcon-ui/pom.xml with update parent pom version <parent> <groupId>org.apache.falcon</groupId> <artifactId>falcon-main</artifactId> <version>0.6-incubating-SNAPSHOT</version> </parent> to <parent> <groupId>org.apache.falcon</groupId> <artifactId>falcon-main</artifactId> <version>0.7-SNAPSHOT</version> </parent>
          Hide
          sriksun Srikanth Sundarrajan added a comment -
          dataqa@hydra:~/sriksun/falcon$ export MAVEN_OPTS="-Xmx1024m -XX:MaxPermSize=256m" && mvn clean install -Ptest-patch,hadoop-2
          [INFO] Scanning for projects...
          ...
          [INFO] ------------------------------------------------------------------------
          [INFO] BUILD FAILURE
          [INFO] ------------------------------------------------------------------------
          [INFO] Total time: 27.362 s
          [INFO] Finished at: 2015-02-06T11:31:01+05:30
          [INFO] Final Memory: 34M/181M
          [INFO] ------------------------------------------------------------------------
          [ERROR] Failed to execute goal org.apache.maven.plugins:maven-antrun-plugin:1.7:run (compile) on project falcon-ui: An Ant BuildException has occured: Execute failed: java.io.IOException: Cannot run program "npm" (in directory "/home/dataqa/sriksun/falcon/falcon-ui"): error=2, No such file or directory
          [ERROR] around Ant part ...<exec dir="/home/dataqa/sriksun/falcon/falcon-ui" executable="npm">... @ 4:70 in /home/dataqa/sriksun/falcon/falcon-ui/target/antrun/build-falcon-ui-compile.xml
          
          Show
          sriksun Srikanth Sundarrajan added a comment - dataqa@hydra:~/sriksun/falcon$ export MAVEN_OPTS="-Xmx1024m -XX:MaxPermSize=256m" && mvn clean install -Ptest-patch,hadoop-2 [INFO] Scanning for projects... ... [INFO] ------------------------------------------------------------------------ [INFO] BUILD FAILURE [INFO] ------------------------------------------------------------------------ [INFO] Total time: 27.362 s [INFO] Finished at: 2015-02-06T11:31:01+05:30 [INFO] Final Memory: 34M/181M [INFO] ------------------------------------------------------------------------ [ERROR] Failed to execute goal org.apache.maven.plugins:maven-antrun-plugin:1.7:run (compile) on project falcon-ui: An Ant BuildException has occured: Execute failed: java.io.IOException: Cannot run program "npm" (in directory "/home/dataqa/sriksun/falcon/falcon-ui"): error=2, No such file or directory [ERROR] around Ant part ...<exec dir="/home/dataqa/sriksun/falcon/falcon-ui" executable="npm">... @ 4:70 in /home/dataqa/sriksun/falcon/falcon-ui/target/antrun/build-falcon-ui-compile.xml
          Hide
          sriksun Srikanth Sundarrajan added a comment -

          Once you get past npm by installing it, you will hit simillar issue around grunt

          Show
          sriksun Srikanth Sundarrajan added a comment - Once you get past npm by installing it, you will hit simillar issue around grunt
          Hide
          sriksun Srikanth Sundarrajan added a comment -

          When I install the required components and attempt to build I get the following issue

          [INFO] ------------------------------------------------------------------------
          [INFO] BUILD FAILURE
          [INFO] ------------------------------------------------------------------------
          [INFO] Total time: 5:07.089s
          [INFO] Finished at: Fri Feb 06 14:28:34 IST 2015
          [INFO] Final Memory: 35M/347M
          [INFO] ------------------------------------------------------------------------
          [ERROR] Failed to execute goal org.apache.rat:apache-rat-plugin:0.7:check (rat-check) on project falcon-ui: Too many unapproved licenses: 13219 -> [Help 1]
          

          Attaching the rat output for reference.

          Show
          sriksun Srikanth Sundarrajan added a comment - When I install the required components and attempt to build I get the following issue [INFO] ------------------------------------------------------------------------ [INFO] BUILD FAILURE [INFO] ------------------------------------------------------------------------ [INFO] Total time: 5:07.089s [INFO] Finished at: Fri Feb 06 14:28:34 IST 2015 [INFO] Final Memory: 35M/347M [INFO] ------------------------------------------------------------------------ [ERROR] Failed to execute goal org.apache.rat:apache-rat-plugin:0.7:check (rat-check) on project falcon-ui: Too many unapproved licenses: 13219 -> [Help 1] Attaching the rat output for reference.
          Hide
          armando.reyna Armando Reyna added a comment -

          Please apply this patch for the Falcon-UI integration.
          *The other patches are deprecated

          Show
          armando.reyna Armando Reyna added a comment - Please apply this patch for the Falcon-UI integration. *The other patches are deprecated
          Hide
          armando.reyna Armando Reyna added a comment - - edited

          Hi Srikanth Sundarrajan, I fixed some things on the falcon-ui and created a new patch patch_1.0.1.patch, please try it and let me know any problems with it.

          Thanks!

          Show
          armando.reyna Armando Reyna added a comment - - edited Hi Srikanth Sundarrajan , I fixed some things on the falcon-ui and created a new patch patch_1.0.1.patch , please try it and let me know any problems with it. Thanks!
          Hide
          sureshms Suresh Srinivas added a comment - - edited

          Armando Reyna, can you please respond to each of the issues raised by Srikanth Sundarrajan and confirm that you have addressed them. Did you follow the procedure he mentioned to ensure there are no issues with the patch anymore?

          Show
          sureshms Suresh Srinivas added a comment - - edited Armando Reyna , can you please respond to each of the issues raised by Srikanth Sundarrajan and confirm that you have addressed them. Did you follow the procedure he mentioned to ensure there are no issues with the patch anymore?
          Hide
          armando.reyna Armando Reyna added a comment -

          Yes I have addressed all of them, I apply the licence to the missing files and exclude the libraries.
          I have followed the same procedure to replicate the error, fix the files, make the patch, and test it.

          Show
          armando.reyna Armando Reyna added a comment - Yes I have addressed all of them, I apply the licence to the missing files and exclude the libraries. I have followed the same procedure to replicate the error, fix the files, make the patch, and test it.
          Hide
          armando.reyna Armando Reyna added a comment -

          Thats correct you should have node installed where the build is made because the falcon-ui uses npm modules with Grunt in order to minimize and copy the files to the webapp folder.

          Show
          armando.reyna Armando Reyna added a comment - Thats correct you should have node installed where the build is made because the falcon-ui uses npm modules with Grunt in order to minimize and copy the files to the webapp folder.
          Hide
          armando.reyna Armando Reyna added a comment -

          I changed the parent version in the path patch_1.0.1.patch this shouldn't be a problem any more.

          Show
          armando.reyna Armando Reyna added a comment - I changed the parent version in the path patch_1.0.1.patch this shouldn't be a problem any more.
          Hide
          sowmyaramesh Sowmya Ramesh added a comment -

          Armando Reyna: I tried building with the latest patch and build fails with

          [INFO] Apache Falcon Web Application ..................... FAILURE [  8.638 s]
          [INFO] Apache Falcon Documentation ....................... SKIPPED
          [INFO] ------------------------------------------------------------------------
          [INFO] BUILD FAILURE
          [INFO] ------------------------------------------------------------------------
          [INFO] Total time: 06:22 min
          [INFO] Finished at: 2015-02-06T15:20:21-08:00
          [INFO] Final Memory: 87M/191M
          [INFO] ------------------------------------------------------------------------
          [ERROR] Failed to execute goal org.apache.rat:apache-rat-plugin:0.7:check (rat-check) on project falcon-webapp: Too many unapproved licenses: 6 -> [Help 1]
          
          Show
          sowmyaramesh Sowmya Ramesh added a comment - Armando Reyna : I tried building with the latest patch and build fails with [INFO] Apache Falcon Web Application ..................... FAILURE [ 8.638 s] [INFO] Apache Falcon Documentation ....................... SKIPPED [INFO] ------------------------------------------------------------------------ [INFO] BUILD FAILURE [INFO] ------------------------------------------------------------------------ [INFO] Total time: 06:22 min [INFO] Finished at: 2015-02-06T15:20:21-08:00 [INFO] Final Memory: 87M/191M [INFO] ------------------------------------------------------------------------ [ERROR] Failed to execute goal org.apache.rat:apache-rat-plugin:0.7:check (rat-check) on project falcon-webapp: Too many unapproved licenses: 6 -> [Help 1]
          Hide
          sriksun Srikanth Sundarrajan added a comment -

          Hi Armando Reyna, Have you had a chance to check out the issue reported by Sowmya Ramesh?

          Show
          sriksun Srikanth Sundarrajan added a comment - Hi Armando Reyna , Have you had a chance to check out the issue reported by Sowmya Ramesh ?
          Hide
          kho@hortonworks.com kenneth ho added a comment -

          this patch has been verified.

          Show
          kho@hortonworks.com kenneth ho added a comment - this patch has been verified.
          Hide
          kho@hortonworks.com kenneth ho added a comment -

          Srikanth Sundarrajan please use the latest patch patch.tar.gz, we have verified it.
          After extracted, please apply in this order:
          FALCON-790.patch , FALCON-790-V1.patch , FALCON-790-V2.patch , FALCON-790-V3.patch

          Show
          kho@hortonworks.com kenneth ho added a comment - Srikanth Sundarrajan please use the latest patch patch.tar.gz, we have verified it. After extracted, please apply in this order: FALCON-790 .patch , FALCON-790 -V1.patch , FALCON-790 -V2.patch , FALCON-790 -V3.patch
          Hide
          sriksun Srikanth Sundarrajan added a comment -

          Can you also include these things additionally please?

          1. Prerequisites for building with this patch (npm, grunt etc) documented in ./Installation-Steps.txt::Building Falcon
          2. Instructions updated in docs/src/site/twiki/InstallationSteps.twiki for building & also bringing up the UI. Please note that there is existing documentation on bringing up the UI(dashboard), please do add notes on how they co-exist.

          Looks good otherwise.

          Also we need to work with build infra to setup npm, grunt etc on all the build servers if they are not present already.

          Show
          sriksun Srikanth Sundarrajan added a comment - Can you also include these things additionally please? 1. Prerequisites for building with this patch (npm, grunt etc) documented in ./Installation-Steps.txt::Building Falcon 2. Instructions updated in docs/src/site/twiki/InstallationSteps.twiki for building & also bringing up the UI. Please note that there is existing documentation on bringing up the UI(dashboard), please do add notes on how they co-exist. Looks good otherwise. Also we need to work with build infra to setup npm, grunt etc on all the build servers if they are not present already.
          Hide
          kho@hortonworks.com kenneth ho added a comment -

          Srikanth Sundarrajan I have uploaded the patch that contains :
          1. build instruction and installation steps
          2. pom.xml with the plugin to install grunt and node.
          3. updated license file.

          Show
          kho@hortonworks.com kenneth ho added a comment - Srikanth Sundarrajan I have uploaded the patch that contains : 1. build instruction and installation steps 2. pom.xml with the plugin to install grunt and node. 3. updated license file.
          Hide
          sriksun Srikanth Sundarrajan added a comment -
          [INFO] --- frontend-maven-plugin:0.0.14:grunt (grunt build) @ falcon-ui ---
          [INFO] Running 'grunt build --no-color' in /projects/incubator-falcon/falcon-ui
          [INFO] 
          [INFO] module.js:340
          [INFO]     throw err;
          [INFO]           ^
          [INFO] Error: Cannot find module '/projects/incubator-falcon/falcon-ui/node_modules/grunt-cli/bin/grunt'
          [INFO]     at Function.Module._resolveFilename (module.js:338:15)
          [INFO]     at Function.Module._load (module.js:280:25)
          [INFO]     at Function.Module.runMain (module.js:497:10)
          [INFO]     at startup (node.js:119:16)
          [INFO]     at node.js:906:3
          [INFO] ------------------------------------------------------------------------
          

          Running into some issues. Will check to see if I have done something wrong.

          Show
          sriksun Srikanth Sundarrajan added a comment - [INFO] --- frontend-maven-plugin:0.0.14:grunt (grunt build) @ falcon-ui --- [INFO] Running 'grunt build --no-color' in /projects/incubator-falcon/falcon-ui [INFO] [INFO] module.js:340 [INFO] throw err; [INFO] ^ [INFO] Error: Cannot find module '/projects/incubator-falcon/falcon-ui/node_modules/grunt-cli/bin/grunt' [INFO] at Function.Module._resolveFilename (module.js:338:15) [INFO] at Function.Module._load (module.js:280:25) [INFO] at Function.Module.runMain (module.js:497:10) [INFO] at startup (node.js:119:16) [INFO] at node.js:906:3 [INFO] ------------------------------------------------------------------------ Running into some issues. Will check to see if I have done something wrong.
          Hide
          sriksun Srikanth Sundarrajan added a comment -

          Error is reproducible in multiple runs across hosts.

          Show
          sriksun Srikanth Sundarrajan added a comment - Error is reproducible in multiple runs across hosts.
          Hide
          sriksun Srikanth Sundarrajan added a comment -

          grunt-cli isn't installed automatically by the build leading to this error.

          Show
          sriksun Srikanth Sundarrajan added a comment - grunt-cli isn't installed automatically by the build leading to this error.
          Hide
          kho@hortonworks.com kenneth ho added a comment -

          please extract patch.tar.gz, and apply in this order:
          FALCON-790.patch , FALCON-790-V1.patch , FALCON-790-V2.patch , FALCON-790-V3.patch, FALCON-790-v4.patch

          Show
          kho@hortonworks.com kenneth ho added a comment - please extract patch.tar.gz, and apply in this order: FALCON-790 .patch , FALCON-790 -V1.patch , FALCON-790 -V2.patch , FALCON-790 -V3.patch, FALCON-790 -v4.patch
          Hide
          sriksun Srikanth Sundarrajan added a comment -

          I had exactly applied them in that order. Will try once more. If I do run into issues, will upload build logs for further debugging.

          Show
          sriksun Srikanth Sundarrajan added a comment - I had exactly applied them in that order. Will try once more. If I do run into issues, will upload build logs for further debugging.
          Hide
          sureshms Suresh Srinivas added a comment -

          kenneth ho, it might be easier if you post a consolidated pitch as well.

          Show
          sureshms Suresh Srinivas added a comment - kenneth ho , it might be easier if you post a consolidated pitch as well.
          Hide
          sriksun Srikanth Sundarrajan added a comment -

          Hi kenneth ho, I am build now. Quick question for you, before I commit this.

          Found some generated code copied to webapp/src/main/webapp/public/, am assuming we will have to exclude them from the commit. Am wondering if they need to be in webapp/src or should it instead be generated under webapp/target ?

          Show
          sriksun Srikanth Sundarrajan added a comment - Hi kenneth ho , I am build now. Quick question for you, before I commit this. Found some generated code copied to webapp/src/main/webapp/public/, am assuming we will have to exclude them from the commit. Am wondering if they need to be in webapp/src or should it instead be generated under webapp/target ?
          Hide
          sriksun Srikanth Sundarrajan added a comment -

          Attaching patch for further changes to avoid writing generated source under webapp/src/main/webapp/public.

          Would like

          falcon-ui/dist
          falcon-ui/node
          falcon-ui/node_modules
          

          to be actually generated under target. For now have added them to .gitignore.

          Please review the additional patch. Will commit once you ack. If it is possible to move out the above listed directories out of falcon-ui into falcon-ui/target, then it would be cleaner

          Show
          sriksun Srikanth Sundarrajan added a comment - Attaching patch for further changes to avoid writing generated source under webapp/src/main/webapp/public. Would like falcon-ui/dist falcon-ui/node falcon-ui/node_modules to be actually generated under target. For now have added them to .gitignore. Please review the additional patch. Will commit once you ack. If it is possible to move out the above listed directories out of falcon-ui into falcon-ui/target, then it would be cleaner
          Hide
          sriksun Srikanth Sundarrajan added a comment -
          Show
          sriksun Srikanth Sundarrajan added a comment - See additional patch
          Hide
          kho@hortonworks.com kenneth ho added a comment -

          Patch looks good.
          webapp/src/main/webapp/public/ should be excluded from the commit.

          Show
          kho@hortonworks.com kenneth ho added a comment - Patch looks good. webapp/src/main/webapp/public/ should be excluded from the commit.
          Hide
          sriksun Srikanth Sundarrajan added a comment -

          Thanks kenneth ho & Armando Reyna. This has been committed. Apologies for dragging it this long. Hopefully next set of features can come in quickly.

          Show
          sriksun Srikanth Sundarrajan added a comment - Thanks kenneth ho & Armando Reyna . This has been committed. Apologies for dragging it this long. Hopefully next set of features can come in quickly.
          Hide
          githubbot ASF GitHub Bot added a comment -

          Github user asfgit closed the pull request at:

          https://github.com/apache/falcon/pull/5

          Show
          githubbot ASF GitHub Bot added a comment - Github user asfgit closed the pull request at: https://github.com/apache/falcon/pull/5

            People

            • Assignee:
              kho@hortonworks.com kenneth ho
              Reporter:
              kho@hortonworks.com kenneth ho
            • Votes:
              0 Vote for this issue
              Watchers:
              10 Start watching this issue

              Dates

              • Created:
                Updated:
                Resolved:

                Development