Infrastructure
  1. Infrastructure
  2. INFRA-4440

Migrate sparql.org to ASF Infrastructure

    Details

    • Type: Wish Wish
    • Status: Closed
    • Priority: Major Major
    • Resolution: Fixed
    • Fix Version/s: Initial Clearing
    • Component/s: None
    • Labels:
      None

      Description

      Currently we have a service demo running which displays use and functionality of the Apache Jena.

      It would be great if we were able to migrate this demo to ASF infrastructure where it now belongs.

      Some basic traffic statistics are provided below.

      -------------------
      sparql.org is around 50 hits per hour (today).

      Quite a lot of syntax errors, which are cheap to process. Failure to count brackets, failure to put <> around URIs, that sort of stuff.

      The demo server also implements query timeout - set to a few 10s of seconds currently - and a limit on the amount of data it wil read from a remote location.

        Activity

        Hide
        Andy Seaborne added a comment -
        Tony,

        Manual duly read this time. Thanks.
        Show
        Andy Seaborne added a comment - Tony, Manual duly read this time. Thanks.
        Hide
        Tony Stevenson added a comment -
        Andy,

        We do not use passwords to gain elevated privileges. We use a one time password mechanism, see here for details: http://www.apache.org/dev/freebsd-jails.html#opie (as per the comments on April 29th, 14:13)
        Show
        Tony Stevenson added a comment - Andy, We do not use passwords to gain elevated privileges. We use a one time password mechanism, see here for details: http://www.apache.org/dev/freebsd-jails.html#opie (as per the comments on April 29th, 14:13)
        Hide
        Andy Seaborne added a comment -
        Small problem - I can't sudo. It asks for the password ... which I don't know. Group wheel does not have write access to anything I can find to get started.

        Could some one let me know by a suitably secret means what the jena jail root password is, please?
        Show
        Andy Seaborne added a comment - Small problem - I can't sudo. It asks for the password ... which I don't know. Group wheel does not have write access to anything I can find to get started. Could some one let me know by a suitably secret means what the jena jail root password is, please?
        Hide
        Andy Seaborne added a comment -
        I have logged in successfully now.

        Thanks for all your help -- Andy
        Show
        Andy Seaborne added a comment - I have logged in successfully now. Thanks for all your help -- Andy
        Hide
        #asfinfra IRC Bot added a comment -
        <danielsh> Try now please. A special case in the jail-create script meant I could login via ssh even though you couldn't
        Show
        #asfinfra IRC Bot added a comment - <danielsh> Try now please. A special case in the jail-create script meant I could login via ssh even though you couldn't
        Hide
        Andy Seaborne added a comment -
        At the moment, it's not accepting ssh

        ssh -v ==>
        ...
        debug1: Next authentication method: publickey
        debug1: Offering RSA public key: /home/afs/.ssh/id_rsa
        debug1: Authentications that can continue: publickey,keyboard-interactive
        ... (other things tried)

        I added the right public key as far as I can see:
        cmp .../people/andy.pub ~/.ssh/id_rsa.pub

        This is the same as people.apache.org

        The probability of user error is high though.
        Show
        Andy Seaborne added a comment - At the moment, it's not accepting ssh ssh -v ==> ... debug1: Next authentication method: publickey debug1: Offering RSA public key: /home/afs/.ssh/id_rsa debug1: Authentications that can continue: publickey,keyboard-interactive ... (other things tried) I added the right public key as far as I can see: cmp .../people/andy.pub ~/.ssh/id_rsa.pub This is the same as people.apache.org The probability of user error is high though.
        Hide
        #asfinfra IRC Bot added a comment -
        <danielsh> I think that concludes infra's tasks here. Reopen here or ask on list if needed.
        Show
        #asfinfra IRC Bot added a comment - <danielsh> I think that concludes infra's tasks here. Reopen here or ask on list if needed.
        Hide
        #asfinfra IRC Bot added a comment -
        <danielsh> Sounds good. Ports are opened as requested. pf(4) is in place if needed.
        Show
        #asfinfra IRC Bot added a comment - <danielsh> Sounds good. Ports are opened as requested. pf(4) is in place if needed.
        Hide
        Andy Seaborne added a comment -
        Port 80 (for httpd). (Port 22 for ssh :-)
        Port 3030 only if that needs to be open for local redirection.

        If localhost:3030 works without opening it to the external network, that would be best but either way works for us.

        Our current setup is

        <VirtualHost *:80>
          ServerName www.sparql.org
          ServerAlias sparql.org
          ProxyRequests off
          <Proxy *>
            Order deny,allow
            Allow from all
          </Proxy>
          ProxyPass / http://127.0.0.1:3030/ max=4
          ProxyPassReverse / http://127.0.0.1:3030/
          ProxyPreserveHost On
        </VirtualHost>

        Same for sparql.net. port 3030 is not open

        However, there isn't some specific magic to that setup - anything that gets port 80 to port 3030 will do. The rate limiting and the ability to add other httpd features like blocking and caching if we ever need to is really why it is there; we haven't so far. It also means that admin of the service is user-space, but that isn't important.
        Show
        Andy Seaborne added a comment - Port 80 (for httpd). (Port 22 for ssh :-) Port 3030 only if that needs to be open for local redirection. If localhost:3030 works without opening it to the external network, that would be best but either way works for us. Our current setup is <VirtualHost *:80>   ServerName www.sparql.org   ServerAlias sparql.org   ProxyRequests off   <Proxy *>     Order deny,allow     Allow from all   </Proxy>   ProxyPass / http://127.0.0.1:3030/ max=4   ProxyPassReverse / http://127.0.0.1:3030/   ProxyPreserveHost On </VirtualHost> Same for sparql.net. port 3030 is not open However, there isn't some specific magic to that setup - anything that gets port 80 to port 3030 will do. The rate limiting and the ability to add other httpd features like blocking and caching if we ever need to is really why it is there; we haven't so far. It also means that admin of the service is user-space, but that isn't important.
        Hide
        #asfinfra IRC Bot added a comment -
        <danielsh> Access granted: andy@jena.zones.apache.org. Fingerprints given above. Root access available via OPIE (www.apache.org/dev/freebsd-jails#opie). What ports do you need open?
        Show
        #asfinfra IRC Bot added a comment - <danielsh> Access granted: andy@jena.zones.apache.org . Fingerprints given above. Root access available via OPIE ( www.apache.org/dev/freebsd-jails#opie ). What ports do you need open?
        Hide
        Andy Seaborne added a comment -
        Key added (andy.pub)
        Show
        Andy Seaborne added a comment - Key added (andy.pub)
        Hide
        #asfinfra IRC Bot added a comment -
        <danielsh> Fingerprints: a7:2e:d0:83:c2:65:74:79:30:64:4f:83:10:0a:b5:41 (rsa), e0:ac:c7:20:d2:dc:1a:93:5c:93:82:4e:00:e6:68:7a (ecdsa)
        Show
        #asfinfra IRC Bot added a comment - <danielsh> Fingerprints: a7:2e:d0:83:c2:65:74:79:30:64:4f:83:10:0a:b5:41 (rsa), e0:ac:c7:20:d2:dc:1a:93:5c:93:82:4e:00:e6:68:7a (ecdsa)
        Hide
        #asfinfra IRC Bot added a comment -
        <danielsh> The jail has been created. Please add SSH keys to svn at the previously-indicated location.
        Show
        #asfinfra IRC Bot added a comment - <danielsh> The jail has been created. Please add SSH keys to svn at the previously-indicated location.
        Hide
        #asfinfra IRC Bot added a comment -
        <danielsh> We'll set you up with a jail then. Please add ssh keys to https://svn.apache.org/repos/infra/infrastructure/trunk/ssh_keys/people/
        Show
        #asfinfra IRC Bot added a comment - <danielsh> We'll set you up with a jail then. Please add ssh keys to https://svn.apache.org/repos/infra/infrastructure/trunk/ssh_keys/people/
        Hide
        #asfinfra IRC Bot added a comment -
        <danielsh> I see no issues with directing sparql.org traffic to a daemon you run (presuming the PMC is happy with this). We'll open the ports on the firewall.
        Show
        #asfinfra IRC Bot added a comment - <danielsh> I see no issues with directing sparql.org traffic to a daemon you run (presuming the PMC is happy with this). We'll open the ports on the firewall.
        Hide
        Andy Seaborne added a comment -
        Looks good -- java/openjdk6 java openjdk6-b24_4

        Providing we can sparql.org traffic routed to our Jetty server running on some port, then Java is all we need. We provided everything for Jetty and local files.

        (In case it makes any difference - Jena has now been approved by the board as a TLP and we now have to do the graduation process).
        Show
        Andy Seaborne added a comment - Looks good -- java/openjdk6 java openjdk6-b24_4 Providing we can sparql.org traffic routed to our Jetty server running on some port, then Java is all we need. We provided everything for Jetty and local files. (In case it makes any difference - Jena has now been approved by the board as a TLP and we now have to do the graduation process).
        Hide
        #asfinfra IRC Bot added a comment -
        <danielsh> Please confirm whether the software and versions on http://tb.apache.org/index.php?action=list_buildports&build=9.0-RELENG-j-tlp meet your needs.
        Show
        #asfinfra IRC Bot added a comment - <danielsh> Please confirm whether the software and versions on http://tb.apache.org/index.php?action=list_buildports&build=9.0-RELENG-j-tlp meet your needs.
        Hide
        Andy Seaborne added a comment -
        Ping. Sorry for the delays - graduation processing ate my Apache time.

        sparql.org needs java6 or more, we can provide rest, including Jetty.

        The software is the distribution of Jena Fuseki, so there is a single jar with all dependencies in it to run. There are some static pages as well all served by Jetty.

        Currently, the front is httpd acting as virtual host controller, and it routes to the right port on localhost.

        I'm not familiar with a fBSD jail but whatever works; I'll learn.
        Show
        Andy Seaborne added a comment - Ping. Sorry for the delays - graduation processing ate my Apache time. sparql.org needs java6 or more, we can provide rest, including Jetty. The software is the distribution of Jena Fuseki, so there is a single jar with all dependencies in it to run. There are some static pages as well all served by Jetty. Currently, the front is httpd acting as virtual host controller, and it routes to the right port on localhost. I'm not familiar with a fBSD jail but whatever works; I'll learn.
        Hide
        #asfinfra IRC Bot added a comment -
        <danielsh> ping
        Show
        #asfinfra IRC Bot added a comment - <danielsh> ping
        Hide
        Tony Stevenson added a comment -
        Andy,

        Do you have more info, OS, Java Version, Jetty version, etc? This might dictate either a Ubuntu VM, or a fBSD Jail.
        Show
        Tony Stevenson added a comment - Andy, Do you have more info, OS, Java Version, Jetty version, etc? This might dictate either a Ubuntu VM, or a fBSD Jail.
        Hide
        Andy Seaborne added a comment -
        Sorry for the delay about sparql.org ...

        sparql.org, or SPARQLer, is not a static site; it runs some services:

        1/ a SPARQL demo (query over a very very small database)
        2/ Some validation services for parsing and checking SPARQL and RDF syntaxes.
        3/ (Occasionally used) some exercises for tutorials.
        4/ A general query service that can itself make HTTP GET request out to the web.

        It's currently running Jena Fuseki (all java) which uses Jetty as the web container and httpd as the front. It's on an AWS small instance but hardly stretches that hardware (load average is 0.08 just at the moment) . Typically load is 10's of hits a day but being public, it does sometimes get a surge of requests.

        The query services have hard timeouts on them to limit silly queries.

        There is no guarantee of availability. A few people would let me know if it's down (e.g. it's used by the W3C RDFa working grouping in their test suite) and they understand it's on a "best effort" basis with absolutely no guarantees whatsoever.

        We don't need DNS management of sparql.org. We can put a CNAME or an A record.

        There is no urgency for this - the current server is provided by a company that uses Jena. Now we have migrated all the other project infrastructure off this server, SPARQLer is the last thing to move but we are not being pushed to return the server just yet.
        Show
        Andy Seaborne added a comment - Sorry for the delay about sparql.org ... sparql.org, or SPARQLer, is not a static site; it runs some services: 1/ a SPARQL demo (query over a very very small database) 2/ Some validation services for parsing and checking SPARQL and RDF syntaxes. 3/ (Occasionally used) some exercises for tutorials. 4/ A general query service that can itself make HTTP GET request out to the web. It's currently running Jena Fuseki (all java) which uses Jetty as the web container and httpd as the front. It's on an AWS small instance but hardly stretches that hardware (load average is 0.08 just at the moment) . Typically load is 10's of hits a day but being public, it does sometimes get a surge of requests. The query services have hard timeouts on them to limit silly queries. There is no guarantee of availability. A few people would let me know if it's down (e.g. it's used by the W3C RDFa working grouping in their test suite) and they understand it's on a "best effort" basis with absolutely no guarantees whatsoever. We don't need DNS management of sparql.org. We can put a CNAME or an A record. There is no urgency for this - the current server is provided by a company that uses Jena. Now we have migrated all the other project infrastructure off this server, SPARQLer is the last thing to move but we are not being pushed to return the server just yet.
        Hide
        Lewis John McGibbney added a comment -
        Hi Gavin, I've sent my mail over to the guys @ jena-dev for this one. As far as I know it is used most by that ASF community and it would be best for them to sort it. I'm watching this issue so I'll try to help out if and where I can over the next while.
        Thank you
        Show
        Lewis John McGibbney added a comment - Hi Gavin, I've sent my mail over to the guys @ jena-dev for this one. As far as I know it is used most by that ASF community and it would be best for them to sort it. I'm watching this issue so I'll try to help out if and where I can over the next while. Thank you
        Hide
        Gavin added a comment -
        Lewis, please explain what you want here please?

        Do you need a FreeBSD Jail or Ubuntu VM or neither?
        What is to happen to the sparql.org domain?
        Show
        Gavin added a comment - Lewis, please explain what you want here please? Do you need a FreeBSD Jail or Ubuntu VM or neither? What is to happen to the sparql.org domain?
        Hide
        #asfinfra IRC Bot added a comment -
        <danielsh> I'm assuming this is only about the app, and NOT about migrating DNS handling or whois ownership of the listed domain.
        Show
        #asfinfra IRC Bot added a comment - <danielsh> I'm assuming this is only about the app, and NOT about migrating DNS handling or whois ownership of the listed domain.
        Hide
        #asfinfra IRC Bot added a comment -
        <danielsh> Is that a static HTTP website? If yes we could host it on our main webserver, if not we could create an Ubuntu VM or FreeBSD jail and let the (P)PMC install+maintain the app on it.
        Show
        #asfinfra IRC Bot added a comment - <danielsh> Is that a static HTTP website? If yes we could host it on our main webserver, if not we could create an Ubuntu VM or FreeBSD jail and let the (P)PMC install+maintain the app on it.

          People

          • Assignee:
            Unassigned
            Reporter:
            Lewis John McGibbney
          • Votes:
            0 Vote for this issue
            Watchers:
            4 Start watching this issue

            Dates

            • Created:
              Updated:
              Resolved:

              Development