Infrastructure
  1. Infrastructure
  2. INFRA-6597

Builds failing on solaris1 Jenkins node due to not enough space

    Details

    • Type: Bug Bug
    • Status: Closed
    • Priority: Major Major
    • Resolution: Fixed
    • Fix Version/s: Initial Clearing
    • Component/s: Jenkins
    • Labels:
      None

      Description

      Multiple builds, like [ZooKeeper-trunk-solaris|https://builds.apache.org/job/ZooKeeper-trunk-solaris/623/consoleFull] , [mahout-collections-trunk|https://builds.apache.org/job/mahout-collections-trunk/1154/consoleFull], and [Mahout-Examples-Cluster-Reuters-II|https://builds.apache.org/job/Mahout-Examples-Cluster-Reuters-II/554/consoleFull] are failing when run on solaris1 Jenkins node with errors like

      {noformat}
      [ERROR] Failed to execute goal org.apache.maven.plugins:maven-jar-plugin:2.4:jar (default-jar) on project mahout-math: Error assembling JAR: Failed to read filesystem attributes for: /zonestorage/hudson_solaris/home/hudson/hudson-slave/workspace/Mahout-Examples-Cluster-Reuters-II/trunk/math/pom.xml: Failed to retrieve numeric file attributes using: '/bin/sh -c ls -1lad /zonestorage/hudson_solaris/home/hudson/hudson-slave/workspace/Mahout-Examples-Cluster-Reuters-II/trunk/math/pom.xml': Error while executing process. Cannot run program "/bin/sh": error=12, Not enough space -> [Help 1]
      {noformat}

      Multiple posts on net suggest that this error means that there is not enough swap space.
      There is a [related page on Jenkins wiki|https://wiki.jenkins-ci.org/display/JENKINS/IOException+Not+enough+space] with more info.

        Activity

        Hide
        Jake Farrell added a comment -
        purged out /tmp and got it back down to 1%. from the builds@ mailing list [~olamy] cleaned up some hung jobs



        It looks some jobs doesn't "shutdown" correctly...
        This means a lot of process are still running whereas no one is active
        in Jenkins

        $ ps -fu hudson
             UID PID PPID C STIME TTY TIME CMD
          hudson 29447 19577 0 Apr 28 ? 0:00
        /home/hudson/tools/java/latest1.5/bin/java -Xmx512m
        -XX:MaxPermSize=128m -Djava
          hudson 7882 7876 0 02:20:16 pts/6 0:00 -sh
          hudson 18363 19577 0 Apr 21 ? 0:00
        /home/hudson/tools/java/latest1.5/bin/java -cp
        /export/home/hudson/hudson-slave
          hudson 10894 19577 0 Mar 23 ? 0:00
        /home/hudson/tools/java/latest1.5/bin/java -cp
        /export/home/hudson/hudson-slave
          hudson 15075 19577 0 Mar 12 ? 0:00
        /home/hudson/tools/java/latest1.5/bin/java -cp
        /export/home/hudson/hudson-slave
          hudson 2629 19577 0 Feb 27 ? 315:19
        /zonestorage/hudson_solaris/home/hudson/hudson-slave/tools/jdk6-u26-auto-instal
          hudson 6249 19577 0 Feb 14 ? 0:00
        /zonestorage/hudson_solaris/home/hudson/hudson-slave/tools/jdk6-u26-auto-instal
          hudson 8915 19577 0 Apr 24 ? 0:00
        /home/hudson/tools/java/latest1.5/bin/java -Xmx512m
        -XX:MaxPermSize=128m -Djava
          hudson 23630 19577 0 Jul 25 ? 0:00
        /export/home/jm/tools/perl586/bin/perl -T -w ../spamd/spamd.raw -D -x
        -s stderr
          hudson 3761 19577 0 Apr 14 ? 0:00
        /zonestorage/hudson_solaris/home/hudson/hudson-slave/tools/jdk6-u26-auto-instal
          hudson 2224 19577 0 May 24 ? 0:00
        /home/hudson/tools/java/latest1.5/bin/java -Xmx512m
        -XX:MaxPermSize=128m -Djava
          hudson 9066 19577 0 Feb 11 ? 0:00
        /home/hudson/tools/java/latest1.5/bin/java -cp
        /export/home/hudson/hudson-slave
          hudson 24760 19577 0 Apr 29 ? 0:00
        /home/hudson/tools/java/latest1.5/bin/java -Xmx512m
        -XX:MaxPermSize=128m -Djava
          hudson 218 19577 0 Mar 18 ? 0:00
        /home/hudson/tools/java/latest1.5/bin/java -cp
        /export/home/hudson/hudson-slave
          hudson 23629 19577 0 Jul 25 ? 0:00
        /export/home/jm/tools/perl586/bin/perl -T -w ../spamd/spamd.raw -D -x
        -s stderr
          hudson 6906 19577 0 Feb 09 ? 0:00
        /home/hudson/tools/java/latest1.5/bin/java -cp
        /export/home/hudson/hudson-slave
          hudson 8142 7882 0 02:20:24 pts/6 0:00 ps -fu hudson
          hudson 3758 19577 0 Mar 26 ? 0:00
        /home/hudson/tools/java/latest1.5/bin/java -cp
        /export/home/hudson/hudson-slave
          hudson 7876 7822 0 02:20:15 ? 0:00 /usr/lib/ssh/sshd


        So that's the reason of this "not enough space" message.
        Yes solaris says that for not enough memory.

        So I cleanup those process
        Will be better now.
        Show
        Jake Farrell added a comment - purged out /tmp and got it back down to 1%. from the builds@ mailing list [~olamy] cleaned up some hung jobs It looks some jobs doesn't "shutdown" correctly... This means a lot of process are still running whereas no one is active in Jenkins $ ps -fu hudson      UID PID PPID C STIME TTY TIME CMD   hudson 29447 19577 0 Apr 28 ? 0:00 /home/hudson/tools/java/latest1.5/bin/java -Xmx512m -XX:MaxPermSize=128m -Djava   hudson 7882 7876 0 02:20:16 pts/6 0:00 -sh   hudson 18363 19577 0 Apr 21 ? 0:00 /home/hudson/tools/java/latest1.5/bin/java -cp /export/home/hudson/hudson-slave   hudson 10894 19577 0 Mar 23 ? 0:00 /home/hudson/tools/java/latest1.5/bin/java -cp /export/home/hudson/hudson-slave   hudson 15075 19577 0 Mar 12 ? 0:00 /home/hudson/tools/java/latest1.5/bin/java -cp /export/home/hudson/hudson-slave   hudson 2629 19577 0 Feb 27 ? 315:19 /zonestorage/hudson_solaris/home/hudson/hudson-slave/tools/jdk6-u26-auto-instal   hudson 6249 19577 0 Feb 14 ? 0:00 /zonestorage/hudson_solaris/home/hudson/hudson-slave/tools/jdk6-u26-auto-instal   hudson 8915 19577 0 Apr 24 ? 0:00 /home/hudson/tools/java/latest1.5/bin/java -Xmx512m -XX:MaxPermSize=128m -Djava   hudson 23630 19577 0 Jul 25 ? 0:00 /export/home/jm/tools/perl586/bin/perl -T -w ../spamd/spamd.raw -D -x -s stderr   hudson 3761 19577 0 Apr 14 ? 0:00 /zonestorage/hudson_solaris/home/hudson/hudson-slave/tools/jdk6-u26-auto-instal   hudson 2224 19577 0 May 24 ? 0:00 /home/hudson/tools/java/latest1.5/bin/java -Xmx512m -XX:MaxPermSize=128m -Djava   hudson 9066 19577 0 Feb 11 ? 0:00 /home/hudson/tools/java/latest1.5/bin/java -cp /export/home/hudson/hudson-slave   hudson 24760 19577 0 Apr 29 ? 0:00 /home/hudson/tools/java/latest1.5/bin/java -Xmx512m -XX:MaxPermSize=128m -Djava   hudson 218 19577 0 Mar 18 ? 0:00 /home/hudson/tools/java/latest1.5/bin/java -cp /export/home/hudson/hudson-slave   hudson 23629 19577 0 Jul 25 ? 0:00 /export/home/jm/tools/perl586/bin/perl -T -w ../spamd/spamd.raw -D -x -s stderr   hudson 6906 19577 0 Feb 09 ? 0:00 /home/hudson/tools/java/latest1.5/bin/java -cp /export/home/hudson/hudson-slave   hudson 8142 7882 0 02:20:24 pts/6 0:00 ps -fu hudson   hudson 3758 19577 0 Mar 26 ? 0:00 /home/hudson/tools/java/latest1.5/bin/java -cp /export/home/hudson/hudson-slave   hudson 7876 7822 0 02:20:15 ? 0:00 /usr/lib/ssh/sshd So that's the reason of this "not enough space" message. Yes solaris says that for not enough memory. So I cleanup those process Will be better now.

          People

          • Assignee:
            Jake Farrell
            Reporter:
            Stevo Slavic
          • Votes:
            1 Vote for this issue
            Watchers:
            3 Start watching this issue

            Dates

            • Created:
              Updated:
              Resolved:

              Development