Uploaded image for project: 'Bigtop'
  1. Bigtop
  2. BIGTOP-3578

Solr smoke test fails on ppc64le

Attach filesAttach ScreenshotVotersWatch issueWatchersCreate sub-taskLinkCloneUpdate Comment AuthorReplace String in CommentUpdate Comment VisibilityDelete Comments
    XMLWordPrintableJSON

Details

    • Bug
    • Status: Resolved
    • Major
    • Resolution: Fixed
    • None
    • 3.0.0
    • deployment, solr, tests
    • None

    Description

      https://ci.bigtop.apache.org/view/Test/job/Bigtop-trunk-smoke-tests/259/COMPONENTS=hdfs.zookeeper.solrcloud@solr,OS=fedora-33-ppc64le-deploy/

      Solr smoke test fails with the following error.

      > Task :bigtop-tests:smoke-tests:solr:test
      Downloading https://repo.maven.apache.org/maven2/org/codehaus/woodstox/wstx-asl/3.2.7/wstx-asl-3.2.7.pom to /tmp/gradle_download5228771710184587059bin
      Downloading https://repo.maven.apache.org/maven2/javax/servlet/jsp/jsp-api/2.1/jsp-api-2.1.pom to /tmp/gradle_download7723517104372501069bin
      Downloading https://repo.maven.apache.org/maven2/org/slf4j/slf4j-log4j12/1.7.10/slf4j-log4j12-1.7.10.pom to /tmp/gradle_download5040172534686132912bin
      Downloading https://repo.maven.apache.org/maven2/javax/servlet/jsp/jsp-api/2.1/jsp-api-2.1.jar to /tmp/gradle_download8433979289897434251bin
      Downloading https://repo.maven.apache.org/maven2/org/codehaus/woodstox/wstx-asl/3.2.7/wstx-asl-3.2.7.jar to /tmp/gradle_download7704021333708043047bin
      Downloading https://repo.maven.apache.org/maven2/org/slf4j/slf4j-log4j12/1.7.10/slf4j-log4j12-1.7.10.jar to /tmp/gradle_download963266749008733834bin
      Caching disabled for task ':bigtop-tests:smoke-tests:solr:test' because:
        Build cache is disabled
      Task ':bigtop-tests:smoke-tests:solr:test' is not up-to-date because:
        No history is available.
      Starting process 'command '/usr/bin/solrctl''. Working directory: /bigtop-home/bigtop-tests/smoke-tests/solr Command: /usr/bin/solrctl instancedir --generate /tmp/smoke_data
      Successfully started process 'command '/usr/bin/solrctl''
      Starting process 'command '/usr/bin/solrctl''. Working directory: /bigtop-home/bigtop-tests/smoke-tests/solr Command: /usr/bin/solrctl instancedir --create smoke /tmp/smoke_data
      Successfully started process 'command '/usr/bin/solrctl''
      Uploading configs from /tmp/smoke_data/conf to f028d5337d92.bigtop.apache.org:2181/solr. This may take up to a minute.
      Starting process 'command '/usr/bin/solrctl''. Working directory: /bigtop-home/bigtop-tests/smoke-tests/solr Command: /usr/bin/solrctl collection --create smoke
      Successfully started process 'command '/usr/bin/solrctl''
      Error: can't discover Solr URI. Please specify it explicitly via --solr.
      
      > Task :bigtop-tests:smoke-tests:solr:test FAILED
      :bigtop-tests:smoke-tests:solr:test (Thread[Execution worker for ':',5,main]) completed. Took 11.357 secs.
      
      FAILURE: Build failed with an exception.
      

      I could reproduce this failure by running the test on the ppc64le server directly. It looks like that solr-server didn't start due to a memory issue.

      jenkins@bigtop1slave:~/bigtop/provisioner/docker$ ./docker-hadoop.sh --exec 1 jps
      1073 NameNode
      1747 GradleDaemon
      1257 DataNode
      553 QuorumPeerMain
      2031 Jps
      jenkins@bigtop1slave:~/bigtop/provisioner/docker$ ./docker-hadoop.sh --exec 1 systemctl status solr-server
      ‚óŹ solr-server.service - LSB: Apache Solr as a Service
           Loaded: loaded (/etc/rc.d/init.d/solr-server; generated)
           Active: active (exited) since Fri 2021-08-13 06:22:02 UTC; 1min 30s ago
             Docs: man:systemd-sysv-generator(8)
      
      Aug 13 06:21:51 31fbfd1187ec systemd[1]: Starting LSB: Apache Solr as a Service...
      Aug 13 06:21:51 31fbfd1187ec su[841]: (to solr) root on none
      Aug 13 06:21:52 31fbfd1187ec su[841]: pam_unix(su:session): session opened for user solr(uid=993) by (uid=0)
      Aug 13 06:22:02 31fbfd1187ec solr-server[931]: Starting Apache Solr server in cloud mode[  OK  ]
      Aug 13 06:22:02 31fbfd1187ec systemd[1]: Started LSB: Apache Solr as a Service.
      jenkins@bigtop1slave:~/bigtop/provisioner/docker$ ./docker-hadoop.sh --exec 1 cat /var/log/solr/solr-8983-console.log
      
      The stack size specified is too small, Specify at least 328k
      Error: Could not create the Java Virtual Machine.
      Error: A fatal exception has occurred. Program will exit.
      

      That was not resolved by extending memory limit to 8GB.

      jenkins@bigtop1slave:~/bigtop/provisioner/docker$ docker stats --no-stream
      CONTAINER ID   NAME                              CPU %     MEM USAGE / LIMIT   MEM %     NET I/O          BLOCK I/O       PIDS
      31fbfd1187ec   20210813_061339_r31566_bigtop_1   0.03%     882.9MiB / 8GiB     10.78%    253MB / 1.27MB   655kB / 259MB   28
      

      Attachments

        Activity

          This comment will be Viewable by All Users Viewable by All Users
          Cancel

          People

            sekikn Kengo Seki
            sekikn Kengo Seki
            Votes:
            0 Vote for this issue
            Watchers:
            1 Start watching this issue

            Dates

              Created:
              Updated:
              Resolved:

              Time Tracking

                Estimated:
                Original Estimate - Not Specified
                Not Specified
                Remaining:
                Remaining Estimate - 0h
                0h
                Logged:
                Time Spent - 0.5h
                0.5h

                Slack

                  Issue deployment