[BEAM-10157] updateOfflineRepository task shows fatal error on flogger, resulting in artifacts not downloaded

• Type: Task
• Status: Resolved
• Priority: P3
• Resolution: Fixed
• Fix Version/s: Missing
• Component/s: build-system
• Assignee: Unassigned
• Reporter: Tomo Suzuki
• Created: 29/May/20 21:33
• Updated: 13/Jan/22 21:48
• Resolved: 21/May/21 15:04

Description

Step to reproduce.

Run updateOfflineRepository

suztomo-macbookpro44% ./gradlew \                                              
    :model:fn-execution:updateOfflineRepository \
    :model:job-management:updateOfflineRepository \
    :model:pipeline:updateOfflineRepository \
    :sdks:java:core:updateOfflineRepository \
    :sdks:java:extensions:google-cloud-platform-core:updateOfflineRepository \
    :sdks:java:extensions:join-library:updateOfflineRepository \
    :sdks:java:extensions:protobuf:updateOfflineRepository \
    :sdks:java:extensions:sql:updateOfflineRepository \
    :sdks:java:extensions:sql:datacatalog:updateOfflineRepository \
    :sdks:java:extensions:sql:zetasql:updateOfflineRepository \
    :sdks:java:fn-execution:updateOfflineRepository \
    :sdks:java:harness:updateOfflineRepository \
    :sdks:java:io:google-cloud-platform:updateOfflineRepository \
    :sdks:java:io:kafka:updateOfflineRepository \
    :sdks:java:io:mongodb:updateOfflineRepository \
    :sdks:java:io:parquet:updateOfflineRepository \
    :sdks:java:testing:nexmark:updateOfflineRepository \
    :runners:core-construction-java:updateOfflineRepository \
    :runners:core-java:updateOfflineRepository \
    :runners:direct-java:updateOfflineRepository \
    :runners:google-cloud-dataflow-java:updateOfflineRepository \
    :runners:google-cloud-dataflow-java:worker:updateOfflineRepository \
    :runners:java-fn-execution:updateOfflineRepository \
    :runners:local-java:updateOfflineRepository \
    :examples:java:updateOfflineRepository \
    :vendor:bytebuddy-1_10_8:updateOfflineRepository\
    :vendor:grpc-1_26_0:updateOfflineRepository \
    :vendor:guava-26_0-jre:updateOfflineRepository\
    :vendor:sdks-java-extensions-protobuf:updateOfflineRepository \
    --no-parallel \
    -PofflineRepositoryRoot=/tmp/offline-plugin
...
> Task :sdks:java:extensions:google-cloud-platform-core:updateOfflineRepository
1 problem was encountered while building the effective model for com.google.flogger:flogger:0.5.1
[FATAL] 'dependencies.dependency. com.google.flogger:flogger:0.5.1' for com.google.flogger:flogger:0.5.1 is referencing itself. @ 
: [[FATAL] 'dependencies.dependency. com.google.flogger:flogger:0.5.1' for com.google.flogger:flogger:0.5.1 is referencing itself. @ com.google.flogger:flogger:0.5.1]

...

This circular dependency is a problem in flogger. https://github.com/google/flogger/issues/152

Set beam project to use /tmp/offline-plugin

suztomo-macbookpro44% git diff
diff --git a/gradle.properties b/gradle.properties
index 22f2c251bd..a5cc583a91 100644
--- a/gradle.properties
+++ b/gradle.properties
@@ -19,7 +19,7 @@ org.gradle.caching=true
 org.gradle.parallel=true
 org.gradle.configureondemand=true
 org.gradle.jvmargs=-Xss10240k
-offlineRepositoryRoot=offline-repository
+offlineRepositoryRoot=/tmp/offline-plugin
 signing.gnupg.executable=gpg
 signing.gnupg.useLegacyGpg=true
 
diff --git a/settings.gradle b/settings.gradle
index 058213798e..07e7ecb0b3 100644
--- a/settings.gradle
+++ b/settings.gradle
@@ -16,6 +16,14 @@
  * limitations under the License.
  */
 
+buildscript {
+  repositories {
+    maven {
+      url '/tmp/offline-plugin'
+    }
+  }
+}
+
 rootProject.name = "beam"
 
 include ":release"

Run the project with offline flag

Offline Gradle build cannot find com.google.cloud:google-cloud-bigquery:1.108.0 and com.google.http-client:google-http-client-protobuf:1.34.0:

suztomo-macbookpro44% ./gradlew --offline :runners:google-cloud-dataflow-java:compileJava
Configuration on demand is an incubating feature.

...

> Task :sdks:java:io:google-cloud-platform:compileJava FAILED

FAILURE: Build failed with an exception.

* What went wrong:
Execution failed for task ':sdks:java:io:google-cloud-platform:compileJava'.
> Could not resolve all files for configuration ':sdks:java:io:google-cloud-platform:compileClasspath'.
   > Could not find com.google.cloud:google-cloud-bigquery:1.108.0.
     Searched in the following locations:
       - file:/tmp/offline-plugin/com/google/cloud/google-cloud-bigquery/1.108.0/google-cloud-bigquery-1.108.0.pom
       - file:/tmp/offline-plugin/com/google/cloud/google-cloud-bigquery/1.108.0/google-cloud-bigquery-1.108.0.jar
     Required by:
         project :sdks:java:io:google-cloud-platform > com.google.cloud:google-cloud-bigquerystorage:0.125.0-beta
   > Could not find com.google.http-client:google-http-client-protobuf:1.34.0.
     Searched in the following locations:
       - file:/tmp/offline-plugin/com/google/http-client/google-http-client-protobuf/1.34.0/google-http-client-protobuf-1.34.0.pom
       - file:/tmp/offline-plugin/com/google/http-client/google-http-client-protobuf/1.34.0/google-http-client-protobuf-1.34.0.jar
     Required by:
         project :sdks:java:io:google-cloud-platform > com.google.cloud.datastore:datastore-v1-proto-client:1.6.3

Git commit I tried

suztomo-macbookpro44% git log -1
commit d87fc525e820f78c4546f3f0f5bd5218e3ce323f (HEAD -> master_offline, origin/master, origin/HEAD, master)
Author: Robert Bradshaw <[e-mail]>
Date:   Fri May 29 12:24:48 2020 -0700

    [BEAM-7746] Fix type errors and enable checks for apache_beam.dataframe.* (#11632)
suztomo-macbookpro44% grep google-cloud-bigquery buildSrc/src/main/groovy/org/apache/beam/gradle/BeamModulePlugin.groovy 
        google_cloud_bigquery                       : "com.google.cloud:google-cloud-bigquery:1.108.0",
        google_cloud_bigquery_storage               : "com.google.cloud:google-cloud-bigquerystorage:0.125.0-beta",
        proto_google_cloud_bigquery_storage_v1beta1 : "com.google.api.grpc:proto-google-cloud-bigquerystorage-v1beta1:0.85.1",