Details
-
Improvement
-
Status: Resolved
-
Major
-
Resolution: Fixed
-
3.4.1
Description
It's easier to invoke methods through reflection through parquet/iceberg DynMethods if the invoked method raises unchecked exceptions, because it doesn't then rewrape the raised exception in a generic RuntimeException
Catching the IOEs and wrapping as UncheckedIOEs makes it much easier to unwrap IOEs after the invocation
Attachments
Issue Links
- relates to
-
HADOOP-18679 Add BulkDelete API for paged delete of files and objects
- Resolved
- links to
Activity
hadoop-yetus commented on PR #6885:
URL: https://github.com/apache/hadoop/pull/6885#issuecomment-2163294710
:confetti_ball: *+1 overall*
Vote | Subsystem | Runtime | Logfile | Comment |
:----: | ----------: | --------: | :--------: | :-------: |
+0 :ok: | reexec | 0m 19s | Docker mode activated. | |
_ Prechecks _ | ||||
---|---|---|---|---|
+1 :green_heart: | dupname | 0m 1s | No case conflicting files found. | |
+0 :ok: | codespell | 0m 0s | codespell was not available. | |
+0 :ok: | detsecrets | 0m 0s | detect-secrets was not available. | |
+1 :green_heart: | @author | 0m 0s | The patch does not contain any @author tags. | |
+1 :green_heart: | test4tests | 0m 0s | The patch appears to include 1 new or modified test files. | |
_ trunk Compile Tests _ | ||||
+1 :green_heart: | mvninstall | 32m 43s | trunk passed | |
+1 :green_heart: | compile | 8m 55s | trunk passed with JDK Ubuntu-11.0.23+9-post-Ubuntu-1ubuntu120.04.2 | |
+1 :green_heart: | compile | 8m 6s | trunk passed with JDK Private Build-1.8.0_412-8u412-ga-1~20.04.1-b08 | |
+1 :green_heart: | checkstyle | 0m 41s | trunk passed | |
+1 :green_heart: | mvnsite | 0m 57s | trunk passed | |
+1 :green_heart: | javadoc | 0m 41s | trunk passed with JDK Ubuntu-11.0.23+9-post-Ubuntu-1ubuntu120.04.2 | |
+1 :green_heart: | javadoc | 0m 32s | trunk passed with JDK Private Build-1.8.0_412-8u412-ga-1~20.04.1-b08 | |
+1 :green_heart: | spotbugs | 1m 28s | trunk passed | |
+1 :green_heart: | shadedclient | 21m 3s | branch has no errors when building and testing our client artifacts. | |
_ Patch Compile Tests _ | ||||
+1 :green_heart: | mvninstall | 0m 31s | the patch passed | |
+1 :green_heart: | compile | 8m 24s | the patch passed with JDK Ubuntu-11.0.23+9-post-Ubuntu-1ubuntu120.04.2 | |
+1 :green_heart: | javac | 8m 24s | the patch passed | |
+1 :green_heart: | compile | 8m 12s | the patch passed with JDK Private Build-1.8.0_412-8u412-ga-1~20.04.1-b08 | |
+1 :green_heart: | javac | 8m 12s | the patch passed | |
+1 :green_heart: | blanks | 0m 0s | The patch has no blanks issues. | |
-0 :warning: | checkstyle | 0m 38s | [/results-checkstyle-hadoop-common-project_hadoop-common.txt](https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-6885/1/artifact/out/results-checkstyle-hadoop-common-project_hadoop-common.txt) | hadoop-common-project/hadoop-common: The patch generated 3 new + 3 unchanged - 1 fixed = 6 total (was 4) |
+1 :green_heart: | mvnsite | 0m 53s | the patch passed | |
+1 :green_heart: | javadoc | 0m 40s | the patch passed with JDK Ubuntu-11.0.23+9-post-Ubuntu-1ubuntu120.04.2 | |
+1 :green_heart: | javadoc | 0m 35s | the patch passed with JDK Private Build-1.8.0_412-8u412-ga-1~20.04.1-b08 | |
+1 :green_heart: | spotbugs | 1m 33s | the patch passed | |
+1 :green_heart: | shadedclient | 21m 30s | patch has no errors when building and testing our client artifacts. | |
_ Other Tests _ | ||||
+1 :green_heart: | unit | 16m 32s | hadoop-common in the patch passed. | |
+1 :green_heart: | asflicense | 0m 39s | The patch does not generate ASF License warnings. | |
136m 27s |
Subsystem | Report/Notes |
----------: | :------------- |
Docker | ClientAPI=1.45 ServerAPI=1.45 base: https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-6885/1/artifact/out/Dockerfile |
GITHUB PR | https://github.com/apache/hadoop/pull/6885 |
Optional Tests | dupname asflicense compile javac javadoc mvninstall mvnsite unit shadedclient spotbugs checkstyle codespell detsecrets |
uname | Linux e38782558de5 5.15.0-106-generic #116-Ubuntu SMP Wed Apr 17 09:17:56 UTC 2024 x86_64 x86_64 x86_64 GNU/Linux |
Build tool | maven |
Personality | dev-support/bin/hadoop.sh |
git revision | trunk / 9581c1b9f57e4958adec2a0bfa5fe1f1264162a1 |
Default Java | Private Build-1.8.0_412-8u412-ga-1~20.04.1-b08 |
Multi-JDK versions | /usr/lib/jvm/java-11-openjdk-amd64:Ubuntu-11.0.23+9-post-Ubuntu-1ubuntu120.04.2 /usr/lib/jvm/java-8-openjdk-amd64:Private Build-1.8.0_412-8u412-ga-1~20.04.1-b08 |
Test Results | https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-6885/1/testReport/ |
Max. process+thread count | 1279 (vs. ulimit of 5500) |
modules | C: hadoop-common-project/hadoop-common U: hadoop-common-project/hadoop-common |
Console output | https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-6885/1/console |
versions | git=2.25.1 maven=3.6.3 spotbugs=4.2.2 |
Powered by | Apache Yetus 0.14.0 https://yetus.apache.org |
This message was automatically generated.
hadoop-yetus commented on PR #6885:
URL: https://github.com/apache/hadoop/pull/6885#issuecomment-2165835177
:confetti_ball: *+1 overall*
Vote | Subsystem | Runtime | Logfile | Comment |
:----: | ----------: | --------: | :--------: | :-------: |
+0 :ok: | reexec | 0m 20s | Docker mode activated. | |
_ Prechecks _ | ||||
---|---|---|---|---|
+1 :green_heart: | dupname | 0m 0s | No case conflicting files found. | |
+0 :ok: | codespell | 0m 0s | codespell was not available. | |
+0 :ok: | detsecrets | 0m 0s | detect-secrets was not available. | |
+1 :green_heart: | @author | 0m 0s | The patch does not contain any @author tags. | |
+1 :green_heart: | test4tests | 0m 0s | The patch appears to include 1 new or modified test files. | |
_ trunk Compile Tests _ | ||||
+1 :green_heart: | mvninstall | 38m 11s | trunk passed | |
+1 :green_heart: | compile | 10m 32s | trunk passed with JDK Ubuntu-11.0.23+9-post-Ubuntu-1ubuntu120.04.2 | |
+1 :green_heart: | compile | 9m 22s | trunk passed with JDK Private Build-1.8.0_412-8u412-ga-1~20.04.1-b08 | |
+1 :green_heart: | checkstyle | 0m 48s | trunk passed | |
+1 :green_heart: | mvnsite | 1m 2s | trunk passed | |
+1 :green_heart: | javadoc | 0m 48s | trunk passed with JDK Ubuntu-11.0.23+9-post-Ubuntu-1ubuntu120.04.2 | |
+1 :green_heart: | javadoc | 0m 34s | trunk passed with JDK Private Build-1.8.0_412-8u412-ga-1~20.04.1-b08 | |
+1 :green_heart: | spotbugs | 1m 34s | trunk passed | |
+1 :green_heart: | shadedclient | 24m 51s | branch has no errors when building and testing our client artifacts. | |
_ Patch Compile Tests _ | ||||
+1 :green_heart: | mvninstall | 0m 33s | the patch passed | |
+1 :green_heart: | compile | 9m 42s | the patch passed with JDK Ubuntu-11.0.23+9-post-Ubuntu-1ubuntu120.04.2 | |
+1 :green_heart: | javac | 9m 42s | the patch passed | |
+1 :green_heart: | compile | 8m 59s | the patch passed with JDK Private Build-1.8.0_412-8u412-ga-1~20.04.1-b08 | |
+1 :green_heart: | javac | 8m 59s | the patch passed | |
+1 :green_heart: | blanks | 0m 0s | The patch has no blanks issues. | |
-0 :warning: | checkstyle | 0m 39s | [/results-checkstyle-hadoop-common-project_hadoop-common.txt](https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-6885/2/artifact/out/results-checkstyle-hadoop-common-project_hadoop-common.txt) | hadoop-common-project/hadoop-common: The patch generated 1 new + 3 unchanged - 1 fixed = 4 total (was 4) |
+1 :green_heart: | mvnsite | 0m 58s | the patch passed | |
+1 :green_heart: | javadoc | 0m 45s | the patch passed with JDK Ubuntu-11.0.23+9-post-Ubuntu-1ubuntu120.04.2 | |
+1 :green_heart: | javadoc | 0m 33s | the patch passed with JDK Private Build-1.8.0_412-8u412-ga-1~20.04.1-b08 | |
+1 :green_heart: | spotbugs | 1m 39s | the patch passed | |
+1 :green_heart: | shadedclient | 25m 7s | patch has no errors when building and testing our client artifacts. | |
_ Other Tests _ | ||||
+1 :green_heart: | unit | 16m 29s | hadoop-common in the patch passed. | |
+1 :green_heart: | asflicense | 0m 40s | The patch does not generate ASF License warnings. | |
154m 55s |
Subsystem | Report/Notes |
----------: | :------------- |
Docker | ClientAPI=1.45 ServerAPI=1.45 base: https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-6885/2/artifact/out/Dockerfile |
GITHUB PR | https://github.com/apache/hadoop/pull/6885 |
Optional Tests | dupname asflicense compile javac javadoc mvninstall mvnsite unit shadedclient spotbugs checkstyle codespell detsecrets |
uname | Linux e3e2e3284b42 5.15.0-106-generic #116-Ubuntu SMP Wed Apr 17 09:17:56 UTC 2024 x86_64 x86_64 x86_64 GNU/Linux |
Build tool | maven |
Personality | dev-support/bin/hadoop.sh |
git revision | trunk / e9c38f5879c3664b07e3bc80b37db296b772f21d |
Default Java | Private Build-1.8.0_412-8u412-ga-1~20.04.1-b08 |
Multi-JDK versions | /usr/lib/jvm/java-11-openjdk-amd64:Ubuntu-11.0.23+9-post-Ubuntu-1ubuntu120.04.2 /usr/lib/jvm/java-8-openjdk-amd64:Private Build-1.8.0_412-8u412-ga-1~20.04.1-b08 |
Test Results | https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-6885/2/testReport/ |
Max. process+thread count | 1264 (vs. ulimit of 5500) |
modules | C: hadoop-common-project/hadoop-common U: hadoop-common-project/hadoop-common |
Console output | https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-6885/2/console |
versions | git=2.25.1 maven=3.6.3 spotbugs=4.2.2 |
Powered by | Apache Yetus 0.14.0 https://yetus.apache.org |
This message was automatically generated.
steveloughran commented on PR #6885:
URL: https://github.com/apache/hadoop/pull/6885#issuecomment-2167753815
checkstyle is complaining about the _ in the method names.
@mukund-thakur this should be good for review.
mukund-thakur commented on code in PR #6885:
URL: https://github.com/apache/hadoop/pull/6885#discussion_r1643437675
##########
hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/util/functional/FunctionalIO.java:
##########
@@ -0,0 +1,99 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.hadoop.util.functional;
+
+import java.io.IOException;
+import java.io.UncheckedIOException;
+import java.util.function.Supplier;
+
+import org.apache.hadoop.classification.InterfaceAudience;
+
+/**
+ * Functional utilities for IO operations.
+ */
+@InterfaceAudience.Private
+public final class FunctionalIO {
+
+ private FunctionalIO()
+
+ /**
+ * Invoke any operation, wrapping IOExceptions with
+ *
.
+ * @param call callable
+ * @param <T> type of result
+ * @return result
+ * @throws UncheckedIOException if an IOE was raised.
+ */
+ public static <T> T uncheckIOExceptions(CallableRaisingIOE<T> call) {
+ try
catch (IOException e)
{ + throw new UncheckedIOException(e); + }+ }
+
+ /**
+ * Wrap a
as a
{@link Supplier}.
+ * This is similar to
, except that
+ * only IOExceptions are caught and wrapped; all other exceptions are
+ * propagated unchanged.
+ * @param <T> type of result
+ */
+ private static final class UncheckedIOExceptionSupplier<T> implements Supplier<T> {
Review Comment:
I don't see any use of this so for though. Assuming it will be used later?
steveloughran opened a new pull request, #6885:
URL: https://github.com/apache/hadoop/pull/6885
-WrappedIO methods raise UncheckedIOEs
-new class org.apache.hadoop.util.functional.FunctionalIO
with wrap/unwrap and the ability to generate a
java.util.function.Supplier around a CallableRaisingIOE.
-Tests
HADOOP-17799. Your PR title ...')?