This update will remove the @Nullable attribute from jsr305, which may propagate through the code

-though as the only place @Nullable is used in the entire Hadoop codebase is apparently TestMetricsSystemImpl, HADOOP-10067 will have fixed it.

hdfs stops compiling with this patch

hadoop-trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/qjournal/server/Journal.java:[68,32] error: cannot find symbol
[ERROR] symbol:   class Ranges

problem HBase had that also arose in hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/TestDataTransferKeepalive.java

patch fixing two compilation errors after Guava dropped classes

-1 overall. Here are the results of testing the latest attachment
http://issues.apache.org/jira/secure/attachment/12613889/HADOOP-10101-002.patch
against trunk revision .

+1 @author. The patch does not contain any @author tags.

+1 tests included. The patch appears to include 1 new or modified test files.

-1 javac. The applied patch generated 1565 javac compiler warnings (more than the trunk's current 1546 warnings).

+1 javadoc. The javadoc tool did not generate any warning messages.

+1 eclipse:eclipse. The patch built with eclipse:eclipse.

+1 findbugs. The patch does not introduce any new Findbugs (version 1.3.9) warnings.

+1 release audit. The applied patch does not increase the total number of release audit warnings.

+1 core tests. The patch passed unit tests in hadoop-hdfs-project/hadoop-hdfs.

+1 contrib tests. The patch passed contrib unit tests.

Test results: https://builds.apache.org/job/PreCommit-HADOOP-Build/3287//testReport/
Javac warnings: https://builds.apache.org/job/PreCommit-HADOOP-Build/3287//artifact/trunk/patchprocess/diffJavacWarnings.txt
Console output: https://builds.apache.org/job/PreCommit-HADOOP-Build/3287//console

This message is automatically generated.

Link to HDFS-5518: the test/review side of the HDFS patch

Deprecation warnings of

• com.google.common.collect.ComparisonChain
• com.google.common.base.Stopwatch

Replacing them with their successors would change more code but reduce the effort on the next guava update, when perhaps these files go away completely

291a292
> [WARNING] /home/jenkins/jenkins-slave/workspace/PreCommit-HADOOP-Build/trunk/hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/util/VersionUtil.java:[112,6] [deprecation] compare(boolean,boolean) in com.google.common.collect.ComparisonChain has been deprecated
293a295,296
> [WARNING] /home/jenkins/jenkins-slave/workspace/PreCommit-HADOOP-Build/trunk/hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/util/JvmPauseMonitor.java:[156,21] [deprecation] Stopwatch() in com.google.common.base.Stopwatch has been deprecated
> [WARNING] /home/jenkins/jenkins-slave/workspace/PreCommit-HADOOP-Build/trunk/hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/util/JvmPauseMonitor.java:[165,32] [deprecation] elapsedMillis() in com.google.common.base.Stopwatch has been deprecated
432a436,439
> [WARNING] /home/jenkins/jenkins-slave/workspace/PreCommit-HADOOP-Build/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/qjournal/server/Journal.java:[372,19] [deprecation] Stopwatch() in com.google.common.base.Stopwatch has been deprecated
> [WARNING] /home/jenkins/jenkins-slave/workspace/PreCommit-HADOOP-Build/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/qjournal/server/Journal.java:[377,22] [deprecation] elapsedTime(java.util.concurrent.TimeUnit) in com.google.common.base.Stopwatch has been deprecated
> [WARNING] /home/jenkins/jenkins-slave/workspace/PreCommit-HADOOP-Build/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/qjournal/server/Journal.java:[378,10] [deprecation] elapsedTime(java.util.concurrent.TimeUnit) in com.google.common.base.Stopwatch has been deprecated
> [WARNING] /home/jenkins/jenkins-slave/workspace/PreCommit-HADOOP-Build/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/qjournal/server/Journal.java:[380,28] [deprecation] elapsedTime(java.util.concurrent.TimeUnit) in com.google.common.base.Stopwatch has been deprecated
435a443,444
> [WARNING] /home/jenkins/jenkins-slave/workspace/PreCommit-HADOOP-Build/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/qjournal/client/IPCLoggerChannel.java:[133,45] [deprecation] Stopwatch() in com.google.common.base.Stopwatch has been deprecated
> [WARNING] /home/jenkins/jenkins-slave/workspace/PreCommit-HADOOP-Build/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/qjournal/client/IPCLoggerChannel.java:[437,30] [deprecation] elapsedMillis() in com.google.common.base.Stopwatch has been deprecated
437a447,448
> [WARNING] /home/jenkins/jenkins-slave/workspace/PreCommit-HADOOP-Build/trunk/hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/qjournal/server/TestJournalNode.java:[330,19] [deprecation] Stopwatch() in com.google.common.base.Stopwatch has been deprecated
> [WARNING] /home/jenkins/jenkins-slave/workspace/PreCommit-HADOOP-Build/trunk/hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/qjournal/server/TestJournalNode.java:[334,18] [deprecation] elapsedMillis() in com.google.common.base.Stopwatch has been deprecated
438a450,453
> [WARNING] /home/jenkins/jenkins-slave/workspace/PreCommit-HADOOP-Build/trunk/hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/TestMultiThreadedHflush.java:[103,21] [deprecation] Stopwatch() in com.google.common.base.Stopwatch has been deprecated
> [WARNING] /home/jenkins/jenkins-slave/workspace/PreCommit-HADOOP-Build/trunk/hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/TestMultiThreadedHflush.java:[106,22] [deprecation] elapsedTime(java.util.concurrent.TimeUnit) in com.google.common.base.Stopwatch has been deprecated
> [WARNING] /home/jenkins/jenkins-slave/workspace/PreCommit-HADOOP-Build/trunk/hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/TestMultiThreadedHflush.java:[279,21] [deprecation] Stopwatch() in com.google.common.base.Stopwatch has been deprecated
> [WARNING] /home/jenkins/jenkins-slave/workspace/PreCommit-HADOOP-Build/trunk/hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/TestMultiThreadedHflush.java:[284,44] [deprecation] elapsedMillis() in com.google.common.base.Stopwatch has been deprecated
447a463,466
> [WARNING] /home/jenkins/jenkins-slave/workspace/PreCommit-HADOOP-Build/trunk/hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/util/TestChunkedArrayList.java:[72,23] [deprecation] Stopwatch() in com.google.common.base.Stopwatch has been deprecated
> [WARNING] /home/jenkins/jenkins-slave/workspace/PreCommit-HADOOP-Build/trunk/hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/util/TestChunkedArrayList.java:[77,51] [deprecation] elapsedMillis() in com.google.common.base.Stopwatch has been deprecated
> [WARNING] /home/jenkins/jenkins-slave/workspace/PreCommit-HADOOP-Build/trunk/hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/util/TestChunkedArrayList.java:[84,23] [deprecation] Stopwatch() in com.google.common.base.Stopwatch has been deprecated
> [WARNING] /home/jenkins/jenkins-slave/workspace/PreCommit-HADOOP-Build/trunk/hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/util/TestChunkedArrayList.java:[89,51] [deprecation] elapsedMillis() in com.google.common.base.Stopwatch has been deprecated

Any update on this Jira?
If no objection, I would like to take-up and replace all deprecated with their successors..

I'm happy for you to take it up!

Attaching the same patch as in HDFS-5518, with Hadoop QA overall +1.

Hi Steve Loughran, Could you take a look at changes...?
Thanks,

This is the previous patch resynced with trunk and all trailing CR's stripped.

Vinay, can you make sure that your editor/OS/SCM tool isn't adding the wrong line endings?

-1 overall. Here are the results of testing the latest attachment
http://issues.apache.org/jira/secure/attachment/12617772/HADOOP-10101.patch
against trunk revision .

+1 @author. The patch does not contain any @author tags.

+1 tests included. The patch appears to include 4 new or modified test files.

+1 javac. The applied patch does not increase the total number of javac compiler warnings.

+1 javadoc. The javadoc tool did not generate any warning messages.

+1 eclipse:eclipse. The patch built with eclipse:eclipse.

+1 findbugs. The patch does not introduce any new Findbugs (version 1.3.9) warnings.

+1 release audit. The applied patch does not increase the total number of release audit warnings.

-1 core tests. The patch failed these unit tests in hadoop-common-project/hadoop-common hadoop-hdfs-project/hadoop-hdfs:

org.apache.hadoop.hdfs.server.blockmanagement.TestBlocksWithNotEnoughRacks

+1 contrib tests. The patch passed contrib unit tests.

Test results: https://builds.apache.org/job/PreCommit-HADOOP-Build/3367//testReport/
Console output: https://builds.apache.org/job/PreCommit-HADOOP-Build/3367//console

This message is automatically generated.

+1 overall. Here are the results of testing the latest attachment
http://issues.apache.org/jira/secure/attachment/12619314/HADOOP-10101-004.patch
against trunk revision .

+1 @author. The patch does not contain any @author tags.

+1 tests included. The patch appears to include 4 new or modified test files.

+1 javac. The applied patch does not increase the total number of javac compiler warnings.

+1 javadoc. The javadoc tool did not generate any warning messages.

+1 eclipse:eclipse. The patch built with eclipse:eclipse.

+1 findbugs. The patch does not introduce any new Findbugs (version 1.3.9) warnings.

+1 release audit. The applied patch does not increase the total number of release audit warnings.

+1 core tests. The patch passed unit tests in hadoop-common-project/hadoop-common hadoop-hdfs-project/hadoop-hdfs.

+1 contrib tests. The patch passed contrib unit tests.

Test results: https://builds.apache.org/job/PreCommit-HADOOP-Build/3368//testReport/
Console output: https://builds.apache.org/job/PreCommit-HADOOP-Build/3368//console

This message is automatically generated.

Because 3rd party Hadoop dependencies leak to the client and apps, this is an incompatible change (Guava 15 removed some classes). This may affect applications that use those classes avail in older versions of Guava.

IMO, we have to wait till Hadoop 3 for this change. Plus, we should add this to the compat guidelines.

1. marked the patch as targeting 3.0
2. are there any specific applications which you know will break! or is this just a theoretical concern?
3. I did write a section on dependency compatibility for the docs, precisely baca use they are visible. It comes down to "there are no guarantees"

Pint #3 doesn't mean that we should push this in to branch 2, just that we explicitly state that we don't make any guarantees about dependency compatibility -I'd we did we would never be able to upgrade anything

#2, if somebody is using the same Guava classes that break HDFS.

#3, we should figure this out somehow, either using classloader magic or shading to hide all 3rd party dep JARs from the user space. But this is a 3.x timeframe thing.

-yes, but which apps are they? As I don't want us to be constrained fir theoretical reasons.
-MR jobs can already override classloaders
-there's some OSGi JIRAs, which won't break compatibility as they only declare what is exported. Once we have them in, downstream apps than want isolation can use OSGi classloaders with having to try to roll their own solutions

You'll only know when you release with Guava 15 and users apps start breaking.

MR jobs can use classloading isolation, yes. But only on the task side and by default is OFF.

OSGi requires changes in existing apps.

My point is, and I think you agree, we cannot do this in 2.x timeframe.

1. yes, this must be a 3.x patch. I'm only putting in low/no risk updates into branch-2.
2. I would like to identify apps at risk! as otherwise they only know of the problem at ship time.
3. I mentioned OSGi as you mentioned 'classpath magic'. We don't need more magic, and OSGi enabling branch 2 would be transparent to existing apps, while isolating new ones from change.
4. Shading has its own flaws! including stopping ops teams from auditing and updating an installation from a security perspective.

Guava dependency injection pre v16 has been broken by Java 7u51; as/when google backport the fix to the 15.x branch we will need to upgrade -as downstream clients will need to do their own fix.

This patch needs to be rebased, and it is probably okay to move guava towards v17.

-1 overall. Here are the results of testing the latest attachment
http://issues.apache.org/jira/secure/attachment/12619314/HADOOP-10101-004.patch
against trunk revision .

-1 patch. The patch command could not apply the patch.

Console output: https://builds.apache.org/job/PreCommit-HADOOP-Build/4096//console

This message is automatically generated.

Haohui Mai, if we make this change, per prior discussion in this JIRA (se above), it will live only in trunk, correct?

Correct me if I'm wrong, this patch should only be committed to trunk based on Steve Loughran's comments.

ok, gr8 we are sync on this going into trunk.

There is another JIRA going on to bump up trunk to jdk1.7.

I think that is related to this as well and we should revive the thread we had in dev@ a while ago to come up with a balance on how to move to newer components without making it too difficult for backporting things to stable releases which cannot move to those new components.

+1 on going to guava 17 on the trunk. If we stuck with 15, it would be obsolete from the moment when hadoop 3 is released.

It would have to be on trunk, guava is a troublespot all round. I'm almost tempted to see if we can repimplement the most common uses (checkArguments, visibleForTesting, and some of the collections) and pull it out.

Some of the fixes for 7u55+ changes are only in more recent versions of Guava, so the update should follow a java 7 migrate in trunk

-1 overall. Here are the results of testing the latest attachment
http://issues.apache.org/jira/secure/attachment/12619314/HADOOP-10101-004.patch
against trunk revision 4be9517.

-1 patch. The patch command could not apply the patch.

Console output: https://builds.apache.org/job/PreCommit-HADOOP-Build/4698//console

This message is automatically generated.

Could Hadoop start using the maven shade plugin to relocate the guava classes it uses into a private package? Then downstream projects that make use of Guava wouldn't be impacted by whatever version Hadoop chooses.

-1 overall. Here are the results of testing the latest attachment
http://issues.apache.org/jira/secure/attachment/12619314/HADOOP-10101-004.patch
against trunk revision 5bd048e.

-1 patch. The patch command could not apply the patch.

Console output: https://builds.apache.org/job/PreCommit-HADOOP-Build/5105//console

This message is automatically generated.

The patch needs to be updated because HADOOP-11032 is committed. BTW, guava v18 removes Enums#valueOfFunction. This patch includes the fix.

-1 overall. Here are the results of testing the latest attachment
http://issues.apache.org/jira/secure/attachment/12695059/HADOOP-10101-005.patch
against trunk revision 9850e15.

+1 @author. The patch does not contain any @author tags.

-1 tests included. The patch doesn't appear to include any new or modified tests.
Please justify why no new tests are needed for this patch.
Also please list what manual steps were performed to verify this patch.

-1 javac. The applied patch generated 1199 javac compiler warnings (more than the trunk's current 1187 warnings).

+1 javadoc. There were no new javadoc warning messages.

+1 eclipse:eclipse. The patch built with eclipse:eclipse.

+1 findbugs. The patch does not introduce any new Findbugs (version 2.0.3) warnings.

+1 release audit. The applied patch does not increase the total number of release audit warnings.

-1 core tests. The patch failed these unit tests in hadoop-common-project/hadoop-common:

org.apache.hadoop.ipc.TestRPCWaitForProxy
org.apache.hadoop.fs.shell.TestXAttrCommands

Test results: https://builds.apache.org/job/PreCommit-HADOOP-Build/5524//testReport/
Javac warnings: https://builds.apache.org/job/PreCommit-HADOOP-Build/5524//artifact/patchprocess/diffJavacWarnings.txt
Console output: https://builds.apache.org/job/PreCommit-HADOOP-Build/5524//console

This message is automatically generated.

The failure of TestXAttrCommands is related. We need to make error handling consistent before and after the change. Attaching new patch to address the issue.

-1 overall. Here are the results of testing the latest attachment
http://issues.apache.org/jira/secure/attachment/12695092/HADOOP-10101-006.patch
against trunk revision caf7298.

+1 @author. The patch does not contain any @author tags.

-1 tests included. The patch doesn't appear to include any new or modified tests.
Please justify why no new tests are needed for this patch.
Also please list what manual steps were performed to verify this patch.

-1 javac. The patch appears to cause the build to fail.

Console output: https://builds.apache.org/job/PreCommit-HADOOP-Build/5527//console

This message is automatically generated.

We need to replace LimitInpuStream with ByteStreams.limit(java.io.InputStream, long) instead in addition to Enums#valueOfFunction.

I think we shouldn't use functions annotated as Beta or Deprecated in upstream Guava.

Attaching patch for fixing it.

Tsuyoshi Ozawa, you might want to take a look at HADOOP-11286 and HADOOP-11470.

-1 overall. Here are the results of testing the latest attachment
http://issues.apache.org/jira/secure/attachment/12695199/HADOOP-10101-007.patch
against trunk revision 7882bc0.

+1 @author. The patch does not contain any @author tags.

-1 tests included. The patch doesn't appear to include any new or modified tests.
Please justify why no new tests are needed for this patch.
Also please list what manual steps were performed to verify this patch.

-1 javac. The applied patch generated 1199 javac compiler warnings (more than the trunk's current 1187 warnings).

+1 javadoc. There were no new javadoc warning messages.

+1 eclipse:eclipse. The patch built with eclipse:eclipse.

+1 findbugs. The patch does not introduce any new Findbugs (version 2.0.3) warnings.

+1 release audit. The applied patch does not increase the total number of release audit warnings.

-1 core tests. The patch failed these unit tests in hadoop-common-project/hadoop-common hadoop-hdfs-project/hadoop-hdfs:

org.apache.hadoop.hdfs.server.namenode.ha.TestRetryCacheWithHA

Test results: https://builds.apache.org/job/PreCommit-HADOOP-Build/5529//testReport/
Javac warnings: https://builds.apache.org/job/PreCommit-HADOOP-Build/5529//artifact/patchprocess/diffJavacWarnings.txt
Console output: https://builds.apache.org/job/PreCommit-HADOOP-Build/5529//console

This message is automatically generated.

1. without this patch, what is the max guava version Hadoop 2.6+ runs against?
2. can this patch (without the actual POM change) go into branch-2
3. ...so then the switch to 18.0 would be the branch-3 change

also, minor, please set a LOCALE_EN in en.toUpperCase() for consistent behaviour round the world. That's a bug in the existing code that could be fixed here

Along that line, if we can maximize the guava compatibility within the hadoop code (which several previous HADOOP JIRAs achieved), that is already a win for users. +1 for making code changes to make things compatible.

As for the patch, note that HADOOP-11286 introduced the Hadoop version of LimitInputStream. It'd be good to use it to remove the dependency on guava's LimitInputStream.

Thanks Steven and Sangjin for your review.

We can upgrade Guava to 14.0.1 without the patch because:
1. LimitInputStream is removed from 15.0: http://docs.guava-libraries.googlecode.com/git-history/5e074af6a526af39a4410a836517e57c72aa50bb/javadoc/com/google/common/io/LimitInputStream.html
2. Enums.valueOfFunction is removed from 18.0: http://docs.guava-libraries.googlecode.com/git-history/v16.0.1/javadoc/com/google/common/base/Enums.html

My understand is as follows:
1. For branch-2, we replace deprecated methods and en.toUpperCase() without pom.xml change(, or should we upgrade the version to 14.0.1?).
2. For trunk, we replace deprecated methods and en.toUpperCase() with pom.xml change(guava will be 18.0?).

Please let me know if I'm wrong. I'll update the patches soon.

YARN-3029 addresses locale problem partially. Linking to the issue.

HADOOP-11286 replaced all use of guava's LimitInputStream with Hadoop's own. It appears a new reference to guava's LimitInputStream has crept in since then.

Since we introduced Hadoop's LimitInputStream, we might as well stick with that approach. I understand it is not perfect (as people can still introduce new references without enforcement), but it still gives us value.

Could you fix that to use Hadoop's LimitInputStream rather than using ByteStreams.limit()? That way it can be committed to branch-2 as well.

If we fix that one, then the compatibility is expanded to prior to 18.0, correct?

Sangjin Lee, Good point, you're correct. We can upgrade guava to 17.0 with hadoop's LimitInputStream instead of ByteStreams.limit.

Am I missing something here? If we push this to branch-2 that would be an incompatible change, we may break applications that that use ByteStreams.limit.

Sorry what I said may sound confusing.

I understand that we are NOT upgrading guava in branch-2. As such, any change that upgrades guava cannot go into branch-2 as is. But code changes that make it compatible with different versions of guava can go into branch-2. I was suggesting pushing those code changes to branch-2. On trunk, we can also upgrade guava.

I noticed that Enums#stringConverter is supported from 16. This change cannot be applied to branch-2 since Guava is not updated in branch-2.

got it, thx

Attaching a patch for trunk:

1. Updating Guava to 18.0.
2. Using org.apache.hadoop.util.LimitInputStream instead of Guava's one.
3. Using Enums.stringConverter instead of Enums.valueOfFunction.
4. Using only Locale.ENGLISH for String#toUpperCase/toLowerCase.

-1 overall. Here are the results of testing the latest attachment
http://issues.apache.org/jira/secure/attachment/12695405/HADOOP-10101-008.patch
against trunk revision e36ef3b.

+1 @author. The patch does not contain any @author tags.

+1 tests included. The patch appears to include 17 new or modified test files.

-1 javac. The applied patch generated 1199 javac compiler warnings (more than the trunk's current 1187 warnings).

+1 javadoc. There were no new javadoc warning messages.

+1 eclipse:eclipse. The patch built with eclipse:eclipse.

-1 findbugs. The patch appears to introduce 26 new Findbugs (version 2.0.3) warnings.

-1 release audit. The applied patch generated 1 release audit warnings.

-1 core tests. The patch failed these unit tests in hadoop-common-project/hadoop-annotations hadoop-common-project/hadoop-auth hadoop-common-project/hadoop-common hadoop-common-project/hadoop-nfs hadoop-hdfs-project/hadoop-hdfs hadoop-hdfs-project/hadoop-hdfs-httpfs hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-app hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-common hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-jobclient hadoop-mapreduce-project/hadoop-mapreduce-examples hadoop-maven-plugins hadoop-tools/hadoop-azure hadoop-tools/hadoop-distcp hadoop-tools/hadoop-extras hadoop-tools/hadoop-gridmix hadoop-tools/hadoop-rumen hadoop-tools/hadoop-streaming hadoop-yarn-project/hadoop-yarn/hadoop-yarn-client hadoop-yarn-project/hadoop-yarn/hadoop-yarn-common hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-applicationhistoryservice hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-common hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager:

org.apache.hadoop.conf.TestJobConf
org.apache.hadoop.mapreduce.TestLargeSort
org.apache.hadoop.mapred.gridmix.TestHighRamJob
org.apache.hadoop.hdfs.server.balancer.TestBalancer

Test results: https://builds.apache.org/job/PreCommit-HADOOP-Build/5542//testReport/
Release audit warnings: https://builds.apache.org/job/PreCommit-HADOOP-Build/5542//artifact/patchprocess/patchReleaseAuditProblems.txt
Findbugs warnings: https://builds.apache.org/job/PreCommit-HADOOP-Build/5542//artifact/patchprocess/newPatchFindbugsWarningshadoop-rumen.html
Findbugs warnings: https://builds.apache.org/job/PreCommit-HADOOP-Build/5542//artifact/patchprocess/newPatchFindbugsWarningshadoop-mapreduce-client-core.html
Findbugs warnings: https://builds.apache.org/job/PreCommit-HADOOP-Build/5542//artifact/patchprocess/newPatchFindbugsWarningshadoop-hdfs-httpfs.html
Findbugs warnings: https://builds.apache.org/job/PreCommit-HADOOP-Build/5542//artifact/patchprocess/newPatchFindbugsWarningshadoop-maven-plugins.html
Javac warnings: https://builds.apache.org/job/PreCommit-HADOOP-Build/5542//artifact/patchprocess/diffJavacWarnings.txt
Console output: https://builds.apache.org/job/PreCommit-HADOOP-Build/5542//console

This message is automatically generated.

• findbugs warnings all seem unrelated
• javac warnings are a mix of spurious and more deprecated warnings; presumably some are guava 18 telling hadoop of for having source compatible with 11.x. Those warnings should be viewed as a sign that those methods will be going away at some point, which will matter for Hadoop-on-Guava 11.

I wouldn't worry about them; just the tests

findbugs warnings all seem unrelated

Raised HDFS-7709 to address the findbug warnings

Thanks Steve for your review. I'll check the javac warnings.

Attaching a patch for trunk. Replaced deprecated methods with proper methods:

• Objects.toStringHelper-> MoreObjects.toStringHelper
• MoreExecutors.sameThreadExecutor -> MoreExecutors.newDirectExecutorService
• Iterators.emptyIterator -> Collections.emptyIterator

-1 overall. Here are the results of testing the latest attachment
http://issues.apache.org/jira/secure/attachment/12696476/HADOOP-10101-009.patch
against trunk revision 42548f4.

+1 @author. The patch does not contain any @author tags.

+1 tests included. The patch appears to include 18 new or modified test files.

-1 javac. The applied patch generated 1151 javac compiler warnings (more than the trunk's current 1150 warnings).

+1 javadoc. There were no new javadoc warning messages.

+1 eclipse:eclipse. The patch built with eclipse:eclipse.

-1 findbugs. The patch appears to introduce 26 new Findbugs (version 2.0.3) warnings.

+1 release audit. The applied patch does not increase the total number of release audit warnings.

-1 core tests. The patch failed these unit tests in hadoop-common-project/hadoop-annotations hadoop-common-project/hadoop-auth hadoop-common-project/hadoop-common hadoop-common-project/hadoop-nfs hadoop-hdfs-project/hadoop-hdfs hadoop-hdfs-project/hadoop-hdfs-httpfs hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-app hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-common hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-jobclient hadoop-mapreduce-project/hadoop-mapreduce-examples hadoop-maven-plugins hadoop-tools/hadoop-azure hadoop-tools/hadoop-distcp hadoop-tools/hadoop-extras hadoop-tools/hadoop-gridmix hadoop-tools/hadoop-rumen hadoop-tools/hadoop-streaming hadoop-yarn-project/hadoop-yarn/hadoop-yarn-client hadoop-yarn-project/hadoop-yarn/hadoop-yarn-common hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-applicationhistoryservice hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-common hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager:

org.apache.hadoop.ipc.TestRPCWaitForProxy
org.apache.hadoop.mapreduce.TestLargeSort
org.apache.hadoop.conf.TestJobConf
org.apache.hadoop.mapred.gridmix.TestHighRamJob

Test results: https://builds.apache.org/job/PreCommit-HADOOP-Build/5579//testReport/
Findbugs warnings: https://builds.apache.org/job/PreCommit-HADOOP-Build/5579//artifact/patchprocess/newPatchFindbugsWarningshadoop-mapreduce-client-core.html
Findbugs warnings: https://builds.apache.org/job/PreCommit-HADOOP-Build/5579//artifact/patchprocess/newPatchFindbugsWarningshadoop-maven-plugins.html
Findbugs warnings: https://builds.apache.org/job/PreCommit-HADOOP-Build/5579//artifact/patchprocess/newPatchFindbugsWarningshadoop-hdfs-httpfs.html
Findbugs warnings: https://builds.apache.org/job/PreCommit-HADOOP-Build/5579//artifact/patchprocess/newPatchFindbugsWarningshadoop-rumen.html
Javac warnings: https://builds.apache.org/job/PreCommit-HADOOP-Build/5579//artifact/patchprocess/diffJavacWarnings.txt
Console output: https://builds.apache.org/job/PreCommit-HADOOP-Build/5579//console

This message is automatically generated.

Steve Loughran I checked Javac warnings: these warnings look not related to the patch. I also checked findbugs are not related. Could you take a look at 009 patch?

I also checked findbugs are not related.

I've attached patch to fix HDFS-7709, kindly review it. Thanks!

Rakesh's patch was merged on HDFS-7709.

Attaching v9 patch again.

-1 overall. Here are the results of testing the latest attachment
http://issues.apache.org/jira/secure/attachment/12698468/HADOOP-10101-009.patch
against trunk revision 3826277.

-1 patch. Trunk compilation may be broken.

Console output: https://builds.apache.org/job/PreCommit-HADOOP-Build/5673//console

This message is automatically generated.

Refreshed a patch.

Refreshed a patch.

-1 overall. Here are the results of testing the latest attachment
http://issues.apache.org/jira/secure/attachment/12698904/HADOOP-10101-010.patch
against trunk revision 6804d68.

+1 @author. The patch does not contain any @author tags.

+1 tests included. The patch appears to include 18 new or modified test files.

-1 javac. The applied patch generated 1152 javac compiler warnings (more than the trunk's current 1151 warnings).

+1 javadoc. There were no new javadoc warning messages.

+1 eclipse:eclipse. The patch built with eclipse:eclipse.

-1 findbugs. The patch appears to introduce 19 new Findbugs (version 2.0.3) warnings.

+1 release audit. The applied patch does not increase the total number of release audit warnings.

-1 core tests. The patch failed these unit tests in hadoop-common-project/hadoop-annotations hadoop-common-project/hadoop-auth hadoop-common-project/hadoop-common hadoop-common-project/hadoop-nfs hadoop-hdfs-project/hadoop-hdfs hadoop-hdfs-project/hadoop-hdfs-httpfs hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-app hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-common hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-jobclient hadoop-mapreduce-project/hadoop-mapreduce-examples hadoop-maven-plugins hadoop-tools/hadoop-azure hadoop-tools/hadoop-distcp hadoop-tools/hadoop-extras hadoop-tools/hadoop-gridmix hadoop-tools/hadoop-rumen hadoop-tools/hadoop-streaming hadoop-yarn-project/hadoop-yarn/hadoop-yarn-client hadoop-yarn-project/hadoop-yarn/hadoop-yarn-common hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-applicationhistoryservice hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-common hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager:

org.apache.hadoop.conf.TestJobConf
org.apache.hadoop.mapred.gridmix.TestHighRamJob

Test results: https://builds.apache.org/job/PreCommit-HADOOP-Build/5696//testReport/
Findbugs warnings: https://builds.apache.org/job/PreCommit-HADOOP-Build/5696//artifact/patchprocess/newPatchFindbugsWarningshadoop-mapreduce-client-core.html
Findbugs warnings: https://builds.apache.org/job/PreCommit-HADOOP-Build/5696//artifact/patchprocess/newPatchFindbugsWarningshadoop-maven-plugins.html
Findbugs warnings: https://builds.apache.org/job/PreCommit-HADOOP-Build/5696//artifact/patchprocess/newPatchFindbugsWarningshadoop-yarn-server-resourcemanager.html
Javac warnings: https://builds.apache.org/job/PreCommit-HADOOP-Build/5696//artifact/patchprocess/diffJavacWarnings.txt
Console output: https://builds.apache.org/job/PreCommit-HADOOP-Build/5696//console

This message is automatically generated.

-1 overall. Here are the results of testing the latest attachment
http://issues.apache.org/jira/secure/attachment/12698912/HADOOP-10101-010.patch
against trunk revision ef950ea.

+1 @author. The patch does not contain any @author tags.

+1 tests included. The patch appears to include 18 new or modified test files.

-1 javac. The applied patch generated 1152 javac compiler warnings (more than the trunk's current 1151 warnings).

+1 javadoc. There were no new javadoc warning messages.

+1 eclipse:eclipse. The patch built with eclipse:eclipse.

-1 findbugs. The patch appears to introduce 19 new Findbugs (version 2.0.3) warnings.

+1 release audit. The applied patch does not increase the total number of release audit warnings.

-1 core tests. The patch failed these unit tests in hadoop-common-project/hadoop-annotations hadoop-common-project/hadoop-auth hadoop-common-project/hadoop-common hadoop-common-project/hadoop-nfs hadoop-hdfs-project/hadoop-hdfs hadoop-hdfs-project/hadoop-hdfs-httpfs hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-app hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-common hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-jobclient hadoop-mapreduce-project/hadoop-mapreduce-examples hadoop-maven-plugins hadoop-tools/hadoop-azure hadoop-tools/hadoop-distcp hadoop-tools/hadoop-extras hadoop-tools/hadoop-gridmix hadoop-tools/hadoop-rumen hadoop-tools/hadoop-streaming hadoop-yarn-project/hadoop-yarn/hadoop-yarn-client hadoop-yarn-project/hadoop-yarn/hadoop-yarn-common hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-applicationhistoryservice hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-common hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager:

org.apache.hadoop.conf.TestJobConf
org.apache.hadoop.mapred.gridmix.TestHighRamJob
org.apache.hadoop.fs.http.client.TestHttpFSFWithWebhdfsFileSystem

Test results: https://builds.apache.org/job/PreCommit-HADOOP-Build/5698//testReport/
Findbugs warnings: https://builds.apache.org/job/PreCommit-HADOOP-Build/5698//artifact/patchprocess/newPatchFindbugsWarningshadoop-maven-plugins.html
Findbugs warnings: https://builds.apache.org/job/PreCommit-HADOOP-Build/5698//artifact/patchprocess/newPatchFindbugsWarningshadoop-mapreduce-client-core.html
Findbugs warnings: https://builds.apache.org/job/PreCommit-HADOOP-Build/5698//artifact/patchprocess/newPatchFindbugsWarningshadoop-yarn-server-resourcemanager.html
Javac warnings: https://builds.apache.org/job/PreCommit-HADOOP-Build/5698//artifact/patchprocess/diffJavacWarnings.txt
Console output: https://builds.apache.org/job/PreCommit-HADOOP-Build/5698//console

This message is automatically generated.

The test failures are not related to the patch. Steve Loughran, do you mind taking a look? I'll create a patches for branch-2 after trunk code.

Talked with Steve offline. The patch is now very large, so we should split them:

1. Fix up usage of Guava's methods for both branch-2 and trunk on HADOOP-11600.
2. Locale bug is addressed on HADOOP-11602.
3. trunk-specific change to upgrade Guava 18.0 on this issue(HADOOP-10101).

3 depends on 1, so I'll submit a patch after 1 is committed.

This patch is only for trunk and can be applied after HADOOP-11600 is committed.

Submitting v11 patch since HADOOP-11600 was committed.

-1 overall. Here are the results of testing the latest attachment
http://issues.apache.org/jira/secure/attachment/12699092/HADOOP-10101-011.patch
against trunk revision f0412de.

+1 @author. The patch does not contain any @author tags.

+1 tests included. The patch appears to include 1 new or modified test files.

-1 javac. The applied patch generated 1152 javac compiler warnings (more than the trunk's current 1151 warnings).

+1 javadoc. There were no new javadoc warning messages.

+1 eclipse:eclipse. The patch built with eclipse:eclipse.

+1 findbugs. The patch does not introduce any new Findbugs (version 2.0.3) warnings.

+1 release audit. The applied patch does not increase the total number of release audit warnings.

+1 core tests. The patch passed unit tests in hadoop-common-project/hadoop-common hadoop-hdfs-project/hadoop-hdfs.

Test results: https://builds.apache.org/job/PreCommit-HADOOP-Build/5727//testReport/
Javac warnings: https://builds.apache.org/job/PreCommit-HADOOP-Build/5727//artifact/patchprocess/diffJavacWarnings.txt
Console output: https://builds.apache.org/job/PreCommit-HADOOP-Build/5727//console

This message is automatically generated.

The javac warning looks not related to the patch. Steve Loughran, could you take a look at v11 patch?

I think Sean Busbey made a good point. I'm sure there are some practical concerns, but as more libraries start to depend on more recent guava versions it becomes more of a problem.
For now I've taken the reverse approach for our integration tests (shading dropwizard 0.8.1 w. relocated guava classes) to allow both hadoop and dropwizard services to run in one jvm.

I hit something similar in tez recently and looking at this jira, I am wondering as to why shading ( relocating all classes ) the guava jar is not an option?

Consider the following approach:

• introduce a new hadoop-guava module which depends on guava version X.
• hadoop-guava relocates all classes in guava to org.apache.hadoop.shaded.guava
• all code in hadoop depends on hadoop-guava and excludes the google-guava dependency

I did try a quick prototype approach on branch-2 and it seems to function without directly pulling in guava ( except of course - the curator guava dependency problem )

-1 overall

This message was automatically generated.

Github user afs commented on the pull request:

https://github.com/apache/jena/pull/87#issuecomment-123468596

org.apache.hadoop:hadoop-common:jar:2.6.0 depends on com.google.guava:guava:jar:11.0.2 (provided).

Just because Jena tests work isn't evidence Guava 16.0.1 works unfortunately. Jena is a library and may be co-resident with other code. See HADOOP-10101 including the initial attempt at migration to 15.0 which did not go in. At this very late stage, it needs a strong advantage for users for change.

(and this is why we shade guava for Jena!)

-1 overall

This message was automatically generated.

Ping, any plan on upgrading Guava in this thread? btw, Guava is on 19.0.

-1 overall

This message was automatically generated.

I think it is high time we bump up the dependency however we should look to hiding it too i.e. shading it as needed so that downstream users can use whatever version of guava they want to use.

Some questions:

• why version 18 and not 19? Guava 19 was released in 2015 or so
• have you confirmed that no deprecated and/or beta apis are being used?
• I believe there are some places where guava objects are being used in public facing APIs. This might not be something to address in this jira but should definitely be looked at as a blocker before the next release in addition to my initial comment.

-1 overall

This message was automatically generated.

whatever the latest version goes hitesh, that's what the title says. the Hadoop code is designed to work with later versions (i.e we've moved off bits that were taken away), but still ship with an older one. I'd love to bump us up to a new one

We hit one problem with Guava version 18 for HDFS, The application we built is based on grpc, guava version 18 is minimum required

Exception in thread "main" java.lang.NoSuchMethodError: com.google.common.base.Enums.valueOfFunction(Ljava/lang/Class;)Lcom/google/common/base/Function;
at org.apache.hadoop.fs.shell.XAttrCommands$GetfattrCommand.<clinit>(XAttrCommands.java:69)
at sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native Method)
at sun.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:62)
at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45)
at java.lang.reflect.Constructor.newInstance(Constructor.java:423)
at org.apache.hadoop.util.ReflectionUtils.newInstance(ReflectionUtils.java:132)
at org.apache.hadoop.fs.shell.CommandFactory.getInstance(CommandFactory.java:125)
at org.apache.hadoop.fs.shell.CommandFactory.getInstance(CommandFactory.java:109)
at org.apache.hadoop.fs.FsShell.run(FsShell.java:290)
at org.apache.hadoop.util.ToolRunner.run(ToolRunner.java:70)
at org.apache.hadoop.util.ToolRunner.run(ToolRunner.java:84)
at org.apache.hadoop.fs.FsShell.main(FsShell.java:352)

> I think it is high time we bump up the dependency however we should look to hiding it too i.e. shading it as needed so that downstream users can use whatever version of guava they want to use.

I agree with Hitesh Shah that we should shade guava. Then, this JIRA is no longer an incompatible change.

-1 overall

This message was automatically generated.

Need to make sure that Guava objects aren't in use in "public" APIs, where public includes those bits of the private APIs which are actually needed to produce working code. There's a lot in YARN, and historically large bits of Hadoop common

Lei Guo based on protobuf experience, I woudn't trust any google lib to have backwards compatibility. Think very hard about that before adopting grpc in any of your own code

I would like to resume working on this jira, based on shading approach as Nicholas and Hitesh mentioned. Please give me to try the approach for a few days.

why version 18 and not 19? Guava 19 was released in 2015 or so

It's because Guava 19 had more incompatible change. Now, 21.0, which requires JDK-8 or upper, has been released. I would like to create a patch to update Guava's version to 21.0, targeting branch-3.

have you confirmed that no deprecated and/or beta apis are being used?

Not yet.

I believe there are some places where guava objects are being used in public facing APIs. This might not be something to address in this jira but should definitely be looked at as a blocker before the next release in addition to my initial comment.

Yeah, let's do this on another jira. I think we should create this wrapper class instead of exposing Guava's objects directly.

I did try a quick prototype approach on branch-2 and it seems to function without directly pulling in guava ( except of course - the curator guava dependency problem )

Curator have also shared Guava. If we can move the version to up to date version, we can avoid the problem.

Need to make sure that Guava objects aren't in use in "public" APIs

Are there any tools that analyze public vs. private usage of dependencies? Seems important but I've not seen any.

where public includes those bits of the private APIs which are actually needed to produce working code.

Can you elaborate on this? My (naive) impression was that anything not "leaked" in method-signatures could be shaded.

Is this addressed by using the shaded client introduced by HADOOP-11804 for Hadoop 3? It shades Guava also.

The issue for APIs is that public APIs can't shade things, as then only callers with matching shade declarations can use them (personal experience there with hive & spark & kryo). It's ok privately, but the issue is defining "private" is a hard one, given that things like UGI have for a long time been tagged as private. Essentially: anything down as @LimitedPrivate for MapReduce has to be assumed as "needed by YARN apps" and touched very, very carefully

Thanks Andrew for sharing HADOOP-11804. Now, we can focus on upgrading guava here. The exposed objects of Guava may be a problem as Steve mentioned. I will create a following jira to address the issue.

Attaching a patch to upgrade guava to run tests.

I chose version 21.0, the latest version of Guava. It requires jdk 8 or later and fits in hadoop 3.0.

-1 overall

This message was automatically generated.

On my local, compilation failed with the patch by changes of Guava's interface. Updating it soon.

Uploading v13 patch.

-1 overall

This message was automatically generated.

IIRC, YarnClient has a guava object being used in one of its public APIs.

OIC, we need to upgrade Curator on YARN-3774 at first. I would like to take Curator's update before solving this issue.

After working to update Curator for a while, I found that Curator 3.x only supports ZooKeeper 3.5.x. On HADOOP-14187, avoiding the drastic change(YARN-3774) now, I would like to suggest updating ZooKeeper dependency to 3.4.9 and Curator dependency to 2.12.0, which supports ZooKeeper 3.4.x. It's because our current focus here is to update Guava - the blocker of this issue is to introduce Curator which includes shaded Guava.

I tested with v13 patch on HADOOP-14187 on my local:

mvn install test -Dtest=TestZKDelegationTokenSecretManager,TestChildReaper,TestKDiag,TestDelegationTokenRenewer,TestSubmitApplicationWithRMHA,TestLeaderElectorService,TestRMHAForNodeLabels,TestZKRMStateStorePerf,TestKillApplicationWithRMHA,TestRMStoreCommands,TestZKRMStateStore,TestRMRestart,TestZKRMStateStoreZKClientConnections,TestReservationSystemWithRMHA

All test passes on my local basically. Flaky tests sometimes fail, but they're not ZK-related. The reason seems to be YARN-5548.

Since HADOOP-14187 is merged, I'm resubmitting Jenkins CI again.

Hitesh Shah thanks for sharing the information. IIUC, is the problem method AMRMClient#waitFor? It takes a com.google.common.base.Supplier instance as an argument.

-1 overall

This message was automatically generated.

Uploading v15 patch. It fixes checkstyle warning. javac warnings and TestKDiag failure are not related to the patch: javac warnings are by using deprecated methods which are not related to the patch, and TestKDiag passes on my local.

-1 overall

This message was automatically generated.

Patch looks good. Thanks for working on this well known hard issue – Hadoop is stuck on Guava 11.

Please take a look the javac/checkstyle warnings. The test failures seem unrelated. Please take a look as well. Thanks.

Fixing checkstyle warning again.

Tsz Wo Nicholas Sze thanks for taking a look! More taking a look at deeper, javac warnings were caused by deprecated warnings after upgrading Guava, as you mentioned.I will update it soon. Canceling a patch.

Fixing javac warnings.

-1 overall

This message was automatically generated.

The above report is with v16, not with v17. v17 includes fixes against javac warnings, so please wait for a report with v17 patch.

-1 overall

This message was automatically generated.

Fixing an additional checkstyle warning - removing unused import.

Another warning checkstyle raised looks checkstyle's bug - indentation should be 6 at the line, and it is now with the patch. https://github.com/checkstyle/checkstyle/issues/1174

./hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/JournalSet.java:66: return ComparisonChain.start().compareFalseFirst(!elis1.isLocalLog(),: 'method def' child have incorrect indentation level 6, expected level should be one of the following: 8, 10, 12. [Indentation]