Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

BIGTOP-4028 : Excluding log dependencies corresponding to hbase in the hive service #1200

Open
wants to merge 1 commit into
base: master
Choose a base branch
from

Conversation

LiJie20190102
Copy link
Contributor

…e hive service

Description of PR

Excluding log dependencies corresponding to hbase in the hive service
before:
image

after:
image

How was this patch tested?

Launch hive CLI to view startup logs

For code changes:

  • [1 ] Does the title or this PR starts with the corresponding JIRA issue id (e.g. 'BIGTOP-3638. Your PR title ...')?
  • [1] Make sure that newly added files do not have any licensing issues. When in doubt refer to https://www.apache.org/licenses/

@iwasakims
Copy link
Member

Why ERROR and WARN messages are not shown at all after the patch is applied? Those messages seem not to be related to multiple slf4j bindings.

@LiJie20190102
Copy link
Contributor Author

Why ERROR and WARN messages are not shown at all after the patch is applied? Those messages seem not to be related to multiple slf4j bindings.

After installing hbase, there may be some log dependencies that conflict with hive's log dependencies:
image

image

@iwasakims
Copy link
Member

I meant the log messages related to HiveConf and FileUtils should be emitted even after fixing duplicate slf4j bindings.

@LiJie20190102
Copy link
Contributor Author

I meant the log messages related to HiveConf and FileUtils should be emitted even after fixing duplicate slf4j bindings.

I think fixing the duplicate SLf4j binding does not affect the previous log output. The following picture shows the scenario when I do not install Hbase:
image

@iwasakims
Copy link
Member

Hmm. I can not reproduce the issue by just installing both Hive and HBase. Is the issue already fixed in master (and maybe branch-3.2) by upgrading HBase? There is no slf4j-reload4j-1.7.33.jar in the output of hbase mapredcp.

root@0f9131e5418b:/# hbase mapredcp | sed -z 's/:/\n/g'
/usr/lib/hbase/lib/shaded-clients/hbase-shaded-mapreduce-2.4.17.jar
/usr/lib/hbase/lib/client-facing-thirdparty/audience-annotations-0.13.0.jar
/usr/lib/hbase/lib/client-facing-thirdparty/commons-logging-1.2.jar
/usr/lib/hbase/lib/client-facing-thirdparty/htrace-core4-4.2.0-incubating.jar
/usr/lib/hbase/lib/client-facing-thirdparty/jcl-over-slf4j-1.7.33.jar
/usr/lib/hbase/lib/client-facing-thirdparty/jul-to-slf4j-1.7.33.jar
/usr/lib/hbase/lib/client-facing-thirdparty/reload4j-1.2.22.jar
/usr/lib/hbase/lib/client-facing-thirdparty/slf4j-api-1.7.33.jar

# depends on HBASE-8438 (hbase-0.94.14+, hbase-0.96.1+) for `hbase mapredcp` command
for x in $($HBASE_BIN mapredcp 2>&2 | tr ':' '\n') ; do
- if [[ $x == *zookeeper* || $x == *protobuf-java* || $x == *guava* ]] ; then
+ if [[ $x == *zookeeper* || $x == *protobuf-java* || $x == *guava* || $x == *log* || $x == *slf4j* ]] ; then
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

*log* looks prone to future bug by matching unintentional files. What is the target jar here?
Same to *slf4j*. Are you really sure that all jars matching the pattern should be excluded?

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
2 participants