[Azkaban] -- Enable logging of underlying exception when fetching NameNodeToken

[himanshukandwal]

Context

We are seeing a quite high amount of errors during the fetching of name-node tokens and the exception does not present much information to perform RC analysis.

13-04-2022 10:01:16 PDT copy-latest INFO - Token service: thrift://ltx1-holdemhcat01.grid.linkedin.com:7552,thrift://ltx1-holdemhcat02.grid.linkedin.com:7552
13-04-2022 10:01:16 PDT copy-latest INFO - Hive metastore token(s) prefetched
13-04-2022 10:01:16 PDT copy-latest INFO - Pre-fetching JH token from job history server
13-04-2022 10:01:16 PDT copy-latest INFO - Connecting to HistoryServer at: ltx1-holdemjh01.grid.linkedin.com:10020
13-04-2022 10:01:16 PDT copy-latest INFO - JH token from job history server pre-fetched, token Kind: MR_DELEGATION_TOKEN, token service: ltx1-holdemjh01.grid.linkedin.com:10020
13-04-2022 10:01:16 PDT copy-latest INFO - Here is the props for obtain.namenode.token: true
13-04-2022 10:01:17 PDT copy-latest ERROR - Job run failed!
azkaban.security.commons.HadoopSecurityManagerException: Failed to get hadoop tokens! nullazkaban.security.commons.HadoopSecurityManagerException: Failed to fetch DFS token for dcoffline/az_ltx1-holdemaz05.grid.linkedin.com_61914650@GRID.LINKEDIN.COM
    at azkaban.security.AbstractHadoopSecurityManager.doPrefetch(AbstractHadoopSecurityManager.java:418)
    at azkaban.security.AbstractHadoopSecurityManager.prefetchToken(AbstractHadoopSecurityManager.java:381)
    at azkaban.jobtype.HadoopJobUtils.getHadoopTokens(HadoopJobUtils.java:165)
    at azkaban.jobtype.HadoopProxy.setupPropsForProxy(HadoopProxy.java:99)
    at azkaban.jobtype.HadoopShell.run(HadoopShell.java:49)
    at azkaban.execapp.JobRunner.runJob(JobRunner.java:991)
    at azkaban.execapp.JobRunner.doRun(JobRunner.java:708)
    at azkaban.execapp.JobRunner.run(JobRunner.java:658)
    at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
    at java.util.concurrent.FutureTask.run(FutureTask.java:266)
    at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
    at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
    at java.lang.Thread.run(Thread.java:748)
Caused by: java.lang.reflect.UndeclaredThrowableException
    at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1911)
    at azkaban.security.HadoopSecurityManager_H_2_0.fetchAllHadoopTokens(HadoopSecurityManager_H_2_0.java:234)
    at azkaban.security.AbstractHadoopSecurityManager.doPrefetch(AbstractHadoopSecurityManager.java:404)
    ... 12 more

Hence, in this RB we are throwing the original exception as part of the stacktrace.

[codecov]

Codecov Report

Merging #3076 (acfed63) into master (38fe7ba) will increase coverage by 0.39%.
The diff coverage is 0.00%.

@@             Coverage Diff              @@
##             master    #3076      +/-   ##
============================================
+ Coverage     41.58%   41.97%   +0.39%     
- Complexity     4658     4781     +123     
============================================
  Files           596      596              
  Lines         40094    40889     +795     
  Branches       4662     4838     +176     
============================================
+ Hits          16672    17164     +492     
- Misses        22038    22329     +291     
- Partials       1384     1396      +12     

Impacted Files	Coverage Δ
.../azkaban/security/HadoopSecurityManager_H_2_0.java	0.00% <0.00%> (ø)
az-core/src/main/java/azkaban/Constants.java	12.50% <0.00%> (-8.93%)	⬇️
...ava/azkaban/executor/ExecutionControllerUtils.java	37.39% <0.00%> (-1.00%)	⬇️
...rc/main/java/azkaban/utils/ExecuteAsUserUtils.java	0.00% <0.00%> (ø)
...in/src/main/java/azkaban/jobtype/HadoopPigJob.java	0.00% <0.00%> (ø)
...azkaban/executor/container/ContainerImplUtils.java	100.00% <0.00%> (ø)
...rver/src/main/java/azkaban/execapp/FlowRunner.java	71.82% <0.00%> (+0.09%)	⬆️
...c/main/java/azkaban/execapp/FlowRunnerManager.java	15.03% <0.00%> (+0.29%)	⬆️
...an/container/models/AzKubernetesV1SpecBuilder.java	90.98% <0.00%> (+0.32%)	⬆️
...src/main/java/azkaban/executor/ExecutableFlow.java	89.71% <0.00%> (+0.71%)	⬆️
... and 5 more

---

Continue to review full report at Codecov.

Legend - Click here to learn more
Δ = absolute <relative> (impact), ø = not affected, ? = missing data
Powered by Codecov. Last update 38fe7ba...acfed63. Read the comment docs.

[coveralls]

Pull Request Test Coverage Report for Build 6840

• 0 of 1 (0.0%) changed or added relevant line in 1 file are covered.
• 407 unchanged lines in 15 files lost coverage.
• Overall coverage increased (+0.04%) to 45.293%

---

Changes Missing Coverage	Covered Lines	Changed/Added Lines	%
azkaban-hadoop-security-plugin/src/main/java/azkaban/security/HadoopSecurityManager_H_2_0.java	0	1	0.0%

Files with Coverage Reduction	New Missed Lines	%
azkaban-common/src/main/java/azkaban/container/models/AzKubernetesV1SpecBuilder.java	1	90.65%
azkaban-common/src/main/java/azkaban/executor/ExecutableFlowBase.java	1	87.71%
azkaban-common/src/main/java/azkaban/user/UserUtils.java	1	74.19%
azkaban-exec-server/src/main/java/azkaban/execapp/event/FlowWatcher.java	1	93.75%
azkaban-common/src/main/java/azkaban/container/models/ImagePullPolicy.java	6	0%
azkaban-common/src/main/java/azkaban/executor/ExecutableFlow.java	8	93.72%
azkaban-common/src/main/java/azkaban/utils/ExecuteAsUserUtils.java	8	0%
az-core/src/main/java/azkaban/Constants.java	10	21.43%
azkaban-common/src/main/java/azkaban/executor/ExecutionControllerUtils.java	23	48.83%
azkaban-common/src/main/java/azkaban/executor/container/watch/KubernetesWatch.java	27	71.79%

Totals
Change from base Build 6790:	0.04%
Covered Lines:	18077
Relevant Lines:	39911

---

💛 - Coveralls

[svnarumugam]

Looks good to me. Thank you so much for making this change!

[amielkelinkedin]

@himanshukandwal, would you like us to merge this for you?