Runaway Databricks Spark .Net Job (Azure Blob Storage vs Data Lake Storage?)

[timothyl1985]

Hi all,

Please advise if this isn't the appropriate forum, however I've recently observed my scheduled Spark .Net job failing to terminate despite completing its work.

After lengthy investigation and trial and error, I've established one possible cause being writing to Azure Data Lake Gen2 as opposed to Azure Blob Storage (is it driver incompatibility, ABFS vs WASB?). With writing to Azure Blog Storage enabling to job to terminate successfully on completion whereas the former, renders a runaway job requiring me to intervene and manually terminate.

I've tinkered with various access models ie. mounting vs direct acess through spark job instantiation and authenticating with access key and service principal all to no avail.

Any guidance on this issue or tools to enable deeper interrogation of logging would be appreciated!

Thanks

Tim.

Log4j output:

1. This is the tail of a successful termination:
   21/12/29 14:47:11 INFO OutputCommitCoordinator$OutputCommitCoordinatorEndpoint: OutputCommitCoordinator stopped!
   21/12/29 14:47:11 INFO SparkContext: Successfully stopped SparkContext
   21/12/29 14:47:11 INFO DotnetRunner: Closing DotnetBackend
   21/12/29 14:47:11 INFO DotnetBackend: Callback server has already been shutdown.
   21/12/29 14:47:11 INFO DotnetRunner: .NET application exited successfully
   21/12/29 14:47:14 INFO ShutdownHookManager: Shutdown hook called
   21/12/29 14:47:14 INFO ShutdownHookManager: Deleting directory /local_disk0/spark-d9202209-a948-4879-89a8-24321193a536
   21/12/29 14:47:14 INFO ShutdownHookManager: Deleting directory /local_disk0/tmp/spark-802f4d0f-c884-431a-8439-7bd5f3e80e11
   21/12/29 14:47:14 INFO ShutdownHookManager: Deleting directory /local_disk0/tmp/spark-527343ca-4d82-48ca-a266-3f6c6dad2290
   21/12/29 14:47:14 INFO ShutdownHookManager: Deleting directory /local_disk0/tmp/spark-b5110e4c-0f9d-454f-ad1a-5c3b8bd2d035
   21/12/29 14:47:14 INFO NativeAzureFileSystem: WASB Filesystem wasbs://root@dbstoragemk42k2y3l3s6k.blob.core.windows.net is closed with isClosed = false

2. In contrast, tail of a non-termination (observe absence of "ShutdownHookManager"):
   21/12/29 01:23:35 INFO MapOutputTrackerMasterEndpoint: MapOutputTrackerMasterEndpoint stopped!
   21/12/29 01:23:35 INFO MemoryStore: MemoryStore cleared
   21/12/29 01:23:35 INFO BlockManager: BlockManager stopped
   21/12/29 01:23:35 INFO BlockManagerMaster: BlockManagerMaster stopped
   21/12/29 01:23:36 INFO OutputCommitCoordinator$OutputCommitCoordinatorEndpoint: OutputCommitCoordinator stopped!
   21/12/29 01:23:36 INFO SparkContext: Successfully stopped SparkContext
   21/12/29 01:23:36 INFO DotnetRunner: Closing DotnetBackend
   21/12/29 01:23:36 INFO DotnetBackend: Callback server has already been shutdown.
   21/12/29 01:23:36 INFO DotnetRunner: .NET application exited successfully

[imback82]

@timothyl1985 Can you try something very simple to write to ABFS and see if you still see the issue? For example:

SparkSession spark = SparkSession.Builder().GetOrCreate();
spark.Range(0, 5).Write().Parquet("your abfs path")
spark.Stop();

[timothyl1985]

Thanks for your suggestion @imback82, sadly no success even with that very simple write action. The parquet was duly written but again, the job didn't terminated.

[imback82]

Looks like you are running this in Databricks? Can you share your environment like runtime version, Spark version, etc.?

Can you also try to repro this with Scala?

val spark = SparkSession.builder.getOrCreate()
spark.range(0, 5).write.parquet("your abfs path")
spark.stop()