Can sparkxgb work through Livy connection?

[mzorko]

Hello, so the local mode is working like expected:

library(sparkxgb)
library(sparklyr)

# Extension is already called
sparklyr::registered_extensions()
# [1] sparkxgb

# Connection works
sc <- sparklyr::spark_connect("local", version = "2.3.3")

Is it possible to use sparkxgb through Livy connection and if yes, what do I have set up?
Do I have to copy .jar file to all the nodes or just to set up livy_config paths?

(.jar file is currently in the tmp folder: "/tmp/sparkxgb-2.3-2.11.jar")

library(sparkxgb)
library(sparklyr)

my_config <- sparklyr::spark_config()

# Was not sure where to add this .jar path
my_config[["sparklyr.jars.default"]] <- "/tmp/sparkxgb-2.3-2.11.jar"
my_config[["spark.jars.packages"]] <- c(
      "com.esotericsoftware.reflectasm:reflectasm:1.08",  "ml.dmlc:xgboost4j-spark:0.81")

my_config_livy <- sparklyr::livy_config(
      config = my_config , driver_memory = "7g", driver_cores = 2, negotiate = FALSE, 
      executor_memory = "7g", executor_cores = 2, num_executors = 40,

      # Was not sure where to add this .jar path
      #jars = "/home/rstudio/R/x86_64-pc-linux-gnu-library/3.6/sparkxgb/java/sparkxgb-2.3-2.11.jar",
      name = session_name, heartbeat_timeout = 0)

# Livy timeout connection error
sc <- sparklyr::spark_connect("ec2-...com:8998", version = "2.3.3", method = "livy", config = my_config_livy )

This was one of several attempts, but the error was always Livy 60 second timeout.