Skip to content

Experiment to get LIVY+ Apache Spark working on DC/OS + MESOS + Docker

License

Notifications You must be signed in to change notification settings

mganta/docker-livy

Repository files navigation

docker-livy

A Docker image for Livy, the REST Spark Server.

Running

which will expose the port 8998 on the Docker host for this image.

Details

Have a look at the official docs to see how to use the Livy REST API.

For exposing the service URL to http://localhost/service/livy-spark2/, you will need the labels as shown in the json below.

Also, breaking docker rules and running spark-mesos-dispatcher in this too :) Had to run the mesos dispatcher as spark would not download jars via http/s. This is due to https://issues.apache.org/jira/browse/SPARK-10643

Here is the DC/OS service JSON

{ "volumes": null, "id": "/livy-spark2", "cmd": null, "args": null, "user": null, "env": null, "instances": 1, "cpus": 1, "mem": 1024, "disk": 0, "gpus": 0, "executor": null, "constraints": null, "fetch": null, "storeUrls": null, "backoffSeconds": 1, "backoffFactor": 1.15, "maxLaunchDelaySeconds": 3600, "container": { "docker": { "image": "mganta/livy2-docker", "forcePullImage": true, "privileged": false, "portMappings": [ { "containerPort": 8998, "protocol": "tcp", "name": "livy", "servicePort": 10007, "hostPort": 0 }, { "containerPort": 4040, "protocol": "tcp", "name": "sparkui", "servicePort": 10008, "hostPort": 0 } ], "network": "USER" } }, "healthChecks": [ { "protocol": "HTTP", "gracePeriodSeconds": 300, "intervalSeconds": 60, "timeoutSeconds": 123, "maxConsecutiveFailures": 3, "ignoreHttp1xx": false } ], "readinessChecks": null, "dependencies": null, "upgradeStrategy": { "minimumHealthCapacity": 1, "maximumOverCapacity": 1 }, "labels": { "DCOS_SERVICE_NAME": "livy-spark2", "DCOS_SERVICE_SCHEME": "http", "DCOS_SERVICE_PORT_INDEX": "0" }, "acceptedResourceRoles": null, "ipAddress": { "networkName": "dcos" }, "residency": null, "secrets": null, "taskKillGracePeriodSeconds": null }

curl for livy job submit

curl -v -H "Content-Type: application/json" -X POST -d '{ "file": "https://.blob.core.windows.net/myjars/spark-examples_2.11-2.0.1.jar", "className":"org.apache.spark.examples.SparkPi", "args": ["100"] }' 'http://localhost/service/livy-spark2/batches'

About

Experiment to get LIVY+ Apache Spark working on DC/OS + MESOS + Docker

Topics

Resources

License

Stars

Watchers

Forks

Releases

No releases published

Packages

No packages published

Languages