This is an old revision of the document!
Table of Contents
Using Apache Spark in the CS Dept Lab
Spark 3.1.1 is installed in the CS Dept. Lab in the folder: /usr/lib/spark. You'll find the Spark binaries and scripts at /usr/lib/spark/bin. Spark is currently configured to use six worker nodes.
If you are a member of the 'spark_users' group, you will be able to access Spark to submit new jobs. Jobs MUST be submitted to the head node (spark.cs.ndsu.edu).
Status pages
The Spark web interface is here Spark Master.
Note: Status and information pages are only available from NDSU CS Department office and lab computers.
Firewall info
The head node is firewalled such that Spark status links are ONLY accessible from ND state networks. The head node, Zoidberg, can be accessed via SSH from anywhere, so job submission is possible from outside of these networks, but it is currently not possible to view the status pages.