NDSU CS Department Wiki

This page is read only. You can view the source, but not change it. Ask your administrator if you think this is wrong.

====== Using Apache Spark in the CS Dept Lab ======

Spark 3.5.0 is installed in the CS Dept. Lab in the folder: /usr/lib/spark. You'll find the Spark binaries and scripts at /usr/lib/spark/bin.
Spark is currently configured to use six worker nodes.

If you are a member of the 'spark_users' group, you will be able to access Spark to submit new jobs. Jobs MUST be submitted to the head node (spark.cs.ndsu.edu).

Spark documentation for 3.5.0 can be found here https://spark.apache.org/docs/3.5.0/ 


===== Status pages =====

The Spark web interface is here [[http://spark.cs.ndsu.edu:8080/|Spark Master]].

**Note:** Status and information pages are only available from NDSU CS Department office and lab computers.

===== Firewall info =====

The head node is firewalled such that Spark status links are ONLY accessible from /* ND state */ CS Dept networks. The Spark head node can be accessed via SSH from anywhere (**not being blocked from your home country or ISP**), so job submission is possible from outside of these networks, but it is currently not possible to view the status pages for security purposes. 

===== Spark Components =====

The current version of Spark installed does NOT support HDSF