Differences

This shows you the differences between two versions of the page.

--- deptlab:spark [2021/11/15 11:20] – created localadmin
+++ deptlab:spark [2024/09/05 11:04] (current) – [Firewall info] localadmin
@@ Line 1: / Line 1: @@
-Using Apache Spark in the CS Dept Lab
+====== Using Apache Spark in the CS Dept Lab ======
-Spark 3.1.1 is installed in the CS Dept. Lab in the folder: /usr/lib/spark. You'll find the Spark binaries and scripts at /usr/lib/spark/bin.
-If you are a member of the 'spark_users' group, you will be able to access Spark to submit new jobs.
+Spark 3.5.0 is installed in the CS Dept. Lab in the folder: /usr/lib/spark. You'll find the Spark binaries and scripts at /usr/lib/spark/bin.
+Spark is currently configured to use six worker nodes.
-Jobs MUST be submitted to the head node (spark.cs.ndsu.edu).
+If you are a member of the 'spark_users' group, you will be able to access Spark to submit new jobs. Jobs MUST be submitted to the head node (spark.cs.ndsu.edu).
-The Spark web interface is here [[http://spark.cs.ndsu.edu:8080/|Spark Master]] from NDSU CS Department computers.
+Spark documentation for 3.5.0 can be found here https://spark.apache.org/docs/3.5.0/
+===== Status pages =====
+The Spark web interface is here [[http://spark.cs.ndsu.edu:8080/|Spark Master]].
+**Note:** Status and information pages are only available from NDSU CS Department office and lab computers.
+===== Firewall info =====
+The head node is firewalled such that Spark status links are ONLY accessible from /* ND state */ CS Dept networks. The Spark head node can be accessed via SSH from anywhere (**not being blocked from your home country or ISP**), so job submission is possible from outside of these networks, but it is currently not possible to view the status pages for security purposes.
+===== Spark Components =====
+The current version of Spark installed does NOT support HDSF