User Tools

Site Tools


deptlab:spark

Differences

This shows you the differences between two versions of the page.

Link to this comparison view

Both sides previous revisionPrevious revision
Next revision
Previous revision
deptlab:spark [2021/11/15 11:27] localadmindeptlab:spark [2024/09/05 11:04] (current) – [Firewall info] localadmin
Line 1: Line 1:
 ====== Using Apache Spark in the CS Dept Lab ====== ====== Using Apache Spark in the CS Dept Lab ======
  
-Spark 3.1.is installed in the CS Dept. Lab in the folder: /usr/lib/spark. You'll find the Spark binaries and scripts at /usr/lib/spark/bin.+Spark 3.5.is installed in the CS Dept. Lab in the folder: /usr/lib/spark. You'll find the Spark binaries and scripts at /usr/lib/spark/bin.
 Spark is currently configured to use six worker nodes. Spark is currently configured to use six worker nodes.
  
 If you are a member of the 'spark_users' group, you will be able to access Spark to submit new jobs. Jobs MUST be submitted to the head node (spark.cs.ndsu.edu). If you are a member of the 'spark_users' group, you will be able to access Spark to submit new jobs. Jobs MUST be submitted to the head node (spark.cs.ndsu.edu).
  
-The Spark web interface is here [[http://spark.cs.ndsu.edu:8080/|Spark Master]] from NDSU CS Department computers.+Spark documentation for 3.5.0 can be found here https://spark.apache.org/docs/3.5.0/ 
  
  
 +===== Status pages =====
 +
 +The Spark web interface is here [[http://spark.cs.ndsu.edu:8080/|Spark Master]].
 +
 +**Note:** Status and information pages are only available from NDSU CS Department office and lab computers.
 +
 +===== Firewall info =====
 +
 +The head node is firewalled such that Spark status links are ONLY accessible from /* ND state */ CS Dept networks. The Spark head node can be accessed via SSH from anywhere (**not being blocked from your home country or ISP**), so job submission is possible from outside of these networks, but it is currently not possible to view the status pages for security purposes. 
 +
 +===== Spark Components =====
 +
 +The current version of Spark installed does NOT support HDSF
deptlab/spark.1636997264.txt.gz · Last modified: 2021/11/15 11:27 by localadmin