Scheduler Component Deployment

Version Requirements

Component	Version Requirement	Remarks
Spark	spark-3.4.4-bin-hadoop3+	Recommended download: 3.4.4

1. Prerequisites

1.1 Create directory

sudo mkdir -p /opt/soft
# Grant permissions
sudo chmod -R 777 /opt/soft

1.2 Upload

Upload spark-3.4.4-bin-hadoop3.tgz to /opt/soft.

1.3 Extract and rename

# Spark 
cd /opt/soft
sudo tar -zxvf spark-3.4.4-bin-hadoop3.tgz
sudo mv spark-3.4.4-bin-hadoop3 spark

2. Spark Cluster Deployment

2.1 Add environment variables

# Add environment variables
nano ~/.bashrc
# Add your environment variable setting at the end of the file: export SPARK_MASTER_HOST=192.168.3.38  
# 192.168.3.38 is the private IP of the current server

# Make environment variables take effect
source ~/.bashrc

2.2 Deploy master

cd /opt/soft/spark
# Start master
./sbin/start-master.sh
# If port 8080 is occupied, add the following content to conf/spark-env.sh
export SPARK_MASTER_WEBUI_PORT=8077

2.3 Deploy slave

A slave can be deployed on multiple servers. To deploy multiple servers, upload the Spark archive to each server, extract it, and start the slave.

cd /opt/soft/spark
# Start slave. 192.168.3.38 is the private IP of the current server.
./sbin/start-slave.sh spark://192.168.3.38:7077