Activity #3744: Spark Installation and Configuration - Big Data Analytics Platform and Tools for Epidemic Forecasting and Mitigation - Health and Technology Group

Actions

Activity #3744

open

Module #3693: Big Data Analytics Platform and Tools for Epidemic Forecasting and Mitigation

Module #3696: Center for Epidemic Forecsting(EPIFORM)

Status:

Resolved

Priority:

Normal

Assignee:

Start date:

06/02/2025

Due date:

% Done:

100%

Estimated time:

Planned Due Date:

Description

Installed Spark 3.4.4 on the cluster.

Configured Spark in two modes:

Standalone Mode: Set up Spark master and worker nodes manually.
      YARN Mode: Integrated Spark with Hadoop YARN for resource management.

Configured spark-env.sh, spark-defaults.conf, and worker nodes list.

Verified Spark job execution in both client and cluster deploy modes.

Monitored job progress via Spark UI and YARN ResourceManager UI.

Actions

Also available in: Atom PDF