Project

General

Profile

Actions
Copy link

Activity #3749

open

Module #3693: Big Data Analytics Platform and Tools for Epidemic Forecasting and Mitigation

Module #3696: Center for Epidemic Forecsting(EPIFORM)

Exploration of Data Pipeline

Added by Prasidh J S about 1 year ago.

Status:
Resolved
Priority:
Normal
Assignee:
Prasidh J S
Start date:
06/11/2025
Due date:
% Done:

100%

Estimated time:
Planned Due Date:

Description

Explored the full end-to-end data pipeline:

Data ingestion using Apache NiFi
       Storage in HDFS / Delta Lake
       Processing with Apache Spark
       Visualization via custom web dashboards

Identified hardware requirements for each stage:

Ingestion (NiFi): 4-core CPU, 8–16 GB RAM, 1 TB disk, 1 Gbps NIC
       Storage (HDFS): High-capacity RAID HDDs on DataNodes
       Processing (Spark): Multi-core CPU and high RAM on Worker nodes
       Serving (Web/Dashboard): Light-weight server with 2-core CPU, 8 GB RAM
       Ensured GPU server is available for future AI/ML pipeline extension.

No data to display

Actions
Copy link

Also available in: Atom PDF