Big Data Analytics Platform and Tools for Epidemic Forecasting and Mitigation

Please update the draft paper to align with the title:
A Scalable Data Ingestion Pipeline for Big Health Data Analytics: Performance Analysis of Apache NiFi and Delta Lake Using Medallion Architecture.

Abstract: Modify to reflect above title and focusing on scalability, performance evaluation, and the health data analytics context.
Related Works:
Include research papers highlighting the importance of efficient data ingestion from multiple health data sources in health analytics.
Add studies or implementations involving Apache NiFi, Delta Lake, and Hadoop clusters to position our work within the current research landscape.
Introduction: Modify to connect the problem of unstructured public health data ingestion with the proposed NiFi–Delta Lake–Medallion architecture solution.
Architecture Diagram:
Prepare a architecture diagram with the following components:
• Web Interface: For uploading daily health reports in PDF format.
• Directory watch integrated with NiFi pipeline
• NiFi Pipeline: Processors for document ingestion, PDF parsing, conversion to CSV/JSON formats, and storage into Delta Lake (in Hadoop HDFS).
• Retrieval API Interface: For querying processed data from Delta Lake.
• Visualization Layer: For analytical dashboards and trend visualization.
3. Performance Parameters
The following performance metrics can be considered and included for evaluation and results discussion. Explore on other relevant parameters also.
• File Parsing Time: Time taken to process each daily PDF.
• Row Extraction Accuracy: Comparison between OCR/text extraction results and manually verified ground truth.
• Storage Efficiency: Comparative storage size and efficiency across raw PDF, CSV, JSON, and Delta Lake (Parquet) formats.
• Query Latency: Time required to retrieve district/disease trends from Delta Lake.