Big Data Analytics Platform and Tools for Epidemic Forecasting and Mitigation

Download and use DDD design document template from QA portal.
Document for the Module: IDSP data ingestion
including the following:

• Define the document schema of IDSP PDF file
• Document the file source location (e.g., SFTP, cloud bucket, manual upload)
• Describe the file naming conventions and frequency (e.g., daily, weekly)
• Process of Apache NiFi pipeline. conversions from PDF to CSV
• Pre-processors, validation checks included
• Record Bronze layer table name and structure (raw ingestion, no transformation)
• Describe Silver layer transformations (e.g., district name normalization, mapping to district_id)
• Specify pre-processors used in silver (for )
• Gold layer outputs (e.g., cleaned and enriched case data by district and date)
• Document partitioning and indexing strategies (e.g., partition by date)
• Capture storage paths and format (e.g., Delta tables stored in <folder format>)
• List data quality checks and logging implemented in the pipeline
• Include version control strategy for schema and mapping tables
• Note performance benchmarks or runtime characteristics; log times in each pipeline processor component
• Describe monitoring, alerts, and error handling mechanisms