Project

General

Profile

Actions
Copy link

Module #3696

open

Module #3693: Big Data Analytics Platform and Tools for Epidemic Forecasting and Mitigation

Center for Epidemic Forecsting(EPIFORM)

Added by Binu Janardhanan about 1 year ago. Updated 10 months ago.

Status:
New
Priority:
High
Assignee:
Sudalaimani C
Start date:
01/10/2025
Due date:
10/29/2025 (about 9 months late)
% Done:

72%

Estimated time:
(Total: 0:00 h)
Planned Due Date:

Description

Center for Epidemic Forecsting(EPIFORM)

Subtasks 61 (45 open — 16 closed)

Activity #3701: Design NiFi Framework for Data ingestion NewThara S Pillai06/09/202510/16/2025

Actions
Activity #3702: Explore the possibilities to develop an automated data Ingestion Pipe line integrating with HDFS layer NewSherin Sebastian06/09/2025

Actions
Activity #4161: Preparation of a Conference Paper on Data Ingestion Pipeline with Apache NiFiNewSherin Sebastian09/22/202510/16/2025

Actions
Activity #3703: Automate the IDSP group data extraction and transforming to JSON file from a NiFi paltform to HDFS NewSherin Sebastian06/09/2025

Actions
Module #3704: Technology ExplorationNew03/21/202510/29/2025

Actions
Activity #3705: Explore Delta LakeClosedSreeshna T03/21/2025

Actions
Activity #3706: Setup tool to query delta lake objectsClosedSreeshna T03/21/2025

Actions
Activity #3707: Create a DataFlow pipelineClosedSreeshna T03/21/2025

Actions
Activity #3708: Explore - Store data onto Delta LakeClosedSreeshna T03/27/2025

Actions
Activity #3709: Explore Delta Lake medallion-architectureClosedSreeshna T03/27/2025

Actions
Activity #3710: Explore - Data ingestion from public APIsClosedSherin Sebastian03/27/2025

Actions
Activity #3711: Explore - Delta Lake data read from multiple files with filteringNewSreeshna T03/27/2025

Actions
Activity #3712: Feasibility of integration of nifi data ingestion and visualizationNewSherin Sebastian03/27/2025

Actions
Activity #3713: Design approaches - Delta Lake -- medallion-architectureNewSherin Sebastian03/27/2025

Actions
Activity #3714: Create NiFi data flow files - read from static filesClosedSherin Sebastian03/28/2025

Actions
Activity #3715: Create a custom processor in NiFiClosedSreeshna T05/30/2025

Actions
Activity #3716: Use NiFi to ingest CSV files from SFTPResolvedSreeshna T05/30/2025

Actions
Activity #3730: Explore back pressure and demo the purpose and applicationClosedSreeshna T06/11/2025

Actions
Activity #3731: NiFi server architectureNewSherin Sebastian06/11/2025

Actions
Activity #3732: Explore provenance and demo purposeNewSherin Sebastian06/11/2025

Actions
Activity #3735: Demonstrate Delta lake - scheme enforcement/evolutionClosedSreeshna T06/11/2025

Actions
Activity #3745: Demo Provenance Events to track data flowNewSherin Sebastian06/11/2025

Actions
Activity #3746: Optimization approachesNewSreeshna T06/11/2025

Actions
Activity #3747: Modify pipeline to submit delta lake write as a spark jobClosedSherin Sebastian06/11/2025

Actions
Activity #3750: Experiment Merge otpion for data insert/update/delete from delta lake ClosedSreeshna T06/11/2025

Actions
Activity #3757: Enable and test Python logging in NiFi ExecuteScript processor with daily log file rotation.ClosedSherin Sebastian06/13/2025

Actions
Activity #3776: Prepare documentation - Delta lake access thorugh DBeaverResolvedSreeshna T06/23/2025

Actions
Sub module #3873: Data Mapping and Harmonization for Delta Lake ConsolidationNewSherin Sebastian07/07/2025

Actions
Activity #3874: Simulate a csv file for temperature data with district names.NewSherin Sebastian07/07/2025

Actions
Activity #3876: Document schema of IDSP fileNewSherin Sebastian07/07/2025

Actions
Activity #3877: Build a Canonical Dimension TableNewSherin Sebastian07/07/2025

Actions
Activity #3878: Ingest Data into Bronze LayerNewSherin Sebastian07/07/2025

Actions
Activity #3879: Mapping and Transformation in Silver LayerNewSherin Sebastian07/07/2025

Actions
Activity #3880: Join and Aggregate in Gold LayerNewSherin Sebastian07/07/2025

Actions
Activity #3881: Query and ValidationNewSherin Sebastian07/07/2025

Actions
Sub module #4065: Paper publishing New08/04/202510/29/2025

Actions
Activity #4066: Literature survey to compareNewSherin Sebastian08/04/2025

Actions
Activity #4342: Research Paper – Data Ingestion using Apache NiFiNewSherin Sebastian10/21/202510/29/2025

Actions
Module #3717: DevelopmentNew06/10/2025

Actions
Module #3718: Technology Stack SetupNew03/28/2025

Actions
Activity #3719: Setup Spark managed Delta LakeClosedSherin Sebastian03/28/2025

Actions
Activity #3720: Setup Spark managed Delta LakeClosedSreeshna T03/28/2025

Actions
Activity #3721: Experimental Setup Data Repository - NiFi - Delta LakeNewSherin Sebastian06/10/2025

Actions
Activity #3722: Setup timer-driven NiFi processorsClosedSherin Sebastian06/10/2025

Actions
Activity #3723: Setup event-driven NiFi processorsNewSherin Sebastian06/10/2025

Actions
Activity #3733: Enable HTTPSNewSherin Sebastian06/11/2025

Actions
Activity #3734: Set up users, roles, and policiesNewSherin Sebastian06/11/2025

Actions
Activity #3736: Big Data Technology ExplorationResolvedPrasidh J S01/10/2025

Actions
Activity #3737: Hadoop Architecture StudyResolvedPrasidh J S01/28/2025

Actions
Activity #3738: Hands-On Hadoop Configuration, Monitoring and ManagementResolvedPrasidh J S02/11/2025

Actions
Activity #3739: Introduction to Disease ModellingResolvedPrasidh J S03/03/2025

Actions
Activity #3740: Compartmental Models ExploredResolvedPrasidh J S03/06/2025

Actions
Activity #3741: Model Implementation and VisualizationResolvedPrasidh J S03/18/2025

Actions
Activity #3742: User Interface Design (PyQt6)ResolvedPrasidh J S04/01/2025

Actions
Activity #3743: Spark Technology ExplorationResolvedPrasidh J S05/01/2025

Actions
Activity #3744: Spark Installation and ConfigurationResolvedPrasidh J S06/02/2025

Actions
Activity #3748: Exploration and Finalization of Efficient Hardware Requirements for Big Data TechnologiesResolvedPrasidh J S04/16/2025

Actions
Activity #3749: Exploration of Data PipelineResolvedPrasidh J S06/11/2025

Actions
Activity #3758: Integration and Usage of Delta Lake with Hadoop and SparkResolvedPrasidh J S06/12/2025

Actions
Module #3883: DocumentationNew07/07/2025

Actions
Module #3889: Prepare a design document of IDSP data ingestionNewSherin Sebastian07/07/2025

Actions
Actions
Copy link
 #1

Updated by Binu Janardhanan about 1 year ago

  • Subtask #3701 added
Actions
Copy link
 #2

Updated by Binu Janardhanan about 1 year ago

  • Subtask #3703 added
Actions
Copy link
 #3

Updated by Thara S Pillai about 1 year ago

  • Subtask #3704 added
Actions
Copy link
 #4

Updated by Thara S Pillai about 1 year ago

  • Subtask #3717 added
Actions
Copy link
 #5

Updated by Thara S Pillai about 1 year ago

  • Subtask #3718 added
Actions
Copy link
 #6

Updated by Prasidh J S about 1 year ago

  • Subtask #3736 added
Actions
Copy link
 #7

Updated by Prasidh J S about 1 year ago

  • Subtask #3737 added
Actions
Copy link
 #8

Updated by Prasidh J S about 1 year ago

  • Subtask #3738 added
Actions
Copy link
 #9

Updated by Prasidh J S about 1 year ago

  • Subtask #3739 added
Actions
Copy link
 #10

Updated by Prasidh J S about 1 year ago

  • Subtask #3740 added
Actions
Copy link
 #11

Updated by Prasidh J S about 1 year ago

  • Subtask #3741 added
Actions
Copy link
 #12

Updated by Prasidh J S about 1 year ago

  • Subtask #3742 added
Actions
Copy link
 #13

Updated by Prasidh J S about 1 year ago

  • Subtask #3743 added
Actions
Copy link
 #14

Updated by Prasidh J S about 1 year ago

  • Subtask #3744 added
Actions
Copy link
 #15

Updated by Prasidh J S about 1 year ago

  • Subtask #3748 added
Actions
Copy link
 #16

Updated by Prasidh J S about 1 year ago

  • Subtask #3749 added
Actions
Copy link
 #17

Updated by Prasidh J S about 1 year ago

  • Subtask #3758 added
Actions
Copy link
 #18

Updated by Thara S Pillai about 1 year ago

  • Subtask #3883 added
Actions
Copy link

Also available in: Atom PDF