Project

General

Profile

Actions
Copy link

Module #3704

open

Module #3693: Big Data Analytics Platform and Tools for Epidemic Forecasting and Mitigation

Module #3696: Center for Epidemic Forecsting(EPIFORM)

Technology Exploration

Added by Thara S Pillai about 1 year ago. Updated 10 months ago.

Status:
New
Priority:
High
Assignee:
-
Start date:
03/21/2025
Due date:
10/29/2025 (about 9 months late)
% Done:

62%

Estimated time:
(Total: 0:00 h)
Planned Due Date:

Subtasks 33 (20 open — 13 closed)

Activity #3705: Explore Delta LakeClosedSreeshna T03/21/2025

Actions
Activity #3706: Setup tool to query delta lake objectsClosedSreeshna T03/21/2025

Actions
Activity #3707: Create a DataFlow pipelineClosedSreeshna T03/21/2025

Actions
Activity #3708: Explore - Store data onto Delta LakeClosedSreeshna T03/27/2025

Actions
Activity #3709: Explore Delta Lake medallion-architectureClosedSreeshna T03/27/2025

Actions
Activity #3710: Explore - Data ingestion from public APIsClosedSherin Sebastian03/27/2025

Actions
Activity #3711: Explore - Delta Lake data read from multiple files with filteringNewSreeshna T03/27/2025

Actions
Activity #3712: Feasibility of integration of nifi data ingestion and visualizationNewSherin Sebastian03/27/2025

Actions
Activity #3713: Design approaches - Delta Lake -- medallion-architectureNewSherin Sebastian03/27/2025

Actions
Activity #3714: Create NiFi data flow files - read from static filesClosedSherin Sebastian03/28/2025

Actions
Activity #3715: Create a custom processor in NiFiClosedSreeshna T05/30/2025

Actions
Activity #3716: Use NiFi to ingest CSV files from SFTPResolvedSreeshna T05/30/2025

Actions
Activity #3730: Explore back pressure and demo the purpose and applicationClosedSreeshna T06/11/2025

Actions
Activity #3731: NiFi server architectureNewSherin Sebastian06/11/2025

Actions
Activity #3732: Explore provenance and demo purposeNewSherin Sebastian06/11/2025

Actions
Activity #3735: Demonstrate Delta lake - scheme enforcement/evolutionClosedSreeshna T06/11/2025

Actions
Activity #3745: Demo Provenance Events to track data flowNewSherin Sebastian06/11/2025

Actions
Activity #3746: Optimization approachesNewSreeshna T06/11/2025

Actions
Activity #3747: Modify pipeline to submit delta lake write as a spark jobClosedSherin Sebastian06/11/2025

Actions
Activity #3750: Experiment Merge otpion for data insert/update/delete from delta lake ClosedSreeshna T06/11/2025

Actions
Activity #3757: Enable and test Python logging in NiFi ExecuteScript processor with daily log file rotation.ClosedSherin Sebastian06/13/2025

Actions
Activity #3776: Prepare documentation - Delta lake access thorugh DBeaverResolvedSreeshna T06/23/2025

Actions
Sub module #3873: Data Mapping and Harmonization for Delta Lake ConsolidationNewSherin Sebastian07/07/2025

Actions
Activity #3874: Simulate a csv file for temperature data with district names.NewSherin Sebastian07/07/2025

Actions
Activity #3876: Document schema of IDSP fileNewSherin Sebastian07/07/2025

Actions
Activity #3877: Build a Canonical Dimension TableNewSherin Sebastian07/07/2025

Actions
Activity #3878: Ingest Data into Bronze LayerNewSherin Sebastian07/07/2025

Actions
Activity #3879: Mapping and Transformation in Silver LayerNewSherin Sebastian07/07/2025

Actions
Activity #3880: Join and Aggregate in Gold LayerNewSherin Sebastian07/07/2025

Actions
Activity #3881: Query and ValidationNewSherin Sebastian07/07/2025

Actions
Sub module #4065: Paper publishing New08/04/202510/29/2025

Actions
Activity #4066: Literature survey to compareNewSherin Sebastian08/04/2025

Actions
Activity #4342: Research Paper – Data Ingestion using Apache NiFiNewSherin Sebastian10/21/202510/29/2025

Actions
Actions
Copy link

Also available in: Atom PDF