Project

General

Profile

Actions
Copy link

Module #3704

open

Module #3693: Big Data Analytics Platform and Tools for Epidemic Forecasting and Mitigation

Module #3696: Center for Epidemic Forecsting(EPIFORM)

Technology Exploration

Added by Thara S Pillai about 1 year ago. Updated 10 months ago.

Status:
New
Priority:
High
Assignee:
-
Start date:
03/21/2025
Due date:
10/29/2025 (about 9 months late)
% Done:

62%

Estimated time:
(Total: 0:00 h)
Planned Due Date:

Subtasks 33 (20 open — 13 closed)

Activity #3705: Explore Delta LakeClosedSreeshna T03/21/2025

Actions
Activity #3706: Setup tool to query delta lake objectsClosedSreeshna T03/21/2025

Actions
Activity #3707: Create a DataFlow pipelineClosedSreeshna T03/21/2025

Actions
Activity #3708: Explore - Store data onto Delta LakeClosedSreeshna T03/27/2025

Actions
Activity #3709: Explore Delta Lake medallion-architectureClosedSreeshna T03/27/2025

Actions
Activity #3710: Explore - Data ingestion from public APIsClosedSherin Sebastian03/27/2025

Actions
Activity #3711: Explore - Delta Lake data read from multiple files with filteringNewSreeshna T03/27/2025

Actions
Activity #3712: Feasibility of integration of nifi data ingestion and visualizationNewSherin Sebastian03/27/2025

Actions
Activity #3713: Design approaches - Delta Lake -- medallion-architectureNewSherin Sebastian03/27/2025

Actions
Activity #3714: Create NiFi data flow files - read from static filesClosedSherin Sebastian03/28/2025

Actions
Activity #3715: Create a custom processor in NiFiClosedSreeshna T05/30/2025

Actions
Activity #3716: Use NiFi to ingest CSV files from SFTPResolvedSreeshna T05/30/2025

Actions
Activity #3730: Explore back pressure and demo the purpose and applicationClosedSreeshna T06/11/2025

Actions
Activity #3731: NiFi server architectureNewSherin Sebastian06/11/2025

Actions
Activity #3732: Explore provenance and demo purposeNewSherin Sebastian06/11/2025

Actions
Activity #3735: Demonstrate Delta lake - scheme enforcement/evolutionClosedSreeshna T06/11/2025

Actions
Activity #3745: Demo Provenance Events to track data flowNewSherin Sebastian06/11/2025

Actions
Activity #3746: Optimization approachesNewSreeshna T06/11/2025

Actions
Activity #3747: Modify pipeline to submit delta lake write as a spark jobClosedSherin Sebastian06/11/2025

Actions
Activity #3750: Experiment Merge otpion for data insert/update/delete from delta lake ClosedSreeshna T06/11/2025

Actions
Activity #3757: Enable and test Python logging in NiFi ExecuteScript processor with daily log file rotation.ClosedSherin Sebastian06/13/2025

Actions
Activity #3776: Prepare documentation - Delta lake access thorugh DBeaverResolvedSreeshna T06/23/2025

Actions
Sub module #3873: Data Mapping and Harmonization for Delta Lake ConsolidationNewSherin Sebastian07/07/2025

Actions
Activity #3874: Simulate a csv file for temperature data with district names.NewSherin Sebastian07/07/2025

Actions
Activity #3876: Document schema of IDSP fileNewSherin Sebastian07/07/2025

Actions
Activity #3877: Build a Canonical Dimension TableNewSherin Sebastian07/07/2025

Actions
Activity #3878: Ingest Data into Bronze LayerNewSherin Sebastian07/07/2025

Actions
Activity #3879: Mapping and Transformation in Silver LayerNewSherin Sebastian07/07/2025

Actions
Activity #3880: Join and Aggregate in Gold LayerNewSherin Sebastian07/07/2025

Actions
Activity #3881: Query and ValidationNewSherin Sebastian07/07/2025

Actions
Sub module #4065: Paper publishing New08/04/202510/29/2025

Actions
Activity #4066: Literature survey to compareNewSherin Sebastian08/04/2025

Actions
Activity #4342: Research Paper – Data Ingestion using Apache NiFiNewSherin Sebastian10/21/202510/29/2025

Actions
Actions
Copy link
 #1

Updated by Thara S Pillai about 1 year ago

  • Subtask #3705 added
Actions
Copy link
 #2

Updated by Thara S Pillai about 1 year ago

  • Subtask #3706 added
Actions
Copy link
 #3

Updated by Thara S Pillai about 1 year ago

  • Subtask #3707 added
Actions
Copy link
 #4

Updated by Thara S Pillai about 1 year ago

  • Subtask #3708 added
Actions
Copy link
 #5

Updated by Thara S Pillai about 1 year ago

  • Subtask #3709 added
Actions
Copy link
 #6

Updated by Thara S Pillai about 1 year ago

  • Subtask #3710 added
Actions
Copy link
 #7

Updated by Thara S Pillai about 1 year ago

  • Subtask #3711 added
Actions
Copy link
 #8

Updated by Thara S Pillai about 1 year ago

  • Subtask #3712 added
Actions
Copy link
 #9

Updated by Thara S Pillai about 1 year ago

  • Subtask #3713 added
Actions
Copy link
 #10

Updated by Thara S Pillai about 1 year ago

  • Subtask #3714 added
Actions
Copy link
 #11

Updated by Thara S Pillai about 1 year ago

  • Subtask #3715 added
Actions
Copy link
 #12

Updated by Thara S Pillai about 1 year ago

  • Subtask #3716 added
Actions
Copy link
 #13

Updated by Thara S Pillai about 1 year ago

  • Subtask #3730 added
Actions
Copy link
 #14

Updated by Thara S Pillai about 1 year ago

  • Subtask #3731 added
Actions
Copy link
 #15

Updated by Thara S Pillai about 1 year ago

  • Subtask #3732 added
Actions
Copy link
 #16

Updated by Thara S Pillai about 1 year ago

  • Subtask #3735 added
Actions
Copy link
 #17

Updated by Thara S Pillai about 1 year ago

  • Subtask #3745 added
Actions
Copy link
 #18

Updated by Thara S Pillai about 1 year ago

  • Subtask #3746 added
Actions
Copy link
 #19

Updated by Thara S Pillai about 1 year ago

  • Subtask #3747 added
Actions
Copy link
 #20

Updated by Thara S Pillai about 1 year ago

  • Subtask #3750 added
Actions
Copy link
 #21

Updated by Thara S Pillai about 1 year ago

  • Subtask #3757 added
Actions
Copy link
 #22

Updated by Thara S Pillai about 1 year ago

  • Subtask #3776 added
Actions
Copy link
 #23

Updated by Thara S Pillai about 1 year ago

  • Subtask #3873 added
Actions
Copy link
 #24

Updated by Thara S Pillai about 1 year ago

  • Subtask #4065 added
Actions
Copy link

Also available in: Atom PDF