Project

General

Profile

Actions
Copy link

Activity #3758

open

Module #3693: Big Data Analytics Platform and Tools for Epidemic Forecasting and Mitigation

Module #3696: Center for Epidemic Forecsting(EPIFORM)

Integration and Usage of Delta Lake with Hadoop and Spark

Added by Prasidh J S about 1 year ago. Updated about 1 year ago.

Status:
Resolved
Priority:
Normal
Assignee:
Prasidh J S
Start date:
06/12/2025
Due date:
% Done:

100%

Estimated time:
Planned Due Date:

Description

explore and implement Delta Lake in a Hadoop-Spark ecosystem.
Installed Delta Lake version 2.4.0, compatible with Spark 3.4.4 and Scala 2.12.
Configured Spark to support Delta Lake
Configured Delta Lake to write and read directly from HDFS.
Ensured proper permissions and accessibility from Spark to the HDFS directory.
Loaded Delta-formatted data from HDFS:
Confirmed schema enforcement and data consistency.

Actions
Copy link
 #1

Updated by Prasidh J S about 1 year ago

  • Parent task set to #3696
Actions
Copy link

Also available in: Atom PDF