Activity #3735
closedModule #3693: Big Data Analytics Platform and Tools for Epidemic Forecasting and Mitigation
Module #3696: Center for Epidemic Forecsting(EPIFORM)
Module #3704: Technology Exploration
Demonstrate Delta lake - scheme enforcement/evolution
100%
Description
Ingest a data csv.
Develop a Rest API using python pySpark API/sparkSQL and Fast API framework to read data from the delta lake.
Create the csv with addition/deletion of columns
Ingets the modified data csv.
Run REST API using jMeter to demonstrate scheme enforcement/evolution works https://www.databricks.com/blog/2019/09/24/diving-into-delta-lake-schema-enforcement-evolution.html
Updated by Thara S Pillai about 1 year ago
- Planned Due Date changed from 06/11/2025 to 06/16/2025
If created parquet files not working, try to download from https://www.tablab.app/parquet/sample and see whether code works.
Refer: https://medium.com/@uzzaman.ahmed/introduction-to-pyspark-parquet-api-read-and-write-with-parameters-b43a4695a0e2
Updated by Sreeshna T about 1 year ago
- Status changed from In Progress to Resolved
Updated by Thara S Pillai about 1 year ago
- Status changed from Resolved to Closed