EPIFOM-DataIngestion

Steps for a Spark managed Delta Lake setup.

1. Copy folders <hstgshare>:\Epiform\DI

2. Set environment variables
HADOOP_CONF_DIR - <local path>\hadoop-3.4.1\etc\hadoop
HADOOP_HOME - <local path>\hadoop-3.4.1
SPARK_HOME - <local path>\spark-3.5.5-bin-hadoop3

3. Create folder
C:\tmp\spark-warehouse
and adjust values in Spark-3.5.5-bin-hadoop3\conf\spark-default.conf
for spark.sql.warehouse.dir data storage location managed by spark.

4. Environment variable Path to include
<local path>\spark-3.5.5-bin-hadoop3\bin
<local path>\hadoop-3.4.1\bin
C:\Program Files\Java\jdk-11\bin

5. Starting spark services:
cd <local path>\spark-3.5.5-bin-hadoop3\bin

6. Start the Spark Master:
spark-class org.apache.spark.deploy.master.Master
Now, Spark Master running at spark://<machine IP>::7077

7. Start the Worker and connect it to the Master:
spark-class org.apache.spark.deploy.worker.Worker spark://<machine IP>::7077
The Worker will start and register with the Master.

8. Start Spark Thrift Server (for SQL access)
spark-submit --class org.apache.spark.sql.hive.thriftserver.HiveThriftServer2 --master spark://<machine IP>:7077
Thrift server web UI: http://&lt;machine IP>:4040/jobs/

9. Use DBeaver IDE to connect to Delta Lake data tables
Create Connection with URL jdbc:hive2://<machine IP>:10000/default