Create a Java Maven Application for Apache Spark in Eclipse

  1. Use Maven as the build system.
  2. Update Project Object Model (POM) file to include the Spark dependencies.
  3. Write your application in JAVA
  4. Generate a JAR file that can be submitted to Spark Cluster.
  5. Submit spark applications using spark-submit.

Prerequisites:

  1. Apache Spark installed on your machine.
  2. Oracle JAVA Development Kit.This article used openjdk version 1.8.0_275
  3. A JAVA IDE.This article uses Eclipse IDE.

Use Eclipse to create new application:

  1. Start Eclipse and select New Project.
  2. Select new Maven Project from the list and select next.
  3. Select Create a simple project (skip archetype selection) checkbox and click on next.
  4. Provide relevant values for GroupId, and ArtifactId. Name The following values are used in this tutorial:
  • Group Id: com.spark.example
  • Artifact Id : SparkSimpleApp
<maven.compiler.source>1.8</maven.compiler.source><maven.compiler.target>1.8</maven.compiler.target>
<dependency><groupId>org.apache.spark</groupId><artifactId>spark-core_2.12</artifactId><version>2.4.0</version></dependency><dependency><groupId>org.apache.spark</groupId><artifactId>spark-sql_2.12</artifactId><version>2.4.0</version></dependency>
import org.apache.spark.SparkConf;import org.apache.spark.sql.Dataset;import org.apache.spark.sql.Row;import org.apache.spark.sql.SparkSession;public class Main {public static void main(String[] args) {// TODO Auto-generated method stubSparkSession spark = SparkSession.builder().appName("json-demo").getOrCreate();Dataset<Row> employees = spark.read().json("/home/rahul/Desktop/eclipse-ee/employees.json");employees.printSchema();employees.show();employees.write().json("file:///home/rahul/Desktop/eclipse-ee/employees_spark.json");}}
Maven Build is successful

Run the application on the Apache Spark cluster

  1. Copy the application JAR to anywhere in your system.We need the path of the location where it is copied.
  2. Run the following command from terminal
spark-submit --class "Main" --master local[2] "<path-to-your-jar-file>"

--

--

Get the Medium app

A button that says 'Download on the App Store', and if clicked it will lead you to the iOS App store
A button that says 'Get it on, Google Play', and if clicked it will lead you to the Google Play store
Rahul Sawra

Rahul Sawra

Software Developer. Passionate about programming.