How To Install Spark On Windows 10

Apache Spark is a fast and general-purpose cluster computing system. It provides high-level APIs in Java, Scala, Python and R, and an optimized engine that supports general execution graphs. It as well supports a rich gear up of higher-level tools including Spark SQL for SQL and structured data processing, MLlib for machine learning, GraphX for graph processing, and Spark Streaming.

In this document, we volition embrace the installation process of Apache Spark on Windows 10 operating system

Prerequisites

This guide assumes that you lot are using Windows 10 and the user had admin permissions.

Organisation requirements:

• Windows 10 Bone
• At least 4 GB RAM
• Free space of at to the lowest degree 20 GB

Installation Process

Step ane: Go to the below official download page of Apache Spark and choose the latest release. For the bundle blazon, cull 'Pre-built for Apache Hadoop'.

The page will expect like below.

Step ii:   Once the download is completed unzip the file, to unzip the file using WinZip or WinRAR or 7-Cipher.

Step 3: Create a folder called Spark under your user Directory like below and copy paste the content from the unzipped file.

C:\Users\<USER>\Spark

It looks like below subsequently copy-pasting into the Spark directory.

Stride 4: Go to the conf folder and open log file called, log4j.properties. template. Modify INFO to WARN (It can be ERROR to reduce the log). This and side by side steps are optional.

Remove. template and then that Spark can read the file.

Before removing. template all files await like below.

After removing. template extension, files will await similar beneath

Step 5: Now we need to configure path.

Go to Command Console -> System and Security -> System -> Avant-garde Settings -> Environment Variables

Add below new user variable (or System variable) (To add together new user variable click on New push nether User variable for <USER>)

Click OK.

Add %SPARK_HOME%\bin to the path variable.

Click OK.

Footstep 6: Spark needs a slice of Hadoop to run. For Hadoop two.seven, yous need to install winutils.exe.

You can find winutils.exe from below page

Download it.

Step 7: Create a folder called winutils in C drive and create a folder chosen bin inside. Then, movement the downloaded winutils file to the bin binder.

C:\winutils\bin

Add the user (or arrangement) variable %HADOOP_HOME% like SPARK_HOME.

Click OK.

Stride viii : To install Apache Spark, Java should be installed on your calculator. If you don't take java installed in your system. Please follow the below process

Java Installation Steps :

• Go to the official Coffee site mentioned below  the page.

Accept Licence Agreement for Coffee SE Development Kit 8u201

• Download jdk-8u201-windows-x64.exe file
• Double Click on Downloaded .exe file, you will the window shown below.

• ClickNext.
• So below window will be displayed.

• Click Side by side.
• Below window will be displayed after some process.

• Click Close.

Exam Java Installation :

Open Control Line and type java -version,and then it should display installed version of Java

Y'all should also check JAVA_HOME and path of %JAVA_HOME%\bin included in user variables (or organization variables)

1. In the finish, the surround variables take three new paths (if you demand to add Java path, otherwise SPARK_HOME and HADOOP_HOME).

two. Create c:\tmp\hive directory. This pace is not necessary for later versions of Spark. When you first start Spark, it creates the folder past itself. However, it is the all-time practice to create a folder.

C:\tmp\hive

Exam Installation :

Open command line and type spark-shell, you get the result as below.

We have completed spark installation on Windows system. Allow'due south create RDD and     Data frame

We create one RDD and Data frame then will end up.

ane. We can create RDD in 3 means, we will use 1 way to create RDD.

Ascertain whatever list then parallelize information technology. It volition create RDD. Below is code and copy paste it one past one on the command line.

val list = Assortment(1,ii,3,4,5) val rdd = sc.parallelize(list)

To a higher place will create RDD.

ii. Now we volition create a Information frame from RDD. Follow the below steps to create Dataframe.

import spark.implicits._ val df = rdd.toDF("id")

To a higher place lawmaking will create Dataframe with id as a column.

To brandish the data in Dataframe use below command.

Df.testify()

Information technology volition display the below output.

How to uninstall Spark from Windows 10 Organisation:

Please follow below steps to uninstall spark on Windows 10.

1. Remove beneath System/User variables from the organization.
2. SPARK_HOME
3. HADOOP_HOME

To remove System/User variables please follow below steps:

Go to Control Panel -> System and Security -> System -> Advanced Settings -> Environment Variables, then notice SPARK_HOME and HADOOP_HOME so select them, and printing DELETE push button.

Find Path variable Edit -> Select %SPARK_HOME%\bin -> Press DELETE Button

Select % HADOOP_HOME%\bin -> Press DELETE Button -> OK Button

Open Control Prompt the type spark-shell so enter, now we get an error. At present nosotros can confirm that Spark is successfully uninstalled from the System.

Source: https://www.knowledgehut.com/blog/big-data/how-to-install-apache-spark-on-windows

Posted by: beahmprothervents.blogspot.com

Share this post