INSTALL SPARK ON WINDOWS 7 INSTALL
INSTALL SPARK ON WINDOWS 7 64 BIT

INSTALL SPARK ON WINDOWS 7 INSTALL

In one sense it is good that it makes us aware of many of the internal things, but on the other hand sometimes you just want to install the stuff and get startedd with it without wanting to spend days configuring it. I wish Java based tools and applications in future will be easier to deploy. In my past exxperience I always found it very difficult to debug issues as the error messages showed some error which may not be the correct source of the problem. Personally this is one thing which I do not like about Java. There was a big stack trace which was difficult to understand. The error which I received showed problems with creation of SqlContext. When the Spark-shell command is executed, there are multiple errors which are reported on the console. This seems to be a known issue as there are multiple resources on the internet which talk about it.

INSTALL SPARK ON WINDOWS 7 64 BIT

Unfortunately on Windows 10 64 bit machine, Spark does not start very well. If everything is installed successfully, we should get a Scala shell to execute our commands. After extracting the contents of the downloaded file, I tried running the Spark-shell command from the commnand prompt. If you prefer you can also download the source code & build the whole package. I chose the version which is pre-built for Hadoop 2.6 and later. I started with downloading a copy of Spark distribution 1.5.2 Nov 9 2015from the Apache website. What is the problem in installing Spark in stand alone mode on Windows machine?

Currently it is one of the most popular projects among the different tools used as part of Hadoop ecosystem. Spark is written in Scala & support Scala, Java, Python and R languages as of writing this post in January 2016. This also means that we do not need to set up Hadoop cluster to get started with Spark. One of the positives of Spark is that it can be run in standalone mode without having to setup nodes in the cluster. It also fits more in the distributed computing paradigm related to big data world. It claims to be upto 100 times more faster compared to Hadoop MapReduce processing methods. Spark is an execution engine which is gaining popularity due to its ability to perform in memory parallel processing. Net background and new to Java way of working might face similar difficulties tat I faced over a day to get Spark up and running.

I got back to bit of programming after a long gap and it was quite evident that I struggledd a bit in configuring the system. This post is about my experice of setting up Spark as a standalone instance on Windows 10 64 bit machine. Hopefully the next ones will be much more frequent. Its been almost 2 years since I wrote a blog post.

Install spark on windows 7