Install spark ubuntu docker image how to#
Install spark ubuntu docker image install#
Install spark ubuntu docker image download#

Run the spark shell, exectue: scala> val textFile = sc.textFile ( 'hdfs://localhost:9000/user/(username)/input/Apache_Spark' ) scala> unt ()

You can also just leave it on the hard-drive, and use its absolute path when reading the file.

Install spark ubuntu docker image download#

Extract to a local directory.Д«efore we actually start to play with spark, letвЂ™s download some text file, say, this page, and put it in hdfs as /user/(Your user name)/input/Apache_Spark. Finally, spark!Дownload spark from here, choose pre-built version of hadoop 2.4.

Install spark ubuntu docker image install#

Extract the tar file to a local directory, say, /usr/local/hadoopДЎ.Make sure your system has ssh and rsync: sudo apt-get install sshДў.Configure environment variables: # set to the root of your Java installation, mine is /usr/lib/jvm/java-7-oracle/ export JAVA_HOME = 3. Note that in sparkвЂ™s official site it only gives pre-compiled version compatible with hadoop up to 2.4, but the one works with 2.4 can also work with 2.5 after testing. At this times the newest is 2.5.2, download it. You can also refer to here for the configuration of the newest version hadoop and skip this part(though in my configuration the procedure is a little different).ДЇirst download hadoop from official site. This section talks about configuring a hadoop instance(also in standalone mode). Still, this is a tutorial so I try to be as comprehensive as possible. However this is optional because you can also specify a local path in spark shell. We can setup hadoop and ask spark to read file from hdfs in later section. If you have multiple installations of Java, you can run this command to choose the one you want to use: sudo update-alternatives -config java 2. Then install the desired version of JDK, I used JDK 7: sudo apt-get install oracle-java7-installer Sudo add-apt-repository ppa:webupd8team/java To install Oracle JDK, first run: sudo apt-get install python-software-properties In this case Oracle JDK is better than OpenJDK. If you donвЂ™t have the patience and just want to play, you can download the docker image from here and follow the instruction at the end of this post.

Install spark ubuntu docker image how to#

HereвЂ™s how to configure a standalone-mode spark instance on ubuntu(I use 12.04). Compared with Hadoop, it offers much nicer abstractions(e.g.

Spark is a computational framework quite popular recently. Check it out!Дў8 November 2014 Setting up standalone Spark on ubuntu(With Docker Image Provided) I'm developing a Mini Course about Web Development.