Scala can be used as a tool to manipulate big data. If it is used in the spark context, we have a possibility to combine two strong tools: spark with its possibility to bypass the MapReduce bottleneck and Scala with its short learning curve.<\/p>\n
The idea that Scala can be closely integrated with Spark is already clear when Scala is started. It can be started with the command “spark-shell” from the terminal. After a few seconds the Spark sign is shown with the note “Using Scala version 2.10.4”. A few lines below, one sees “Spark context available as sc”. This is the statement that we will use.<\/p>\n
We have quite a number of files in an HDFS directory. They can be made accessible as a RDD via:<\/p>\n
\nval rawblocks = sc.textFile(\"\/linkage\")\n<\/pre>\nSuch a RDD is a dataset that is composed of a number of lines. The number of lines can be seen with rawblocks.count() that yields: res63: Long = 5749142.
\nTo do anything with this RDD, we must 1\/ remove lines that are used as header, 2\/ split the lines and 3\/identify numbers as numbers. Let us do so.<\/p>\nRemoval of lines that act as a header can be done with this function:<\/p>\n
\ndef isHeader(line: String): Boolean = {\n line.contains(\"id_1\")\n }\n\nfollowed by\n\nval noheader = rawblocks.filter(x => !isHeader(x))\n<\/pre>\nSplitting the lines and identifying numbers as numbers is a bit cumbersome, but the code looks pretty straightforward:<\/p>\n
\ndef toDouble(s:String) = {\n if (\"?\".equals(s)) Double.NaN else s.toDouble\n }\n\ncase class MatchData(id1: Int, id2: Int,\n scores:Array[Double],matched: Boolean)\n\ndef parse(line: String) = {\n val pieces = line.split(',')\n val id1 = pieces(0).toInt\n val id2 = pieces(1).toInt\n val scores = pieces.slice(2, 11).map(toDouble)\n val matched = pieces(11).toBoolean\n MatchData(id1, id2, scores, matched)\n }\n\nfollowed by \n\nval parsed = noheader.map(line => parse(line))\n<\/pre>\nThen we are ready to create the histogram:<\/p>\n
\nval matchCounts = parsed.map(md => md.matched).countByValue()\nval matchCountsSeq = matchCounts.toSeq\nmatchCountsSeq.foreach(println)\n<\/pre>\n
\nFinally a less complicated script that calculates the mean of a column:<\/p>\n\nval famblocks = sc.textFile(\"\/user\/hdfs\/fam\")\ncase class MatchData(id1: Int, id2: String)\ndef parse(line: String) = {\n val pieces = line.split(',')\n val id1 = pieces(0).toInt\n val id2 = pieces(1).toString\n MatchData(id1,id2)\n }\nval parsed = famblocks.map(line => parse(line))\nval matchCounts = parsed.map(md => md.id1).mean()\n<\/pre>\nwith fam an HDFS file containing lines with a number and name, comma separated. (1,tom etc).<\/p>\n
<\/p>\n","protected":false},"excerpt":{"rendered":"
Scala can be used as a tool to manipulate big data. If it is used in the spark context, we have a possibility to combine two strong tools: spark with its possibility to bypass the MapReduce bottleneck and Scala with its short learning curve. The idea that Scala can be closely integrated with Spark is […]<\/p>\n","protected":false},"author":1,"featured_media":1520,"comment_status":"closed","ping_status":"open","sticky":false,"template":"","format":"standard","meta":{"footnotes":""},"categories":[1],"tags":[],"class_list":["post-1519","post","type-post","status-publish","format-standard","has-post-thumbnail","hentry","category-uncategorized"],"_links":{"self":[{"href":"http:\/\/archief.van-maanen.com\/index.php?rest_route=\/wp\/v2\/posts\/1519","targetHints":{"allow":["GET"]}}],"collection":[{"href":"http:\/\/archief.van-maanen.com\/index.php?rest_route=\/wp\/v2\/posts"}],"about":[{"href":"http:\/\/archief.van-maanen.com\/index.php?rest_route=\/wp\/v2\/types\/post"}],"author":[{"embeddable":true,"href":"http:\/\/archief.van-maanen.com\/index.php?rest_route=\/wp\/v2\/users\/1"}],"replies":[{"embeddable":true,"href":"http:\/\/archief.van-maanen.com\/index.php?rest_route=%2Fwp%2Fv2%2Fcomments&post=1519"}],"version-history":[{"count":0,"href":"http:\/\/archief.van-maanen.com\/index.php?rest_route=\/wp\/v2\/posts\/1519\/revisions"}],"wp:attachment":[{"href":"http:\/\/archief.van-maanen.com\/index.php?rest_route=%2Fwp%2Fv2%2Fmedia&parent=1519"}],"wp:term":[{"taxonomy":"category","embeddable":true,"href":"http:\/\/archief.van-maanen.com\/index.php?rest_route=%2Fwp%2Fv2%2Fcategories&post=1519"},{"taxonomy":"post_tag","embeddable":true,"href":"http:\/\/archief.van-maanen.com\/index.php?rest_route=%2Fwp%2Fv2%2Ftags&post=1519"}],"curies":[{"name":"wp","href":"https:\/\/api.w.org\/{rel}","templated":true}]}}