After downloading ScalaIDE and after installing https://github.com/sonatype/m2eclipse-scala, still ran into the issue of not finding the right arch-type. Solved by adding a remote catalog: http://repo1.maven.org/maven2/archetype-catalog.xml
Here is the POM XML
1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 41 42 43 44 45 46 47 48 49 50 51 52 53 54 55 56 57 58 59 60 61 62 63 64 65 66 67 68 69 70 71 72 73 74 75 76 77 78 79 80 81 82 83 84 85 86 87 88 89 90 91 92 93 94 95 96 97 98 99 100 101 102 103 104 105 106 107 108 109 110 111 112 113 114 115 116 117 118 119 120 121 122 123 124 125 126 127 128 129 130 131 132 133 134 135 136 137 138 139 140 141 142 143 144 145 146 147 148 149 150 151 152 153 154 155 156 157 158 159 160 161 162 163 164 165 166 167 168 169 170 | <project xmlns="http://maven.apache.org/POM/4.0.0" xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance" xsi:schemaLocation="http://maven.apache.org/POM/4.0.0 http://maven.apache.org/maven-v4_0_0.xsd"> <modelVersion>4.0.0</modelVersion> <groupId>com.vitarkah.scala</groupId> <artifactId>first</artifactId> <version>0.0.1-SNAPSHOT</version> <name>${project.artifactId}</name> <description>My wonderfull scala app</description> <inceptionYear>2015</inceptionYear> <licenses> <license> <name>My License</name> <url>http://....</url> <distribution>repo</distribution> </license> </licenses> <properties> <maven.compiler.source>1.6</maven.compiler.source> <maven.compiler.target>1.6</maven.compiler.target> <encoding>UTF-8</encoding> <scala.version>2.10.6</scala.version> <scala.compat.version>2.10</scala.compat.version> </properties> <pluginRepositories> <pluginRepository> <id>scala-tools.org</id> <name>Scala-tools Maven2 Repository</name> <url>http://scala-tools.org/repo-releases</url> </pluginRepository> </pluginRepositories> <dependencies> <dependency> <groupId>org.apache.spark</groupId> <artifactId>spark-core_2.10</artifactId> <version>1.6.0</version> </dependency> <dependency> <groupId>org.apache.spark</groupId> <artifactId>spark-mllib_2.10</artifactId> <version>1.6.0</version> </dependency> <dependency> <groupId>junit</groupId> <artifactId>junit</artifactId> <version>4.1</version> <scope>test</scope> </dependency> <dependency> <groupId>org.scalactic</groupId> <artifactId>scalactic_2.10</artifactId> <version>2.2.6</version> </dependency> <dependency> <groupId>org.scalatest</groupId> <artifactId>scalatest_2.10</artifactId> <version>2.2.6</version> <scope>test</scope> </dependency> </dependencies> <build> <plugins> <!-- mixed scala/java compile --> <plugin> <groupId>org.scala-tools</groupId> <artifactId>maven-scala-plugin</artifactId> <executions> <execution> <id>compile</id> <goals> <goal>compile</goal> </goals> <phase>compile</phase> </execution> <execution> <id>test-compile</id> <goals> <goal>testCompile</goal> </goals> <phase>test-compile</phase> </execution> <execution> <phase>process-resources</phase> <goals> <goal>compile</goal> </goals> </execution> </executions> </plugin> <plugin> <artifactId>maven-compiler-plugin</artifactId> <configuration> <source>1.7</source> <target>1.7</target> </configuration> </plugin> <!-- for fatjar --> <plugin> <groupId>org.apache.maven.plugins</groupId> <artifactId>maven-assembly-plugin</artifactId> <version>2.4</version> <configuration> <descriptorRefs> <descriptorRef>jar-with-dependencies</descriptorRef> </descriptorRefs> </configuration> <executions> <execution> <id>assemble-all</id> <phase>package</phase> <goals> <goal>single</goal> </goals> </execution> </executions> </plugin> <plugin> <groupId>org.apache.maven.plugins</groupId> <artifactId>maven-jar-plugin</artifactId> <configuration> <archive> <manifest> <addClasspath>true</addClasspath> <mainClass>fully.qualified.MainClass</mainClass> </manifest> </archive> </configuration> </plugin> </plugins> <pluginManagement> <plugins> <!--This plugin's configuration is used to store Eclipse m2e settings only. It has no influence on the Maven build itself. --> <plugin> <groupId>org.eclipse.m2e</groupId> <artifactId>lifecycle-mapping</artifactId> <version>1.0.0</version> <configuration> <lifecycleMappingMetadata> <pluginExecutions> <pluginExecution> <pluginExecutionFilter> <groupId>org.scala-tools</groupId> <artifactId> maven-scala-plugin </artifactId> <versionRange> [2.15.2,) </versionRange> <goals> <goal>compile</goal> <goal>testCompile</goal> </goals> </pluginExecutionFilter> <action> <execute></execute> </action> </pluginExecution> </pluginExecutions> </lifecycleMappingMetadata> </configuration> </plugin> </plugins> </pluginManagement> </build> </project> |
Here is the “hello world” Scala app
1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 | package com.vitarkah.scala.first import org.apache.spark.SparkContext import org.apache.spark.SparkContext._ import org.apache.spark.SparkConf /** * @author ${user.name} */ object App { def main(args: Array[String]) { // initialise spark context val conf = new SparkConf().setAppName("HelloWorld").setMaster("local") val sc = new SparkContext(conf) // do stuff println("Hello, world!") // terminate spark context sc.stop() } } |
Here is the output
1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 41 | Using Spark's default log4j profile: org/apache/spark/log4j-defaults.properties 16/02/14 21:04:59 INFO SparkContext: Running Spark version 1.6.0 16/02/14 21:04:59 WARN NativeCodeLoader: Unable to load native-hadoop library for your platform... using builtin-java classes where applicable 16/02/14 21:05:00 WARN Utils: Your hostname, vichu-Lenovo-Z50-70 resolves to a loopback address: 127.0.1.1; using 192.168.1.140 instead (on interface wlan0) 16/02/14 21:05:00 WARN Utils: Set SPARK_LOCAL_IP if you need to bind to another address 16/02/14 21:05:00 INFO SecurityManager: Changing view acls to: vichu 16/02/14 21:05:00 INFO SecurityManager: Changing modify acls to: vichu 16/02/14 21:05:00 INFO SecurityManager: SecurityManager: authentication disabled; ui acls disabled; users with view permissions: Set(vichu); users with modify permissions: Set(vichu) 16/02/14 21:05:00 INFO Utils: Successfully started service 'sparkDriver' on port 51488. 16/02/14 21:05:00 INFO Slf4jLogger: Slf4jLogger started 16/02/14 21:05:01 INFO Remoting: Starting remoting 16/02/14 21:05:01 INFO Remoting: Remoting started; listening on addresses :[akka.tcp://sparkDriverActorSystem@192.168.1.140:38851] 16/02/14 21:05:01 INFO Utils: Successfully started service 'sparkDriverActorSystem' on port 38851. 16/02/14 21:05:01 INFO SparkEnv: Registering MapOutputTracker 16/02/14 21:05:01 INFO SparkEnv: Registering BlockManagerMaster 16/02/14 21:05:01 INFO DiskBlockManager: Created local directory at /tmp/blockmgr-37186a0b-a60a-41d7-8561-fd4054ef67f7 16/02/14 21:05:01 INFO MemoryStore: MemoryStore started with capacity 1088.6 MB 16/02/14 21:05:01 INFO SparkEnv: Registering OutputCommitCoordinator 16/02/14 21:05:01 INFO Utils: Successfully started service 'SparkUI' on port 4040. 16/02/14 21:05:01 INFO SparkUI: Started SparkUI at http://192.168.1.140:4040 16/02/14 21:05:01 INFO Executor: Starting executor ID driver on host localhost 16/02/14 21:05:01 INFO Utils: Successfully started service 'org.apache.spark.network.netty.NettyBlockTransferService' on port 50884. 16/02/14 21:05:01 INFO NettyBlockTransferService: Server created on 50884 16/02/14 21:05:01 INFO BlockManagerMaster: Trying to register BlockManager 16/02/14 21:05:01 INFO BlockManagerMasterEndpoint: Registering block manager localhost:50884 with 1088.6 MB RAM, BlockManagerId(driver, localhost, 50884) 16/02/14 21:05:01 INFO BlockManagerMaster: Registered BlockManager Hello, world! <<--- There it is! 16/02/14 21:05:01 INFO SparkUI: Stopped Spark web UI at http://192.168.1.140:4040 16/02/14 21:05:01 INFO MapOutputTrackerMasterEndpoint: MapOutputTrackerMasterEndpoint stopped! 16/02/14 21:05:01 INFO MemoryStore: MemoryStore cleared 16/02/14 21:05:01 INFO BlockManager: BlockManager stopped 16/02/14 21:05:01 INFO BlockManagerMaster: BlockManagerMaster stopped 16/02/14 21:05:01 INFO OutputCommitCoordinator$OutputCommitCoordinatorEndpoint: OutputCommitCoordinator stopped! 16/02/14 21:05:01 INFO SparkContext: Successfully stopped SparkContext 16/02/14 21:05:01 INFO RemoteActorRefProvider$RemotingTerminator: Shutting down remote daemon. 16/02/14 21:05:01 INFO RemoteActorRefProvider$RemotingTerminator: Remote daemon shut down; proceeding with flushing remote transports. 16/02/14 21:05:01 INFO ShutdownHookManager: Shutdown hook called 16/02/14 21:05:01 INFO ShutdownHookManager: Deleting directory /tmp/spark-44aa9e76-6aec-4162-8856-c1605c8f060c |