Spark-Error-Common-创新互联

      IDEA 开发 WordCount（Spark）报错

WordCount {

}

WordCount{
  (args: Array[]) {
    hadoopAddr = conf = SparkConf().setAppName()
    sc = SparkContext(conf)
    mapRdd = sc.textFile(hadoopAddr)
    result = mapRdd.flatMap(_.split()).map((_)).reduceByKey(_ + _).collect()
  }
}

Using Spark's default log4j profile: org/apache/spark/log4j-defaults.properties

15/12/08 22:07:16 INFO SparkContext: Running Spark version 1.4.0

15/12/08 22:07:17 WARN NativeCodeLoader: Unable to load native-hadoop library for your platform... using builtin-java classes where applicable

15/12/08 22:07:18 ERROR Shell: Failed to locate the winutils binary in the hadoop binary path

java.io.IOException: Could not locate executable null\bin\winutils.exe in the Hadoop binaries.

at org.apache.hadoop.util.Shell.getQualifiedBinPath(Shell.java:355)

at org.apache.hadoop.util.Shell.getWinUtilsPath(Shell.java:370)

at org.apache.hadoop.util.Shell.(Shell.java:363)

at org.apache.hadoop.util.StringUtils.(StringUtils.java:79)

at org.apache.hadoop.security.Groups.parseStaticMapping(Groups.java:104)

at org.apache.hadoop.security.Groups.(Groups.java:86)

at org.apache.hadoop.security.Groups.(Groups.java:66)

at org.apache.hadoop.security.Groups.getUserToGroupsMappingService(Groups.java:280)

另外有需要云服务器可以了解下创新互联scvps.cn，海内外云服务器15元起步，三天无理由+7*72小时售后在线，公司持有idc许可证，提供“云服务器、裸金属服务器、高防服务器、香港服务器、美国服务器、虚拟主机、免备案服务器”等云主机租用服务以及企业上云的综合解决方案，具有“安全稳定、简单易用、服务可用性高、性价比高”等特点与优势，专为企业上云打造定制，能够满足用户丰富、多元化的应用场景需求。