Sqoop从Oracle导出数据出错：TheNetwork

Sqoop执行命令：

成都创新互联专注于神农架林区企业网站建设,响应式网站开发,商城系统网站开发。神农架林区网站建设公司,为神农架林区等地区提供建站服务。全流程定制设计，专业设计，全程项目跟踪，成都创新互联专业和态度为您提供的服务

    ./sqoop  import  --connect jdbc:oracle:thin:@10.112.101.251:1621:crmadbmr --username bass_etl --password 75!n!u6J --table DBPMSADM.PD_USERSVC_INFO_00 -m 4 --target-dir /ext/ods/PD_USERSVC_INFO_00/2014071906

执行日志：

crmd3n:/d2_data0/user/ocdc/bin/sqoop-1.4.2-cdh5.2.1/bin> 4 --target-dir /ext/ods/PD_USERSVC_INFO_00/2014071906 <

14/07/19 15:31:30 WARN tool.BaseSqoopTool: Setting your password on the command-line is insecure. Consider using -P instead. 

14/07/19 15:31:30 INFO manager.SqlManager: Using default fetchSize of 1000

14/07/19 15:31:30 INFO tool.CodeGenTool: Beginning code generation

14/07/19 15:31:31 INFO manager.OracleManager: Time zone has been set to GMT

14/07/19 15:31:31 INFO manager.SqlManager: Executing SQL statement: SELECT t.* FROM DBPMSADM.PD_USERSVC_INFO_00 t WHERE 1=0

14/07/19 15:31:31 INFO orm.CompilationManager: HADOOP_MAPRED_HOME is /d2_data0/user/ocdc/bin/hadoop-2.0.0-mr1-cdh5.2.1

注意：/tmp/sqoop-ocdc/compile/f2f2da5cca8a7ec0811e1390418222a8/DBPMSADM_PD_USERSVC_INFO_00.java 使用或覆盖了已过时的 API。

注意：要了解详细信息，请使用 -Xlint:deprecation 重新编译。

14/07/19 15:31:33 INFO orm.CompilationManager: Writing jar file: /tmp/sqoop-ocdc/compile/f2f2da5cca8a7ec0811e1390418222a8/DBPMSADM.PD_USERSVC_INFO_00.jar

14/07/19 15:31:33 INFO manager.OracleManager: Time zone has been set to GMT

14/07/19 15:31:33 WARN manager.OracleManager: The table DBPMSADM.PD_USERSVC_INFO_00 contains a multi-column primary key. Sqoop will default to the column SVC_ID only for this job.

14/07/19 15:31:33 INFO manager.OracleManager: Time zone has been set to GMT

14/07/19 15:31:33 WARN manager.OracleManager: The table DBPMSADM.PD_USERSVC_INFO_00 contains a multi-column primary key. Sqoop will default to the column SVC_ID only for this job.

14/07/19 15:31:33 INFO mapreduce.ImportJobBase: Beginning import of DBPMSADM.PD_USERSVC_INFO_00

14/07/19 15:31:33 WARN util.NativeCodeLoader: Unable to load native-hadoop library for your platform... using builtin-java classes where applicable

14/07/19 15:31:33 INFO manager.OracleManager: Time zone has been set to GMT

14/07/19 15:31:34 WARN mapred.JobClient: Use GenericOptionsParser for parsing the arguments. Applications should implement Tool for the same.

14/07/19 15:31:36 INFO db.DataDrivenDBInputFormat: BoundingValsQuery: SELECT MIN(SVC_ID), MAX(SVC_ID) FROM DBPMSADM.PD_USERSVC_INFO_00

14/07/19 15:31:37 WARN db.TextSplitter: Generating splits for a textual index column.

14/07/19 15:31:37 WARN db.TextSplitter: If your database sorts in a case-insensitive order, this may result in a partial import or duplicate records.

14/07/19 15:31:37 WARN db.TextSplitter: You are strongly encouraged to choose an integral split column.

14/07/19 15:31:38 INFO mapred.JobClient: Running job: job_201407151514_3489

14/07/19 15:31:39 INFO mapred.JobClient: map 0% reduce 0%

14/07/19 15:31:50 INFO mapred.JobClient: Task Id : attempt_201407151514_3489_m_000000_0, Status : FAILED

java.lang.RuntimeException: java.lang.RuntimeException: java.sql.SQLException: Io exception: The Network Adapter could not establish the connection

        at org.apache.sqoop.mapreduce.db.DBInputFormat.setConf(DBInputFormat.java:167)

        at org.apache.hadoop.util.ReflectionUtils.setConf(ReflectionUtils.java:70)

        at org.apache.hadoop.util.ReflectionUtils.newInstance(ReflectionUtils.java:130)

        at org.apache.hadoop.mapred.MapTask.runNewMapper(MapTask.java:636)

        at org.apache.hadoop.mapred.MapTask.run(MapTask.java:331)

        at org.apache.hadoop.mapred.Child$4.run(Child.java:268)

        at java.security.AccessController.doPrivileged(Native Method)

        at javax.security.auth.Subject.doAs(Subject.java:415)

        at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1408)

        at org.apache.hadoop.mapred.Child.main(Child.java:262)

Caused by: java.lang.RuntimeException: java.sql.SQLException: Io exception: The Network Adapter could not establish the connection

        at org.apache.sqoop.mapreduce.db.DBInputFormat.getConnection

14/07/19 15:31:51 INFO mapred.JobClient: Task Id : attempt_201407151514_3489_m_000001_0, Status : FAILED

java.lang.RuntimeException: java.lang.RuntimeException: java.sql.SQLException: Io exception: The Network Adapter could not establish the connection 

        at org.apache.sqoop.mapreduce.db.DBInputFormat.setConf(DBInputFormat.java:167)

        at org.apache.hadoop.util.ReflectionUtils.setConf(ReflectionUtils.java:70)

        at org.apache.hadoop.util.ReflectionUtils.newInstance(ReflectionUtils.java:130)

        at org.apache.hadoop.mapred.MapTask.runNewMapper(MapTask.java:636)

        at org.apache.hadoop.mapred.MapTask.run(MapTask.java:331)

        at org.apache.hadoop.mapred.Child$4.run(Child.java:268)

        at java.security.AccessController.doPrivileged(Native Method)

        at javax.security.auth.Subject.doAs(Subject.java:415)

        at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1408)

        at org.apache.hadoop.mapred.Child.main(Child.java:262)

Caused by: java.lang.RuntimeException: java.sql.SQLException: Io exception: The Network Adapter could not establish the connection

        at org.apache.sqoop.mapreduce.db.DBInputFormat.getConnection

14/07/19 15:31:51 INFO mapred.JobClient: Task Id : attempt_201407151514_3489_m_000003_0, Status : FAILED

java.lang.RuntimeException: java.lang.RuntimeException: java.sql.SQLException: Io exception: The Network Adapter could not establish the connection 

        at org.apache.sqoop.mapreduce.db.DBInputFormat.setConf(DBInputFormat.java:167)

        at org.apache.hadoop.util.ReflectionUtils.setConf(ReflectionUtils.java:70)

        at org.apache.hadoop.util.ReflectionUtils.newInstance(ReflectionUtils.java:130)

        at org.apache.hadoop.mapred.MapTask.runNewMapper(MapTask.java:636)

        at org.apache.hadoop.mapred.MapTask.run(MapTask.java:331)

        at org.apache.hadoop.mapred.Child$4.run(Child.java:268)

        at java.security.AccessController.doPrivileged(Native Method)

        at javax.security.auth.Subject.doAs(Subject.java:415)

        at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1408)

        at org.apache.hadoop.mapred.Child.main(Child.java:262)

Caused by: java.lang.RuntimeException: java.sql.SQLException: Io exception: The Network Adapter could not establish the connection 

        at org.apache.sqoop.mapreduce.db.DBInputFormat.getConnection

14/07/19 15:31:53 INFO mapred.JobClient: map 25% reduce 0% 

14/07/19 15:31:58 INFO mapred.JobClient: Task Id : attempt_201407151514_3489_m_000001_1, Status : FAILED

java.lang.RuntimeException: java.lang.RuntimeException: java.sql.SQLException: Io exception: The Network Adapter could not establish the connection 

        at org.apache.sqoop.mapreduce.db.DBInputFormat.setConf(DBInputFormat.java:167)

        at org.apache.hadoop.util.ReflectionUtils.setConf(ReflectionUtils.java:70)

        at org.apache.hadoop.util.ReflectionUtils.newInstance(ReflectionUtils.java:130)

        at org.apache.hadoop.mapred.MapTask.runNewMapper(MapTask.java:636)

        at org.apache.hadoop.mapred.MapTask.run(MapTask.java:331)

        at org.apache.hadoop.mapred.Child$4.run(Child.java:268)

        at java.security.AccessController.doPrivileged(Native Method)

        at javax.security.auth.Subject.doAs(Subject.java:415)

        at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1408)

        at org.apache.hadoop.mapred.Child.main(Child.java:262)

Caused by: java.lang.RuntimeException: java.sql.SQLException: Io exception: The Network Adapter could not establish the connection 

        at org.apache.sqoop.mapreduce.db.DBInputFormat.getConnection

14/07/19 15:31:58 INFO mapred.JobClient: Task Id : attempt_201407151514_3489_m_000000_1, Status : FAILED

java.lang.RuntimeException: java.lang.RuntimeException: java.sql.SQLException: Io exception: The Network Adapter could not establish the connection 

        at org.apache.sqoop.mapreduce.db.DBInputFormat.setConf(DBInputFormat.java:167)

        at org.apache.hadoop.util.ReflectionUtils.setConf(ReflectionUtils.java:70)

        at org.apache.hadoop.util.ReflectionUtils.newInstance(ReflectionUtils.java:130)

        at org.apache.hadoop.mapred.MapTask.runNewMapper(MapTask.java:636)

        at org.apache.hadoop.mapred.MapTask.run(MapTask.java:331)

        at org.apache.hadoop.mapred.Child$4.run(Child.java:268)

        at java.security.AccessController.doPrivileged(Native Method)

        at javax.security.auth.Subject.doAs(Subject.java:415)

        at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1408)

        at org.apache.hadoop.mapred.Child.main(Child.java:262)

Caused by: java.lang.RuntimeException: java.sql.SQLException: Io exception: The Network Adapter could not establish the connection

        at org.apache.sqoop.mapreduce.db.DBInputFormat.getConnection

14/07/19 15:31:59 INFO mapred.JobClient: Task Id : attempt_201407151514_3489_m_000003_1, Status : FAILED

java.lang.RuntimeException: java.lang.RuntimeException: java.sql.SQLException: Io exception: The Network Adapter could not establish the connection 

        at org.apache.sqoop.mapreduce.db.DBInputFormat.setConf(DBInputFormat.java:167)

        at org.apache.hadoop.util.ReflectionUtils.setConf(ReflectionUtils.java:70)

        at org.apache.hadoop.util.ReflectionUtils.newInstance(ReflectionUtils.java:130)

        at org.apache.hadoop.mapred.MapTask.runNewMapper(MapTask.java:636)

        at org.apache.hadoop.mapred.MapTask.run(MapTask.java:331)

        at org.apache.hadoop.mapred.Child$4.run(Child.java:268)

        at java.security.AccessController.doPrivileged(Native Method)

        at javax.security.auth.Subject.doAs(Subject.java:415)

        at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1408)

        at org.apache.hadoop.mapred.Child.main(Child.java:262)

Caused by: java.lang.RuntimeException: java.sql.SQLException: Io exception: The Network Adapter could not establish the connection

        at org.apache.sqoop.mapreduce.db.DBInputFormat.getConnection

14/07/19 15:32:05 INFO mapred.JobClient: Task Id : attempt_201407151514_3489_m_000000_2, Status : FAILED

java.lang.RuntimeException: java.lang.RuntimeException: java.sql.SQLException: Io exception: The Network Adapter could not establish the connection 

        at org.apache.sqoop.mapreduce.db.DBInputFormat.setConf(DBInputFormat.java:167)

        at org.apache.hadoop.util.ReflectionUtils.setConf(ReflectionUtils.java:70)

        at org.apache.hadoop.util.ReflectionUtils.newInstance(ReflectionUtils.java:130)

        at org.apache.hadoop.mapred.MapTask.runNewMapper(MapTask.java:636)

        at org.apache.hadoop.mapred.MapTask.run(MapTask.java:331)

        at org.apache.hadoop.mapred.Child$4.run(Child.java:268)

        at java.security.AccessController.doPrivileged(Native Method)

        at javax.security.auth.Subject.doAs(Subject.java:415)

        at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1408)

        at org.apache.hadoop.mapred.Child.main(Child.java:262)

Caused by: java.lang.RuntimeException: java.sql.SQLException: Io exception: The Network Adapter could not establish the connection 

        at org.apache.sqoop.mapreduce.db.DBInputFormat.getConnection

14/07/19 15:32:06 INFO mapred.JobClient: Task Id : attempt_201407151514_3489_m_000003_2, Status : FAILED

java.lang.RuntimeException: java.lang.RuntimeException: java.sql.SQLException: Io exception: The Network Adapter could not establish the connection

        at org.apache.sqoop.mapreduce.db.DBInputFormat.setConf(DBInputFormat.java:167)

        at org.apache.hadoop.util.ReflectionUtils.setConf(ReflectionUtils.java:70)

        at org.apache.hadoop.util.ReflectionUtils.newInstance(ReflectionUtils.java:130)

        at org.apache.hadoop.mapred.MapTask.runNewMapper(MapTask.java:636)

        at org.apache.hadoop.mapred.MapTask.run(MapTask.java:331)

        at org.apache.hadoop.mapred.Child$4.run(Child.java:268)

        at java.security.AccessController.doPrivileged(Native Method)

        at javax.security.auth.Subject.doAs(Subject.java:415)

        at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1408)

        at org.apache.hadoop.mapred.Child.main(Child.java:262)

Caused by: java.lang.RuntimeException: java.sql.SQLException: Io exception: The Network Adapter could not establish the connection

        at org.apache.sqoop.mapreduce.db.DBInputFormat.getConnection

14/07/19 15:32:08 INFO mapred.JobClient: map 50% reduce 0% 

14/07/19 15:32:17 INFO mapred.JobClient: Job complete: job_201407151514_3489

14/07/19 15:32:17 INFO mapred.JobClient: Counters: 24

14/07/19 15:32:17 INFO mapred.JobClient: File System Counters

14/07/19 15:32:17 INFO mapred.JobClient: FILE: Number of bytes read=0

14/07/19 15:32:17 INFO mapred.JobClient: FILE: Number of bytes written=369072

14/07/19 15:32:17 INFO mapred.JobClient: FILE: Number of read operations=0

14/07/19 15:32:17 INFO mapred.JobClient: FILE: Number of large read operations=0

14/07/19 15:32:17 INFO mapred.JobClient: FILE: Number of write operations=0

14/07/19 15:32:17 INFO mapred.JobClient: HDFS: Number of bytes read=256

14/07/19 15:32:17 INFO mapred.JobClient: HDFS: Number of bytes written=8

14/07/19 15:32:17 INFO mapred.JobClient: HDFS: Number of read operations=2

14/07/19 15:32:17 INFO mapred.JobClient: HDFS: Number of large read operations=0

14/07/19 15:32:17 INFO mapred.JobClient: HDFS: Number of write operations=2

14/07/19 15:32:17 INFO mapred.JobClient: Job Counters

14/07/19 15:32:17 INFO mapred.JobClient: Failed map tasks=1

14/07/19 15:32:17 INFO mapred.JobClient: Launched map tasks=12

14/07/19 15:32:17 INFO mapred.JobClient: Total time spent by all maps in occupied slots (ms)=91182

14/07/19 15:32:17 INFO mapred.JobClient: Total time spent by all reduces in occupied slots (ms)=0

14/07/19 15:32:17 INFO mapred.JobClient: Total time spent by all maps waiting after reserving slots (ms)=0

14/07/19 15:32:17 INFO mapred.JobClient: Total time spent by all reduces waiting after reserving slots (ms)=0

14/07/19 15:32:17 INFO mapred.JobClient: Map-Reduce Framework

14/07/19 15:32:17 INFO mapred.JobClient: Map input records=0

14/07/19 15:32:17 INFO mapred.JobClient: Map output records=0

14/07/19 15:32:17 INFO mapred.JobClient: Input split bytes=256

14/07/19 15:32:17 INFO mapred.JobClient: Spilled Records=0

14/07/19 15:32:17 INFO mapred.JobClient: CPU time spent (ms)=8910

14/07/19 15:32:17 INFO mapred.JobClient: Physical memory (bytes) snapshot=698327040

14/07/19 15:32:17 INFO mapred.JobClient: Virtual memory (bytes) snapshot=7732314112

14/07/19 15:32:17 INFO mapred.JobClient: Total committed heap usage (bytes)=2022703104

14/07/19 15:32:17 INFO mapreduce.ImportJobBase: Transferred 8 bytes in 43.3403 seconds (0.1846 bytes/sec)

14/07/19 15:32:17 INFO mapreduce.ImportJobBase: Retrieved 0 records.

14/07/19 15:32:17 ERROR tool.ImportTool: Error during import: Import job failed!

    昨天发了从DB2中导入数据的错误日志，后来定位为JDBC驱动包的问题，但是替换之后还是没用，然后又定位为是由于JDBC线程未关闭，然后换成了从Oracle中导入数据，又来了另外一个问题：连接有时能通，有时不能通，对于-m 4是基本通不了，原因在于Mapreduce是自动分发匹配的，所以有些结点可能是不可见的，这样对我们连接又会出现网络不通的问题，初步解决方案：自行修改源代码，找到任务分发代码，直接指定一个专门用来做数据导入的结点。正在实践ing。。。。