博客
关于我
强烈建议你试试无所不能的chatGPT,快点击我
【甘道夫】Sqoop1.99.3基础操作--导入Oracle的数据到HDFS
阅读量:4282 次
发布时间:2019-05-27

本文共 5893 字,大约阅读时间需要 19 分钟。

第一步:进入客户端Shell
fulong@FBI008:~$ 
sqoop.sh client
Sqoop home directory: /home/fulong/Sqoop/sqoop-1.99.3-bin-hadoop200
Sqoop Shell: Type 'help' or '\h' for help.
sqoop:000> 
set server --host FBI003 --port 12000 --webapp sqoop
Server is set successfully
sqoop:000> 
show version --all
client version:
  Sqoop 1.99.3 revision 2404393160301df16a94716a3034e31b03e27b0b
  Compiled by mengweid on Fri Oct 18 14:15:53 EDT 2013
server version:
  Sqoop 1.99.3 revision 2404393160301df16a94716a3034e31b03e27b0b
  Compiled by mengweid on Fri Oct 18 14:15:53 EDT 2013
Protocol version:
  [1]

第二步:创建连接
sqoop:000>
 create connection --cid 001
Creating connection for connector with id 
1
Please fill following values to create new connection object
Name: 
147patents
Connection configuration
JDBC Driver Class: 
oracle.jdbc.driver.OracleDriver
JDBC Connection String: 
jdbc:oracle:thin:@192.168.0.147:1521:ORCLGBK
Username: 
zhaobiao
Password: 
******
JDBC Connection Properties:
There are currently 0 values in the map:
entry#
Security related configuration options
Max connections: 
100
New connection was successfully created with validation status FINE and persistent id 1

第三步:创建Job
我这里顺便尝试了一下update命令,所以第一遍创建job时输入了错误的tablename:
sqoop:000> create job
Required argument --xid is missing.
sqoop:000> 
create job --xid 1 --type import
Creating job for connection with id 1
Please fill following values to create new job object
Name: 
ImportPatents
Database configuration
Schema name: 
zhaobiao
Table name: 
zhaobiao
Table SQL statement:
Table column names:
Partition column name:
Nulls in partition column:
Boundary query:
Output configuration
Storage type:
  0 : HDFS
Choose: 
0
Output format:
  0 : TEXT_FILE
  1 : SEQUENCE_FILE
Choose: 
0
Compression format:
  0 : NONE
  1 : DEFAULT
  2 : DEFLATE
  3 : GZIP
  4 : BZIP2
  5 : LZO
  6 : LZ4
  7 : SNAPPY
Choose: 
0
Output directory: 
/Data/zhaobiao
Throttling resources
Extractors:
Loaders:
New 
job was successfully created with validation status FINE  and persistent 
id 1
sqoop:000> update job
Required argument --jid is missing.
sqoop:000> 
update job --jid 1
Updating job with id 1
Please update job metadata:
Name: ImportPatents
Database configuration
Schema name: zhaobiao
Table name: 
CMS_NEWS_0625
Table SQL statement:
Table column names:
Partition column name:
Nulls in partition column:
Boundary query:
Output configuration
Storage type:
  0 : HDFS
Choose: 0
Output format:
  0 : TEXT_FILE
  1 : SEQUENCE_FILE
Choose: 0
Compression format:
  0 : NONE
  1 : DEFAULT
  2 : DEFLATE
  3 : GZIP
  4 : BZIP2
  5 : LZO
  6 : LZ4
  7 : SNAPPY
Choose: 0
Output directory: /Data/zhaobiao
Throttling resources
Extractors:
Loaders:
Job was successfully updated with status FINE
总结:
1.create job时必须指定一个连接id
2.create job完成后,自动分配了一个job id
3.update job时,需要指定job id
4.创建Job过程中指定的
Output directory指定的是HDFS中的路径

第四步:执行job
sqoop:000> 
start job --jid 1
2014-06-27 12:44:17 CST: FAILURE_ON_SUBMIT
Exception: org.apache.hadoop.mapred.FileAlreadyExistsException: Output directory hdfs://fulonghadoop/Data/zhaobiao already exists
sqoop:000> start job --jid 1
Submission details
Job ID: 1
Server URL: http://FBI003:12000/sqoop/
Created by: fulong
Creation date: 2014-06-27 12:44:53 CST
Lastly updated by: fulong
External ID: job_1403248336137_0005
        
http://FBI003:8088/proxy/application_1403248336137_0005/
2014-06-27 12:44:53 CST: BOOTING  - Progress is not available
sqoop:000> 
status job --jid 1
Submission details
Job ID: 1
Server URL: http://FBI003:12000/sqoop/
Created by: fulong
Creation date: 2014-06-27 12:44:53 CST
Lastly updated by: fulong
External ID: job_1403248336137_0005
        http://FBI003:8088/proxy/application_1403248336137_0005/
2014-06-27 12:50:11 CST: 
RUNNING  - 0.00 %
sqoop:000> status job --jid 1
Submission details
Job ID: 1
Server URL: http://FBI003:12000/sqoop/
Created by: fulong
Creation date: 2014-06-27 12:44:53 CST
Lastly updated by: fulong
External ID: job_1403248336137_0005
        http://FBI003:8088/proxy/application_1403248336137_0005/
2014-06-27 13:36:49 CST: 
RUNNING  - 25.00 %
sqoop:000> status job --jid 1
Submission details
Job ID: 1
Server URL: http://FBI003:12000/sqoop/
Created by: fulong
Creation date: 2014-06-27 12:44:53 CST
Lastly updated by: fulong
External ID: job_1403248336137_0005
        http://FBI003:8088/proxy/application_1403248336137_0005/
2014-06-27 13:39:24 CST: 
RUNNING  - 35.00 %
sqoop:000> status job --jid 1
Submission details
Job ID: 1
Server URL: http://FBI003:12000/sqoop/
Created by: fulong
Creation date: 2014-06-27 12:44:53 CST
Lastly updated by: fulong
External ID: job_1403248336137_0005
        http://FBI003:8088/proxy/application_1403248336137_0005/
2014-06-27 13:47:19 CST: 
RUNNING  - 40.00 %
sqoop:000> 
status job --jid 1
Submission details
Job ID: 1
Server URL: http://FBI003:12000/sqoop/
Created by: fulong
Creation date: 2014-06-27 12:44:53 CST
Lastly updated by: fulong
External ID: job_1403248336137_0005
        http://FBI003:8088/proxy/application_1403248336137_0005/
2014-06-27 14:08:36 CST: SUCCEEDED
Counters:
        org.apache.hadoop.mapreduce.JobCounter
                SLOTS_MILLIS_MAPS: 16499226
                TOTAL_LAUNCHED_MAPS: 10
                SLOTS_MILLIS_REDUCES: 0
                OTHER_LOCAL_MAPS: 15
        org.apache.hadoop.mapreduce.lib.output.FileOutputFormatCounter
                BYTES_WRITTEN: 421226094
        org.apache.hadoop.mapreduce.lib.input.FileInputFormatCounter
                BYTES_READ: 0
        org.apache.hadoop.mapreduce.TaskCounter
                MAP_INPUT_RECORDS: 0
                MERGED_MAP_OUTPUTS: 0
                PHYSICAL_MEMORY_BYTES: 1185689600
                SPILLED_RECORDS: 0
                COMMITTED_HEAP_BYTES: 627179520
                CPU_MILLISECONDS: 307570
                FAILED_SHUFFLE: 0
                VIRTUAL_MEMORY_BYTES: 6542311424
                SPLIT_RAW_BYTES: 1437
                MAP_OUTPUT_RECORDS: 644083
                GC_TIME_MILLIS: 5279
        org.apache.hadoop.mapreduce.FileSystemCounter
                FILE_WRITE_OPS: 0
                FILE_READ_OPS: 0
                FILE_LARGE_READ_OPS: 0
                FILE_BYTES_READ: 0
                HDFS_BYTES_READ: 1437
                FILE_BYTES_WRITTEN: 931780
                HDFS_LARGE_READ_OPS: 0
                HDFS_WRITE_OPS: 20
                HDFS_READ_OPS: 40
                HDFS_BYTES_WRITTEN: 431907904
        org.apache.sqoop.submission.counter.SqoopCounters
                ROWS_READ: 644083
Job executed 
successfully

接下来的问题:如何导入数据到Hbase中?

转载地址:http://ebcgi.baihongyu.com/

你可能感兴趣的文章
nyist-组队赛(六)
查看>>
nyist-组队赛(七)
查看>>
nyist-508(余数求和)
查看>>
zoj-3624(Count Path Pair)组合数+乘法逆元
查看>>
nyoj - 947(Max Xor)字典树
查看>>
单链表
查看>>
双向链表
查看>>
UVA Live 6068
查看>>
UVa Cellular Structure
查看>>
UVa Distinct Subsequences
查看>>
nyoj-字和串之差
查看>>
UVA Compromise(最长公共子序列 + 打印路径)
查看>>
01背包之打印路径
查看>>
01背包之(数之组合)
查看>>
reverse() 几种操作
查看>>
cf-123B
查看>>
poj -- 1185 炮兵阵地
查看>>
hdu-4539 郑厂长系列故事――排兵布阵(状态压缩)
查看>>
hdu-1074 Doing Homework
查看>>
hdu-5063 Operation the Sequence
查看>>