DataStage开发经验汇总
Datastage开发经验汇总 润信科技
文档信息
项目名称 文档名称 DataStage开发经验汇总 项目经理 批复信息
拟制 审核 批准 负责人 日期 意见 文件控制
版本记录 日期 2008.1.1 审阅记录 日期 分发记录 日期 作者 张定行 审阅者 接受人 版本 V1.00 意见 地点 变更说明 建立文档 第1页
Datastage开发经验汇总 润信科技
目 录
1
引言 ........................................................................................ 1
1.1 1.2 1.3
编写目的 .......................................................................................... 1 参考资料 .......................................................................................... 1 说明.................................................................................................. 1
2 DS软件简介 ........................................................................... 2
2.1
DataStage企业版概述 ..................................................................... 2
2.1.1 2.1.2 2.1.3 2.1.4 2.1.5 2.1.6 2.1.7 2.1.8
绪论 .......................................................................................................... 2 Advanced Development and Maintenance ............................................. 2 Complete Development Environment ...................................................... 4 Enterprise Deployment and Management ............................................... 7 Highly Scalable Architecture .................................................................... 9 End－to－End Enterprise Meta Data .................................................... 10 附录A..................................................................................................... 13 附录B..................................................................................................... 18
2.2
2.3 DataStage client 介绍 .................................................................... 19 Administrator .................................................................................. 20
2.3.1 2.3.2 2.3.3
全局设置 ................................................................................................. 20 工程属性设置 ......................................................................................... 20 Environment ........................................................................................... 22 菜单栏介绍 ............................................................................................. 24 工具栏介绍 ............................................................................................. 26 资源目录介绍 ......................................................................................... 26 显示区介绍 ............................................................................................. 27 菜单栏介绍 ............................................................................................. 28 工具栏介绍 ............................................................................................. 30 作业目录介绍 ......................................................................................... 31 显示区介绍 ............................................................................................. 31 工具栏介绍 ............................................................................................. 32 菜单栏介绍 ............................................................................................. 33 资源目录介绍 ......................................................................................... 35 设计区介绍 ............................................................................................. 36 Job属性 ................................................................................................. 36 Job类型 ................................................................................................. 37
2.4 Manager ......................................................................................... 23
2.4.1
2.4.2 2.4.3 2.4.4
2.5 Director .......................................................................................... 27
2.5.1 2.5.2 2.5.3 2.5.4
2.6 Designer ........................................................................................ 31
2.6.1 2.6.2 2.6.3 2.6.4 2.6.5 2.6.6
3 DS安装与配置 ..................................................................... 38
3.1 3.2
安装与初始化 ................................................................................. 38 环境设置 ........................................................................................ 38
第1页
Datastage开发经验汇总 润信科技
3.3 3.4 3.5 3.6 3.7 3.8 3.9 3.10 安装步骤 ........................................................................................ 39 安装驱动程序 ................................................................................. 52 修改目录权限 ................................................................................. 53 修改dsenv ..................................................................................... 53 修改datastage字符集 ................................................................... 54 进程的启动和停止 .......................................................................... 54 验证DataStage的安装 .................................................................. 55 Client安装 ..................................................................................... 55
4 DS开发流程介绍 .................................................................. 56
4.1 4.2 4.3
基本步骤 ........................................................................................ 56 运行机制 ........................................................................................ 56 注意事项 ........................................................................................ 56
5 Parallel Job Stage ................................................................ 58
5.1
Development/Debug Stages .......................................................... 58
5.1.1 5.1.2
Peek Stage............................................................................................. 58 Row Generator Stage ............................................................................ 58 Data Set Stage ....................................................................................... 58 Sequential File Set Stage ...................................................................... 60 File Set Stage ........................................................................................ 63 Lookup File Set Stage ........................................................................... 64 Oracle Enterprise Stage ........................................................................ 66 Aggregator Stage ................................................................................... 67 Change Capture Stage .......................................................................... 68 Compare Stage ...................................................................................... 70 Compress Stage .................................................................................... 70 Copy Stage ............................................................................................ 71 Difference Stage .................................................................................... 72 Expand Stage......................................................................................... 74 Filter Stage ............................................................................................. 74 Funnel Stage .......................................................................................... 75 Join Stage .............................................................................................. 76 Lookup Stage ......................................................................................... 78 Merge Stage .......................................................................................... 79 Modify Stage .......................................................................................... 81 Remove Duplicates Stage ..................................................................... 82 Sort Stage .............................................................................................. 83 Switch Stage .......................................................................................... 84 Transformer Stage ................................................................................. 85
5.2 File Stages ..................................................................................... 58
5.2.1
5.2.2 5.2.3 5.2.4
5.3 5.4
Database Stages............................................................................ 66
5.3.1 5.4.1 5.4.2 5.4.3 5.4.4 5.4.5 5.4.6 5.4.7 5.4.8 5.4.9 5.4.10 5.4.11 5.4.12 5.4.13 5.4.14 5.4.15 5.4.16 5.4.17
Processing Stages ......................................................................... 67
6 开发经验 ............................................................................... 87
第2页
Datastage开发经验汇总 润信科技
6.1 技巧汇总 ........................................................................................ 87
6.1.1 6.1.2 6.1.3 6.1.4 6.1.5 6.1.6 6.1.7 6.1.8 6.1.9
Filler和sqlload的关键字冲突问题。 .................................................... 87 自定义sql ............................................................................................... 87 Table name和tab definition一致的问题 ............................................... 87 数据的排序非常重要 .............................................................................. 87 数据排序实现的两种方式 ....................................................................... 87 聚合数据的两种方式 .............................................................................. 88 最优化的并行机制 .................................................................................. 88 SchemaFiles的使用 .............................................................................. 89 实现调度的方法 ...................................................................................... 90 Informix数据库连接 ............................................................................... 91 Oracle数据库连接 ................................................................................. 92 优化策略 ................................................................................................. 93 关键问题分析 ......................................................................................... 98 并行度 .................................................................................................... 98 处理建议 ................................................................................................. 99 其它 ...................................................................................................... 100 机器的对称性 ....................................................................................... 100 并行调度测试说明 ................................................................................ 100
6.2 常用数据库连接 .............................................................................. 91
6.2.1
6.2.2
6.3 性能调优 ........................................................................................ 93
6.3.1 6.3.2 6.3.3 6.3.4 6.3.5 6.3.6 6.3.7
7 问题汇总 ............................................................................. 102
7.1
开发问题 ...................................................................................... 102
7.1.1 7.1.2 7.1.3
数据库连接错误 .................................................................................... 102 Oracle入库出现空格 ............................................................................ 102 连接oracle时出错 ............................................................................... 102 Datastage重新启动无法登陆 ............................................................... 103 DS作业无法Reset，无法重新启动 ..................................................... 103 作业Link信息取不到 ........................................................................... 103 启动Server时，没有任何提示就且失败退出 ...................................... 103
7.2 系统问题 ...................................................................................... 103
7.2.1
7.2.2 7.2.3 7.2.4
第3页
