ETL Data Validation
Atul Singh
May 10, 2016
Organizations typically have “dirty data” that must be cleansed or scrubbed before being loaded into the data warehouse. In an ideal ...
Tuesday, May 10, 2016
Sunday, May 8, 2016
The DW Testing Life Cycle
Atul Singh
May 08, 2016
As with any other piece of software a DW implementation undergoes the natural cycle of Unit testing, System testing, Regression testing, I...
Friday, May 6, 2016
DS Fatal Error: Destination "APT_TRinput0Rec0" is already bound
Atul Singh
May 06, 2016
Fatal Error: Destination "APT_TRinput0Rec0" is already bound - Transformer Stage Error Solutions: * Check if the output sta...
Tuesday, May 3, 2016
Otherwise Constraint - A Quick DataStage Recipe
Atul Singh
May 03, 2016
Recipe: How to use "Otherwise" constraint in Transformer Stage How To: To use "Otherwise" constraint in Trans...
Tuesday, April 26, 2016
Tuning SQL Statement #2
Atul Singh
April 26, 2016
Tuning SQL Statement #1....... Tuning SQL Statement #3....... • The UNION statement performs the equivalent of a SELECT DISTINCT on t...
Monday, April 25, 2016
Tuning SQL Statement #1
Atul Singh
April 25, 2016
• Try to use explicit columns and try to read columns in index orders if possible, even to the point of reading indexes and not tables....
Sunday, April 24, 2016
Suppress Warnings - A Quick DataStage Recipe
Atul Singh
April 24, 2016
Recipe: How to suppress job warnings in DS job log HowTo: Add "rule" in Message Handler Method: From DS Director, f...
Tuesday, April 19, 2016
Shell Script Scenario #8 - Odd and Even No
Atul Singh
April 19, 2016
Write a shell script to find whether an input integer is even or odd. Like the below page to get update https:/...
Monday, April 18, 2016
Best book for Linux Shell Scripting - Intermediate User
Atul Singh
April 18, 2016
Today, I am gonna share my secret of Linux Scripting. The book, shared below, is core responsible for my shell scripting technique and ease...
10 Scenario based Interview Questions #2 - DataStage
Atul Singh
April 18, 2016
11. Design a job which insert the data if not exists and update if exists in target 12. Design a job which includes a header and footer i...
Wednesday, April 13, 2016
ETL Code Review Checklist
Atul Singh
April 13, 2016
Guideline Design jobs for restartability/ if not designed then what is the reason ? Do not follow Sequential File s...
Tuesday, April 12, 2016
Export DataStage job log from Director Client
Atul Singh
April 12, 2016
These steps are not for exporting log for more than 3-5 jobs as these are manual steps and need to repeat for each job (Yes, each job indi...
Sunday, April 10, 2016
5 Tips For Better DataStage Design #12
Atul Singh
April 10, 2016
1. Minimum number of sort stages should be use in a datastage job. “Don’t sort if previously sorted” in sort Stage, this option should be ...
Monday, April 4, 2016
DataStage Scenario #16 - Cross duplicate Removal
Atul Singh
April 04, 2016
Need to remove duplicate where source or destination can be switched. Input: source destination distance cit...
Friday, April 1, 2016
jupyter notebook tip #1
Atul Singh
April 01, 2016
How we can start jupyter notebook from a specific directory? Below, I am sharing a very tiny batch (and linux shell) script which will k...
Wednesday, March 30, 2016
echo the 'echo'
Atul Singh
March 30, 2016
"echo" command is the very basic command of Linux/Unix. Most of us known this command to print whatever is being passed to it ...
Monday, March 28, 2016
10 Scenario based Interview Questions #1 - DataStage
Atul Singh
March 28, 2016
1. Design a job which convert single source row to three target row. 2. Design a job which can identify the row if they are duplicate in i...
Saturday, March 26, 2016
Data Warehouse Glossary #3
Atul Singh
March 26, 2016
Drill Through: Data analysis that goes from an OLAP cube into the relational database. Data Warehousing: The process of designing, buil...
Wednesday, March 23, 2016
5 Tips For Better DataStage Design #11
Unknown
March 23, 2016
When writing intermediate results that will only be shared between parallel jobs, always write to persistent data sets (using Data Set ...
Tuesday, March 22, 2016
Transformer Stage alternative - A Quick DataStage Recipe
Atul Singh
March 22, 2016
What to use instead of "Transformer" Stage Copy Stage Use "Copy" stage instead of "Transformer" Stage fo...
Monday, March 21, 2016
Data Warehouse Glossary #2
Atul Singh
March 21, 2016
Dimension: A variable, perspective or general category of information that is used to organize and analyze information in a multi-dimensio...
Thursday, March 17, 2016
Data Warehouse Glossary #1
Atul Singh
March 17, 2016
Ad Hoc Query: A database search that is designed to extract specific information from a database. It is ad hoc if it is designed at the...
Tuesday, March 15, 2016
yet another Shell script to access DataStage job logs
Atul Singh
March 15, 2016
Introduction We have various data warehouses hosted on AIX/Linux/Unix operating systems. DataStage Director is one of the ETL monitori...
Monday, March 14, 2016
Machine Learning - Learning Steps
Atul Singh
March 14, 2016
Step 1 INTRODUCTION TO MACHINE LEARNING Overview of Machine Learning Supervised vs. Unsupervised Learning Classification vs. Regressio...
Machine Learning Links you must Visit
Atul Singh
March 14, 2016
1. Scikit-Learn Tutorial Series - http://buff.ly/1XnDWv6 2. 7 Free Machine Learning Courses - http://buff.ly/1XoBapa 3. k-nearest ne...
DataScience Links you must Visit
Atul Singh
March 14, 2016
1. Python (and R) for Data Science - sample code, libraries, projects, tutorials - http://buff.ly/22bwhXi 2. 19 Worst Mistakes at Data S...
Saturday, March 12, 2016
Capture Unmatched records from Join Stage - A Quick DataStage Recipe
Atul Singh
March 12, 2016
How to capture unmatched records from "Join" Stage. This stage does not provide reject handling for unmatched records (such as i...
Friday, March 11, 2016
Python Points #12 - execute OS commads
Atul Singh
March 11, 2016
Like the below page to get update https://www.facebook.com/datastage4you https://twitter.com/datagenx https://plus.goog...
