Read IBM Cognos TM1 Data in IBM SPSS Modeler

One of the new feature added in SPSS Modeler 16, is the support of reading data from IBM Cognos TM1 and exporting data to it.  In this post we will look at:

1. Prerequisite for reading data from IBM Cognos TM1 into IBM SPSS Modeler
2. Reading data from IBM Cognos TM1 cube into IBM SPSS Modeler

We can therefore bring in the Enterprise Planning (or other data from IBM Cognos TM1) functionality into the Predictive Analytics capabilities of SPSS.  This brings in synergy between the two products.

Prerequisites

1. SPSS Modeler must be ver 16 or above and IBM Cognos TM1 needs to be 10.2 or higher
2. Before you begin working with IBM Cognos TM1 in SPSS Modeler (either for import or export), you will need to copy following three processes (.pro files) from IBM SPSS Modeler <Modeler install directory>/ext/bin/pasw.tm1/scripts to the DATA directory of your IBM Cognos TM1 server.  To do this, stop the Cognos TM1 service, copy these files and start the Cognos TM1 service
   • ExportToSPSS.pro
   • ImportFromSPSS.pro
   • SPSSCreateNewMeasures.pro
3. If you have multiple TM1 servers on a given TM1 Admin Host, then you'd need to copy these 3 .pro files to all those DATA directories, with which you want to connect in IBM SPSS Modeler
4. If you have done TM1 integration before applying IBM SPSS Modeler 16.0 FP1, then after FP1 is applied, you will need to copy the 3 .pro files all over again ... i.e. you will need to repeat the step #2.. Please read this IBM KB article
5. Measure dimension needs to be defined on the IBM CognosTM1 cube, from which you are importing data.  To do this, right click on the cube, choose properties and pick the measure dimension
   • There must be at least one public view defined on the cube.  Private views cannot be read from IBM SPSS Modeler
6. Per IBM Knowledge Center, the data that needs to be read must be in UTF 8 format

Read data from IBM Cognos TM1

Please take few minutes to read and follow the prerequisites.  After you have done, you are all set to get data from powerful in-memory database into your predictive analytics tool.

In your stream, click on the Sources and place the IBM Cognos TM1 Import palette on your canvas.  Double click on the palette and set its properties.  First enter the PM hub URL and click on Connect

In the TM1 Server drop down there will be list of TM1 servers, pick the one where the cube resides.  For our test purposes, we will use Planning Sample.  Click on Login, enter the credentials and click Ok.  Notice the list of IBM Cognos TM1 Cubes in shown in SPSS as well as in Architect.

A test cube, which I created "SK_Cube1" is missing in the SPSS list.  There is no measure dimension defined for this cube.  Consequently it does NOT appear in the cube list within SPSS Modeler (as mentioned in #5 prerequisite).  We will go ahead and define a measure for this cube.  It can be either done by right clicking on the cube or entering the suitable text in }CubeProperties cube.  Once you are done, click on refresh in SPSS Modeler

Take a look at the screen below.

• In the blue box on right side, it shows the views available on this cube as well as the sample data entered from TM1 Architect.  Observe that there are Public Views available.  "View 1" is a private view
• In the red box on left side, there are views listed under the cube in SPSS Modeler.  Since there are no public views available, you cannot work with this TM1 cube, yet

We will open the private view and save it as Public View.  Then click refresh in SPSS Modeler.  Select the view and click on Import.  Then click on Preview, to see the sample data imported.  Voila!

       

When you hit the Preview button, it is at this stage "ExportoSPSS" process is executed.  If this process is not present in your TM1 server, you will get error right away.

IBM Cognos TM1 and Stored Procedures

We often use ODBC as Datasource Type in a Turbo Integrator (TI) process and load data into IBM Cognos TM1 cubes or build metadata.  In majority of the cases, we end up writing SQL code in Query box, to fetch the information that is required to perform the intended task.  Once in a while, we are required to work with a Stored Procedure (SP) in IBM Cognos TM1.

We will  look at how to execute an SP from TI. I will be using DB2' sample database to demonstrate the example.  SP name is TM1SP1 and its body is as follows.  A very simple SP, which accepts department number as a parameter, it then brings back the name and manager number associated with that department.

CREATE OR REPLACE PROCEDURE TM1.TM1SP1 (IN SrchNo char (3))
    DYNAMIC RESULT SETS 1
    LANGUAGE SQL
BEGIN
    DECLARE CUR1 CURSOR WITH RETURN TO CALLER FOR
        SELECT DeptNo, DeptName, MgrNo FROM TM1.DEPARTMENT WHERE DeptNo = SrchNo;
    OPEN CUR1;
END

Create a new TI process using ODBC as datasource type.  Choose the appropriate DSN, key in username and password.

For MS SQL if single-signon is enabled then, there is no need to key the username and password.  In our case, DB2 is not configured.  Therefore a named user and password is required to connect to the database.

 To run a stored procedure in DB2, you will need to use CALL function.  In our case, to run the TM1SP1 stored proc, the command will be 

In MS SQL or Oracle, you will use Execute SP_Name to run a SP.  Alternately you could abbreviate the Execute command to Exec and it will still work.

The preview pane of the TI task will now show the output of the stored procedure.  The test SP was written to return 3 columns from the Department table.  Preview pane in the TI window will show these 3 columns.

You can now switch to the Variable tab in TI window and work forward from there.

So far so good. This method will work.  As long as the stored procedure does NOT modify the data (i.e. there are no insert/update/delete statements in it).

Let's move on to Employee table in our sample database and look at Salary of Ted

SELECT EmpNo, FirstNme, LastName, Salary FROM TM1.Employee where EmpNo = '000100'; 

We will write an SP that increments salary of an employee by 1,000

CREATE OR REPLACE PROCEDURE TM1.TM1SP2 (IN SrchNo char (6))
    DYNAMIC RESULT SETS 1
    LANGUAGE SQL
BEGIN
    UPDATE TM1.Employee
    SET Salary = Salary + 1000
    WHERE EmpNo = SrchNo;
END

Every time a call to the SP is made, it increments the salary of employee number (passed as parameter) by 1,000.  Going by the above steps, if we end up creating a SP and write our TI as below

When you hit preview pane, you will see the results as below:

Ted's salary is already incremented by 1,000 to 87,150.  Let's proceed to variables tab and define the available 4 variables as Other.  In data tab, let's put this statement

AsciiOutput (GetProcessErrorFileDirectory | 'sk.txt', EmpNo, FirstNme, LastName, NumbertoString (salary));

Save the process and run it.  After the process is run, open the text file - sk.txt in our case.  Salary of Ted is incremented by another 1,000 to 88,150!


While the intention was to run the SP once and increment the Salary for given employee, by 1,000.  Net effect is that Salary is now incremented by 2,000.  Another unintended consequence of this approach is that, if someone just opens this TI process in TM1 Architect, then also the Salary is incremented!

Therefore, if the SP we are working with modifies the data, then it is better to run it in the prolog by using TI function DatasourceQuery.  

In our example, while we want to run the Stored Procedure in Prolog of Cognos TM1's TI, in the initial query pane, there is a need to have a valid SQL.  Since the Stored Procedure returns 4 columns, we will write out a sql that gets 4 columns but does nothing.

 When you hit Preview this is what is seen

Prolog tab will have the following code.

vs_SQL = 'Call TM1.TM1SP2 (''000100'')';
DatasourceQuery = vs_SQL;

Let the AsciiOutput statement remain as-is in the Data tab.  Set Ted's Salary back to 86,150.  

From TM1 Architect, run the modified TI process.  If we check the output file as well as the DB2 Table contents, we will now see that Salary is incremented by 1,000 alone.  Even if someone repeatedly opens the TI, the Salary is not changed - unless the process is run.

Summary:

• To run a stored procedure from a TI in IBM Cognos TM1, we can write the statement in the query pane.  This is ok as long as the SP fetches the data
• If however, the SP we want to work with modifies the data in some fashion (be it insert, update, delete or some combination of these) then writing the statement in the query pane, will lead to double/triple/quadruple/multiple executions, causing lot of grief