Stardust Database Access Tuning

A general approach for Stardust database access tuning consists of the following tasks:

Database Monitoring

First, you have to identify

The appropriate means for this task in general are database specific tools. Additionally, you may set the Stardust server property:

Carnot.Engine.Tuning.DB.slowStatementTracingThreshold = 500.

This property sets the threshold duration in milliseconds for all database statements launched by Stardust . If the threshold is exceeded, the corresponding statement is logged in the Stardust log. with WARN priority, e.g.

[06.12.05 12:01:53:443] [ORB.thread.pool : 4] [ WARN] Slow query: 1085ms SELECT COUNT(*) FROM activity_instance o INNER JOIN process_instance PR_PI ON (o.processInstance = PR_PI.oid) INNER JOIN data_value PR_DV1 ON (PR_PI.rootProcessInstance = PR_DV1.processInstance AND PR_DV1.data = ?) INNER JOIN ikr_dcluster_1 PR_DVCL1 ON (PR_PI.rootProcessInstance =

PR_DVCL1.processInstance) WHERE (o.state IN (?, ?) AND (PR_DVCL1.tk_an_bereich_bs = ? AND PR_DVCL1.sval_an_bereich_bs IN (?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?)) AND (PR_DV1.type_key = ? AND PR_DV1.string_value = ?) AND (PR_DVCL1.tk_an_rolle = ? AND PR_DVCL1.sval_an_rolle = ?) AND (o.currentUserPerformer = ? OR o.currentPerformer = ?))

A useful approach is to identify categories of critical statements with:

<session-name>.UsePreparedStatements = true

and to then deactivate prepared statement to obtain the concrete parameter values for critical queries (data IDs, process OIDs).

Analysis of the Execution Plan of Database Queries

Execution plans for critical queries are obtained with the tools provided by the DBMS in use. You may consult your database administrator for further details.

Cluster Definition and Indexing

The obtained list of critical statements can be used in two ways:

(A) A statement with multiple joins on the DATA_VALUE table - especially if launched very often - is a good candidate for a cluster slot.

Data being used together in predicates

... WHERE ... Data1=? AND Data2=?

should reside in the same cluster.

Additionally, indexes on these clusters might be considered according to the following principles:

of the index definition.

(B) With or without clusters being defined, the execution path (table scans, index scans) has to be considered in detail and known database tuning mechanisms, like

on cluster and non-cluster tables have to be applied. To decide on the appropriate approach you have to obtain the UPDATE/SELECT ratio for the tables to be indexed. In any case, theses considerations are not Stardust -specific and can be recommended by your database administrator without further knowledge of the Stardust Process Engine.

The steps described below can be performed repeatedly with successively reducing the threshold for critical statements.

Deadlocks

UPDATE-Deadlocks on data clusters caused by parallel access of two different threads on disjoint cluster slots can be avoided by splitting up the cluster. Critical clusters can be obtained from the fields listed in the corresponding log.

It might also be an approach to change the process model accordingly.

Deadlocks on other Stardust tables can be avoided by specifying additional indexes on these tables. This approach is also a standard database administration task and does not require specific knowledge of the Stardust products. Hence, you might consult your database administrator for further recommendations.

Retrieving Generated Keys Directly

Per default, database generated keys are retrieved on non-sequence databases after the INSERT statement. To avoid this extra round trip, you can use the property AuditTrail.UseJdbcReturnGeneratedKeys. If you add this property to the carnot.properties file and set it to true, the support provided by JDBC 1.4 for retrieving database generated keys directly from the INSERT statement will be used instead.

Archiving Audit Trail Contents

An audit trail database containing data for a large number of processes may reach a size, where the scalability limits of the used database system are reached. Usually, a high percentage of the process instances, whose data are stored in the audit trail database are already terminated. Hence, these process data can be exported to off line data sources and access only on demand.

Cleaning up Process Instance Hierarchy Entries upon Process Termination

To increase performance, you have the option to remove process instance hierarchy entries upon process termination. If you set the server-side property Carnot.Engine.Tuning.ProcessTerminationHierarchyCleanup in your carnot.properties file to true, process instance hierarchy entries are removed from the table procinst_hierarchy upon process termination. Per default this property is set to false. For details on table procinst_hierarchy, refer to the PROCINST_HIERARCHY TABLE description in chapter The Repository Model.

Using Prepared Statements

To reduce the number of statements the database needs to parse and optimize you can use prepared statements. In general, any query that is meant to be used more than once should be a prepared statement.

Using Prepared SQL Statements is usually estimated to raise performance by about 10%, no additional action needed. This has to be considered of special importance for complex queries like the ones issued from the Query API implementation. Now all queries and updates on the audit trail honor the session property to use Prepared SQL Statements:

<session-name>.UsePreparedStatements = false/true/mixed

Note that value false should only be used for debugging purposes! In production scenarios please use true or mixed instead.

We recommend to use the mixed mode. For example, if querying for process instances states and activity instance states, the data can be "unbalanced". Mostly there are a lot of completed and only a few non-completed states. Regarding the performance, in case such data occur, it is helpful for the database query analyzer and data access path builder to have the real values for these unbalanced data and additionally the prepared statements for general business data.

Note that the mixed mode is deprecated. In a later release, mixed SQL will be done by the engine internally if the property is true.

If prepared statements are used, the corresponding SQL queries will contain bound values like ? characters instead of concrete values.

Restrictions on using Prepared Statements

Note that prepared statements are explicitly ignored for the following scenarios:

  1. In case a data cluster is configured, the data slot is configured with ignorePreparedStatements=true and a query uses this slot.
  2. If one of the following database fields are used in a query:

Optimizing the Usage of Prepared Statements by using Oracle's Adaptive Cursor Sharing

Adaptive Cursor Sharing can be used for optimization as it allows the server to compare the effectiveness of execution plans between executions with different bind variable values. Please refer to Oracle's online documentation for details on the Oracle cursor sharing option, e.g. Improving Real-World Performance Through Cursor Sharing of the Oracle Tuning guide.

Histogram creation

Using Oracle's adaptive cursor sharing requires a histogram creation. For example use the following commands to force a histogram creation on column STRING_VALUE of table DATA_VALUE:

EXEC DBMS_STATS.gather_table_stats(NULL, 'DATA_VALUE', method_opt=>'FOR COLUMNS SIZE 254 STRING_VALUE', cascade=>TRUE);

SELECT column_name, histogram FROM user_tab_cols WHERE  table_name = 'DATA_VALUE';

Note that the table must not be empty. It has to contain values to force histogram creation.

The following table shows the created histogram for column STRING_VALUE:

COLUMN_NAME                    HISTOGRAM
------------------------------ ---------------
OID                            NONE
MODEL                          NONE
DATA                           NONE
STRING_VALUE                   HEIGHT BALANCED
NUMBER_VALUE                   NONE
DOUBLE_VALUE                   NONE
TYPE_KEY                       NONE
PROCESSINSTANCE                NONE

If the current schema is different from the schema the histogram table should be created in, use the following syntax:

EXEC DBMS_STATS.gather_table_stats('CARNOT', 'DATA_VALUE', method_opt=>'FOR COLUMNS SIZE 254 STRING_VALUE', cascade=>TRUE);

SELECT column_name, histogram FROM ALL_TAB_COLUMNS WHERE owner='CARNOT' AND  table_name = 'DATA_VALUE';

Query Examples

In the first example, we run a query that uses a bind variable. The table DATA_VALUE has one entry that matches data% for string_value. The execution plan shows that the query has used the index.

SQL> VARIABLE l_string_value VARCHAR2(128 CHAR);
SQL> EXEC :l_string_value := 'data%';

PL/SQL procedure successfully completed.

SQL>
SQL> SELECT MAX(oid) FROM carnot.data_value WHERE string_value LIKE :l_string_value;

  MAX(OID)
----------
    100006

SQL> SELECT * FROM TABLE(DBMS_XPLAN.display_cursor);

PLAN_TABLE_OUTPUT
---------------------------------------------------------------------------------------------------------
SQL_ID  gzms5fbsmqwmp, child number 1
-------------------------------------
SELECT MAX(oid) FROM carnot.data_value WHERE string_value LIKE
:l_string_value

Plan hash value: 1688810086

---------------------------------------------------------------------------------------------------
| Id  | Operation                    | Name               | Rows  | Bytes | Cost (%CPU)| Time     |
---------------------------------------------------------------------------------------------------
|   0 | SELECT STATEMENT             |                    |       |       |     3 (100)|          |

PLAN_TABLE_OUTPUT
---------------------------------------------------------------------------------------------------------
|   1 |  SORT AGGREGATE              |                    |     1 |    23 |            |          |
|   2 |   TABLE ACCESS BY INDEX ROWID| DATA_VALUE         |     1 |    23 |     3   (0)| 00:00:01 |
|*  3 |    INDEX RANGE SCAN          | DATA_VALUES_INDEX5 |     1 |       |     2   (0)| 00:00:01 |
---------------------------------------------------------------------------------------------------

Now we run a query that uses a bind variable containing a value that matches multiple entries. The optimizer selects an execution plan that uses full table access.

VARIABLE l_string_value VARCHAR2(128 CHAR);
EXEC :l_string_value := '%test2%';

SELECT MAX(oid) FROM carnot.data_value WHERE string_value LIKE :l_string_value;
SELECT * FROM TABLE(DBMS_XPLAN.display_cursor);

SQL> VARIABLE l_string_value VARCHAR2(128 CHAR);
SQL> EXEC :l_string_value := '%test2%';

PL/SQL procedure successfully completed.

SQL>
SQL> SELECT MAX(oid) FROM carnot.data_value WHERE string_value LIKE :l_string_value;

  MAX(OID)
----------
     99986

SQL> SELECT * FROM TABLE(DBMS_XPLAN.display_cursor);

PLAN_TABLE_OUTPUT
----------------------------------------------------------------------------------------------------
SQL_ID  gzms5fbsmqwmp, child number 3
-------------------------------------
SELECT MAX(oid) FROM carnot.data_value WHERE string_value LIKE
:l_string_value

Plan hash value: 3200669370

---------------------------------------------------------------------------------
| Id  | Operation          | Name       | Rows  | Bytes | Cost (%CPU)| Time     |
---------------------------------------------------------------------------------
|   0 | SELECT STATEMENT   |            |       |       |   141 (100)|          |

PLAN_TABLE_OUTPUT
----------------------------------------------------------------------------------------------------
|   1 |  SORT AGGREGATE    |            |     1 |    23 |            |          |
|*  2 |   TABLE ACCESS FULL| DATA_VALUE |  5000 |   112K|   141   (2)| 00:00:02 |
---------------------------------------------------------------------------------

Now we run the following query:

SQL> SELECT sql_id, child_number, is_bind_sensitive, is_bind_aware
  2  FROM   v$sql
  3  WHERE  sql_text = 'SELECT MAX(oid) FROM carnot.data_value WHERE string_value LIKE :l_string_value';

Having a look a the output, we can see that, as column is_bind_sensitive displays value Y, this query requires different execution plans depending on the bind variable value. Column is_bind_aware shows that a more efficient execution plan for this query is used.

SQL_ID        CHILD_NUMBER I I
------------- ------------ - -
gzms5fbsmqwmp            0 Y N
gzms5fbsmqwmp            1 Y Y
gzms5fbsmqwmp            2 Y Y
gzms5fbsmqwmp            3 Y Y

Optimizing the Usage of Prepared Statements with a DB2 Profile

You can optimize the usage of prepared statements with a DB2 profile, which is described in this section. With this setup, a new execution plan is created for each executed statement specified in the profile.

Please refer to the IBM Knowledge Center site Using the REOPT bind option with input variables in complex queries and to article Improve SQL execution times using the DB2 statement concentrator feature and the DB2 statement reoptimization feature for details on using the DB2 statement re-optimization feature.

To create and setup a profile for optimization:

  1. Create a profile xml file, e.g. profile1.xml, with the following content:
    <?xml version="1.0" encoding="UTF-8"?>
    <OPTPROFILE VERSION="9.7.2.0">
       <STMTPROFILE ID="REOPT_QUERY">
          <STMTKEY>
             <![CDATA[SELECT MAX(oid) FROM data_value WHERE string_value LIKE ?]]>
          </STMTKEY>
          <OPTGUIDELINES>
             <REOPT VALUE="ALWAYS"/>
          </OPTGUIDELINES>
       </STMTPROFILE>
    </OPTPROFILE>
    
  2. To activate the use of optimization profiles in DB2 the variable DB2_OPTPROFILE has to be set to YES. After the change the DB2 server has to be restarted:
    db2set DB2_OPTPROFILE=YES
    db2stop
    db2start
  3. Use the SYSPROC.SYSINSTALLOBJECTS procedure to create the SYSTOOLS.OPT_PROFILE table for storing the optimization profile:
    db2 CALL SYSPROC.SYSINSTALLOBJECTS('OPT_PROFILES', 'C', CAST(NULL AS VARCHAR(128)), CAST(NULL AS VARCHAR(128)))
  4. Create a file, e.g. profile1.del, with the following content:
    "DEMO","PROFILE1","profile1.xml"
  5. Import the optimization profile in table SYSTOOLS.OPT_PROFILE:
    db2 IMPORT FROM profile1.del OF DEL MODIFIED BY LOBSINFILE INSERT INTO SYSTOOLS.OPT_PROFILE
  6. Flush the DB2 optimization profile cache to activate changes to the optimization profiles:
    db2 FLUSH OPTIMIZATION PROFILE CACHE ALL;
  7. Adjust the jdbc URL string of the data source and add property optimizationProfile:
    jdbc:db2://localhost:50000/ipp:optimizationProfile=DEMO.PROFILE1;