How to Tune CTE “WITH” SQL statement?

CTE stands for common table expression. A CTE allows you to define a temporary named result set that available temporarily in the execution scope of a statement such as SELECT, INSERT, UPDATE, DELETE.

The following shows an example of a CTE in MySQL:

WITH
    cte1 AS (SELECT a, b FROM table1),
    cte2 AS (SELECT c, d FROM table2)
SELECT b, d FROM cte1 JOIN cte2
WHERE cte1.a = cte2.c;

The following is an example shows a SQL statement with CTE WITH expression. The retrieve records from EMPLOYEE that EMP_GRADE and EMP_DEPT have to satisfy the CTE selection result.

with DT1 as
(SELECT
                     EMP_GRADE GRADE,EMP_DEPT DEPT
  FROM     DEPARTMENT,
                     EMP_SUBSIDIARY
 WHERE   DPT_ID = EMP_DEPT
         AND   DPT_AVG_SALARY<500000
         AND   EMP_DEPT<‘D’
         and     EMP_SALARY<1200000
)
select * from  EMPLOYEE where (EMP_GRADE,EMP_DEPT) in
(select GRADE,DEPT from DT1)

Here the following are the query plan of this SQL, it takes 55.9 seconds to finish. The query shows a “Subquery2” with a Nested Loop from sub_emp_salary_inx to DEPARTMENT_PK.

I found the Rows=69606 of step 1 (1 Index Range Scan –  EMP_SUBSIDIARY –  sub_emp_salary_inx) is significant high, it is not reasonable for MySQL SQL optimizer to such path from EMP_SUBSIDIARY to DEPARTMENT. I believe that MySQL optimizer cannot do a good transitivity improvement DPT_ID.  So, I manually add a new condition as “and DPT_ID<‘D’“and a “group by 1,2” to narrow down the result set from CTE.

with DT1
           as (select   EMP_GRADE GRADE,
                                   EMP_DEPT  DEPT
                   from     DEPARTMENT,
                                   EMP_SUBSIDIARY
                 where    DPT_ID = EMP_DEPT
                                   and DPT_ID < ‘D’
                                   and DPT_AVG_SALARY < 500000
                                   and EMP_DEPT < ‘D’
                                   and EMP_SALARY < 1200000
                 group by 1,
                                    2)
  select *
  from         EMPLOYEE
  where      (EMP_GRADE,EMP_DEPT) in (select  GRADE,
                      DEPT
                     from     DT1)

Here is the query plan of the rewritten SQL and it is running faster. The new query plan shows correct driving path from DEPARTMENT to EMP_SUBSIDIARY, the estimated Rows now are closer to reality. There are two new steps of GROUP and DT1 (materialized) to narrow down the result set of CTE to future improve the performance.

This kind of rewrites can be achieved by Tosska SQL Tuning Expert for MySQL automatically, it shows that the rewrite is more than 2 times faster than the original SQL. There are some other rewrites shown in this screen with comparable results too.

https://tosska.com/tosska-sql-tuning-expert-tse-for-mysql-2/

How to Tune SQL Statement with multiple MAX() functions for Oracle?

Here the following is a very simple SQL statement with two Max() functions in select list.

select max(emp_salary),max(emp_id)
from employee;

Here the following are the query plans of this SQL, it takes 8.82 seconds to finish. The query shows a Full Table Scan of Employee table.

You can see that this SQL cannot utilize index scan even though the emp_id and emp_salary are indexed. If I change the SQL to select max(emp_salary) only like the following: select max(emp_salary) from employee;

The SQL will run much faster and the emp_salary index will be used.

In order to solve this problem, let me rewrite the SQL into the following syntax. I use two WITH (common table expression) to select each max() function independently and it fully utilize the index in each column.

WITH dt1
     AS (SELECT Max(emp_salary)
             FROM   employee),
     dt2
     AS (SELECT Max(emp_id)
             FROM   employee)
  SELECT *
  FROM   dt1,
                   dt2;

Here is the query plan of the rewritten SQL and the speed is 0.00 seconds which cannot be detected in our timing scale. The new query plan shows that two Index Full Scan (MIN/MAX) for each column are used now.

This kind of rewrite can be achieved by Tosska SQL Tuning Expert for Oracle automatically, it shows that the rewrite is much faster than the original SQL

https://tosska.com/tosska-sql-tuning-expert-pro-tse-pro-for-oracle/

How to Tune “Not Exists” SQL statement?

The following is an example shows a SQL statement with “Not Exists” expression. The SQL retrieve records from emp_subsidiary that satisfy with the “Not  Exists” subquery.

select * from emp_subsidiary sub
where not (exists
(select ‘x’ from employee emp
      where emp.emp_salary<1000
           and emp.emp_dept=sub.emp_dept ))
and sub.emp_grade<1200

Here the following is the query plan of this SQL, it takes 13.36 seconds to finish. The query shows a Nested Loop from emp_subsidiary to the “Materialized Subquery2” from a full table scan of employee.

I found the Rows=2950038 of “Full Table Scan of employee” of step 2 is significantly high to constitute the materialized subquery2(view). In order to reduce the actual number of rows scan of this materialized subquery2(view). I moved the subquery of “Not Exists” to a CTE “WITH” statement and added a “group by 1” to reduce the result set of the CTE in the following.

with DT1
          as (select     emp.emp_dept
                from         employee emp
                where      emp.emp_salary < 1000
                group by 1)
    select *
    from     emp_subsidiary sub
    where  not ( sub.emp_dept in (select emp_dept
                                                                     from   DT1) )
                    and sub.emp_grade < 1200

The following is the query plan of the rewritten SQL and it takes only 2.32 seconds to complete. The new query plan shows an “Index Range Scan” to the Employee table plus a “GROUP” operation to narrow down the result set of the materialized subquery2.

This kind of rewrites can be achieved by Tosska SQL Tuning Expert for MySQL automatically, it shows that the rewrite is more than 5 times faster than the original SQL. There are some other rewrites shown in this screen with even better performance, but they are more complicated in SQL syntax transformation and not suitable to discuss here in this short article.

https://tosska.com/tosska-sql-tuning-expert-tse-for-mysql-2/

How to Tune UPDATE SQL statement with IN subquery (I) ?

The following is an example shows an Update SQL statement with an “IN” subquery. It updates records from emp_subsidiary that satisfies the “IN” subquery conditions.

update emp_subsidiary set emp_name =‘Deleted Name’
where   emp_dept in
(select dpt_id from department
 where dpt_avg_salary<=6000);

Here the following is the query plan of this SQL, it takes 7.55 seconds to finish the update. The query shows an attached_subqueries attached to a Full Index Scan of emp_subsidiary table. It means that the 295344 rows in emp_subsidiary is going to check the subquery’s conditions one by one.

Let me rewrite the SQL into the following join update syntax.

update emp_subsidiary e1, department d1
set    e1.emp_name=‘Deleted Name’
where  e1.emp_dept = d1.dpt_id
       and d1.dpt_avg_salary <= 6000

The following is the query plan of the rewritten SQL and it takes only 1.22 seconds to complete. The new query plan shows a “Nested Loop” from Department table to Emp_subsidiary table, due to the condition “dpt_avg_salary <= 6000” has been executed before it is going to loop the Emp_subsidiary table, it saved a lot of unnecessary time to detect every record in Emp_subsidiary table.

This kind of rewrites can be achieved by Tosska SQL Tuning Expert for MySQL automatically, it shows that the rewrite is more than 6 times faster than the original SQL.

https://tosska.com/tosska-sql-tuning-expert-tse-for-mysql-2/

What Optimization in SQL is, and Why It’s Necessary for DBAs

SQL statements or queries are designed to retrieve information from the database. A user can achieve the same results through optimization in SQL; using a tuned query is especially useful from an execution perspective. 

Tuning a database is a vital step in organizing and accessing the information in a database. Performance tuning in SQL requires streamlining and homogenizing the environment of a database and the files in it. This simplifies the way users access data in a big way. 

Why Companies Need to Consider Optimization in SQL

Several organizations own databases, but not all of them hire IT staff knowledgeable in the ways of optimization in SQL. Only professionals who have tuning skills and experience along with insider information about the working of databases should do this. 

In case your company has a database but it hasn’t undergone performance tuning, you might encounter inadequate responses to queries and face unnecessary complications when handling data. Don’t let your efficiency get affected because of something avoidable like this! 

Performance Tuning in SQL: What It Involves

Tasks related to performance tuning include optimization in SQL database, creating and managing indexes, and other related tasks to maintain or improve database performance. The goal of MySQL query optimization is to increase the speed and brevity of query responses and to simplify data retrieval. 

Let’s look at three major reasons why companies need to take performance tuning and seriously – 

1. To enhance the rate of data fetching options

If your database lacks optimization, then fetching data can get slower with increasing data loads. Optimizing queries enables users to create indexes and eradicate issues that may be slowing down data retrieval. After all, it can get quite frustrating for your employees to wait for the database to perform its operations, which can pass on to customers forced to wait for the same.

2. To refrain from coding loops

Making your database go through a coding loop is akin to hammering it repeatedly. That’s because the same query is executed several times when it is placed in a loop. However, once you remove the query from the loop, you will experience a definite surge in performance because the query is run only once rather than going through multiple iterations. 

3. To increase the performance of your SQL statements 

Query tuning in SQL includes changing previous query patterns and habits that were affecting the speed of data storage and retrieval. For example, the use of SELECT is reduced by opting for separate column declaration and eliminating correlated subqueries. Queries are also simplified by obviating temporary tables at times, aside from many other techniques of optimization in SQL

Your database will be able to manage much more data after the application of all these improvements as these will increase its efficiency, making it scalable as well. Once your database has scalability, it also overcomes lower performance and ensures user satisfaction in terms of experience. 
If you require professional tools to manage MySQL query optimization and tuning, then Tosska can help you. Tosska provides highly intuitive tools that can simplify query tuning beyond your imagination, and it does this with the help of innovative AI technologies. Contact us today to learn more about our range of query optimization products and services.

Improve Performance of SQL Query with these Great Techniques

SQL performance tuning can be an extremely complicated task, especially where data in huge quantities is concerned. When implementing queries to insert data in large quantities, even the tiniest of changes can have a major impact on performance – for better or for worse. 

If you are new to databases, you may be wondering what SQL performance tuning is and how you can use it with sound knowledge of the fundamentals and a few tricks up your sleeve. In this blog, you will find some fundamental techniques for SQL tuning to improve performance of SQL query being entered in the database. 

Techniques to Improve the Performance of SQL Queries

Consider these five tips and techniques to enhance database performance – 

Indexing

Indexes are quite effective in SQL tuning but are often overlooked at the time of development. Basically, an index is a data structure that can boost data retrieval speeds in tables by supplying quick random lookups and prompt access to requested records. This implies that once you have made an index, selecting, SQL performance monitoring, and sorting operations are faster. 

They are also useful in defining a primary key that will prevent other columns from having the same values. Naturally, database indexing is a vast topic that deserves its own set of blogs, but for now, it is important to understand that the aim is to index the larger columns intended for searching and ordering.

  • Keep in mind, however, that indexes must be modified after INSERT, UPDATE, and DELETE operations, which means they could actually worsen the performance of the database if your tables are receiving a large number of these commands. 
  • Furthermore, Database Administrators usually discard their indexes before executing gigantic batch inserts involving millions of rows, to hasten the insertion process. Once the task is complete, they then create the indexes all over again. It is important to remember, in such cases, that when the indexes are dropped in this manner, it affects all the queries being executed in that table. Hence, to improve performance of SQL query, this approach is typically taken in certain situations that require a single sizable insertion.

Execution Plan Tool in SQL Server

This tool helps create indexes and it shows all the data retrieval techniques selected by the query optimizer. There are walkthroughs available that will help newcomers learn more about this tool.

  • If you are using the SQL Server Management Studio, you can fetch the execution plan by pressing on Ctrl+M to select the “Include Actual Execution Plan” option before executing your query. This leads to a third tab named “Execution Plan” that will show any missing indexes that it has detected.

Steer Clear of Coding Loops

Suppose you need to insert a thousand queries in your database in one go. In that case, you may be tempted to do it using a loop but you must, in fact, refrain from doing so. 

  • Instead, consider changing the snippets containing the loop in unique INSERT or UPDATE statements that have additional rows and values. 
  • Make sure that your WHERE clause avoids updating the stored value if it matches the existing value. Such a trivial optimization can dramatically improve performance of SQL query by updating only hundreds of rows instead of thousands.

Checking Whether a Record Exists 

This is a handy SQL optimization approach that concerns the use of EXISTS(). 

  • In case you want to know if a certain record is present in the database, make a preference for EXISTS() instead of COUNT(). That’s because EXISTS() can give you much better performance with more coherent code as it leaves the table the moment it gets the data it needs. On the other hand, COUNT() scans the whole table every single time, counting up each and every entry that matches your condition.

How You Need Tuning to Improve MySQL Database Performance

SQL performance tuning is about saving you a great deal of money by improving various aspects of database management. Be it changing inefficient queries or replacing unnecessarily large indexes, tuning your database can really help you regulate your expenses.  

SQL tuning will not only improve MySQL database performance, but it will also help you save multiple resources, whether you operate your database servers on-premises or migrate your data over to the cloud.

Performance Tuning – More Than Just Optimization of SQL Queries

A majority of database professionals mostly focus on ensuring uptime by keeping tabs on the allocation of resources such as storage and memory. These formed a sizable portion of database management until companies started moving their databases to enormous cloud resources like AWS and Azure. 

This gave importance to other aspects like performance tuning in SQL, which DBAs work on once they ensure that things are running somewhat smoothly. However, a lot of database professionals may have to tweak a SQL Server that they aren’t familiar with, and there’s not much help available for such instances. 

Identifying Problem Areas During Performance Tuning

It can get intensive trying to figure out all the issues and resolving them one after another. At first, your problems may not even involve the optimization of SQL queries, and you may begin your search at the user/session-level. 

What some professionals do at this point is, they listen to the users and ask questions such as – 

  • Are there any specific times when the application slows down, for example, when they create an extra ticket or open an active one?
  • Which data takes excessive time to render?
  • How long does saving a record take? (a specific answer may be preferable)

Improve MySQL Database Performance in SQL Server with These Tools

Fortunately, SQL Server has multiple features that can help you improve MySQL database performance with greater convenience. Some of them are – 

Plan Guides

Plan guides enable you to adjust the way SQL Server executes a query, affecting performance. These are especially useful in case of queries written by another vendor, where some experts may not be willing to change them right away, as you can add a query hint to influence its operation. 

However, plan guides may get obsolete over time because the scenario may change after a while but they cannot. To overcome this, make a note of reviewing them from time to time. 

Query Store

This feature helps in the optimization of SQL queries too. It allows the user to determine which queries are taking up the most resources, and then tune them. Apparently, the Query Store feature is not enabled in some SQL databases because the user rarely needs it at first, but enabling it is easy.

Some DBAs are not aware of the Query Store, while others know of it but haven’t explored it enough, so the feature may as well be disabled. They can begin its use once they know how it works so that they can analyze various performance fluctuations caused because of modifications in the query plan.

Database Engine Tuning Advisor

This function examines workflows and suggests indexes or other strategies to improve MySQL database performance. On the other hand, don’t run this tool until your database has sufficient data stored, as the recommendations are more productive at that point. 

It won’t be as useful for a newer application that has, say, only a thousand rows in its tables, than after the app has expanded a bit.

How to Tune SQL with OR statements?

It is common that the performance is not good if a SQL statement with OR conditions. Let’s have an example show you how to tune those SQL statements in certain situations. Here is an example SQL that extract records from EMPLOYEE table if (emp_grade < 1050 or emp_id<730000). Emp_grade and emp_id are indexed and they are not null field.

select * from employee
where emp_grade < 1050 or emp_id<730000

You can see MySQL SQL Optimizer use an Index Merge of emps_grade_inx and employee_pk to process the SQL, the performance is not good as expected since the result set is quite big for sort_union operation. It takes more than 40 seconds to finish the data retrieval. Let me rewrite the OR condition into the following UNION ALL statement, please make sure the emp_grade and emp_id are not null column, otherwise it may generate error result. The rewrite is simple that the first part extract data with emp_grade<1050, the second part of the UNION ALL retrieve records that satisfied with emp_id<730000, but it is not retrieved in the first part of the UNION ALL.

select    *
from       employee
where    emp_grade < 1050
union all
select    *
from       employee
where    not ( emp_grade < 1050 )
                  and emp_id < 730000

Here the following is the query plan of this SQL, it takes 12.46 seconds to finish. The query shows two “Index Range Scan” of EMPLOYEE_PK and EMPS_GRADE_INX to the employee table.

This kind of rewrites can be achieved by Tosska SQL Tuning Expert for MySQL automatically, it shows that the rewrite is more than 3 times faster than the original SQL. There are some other rewrites shown in this screen with comparable results too.

https://tosska.com/tosska-sql-tuning-expert-tse-for-mysql-2/

How to Tune SQL with COUNT(*) statements ?

It is common that we used to count the number of records in a table.  You may encounter unexpected performance degradation in certain situations. Here is an example SQL that count number of records from EMPLOYEE table. There are number of indexes are built such as emp_id, emp_dept, emp_grade, emp_hire_date and etc….

SELECT COUNT(*)
   FROM EMPLOYEE;

You can see MySQL SQL Optimizer use a Full Index Scan of EMP_HIRE_DATE index, the performance is bad since unnecessary random reads is needed and it takes 3 minutes and 6 seconds to count a 3 million records in my computer.  I want to make use of Index Range Scan for specific index, let me rewrite the above SQL into the following syntax. If you know EMP_GRADE is indexed and it is not a nullable column, you can add a dummy condition EMP_GRADE>=’’. It fools MySQL SQL optimizer to consider using EMP_GRADE range index to retrieve the records and it is successfully generate a new plan in the following:

select    COUNT(*)
from       EMPLOYEE
where    EMP_GRADE >=  ‘ ‘

Here the following is the query plan of this SQL, it takes 2.6 seconds to finish. The query shows an “Index Range Scan” of employee table.

This kind of rewrites can be achieved by Tosska SQL Tuning Expert for MySQL automatically, it shows that the rewrite is almost 71 times faster than the original SQL. There are some other rewrites shown in this screen with comparable results too.

https://tosska.com/tosska-sql-tuning-expert-tse-for-mysql-2/

How to Tune substr(emp_name,5,4) SQL Statement?

There may be some business requirements that need to compare certain part of a column as a data retrieval criteria. Here is an example SQL that retrieves data from EMPLOYEE table employee’s name with a string pattern “Acco” start from 5 character of the emp_name.

select    *
  from    employee
where    substr(emp_name,5,4)=‘Acco’

Here the following are the query plans of this SQL, it takes 17 seconds to finish. The query shows a “Full Table Scan Employee”  

You can see that this SQL cannot utilize index scan even the emp_name is indexed field. Let me add a “Force Index(emp_name_inx)“ hints to the SQL and hope it can help MySQL SQL optimizer to use index scan, but it fails to enable the index scan anyway, so I add one more dummy condition emp_name >= ‘ ‘ , it is an always true condition that emp_name should be greater or equal to a smallest empty character.

select    *
from       employee force index(emp_name_inx)
where    substr(emp_name,5,4) = ‘Acco’
                 and emp_name >= ‘ ‘

Here is the query plan of the rewritten SQL and it is running faster. The new query plan shows that an Index Range Scan is used now.

This kind of rewrite can be achieved by Tosska SQL Tuning Expert for MySQL automatically, it shows that the rewrite is almost 6 times faster than the original SQL.

https://tosska.com/tosska-sql-tuning-expert-tse-for-mysql-2/