I am curious what you are using CTE’s for?

Please post a reply with the top things you use CTE’s for.

Thanks!!

These queries will be using that database that was set up in a previous posting on the CTE_DEMO Sample Database, if you haven’t set up the sample database, download it and set it up now.

The concept of data paging is taking a large set of results and breaking it up into smaller pages of result sets. For instance if you search your favorite internet search engine for a term, you are usually shown results 1 to 20 of X where X is a very large number.  For instance page 1 is results 1 to 20 of 652,443 results.

CTE Data Paging

declare @pageNum as int;
declare @pageSize as int;
set @pageNum = 2;
set @pageSize = 10;

;WITH TablesAndColumns AS
(
SELECT OBJECT_NAME(sc.object_id) AS TableName,
 name AS ColumnName,
 ROW_NUMBER() OVER (ORDER BY OBJECT_NAME(sc.object_id)) AS RowNum
 FROM sys.columns sc
)
SELECT *
 FROM TablesAndColumns
 WHERE RowNum BETWEEN (@pageNum - 1) * @pageSize + 1
 AND @pageNum * @pageSize ;

Now lets wrap it in a procedure to simplify things.

CREATE PROCEDURE TablesAndColumnsPager @pageNum int, @pageSize int
AS
BEGIN
SET NOCOUNT ON;

;WITH TablesAndColumns AS
  (
SELECT OBJECT_NAME(sc.object_id) AS TableName,
  name AS ColumnName,
  ROW_NUMBER() OVER (ORDER BY OBJECT_NAME(sc.object_id)) AS RowNum
  FROM sys.columns sc
  )
  SELECT *
  FROM TablesAndColumns
  WHERE RowNum BETWEEN (@pageNum - 1) * @pageSize + 1
  AND @pageNum * @pageSize ;
END

Then to call the paging function it is a bit cleaner.

exec TablesAndColumnsPager 1, 10;
exec TablesAndColumnsPager 2, 10;
exec TablesAndColumnsPager 3, 10;

Which produces the following output.

Related Links:

• My Book on Common Table Expressions
• cte_demo sample database
• TSQL 2012 – OFFSET and FETCH

Common Table Expressions Book

If you enjoyed this posting, and want to learn more about common table expressions, please take a look at my book on CTE’s at Amazon.com. The book is titled Common Table Expressions - Joes 2 Pros® - A CTE Tutorial on Performance, Stored Procedures, Recursion, Nesting and the use of Multiple CTEs.

• June 14th - CTE Data Paging
• June 13th - Query To Simulate CPU Load
• June 12th – Nested CTE’s Gone Wild – The Video
• June 11th - Nested CTEs
• June 10th - Database Health Reports – Seven Hundred Downloads So Far
• June 10th - Multiple CTEs in a Query
• June 9th - Expanding on Recursive Dates CTE

I think my favorite for the week was the Seven Hundredth download of the Database Health Reports project. This project is really starting to take off. I am glad that people are using it and getting value out of it.

A successful week 2 of Common Table Expression month at SteveStedman.com

These queries will be using that database that was set up in a previous posting on the CTE_DEMO Sample Database, if you haven’t set up the sample database, download it and set it up now.

The concept of data paging is taking a large set of results and breaking it up into smaller pages of result sets. For instance if you search your favorite internet search engine for a term, you are usually shown results 1 to 20 of X where X is a very large number.  For instance page 1 is results 1 to 20 of 652,443 results.

Data Paging – Before SQL 2012

First lets take a look at a query with no data paging Shows all tables on this database, and all columns, and returns many rows.

SELECT OBJECT_NAME(sc.object_id) as TableName,
       name as ColumnName
  FROM sys.columns sc
ORDER BY OBJECT_NAME(sc.object_id);

When this is run we get 702 rows in the result set:

Now to introduce data paging with a CTE,  Here we have a CTE example to get data paging. This example assumes that the page size is 10 the first page would display the first 10 rows, the the second page would display rows 11 to 20, and the third page would display rows 21 to 30. Here we are using the ROW_NUMBER() function with the OVER clause for the windowing functionality to get the row number from the output. In a non-CTE query you can’t use ROW_NUMBER() in the where statement, but if it is wrapped in a CTE, and given a column alias we can then use it in the WHERE clause selecting from the CTE.

declare @pageNum as int;
declare @pageSize as int;
set @pageNum = 2;
set @pageSize = 10;

;WITH TablesAndColumns AS
(
SELECT OBJECT_NAME(sc.object_id) AS TableName,
 name AS ColumnName,
 ROW_NUMBER() OVER (ORDER BY OBJECT_NAME(sc.object_id)) AS RowNum
 FROM sys.columns sc
)
SELECT *
 FROM TablesAndColumns
 WHERE RowNum BETWEEN (@pageNum - 1) * @pageSize + 1
 AND @pageNum * @pageSize ;

When we run this we get the following results for page 2:

Now if we change the @pageNum variable to be 3 we get the following:

SQL 2012 Data Paging

If you are using SQL Server 2012, there is a new option added to the SELECT statement, this is called OFFSET and FETCH. You have an option to do data paging without the CTE, the OFFSET and FETCH option allows you to do the following:

</pre>
declare @pageNum as int;
declare @pageSize as int;
set @pageNum = 2;
set @pageSize = 10;

SELECT OBJECT_NAME(sc.object_id) AS TableName,
name AS ColumnName
 FROM sys.columns sc
 ORDER BY TableName
OFFSET (@pageNum - 1) * @pageSize ROWS FETCH NEXT @pageSize ROWS ONLY;
<pre>

Which produces similar results for page 2 as the CTE version as shown here:

Which is Better?

The only correct answer is “It Depends”.  It depends on a number of things. First if you are writing code that needs to run on an older version of SQL Server than 2012, then you can’t use the  OFFSET and FETCH way 0f doing it, your best option is the CTE data paging. If you are running on only SQL Server 2012 or newer, then you have a choice. The size of your result set, the design of your tables, the number of JOINs in the query will all impact the overall performance. I would suggest if you are using SQL 2012 and you have a choice, code it both ways, and performance test it. See which runs best base on your specific query.

Related Links:

• My Book on Common Table Expressions
• cte_demo sample database
• TSQL 2012 – OFFSET and FETCH

Common Table Expressions Book

If you enjoyed this posting, and want to learn more about common table expressions, please take a look at my book on CTE’s at Amazon.com. The book is titled Common Table Expressions - Joes 2 Pros® - A CTE Tutorial on Performance, Stored Procedures, Recursion, Nesting and the use of Multiple CTEs.

Here is that I came up with for a plan. Keeping in mind the goal here is to write queries that will use up as much CPU as possible. This is counter-intuitive, completely against everything that I practice on a daily basis, but here goes.

First create a table with poor design. Using UNIQUEIDENTIFIERS for a primary key and a foreign key (parent_id) is probably ugly enough.

CREATE TABLE SplitThrash
(
 id UNIQUEIDENTIFIER default newid(),
 parent_id UNIQUEIDENTIFIER default newid(),
 name VARCHAR(50) default cast(newid() as varchar(50))
);

Next we fill the table up with lots and lots of rows, specifically 1,000,000 rows, remember here the goal is to simulate CPU load. If this isn’t enough I often times run this script several times. Keep in mind the GO statement followed by a number says to execute the batch that many times.

SET NOCOUNT ON;
INSERT INTO SplitThrash DEFAULT VALUES;
GO  1000000

Next, this part makes me just feel nasty. Create a CLUSTERED index on the table that we just filled up, and cluster on BOTH columns that were UNIQUEIDENTIFIERS.

CREATE CLUSTERED INDEX [ClusteredSplitThrash] ON [dbo].[SplitThrash]
(
 [id] ASC,
 [parent_id] ASC
);

At this point is is a bit ugly, but it still doesn’t use much memory. You are probably wondering why I called the table split thrash. I gave it this name so that updating the UNIQUEIDENTIFER would cause as many page splits or new page allocations as possible. So we update the parent_id which is part of the clustered index

</pre>
UPDATE SplitThrash
SET parent_id = newid(), id = newid();
GO 100
<pre>

This update statement causes chaos in the page structure for the table as updating the unique identifiers causes quite a bit of processor work.

On my wimpy VM for this development environment I need to repeat this entire process creating 4 or 5 tables, and doing the update in 4 or 5 SSMS windows in order to use up all of the CPU on the database.

Don’t try this on a production system, but it is a great test to run on a development server.

See Also:

If you like this you might also like my article on how to use up all of your memory with a common table expression.

These queries will be using that database that was set up in a previous posting on the CTE_DEMO Sample Database, if you haven’t set up the sample database, download it and set it up now.

How to use up all of your SQL Server available memory with a single CTE query

The article could be named, “How to use up all of your SQL Server available memory with a single CTE query.” Another name for the article could just be “SQL Server Bug Report” depending on how you look at it.

When presenting unleashing Common Table Expressions at SQL Saturday a while back, I was asked a couple of great questions that I didn’t know the answer to. So I did the research and tracked it down:

1. How many levels of recursion can you have in a CTE?

2. How many levels of nesting can you have in a CTE?

So I started doing the research and doing some testing to figure it out.

How many levels of recursion can you have in a CTE?

This is the easier one to answer between the two questions. The answer is, you can have more levels of recursion that you would ever need for standard recursion. I have tested CTEs with up to 1 million levels of recursion, and the have performed pretty well. If you are writing queries that need more than 1 million levels of recursion, perhaps you should take a look at a different approach.

How many levels of nesting can you have in a CTE?

This is where it gets really interesting. SQL Server does a really good job with overall performance on CTEs, but where it completely breaks down is on deep nested CTE queries. By deep nested I mean more than a thousand or two thousand. To answer the question of how many levels can you have, I would answer this as you can nest more than you would ever really need. If you really need more than a thousand levels of nesting in a CTE, you might want to rethink your approach.

On SQL Server 2005 the limit is 255. But in SQL Server 2008 and newer this limit was extended, and appears to not have a fixed limit, rather the limit is based on the amount of memory available for the query to use.

So to build the crazy CTE, I used excel to build out several thousands of rows of nested CTEs. One calling another, calling the next, and so on. What I found was that this was a very easy way to use up almost all the memory on your database. DO NOT TRY THIS ON A PRODUCTION DATABASE. The reason that DBAs and developers have test or development databases is to play around with things that may be dangerous on a production server. This is one that you could try on a test server.

The other interesting thing that the query does when it uses up all the memory, somehow it dumps some of the connections that are currently active on the SQL Server at that point.

Watch the video to find out how a nested CTE query could use up all of the memory on your SQL Server.

If anyone has a test server with 512GB, or 1TB of available, memory, I would love to see how this type of query performs. Give it a try and let me know.

You can download the script NestedCTE.zip here. The zip file contains 2 files, one called CTE.sql, and one called CTE2500.sql.  The 2500 version is the one that I used in the demo, and the CTE.sql has 32767 nestings.  I would love to see the bigger on run on a server with a TB or more of memory to see how it performs.

Related Links:

• My Book on Common Table Expressions
  
• cte_demo sample database

Common Table Expressions Book

If you enjoyed this posting, and want to learn more about common table expressions, please take a look at my book on CTE’s at Amazon.com. The book is titled Common Table Expressions - Joes 2 Pros® - A CTE Tutorial on Performance, Stored Procedures, Recursion, Nesting and the use of Multiple CTEs.

These queries will be using that database that was set up in a previous posting on the CTE_DEMO Sample Database, if you haven’t set up the sample database, download it and set it up now.

Russian Nesting Dolls

If you have ever seen the Matryoshka dolls known as the Russian nesting dolls or babushka dolls they are very interesting. They start with a large wooden doll that when opened contains a slightly smaller doll, then inside of that one an even smaller doll. These dolls often have five to eight dolls nested inside of each other.

The interesting thing about these dolls is that they have one specific way that they can all fit together; the smallest doll must be the first one put inside of the next smallest. If you were to lose one of the medium dolls, they could still be placed together; there would just be some extra space inside.

Nested CTEs often remind me of the Russian nesting dolls. Think of the first CTE in a multiple CTE query as the smallest doll. That first CTE can’t reference any other CTEs, much like the smallest doll can’t fit any other dolls inside of it.

Think of the second CTE in a multiple CTE query as the second smallest doll in the Russian dolls. It can only have one doll fit inside of it. CTEs are similar in that the only CTE that can be referenced by the second CTE in a multiple CTE query is the first CTE.

The third CTE is like the third smallest Russian doll. The third smallest doll can only have the smallest or second smallest doll placed inside of it. With the third CTE it can only access the first or the second CTE declared, and not any of the CTEs declared later.

The same concept applies for all of the CTEs in a nested CTE query. Any CTE query can access the CTEs declared prior to that one, but not the ones after.

Nested CTEs

Take an example of a multiple CTE query with 4 CTEs as follows with the … being replaced with a T-SQL query.

;WITH CTE1 AS
  ( .... ),

CTE2 AS
  ( .... ),

CTE3 AS
  ( .... ),

CTE4 AS
  ( .... )

SELECT *
    FROM CTE4;

Related Links:

• My Book on Common Table Expressions
  
• cte_demo sample database

Common Table Expressions Book

If you enjoyed this posting, and want to learn more about common table expressions, please take a look at my book on CTE’s at Amazon.com. The book is titled Common Table Expressions - Joes 2 Pros® - A CTE Tutorial on Performance, Stored Procedures, Recursion, Nesting and the use of Multiple CTEs.

First I would like to say Thank You to everyone using it so far.

Beta 6 introduced a few new features, but mostly focused on overall stability and bug fixing.

• New Inefficient Indexes report added. Useful to find indexes that aren’t being used.
• Scrollbar added to the historic waits report to allow for easier browsing of wait stats.
• Overall improved CTRL-C and CTRL-A clipboard support.
• Adding a setting dialog to consolidate some of the basic settings.
• Added a threshold setting (to the new setting dialog) to set the threshold to search on the big clustered indexes report.
• Added sorting to all the columns in all the grids.
• Added the application name into the connect string.
• Improved the call to check for updates when the application starts.
• Adding another warning to the database overview panel for huge log files.
• Added additional wait type documentation.
• Improved error handling.
• Added the update every X seconds functionality to the database connections report, this includes a setting in the setting dialog to change the every X seconds interval. If you don’t want it to update, just set it to a very large number.
• New update interval settings for the database overview and server overview panels.
• Adding Historic Plan Cache Hit Ratio Chart to the Historic Reporting section.

Still my favorite part of the overall product is the historic wait time monitoring. Once historic monitoring is turned, it records data about what may having been causing problems at some point in time. This way when someone make a comment like “The database appeared slow yesterday at 4:25am”, you have the ability to go find out why it may have been slow.

On June 6th I received a nice comment on the Raving Fans page. This page was set up to share what people have to say about this application. Since the application is free of charge the only thing I ask is for someone to share what they find with the application:

Database Health Reports is a free application that I provide to the SQL Server community to help analyze and fix performance problems on your MS Sql Server.

These queries will be using that database that was set up in a previous posting on the CTE_DEMO Sample Database, if you haven’t set up the sample database, download it and set it up now.

One of the questions that comes up when discussing CTE Scope is can I do multiple CTEs. The answer is yes you can do multiple CTEs but the scope is is still constrained to a single SQL Statement.

Multiple CTEs in a Single Statement

Lets take a look when multiple CTEs in a single statement might be useful. Take the example of trying to generate random names in order to fill up tables in a test database to simulate a full database during the development phase of a new project. This example is explained in much further detail in chapter 6 in my Book on Common Table Expressions. Lets take a look here at this example from my SQL Saturday presentation. The following is a CTE that returns a list of first names.

;WITH Fnames (Name) AS
(

SELECT 'John'
 UNION
 SELECT 'Mary'
 UNION
 SELECT 'Bill'

)
SELECT F.Name FirstName
 FROM Fnames F;

When run we see the following:

Now if we want to expand on that we can add a second CTE for last names and CROSS JOIN the results of the two CTEs. A refresher on CROSS JOIN, the CROSS JOIN takes every row from one side and matches it up with every rows for the other side, with no ON clause. This effectively creates a Cartesian product of the two tables, or joins up every possible combination.

;WITH Fnames (Name) AS
(
SELECT 'John'
 UNION
 SELECT 'Mary'
 UNION
 SELECT 'Bill'
 ),

Lnames (Name) AS
(
SELECT 'Smith'
 UNION
 SELECT 'Gibb'
 UNION
 SELECT 'Jones'
 )

SELECT F.Name AS FirstName, L.Name AS LastName
 FROM Fnames AS F
 CROSS JOIN Lnames AS L;

Here you will notice that the initial CTE is separated from the new CTE with a comma, and that the second CTE doesn’t start with the WITH keyword. Then in the final SELECT statement it just references both of the CTE statements declared above. When run we see the following results.

Here you can see that every first name is joined up with every last name, with two input tables of 3 rows each the result set has 3×3 or 9 rows of output.

You could also add a third CTE with middle names, and possibly a 4th with login names, and very quickly end up with a very large result set multiplying each time there is another cross join added.

Related Links:

• My Book on Common Table Expressions
  
• cte_demo sample database
• CTE Scope

Common Table Expressions Book

If you enjoyed this posting, and want to learn more about common table expressions, please take a look at my book on CTE’s at Amazon.com. The book is titled Common Table Expressions - Joes 2 Pros® - A CTE Tutorial on Performance, Stored Procedures, Recursion, Nesting and the use of Multiple CTEs.

These queries will be using that database that was set up in a previous posting on the CTE_DEMO Sample Database, if you haven’t set up the sample database, download it and set it up now.

Recursive Review

Yesterday’s topic was the introduction to recursive CTE’s.

In the introduction to recursive CTE’s we covered the declaration of the CTE, the Anchor Query which starts the recursive process, the Recursive Query which continues the recursion, and the Query that calls the CTE.

Dates CTE Review

In the following picture, the CTE is named Dates, the anchor query start out by just selecting January 1st of 2013. Next the recursive part selects CalendarDate from the Dates CTE and it adds a single day to it. This all continues recursively as long as the date is less than January 1st 2014.

Adding In More Details

For today’s example the CTE will be exactly the same as yesterday, but the query calling it is different. The part highlighted below shows the difference from yesterday.

When run we get the following output:

Next I borrow part of the date formatting from @DevonDBA from his post at http://devondba.blogspot.co.uk/2013/06/create-table-of-dates-showing-holidays.html, and slightly modify it as shown below. Thanks Martyn.

Now we can see if the day that we are looking at is a weekday or a weekend as shown here:

I hope you are enjoying CTE Month at SteveStedman.com.

Related Links:

• My Book on Common Table Expressions
  
• cte_demo sample database
• Recursive CTE for Dates In A Year
• Introduction to Recursive CTEs

Common Table Expressions Book

If you enjoyed this posting, and want to learn more about common table expressions, please take a look at my book on CTE’s at Amazon.com. The book is titled Common Table Expressions - Joes 2 Pros® - A CTE Tutorial on Performance, Stored Procedures, Recursion, Nesting and the use of Multiple CTEs.