My SQL Server is Out Of Disk Space

As the DBA, you get a call from the IT Team who monitors disk space and they tell you the following:

“One of our server that has SQL Server running on it is running low on disk space. It has gone from around 60% disk utilization to 95% disk utilization just overnight. Whats the problem?”

In this case it happens that this is a server that you have never seen before. Someone installed a SQL Server to test out something product along the way, they never told the DBA about it, and suddenly this sever has become an important part of the company internal tools.

Sound familiar?  I know I have been hit with this before.

 

Right off, you have never seen this server before, you have no established baseline to compare it to, and you need to find out why it is running out of disk space rapidly. For me, some of the first things that would cross my mind would be:

  • Has anyone installed a new database or product new on this server recently?
  • Are any of the databases growing rapidly?
  • Are backup files being saved locally?
  • Are backups being run at all?
  • How big are each databases data files and log files?
  • Is there something outside of SQL Server taking up space?

Before I start troubleshooting I am going to just remote desktop to the server and double check the disk space myself to be sure that someone hasn’t misinterpreted the disk utilization.

DiskSpace1

 

At first glance the free disk space is 4.39GB out of 63GB.

DiskSpace2

 

 

As I am thinking about where to look first, the disk space drops to 2.28GB free.  Now I am very concerned.  I know that the C: Drive is filling up, this sever only has a C: drive, and if the Windows operating system runs out of disk space on the boot drive C:, then the server will crash, and often times will no longer boot.  Now things are serious, and must be dealt with quickly.

Now to start hunting, I could dig out several dozen of my favorite trouble shooting queries, or I could just run Database Health Monitor to start troubleshooting the issue. Database Health Monitor is my quick choice to track down problems like this.

I start up Database Health Monitor and connect to the SQL Server that is running out of space, to start looking around.

Initial View

 

Not immediately jumps out.  I see 12 databases on this SQL Server, and I see that the PerformanceTrouble database is using more CPU than any other database on the system.  Now to start looking at individual databases. The first database listed is called BadDB.  For this demo I am using a SQL Server that I use to test the Database Health Monitor Application, and I tend to use names that are descriptive of what the database is intended to do.

FirstDatabase

 

From this database I can see a few red flags. Well, we learn that the Database and Log files all exists on the C: Drive, and that this could cause the database to run out of disk space if the database grows, but this database isn’t very big, its only a few Megabytes, so I am not worried here.

Next I take a look at the Real Time Reports for Backup Sizes:

BackupSizes

 

For this database I can see that it has grow over the last year from around 2 megabytes to around 10 megabytes. All really small.  I also notice from the File names that the backups are being saved on the C: drive too.

As I click through the rest of the databases I see very similar results, its not great, but none of them are very big until I get to the database called PerformanceTrouble. Database Health Monitor shows this overview.

ProblemDatabase

 

Now this page points out a few things.  In bright red, there is a notice that shows the amount of log space allocated and being used by this database is around 24GB, for a database that is only 341MB in size.  This is an indication that backups are not being run, or that backups are not being run often enough, for instance log backups.  On this same page, I can see that the last full backup was run today at 5:00am.  I then click through the rest of the databases and nothing jumps out anywhere near this problem.

At this point I have spent about 3 minutes running Database Health Monitor, and I have found one problem that is using up about 33% of our entire hard drive.

 

How do we fix this issue?

First if I just doubleclick the Large Log File warning at the bottom of the application, my browser will open with one solution. There is a solution presented that looks a lot like this:

First we run this query to get the names of the log files.


-- check the size of the files.
SELECT size / 128.0 as sizeMB, name
FROM sys.database_files;

LogFileSizing

From that query we see that the name of the log file is PerformanceTrouble_log, the script below would get updated that your actual database name and log file name. The following script will shrink the log file for this database to around 1mb.

WARNING: Running this script will disconnect all other users from the database. Only run this on a production database if you have no other option.

-- Truncate the log by changing the database recovery model to SIMPLE.
ALTER DATABASE PerformanceTrouble SET RECOVERY SIMPLE;
GO
-- Shrink the truncated log file to 1 MB.
DBCC SHRINKFILE (PerformanceTrouble_log, 1);
GO
-- Reset the database recovery model.
ALTER DATABASE PerformanceTrouble SET RECOVERY FULL;
GO
-- Be sure to do a full backup, then kick off transaction log backups

-- check the size of the files.
SELECT size / 128.0 as sizeMB, name
FROM sys.database_files;

Once the script is run we can see that the log file has been shrunk from the 24GB to be around 1MB.

When we check the drive space on the server it now looks like this:
CDriveAfterScript
Now we are no longer at the the critical point where we think the server is going to run out of space, but there are two things that should be done next.

1. Run a full backup of this database.  If your SQL Server were to crash before the full backup is run you would have no way to recover back to the current point in time.

2. Monitor and confirm if the database log is continuing to grow, or if this was maybe a one time event that caused it to grow this large. If you determine that it is still growing then research and find out what is causing the transaction log to fill up.  You can use many of the features of Database Health Monitor to do this research.

 

The overall troubleshooting process took about 3 minutes, and the time to fix this issue only took around 2 minutes.

SQL Forensics

If you owned a convenience store, or other business that was likely to get robbed you would likely have surveillance cameras with the intention of having evidence to provide the police with the information that they need in order to track down and prosecute the person who may rob the store.

cStoreCamera

Most of our databases have far more value to our company than the entire contents of a small convenience store, and whether you are willing to admit it or not they are probably more likely to get hacked than a convenience store is to get robbed. But we don’t have the surveillance system in place to monitor what happened when and why.

Once you have been hacked it is very challenging to find out what someone may have done to your databases, especially when you don’t find out for several months.

As a DBA, imagine your manager coming to you saying that 3 months ago between the 10th of the month and the 26th of the month the IT team has determined that a hacker obtained unauthorized access to the network. They have a log of all files that were transferred out of the network.  What we need from you is to know that parts of the database did the change or modify. Did they install a job anywhere that checks for confidential information and emails it to them on a regular basis, did they alter a stored procedure to move money into their account? Did they add another user to the database that will allow them unlimited access whenever they want? How would you answer these questions.

It would be nice if we could just ask SQL Server for a forensic analysis of what occurred between the dates in question and get a quick summary of what was changed, who was in the system and what might have been done. Unfortunately SQL Server doesn’t have a good way to do this, you could get a product that would scan the transaction logs to show what may have changed, if you still have the transaction logs from 3 months ago. You could get some of what you are looking for from Extended Events, maybe some from SQL Source control if you are using a source control product.

All those after the fact solutions are like working with a sketch artist for a convenience store robbery because you didn’t have a functioning camera.

If your company is large enough to have a data center, or to use a data center co-location facility there are probably plenty of cameras. But none of the cameras will catch the hacker with remote access to your systems.

surveillanceCamera

Many of us when asked what changed on our SQL Server 3 months ago over a 2 week period would probably do some research and eventually come up with the “I don’t know” answer. Management may see it like this… “You don’t know if anyone did anything to compromise the integrity of your database, What good are you?”

How do we solve this. Plan ahead, it can actually be easier that installing video cameras. Anyone who has ever installed surveillance cameras knows how painful it can be to get them in the right place, the right angle, the right view.

Github Open Source Project

Introducing SQL Forensics, an open source project that I have just started hosted at Github.  https://github.com/SteveStedman/SqlForensics

The reason that I have created this as an open source project is to have the highest level of transparency in the project. All the source code is there so look it over, don’t trust me, confirm for yourself everything that it is doing to monitor your system, and if you don’t like something it is doing, then just change it.

What does it do today… Not much since I just started working on it yesterday. But here is where I plan to go with it.

  • Tracking of who changed what and when
    • Stored Procedures
    • Functions
    • Tables
    • Triggers
    • Foreign Keys
    • Schemas
    • Users
    • Logins
    • Permissions
    • Add/Remove a Database
    • Database configurations
    • Instance / Server configurations
  • Alerting
  • Investigation Tools

So far if you install the SQL Forensics database it tracks any changes made to the global configuration settings for the current server, sp_configure. You can see the baseline, and any changes by querying the [Log] table.

SELECT li.[name], cast(l.[whenRecorded] as datetime), l.[value]
  FROM [ForensicLogging].[Log] l
  INNER JOIN [ForensicLogging].[LogItems]  li on li.[id] = l.[itemId]
  ORDER BY [whenRecorded] DESC;

sp_configure_advanced_optionsThis shows that sp_configure ‘show advanced options’ was changed 3 times, but reconfigure was never run.  The value column shows the configuration value, as well as the value currently in use.

 

advanced_options

What would you need to know?

What do you need to know for a proper SQL Forensic investigation to know if someone is in the system?  Drop me an email or post a response to this message with your needs.

Links:

Database Health Monitor – Beta 8 – Soft launch

Today I launched Database Health Reports Beta 8 as a soft launch. What I mean by the soft launch is that other than updating the DatabaseHealth website and this blog posting, I haven’t done much to promote it.  Why you might ask?  Due to it being a holiday week between Christmas and New Years, there are many people out of the office. I figured I would wait until after January 1st to make a big splash.

Beta 8 is out. The biggest change is the charting. About 75% of the charts in the system have been completely rewritten. I was using a charting module that didn’t allow for the flexiblity that was needed to make the charts look the way I wanted. In September right after the Beta 7 release I started rewriting the charts from scratch.  Between now and the next Beta I will work to get more of the charts converted over to the new look.

Here is an example of some of the newer charts. The red and green bars on the plan cache are used to indicate change.  Red indicates that the value was worse, and green indicates that it improved or stayed the same. The CPU by database chart was rewritten to make better use of the available space.

New Charts

 

Also shown in the above screen shot is the new Server Configuration panel with details on the specific version of SQL Server, when it was installed, the number of processors and more.

Beta 8 Release Notes

Beta 8 Released 12/29/2013.

The following changes have been made since Beta 7. The big feature in the Beta 8 is the rewrite of many of the charts.

New Features

  • Renamed to Database Health Monitor, attempting to avoid confusion with SSRS Reports.
  • Additional checks for obsolete or unusual settings (SHRINK_DATABASE, TORN_PAGE_DETECTION).
  • Blocking reporting with hierarchical drilldown on the blocking queries.
  • Server details panel showing logical and physical cpu counts, SQL Server Start time, SQL Server install data, Server Name, SQL Server version info, and information on real or virtual server.

Bug Fixes

  • Multithreading the re-connect of databases, vastly improving the startup time if one or more databases is not available.
  • A large amount of the project has been refactored to help mature the product and allow for additional feature growth, and reduce bugs.
  • Improved background threading.
  • Updates and bug fixes on SQL Technical Debt.

SQL Server Performance for Developers

For the .NET programmer, Visual Basic programmer or PHP programmer, if you are accessing a SQL Server database there are some things you should know to performance tune your queries. Learn how to improve query performance with Indexes, how to properly use parameterized queries, using the query analyzer, and avoiding common SQL Server performance pitfalls.

PerformanceTuning

This presentation is a lot of fun. This is one of the few presentations where there is audience participation. Four luck participants will be selected to help simulate the work that SQL Server does when accessing tables structured with different types of indexes.

Download the presentation here:

PerformanceTuning.zip

If you are interested in performance, please take a look at the Database Heath Monitor.

SQL Server FILESTREAM and FileTables

Monday: Using FILESTREAM and FILETABLES in SQL Server

Configuring and Using FILESTREAM and FILETABLES in SQL Server.  Developers love to use SQL Server to store files, but this causes headaches for the DBA, finally a reasonable solution for file storage in SQL Server FILETABLES and FILESTREAM. SQL Server 2008 and 2012 add the new features of FILESTREAM and FILETABLES. Learn how to configure and manipulate files in your SQL Server with FILESTREAM, then learn how to do everything that FILESTREAM sounds like it should do with FILETABLES. With FILETABLES inserting is as easy as drag and drop.

Here is the outline for the presentation:

  • FILESTREAM – SQL Server 2008 and newer
    • Introduction and Configuration
    • Creating a Table Using FILESTREAM
    • TSQL FILESTREAM Access
  • FileTables – SQL Server 2012 and newer
    • Configuring and Creating FileTables
    • Insert, Update and Delete with a FileTable
    • Drag and drop with the file system

 

Here are the slides from the presentation and the supporting sample files.

Filestream and FileTables.zip

 

Speaking at Vancouver DevTeach this week.

This week I will be attending and speaking at Vancouver DevTeach. This event taking place on December 2nd to 4th 2013 at the Vancouver Sheraton Wall Center Hotel. Monday and Tuesday I have morning presentations which leaves the rest of the day to attend a few other sessions.

DevTeach

Here are the sessions that I will be presenting.

Monday: Using FILESTREAM and FILETABLES in SQL Server

Configuring and Using FILESTREAM and FILETABLES in SQL Server.  Developers love to use SQL Server to store files, but this causes headaches for the DBA, finally a reasonable solution for file storage in SQL Server FILETABLES and FILESTREAM. SQL Server 2008 and 2012 add the new features of FILESTREAM and FILETABLES. Learn how to configure and manipulate files in your SQL Server with FILESTREAM, then learn how to do everything that FILESTREAM sounds like it should do with FILETABLES. With FILETABLES inserting is as easy as drag and drop.

Tuesday: SQL Server Performance for Developers

For the .NET programmer, Visual Basic programmer or PHP programmer, if you are accessing a SQL Server database there are some things you should know to performance tune your queries. Learn how to improve query performance with Indexes, how to properly use parameterized queries, using the query analyzer, and avoiding common SQL Server performance pitfalls.

Looking at the lineup of other speakers this looks like it will be a great few days of education. It is nice to attend a conference so close to home, no flights involved for me this time.