Tag Archives: windows

5 (more) New Features in SQL Server 2017

FETCH NEXT 5

About a week ago I posted about the Top 5 Things I’m most excited to see in SQL Server 2017. As you may have noticed, I just focused on the Engine/Agent and not on anything else. To be fair (and I’m always fair to myself), most of the big exciting changes were there.

However, I did want to give the rest of the SQL Server features their due. To rectify that oversight, here’s 5 more new things in SQL Server 2017 that I’m excited about.

1. SSIS On Linux
A Linux Cluster.
Image Source: Visual Hunt

This is a bit of cheat, because I already went over SQL Server on Linux, but this I thought deserved special notice. Did you know you can run your SSIS packages from your Linux box with SQL Server now? You can.

Just pop in this little one-liner and you’re off to the races*.

$ dtexec /F \<package name \> /DE <protection password>

If push came to shove you could probably put that on a cron job should you want. You still need a Windows server to create and maintain the packages, but you can run them locally from the box if you’re trying to keep the family together for the kids.

* Certain Terms and Conditions may apply. See your dealer for details.

2. Machine Learning Services
Python!

This is a feature that kinda existed previously, but it was just called “R” Services. The big thing of note is that it now supports Python and the associated libraries. See previous post in this series to catch my sarcasm about Python not being included in the first place.

Thing to note about Machine Learning Services is that it’s not supported in-database on Linux. You can still do things like native scoring (PREDICT), but that’s just about the long and the short of it. Microsoft is making noises like they’re going to address this in the somewhat-near future.

3. SSIS Scale-out

This is a pretty neat feature that I hadn’t thought about before. What if you had several servers that potentially COULD handle an SSIS workload (in an HA scenario or something), but you didn’t want to always target the same instance. You know, spread the love around.

SQL Server 2017 allows you to set up a master on your main instance and then workers on the servers you want to be able to scale-out to. After a bit of setup on your worker machines you can then either target machines with specific packages or let SSIS decide. Check out this walkthrough for more.

You turn this feature on (assuming you’ve set it up properly) in the SSIS Catalog Properties.
Image Source: Microsoft

 

4. New SSRS Web portal

This is less a new feature, and more of a major revamp to something that already existed. The new Reporting Services default Web Portal is a lot snazzier and has some new things. You can customize branding the instance and even develop KPIs that are contextual to the folder you are currently viewing.

Purdy.
Image Source: Microsoft

 

5. MDS Performance Improvements

Master Data Services has had a rough life. Beginning life as far back as SQL Server 2008R2, this has been the red-headed stepchild of the SQL Server offerings. It started out, in my humble opinion, barely usable, unnecessarily complex and just feature-poor. I’m not alone in this opinion.

Subsequent releases have helped it, but even after several versions it still was pretty weak and only useful for very specific cases. MDS only really came into its own in 2016, but with some performance limitations.

Pictured: Master Data Services circa 2008R2.
Image Source: Visual Hunt

Edging ever closer to a more perfect product, SQL Server 2017 features some much needed performance optimization allowing it to stage millions of rows in a reasonable amount of time. It was painfully slow previously with only a few hundred thousand records.

Lastly, they fixed the slow UI movement when doing things like expanding folders on certain pages.

Honorable Mentions

Not any this time, unless you want to talk about SSAS object-level Security or DAX finally getting an IN operator. Those seem pretty useful.

The End?

That’s it for 2017. There are, of course, many many more changes and new features in SQL Server 2017, but I think 10 or so is good enough to give you a taste. There are changes all across the product and I encourage you to look them over yourself.

-CJ Julius

Top 5 New Features in SQL Server 2017 (that I care about)

New Year, New Database Engine

http://icons8.com/
This is either a database or a stack of licorice pancakes.

It’s finally here! A few days ago as of this writing, SQL Server 2017 was released for Windows, Ubuntu, RedHat and Docker. There are a lot of new things in SQL Server 2017 from Python support in Machine Learning* to better CLR security. But I thought I’d narrow down the list to changes that I’m most interested in.

1. sudo Your Way To A Better SQL Server

SQL SERVER IS ON LINUX NOW! No surprise this is the first thing on my list as I keep going on and on about it. But it’s really here!

Installation is a cinch, especially if you’re a Linux or Unix person. Curl the GPG keys, add the repository, apt-get (if you’re on a real Distro) the installer and run the setup. It’s really that easy.

If you want it even easier, then check out Docker. Slap it in and go!

All the main features are there, the engine (of course), agent, Full-text search, DB Mail, AD authentication, SQL command-line tools etc. Pretty much everything you need to get going on Linux even if you’re in a Windows-dominated environment.

2. It’s Like A Car That Repairs Itself

So this is a big one. With automatic tuning on, SQL Server can detect performance problems, recommend solutions and automatically fix (some) problems. There are two flavors of this, one in SQL Server 2017 and one in Azure; I’ll be talking about the one in SQL 2017 here.

Automatic plan choice correction is the main feature and it checks whether a plan has regressed in performance. If the feature is so enabled, it reverts to the old plan. As you can see below, a new plan was chosen (4) but it didn’t do so well. SQL 2017 reverted to the old plan (3) automatically and got most of the performance back.

Image Source: Microsoft

I’m sure that Microsoft will be expanding this feature in the future and we can expect to see more from this. Azure already has automatic Index tuning in place, so we’ll probably see that in the On-Prem version eventually.

3. Indexes That Start, Stop And Then Start Again

This is a feature I didn’t know I needed. Basically, it allows an online index rebuild that has stopped or failed (say, it ran out of disk space) to be resumed. The index build fails, you fix whatever made it fail, and then resume the rebuild, picking up from where it left off. The rebuild will be in the ‘PAUSED’ state until you’re ready to RESUME or ABORT it.

Code:
-- Start a Resumable index rebuild
ALTER INDEX [NCIX_SomeTable_SomeColumn] on [dbo].[SomeTable]
REBUILD WITH (ONLINE=ON, RESUMABLE=ON, MAX_DURATION=60)

-- PAUSE the rebuild:
ALTER INDEX [NCIX_SomeTable_SomeColumn] on [dbo].[SomeTable] PAUSE

/* If you'd like to resume either after a failure or because we paused it. This syntax will also cause the resume to wait 5 minutes and then kill all blockers if there are any. */
ALTER INDEX [NCIX_SomeTable_SomeColumn] on [dbo].[SomeTable]
RESUME WITH (MAX_DURATION= 60 MINUTES,
WAIT_AT_LOW_PRIORITY (MAX_DURATION=5, ABORT_AFTER_WAIT=BLOCKERS))

/*Or if you just want to stop the whole thing because you hate unfragmented indexes */
ALTER INDEX [NCIX_SomeTable_SomeColumn] on [dbo].[SomeTable] ABORT

This is also great if you want to pause a rebuild because it’s interfering with some process. You can PAUSE the rebuild, wait for the transaction(s) to be done and RESUME. Pretty neat.

4. Gettin’ TRIM
A cute Sheepy.
There hasn’t been a picture in a while and I was afraid you might be getting bored.

This one is kind of minor, but it excited me a lot because it was one of the first things I noticed as a DBA that made me say ‘Why don’t they have a function that does that?’ (note the single quotes). TRIM will trim a string down based on the parameters you provide.

If you provide no parameters it will just cut off all the spaces from both sides. It’s equivalent to writing RTRIM(LTRIM(‘ SomeString ‘))

Code:
SELECT TRIM( '.! ' FROM ' SomeString !!!') AS TrimmedString;

Output:
SomeString

5. Selecting Into The Right Groups

Another small important change. In previous versions of SQL Server, you could not SELECT INTO a specific Filegroup when creating a new table. Now you can, and it uses the familiar ON syntax to do it.

Code:
SELECT [SomeColumn] INTO [dbo].[SomeNewTable] ON [FileGroup] from [dbo].[SomeOriginalTable];

Also, you can now SELECT INTO to import data from Polybase. You know, if you’re into that sort of thing.

Honorable Mentions!

Here are some things that caught my eye, but didn’t really need a whole section explaining them. Still good stuff, though.

Query Store Can Now Wait Like Everybody Else

Query store was a great feature introduced in SQL Server 2016. Now they’ve added the ability to capture wait stats as well. This is going to be useful when trying to correlate badly performing queries and plans with SQL Server’s various waits.

See the sys.query_store_wait_stats system table in your preferred database for the deets. Obviously, you’ll need to turn on Query Store first.

Con The Cat With Strings

Just as minor as TRIM in some people’s books, but this is a great function for CONCAT_WS’ing (it’s not Concatenating, right? That’s a different function) strings with a common separator, ignoring NULLs.

Code:
SELECT CONCAT_WS('-','2017', '09', NULL, '22') AS SomeDate;

Output:
2017-09-22

Get To Know Your Host

sys.dm_os_host_info – This system table returns data for both Windows and Linux. Nothing else, just thought that was neat.

Get Your Model Serviced

Not something I’m jumping for joy about, but it is a good change of pace (I think). No more Service Packs, only Zuul… er… just Cumulative Updates. Check out my article on it if you want to know more about all the Service Model changes.

And Lots Of Other Things

Of course, there’s hundreds other things that are in SQL Server 2017 and related features. I didn’t even touch on the SSIS, SSAS, SSRS, MDS or ML stuff. Check out the shortened list here, broken down by category. Exciting new toys!

-CJ Julius

* Seriously, how do you release a Data Science platform and not include Python? That’s like releasing a motorcycle with only the rear tire. Yes, you can technically use it, but you you’re limiting yourself to a customer base with a very selective skill-set.

SQL Server on Linux: First 30 minutes

They put a ring on it.
They put a ring on it.

What I gushed over a few posts ago has finally happened! SQL Server has a come to Linux (sort of). The database engine is now available as CTP1 and you can get it by adding the repository and running the setup script.

You can follow the walk through for your favorite flavor of Linux, so I won’t repeat that here. it’s really very simple, just a matter of pointing to the correct repository and then apt-get install (Ubuntu). It comes with a setup script that pretty much does all the heavy lifting for you. Keep in mind that this is just for preview so there’s not a lot of options and it sticks everything in a single set of directories (logs/data/tempdb).

I had a small problem when I did the install, but it turned out I just needed to update a few packages. In the event you’re not a Linux person, here’s the easiest way to fix this:

$ sudo apt-get update
$ sudo apt-get upgrade

There’s a lot of stuff to dig into in this release, and as newer versions come out I’ll get more in-depth, but I just wanted to make a quick post about what I did in my first thirty minutes.

Behold in awe my INSERT abilities.
Behold in awe my INSERT abilities.

After the install, I connected via SQLCMD, as there is no SSMS in Linux yet, using the sa and sa password set in the install. I then created a table, dropping a single row into it and then selecting. Not terribly complex stuff.

I took care to try different cases, adding and neglecting brackets ‘[]’ and semicolons. It responded how I expected it to react if I was on a Windows system, which is very reassuring. It’s nice that my T-SQL skills translate seamlessly to the Linux environment, at least internally to SQL Server.

Connected via a my own username.
It doesn’t look or act any different than it I would have if connected to a Windows SQL Server instance.

Next, I put my box ‘U64’ on the network and lo-and-behold I was able to remote into it by its Linux hostname from SSMS 2016 on a Windows machine. No additional setup was required. Microsoft appears to be taking this integration of the Linux and Windows environments seriously.

I then created a SQL login for myself and logged in that way. No issues.

Now, as fun as this was, there’s a whole lot missing. The list includes, but is not limited to:

  • Full-text Search
  • Replication
  • Extended Stored Procedures
  • AD authentication
  • SQL Server Agent
  • SSIS
  • SSAS

This is of course just for CTP1, so a lot of these items will probably show up later. I mean, SQL Server without the SQL Server Agent? That doesn’t even make sense (I’m looking at you Express Edition). There is sort of cascade effect as other items like Maintenance Plans and such that rely on these missing features also being MIA.

The gang's all here!
The gang’s all here!

Also, larger items like Availability groups will also be absent because there’s no Linux analogue for them currently. From what the SQL Server team said in their AMA on reddit they’re toying around with RedHat clustering as a replacement for this in the Linux environment.

The last thing I did before the end of my 30 minutes was to look at the version. As you may or may not know, the Linux version is based on SQL Server vNext, which (as the name implies) is the NEXT version of SQL Server. There was some talk about it being a port of SQL Server 2016, which does not appear to be the case.

SELECT @@VERSION
----------------------------------------------------
Microsoft SQL Server vNext (CTP1) - 14.0.1.246 (X64)
Nov 1 2016 23:24:39
Copyright (c) Microsoft Corporation
on Linux (Ubuntu 16.04.1 LTS)

Note that SQL Server 2016 is version 13.0.

And that’s it! As mentioned before I’ll be doing deeper dives into this as time goes on, at the very least with each CTP. But I have to say I’m happy with the results so far. Everything (that was available) worked as I expected it to work. Nice work MS!

-CJ Julius

(Almost) Everything is Going Open Source Now… and I LOVE it.

Why can't we be friends?
Why can’t we be friends?

While I’m putting together my big update on Inventory Manager, I thought I’d take some time to throw confetti into the air. There may be some excited clapping as well. I warned you.

I largely see myself as platform-agnostic. While I think that certain companies do individual products well, I also believe it’s fair to say that none of them do everything well. I use Android phones and Apple tablets, Linux for home (mostly) and Windows at work. Heck, I’ve got a Roku and a Chromecast because they both do things that the other doesn’t.  I’m all over the map, but all over the map is a great place to be, especially in the tech industry now.

Despite all of this, I have to admit I am partial to Free Open-Source Software (FOSS). Give me a choice between Ubuntu and Windows, and all other things being equal, I’ll choose the Debian-based option. I’ll admit my biases.

So, when MS started moving in this direction I was happy. I wanted to see this trend continue, and boy has it. First of all…

1. .NET Core is now running on Redhat.

When Microsoft announced that .Net was going open-source, I was cautiously optimistic. I’m not a big .Net coder, but I could see the benefit and was hopeful that MS would continue down this path.  This lead to some cool things that I thought I’d never see in a million years, like .Net running on Redhat.

There’s understandably some cynicism about Microsoft’s true intentions, as well as their long term goals, but this is the cross-over that I’ve been wanting to happen for a while. Blending the strengths of RHEL with .NET on top is a great start. If the .NET development platform can be ported, why not parts of the Windows Management Framework? We could even one day see…

2. Powershell on OSX and Linux.

I didn’t always like Powershell, in fact prior to Powershell 3, I just referred to it as PowerHell. Since 4.0, however, it’s no secret that I’m a fan; one look at my github will tell you that. I like its logical approach to (most) things and that it works for simple scripts quite easily, while being a powerhouse (no pun intended) behind the scenes.

Sorry, THIS is the coolest thing ever.
This is the coolest thing ever.

This shell coming to OSX and Linux will be a boon for both systems. While I am, and will probably always be, a bash scripting guy, Powershell in Windows just makes everything so gosh-darn easy. If I could whip up a PS1 script with a few imported modules and attach it to a cron job with ease, then I think everybody wins,  mostly me. But, if I decide that I want to use bash instead, that’s okay because…

3. Bash is running on Windows.

This isn’t a one way transition. Microsoft is making a trade, bringing one of the most widely used shells to Windows. This not only makes scripts more portable, but also knowledge.

Have some ultra-fast Linux bash script that works wonders? Super, you now have it Windows, too. Wrote a script to do some directory work in Powershell? Great, you now know how to do it in Linux.

You can't tell me that isn't the coolest thing ever.
I’m sorry, THIS is the coolest thing ever.

There are very few downsides to this, other than the obvious security issues and that it isn’t truly a stand-alone shell (it’s part of Ubuntu on Windows). In any case, it allows interoperability  between software from different systems. This is great now that…

4. SQL Server is on Linux.

This isn’t technically going open source, as it will run inside a container, but the idea that this will now be possible and supported is like something out of my greatest dreams.

I have a maybe-controversial opinion that SQL Server is the best relational database system out there. For all its faults, I’d rather use SQL Server 2005 SP1 than Oracle 12c. Just the way I feel, and for reasons I won’t go into here. I hope the things I like about SQL Server translate to the Linux environment.

The fact that Ubuntu is supporting this with Microsoft is great. I can’t wait to use my favorite OS with my favorite database engine on the same system.

Last thoughts

There are other items I’ve glossed over, but these are the big ones to me. Soon, we will be able to run SQL Server on Ubuntu Linux with cron jobs executing Powershell for a .Net application that resides on an RHEL box. *excited clapping* (I warned you.)

It’s a great time to be in the tech industry.

-CJ Julius

 

Creating a Simple Database Inventory Manager with Powershell – Part IV: GUI Front-End

Last Time, on Inventory Manager…
Now we make things pretty.
Let’s make things pretty.

Our data pull script has run, the database contains all our server  \ instance .database information and flowers are blooming; things are good. If this doesn’t sound like anything you have done, head back to the Introduction to see if you missed something.

Now it’s time to get everything connected so we can just fire up a GUI and press some buttons, to get the data we need fast.

How the Sausage is Made

If you’ve been following along with this series, and you’ve set up everything as instructed, then you should be able to download the pre-made GUI script and run it out of the box. If you’re pointing to a custom instance or database just change the $RepositoryInstance and/or $RepositoryDB before firing it up. If you want to learn more about how this was put together, keep reading. If you don’t care how the whazits work, you’re done.

At the top of our list is to create a form with buttons and give them names so we can call them in our Powershell. You can either build the form manually with this guide here, or use Visual Studio*. I’m going to be using the latter method because it’s the most versatile, and frankly the easiest. If you use the former, then you’re kind of on your own. Sorry.

In the  Visual Studio method (I’ll be using 2013 Ultimate) you’ll be utilizing Windows Forms and then running them through a “cleaner” to make them Powershell ready. This guide at FoxDeploy explains the whole thing spectacularly and shows you how to create some very complicated UI’s that are Powershell-friendly. I’d recommend going through Parts I and II as they’ll be the things you’ll need to create what we’re going to use here and then come back. Don’t worry, I’ll wait.

Then We Build

Got the GUI code? Cool. The first part of Stephen’s code uses a -replace to filter the Windows Form code and make it work in Powershell. I took that piece and made it a second script so I could just have the clean version of the XAML in my final code. You can find that code here.

Just copy/paste your <Window>…</Window> code over the commented area and run the script. It will spit out the final code and tell you all of the objects you can tie actions to (Name, Value). Then drop in Stephen’s XAML reader code to the main script with the cleaned code and you should have a GUI… that does nothing.

Whee.
Whee.

As I mentioned before, when you pushed the XAML code through WPF_to_PSForm.ps1 it will tell you what the objects are on your form. For our purposes, this is simply a few buttons that need to be tied to stored procedures. We do this though .Add_Click() as in the example below:

$WPFBt_All_Data.Add_Click(
{
$sqlCommand = "
EXEC dbo.prGetInventory;
"
$dataset = Invoke-SQL -datasource $RepositoryInstance -database $RepositoryDB -sqlCommand $sqlCommand
Write-Host $dataset
$dataset | Out-GridView -Title "Database Inventory"
}
)

Nothing crazy that we haven’t been doing other than using Out-GridView. This cool little cmdlet pushes datasets out to a customizable table with filtering, sorting the ability to remove columns etc. -Title “SomeTitle” is the window title.

Sample sorted Database List with a few columns removed.
Sample sorted Database List with a few columns removed.

Once you’ve coded all of the buttons, then add the form display at the bottom, using out-null to suppress messages:

$Form.ShowDialog() | out-null

And that’s done. A Winner is You!

What now?

Using these scripts, you can go out and grab any information from the servers\instances you specify, pull it back into a centralized location and then use a GUI front-end to make fine tuning and retrieval easy. As I stated previously, this is a bare-bones system to centralize your database information. You can gather any piece of information from the Server, Instance or Database level by using the same tools that are currently collecting and retrieving this information.

It’s been a long journey, but thanks for sticking with it! If you want to make any alterations to the code or tighten it up (Odin knows that it needs it), feel free to make the changes and shoot them back to me. I’ll definitely give you credit for significant changes in this blog or the code itself.

Also, and I think this goes without saying, but if you want to use this in your personal or business environment: have at it! Just please make sure you give me proper credit, with maybe a link back to my blog/Twitter/Linkedin? That’d be super cool of you.

Thanks again and happy Inventorying!

–CJ Julius

*Full disclosure: I have not tried this with the Community version of Visual Studio, so all the features may not be there.

Creating a Simple Database Inventory Manager with Powershell – Part III: Data Pull

Powershell time; no really.
DB_inven_Mgmt_PS1_sm_III
I come bearing scripts.

Now it’s time to get to get this thing moving. We’re going to go out to each of our server\instances and pull back the information for our tables, updating them with the stored procedures from the last section.

We’re going to be looking at this script [DB-DataPull.ps1]. It’s about as simple as I could get it for our needs. There’s not a lot of frills, but it’s a good cop and it. gets. results.

If you think you missed something you can go back to Part II: Stored Procedures or check out the Introduction.

Get this Jalopy on the Road

The only thing you need to do is specify where the repository is. If the repository is on your local machine in the DBAdmin database then you need to change nothing.

$RepositoryInstance = '(local)'
$RepositoryDB = 'DBAdmin'

After that you’re done. Seriously. The rest of this post is going to be about the nuts and bolts of the script and what does what and why. If you’re looking to just get it fired up then you’re done. Be gone with you.

What’s in the box?

The first few functions (Get-Type and Out-DataTable) are required to turn multi-line WMI-Object output into DataTables so we can insert them into the Repository. These have been cleaned up and/or modified to fit our needs but are based on the code in the two links I provided.

The Invoke-SQL function is a pared-down version of a pretty popular script for sending dynamic SQL directly to a SQL server. There’s not much to be said about this one other than it opens a connection, sends the command and returns the results as a datatable.

Time to get into the meat of the process. First up, let’s grab all the Instance information using the stored procedure we built in the last post.

$ConnectionString = Invoke-SQL -datasource $RepositoryInstance -database $RepositoryDB -sqlCommand "
EXEC dbo.prGetConnectionInformation;
"

Using a foreach loop to cycle through the rows and thus connecting to each instance to pull the information. We’ll remove the ‘\\MSSQLSERVER’ part since that will actually break our connection, even though it’s the name of the instance (For more information on why this is, see every other Microsoft product ever created).

foreach ($Row in $ConnectionString.Rows)
{
Try
{
$SubConnection = $($Row[0]) -replace '\\MSSQLSERVER',''
$InstanceID = $($Row[2])
Write-Debug $InstanceID
Write-Debug $SubConnection
$Version = Invoke-SQL -datasource $SubConnection -database master -sqlCommand "
SELECT SERVERPROPERTY('productversion'), SERVERPROPERTY ('productlevel'), SERVERPROPERTY ('edition'), @@VERSION
"
}
...

And then with another loop we use dbo.UpdateInstanceList to push all that into our database.

Invoke-SQL -datasource $RepositoryInstance -database $RepositoryDB -sqlCommand "
EXEC dbo.prUpdateInstanceList
@MSSQLVersionLong = '$MSSQLVersionLong'
,@MSSQLVersion = '$MSSQLVersion'
,@MSSQLEdition = '$MSSQLEdition'
,@MSSQLServicePack = '$MSSQLServicePack'
,@InstanceId = $InstanceID
"

That’s it for the Instance information, let’s get the database information. We use the same process to generate the connections as we did before, so I’m going to skip that. The only change you should note is the inclusion of the statement TRUNCATE TABLE dbo.DatabaseList since we are going to completely repopulate it. This way no matter if databases are added or removed, we’re starting each pull with a clean slate.

We get our data via a cte…

$DataPull = Invoke-SQL -datasource $SubConnection -database master -sqlCommand "
with fs
as
(
select database_id, type, size * 8.0 / 1024 size
from sys.master_files
)
select
$InstanceID AS 'InstanceId',
name,
(select sum(size) from fs where type = 0 and fs.database_id = db.database_id) AS DataFileSizeMB
from sys.databases db
ORDER BY DataFileSizeMB
"

…and push it into the Repository via our stored procedure.

Invoke-SQL -datasource $RepositoryInstance -database $RepositoryDB -sqlCommand "
EXEC dbo.prInsertDatabaseList
@DatabaseName = '$DatabaseName'
,@InstanceListId = '$InstanceListId'
,@Size = $Size"

Lastly, we’ll get Service and Server information with the same rinse-and-repeat method, with one notable exception. If you try to return the results of a WMI-Object and parse it into a SQL table, then you’re going to have a bad time.

This is where our two functions from the beginning come in to play. Out-DataTable and its sidekick Get-Type return the results into the proper type for our foreach loop.

$ServerInfo = Get-WmiObject win32_Service -Computer $Row[0] |
where {$_.DisplayName -match "SQL Server"} |
select SystemName, DisplayName, Name, State, StartMode, StartName | Out-DataTable

Now, if you run EXEC dbo.prGetInventory on your Repository database, you should see all of the information you could ever want right there. Magic.

But Wait, There’s More!

Now we’ve got all the data in one place, which is nice and all, but what if we want to get this information quickly? Sure we can jump into SSMS and run the procedures that have the data we want. However, I propose we make a GUI front-end so we can win friends and get free drinks.

DB_DataPull_FrontEnd_GUI_2
Something like this?

We’ll do that in Part IV: The Voyage Home GUI Front-End.

–CJ Julius

Creating a Simple Database Inventory Manager with Powershell – Part I: Building the Repository Database

I've got it. We'll put the databases in a database
I’ve got it! We’ll put the databases in a database.

This first part is simply setting up the database and the tables underneath of them. I’ve tried to make this as painless as possible by providing scripts do do most of this for you. I’ll use this infrastructure piece to explain some of the data that we’ll be pulling back.

This is by no means an exhaustive list of all the information that can be retrieved, but it serves as a foundation to show how all data could be pulled back to these tables. The only limit is determining how you’re going to retrieve this data.

This part will require some legwork from you as entering the ServerName and InstanceName with corresponding Id fields will be necessary. You should only have to do this once*.

If you’re unclear about what all this is then, and think you missed something, check out the Introduction.

[DBAdmin_QA]

[DBAdmin_QA.sql] – Setup script for 2012 can be retrieved here.

This is the database that will act as the repository for all of the information we want to collect. You can call it whatever you want, as you can change this in the code later, but I don’t recommend it the first time around. Why make it more complicated than it needs to be?

Make sure you take a look at the settings and verify they fit to your environment. It is intentionally small and grows slowly as we most likely will not be making it larger by leaps and bounds.

You will want to also make sure that the user you will be making updates/deletes/etc as, has full access to this database. This is a ‘durh’, but I always have to say it.

[dbo].[ServerList]

[dbo.ServerList.sql] – Setup script for 2012 can be retrieved here.

This is our master list of all Servers that contain SQL instances we want to know about. Initially, you’ll want to insert all the Server Names (fully qualified if necessary for your environment) leaving the other fields blank. I don’t put things in bold unless they’re important.

An example of T-SQL code to insert these items:

INSERT INTO dbo.ServerList
( ServerName )
VALUES
('SOMESERVER'),
('SOMEOTHERSERVER')

Notable Columns:

Id – Identity Column for all Servers
ServerName – Manually entered Server Name
IPAddress – IP Address of Server
OSName – Name of the Operating System
OSServicePack – Number of the Service Pack

[dbo].[InstanceList]

[dbo.InstanceList.sql] – Setup script for 2012 can be retrieved here.

Contains all of the Instances and the proper ServerListId (Foriegn Key to dbo.ServerList.Id) as well as some related information. You’ll need to INSERT all of the Instances and match them up to the ServerList.Id column when you inserted the servers.

At some point in the future I may add to the code where it dynamically builds the InstanceList, but that has not been done. This is because there are some instances that ‘exist’ on servers are shut down or disabled in some way. This allows the Instance to be listed in your Full Inventory even if it can’t be accessed. Wouldn’t be much of an Inventory if we weren’t able to inventory it.

An example of T-SQL code to insert these items:

INSERT INTO dbo.InstanceList
(InstanceName,ServerListId )
VALUES
('NAMEDINSTANCE',1)
,('MSSQLSERVER',2)

If you’re a lazy bum and don’t like matching ServerList.Id’s to InstanceList.ServerListId:

INSERT INTO dbo.InstanceList
( InstanceName
,ServerListId
)
SELECT 'NAMEDINSTANCE',
sl.Id
FROM dbo.ServerList sl
WHERE sl.ServerName = 'SOMESERVER'

UPDATE: I’ve made this even easier with a new stored procedure Utility.prInsertNewServerAndInstance. If this is your first time seeing this then there was no update and this procedure has always been here. Look! Over there! Something distracting!

Notable Columns:

InstanceName – Manually entered name of the Instance
ServerListId  – Manually entered FK to dbo.ServerList.Id
MSSQLVersion – Version of MSSQL running this Instance
MSSQLVersionLong – The long-form version of previous column
MSSQLServicePack – Current Service Pack of the MSSQL engine
MSSQLEdition – Edition of the MSSQL engine
isProduction – Manual bit flag to designate a production instance

[dbo].[ServiceList]

[dbo.ServiceList.sql] – Setup script for 2012 can be retrieved here.

The dbo.ServiceList table contains information on MSSQL services running on the server. That does include the MSSQL Database Engine, but also other items such as Reporting Services or Full Text Services.

dbo.ServiceList is dynamically filled so there is no need to add any information to this table.

Notable Columns:

ServiceDisplayName – The Display Name of the Service
ServiceName – The system name for the service
ServiceState – State of the Service (ie “Running”)
ServiceStartMode – Start Mode of Service (ie “Auto”)
ServiceStartName – User the Service is running as

[dbo].[DatabaseList]

[dbo.DatabaseList.sql] – Setup script for 2012 can be retrieved here.

This table is the dynamically generated list of all the tables in the database. As you’ll see later in the Powershell, anything we can pull back from sys.databases (or anything we can join to it) can be put in this table or a related table.

Notable Columns:

DatabaseName – Name of the database (crazy, right?)
SizeInMB – Size of the database returned in Megabytes

dbo.DatabaseList is dynamically filled so there is no need to add any information to this table.

Database Ready

And that’s it for the tables and database. Keep in mind that this is a basic structure that can be the core of any setup you’d like. Any data you can capture via T-SQL or Powershell can be compiled and put into these tables or better yet, other related tables.

In the next post, I’ll talk about how we’re going to be allowing this information to be put into the database via stored procedures.

 

*Unless you forgot to make backups and accidentally wipe the tables. In this case, you can think about your mistake while you re-enter all the data again.

–CJ Julius

Creating A Simple Database Inventory Manager with Powershell – Introduction

Which databases names have the letter “B”?

And then we shall rule the world!
And then we shall rule the world!

All DBAs should keep track of their Servers/ Instances/ etc not only for their own edification, but for Management and security reasons as well. If you’re not, then you need to, as it comes in incredibly handy even if it isn’t a requirement of the job.

Most of the time, this information is compiled into a spreadsheet of some kind or possibly in a word processing document somewhere. Keeping this data up-to-date and accurate is a pain, especially when you have to break it out into multiple tabs and/or over multiple documents.

You could get a full-blown inventory manager that collects and compiles all the data and organizes it for you. But there’s a definite cost to that solution and not one that all companies will find useful (Read: “It’s not in the budget this quarter”).

What if you can’t get someone to shell out the money for a product like that? Then you have to either keep with the spreadsheets (yuck) or you need to find another solution with the tools you have.

What’s the Catch?

A simple frontend for your Database Inventory
A simple front-end for your Database Inventory

So, this is my attempt to resolve this issue using two tools that any MSSQL DBA should have: Powershell and SQL Server. I will point to other software products or versions both paid and free below, but the core code should run using things you should already have. That said, here’s my software list:

Required:

Preferred:

Also I will be making a few assumptions:

  1. Your infrastructure security is set up using Active Directory.
  2. Setting up a new instance is already done and you can connect to it.
  3. Your personal login or the login you are using to execute the code is able to query the relevant system tables and server info on each of the target systems.
  4. You know your current Server\Instance setup.
  5. There is no fifth thing.

What You Get

By the time this series is finished you’ll have a simple GUI front-end (shown above) for current data with all of your servers, instances and all the information you could want about them.

We can pull back and organize any data that SQL Server or Windows Server can spit out including, but not limited to:

  • OS Version, Service Pack.
  • SQL Services running, their statuses and logon information.
  • SQL Server Instance names, versions, and editions.
  • SQL Server Database names and sizes.
  • Ability to dynamically re-pull any of this information if needed.

I will walk you through my solution piece by piece over 4 posts (5 including this intro) that will consist of the following parts. This list will be updated with links to the different sections as they are released.

Part I: Building the Repository Database and Tables
Part II: Creating the Repository Stored Procedures
Part III: Coding the Data-Pulling Powershell
Part IV: Putting together the GUI Front-end

Updates:
2016-11-27
Addendum I: Simple Database Inventory Manager 2.1
2017-05-20
Addendum II: Simple Database Inventory Manager 2.3

See you soon in Part I: Building the Repository Database and Tables

–CJ Julius

MongoDB – Deleting Lots of Data Without Lock

MongoLock
Newer versions of MongoDB (3.x) handle locking better.

MongoDB doesn’t handle locking too well. It’s getting better in new versions (3.2 looks to be a strong contender) with collection and object level locking, but it’s still not great.

In an environment I was working on some time back, there was a need to remove lots of data on a regular basis to keep the size of the database manageable. The developers had implemented a snazzy maintenance system that NULL-ed the large fields that contained payloads for the data in order to save space after their utility was over. This maintenance system was run once a day and locked the collections/databases for that time; the brief period of locking was considered acceptable in this environment.

However, this didn’t free up any space in the system, because of the way MongoDB, (at the time 2.6, but 3.0 seems to have the same issue without Wired Tiger) NULL-ing the field did free up the space, but did not actually allow for new data to be inserted into its place.  The only way to free the space was to take the database offline and repair or compact it once a month. Not an option.

Because of the needs of the system, we were able to delete the whole object and reinsert it without the payload, but this caused excessive overhead and just turned our disk space problem into a CPU and memory one.

It was decided that since we did nightly backups, deleting the entire object was the way to go, but we’d need to do it manually until development could make the appropriate changes to the maintenance system. The problem then arose of deleting at least a week’s worth of data all in one go. This could take several hours and lots of locking if done all at once.

The solution was to have a process that deleted a set of the data from the database, pause for a moment to let other systems insert/delete/update, and it couldn’t require a large development cost (ie the DBA [spoiler: me] had to do it).

My solution as  a patch job was to have a java-script “job” run by a Windows batch script every week to delete data older than a certain date.  You can view my simple mongoMaint batch script on my github here and the javascript file that it calls here.

 

-CJ Julius

A Collection of Collections of Free Microsoft Books

Image modified from a free one provided by http://www.norebbo.com/
Microsoft has lots of free stuff out there. Image provided by norebbo.

There are a lot of free materials out there for learning Microsoft products, and suprisingly (or not?) a lot of them are from Microsoft themselves. I thought I’d take a moment to organize and collect my list of free resources in the hopes that not only will it help me organize and find what I need, but also help others of you who don’t know about this stuff.

The one main source I’m using here is the MSDN MSsmallBiz  blog with posts by Eric Ligman. There are a massive number of titles to look at, but I’ve not seen them compiled into one place. Keep in mind that some of these are older and all the links may not work. I will update this list in the future if I find new/interesting free education materials in this genre.

The Collections

Huge Collection of 60+ MS titles on various topics

This was the first list to go up and start the whole series. Almost all of the offerings come in multiple formats (PDF, EPUB, MOBI).

Noteworthy sections:

Visual Studio 2010 – Office 365 – Windows 8 – SQL Server 2012


 

Large Collection of 20+ MS titles on various topics

The second in the series, and the least interesting of the groups, but it does come with some interesting titles. This group only comes in one format: PDF.

Noteworthy sections:

Own Your Space (a book for teens, no really) – SQL Server 2012 Dev Training Kit


 

Gigantic Collection of 200+ MS titles on various topics

The last group contains quite a few of the previous two sections (but not all, I’ve found). Most are in PDF or DOCX (word) format with a few in portable and non-portable formats thrown in.

Noteworthy sections:

MS Office – Powershell 4.0 (this stuff is really good) – CRM – Quick Start Guide group – even more SQL Server 2012

Bonus

If you’re looking for information on specific Microsoft technologies or if you’re gearing up for an MS cert, check out the Microsoft Virtual Academy.  They’ve got kind of a neat gamification thing going on where you get points for completing certain courses.

 

-CJ Julius