Configuring Alternate Access Mapping in SharePoint 2010

It took me quite a while to get back to writing articles for both MSSQLTips and MSSharePointTips. Now, I'm back and the first article to be published this year is all about configuring alternate access mappings on SharePoint 2010. Check out the article here. There's a corresponding PowerShell command to do the same task so watch out for that article as well

Disaster Recovery Is More Than Just Technology Part 3: The Lion, The Switch and The Wardrobe

You were in your favourite bar one Saturday night when, suddenly, you hear your mobile phone ring. You pick up the phone and heard the sound of a screaming voice on the other end (no, it’s not your wife telling you to go home and take out the trash). The background noise is preventing you from understanding what is actually being said. You checked on the phone number that registered on the phone – it’s your manager. You get out of the bar to clearly hear what is being said until you barely hear the last phrase, “the production database is in recovering state for more than an hour now…” And, then, your battery went dead. Sounds familiar?





In a previous blog post, I talked about the different acronyms that come with the term disaster recovery. In this blog post, I’ll talk about key items that we sometimes tend to ignore when creating a disaster recovery strategy – the lion, the “switch” and the wardrobe (I’ve been a fan of the Narnia movie series from which I got the idea). And, yes, I did get a phone call similar to that while I was driving with my family that I had to pull over and guide the other person on the line as they try to recover the database.





The Lion



I call them the lions because they represent people with authority and responsibility over the infrastructure that you are preparing the disaster recovery strategy for. You definitely need to include them in. If you’re the DBA and you’re designing a disaster recovery strategy for the database server, then that lion is you. However, there are cases where the server administrator is not necessarily the DBA. That other lion is the server administrator. Oh, and isn’t your database server connected to the network? Then, you have another lion in the pack – the network administrator. And isn’t that a faulty hard drive that caused the disaster? Do you know who the supplier was for that hardware? Yes, that lion belongs to your pack as well. I can go on and on and include a ton of people in this list – the service provider for your network link, the company that stores your tapes offsite, the junior staff that needs to know what to do in case you’re on vacation, your IT manager who needs to make the tough calls in case the need arise. Make sure you know who the lions are in your pack and how to get in touch with them. Document who is responsible for what because a missing lion in the pack will definitely affect your service level agreement.



The “Switch”



This is intentional. The switch (not the witch) represents the other types of “hardware” that affect your service reliability. I’ve had some discussions in the past with one of my former customers who happened to have high availability built into their database servers. They had their SQL Server instances running on top of Windows Failover Cluster which they designed after upgrading to Windows Server 2008. However, one of their past outages clearly show that Windows Failover Clustering is totally meaningless if you do not consider the other components of the hardware stack. While multiple nodes of the cluster have provided high availability for their cluster, the main culprit for the outage was the shared storage. Their SAN happened to be on a dedicated network that was causing a bit of an issue with routing. To make matters worse, the “switch” on which the SAN is connected to was shutting down unexpectedly due to power outages that might have been caused by improper wiring on the UPS. They focused so much on the availability for the cluster that they didn’t look at the storage and the network to be potential causes of outages.



Also, in my previous life as a data center engineer, we have had an incident where the production server suddenly experienced performance issues. We couldn’t figure out why because even our remote access sessions won’t go thru to allow us to perform troubleshooting. Until one of the heat sensors in the data center went off. The high CPU utilization was caused by overheating. One of the air conditioners shut down, thus, causing drastic temperature increase inside the data center. Those who spend time visiting a data center know that you need to be wearing a thick enough coat to keep yourself warm while working. Air conditioners are used to control the temperature and humidity to help prevent equipment overheating and, potentially, disaster. While fixing a faulty air conditioner won’t happen in less than a day, designing the data center to allow for such incidents should be considered as part of the disaster recovery strategy (we have had to bring in electric fans and portable air conditioners to temporarily prevent the temperature from rising while the air conditioners are being fixed). You also need to know the lion in the pack responsible for the data center management in case you have your servers co-located somewhere. Bottom line is that you need to consider the other types of hardware that affect your service reliability and should be included in your disaster recovery strategy.

The Wardrobe


The wardrobe represents storage of stuff. And stuff could be anything that affects your service reliability. One of my favourite wardrobes as far as disaster recovery is concerned is the runbook. It stores the information for a particular system that can be used by anybody should the need arise. Not too many DBAs or IT professionals like the idea of documentation but, as I’ve heard from a few, it’s a necessary evil. If you need to rebuild the server because of hardware upgrades or, worse, disaster recovery, the runbook will be your guide to have the server rebuilt just as it was before. If you don’t have one, chances are that you won’t be able to rebuild your server with the exact same configuration as before. With a runbook, you can have junior staff go thru the process themselves by simply following it in written format. You can even include processes for recovering databases based on your backup strategy. A common rule of thumb for runbooks is simply this – write it so that even the most junior staff can figure it out. The challenge is keeping the runbook updated with the changes made on the system. However, runbooks are definitely a must for disaster recovery strategies.
 What about backups? Where are you keeping them? Do you have access to your backups? Are the backup tapes labelled intuitively? Are they safe? Are they stored offsite? This type of wardrobe should be documented as well so you will know where to look for your backups when you need them.


How about storage media? Yes, the media for installing your operating system, your database server, your patches, service packs, application software, etc. Have you heard about a legacy application that is only supported on Windows 2000 Server only to find out that the installation media is missing after the server crashed? Create backups of storage media and document them accordingly so that you can be sure they’re there when you need them.

And, have you even considered yourself as a wardrobe? Yes, you’ve got a ton of information on your head that needs to be shared within your team so that you don’t get as much of those emergency phone calls. You can set up a mentoring session with the junior staff, write documentation (or a blog post like this), do an internal presentation on how to perform a test restore of a backup – anything to make sure you’re not the only person on your team who can do the job. Most people don’t like this idea for fear of losing their job. But this is one thing that would make you more invaluable to the organization. For now, I’ll leave this topic for a professional development blog post.



I’m tempted to dive into the technical details of SQL Server disaster recovery and high availability but I realized that in order to really appreciate the technology aspects of disaster recovery, we need to understand the other aspects that affect and influence it. And just to give you a heads up, I’ll be in Orlando, FL in May for the first ever PASS SQLRally where I’ll be talking about Disaster Recovery Techniques for the SQL Server DBA. If you happened to be within the area or are planning to attend, come see me at my presentation

Check for Free Space on your Drives - LazyAdmin PowerShell version

When tasked to automate a task, I rely on my scripting background to get the job done. Back then, it was all VBScript. Now that more and more customers are moving to Windows Server 2008 servers, I try to convince my customers to use PowerShell for all their scripting needs.

A few weeks back, a friend of mine asked me to write a script that will check the disks on all of their servers and send a report via email.  They have over a hundred or so servers in their data center and it will be a bit cumbersome to log in to each server just to check for free space on their disks. While their network operations center have access to reports generated by their monitoring tools, she didn't. Which is why she opted to do it on her own. I tried to demonstrate why PowerShell was my scripting language of choice for this task by highlighting how this task can be done in a single line of code. Not that it can't be done otherwise but this is an effective way to tell people how easy it is to use Windows PowerShell. Below is the script to do the task (BTW, this can be written in a single line. The blog engine just made it as it is due to space constraints)

Get-WmiObject Win32_Volume -computername "localhost" |

Select-Object __SERVER, Name, @{Name="Size(GB)";Expression={"{0:N1}" -f($_.Capacity/1gb)}},@{Name="FreeSpace(GB)";Expression={"{0:N1}" -f($_.freespace/1gb)}},@{Name="FreeSpacePerCent";Expression={"{0:P0}" -f($_.freespace/$_.capacity)}} |
Where-Object -FilterScript {$_.FreeSpacePerCent -lt 15} |
Sort-Object -property "FreeSpacePerCent"
Format-Table

Now this may seem intimidating at first but let's disect the script to understand what it is doing. The Get-WmiObject cmdlet calls the Win32_Volume class to scan thru a list of disks (I'm using the -computer parameter to highlight an important concept later on.) This includes mountpoints, local drives and USB drives. If you're only concerned about local disks and mountpoints, you can exclude USB drives by filtering via the DriveType attribute of the Win32_Volume class.

Get-WmiObject Win32_Volume -computername "localhost"

Since we are interested in the size of the disk, the free space in GB and in per cent values, we will use the Capacity and FreeSpace attributes of the Win32_Volume class. We do need to perform some calculations to make sure that we get the values we are accustomed with - GB for capacity and per cent in free space value. That's what the calculations are for, noting that capacity and free space are expressed in bytes. The Select-Object cmdlet simply creates a new object by defining a new attribute from the original Win32_Volume class - server hostname, drive or mountpoint name, capacity and free space - named FreeSpacePerCent. This will definitely give you all the disks on a server with the defined attributes.

Select-Object __SERVER, Name, @{Name="Size(GB)";Expression={"{0:N1}" -f($_.Capacity/1gb)}},@{Name="FreeSpace(GB)";Expression={"{0:N1}" -f($_.freespace/1gb)}},@{Name="FreeSpacePerCent";Expression={"{0:P0}" -f($_.freespace/$_.capacity)}}

But we don't want all of the drives. We only want those that have FreeSpacePerCent value less than your allowable threshold. In this example, less than 15 %. So, we use the Where-Object to filter the results.

Where-Object -FilterScript {$_.FreeSpacePerCent -lt 15}

As an administrator, we need to make sure that we address issues that are more critical than others. This is where the Sort-Object cmdlet comes in. We sort the results in order of increasing FreeSpacePerCent so we can immediately address those disks with very little free space left.

Sort-Object -property "FreeSpacePerCent"

Finally, the Format-Table cmdlet is just for aesthetics. The reason is because I will be sending the results of this script as an email attachment.

Now, that wasn't so hard, was it? Understanding what each cmdlet is doing and how you can pipe the results to another cmdlet is the key to maximizing the use of PowerShell. But if you have a hundred or more servers, you wouldn't want to copy the script on all of your servers and run it from there, would you? There are a few ways to accomplish this. One of which is to ue PowerShell Remoting. This will be another topic for a blog post as there is more to it than just simply writing and executing a PowerShell script remotely. What I opted to do here is simpler since I'm assuming that my friend nor I don't have access to Active Directory to create Group Policies to enable PowerShell Remoting on all of the servers. Since the Get-WmiObject cmdlet has a -computer parameter, I can use that to execute queries against remote computers. What I can do is simply read thru a list - probably a list of computers in Active Directory or even as simple as a text file list. I opted for the text file list for the same reason that I didn't go for PowerShell Remoting. So, what I did was to convert the script above to a function that I will call while passing values to the -computer parameter of the Get-WmiObject


function getDiskFreeSpace
{
Get-WmiObject Win32_Volume -computername $args |
Select-Object __SERVER, Name, @{Name="Size(GB)";Expression={"{0:N1}" -f($_.Capacity/1gb)}},@{Name="FreeSpace(GB)";Expression={"{0:N1}" -f($_.freespace/1gb)}},@{Name="FreeSpacePerCent";Expression={"{0:P0}" -f($_.freespace/$_.capacity)}} |
Where-Object -FilterScript {$_.FreeSpacePerCent -lt 15}  |
Sort-Object -property "FreeSpacePerCent"  |
Format-Table
}

I will call this function as I read thru a list of servers in a text file passing the computer names as parameters. And since I want the results of the query in a single output file, I used the Out-File cmdlet to save the results in a text file


ForEach($s in Get-Content C:\serverlist.txt)
{
getDiskFreeSpace $s  | Out-File C:\diskFreeSpaceResults.txt -append
}

Sending emails


I used a simple script in the past to send emails via Windows PowerShell. This was prior to PowerShell v2.0

$SmtpClient = new-object system.net.mail.smtpClient
$SmtpServer = "smtp.yourmailserver.local"
$SmtpClient.host = $SmtpServer

$From = "Friendly Reminder "
$To = "recepient@yourmailserver.net"
$Title = "Subject Matter"
$Body = "Body Text"


$SmtpClient.Send($from,$to,$title,$Body)

With PowerShell v2.0, the Send-MailMessage cmdlet was made available to send an email message. This made it easier to integrate sending email functionalities in PowerShell scripts. Adding one more line to the script above, I've included email sending functionality with attachment

Send-MailMessage -to "recepient@mail.com" -from "sender@mail.com" -subject "Servers Disk Free Space Report" -Attachment "C:\diskFreeSpaceResults.txt" -SmtpServer "yourSMTPserver.mail.com"

Now, you can start using this script with a text file that lists all of your servers. Make sure that the server on which you will be running this script has Windows PowerShell v2.0 installed and that you can communicate with those servers. I've seen servers on different VLANs that are isolated from each other but are in the same Active Directory domain. This causes the script to fail due to connectivity issues. Check with your network administrator to be sure.

Disaster Recovery Is More Than Just Technology Part 2: The Alphabet Soup

In my previous blog post, I talked about high availablity and disaster recovery (HADR) and how it is more than just the underlying technology that keeps the entire strategy intact. In this blog post, I’ll describe a few acronyms – sometimes called buzzwords – that are commonly referred to in HADR projects and implementations (I know I use them a lot when addressing questions regarding HADR.) These acronyms fall under the second P in my PPT for HADR – PROCESS. Every HADR project or implementation should first be able to define these acronyms well before they even purchase the hardware, software and technologies they intend to use. Let’s get going.



Recovery Point Objective (RPO). Simply put, RPO answers the question, “How much data can we afford to lose?” Every HADR project should be able to determine the acceptable amount of data loss and is usually measured in units of time. For example, if a highly critical application runs 24X7 and the stakeholders have defined the RPO to be one (1) hour, if the database that stores the data for the application crashes at 5:45AM and you are running regular log backups (for SQL Server) or redo log backups (for Oracle) every hour starting at 12:30AM (I’m pretty sure the Oracle guys would jump up on me by using this as an example), the 15-minute data loss would be acceptable as you have an RPO value of one (1) hour. The 15-minute data loss was derived from hourly backups running starting at 12:30AM and that the last backup ran at 5:30AM – 15 minutes before the database crashed. Now, defining the hourly backups was not decided upon based on guesswork, although, I see a lot of DBAs simply just use it as a standard. If it was, it’s time to define your RPO and determine the amount of acceptable data loss and review your backup strategies.



Recovery Time Objective (RTO). RTO answers the question, “When is my application coming back online after a disruption?” Together with RPO, RTO is also measured in units of time. Looking back at the earlier example, if the stakeholders have defined the RTO to be two (2) hours, the database, the application and whatever is necessary to use the application should be back online by 7:45AM.

 Service Level Agreement (SLA). As Wikipedia defines it, SLA is a part of a service contract where the level of service is formally defined. In my experience, this is commonly agreed upon by a customer and a service provider. You might be thinking that if you have an internal IT management team, chances are that you won’t have to deal with SLAs. However, bear in mind that in order for a computer application to be online, it relies on hardware which needs to be covered by a vendor warranty with associated service levels should the hardware needs to be serviced or replaced during a disaster, a media on which it can be accessed – either via the Internet or your local network – which also needs to be covered by vendor or in-house service level agreements. This is a very important item in your HADR project as anything external to your team will definitely affect your RPO and RTO. For example, if your highly critical application has been restored within two hours and the data loss was less than an hour, you may have met your RPO and RTO but if the Internet connection that allows your users to access the application is still not restored after two hours, forget about achieving your RPO and RTO. Technically, from the point of view of users, your application is still not accessible. Which is why when you’re dealing with vendors or service providers, make sure that your agreed upon SLAs meets your RPO and RTO.



I’ve only scratched the tip of the iceberg on the different components that make up the PROCESS part of an HADR project. What I find surprising is that whenever I start asking about RPO/RTO/SLA values from customers asking for an HADR solution, they immediately respond with “I want zero downtime and zero data loss for my application.” They simply think that their application deserves an RPO and RTO value of zero (0). What they don’t realize quite yet is that as RPO and RTO approaches zero (borrowing jargons from integral calculus that, as your limits approach zero), the cost exponentially increases. And when we start talking about costs, customers start re-evaluating their HADR strategies the way they are supposed to be. This is where I really like the discussions to go because they will look at each application and the corresponding database differently and categorize them accordingly – from not-so-critical to highly critical. And they start crunching numbers to determine how highly critical an application can be and if it does merit a near-zero RPO and RTO. Take for instance an e-commerce site that generates an average of 50 transactions per minute (which is a relatively low volume these days) at US$10 per transaction. That is equivalent to US$500 per minute. Losing an hour’s worth of data due to downtime or data loss would mean US$30,000. Having an HADR solution in place should be justifiable enough to protect a US$30,000 worth of transactions in an hour. Your strategy would also consider if the transactions are only coming in between 5:00AM until 9:00PM as you wouldn’t want to be investing a lot for a solution that doesn’t protect anything after those hours.

 So when you plan your next HADR project, think about these concepts and define your RPO/RTO/SLA. It will definitely keep your perspectives right and plan accordingly. In my next blog post, I will be talking about high availability implementations, more examples and how they should address your RPO/RTO/SLA. Keep posted. Plus, if you’re in the Washington DC area this weekend, catch me do a presentation on SQL Server Disaster Recovery Techniques for SQL Saturday #61