Windows disk management with .NET framework

Windows has a very powerful programming API that allows developers almost total control over the way storage is managed. There are 3 main API subsets:

• VDS (Virtual Disk Service) allows to programmatically manage disks, volumes, and access features of Windows software RAID implementation. (http://msdn.microsoft.com/en-us/library/bb986750(v=VS.85).aspx)
• VSS (Volume Shadow Copy Service) lets one create snapshots of the current disk state. This can be used for a backup (these snapshots are internally consistent, because VSS exposes - and many software products consume - a subscription interface that allows application to write their state to disk before the snapshot is taken), as well as to preserve the current state of the volume state in time, so it (or individual files) could be rolled back at a later date. (http://msdn.microsoft.com/en-us/library/bb968832(VS.85).aspx)
• Block device access through device IOCTL and direct I/O (http://1-800-magic.blogspot.com/2008/02/taking-disk-image-snapshots.html)

This article explores the undocumented API to VDS for managed code available in Windows Server 2008 and Windows Server 2008 R2.

Unfortunately, .NET framework does not ship with an official wrapper, so there are two ways to access VDS from managed code. VDS is exposed through COM, but has no automation support, so one could use .NET COM interop, but would have to declare the interface by hand. There is a good example of how to do it here: http://stackoverflow.com/questions/2755458/retrieving-virtual-disk-file-name-from-disk-number/2892042.

However, VDS is a relatively big API, and so doing the marshalling by hand is a challenging job. Of course, if you do this, the humanity will be eternally grateful (especially if you post the results to http://pinvoke.net/)!

For those of us who are lazy, the next best thing is to use the DLL that is present on Windows Server 2008 and above - Microsoft.Storage.Vds.dll. It lives in both GAC and SxS. In both cases the paths to the files are absolutely awful, and I will dare not utter them here. Instead, go to you Windows directory and do "dir /s Microsoft.Storage.Vds.dll". The resulting DLL you can then copy to your application and add a reference to it in your project.

Important note: the DLL is ONLY present in Server OS - not in consumer Windows - but the interop logic that it exposes will work on Windows 7 as well. You just have to carry the file around with your app (note of course that Microsoft does not grant you any rights to redistribute parts of Windows!) .

Another important note: to use VDS on a local machine, your application must have an admin token in its process. When debugging in Visual Studio, it helps to start VS "as Administrator". To ensure that the right privileges exist when running the application, add the manifest file as follows:

<?xml version="1.0" encoding="utf-8"?>
<asmv1:assembly manifestVersion="1.0" xmlns="urn:schemas-microsoft-com:asm.v1" xmlns:asmv1="urn:schemas-microsoft-com:asm.v1" xmlns:asmv2="urn:schemas-microsoft-com:asm.v2" xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance">
  <assemblyIdentity version="1.0.0.0" name="MyApplication.app"/>
  <trustInfo xmlns="urn:schemas-microsoft-com:asm.v2">
    <security>
      <requestedPrivileges xmlns="urn:schemas-microsoft-com:asm.v3">
        <requestedExecutionLevel level="requireAdministrator" uiAccess="false" />
      </requestedPrivileges>
    </security>
  </trustInfo>
</asmv1:assembly>

The most important note: The managed API lacks one member of IVdsPack interface - the function that is absolutely crucial to creating RAID arrays (IVdsPack::MigrateDisks). As you can see in reflector, the IVdsPack is not completely marshalled ("slot7" occupies the place where MigrateDisks should have been):

This means that a new disk cannot be made dynamic using this interface, and therefore it cannot be used to build new disk arrays. If you need this functionality, you are screwed have to do COM interop by hand. You can still use this library with the reflector to extract much of the interop code. However, all the code needed to check the disk system health, and to replace a disk is present, which means that the library is very useful for many interesting scenarios. (Again, Microsoft does not give you a right to reverse-engineer parts of Windows to incorporate them into your application!)

With this in mind, on to managed VDS!

To use VDS, you have to first connect to VDS service. You can connect to a service both on your local machine as well as on a remote computer (providing that your user has admin rights on it:

using Microsoft.Storage.Vds;
...
ServiceLoader loader = new ServiceLoader();
Service vds = loader.LoadService("computername"); // or null
vds.WaitForServiceReady();

The object hierarchy in VDS is as follows. The VDS object contains a list of providers, and a list of unallocated disks. Unallocated disks are the disks that have not been initialized. You can access them as follows:

foreach (Disk d in vds.UnallocatedDisks)
{
    Console.WriteLine("Found unused disk {0} {1} {2} {3} {4}",
        d.FriendlyName, d.Name, d.DevicePath, d.Size, d.Health);
}

On most computers (excluding the very rare cases where hardware manufacturers exposed their custom storage systems through VDS, which can happen with SANs), there are two providers, a dynamic provider and basic provider. A dynamic provider contains dynamic disks - the disks that can contain RAID volumes (simple volumes, spanned volumes, RAID-0, RAID-1, and RAID-5). You make the disk dynamic in disk management applet:
 

The basic provider contains all the "normal", non-dynamic disks. To get dynamic and basic providers, do the following:

private static SoftwareProvider GetDynamicProvider(Service vds)
{
    foreach (Provider p in vds.Providers)
    {
        if ((p.Flags & ProviderFlags.Dynamic) != 0)
            return p as SoftwareProvider;
    }
    return null;
}


private static SoftwareProvider GetBasicProvider(Service vds)
{
    foreach (Provider p in vds.Providers)
    {
        if ((p.Flags & (ProviderFlags.Dynamic |
                ProviderFlags.InternalHardwareProvider)) == 0)
            return p as SoftwareProvider;
    }
    return null;
}

Provider contains a list of packs. There is usually only one pack in the dynamic provider, unless a set of dynamic disks initialized on a different computer was added, but not migrated (they appear as "foreign" disks in disk management applet). Basic disks are one per pack, so if there several basic disks in the system, there would be multiple packs in basic provider.

Here is how to print information on all the disks in a system:

SoftwareProvider dynamicProvider = GetDynamicProvider(vds);
foreach (Pack p in dynamicProvider.Packs)
{
    foreach (Disk d in p.Disks)
    {
        Console.WriteLine("Found dynamic disk {0} {1} {2} {3} {4}",
            d.FriendlyName, d.Name, d.DevicePath, d.Size, d.Health);
    }
}


SoftwareProvider basicProvider = GetBasicProvider(vds);
foreach (Pack p in basicProvider.Packs)
{
    foreach (Disk d in p.Disks)
    {
        Console.WriteLine("Found basic disk {0} {1} {2} {3} {4}",
            d.FriendlyName, d.Name, d.DevicePath, d.Size, d.Health);
    }
}


foreach (Disk d in vds.UnallocatedDisks)
{
    Console.WriteLine("Found unused disk {0} {1} {2} {3} {4}",
        d.FriendlyName, d.Name, d.DevicePath, d.Size, d.Health);
}

Among other things, Disk.Heath is a property that allows you to ferret out disks that are missing or unhealthy. From the reflector:

public enum Health

{
    Unknown = 0,
    Healthy = 1,
    Rebuilding = 2,
    Stale = 3,
    Failing = 4,
    FailingRedundancy = 5,
    FailedRedundancy = 6,
    FailedRedundancyFailing = 7,
    Failed = 8,
}

The pack also contains volumes, e.g.:

foreach (Volume v in p.Volumes)

{
    Console.WriteLine("Found volume {0}: {1} {2} {3} {4}",
        v.DriveLetter, v.Name, v.Type, v.Size, v.Health);
}

And, of course, given a disk, it is easy to figure out which volumes are on this disk (in case of RAID arrays, of course, the volume might not be on just one disk of course). A Disk structure has a list of extents in it. An extent has a volume id, which can be translated into a Volume object:

foreach (DiskExtent de in d.Extents)

{
    Volume v = vds.GetObject<Volume>(de.DiskId);
}

The reverse - finding out what disks a volume occupies - is only slightly more involved:

foreach (VolumePlex vp in v.Plexes)

{
    foreach (DiskExtent de in vp.Extents)
    {
        Disk d = vds.GetObject<Disk>(de.DiskId);
    }
}

Another useful piece of functionality in managed VDS interface is ability to replace a dead disk easily by using Pack.BeginReplaceDisk.

Time for a complete example. The following program scans a disk system on a running computer, finds disks that went missing/unhealthy, and sends an email to a pre-defined account.

Program.cs:

//-----------------------------------------------------------------------

// <copyright>
// Copyright (C) Sergey Solyanik.
//
// This file is subject to the terms and conditions of the Microsoft Public License (MS-PL).
// See http://www.microsoft.com/opensource/licenses.mspx#Ms-PL for more details.
// </copyright>
//-----------------------------------------------------------------------


using System;
using System.Collections.Generic;
using System.Configuration;
using System.Linq;
using System.Net.Mail;
using System.Text;

using Microsoft.Storage.Vds;

namespace DiskMonitor
{
    /// <summary>
    /// Implements a simple disk monitor. The program is a periodic task
    /// that looks for an unhealthy disk and sends an email if it finds one.
    /// </summary>
    class Program
    {
        /// <summary>
        /// Main program.
        /// </summary>
        /// <param name="args"> Arguments from the command line. None expected. </param>
        static void Main(string[] args)
        {
            ServiceLoader loader = new ServiceLoader();
            Service vds = loader.LoadService(null);
            vds.WaitForServiceReady();


            StringBuilder sb = new StringBuilder();

            SoftwareProvider dynamicProvider = GetDynamicProvider(vds);
            foreach (Pack p in dynamicProvider.Packs)
            {
                foreach (Disk d in p.Disks)
                {
                    if (d.Health != Health.Healthy && d.Health != Health.Rebuilding)
                    {
                        sb.Append(string.Format("Disk {0} has health status {1}\n",
                            d.Name, d.Health));
                    }
                }
            }


            SoftwareProvider basicProvider = GetBasicProvider(vds);
            foreach (Pack p in basicProvider.Packs)
            {
                foreach (Disk d in p.Disks)
                {
                    if (d.Health != Health.Healthy && d.Health != Health.Rebuilding)
                    {
                        sb.Append(string.Format("Disk {0} has health status {1}\n",
                            d.Name, d.Health));
                    }
                }
            }

            if (sb.Length != 0)
            {
                Console.Error.Write(sb);
            }
            else
            {
                Console.WriteLine("Everything is great!");
            }

            MailMessage m = new MailMessage();
            m.To.Add(ConfigurationManager.AppSettings["NotificationEmail"]);
            m.From = new MailAddress(
           ConfigurationManager.AppSettings["FromEmail"]);
           m.Sender = new MailAddress(
           ConfigurationManager.AppSettings["FromEmail"]);
           m.IsBodyHtml = false;

            if (sb.Length != 0)
            {
                m.Subject = string.Format(
                    "Problems with disk array on {0}", Environment.MachineName);
                m.Body = sb.ToString();
            }
            else if (bool.Parse(ConfigurationManager.AppSettings["VerboseMail"]))
            {
                m.Subject = string.Format(
                    "Checked out disk system on {0}: everything is healthy!",
                    Environment.MachineName);
                m.Body = "Nothing to report.";
            }


            SmtpClient client = new SmtpClient(
            ConfigurationManager.AppSettings["SmtpServer"]);
            client.UseDefaultCredentials = true;
            client.EnableSsl = bool.Parse(
                ConfigurationManager.AppSettings["UseSslForSmtp"]);
            client.Send(m);
        }
        /// <summary>
        /// Finds the dynamic software provider.
        /// </summary>
        /// <param name="vds"> VDS service. </param>
        /// <returns> Basic software provider or null of none exists. </returns>
        private static SoftwareProvider GetDynamicProvider(Service vds)
        {
            foreach (Provider p in vds.Providers)
            {
                if ((p.Flags & ProviderFlags.Dynamic) != 0)
                    return p as SoftwareProvider;
            }
            return null;
        }

        /// <summary>
        /// Finds the basic software provider.
        /// </summary>
        /// <param name="vds"> VDS service. </param>
        /// <returns> Basic software provider or null of none exists. </returns>
        private static SoftwareProvider GetBasicProvider(Service vds)
        {
            foreach (Provider p in vds.Providers)
            {
                if ((p.Flags & (ProviderFlags.Dynamic |
                        ProviderFlags.InternalHardwareProvider)) == 0)
                    return p as SoftwareProvider;
            }
            return null;
        }
    }
}

app.config:

<?xml version="1.0" encoding="utf-8" ?>

<configuration>
  <appSettings>
    <add key="SmtpServer" value="smtp.server.domain.com" />
    <add key="NotificationEmail" value="target@domain.com" />
    <add key="FromEmail" value="user@domain.com" />
    <add key="UseSslForSmtp" value="true" />
    <add key="VerboseMail" value="true" />
  </appSettings>
</configuration>

app.manifest:

<?xml version="1.0" encoding="utf-8"?>
<asmv1:assembly manifestVersion="1.0" xmlns="urn:schemas-microsoft-com:asm.v1" xmlns:asmv1="urn:schemas-microsoft-com:asm.v1" xmlns:asmv2="urn:schemas-microsoft-com:asm.v2" xmlns:xsi=http://www.w3.org/2001/XMLSchema-instance>
  <assemblyIdentity version="1.0.0.0" name="DiskMonitor"/>
  <trustInfo xmlns="urn:schemas-microsoft-com:asm.v2">
    <security>
      <requestedPrivileges xmlns="urn:schemas-microsoft-com:asm.v3">
        <requestedExecutionLevel level="requireAdministrator" uiAccess="false" />
      </requestedPrivileges>
    </security>
  </trustInfo>
</asmv1:assembly>