Formatting a large RAID-5 drive

Old hard drives - I have had them for 3 years - on my servers at home started to fail, so I had to replace one RAID-5 array. I bought 5 2TB drives and used software RAID-5 function in Windows Server 2008 R2 to create a monstrous 7.5TB (usable space) drive out of these 5 disks.

I only use soft RAID because the disks can be read by any Windows Server, whereas hardware RAID protects well against a drive failure, but if it is the controller that dies, you may be out of luck - who knows if this model would be even available a few years down the road, and most controllers use proprietary data formats, so the disk arrays are not portable between them.

Anyway, I built the array and starting formatting it. After the first 24 hours, the format was 13% complete. Next day (today) it is up to 26%.

Puzzle for the readers - how long do you think the format is going to take in total? Do not post the reasoning, just the number :-). I will post the final answer (and why) when the format completes.

Dictionary attacks on my home network

I am running Small Business Server 2003 at home. One service that it provides is sending daily reports about what happened to the system in the last 24 hours.

This report includes summaries of the event logs, and in the last several days the security logs were overflowing with logon audit failures caused by what looks like a distributed dictionary attack on my server.

The attack goes as follows: for a few hours someone (a script, really) is trying to connect to the system using various "likely" account names like aloha, admin, Administrator, master, root, randy, etc. The logon attempt for every user name repeats a few dozen times, most likely with different passwords, although security logs obviously don't show them, and then the user name changes.

After a few thousand attempts the attack subsides only to resume the next day from a different IP address.

Obviously, all the passwords used by our family are rather complex, and it is very unlikely that the thing will ever guess them, but it's discomforting nevertheless, so today I decided to get rid of the attackers altogether.

Small Business Server 2003 does not include standard Windows Firewall, instead it uses filters in its routing service to block unwelcome traffic.

The UI for this "firewall" of sorts can be accessed via Routing and Remote Access MMC available in the Administrative Tools menu. From there, one can expand IP Routing, the NAT/Basic Firewall, and double-clicking on the interface name brings up a properties dialog box with the "Inbound Filters" button.

I had no idea how to access it programmatically. Luckily, most of Windows networking management is scriptable through netsh, and netsh has a very nice property called dump which prints out a script that could be used to reproduce the current configuration.

So I created a rule with the fake firewall rule with an address that I could recognize, ran netsh dump, and searched the output for that ip address. As it turns out then, to add a firewall rule on Server 2003 with routing enabled, this is what needs to be done:

routing ip add filter name="Network Connection"
    filtertype=INPUT srcaddr=xx.xx.xx.xx srcmask=255.255.255.255
    dstaddr=0.0.0.0 dstmask=0.0.0.0 proto=ANY

where xx.xx.xx.xx is the placeholder for the ip address.

Poking around a bit more, it turned out that to list the existing rules (thus determining which IP addresses have already been blocked), one could do this:

routing ip show filter name="Network Connection"

The output is a table which can easily be picked apart with a regular expression:

private static Regex presentIps = new Regex(
    @"^\s*(\d{1,3}\.\d{1,3}\.\d{1,3}\.\d{1,3})\s+255.255.255.255\s+.*$",
    RegexOptions.Multiline);
private static HashSet<string> GetCurrentlyBlockedIps()
{
    string filters = NetSh(
        "routing ip show filter name=\"Network Connection\"");
    MatchCollection allMatches = presentIps.Matches(filters);
    HashSet<string> blockedIps = new HashSet<string>();
    foreach (Match m in allMatches)
    {
        Console.WriteLine("This ip is currently being blocked: {0}",
            m.Groups[1].Value);
        blockedIps.Add(m.Groups[1].Value);
    }
    return blockedIps;
}

NetSh() is the function that runs netsh.exe. It's a bit complex because it reads both STDOUT and STDERR. This has to be done asynchronously or else a read operation on STDOUT might block while STDERR overflows:

private delegate string ReaderDelegate();
private static string NetSh(string arguments)
{
    Process process = new Process();
    process.StartInfo.Arguments = arguments;
    process.StartInfo.CreateNoWindow = true;
    process.StartInfo.FileName = "netsh.exe";
    process.StartInfo.RedirectStandardError = true;
    process.StartInfo.RedirectStandardOutput = true;
    process.StartInfo.UseShellExecute = false;

    process.Start();

    ReaderDelegate stdoutReader =
        new ReaderDelegate(process.StandardOutput.ReadToEnd);
    ReaderDelegate stderrReader =
        new ReaderDelegate(process.StandardError.ReadToEnd);
    IAsyncResult stdoutResult = stdoutReader.BeginInvoke(null, null);
    IAsyncResult stderrResult = stderrReader.BeginInvoke(null, null);

    WaitHandle[] handles =
    {
        stdoutResult.AsyncWaitHandle,
        stderrResult.AsyncWaitHandle
    };

    if (!WaitHandle.WaitAll(handles))
        throw new Exception("netsh.exe was aborted");

    string stdout = stdoutReader.EndInvoke(stdoutResult);
    string stderr = stderrReader.EndInvoke(stderrResult);

    process.WaitForExit();

    if (!string.IsNullOrEmpty(stderr))
    {
        Console.Error.WriteLine(
            "Failed netsh {0}", process.StartInfo.Arguments);

        if (stdout != null)
            Console.Error.WriteLine("{0}", stdout);

        Console.Error.WriteLine("{0}", stderr);

        throw new Exception("netsh.exe failed");
    }

    process.Dispose();

    return stdout;
}

We're almost there. All I now need to do is to walk the security event log, picking out the IP address of the attacker out of the relevant entries:

using (EventLog ev = new EventLog("Security"))
{
    EventLogEntryCollection entries = ev.Entries;
    for (int index = entries.Count - 1; index >= 0; index--)
    {
        EventLogEntry ele = entries[index];
...
        if (ele.CategoryNumber != 2 || ele.InstanceId != 529 ||
            ele.EntryType != EventLogEntryType.FailureAudit)
            continue;

        if (ele.ReplacementStrings.Length < 11 ||
            knownGoodIps.IsMatch(ele.ReplacementStrings[11]))
            continue;

        string ip = ele.ReplacementStrings[11];
...

If the logon failure happens, say, more then 5 times in two minutes, we simply block out the whole ip address from accessing the server, as follows (you'll notice that I maintain a queue of events for all relevant ip addresses that allows me to detect the number of failures per time interval):

if (blockedIps.Contains(ip))
    continue;

LinkedList<DateTime> queue;
if (!trackedIps.ContainsKey(ip))
{
    queue = new LinkedList<DateTime>();
    queue.AddFirst(ele.TimeGenerated);
    trackedIps[ip] = queue;

    continue;
}

queue = trackedIps[ip];
queue.AddFirst(ele.TimeGenerated);

if (queue.Count < failEvents)
    continue;

while (queue.Count > failEvents)
    queue.RemoveLast();

TimeSpan period = queue.Last.Value - queue.First.Value;
if (period.Milliseconds > failPeriod)
    continue;

blockedIps.Add(ip);

NetSh("routing ip add filter name=\"Network Connection\" filtertype=INPUT srcaddr="
    + ip + " srcmask=255.255.255.255 dstaddr=0.0.0.0 dstmask=0.0.0.0 proto=ANY");

The rest is just accounting. Here's the full program for your enjoyment:

//-----------------------------------------------------------------------
// <copyright>
// Copyright (C) Sergey Solyanik.
//
// This file is subject to the terms and conditions of the Microsoft Public License (MS-PL).
// See http://www.microsoft.com/opensource/licenses.mspx#Ms-PL for more details.
// </copyright>
//----------------------------------------------------------------------- 
using System;
using System.Diagnostics;
using System.Collections.Generic;
using System.Configuration;
using System.Linq;
using System.Net.Mail;
using System.Text;
using System.Text.RegularExpressions;
using System.Threading;

namespace Defender
{
    /// <summary>
    /// This implements code that scans security event log and uses Windows Server 2003
    /// ip routing filters to block dictionary attacks.
    /// </summary>
    class Program
    {
        /// <summary>
        /// Regular expression that parses existing filters out of
        /// netsh ip show filter output.
        /// </summary>
        private static Regex presentIps = new Regex(
            @"^\s*(\d{1,3}\.\d{1,3}\.\d{1,3}\.\d{1,3})\s+255.255.255.255\s+.*$",
            RegexOptions.Multiline);

        /// <summary>
        /// Delegate used to read process output.
        /// </summary>
        /// <returns> STDOUT/ERR stream converted to a string. </returns>
        private delegate string ReaderDelegate();

        /// <summary>
        /// Runs netsh.exe with the given argument. Throws if there is an error, else
        /// returns the output as one string.
        /// </summary>
        /// <param name="arguments"> netsh.exe command arguments. </param>
        /// <returns> STDOUT converted to one string. </returns>
        private static string NetSh(string arguments)
        {
            Process process = new Process();
            process.StartInfo.Arguments = arguments;
            process.StartInfo.CreateNoWindow = true;
            process.StartInfo.FileName = "netsh.exe";
            process.StartInfo.RedirectStandardError = true;
            process.StartInfo.RedirectStandardOutput = true;
            process.StartInfo.UseShellExecute = false;

            process.Start();

            ReaderDelegate stdoutReader =
                new ReaderDelegate(process.StandardOutput.ReadToEnd);
            ReaderDelegate stderrReader =
                new ReaderDelegate(process.StandardError.ReadToEnd);
            IAsyncResult stdoutResult = stdoutReader.BeginInvoke(null, null);
            IAsyncResult stderrResult = stderrReader.BeginInvoke(null, null);

            WaitHandle[] handles =
            {
                stdoutResult.AsyncWaitHandle,
                stderrResult.AsyncWaitHandle
            };

            if (!WaitHandle.WaitAll(handles))
            {
                throw new Exception("netsh.exe was aborted");
            }

            string stdout = stdoutReader.EndInvoke(stdoutResult);
            string stderr = stderrReader.EndInvoke(stderrResult);

            process.WaitForExit();

            if (!string.IsNullOrEmpty(stderr))
            {
                Console.Error.WriteLine(
                    "Failed netsh {0}", process.StartInfo.Arguments);

                if (stdout != null)
                    Console.Error.WriteLine("{0}", stdout);

                Console.Error.WriteLine("{0}", stderr);

                throw new Exception("netsh.exe failed");
            }

            process.Dispose();

            return stdout;
        }

        /// <summary>
        /// Gets the collection of ips that are already blocked.
        /// </summary>
        /// <returns> HashSet of blocked ips. </returns>
        private static HashSet<string> GetCurrentlyBlockedIps()
        {
            string filters = NetSh(
                "routing ip show filter name=\"Network Connection\"");
            MatchCollection allMatches = presentIps.Matches(filters);
            HashSet<string> blockedIps = new HashSet<string>();
            foreach (Match m in allMatches)
            {
                Console.WriteLine("This ip is currently being blocked: {0}",
                    m.Groups[1].Value);
                blockedIps.Add(m.Groups[1].Value);
            }

            return blockedIps;
        }

        /// <summary>
        /// Convers a comma-separated list of "known good" ips into a filtering
        /// regular expression.
        /// </summary>
        /// <param name="ipList">Comma-separated list of known-good ips.</param>
        /// <returns>Regular expression that matches this list.</returns>
        private static Regex BuildRegexForIpList(string ipList)
        {
            return new Regex(
                "^(" + ipList.Replace(".", "\\.").Replace(',', '|') + ").*$");
        }

        /// <summary>
        /// Sends notification email.
        /// </summary>
        /// <param name="mailBody"> Body of the email. </param>
        private static void SendMail(StringBuilder mailBody)
        {
            MailMessage email = new MailMessage();
            email.To.Add(ConfigurationSettings.AppSettings["NotificationEmail"]);
            email.Subject = "New attack(s) detected. IPs blocked.";
            email.From = new MailAddress(
                ConfigurationSettings.AppSettings["FromEmail"]);
            email.Sender = new MailAddress(
                ConfigurationSettings.AppSettings["FromEmail"]);
            email.Body = mailBody.ToString();
            email.IsBodyHtml = false;

            SmtpClient client = new SmtpClient(
                ConfigurationSettings.AppSettings["SmtpServer"]);
            client.UseDefaultCredentials = true;
            client.EnableSsl = bool.Parse(
                ConfigurationSettings.AppSettings["UseSslForSmtp"]);

            client.Send(email);
        }

        /// <summary>
        /// Runs one round of periodic processing.
        /// </summary>
        /// <param name="lastProcessedIndex"> Previously seen event index. Processes
        /// all events that are newer than this. </param>
        /// <returns> The new watermark for event index. </returns>
        private static int Process(int lastProcessedIndex)
        {

            StringBuilder mailBody = new StringBuilder();

            int failEvents = int.Parse(
                ConfigurationSettings.AppSettings["LogonFailuresPerPeriod"]);
            int failPeriod = 1000 * 60 * int.Parse(
                ConfigurationSettings.AppSettings["FailurePeriodMinutes"]);
            Regex knownGoodIps = BuildRegexForIpList(
                ConfigurationSettings.AppSettings["KnownGoodIpList"]);

            Dictionary<string, LinkedList<DateTime>> trackedIps =
                new Dictionary<string, LinkedList<DateTime>>();

            HashSet<string> blockedIps = GetCurrentlyBlockedIps();
            
            Dictionary<string, List<string>> auditFailures =
                new Dictionary<string, List<string>>();

            HashSet<string> newlyBlockedIps = new HashSet<string>();

            using (EventLog ev = new EventLog("Security"))
            {
                EventLogEntryCollection entries = ev.Entries;

                for (int index = entries.Count - 1; index >= 0; index--)
                {
                    EventLogEntry ele = entries[index];
                    if (ele.Index < lastProcessedIndex)
                        break;

                    if (ele.CategoryNumber != 2 || ele.InstanceId != 529 ||
                        ele.EntryType != EventLogEntryType.FailureAudit)
                        continue;

                    if (ele.ReplacementStrings.Length < 11 ||
                        knownGoodIps.IsMatch(ele.ReplacementStrings[11]))
                        continue;

                    string ip = ele.ReplacementStrings[11];

                    if (!auditFailures.ContainsKey(ip))
                        auditFailures[ip] = new List<string>();

                    auditFailures[ip].Add(ele.Message);

                    if (blockedIps.Contains(ip))
                        continue;

                    LinkedList<DateTime> queue;
                    if (!trackedIps.ContainsKey(ip))
                    {
                        queue = new LinkedList<DateTime>();
                        queue.AddFirst(ele.TimeGenerated);
                        trackedIps[ip] = queue;

                        continue;
                    }

                    queue = trackedIps[ip];
                    queue.AddFirst(ele.TimeGenerated);

                    if (queue.Count < failEvents)
                        continue;

                    while (queue.Count > failEvents)
                        queue.RemoveLast();

                    TimeSpan period = queue.Last.Value - queue.First.Value;
                    if (period.Milliseconds > failPeriod)
                        continue;

                    string msg = string.Format(
                        "{0} Adding the following ip to blocked list: {1}",
                        DateTime.Now, ip);

                    blockedIps.Add(ip);
                    newlyBlockedIps.Add(ip);

                    NetSh("routing ip add filter name=\"Network Connection\" filtertype=INPUT srcaddr="
                        + ip + " srcmask=255.255.255.255 dstaddr=0.0.0.0 dstmask=0.0.0.0 proto=ANY");

                    Console.WriteLine(msg);
                    mailBody.Append(msg);
                    mailBody.Append("\r\n");
                }

                lastProcessedIndex = entries[entries.Count - 1].Index;
            }

            if (mailBody.Length > 0)
            {
                foreach (string ip in newlyBlockedIps)
                {
                    foreach (string message in auditFailures[ip])
                    {
                        mailBody.Append("-----\r\n");
                        mailBody.Append(message);
                    }
                }

                SendMail(mailBody);
            }

            return lastProcessedIndex;
        }

        /// <summary>
        /// Main entry point, does all the work.
        /// </summary>
        /// <param name="args"> Program arguments. Not used, all configuration is
        /// through app.config. </param>
        static void Main(string[] args)
        {
            int lastProcessedIndex = 0;
            for (; ; )
            {
                lastProcessedIndex = Process(lastProcessedIndex);

                GC.Collect();

                int sleep = int.Parse(
                    ConfigurationSettings.AppSettings["SleepIntervalMinutes"]);
                Console.WriteLine("Sleeping for {0} minutes @ {1}...",
                    sleep, lastProcessedIndex);
                Thread.Sleep(sleep * 60 * 1000);
            }
        }
    }
}

It also requires an app.config with the following settings:

<?xml version="1.0" encoding="utf-8" ?>
<configuration>
  <appSettings>
    <add key="KnownGoodIpList" value="192.168.,131.107."/>
    <add key="LogonFailuresPerPeriod" value="5" />
    <add key="FailurePeriodMinutes" value="2" />
    <add key="SmtpServer" value="xxx" />
    <add key="NotificationEmail" value="yyy" />
    <add key="FromEmail" value="zzz" />
    <add key="UseSslForSmtp" value="true" />
    <add key="SleepIntervalMinutes" value="2"/>
  </appSettings>
</configuration>

The whole thing took barely 2 hours, and have stopped short two attacks just this evening!

J. K. Rowling on elections (via Reddit)

http://www.timesonline.co.uk/tol/comment/columnists/guest_contributors/article7096786.ece?print=yes