Thursday, December 18, 2008

Simple ICMP Heartbeat Monitor

I needed to make sure that if one of my servers goes down I know about it in the next 15 minutes so I wrote a little ICMP monitor program. Why write another ping utility when there are tons of tools available already, which also reveal more information about what went wrong than a simple ping tool can do?

Well, I started from an Internet research, which didn’t satisfy me quite. The tools I found either had reporting features which didn’t fit what I needed, or were too hard to operate or set up, or were pricey. So I spent 2 evenings and wrote my own tool just like a carpenter would put something simple together out of the stuff in his garage saving himself a trip to a Home Depot. My tool did what I needed and ended up being a decent exercise with .NET FCL so it is worth blogging about. I hope it can save someone the time. You can download the binaries here, and the source code here.

Features

The tool is a console application named Monitor.exe, which is intended to be invoked primarily by a Windows scheduled task and has these characteristics:
  • It sends ICMP ping requests to each IP address in a list of addresses set in its configuration file.

  • If ping to at least one address fails then it sends an Email notification to a preconfigured destination. The message includes succeeded and failed IPs, and a NetBIOS name of the server which the ping request has originated from.

  • In order to avoid an "alert storm" when there are ping errors, the tool will send a maximum of 4 notifications, keeping track of them in current user environment variable, which gets reset automatically once all configured IPs are pinged successfully.

  • Sensitive Email account information is encrypted in configuration file. The monitor utility uses Windows DPAPI for handling protection of this information. A utility named Cipher.exe allows encrypting the information so it can be placed into monitor’s config file, and the application itself uses DPAPI to decrypt it.


Installation

Installation is simple, yet it requires editing configuration file and encrypting a few pieces of information. I use Gmail as my SMTP server – they kindly allow relaying messages through their smtp.gmail.com server through port 587. It’s pretty convenient. Since I happen to have an iPhone I get my Email notification in my pocket within 15 minutes of a notification event.

So here are all the installation steps you need to follow:
  1. Extract binaries to a folder of your choice.

  2. You need to encrypt 4 pieces of information for the Email to work.

    • Sender email address.

    • Destination email address.

    • SMTP server login account.

    • SMTP server password.


    Open up command prompt and navigate to the folder where you extracted the binaries to. Use the Cipher.exe utility to encrypt the information. The generated cipher is quite long so it may be hard to copy it directly from command prompt. You can pipe the output to a text file and then copy it from there. For example:

    Cipher.exe sender@mycompany.com > sender.txt
    Cipher.exe destination@destination.com > destination.txt
    Cipher.exe SomeAccountLogin > login.txt
    Cipher.exe SomePassword > pwd.txt

  3. Make sure that the list of IP addresses to ping is proper, as well as the email server address, port, message subject template and body template. Here is an example config file which uses Gmail SMTP server and Email account:
    <configuration>
    <appSettings>
    <add key="IPsToCheck" value="192.168.0.1, 192.168.0.2, 192.168.0.3"/>
    <add key="MailServerAddress" value="smtp.gmail.com"/>
    <add key="MailServerPort" value="587"/>
    <add key="MessageSubjectTemplate" value="Ping failure alert from {0} #{1} of {2}"/>
    <add key="MessageBodyTemplate" value="Heartbeat ping errors have occurred:{0}."/>
    <add key="EncryptedSenderEmail" value="Insert encrypted value"/>
    <add key="EncryptedDestinationEmail" value="Insert encrypted value"/>
    <add key="EncryptedMailServerLogin" value="Insert encrypted value"/>
    <add key="EncryptedMailServerPassword" value="Insert encrypted value"/>
    </appSettings>
    </configuration>
  4. Configure scheduled task to run the Monitor.exe program. Make sure that firewall does not block configured SMTP server port.

  5. A good way to test whether Email notification is working is to temporarily enter at least one IP, which times out on an ICMP request to configuration file, then run the Monitor.exe from command line. This should trigger an Email message.



How it Works

The program is quite simple, yet there are 4 areas in it worth a note – ping implementation, dispatching notifications, security measures and diagnostics.


Ping Implementation

To ping a remote IP I have used WMI Win32_PingStatus class. Using mgmtclassgen.exe command-line tool coming with Visual Studio I have generated a PingStatus.cs C# proxy class abstracting WMI query generation and invokation, thanks to the great blog post by Daniel Velasquez.
On top of that I have my Icmp class, which lets me decouple business rules and WMI implementation even further. This is how the Icmp class looks like:

static class Icmp
{
private const int PingRetriesCount = 4;
private const int PingRetryInterval = 1000;
private const int PingRequiredSuccessCount = 2;


public static bool Ping(string ipAddress)
{
string condition = String.Format(
"Address='{0}'",
ipAddress);
int pingSucceesses = 0;

for (int i = 0; i < PingRetriesCount; ++i)
{
PingStatus.PingStatusCollection instances = PingStatus.
GetInstances(condition);

IEnumerator enumerator = instances.GetEnumerator();

if (enumerator.MoveNext())
{
using (PingStatus instance = (PingStatus)enumerator.Current)
{
if (PingStatus.StatusCodeValues.Success ==
instance.StatusCode)
{
pingSucceesses++;
}
}
}

if (PingRequiredSuccessCount == pingSucceesses)
{
return true;
}
else
{
Thread.Sleep(PingRetryInterval);
}
}

return false;
}
}
As you can see, the Icmp.Ping() method returns truth if it manages to get successful response back from the destination twice out of maximum of 4 attempts, and fails otherwise.

Dispatching Notifications

The application calls Icmp.Ping() method for each of the configured IP addresses and increments error counter once it receives false back. If error count is greater than 0, and if number of previous Email notifications is less than 4 then notification Email is constructed and sent out. The number of notifications is stored in user environment variable and is updated with each successful notification dispatch:

if (numberOfErrors > 0)
{
string value = Environment.GetEnvironmentVariable(
ErrorCountEnvironmentVariable,
EnvironmentVariableTarget.User);
int numberOfNotifications = 0;
Int32.TryParse(value, out numberOfNotifications);

if (numberOfNotifications < MaxNumberOfErrorNotifications)
{
numberOfNotifications++;

string failedIPs = failures.ToString();
failedIPs = failedIPs.Remove(
failedIPs.LastIndexOf(','));
string succeededIPs = successes.ToString();
succeededIPs = succeededIPs.Remove(
succeededIPs.LastIndexOf(','));

Email.SendErrorReport(
Environment.MachineName,
numberOfSuccesses,
numberOfErrors,
succeededIPs,
failedIPs,
numberOfNotifications,
MaxNumberOfErrorNotifications);

Environment.SetEnvironmentVariable(
ErrorCountEnvironmentVariable,
numberOfNotifications.ToString(),
EnvironmentVariableTarget.User);
}

Security Measures

The complexity of encryption is abstracted from us by the System.Security.Cryptography.ProtectedData class, for which I have written a simple facade type named Protection and located in the Cryptography.dll assembly. This DLL is shared between Monitor.exe, Cipher.exe and Decipher.exe. Below is the simple implenentation of the type:

public static class Protection
{
public static string Encrypt(string input)
{
byte[] encryptedData = ProtectedData.Protect(
Encoding.Unicode.GetBytes(input),
GetEntropy(),
DataProtectionScope.CurrentUser);
return Convert.ToBase64String(encryptedData);
}

public static string Decrypt(string input)
{
byte[] decrypted = ProtectedData.Unprotect(
Convert.FromBase64String(input),
GetEntropy(),
DataProtectionScope.CurrentUser);
string decryptedString = Encoding.Unicode.GetString(decrypted);
return decryptedString;
}

private static byte[] GetEntropy()
{
byte[] salt = Encoding.Unicode.GetBytes(
Assembly.GetExecutingAssembly().FullName);
return salt;
}
}
Now the rather interesting question is how secure is this protection? Well, firstly DPAPI itself is secure; at least I can rely on it as much as I do on Windows for protecting my logon information. Secondly, passing in DataProtectionScope.CurrentUser enumeration value to the ProtectedData.Protect() method ensures that other users, even those with admin rights, cannot decrypt the cipher you created. If several applications run under the same Windows user account then the only thing preventing them from deciphering the secret for another application is the fact that optional entropy used by the applications should differ.

So if a malicious user knows the account credentials under which the ICMP Monitor is executed then its secret (encrypted data) can be compromised. Even if an attacker does not know Windows credentials of the ICMP Monitor process, with sufficient rights he can read the values from the process memory after they have been decrypted and stored in a System.String variable, which is also not destroyed until the next garbage collection. Yes, there is a special FCL type addressing this particular vulnerability - System.Security.SecureString, but it cannot be used with many common APIs, such as one for sending Emails using System.Net.Mail.SmtpClient type for instance.

Concluding the security notes, make sure to use such accounts and email addresses with this tool which if compromised would not cause a major adverse impact on the rest of your system.


Diagnostics

The diagnostic means for a simple tool is simple: unhandled exceptions are captured and logged into a text file, which gets overwritten each time this happens so it stores information about the last error only. The file name is IcmpMonitor_LastError.txt. It is located under
%UserProfile%\appData\Local
folder on Windows Vista and Windows 2008 Server or under %UserProfile%\Local Settings\Application Data on Windows 2003 Server or Windows XP.