Real-Time Error Notifications in Basecamp 3

One of the most important aspect of monitoring your application for errors is to instantly get notified when a hiccup occurs. Getting access to this information quickly can help you and your team jump on an issue immediately, if need be.

For instance - if we take Frontdoor for example - imagine that a landlord confirmed a showing request with a user, which is due in a few hours. However, an error occurred in the request (yikes!) and the user was never notified. Needless to say that, in this case, it is important that I jump on this instantly and make sure that the user is notified on time, so that they can show up to the visit.

PS: don't worry - this example is purely fictional :)

Here's an overview of how I integrated Sentry in our node.js backend.

Adding a Monitoring Library in an Existing Project

Before I discuss the notifications within Basecamp, I'd like to go over how I integrated a monitoring library in our app. For our uses at Frontdoor, I decided to use Sentry, mainly because I had some previous experience with it and I like their product.

However, I had only been using Winston up until now and I only added Sentry quite well into the project, which made things a little more interesting. Since I did not want to change every call to the logger across my app, I created a wrapper for it, made sure to keep the methods the same same, and simply replaced the require from Winston to my new wrapper.

On top of not having to change my calls to the logger across the whole app, creating my own wrapper gave me more flexibility and opportunities than simply logging. For instance, calling logger.info() still outputs to a log file, but calling logger.error() now outputs to a log file, and sends the error to Sentry and Basecamp. I also customized it so that logger.warn() outputs to a log file and sends the message to Basecamp, without sending it to Sentry.

Integrating with Basecamp 3

Adding Sentry was pretty useful as is, but I felt the need of integrating it with Basecamp 3, so that I can get notified of errors in real-time, in a tool I use very often. It's nice that Sentry sends me emails when an error occurs, but I tend to check my Basecamp notifications more often than I check my emails, and emails can easily be missed in your inbox. Also, Chatbots were just released by Basecamp and I really wanted to give it a spin.

What I used for this integration were Basecamp's Reporting chatbots, which, to quote Basecamp, are bots that just send information to Campfire on their own without user interaction. Using chatbots, you basically get a webhook specific to your Basecamp project that you can use to report information to through a POST request. The request's body is a simple JSON object:

json: {
 'content': '<b>' + message + '</b>'
}

All in all, sending a message to your Basecamp looks like this:

var options = {
    url: config.Basecamp.alerts,
    method: 'POST',
    json: {
        'content': '<b>' + message + '</b>'
    }
};

request.post(options, function (err, response) {
    if (err) {
        logger.error('Failed to notify Basecamp. StatusCode: %s. Error: %s.', response.statusCode, err);
        return;
    }
});

I highly recommend sending these alerts into a separate Basecamp that is not used by humans in order to minimize the extra noise and make sure that the alerts stand out.

At Frontdoor, I created a new Basecamp called Alerts that is explicitly used for bots and alerts.

Custom Wrapper and Conclusion

For the curious ones who would like to see the full code, below is the custom wrapper that I mentioned earlier. It was carefully built so that I did not have to change my logging calls in the app (e.g. logging.info()), and makes sure to funnel the correct error messages to Sentry and / or Basecamp, as needed.

In conclusion, writing your own wrapper is not complicated and allows you to integrate seamlessly with third parties, such as Sentry and Basecamp, in order to stay on top of any error that may occur in production.

var config = require('./config/global.json');
var raven = require('raven');
var request = require('request');
var util = require('util');
var winston = require('winston');

winston.loggers.add('mainLogger', {
    console: {
        level: 'debug',
        colorize: true
    },
    file: {
        filename: './logs/yourLogFile.log',
        handleExceptions: true,
        humanReadableUnhandledException: true
    }
});

var logger = winston.loggers.get('mainLogger');

var client = null;

function initSentry(app) {
    client = new raven.Client(config.SentryDsn);

    // error sending to Sentry
    client.on('error', function (e) {
        logger.error(e);
    });

    app.use(raven.middleware.express.requestHandler(config.SentryDsn));
}

function initSentryErrorHandler(app) {
    app.use(raven.middleware.express.errorHandler(config.SentryDsn));
}

function notifyBasecamp(message) {
    if (!config.Basecamp || !config.Basecamp.alerts || config.Environment === 'development') {
        return;
    }
    var options = {
        url: config.Basecamp.alerts,
        method: 'POST',
        json: {
            'content': '<b>' + message + '</b>'
        }
    };
    request.post(options, function (err, response) {
        if (err) {
            logger.error('Failed to notify Basecamp. StatusCode: %s. Error: %s.', response.statusCode, err);
            return;
        }
    });
}

function logMessage(type, messages) {
    if (!logger || !type || !messages || messages.length < 1)
        return;
    var message = util.format.apply(null, messages);
    if (type === 'debug')
        logger.debug(message);
    else if (type === 'error') {
        logger.error(message);
        if (client)
            client.captureException(message);
        notifyBasecamp(message);
    }
    else if (type === 'warn') {
        logger.warn(message);
        notifyBasecamp(message);
    }
    else if (type === 'info')
        logger.info(message);
}

function debug() {
    logMessage('debug', arguments);
}

function info() {
    logMessage('info', arguments);
}

function error() {
    logMessage('error', arguments);
}

function warn() {
    logMessage('warn', arguments);
}

exports.debug = debug;
exports.error = error;
exports.info = info;
exports.initSentry = initSentry;
exports.initSentryErrorHandler = initSentryErrorHandler;
exports.warn = warn;