Stateful AWS lambdas part3: bootstrapping like an EC2, reusing resources and node_modules in /tmp

This one falls under the category of “you should probably never do this”, but is also an interesting case study to understand better how lambda works, and how much of the infrastructure powering our lambdas actually get reused between invocations.

We already know from many clues (i.e. role permissions for certain things require the “ec2:” prefix) that our lambda hosts are EC2s, but functions probably run inside ECS containers atop EC2 clusters. We have no control over any of this, Amazon guaranteeing only that our code will execute when it is triggered, everything else fading away between those invocations.

The application

Lambda has pretty tight limits as to how much code can be uploaded into a zip/jar file (50 MB), so let’s suppose we have a very large npm dependency (i.e. phantomjs) that we don’t want to bundle and upload every time we update our function.

Could we download the file to disk, say /tmp (of which we have 512MB) and only re-download it when our container gets switched over / cleared / disappears for whatever reason? How often would that happen?

Control flow

  • Check if our node_modules in /tmp already exists
  • If it doesn’t, download the .zip package, unzip it on disk and keep going
  • If it does, just keep going

The actual code

For this example, we’ll use a zip package of lodash, but this could be anything. Make sure to change the URL to download. To keep things fast, put your file on S3 in the same region, odds are you’ll be hitting a server not too far from your lambda.

var http = require('http');
var fs = require('fs');
var _;
var spawn = require('child_process').spawn;
process.env['NODE_PATH'] = process.env['NODE_PATH'] + ':/tmp';

var lodashUrl = 'http://s3.amazonaws.com/your-bucket-name/lodash.zip'; //change this!
var lodashLocation = '/tmp/lodash.zip';
var nodeModulesLocation = '/tmp';

function doNormalStuff(callback) {
    _ = require(nodeModulesLocation+'/lodash');
    console.log(_.filter([]));
    return callback(null, 'done');
}

exports.handler = function (event, context, callback) {

    if (!fs.existsSync(nodeModulesLocation)){
        fs.mkdirSync(nodeModulesLocation); // if our tmp folder does not exist, create it
    }

    function unzip() { // unzip the downloaded file
        return new Promise(function (resolve) {
            console.log('Unzipping file');
            const unzip = spawn('unzip', ['-q', '-o', lodashLocation, '-d', nodeModulesLocation]);
            unzip.on('close', (code) => {
                console.log(`child process exited with code ${code}`);
                resolve();
            });
        })
    }

    function download(url, dest) { // download the file
        return new Promise(function (resolve, reject) {
            var file = fs.createWriteStream(dest);
            http.get(url, function (response) {
                response.pipe(file);
                file.on('finish', function () {
                    console.log('Download done.');
                    file.close(resolve);
                });
            }).on('error', reject);
        }).catch(console.log)
    }

    // our main control function
    if (!fs.existsSync(lodashLocation)) {
        console.log('File does not exist, downloading...');
        download(lodashUrl, lodashLocation)
            .then(() => console.log('Download has finished, unzipping.'))
            .then(() => unzip())
            .then(() => doNormalStuff(callback))
    }else{
        return doNormalStuff(callback);
    }
};

The results

Now the interesting part…how often did we need to re-download our bundle? Would this actually be a usable strategy to bootstrap a lambda?

I’ve setup a Cloudwatch event trigger each minute to run this function, and let it go for a few hours.

  • Invocations: 133
  • File re-downloaded, according to the logs: 5

So this ain’t bad – we only had to re-fetch the file 3-4% of the time, meaning that our container was stable for long periods of time, keeping its disk state the same.

Obviously, there are zero guarantees as to whether this would be the same for any function, but it gives interesting insight into how often state changes behind the scenes.

Leave a Reply

Fill in your details below or click an icon to log in:

WordPress.com Logo

You are commenting using your WordPress.com account. Log Out / Change )

Twitter picture

You are commenting using your Twitter account. Log Out / Change )

Facebook photo

You are commenting using your Facebook account. Log Out / Change )

Google+ photo

You are commenting using your Google+ account. Log Out / Change )

Connecting to %s