Managing Patches on AWS

During a quiet time over the Christmas break, I spent a little time looking for a better way to manage patching EC2 instances on AWS. It took a little while to put it all together so I thought this might help a few other people if I wrote it up.

We’re running EC2 instances (AWS Linux) as hosts for docker containers, so there isn’t a huge amount to patch. All the same, even a minimal install is vulnerable to attack through unpatched software (eg ssh).

Under the circumstances we want:

• automatic patching with no human intervention
• installing patches out of hours

AWS Patch Manager

Google led me to AWS Patch Manager, which looked promising.

I was far less impressed when I read the details of how this was implemented for AWS Linux. The 7 step process had a handy summary at the bottom:

Note

The equivalent yum command for this workflow is:

sudo yum update-minimal --security --bugfix 

So, essentially it’s just a wrapper around yum update. It’s really not clear how much benefit AWS patch manager brings for a smaller scale installation.

Installing the updates

Installing the updates was pretty simple:

yum update-minimal --security --bugfix -y

This installs just security patches, answering yes to all questions.

In some cases updates require a reboot. While researching this, I found this great discussion, and added this:

needs-restarting -r || shutdown -r

To put it all together into a shell script:

#!/bin/bash
yum update-minimal --security --bugfix -y
needs-restarting -r || shutdown -r now

Now we’ve got a script to install updates and handle reboots if required.

Running the update script

The scripts isn’t much help if we don’t have a way to trigger it. The most obvious thing to do is to add this as a cron task. Easy to do if you’re logged into the machine, but you really want this built into your infrastructure configuration.

For EC2, the best way to do this is through UserData scripts, configured through Cloud Formation templates. There are some great examples of this here (see the section “Parameters Section with Parameter Value Based on Pseudo Parameter”), the YAML syntax is easier to work with.

I want to drop the update script into cron.weekly, which I can do using echo.

To build the update script, this becomes:

echo "#!/bin/bash" > /etc/cron.weekly/install_updates.sh
echo "yum update-minimal --security --bugfix -y" >> /etc/cron.weekly/install_updates.sh
echo "needs-restarting -r || shutdown -r now" >> /etc/cron.weekly/install_updates.sh

There are a couple of other adjustments to make:

• needs-restarting isn’t installed by default, so I need to install that: yum install -y yum-utils
• The script needs permissions to be executed:
  chmod +x /etc/cron.weekly/install_updates.sh

The full script is:

yum install -y yum-utils
echo "#!/bin/bash" > /etc/cron.weekly/install_updates.sh
echo "yum update-minimal --security --bugfix -y" >> /etc/cron.weekly/install_updates.sh
echo "needs-restarting -r || shutdown -r now" >> /etc/cron.weekly/install_updates.sh
chmod +x /etc/cron.weekly/install_updates.sh

Controlling when it executes

One last issue, we don’t really want this installing updates at the wrong time. The simplest way to do this to adjust the hours when cron (actually anacron) runs. This means adjusting START_HOURS_RANGE and I also wanted to reduce the random interval range when the job runs within this range (RANDOM_DELAY).

It’s relatively simple to adjust the config using sed:

sed -i 's/START_HOURS_RANGE=.*/START_HOURS_RANGE=14-19/' /etc/anacrontab
sed -i 's/RANDOM_DELAY=.*/RANDOM_DELAY=20/' /etc/anacrontab

Putting it all together

The final UserData script is:

#!/bin/bash -xe
sed -i 's/START_HOURS_RANGE=.*/START_HOURS_RANGE=14-19/' /etc/anacrontab
sed -i 's/RANDOM_DELAY=.*/RANDOM_DELAY=20/' /etc/anacrontab

yum install -y yum-utils
echo "#!/bin/bash" > /etc/cron.weekly/install_updates.sh
echo "yum update-minimal --security --bugfix -y" >> /etc/cron.weekly/install_updates.sh
echo "needs-restarting -r || shutdown -r now" >> /etc/cron.weekly/install_updates.sh
chmod +x /etc/cron.weekly/install_updates.sh

Putting this in context, with the YAML template:

UserData:
  Fn::Base64: !Sub |
    #!/bin/bash -xe
    sed -i 's/START_HOURS_RANGE=.*/START_HOURS_RANGE=14-19/' /etc/anacrontab
    sed -i 's/RANDOM_DELAY=.*/RANDOM_DELAY=20/' /etc/anacrontab

    yum install -y yum-utils
    echo "#!/bin/bash" > /etc/cron.weekly/install_updates.sh
    echo "yum update-minimal --security --bugfix -y" >> /etc/cron.weekly/install_updates.sh
    echo "needs-restarting -r || shutdown -r now" >> /etc/cron.weekly/install_updates.sh
    chmod +x /etc/cron.weekly/install_updates.sh

This works fine for either individual EC2 instances or Autoscale Launch Configurations.

Final Thoughts

It’s a very long time since I’ve had to look at anything like this. The last time I looked at something like this was the early 2000s when I was hosting my own mailserver and fileserver on debian Linux.

At the time I used to add a script in cron.daily:

apt-get update
apt-get upgrade

What I find amazing is how much this hasn’t changed in over 15 years.