Working with Remote Development Teams

Over my career I’ve worked with a number of remote teams in different contexts. From this I’ve collected a few lessons I’ve learnt. Like the best lessons, some of these have been learnt the hard way.

The TLDR version is: working with remote teams is similar to working with any development team, with some additional challenges. You have the same issues as you have with other teams, just with the added distance caused by timezones, communication and culture.

This advice is based my experience of working with teams that have been split between multiple locations, offshore development contracts or remote teams. There are real benefits to improving the working relationship, even if this is a shorter term contract relationship on a fixed price contract.

If you share an office with your team, you won’t have to think about these things, they come naturally through daily contact.

1. Remote people are people too

It’s easy to treat remote people as though they’re just the service they provide. The distance makes it so much easier to treat them as just a service. You feed them jira issues/Trello cards/emails, and they deliver the work. Simple, right?

That’s not how you’d want to be treated. There are people behind the code (or testing). Don’t think of them as a machine that you feed work into and get a result.

Unless you’re a sociopath, you wouldn’t think this of people you see every day in the office. However when you have no face to face contact with people, it’s easier to de-personalise them.

You need to actively strive to understand who they are. What do they enjoy doing most? How do they like to be managed? Where do they want to go with their career? What do they do outside work?

You know, the kind of stuff you’d do with someone who works in the same office as you.

2. Actively communicate

The reality of the distance between remote and local teams is such an issue that you need to take active steps to bridge the gap. The area where this is most seen is in communication. You need to build in processes that help people to communicate. This can be a real challenge when working with developers as they’re not the most communicative people in world to start with.

Some ways you can actively build this include:

  • Have a standup (this is really a basic)
  • Schedule one on one catch ups with them
  • Use more personal forms of electronic communication. Video is better than audio, audio is better than chat, chat is better than email.
  • Use tools that make keeping in contact easy, eg slack
  • Foster a shared culture
  • Ask for their input as much as possible

Of course, the best form of communication is still face to face. So if possible, either go and meet the people of your team or have them come to meet you.

3. Include them in your plans

Communication works both ways, so you need to look at communicating to them what is on your mind. Tell them what is important to you, tell them where you’re going.

Again, if you all work in the same office, people often learn this without having to be told. They overhear conversations and hear the emotion in people’s voices.

If you need to hit a critical deadline, explain what the impact is if they miss it and ask them to surface issues early. If quality is important, then make that clear, and explain why.

It’s important to help them understand not just what you want to do, but why.

4. Understand the culture

While people are basically people no matter where they are in the world, the culture people live in has a big impact on how they see the world. This is often the culture they have grown up, been educated in and worked in. It shapes how they work.

While not everyone is the same, people from the same culture often have a lot in common. The culture they have in common might be very different to your culture. You need to work at this, as by default you tend to assume that people will approach things the way you do.

For example, as someone who has grown up in Australia, I tend to have a lower respect for authority than many other cultures. As a result I’m inclined to challenge people more senior than me. As an Australian, I assume that other people will challenge me when I’m wrong. In Asian cultures, authority tends to be much more respected. If I were managing a remote team in Asia, I’d need to be very careful to ensure I gave my team lots of space to give feedback and encourage it when it’s given.

Read up about the cultures you’re working with and be aware. One of the best questions to ask yourself is: What do they value (achievement, impact, family, education, status etc)?

5. Be Genuine

Be yourself. Don’t pretend to be someone else to try to build a connection with the team. You might have to moderate how you express yourself, but you should always be yourself.

You want to build relationships with the people you’re working with, and you’ll only do that if you genuinely show who you are.

Conclusion

Working with any team of developers is often quite challenging. Working with remote teams can make some of the challenges even harder. However with it can be a very rewarding experience, where you learn more about other cultures and your own biases.

Managing Patches on AWS

During a quiet time over the Christmas break, I spent a little time looking for a better way to manage patching EC2 instances on AWS. It took a little while to put it all together so I thought this might help a few other people if I wrote it up.

We’re running EC2 instances (AWS Linux) as hosts for docker containers, so there isn’t a huge amount to patch. All the same, even a minimal install is vulnerable to attack through unpatched software (eg ssh).

Under the circumstances we want:

  • automatic patching with no human intervention
  • installing patches out of hours

AWS Patch Manager

Google led me to AWS Patch Manager, which looked promising.

I was far less impressed when I read the details of how this was implemented for AWS Linux. The 7 step process had a handy summary at the bottom:

Note

The equivalent yum command for this workflow is:

sudo yum update-minimal --security --bugfix 

So, essentially it’s just a wrapper around yum update. It’s really not clear how much benefit AWS patch manager brings for a smaller scale installation.

Installing the updates

Installing the updates was pretty simple:

yum update-minimal --security --bugfix -y

This installs just security patches, answering yes to all questions.

In some cases updates require a reboot. While researching this, I found this great discussion, and added this:

needs-restarting -r || shutdown -r

To put it all together into a shell script:

#!/bin/bash
yum update-minimal --security --bugfix -y
needs-restarting -r || shutdown -r now

Now we’ve got a script to install updates and handle reboots if required.

Running the update script

The scripts isn’t much help if we don’t have a way to trigger it. The most obvious thing to do is to add this as a cron task. Easy to do if you’re logged into the machine, but you really want this built into your infrastructure configuration.

For EC2, the best way to do this is through UserData scripts, configured through Cloud Formation templates. There are some great examples of this here (see the section “Parameters Section with Parameter Value Based on Pseudo Parameter”), the YAML syntax is easier to work with.

I want to drop the update script into cron.weekly, which I can do using echo.

To build the update script, this becomes:

echo "#!/bin/bash" > /etc/cron.weekly/install_updates.sh
echo "yum update-minimal --security --bugfix -y" >> /etc/cron.weekly/install_updates.sh
echo "needs-restarting -r || shutdown -r now" >> /etc/cron.weekly/install_updates.sh

There are a couple of other adjustments to make:

  • needs-restarting isn’t installed by default, so I need to install that: yum install -y yum-utils
  • The script needs permissions to be executed:
    chmod +x /etc/cron.weekly/install_updates.sh

The full script is:

yum install -y yum-utils
echo "#!/bin/bash" > /etc/cron.weekly/install_updates.sh
echo "yum update-minimal --security --bugfix -y" >> /etc/cron.weekly/install_updates.sh
echo "needs-restarting -r || shutdown -r now" >> /etc/cron.weekly/install_updates.sh
chmod +x /etc/cron.weekly/install_updates.sh

Controlling when it executes

One last issue, we don’t really want this installing updates at the wrong time. The simplest way to do this to adjust the hours when cron (actually anacron) runs. This means adjusting START_HOURS_RANGE and I also wanted to reduce the random interval range when the job runs within this range (RANDOM_DELAY).

It’s relatively simple to adjust the config using sed:

sed -i 's/START_HOURS_RANGE=.*/START_HOURS_RANGE=14-19/' /etc/anacrontab
sed -i 's/RANDOM_DELAY=.*/RANDOM_DELAY=20/' /etc/anacrontab

Putting it all together

The final UserData script is:

#!/bin/bash -xe
sed -i 's/START_HOURS_RANGE=.*/START_HOURS_RANGE=14-19/' /etc/anacrontab
sed -i 's/RANDOM_DELAY=.*/RANDOM_DELAY=20/' /etc/anacrontab

yum install -y yum-utils
echo "#!/bin/bash" > /etc/cron.weekly/install_updates.sh
echo "yum update-minimal --security --bugfix -y" >> /etc/cron.weekly/install_updates.sh
echo "needs-restarting -r || shutdown -r now" >> /etc/cron.weekly/install_updates.sh
chmod +x /etc/cron.weekly/install_updates.sh

Putting this in context, with the YAML template:

UserData:
  Fn::Base64: !Sub |
    #!/bin/bash -xe
    sed -i 's/START_HOURS_RANGE=.*/START_HOURS_RANGE=14-19/' /etc/anacrontab
    sed -i 's/RANDOM_DELAY=.*/RANDOM_DELAY=20/' /etc/anacrontab

    yum install -y yum-utils
    echo "#!/bin/bash" > /etc/cron.weekly/install_updates.sh
    echo "yum update-minimal --security --bugfix -y" >> /etc/cron.weekly/install_updates.sh
    echo "needs-restarting -r || shutdown -r now" >> /etc/cron.weekly/install_updates.sh
    chmod +x /etc/cron.weekly/install_updates.sh

This works fine for either individual EC2 instances or Autoscale Launch Configurations.

Final Thoughts

It’s a very long time since I’ve had to look at anything like this. The last time I looked at something like this was the early 2000s when I was hosting my own mailserver and fileserver on debian Linux.

At the time I used to add a script in cron.daily:

apt-get update
apt-get upgrade

What I find amazing is how much this hasn’t changed in over 15 years.

Password Storage Done Right

Storing & managing passwords is really hard these days. Passwords are easier to crack. Breaches to one system mean other accounts can be exposed. Even the FBI’s most wanted can’t get it right.

Even the systems you might depend on break.

Best Practices for Passwords

Lets just review best practices for passwords:
  1. Use a complex password. A complex password is one that doesn’t include dictionary words or common phrases that might be used in a dictionary attack. This is getting hard as previously uncommon phrases can become part of a new dictionary. Ideally use a random password.
  2. Use a long password. This will help defend against dictionary attacks.
  3. Don’t use the same password. Ever. This is because if the password is broken in one system, it can be used to log into the other systems.

It needs to…

Just to make things easier, I personally use a range of platforms so I’m looking for something that runs on:
  • Windows
  • OSX
  • iOS
I also need to be able to retrieve, add and edit passwords on any platform. Changes on any one platform should sync to all other others.

It doesn’t need to…

I’m not looking for something that integrates directly into the browser. I’m concerned that something that is directly integrated into the browser is too much of a target for attack. This knocks out using something like Chrome to store passwords, aside from the issue that it can only store web based passwords.
Password based services like LastPass are also an issue, given that someone cracking LastPass can then access your entire set of passwords. These systems are a huge target.

Solution

Best practices are to use a password manager. Syncing was once a challenge, but now with platforms like google drive and dropbox are far easier.

So I use:

  1. KeePass2 – works on Windows
  2. MacPass – works on OSX
  3. KyPass3 – works on iOS
  4. Dropbox – to sync them everywhere
KyPass 3 supports syncing with a dropbox folder and the other apps just talk to the filesystem. It’s an awesome solution that lets you view / edit your passwords anywhere.

Windows 8

Now Windows 10 has been released, I feel it’s worth putting together a highly subjective review of Windows 8.

What does it feel like?

For me, from the first preview buil, Windows 8 felt like the movie From Disk till Dawn.

For those who haven’t seen the movie, it is a collaboration between Quentin Tarantino and Robert Rodriguez. Quentin Tarantino is best known for gangster movies, Robert Rodriguez (at that time) was best known for horror movies.

The movie starts like a normal Quentin Tarantino movie, guns, hold ups, tough talk. Two fugitives are on their way to Mexico. They fight their way over the border to a bar.

Suddenly half the people in the bar turn to Vampires and the movie switches to a horror movie. The survivors band together to fight the vampires.

It’s not really one movie. It’s two movies with two directors with an abrupt transition from one to the other.

We’ve got Tarantino:

Then Rodriguez:

It’s a great movie. You should see if if you haven’t already.

Windows 8 isn’t one operating system

I do have a point here, I promise.

Windows 8 isn’t one operating system, it’s two. On one hand you’ve got the traditional windows desktop, largely the same sort of experience since Windows 98. You know, windows, desktop, explorer.

Then you’ve for the Metro Modern interface. This is the full screen, touch optimized interface.
There is zero continuity between them.

So what does it feel like?

This depends on the device you are using.

Tablet/Convertible – hey this is pretty neat
Laptop – doesn’t really make sense, but ok
Dual Screen Desktop – I want to stab someone

This didn’t really come home to me until I started using windows 8 on a dual screen desktop extensively. The metro modern interface, optimised for touch, doesn’t translate well to large screen displays. Or more than one screen.

The start screen taking over the whole screen is hugely jarring.

My pattern of usage is to start apps using Windows Key + start typing app name. Prior to Window 8, this was a relatively unobtrusive start menu. With Windows 8 it takes over the whole screen. And on a 24″ monitor I hate it.

With the release of Windows 10, it looks like Microsoft has acknowledged this.

Fixing a broken windows installation

I recently had a rather sick server that was exhibiting all sorts of weird behaviour. Applications pools under IIS were shutting down as soon as the site was hit. All sorts of weird stuff was happening. Event logs were providing nothing useful. Even trying to install IIS diagnosis tools failed.

Clearly a very sick machine.

Diagnosis

One of the challenges was what to trust. Clearly there were some underlying issues with the operating system. Once you go down that rabbit hole, where do you stop? Is the event log still working?

Where do you start?

I had the following info:

  1. A web application that was working 6 hours before was now failing, without any useful errors in the event log. There was an exit code but I couldn’t find any info.
  2. Diagnosis tools were failing to install
  3. A colleague was reporting that some apps wouldn’t run

My best guess was that some of the operating system was corrupted in some way.

Solution!

In my extensive research (googling frantically) on this, I came across System File Checker. This runs through all system files and finds any files that might be corrupt.
I also found this rather helpful link that covers how to read the log files.

Resolution

The scan indicated an issue with a specific dll. I passed this onto a colleague who was able to fix the issue by uninstalling and re-installing windows updates. It’s a great tool to add to the toolbox.

The 

System File Checker

System File Checker

Welcome to the nightmare factory

In my earlier post I called software development the dream factory.

Unfortunately the very things that make software the stuff of dreams can also create nightmares.

World without rules

Once you’ve started to write some software, you are setting up rules for your new world.

Those rules define what you can or can do. Once they are defined, it’s often very hard to change them.

There lies the problem. Software is so unbounded by rules that inevitably people make the wrong decisions and are then locked into those decisions.

Once you have created your world you have to live in it. You must live with all the decisions you’ve made.

Looking into the future

Software changes so rapidly that it becomes a challenge to choose tools and techniques that will stand the test of time. Today’s best practice becomes tomorrow’s worst practice.

I recall when bitshifting was a clever optimisation. I was there when winforms was the best way to build desktop applications. I saw service locator go from being a best practice to a worst practice, replaced by IoC containers.

Right now, if you wanted to build a SPA, which javascript framework would you choose? How many of the current favourites will still be in use and being maintained just 2 years from now?

Imbalance

    One of the problems with software is that is it really, really flexible … until it isn’t. One moment you are defining the rules, then you are bound by the rules. Depending on the choices made up front, some of the flexibility might remain.

    Imagine for a moment if a software project were building a bridge over a canyon (I know it’s an overused example). Some of the things that might be completely impossible in the real world are easy, but things that easy in the real world might be close to impossible, depending entirely on the decisions made earlier. For example, it might easy to move the bridge 200m down the canyon, but impossible to add a new coat of paint.

    This is terribly confusing for consumers of the software as the rules aren’t consistent. In the last ‘bridge’ they requisitioned, moving the bridge was hard but painting was easy, why is it so hard this time?

    Welcome to the nightmare factory

    The dream factory also creates nightmares. There are no rules. You are in full control.
    You are going to make the wrong decisions with long term consequences.
    Welcome to the nightmare factory, enjoy your stay 1.

    1 On the bright side, not all dreams turn to nightmares

    Welcome to the Dream Factory

    I’ve often heard software compared to all sorts of things: construction, engineering, even art.

    For me, I always think of software development is the dream factory. Software is so free of any rules, you can do whatever you want to do. It’s like a blank sheet of paper. In 3D. Or as many dimensions as you chose to define, because you genuinely can choose how many dimensions you want.

    In construction, if someone is building a house, they can’t suddenly ask to add another storey. In software this isn’t an unreasonable request. Artists are still limited by the materials they use.

    When people write games they have to build or use a physics engine. Pause to think about that for a moment. Even the rules of physics don’t apply!

    Every piece of software is like a whole new world.

    Engineering

    I originally trained as an engineer. A real engineer, not a software engineer.
    Real engineering is heavily restricted by physical limitations. If you are designing a frame, you’re pretty sure you are going build it out of steel. Sure more exotic materials might be an option (carbon fibre, titanium, nanotubes etc), but most are too expensive to be viable. So you’ll design it to be built out of steel, the same way an engineer 50 years ago might have done.
    Engineering means designing and building from known patterns and materials. You’ll use the same basic mechanisms, materials and machines. The configuration might change but the basic materials and patterns are the same.

    Aren’t there limits in software?

    Now software does have some real bounds. Compute, latency, bandwidth and storage are all limited. However in the age of cloud computing, only latency and bandwidth are truly limited. Even these limitations change rapidly, compared to other industries.

    This is the industry that runs on Moore’s Law, where computing performance roughly doubles every 2 years1. Imagine being an engineer, where the materials you work with doubled in strength every 2 years!
    Most software is limited only by willingness to spend time or money creating it. The only real bounds are the ones you create yourself.

    Welcome to the Dream Factory

    This is why I call software the dream factory. It is truly the stuff of dreams, limited only by the skill and imagination of it’s creators. Software itself does not impose any real bounds.
    Welcome to the dream factory, enjoy your stay.

    1 Moore’s law is more a measure of the number of transistors on a chip doubles every 2 years, however performance is a rough approximation