Setting up a new development environment

updated July 15, 2015
João Moreira

Setting up your development environment on a new computer can be a pain. You start by installing your favorite editor (sublime, vim, emacs, spacemacs, etc.). After that you need to install all your essential packages and plugins (the ones that make you go How can your live without XX? This is impossible to work. every time a colleague asks you to check out something on their computer). Then, if you work with a scripting language like python or ruby, you also need to download an interpreter and configure it for your projects. In our lab we use pyenv which makes this part super easy. Finally you go into the little details and configure the look and feel of the operating system (shell prompt, aliases, handy functions etc), and also set up all the environment variables that connect all the part... Like I said, setting up a dev environment on a new computer can be a pain.

Fortunately a lot of programmers have had this problem before, and since most of them hate having to do the same task more than once, they came up with ways to automate this painful and tedious process.

Nowadays a simple internet search will give you access to plenty of scripts for setting up a new dev environment on your system of choice. However during my searches I couldn’t find a script that I really liked. They either didn’t automate enough parts or automated way more parts than I used. So I decided to write my own custom script, and instead of just sharing it and have it be useful for no one besides myself, I thought I would instead explain how you can make your own setup script.

Note: this guide will help you set up a development environment on a new Unix system only. That means either Linux flavors or OSX. Sorry Windows!

Organizing your dotfiles

First things first, in order to automate the setup of your environment you need to have one to begin with. On Unix systems most of the relevant settings for developers are stored in the so-called "dot files". These are are a set of files whose name starts with a “.” and sit in your home directory. Using dotfiles you can customize the look of the terminal, define handy functions aliases and keyboard shortcuts, and configure text-based editors, version control systems, plotting libraries, and much, much more. I’m going to assume you have your own set of dotfiles you are happy with. If you don’t have one, just search the web for "sensible dotfiles" and you will find tons of examples. Here’s some tips to get you started:

  • You should always have both a .bashrc and a .bash_profile. Most Unix systems run a non-login shell which, by default, reads first the .bash_profle and then .bashrc (if they exist); on OSX the shell is a login shell, which *does not8 read the .bashrc (What's this about login and non-login shells?. So, a good way to keep your configuration multi-system, is to have a .bash_profile that just reads from the .bashrc:

       $cat ~/.bash_profile
  [ -n "$PS1" ] && [ -f ~/.bashrc ] && source ~/.bashrc

    Then you can write all your configurations in the .bashrc.

  • Organize your .bashrc settings by sections. The order is not very important. In my .bashrc I have: Bash options, exports, PATH variable, aliases, bash completions, prompt, and finally functions. Some terminal commands behave slightly differently on OSX than they do on Linux. This is because OSX is based on BSD (another type of Unix distribution), which defines its own version of popular commands such as ls, ln, cat, head, tail, and others. On OSX, if you use Homebrew you can install the Linux versions, which are actually written by the GNU foundation:

    $ brew install coreutils.

    Then modify your PATH like so:

    PATH="/usr/local/bin:$(brew --prefix coreutils)/libexec/gnubin:$PATH"

    This ensures the GNU versions of those commands are the ones loaded in the shell. If you want to keep your configuration general, you need to add some checks in .bashrc to make sure your aliases work as expected in both systems. Here’s how you can configure ls to have colors in both OSX and other Unixes:

    (in your bash_profile)

    # Detect which `ls` flavor is in use
# `ls` color generator: http://geoff.greer.fm/lscolors/
# GNU `ls`
if ls --color > /dev/null 2>&1; then
    COLORFLAG='--color'
    GROUPDIRS='--group-directories-first'
    export LS_COLORS='di=1;34;40:ln=35;40:so=32;40:pi=33;40:ex=31;40:bd=34;46:cd=34;43:su=0;41:sg=0;46:tw=0;42:ow=0;43:'
# OS X `ls`
else
    COLORFLAG='-G'
    # OS X `ls` does not have this option 
    GROUPDIRS=''
    export LSCOLORS=Exfxcxdxbxegedabagacad
fi

alias ls="ls  ${GROUPDIRS} ${COLORFLAG}"

  • If you .bashrc is getting too large, it might be good to move some sections to their own separate files and then source them from the .bashrc. Here I have my prompt customization and my bash functions defined elsewhere:

    for FILE in ~/.{bash_prompt,bash_functions}; do
    [ -f "$FILE" ] && . "$FILE";
done
unset FILE

  • It’s not that important, but to keep your shell environment clean, be sure to unset any variables in your bash files that are not exported nor defined as local. Otherwise they will always be defined in your shell sessions.

So these are my dotfiles that I want to be in place in all my development environments:

  • ~/.bash_functions
  • ~/.bash_prompt
  • ~/.bash_profile
  • ~/.bashrc
  • ~/.gitconfig
  • ~/.gitignore
  • ~/.hgrc
  • ~/.hgignore
  • ~/.vimrc
  • ~/.matplotib/matplotlibrc

Because I use ssh a lot to connect between the lab computers I accumulated my own set of ssh configurations so I’m adding this file as well.

  • ~/.ssh/config

Sublime Text configuration files

Some editors keep their configurations files in a dedicated location. If that’s your case you need to find it and take note of the files/folders you need when starting again from scratch. Sublime Text, my editor of choice, keeps files in of these locations:

  • Ubuntu: ~/.config/sublime-text2/
  • OSX: ~/Library/Application Support/Sublime Text 2/

For Sublime you actually should only copy files from your User folder, namely settings, snippets, and theme files:

  • *.sublime-settings
  • *.sublime-snippet
  • *.sublime-theme
  • *.tmTheme

Writing the installer script

Now the real work begins! We want a script or series of scripts that we can run on any new system that copies all the files for us. We want this script to be available from anywhere so we should really place it somewhere online. Github and Bitbucket are some of the most popular platforms for this purpose. Pick whichever one you’re most comfortable with, noting that by default Github repos are public while Bitbucket repos are private. I’m going with Bitbucket since I already have a few repos there.

$ mkdir /path/to/your/project
$ cd /path/to/your/project
$ git init
$ git remote add origin git@bitbucket.org:joao_moreira/dev-setup.git

Now we just copy our files inside the repo. Here’s what mine looks like.

$ l -R -1 **
locate_sublime_config.sh
install.sh

dotfiles:
bash_functions
bash_profile
bash_prompt
bashrc
gitconfig
gitignore
hgrc
hgignore
matplotlibrc
ssh_config
vimrc

sublime_configs/Packages/User:
(All my Sublime Text stuff)

$ git add .
$ git commit -m “added my dotfiles and sublime config”
$ git push

Alright. We have our files safely in our repo. Now we just need a script to install these files on demand:

$ cat setup_dev_env.sh
#!/bin/bash

# Find full directory name where this script is located
# http://stackoverflow.com/a/246128
CWD=$( cd "$( dirname "${BASH_SOURCE[0]}" )" && pwd )

echo -n "Sym-linking dotfiles..."
ln -fs $CWD/dotfiles/gitconfig ~/.gitconfig
ln -fs $CWD/dotfiles/gitignore ~/.gitignore
ln -fs $CWD/dotfiles/hgrc ~/.hgrc
ln -fs $CWD/dotfiles/hgignore ~/.hgignore
ln -fs $CWD/dotfiles/ssh_config ~/.ssh/config
ln -fs $CWD/dotfiles/vimrc ~/.vimrc
ln -fs $CWD/dotfiles/bash_prompt ~/.bash_prompt
ln -fs $CWD/dotfiles/bash_functions ~/.bash_functions
ln -fs $CWD/dotfiles/bash_profile ~/.bash_profile
ln -fs $CWD/dotfiles/matplotlibrc ~/.matplotlib/matplotlibrc
echo "done!"

As you can see, this script actually creates a symbolic link to the dotfiles in our home directory. This way every time you change your dotfiles you don’t have to worry about copying your changes to the "gold standard" dotfiles, you already did it! Just remember to commit your changes to the repo, so they will apply the next time you need to start fresh. NOTE: If you are already have some dotfiles in your home folder make a copy somewhere else! Otherwise they will be overwritten by the symlinks.

Next up is the sublime text configuration. Here I can’t use symlinks because they won’t capture any eventual files that will be created if I install any new sublime packages. To account for this I use rsync to keep the configuration synced to the repo.

(... rest of setup_dev_env.sh)

echo -n "Configuring Sublime Text 2..."
SUBL=$($CWD/locate_sublime_config.sh)
rsync -tprhm $CWD/sublime_configs/ "$SUBL"
echo "done!"

This will sync the configuration files from the repo to the sublime installation location.
Then in my .bash_functions.sh I have a function that does the reverse:

# Backup sublime text settings
backup_sublime() {
    local REPO_LOCATION=$(locate_dev_repo)
    local SUBL=$($REPO_LOCATION/locate_sublime_config.sh)

    echo
    echo -n "Backing up Sublime Text 2 settings files..."
    rsync -tprhm \
        --include="/Packages/User/*.sublime-settings" \
        --include="/Packages/User/*.sublime-snippet" \
        --include="/Packages/User/*.sublime-theme" \
        --include="/Packages/User/*.tmTheme" \
        --include "*/" \
        --exclude="*" \
        --delete-excluded \
        "$SUBL/" "$REPO_LOCATION/sublime_configs"
    echo "done!"
}

If I change any sublime settings all I need to do is:

$ backup_sublime

and those settings will be backed up!

The line local REPO_LOCATION=$(locate_dev_repo) calls a function that does a more complicated version of "Find full directory name where this script is located". You can find it here. The reason we can’t use the one-liner from earlier (see definition of $CWD) is that when you call this function, .bash_functions.sh will be a symlink, and we need a recursive expression that can "dereference" the link.

Also, if you’re scratching your head at the rsync --include syntax… I’ve been there too. It took me quite a while to get it to work properly. Basically the reason you have to be so verbose in the file specification is that rsync include/exclude rules are applied to each directory in a depth-first-search manner. If you don’t do it this way you either sync all files in all directories or no files at all!

locate_sublime_config.sh is a handy little script that tries to guess which system you’re running and returns the location to sublime settings location:

$ cat locate_sublime_config.sh
#!/bin/bash

# Finds the location of Sublime Text configurations in several systems

OSX_DEFAULT="$HOME/Library/Application Support/Sublime Text 2"
LINUX_DEFAULT="$HOME/.config/sublime-text-2"

if [ -d "$OSX_DEFAULT" ]; then
    SUBL="$OSX_DEFAULT"
elif [ -d "$LINUX_DEFAULT" ]; then
    SUBL="$LINUX_DEFAULT"
else
    SUBL=""
fi

echo "$SUBL"
unset SUBL
unset OSX_DEFAULT
unset LINUX_DEFAULT

Finally we automate the installation of our favorite interpreter. Here in our lab we program in python. Instead of being dependent on whichever version ship with the OS we actually use pyenv to have concurrent, self-contained, python installations. If you’re interested in trying it out check out our pyenv install guide!

Fortunately pyenv has an automatic installer, so we just need one more line in our install script:

(... rest of setup_dev_env.sh)
curl -L https://raw.githubusercontent.com/yyuu/pyenv-installer/master/bin/pyenv-installer | bash

Alternatively, you can install pyenv with Homebrew:

$ brew install pyenv

And that’s it! With this basic installer you can be sure to speed up the setup of your development environment.

More suggestions

If you want to get fancier with your dev setup here are some suggestions:

  • Add some command line flags to check for existing files, select what to install, etc.
  • Add a section in setup_dev_env.sh that actually installs Sublime Text. That’s my next task.
  • For OSX, add a section in setup_dev_env.sh that installs Homebrew and essential packages.
  • Also for OSX, you can add a dotfile to configure pretty much anything in the OS. More info here

Acknowledgments

When writing this guide and my own dotfiles I relied heavily on these great github repos:

Their authors deserve most of the credit for this guide!