Setting up continuous deployment over SFTP for a Jekyll site using GitLab

GitLab, where I have my version control for wllm.no, offer continuous integration/continuous deployment (CI/CD) with a free tier available even for private projects. I wanted to set up continuous deployment so I could make and publish smaller changes without a build environment on my local machine. I also wanted to automate uploading the site to my hosting provider.

A small note – this is not using GitLab Pages. This is a standalone Jekyll build process deploying to an external hosting provider, in my case Domeneshop.

GitLab provides documentation and example configurations for you to get started. This blogpost by Xavier Decuyper was also very helpful in setting up deployment using lftp.

Setting up the pipeline

The build and deploy process can be outlined in a few steps:

1. Set up Ruby build environment and install dependencies
2. Build the site using Jekyll
3. Deploy over SFTP using lftp

Before starting on the config, save the SFTP user and passphrase as variables under the repository Settings -> CI/CD -> Variables tab. Don’t commit secrets such as passwords in your config file. Reading values stored as variables is trivial and much more secure.

First we want a base image with Ruby ready to go. We also want to cache RubyGems between builds to increase build performance.

image: 'ruby:2.6'

cache:
  paths:
    - vendor/ruby

The before_scripts section can be used to set up the build environment and install dependencies. This step will vary based on your needs, but for me the section looks like this:

before_script:
  - apt-get update -qy
  - apt-get install -y lftp
  - ruby -v
  - gem -v
  - gem install bundler --version '~> 2.0.2'
  - bundle -v
  - bundle install -j $(nproc) --path vendor

What I’m doing above is updating the software sources – the image is Debian based – to make sure I’m installing the latest version of lftp which I need later. I also upgrade bundler before I use it to install the dependencies declared in my Gemfile. I also print the versions of Ruby, gem, and bundle for debugging.

Next is building the site and marking the result as a build artifact:

build:
  script:
    - bundle exec jekyll build
  artifacts:
    paths:
      - _site

The above is done on all branches so I get confirmation when creating a merge request that the site is still properly configured.

Finally, when the merge request completes and the pipeline runs for the master branch I run the deploy step:

deploy:
  type: deploy
  environment: production
  script:
    - lftp -e "set sftp:auto-confirm yes; open sftp://ftp.yourprovider.com; user $SFTP_USER $SFTP_PASSPHRASE; mirror --reverse --verbose --delete _site/ www/; bye"
  only:
    - master

A quick explanation of the lftp-command above:

• Auto-accept the SSH host questions
• Open a connection using the sftp protocol
• Read the user and password from the variables configured in GitLab
• Mirror the contents of the local folder _site/ to www/ on the host. This deletes any files in www/ that are no longer in _site. Back up www/ if you have content there already in case the command deletes something unexpected.
• Close the connection

Here is the complete .gitlab-ci.yml, also available as a snippet on GitLab.

# Official language image. Look for the different tagged releases at:
# https://hub.docker.com/r/library/ruby/tags/
image: 'ruby:2.6'

# Cache gems in between builds
cache:
  paths:
    - vendor/ruby

before_script:
  # Update sources
  - apt-get update -qy
  # Install lftp for deploy
  - apt-get install -y lftp
  # Print out ruby version for debugging
  - ruby -v
  # Print out rubygem version for debugging
  - gem -v
  # Upgrade bundle. The default version is 1.17.x.
  - gem install bundler --version '~> 2.0.2'
  # Print out bundle version for debugging
  - bundle -v
  # Install dependencies into ./vendor/ruby
  - bundle install -j $(nproc) --path vendor

build:
  script:
    - bundle exec jekyll build
  artifacts:
    paths:
      - _site

deploy:
  type: deploy
  environment: production
  script:
    - lftp -e "set sftp:auto-confirm yes; open sftp://ftp.yourprovider.com; user $SFTP_USER $SFTP_PASSPHRASE; mirror --reverse --verbose --delete _site/ www/; bye"
  only:
    - master

Troubleshooting

When I first set up the build I did not upgrade bundler. The first time I tried to build it broke on the bundle install step with the message “You must use Bundler 2 or greater with this lockfile”.

This is where I tell you I’m not a Ruby developer 😄

The Jekyll docs instruct users to run gem install bundler jekyll. This meant I ended up with version 2.0.2 on my machine, and that version generated Gemfile.lock.

I’m not the only one having problems with bundler versions when using the official Ruby Docker-image. I don’t know the Ruby ecosystem well enough to have an opinion on it, but (probably for the best) the decision has been made not to ship 2.x by default yet in the images.

So it’s up to us users to upgrade ourselves. Add gem install bundler --version '~> 2.0.2' to your build to upgrade. Optionally, if you can’t or don’t want to upgrade your build pipeline you can downgrade bundler on the machine you use to generate the lockfile. Remove the lockfile, uninstall bundler and install it again while providing the version you need, then regenerate the lockfile.

Kategorisert under: jekyll, gitlab