Caching Dependencies

Time saved with dependency caching can help improve the speed of pipelines, and we've done quite a bit of work in this area but ended up concluding that implementing a dumb cache (ignorant of the version of the dependencies) is not the way to go. 

The following is how we are currently tackling caching for our customers. This tried and tested method uses your local files to improve the quality of the cache, making the integration seamless.

The caching mechanism is reasonably good because you are caching based on the current branch. In a case like this, it can be assumed that generational changes to the dependencies will be consistent. However, there are cases where you may want to debug or test a change to a dependency without caching, so add a config to your pipeline to turn off the cache.

CACHE_DATA_STORE is available on all build agents.

// CACHE_DATA_STORE var is defined at Jenkins global config
def downloadFileFromCache(String fileName, String branch){
    sh """
        if [ -z "\$(aws s3 ls s3://$CACHE_DATA_STORE/${branch}/${fileName})" ]; then
            echo "Cache doesn't exist"
        else
            echo "Cache does exist, downloading and untar"
            aws s3 cp s3://$CACHE_DATA_STORE/${branch}/${fileName} ${fileName} --no-progress
            tar -xf ${fileName} && rm -f ${fileName}
        fi
    """
}
// CACHE_DATA_STORE var is defined at Jenkins global config
def uploadFileToCache(String fileName, String files, String branch){
    sh """
        tar -czf ${fileName} ${files}
        aws s3 cp ${fileName} s3://$CACHE_DATA_STORE/${BRANCH_NAME}/${fileName} --no-progress
    """
}

            stage('Setup Yarn and install the packages') {
                steps {
                    script{
                        if(config.useCache){
                            commonHelper.downloadFileFromCache('dependency-cache.tar.gz',BRANCH_NAME)
                        }
                        sh 'echo $NPM_TOKEN > npmAuthToken'
                        // we need to install yarn here - installing it globally in jenkins setting does not work
                        sh 'npm install -g yarn'
                        sh "yarn config set enableGlobalCache false"
                        sh 'yarn install --immutable --inline-builds'
                    }
                }
                post {
                    success {
                        script{
                            if (config.useCache){
                                commonHelper.uploadFileToCache('dependency-cache.tar.gz',"node_modules .yarn/cache",BRANCH_NAME)
                            }
                        }
                    }
                }
            }

Docker Caching

Docker has built-in features to improve performance, most of which don't require any additional dependencies. 

For the official documentation on this subject, go here for build enhancements and here for a method of specifying external sources. 

In the past, we've used the --cache-from flag in the docker build command, which works well compared to doing nothing. However, it isn't good at granular caching, and you will find that your more time-consuming layers are skipped due to variations in how the layer is assembled. We recommend the following approach to improve the quality of your layer caching. 

Enable Build kit

Enable Buildkit with the environment variable DOCKER_BUILDKIT = 1

# With the environment variable
DOCKER_BUILDKIT         = 1

In the docker build command, specify the flag --build-arg BUILDKIT_INLINE_CACHE=1 and specify the --cache-from flag with a source to a previous build.

docker build --build-arg BUILDKIT_INLINE_CACHE=1 \
--cache-from 123456789.dkr.ecr.eu-west-1.amazonaws.com/app-web-demo:develop-latest \
-t 123456789.dkr.ecr.eu-west-1.amazonaws.com/app-web-demo:develop-latest \
-t 123456789.dkr.ecr.eu-west-1.amazonaws.com/app-web-demo:develop .

Then to complete the process push back a tag to save your latest build from speeding up the next build. Like this.

# docker tag 123456789.dkr.ecr.eu-west-1.amazonaws.com/app-web-demo:develop-latest
docker push 123456789.dkr.ecr.eu-west-1.amazonaws.com/app-web-demo:develop 
docker push 126306479661.dkr.ecr.eu-central-1.amazonaws.com/uc-apps-web-customer:develop-latest

You should see a massive improvement in your docker caching after these improvements.