Caching Dependencies

Created by Tass Skoudros, Modified on Tue, 22 Nov, 2022 at 6:11 PM by Tass Skoudros

Time saved with dependency caching can help improve the speed of pipelines, and we've done quite a bit of work in this area but ended up concluding that implementing a dumb cache (ignorant of the version of the dependencies) is not the way to go. 

The following is how we are currently tackling caching for our customers. This tried and tested method uses your local files to improve the quality of the cache, making the integration seamless.

The caching mechanism is reasonably good because you are caching based on the current branch. In a case like this, it can be assumed that generational changes to the dependencies will be consistent. However, there are cases where you may want to debug or test a change to a dependency without caching, so add a config to your pipeline to turn off the cache.

CACHE_DATA_STORE is available on all build agents.
// CACHE_DATA_STORE var is defined at Jenkins global config
def downloadFileFromCache(String fileName, String branch){
    sh """
        if [ -z "\$(aws s3 ls s3://$CACHE_DATA_STORE/${branch}/${fileName})" ]; then
            echo "Cache doesn't exist"
            echo "Cache does exist, downloading and untar"
            aws s3 cp s3://$CACHE_DATA_STORE/${branch}/${fileName} ${fileName} --no-progress
            tar -xf ${fileName} && rm -f ${fileName}
// CACHE_DATA_STORE var is defined at Jenkins global config
def uploadFileToCache(String fileName, String files, String branch){
    sh """
        tar -czf ${fileName} ${files}
        aws s3 cp ${fileName} s3://$CACHE_DATA_STORE/${BRANCH_NAME}/${fileName} --no-progress
            stage('Setup Yarn and install the packages') {
                steps {
                        sh 'echo $NPM_TOKEN > npmAuthToken'
                        // we need to install yarn here - installing it globally in jenkins setting does not work
                        sh 'npm install -g yarn'
                        sh "yarn config set enableGlobalCache false"
                        sh 'yarn install --immutable --inline-builds'
                post {
                    success {
                            if (config.useCache){
                                commonHelper.uploadFileToCache('dependency-cache.tar.gz',"node_modules .yarn/cache",BRANCH_NAME)

Docker Caching

Docker has built-in features to improve performance, most of which don't require any additional dependencies. 

For the official documentation on this subject, go here for build enhancements and here for a method of specifying external sources

In the past, we've used the --cache-from flag in the docker build command, which works well compared to doing nothing. However, it isn't good at granular caching, and you will find that your more time-consuming layers are skipped due to variations in how the layer is assembled. We recommend the following approach to improve the quality of your layer caching. 

Enable Build kit

Enable Buildkit with the environment variable DOCKER_BUILDKIT = 1

# With the environment variable

In the docker build command, specify the flag --build-arg BUILDKIT_INLINE_CACHE=1 and specify the --cache-from flag with a source to a previous build.

docker build --build-arg BUILDKIT_INLINE_CACHE=1 \
--cache-from \
-t \
-t .

Then to complete the process push back a tag to save your latest build from speeding up the next build. Like this.

# docker tag
docker push 
docker push

You should see a massive improvement in your docker caching after these improvements.

Was this article helpful?

That’s Great!

Thank you for your feedback

Sorry! We couldn't be helpful

Thank you for your feedback

Let us know how can we improve this article!

Select at least one of the reasons
CAPTCHA verification is required.

Feedback sent

We appreciate your effort and will try to fix the article