Prevent Pipeline from rebuilding shared parent stage in multi-stage build

[DrewVartanian]

Please excuse any incorrect terminology, as I'm pretty new to both Copilot and docker in general. I have multiple services that have a shared parent stage in a multi-stage docker build. Something akin to creating web and worker images like shown here:

FROM ubuntu:18.04 AS base
...

FROM base AS web
...

FROM base AS worker
...

When I deploy with Pipeline set up through Copilot, it builds both web and worker all the way from ubuntu:18.04, rather than just building base once for the first image, and then reusing it for the second. Is there any way to have Pipeline only build base once? Base dose take in ARG(s), but they are set to the same value in the web and worker manifest (not sure if that affects things).

[dannyrandall]

Hey @DrewVartanian! Using ARG in docker build makes it so future instructions are not cached. Some details around that are available here.

One potential solution you could try is moving the ARG command into the web and worker stages like this:

FROM base AS web
ARG name

FROM base AS worker
ARG name

This way, the base stage layers will be cache-able and should only run once in your pipeline. Let me know if this works for you!:blush:

EDIT

Another potential solution: #3510 (comment)

[DrewVartanian]

Hi @dannyrandall . Thanks for the response! I tried replacing my ARGs with hardcoded ENVs to test, as they are used in the base stage build, but it still seems to be rebuilding base for each container. Is there anything else that prevents the base layer from being cache-able (like a COPY)?

[DrewVartanian]

That's correct. I haven't changed anything in the buildspec.yml that was generated by copilot when setting up the pipeline, and I'm using the dockerfile, context, and target fields in my manifests' image.build section (with the targets pointing to web and worker).

[dannyrandall]

Hmm...interesting.🤔 Could you share more of your Dockerfile and the image configuration for both services in your manifests?

[dannyrandall]

Actually - if you're copying the entire context directory (COPY . <dest>), I wonder if this would help? #3510 (comment)

[DrewVartanian]

I am copying the entire context directory! Good catch! I added the default output dir/infrastructure/ to .dockerignore, and it's now caching properly. Do you know if there is some way to set the output-dir in the copilot manifests? Either way, thanks for the help!

[dannyrandall]

Yay!:tada: There isn't a way to specify the output-dir from the manifest, but you can modify the buildspec (default lines 59 and 66) and specify the output directory there.