Building multiple pipelines from different branches of a single repo

[huanjani]

We have a more baked-in solution in the works, but in the meantime, here is a workaround for creating different pipelines for the branches of a source repository.

For the first pipeline:

1. From the first git branch, run copilot pipeline init.
2. After the buildspec.yml and pipeline.yml files are created in the copilot directory, check that the pipeline manifest (pipeline.yml) has the desired branch. If not, edit the manifest to change it to the name of the current/desired branch.
3. Add, commit, and push to the git remote.
4. Run copilot pipeline update to build the pipeline.

For the second pipeline:

1. git checkout the second git branch. Run copilot pipeline init. Select the same repository url as you did above.
2. This time, after the buildspec.yml and pipeline.yml files are created, go into the pipeline manifest file (pipeline.yml) and change the name: field to a pipeline name that is different than that of the first pipeline, and confirm that the branch is the desired second branch.
3. Add, commit, and push to the git remote from this second branch.
4. Run copilot pipeline update to build the second pipeline.

Sample pipeline.yml snippets (pay particular attention to the name and branch fields):

# "pipeline.yaml" to trigger a pipeline from a "dev" branch.

name: pipeline-appName-repoName-dev # Add "-dev" suffix in your pipeline name.
version: 1
source:
  provider: CodeCommit
  properties:
    branch: dev                     # Make sure your branch points to "dev"
    repository: https://us-west-2.console.aws.amazon.com/codesuite/codecommit/repositories/repoName/browse

# Second "pipeline.yml" in the "prod" branch.

name: pipeline-appName-repoName-prod # Add "-prod" suffix in your pipeline name.
version: 1
source:
  provider: CodeCommit
  properties:
    branch: prod                     # Make sure your branch points to "prod"
    repository: https://us-west-2.console.aws.amazon.com/codesuite/codecommit/repoName/aws-sample/browse

Of course, eventually you will probably want to merge dev into prod, but still maintain the separate pipelines. To do this:

1. Add a .gitattributes file in the branch to be merged into the other (here, dev)
2. Add pipeline.yml merge=ours to the file.
3. Add and commit the .gitattributes file.

Locally, this will prevent the pipeline manifest from being merged with the other branch and thereby maintain distinct pipelines. (See https://git-scm.com/book/en/v2/Customizing-Git-Git-Attributes.) To merge with a remote host, you'll have to manually reconcile conflicts in the pipeline.yml, then revert the pipeline name after merging.

Let us know how this works for you!

[joshgoodson]

@huanjani have these discussions involved cross-account pipelines at all? Say I have a prod account and dev account and would like to create a cross-account pipeline to go from qa in our dev account to ultimately a production deployment in another account?

Or do you know of any examples in which this is possible today?

[huanjani]

Hi, @joshgoodson!

If I'm understanding correctly, what you're describing is actually the workflow we had in mind when we designed Copilot pipelines. If you initialize environments in different accounts, you can add those environments as deployment stages when you run pipeline init. See https://aws.github.io/copilot-cli/docs/credentials/#environment-credentials.

Does that help?

[dannysteenman]

Hi @huanjani! From what I tested it doesn't work to create a cross-account pipeline yet. I created an issue for that:

#2533

[joshgoodson]

@dannysteenman to get cross account pipelines working, I only deployed the pipeline in a single account and then deployed the env in the other account - I can then place the envs in the pipeline manifest and it appears like it will deploy using the single pipeline in the account you ran copilot pipeline init/update on. I haven't actually released a change through the pipeline from a qa to a prod like env yet, but I was able to update the pipeline and have it added with manual approval turned on for it. I just have two different aws profiles I use locally and specified it when I created each env

copilot app init AppName --domain AppDomain.com 
copilot env init -a AppName -n qa --profile dev
copilot env init -a AppName -n prod --profile prod

Then in my pipeline manifest am able to update the pipeline with this

stages:
    -
      name: qa
      test_commands: [echo 'running tests', make test]
    -
      name: prod
      requires_approval: true

[wstewarttennes]

This works but it's a pretty hacky solution because you can't view the multiple pipelines in a single file and isn't really sustainable if you want to have a bunch of pipelines (example: we need about 15 different QA sites for remote non-technical employees to be able to test distinct features). Would love to be able to add pipelines the same way we can add services, with copilot pipeline init

[huanjani]

Hello, @wstewarttennes!
Thanks for this feedback; perfect timing, as we are in the process of designing what multiple pipelines will look like and how it will work.
Would your pipelines map to different environments of the same app? Or different branches of the same repo? We'd love to hear more!

[wstewarttennes]

Hi @huanjani,

Really happy to hear that! I think the answer is ideally both, but I would rather have different branches of the same repo over different environments.

My production deployments mainly have the same code, so those could be represented via one pipeline and different environments. However, there are times where we need to fork some code to one user on one deployment. Right now I need to do this manually.

QA environments need different code pushed so would need separate branches and that is my main wish.

Thinking outside the current pipeline model, my ideal solution would be to just specify a branch per environment and when code is pushed to that branch it updates the associated environment.

[jsutton]

It is worth noting, that you may get some unexpected behavior if you update the pipeline file in one of the branches after setting this up. The .gitattributes doesn't apply when a FF merge can be done meaning if you update in dev and prod doesn't have a conflicting commit, it may just merge the change right over.

Reference: https://stackoverflow.com/a/27234774

[huanjani]

Work on this feature is underway!
To follow along or contribute feedback, see #3231.
❤️

[efekarakus]

Hi folks! this feature is now released with https://github.com/aws/copilot-cli/releases/tag/v1.16.0 ! 🥳
Blog post for the announcement: https://aws.github.io/copilot-cli/blogs/release-v116/

[wstewarttennes]

wooooo thank y'all!! really appreciate it:)