Recording builds from multiple repositories

Multiple repositories combined in one build then tested

If you produce a build by combining code from several repositories (like the diagram below), invoke launchable record build with multiple --source options to denote them.

✕

To differentiate them, provide a label for each repository in the form of LABEL=PATH:

# record the build
launchable record build --name <BUILD NAME> --source main=./main_repo --source lib=./lib_repo

# create the build
bundle install

Multiple repositories built/deployed separately then tested together (e.g. microservices)

Some teams run regression tests against an environment where several services have been deployed. Each service is built from code from its own repository (or set of repositories), as shown in the diagram below.

✕

The intent of recording a build is to capture the version of the software being tested. In this scenario, the version of the software being tested is effectively the combination of all versions of the components deployed to the test environment.

For example, if the currently deployed version of service 1 is d7bf8b7c (from repo 1) and the currently deployed version of service 2 is c39b86a1 (from repo 2), then the effective version of the software being tested can be thought of as something like:

[
  {
    repository: "repo1",
    commit: "d7bf8b7c"
  },
  {
    repository: "repo2",
    commit: "c39b86a1"
  }
]

This mental model is no different than the example above. However, because you want to capture the versions of all of the deployed software being tested, you need to run launchable record build right before running tests — i.e. in the green box in the diagram above.

This presents a challenge because the repos for each service are usually not available at this stage (and cloning them just for this purpose would be inefficient). Luckily, when you run launchable record build, the CLI actually performs two functions that we can split up to support this use case:

1. Recording all new commits from included repositories, and

2. Recording the build itself, 'tagged' with the HEAD commit from each included repository

The CLI provides options to separate these: you can record commits in each component's build process and then record the "combined" build itself right before you run tests.

The commands and options that enable this are:

1. launchable record commit --source /PATH/TO/REPO , which lets you record commits separately in each component's build process, and

2. Two launchable record build options:

   1. --no-commit-collection which disables commit collection (since you're doing it separately), and

   2. --commit REPO=HASH which lets you 'tag' the build with each repository
      (Note: This means that the deployed version of each service needs to be available to the process where you run tests.)

These commands and steps are shown in the white boxes in the expanded diagram below.

✕

Incremental build over multiple repositories

Some teams have a software project that spans a large number of repositories, with a build system that supports the incremental build with object cache. Only the "changed" repositories are cloned locally and built, and the reamining artifacts for the "unchanged" repositories get pulled from a remote/distributed object cache. The end result is that each build clones a different subset of repositories. Android + RBE is one such example.

See the diagram below. Two repositories (zooma & fft2d) are "changed" and thus cloned locally to be built. One repository (av framework) is "unchanged" and its build artifacts will be pulled from a remote cache, so there's no local clone of this repository. Even though this repository is not cloned, there's still a commit hash (d3427bd) associated with it.

✕

The intent of recording a build is to capture the version of software being assembled, which includes portions that are not built and simply cached in this case. In this case, the behaviour we'd like to achieve is that for the "changed" repositories, we let Launchable collect new commits from a local clone, and for the "unchanged" repositories, we just tell Launchable that the build will use a certain commit that we don't locally have. The commands and options that achieve this are:

1. For each "changed" repository, there should be a local clone, so run launchable record commit --source path/to/repo , which lets you record commits from this repository.

2. Make one launchable record build invocation with the followings:

   1. --commit REPO=HASH for each repository, changed or unchanged. If the repository was changed, the record commit command earlier should have collected this commit hash. If the repository was not changed, there should have been some earlier builds that cloned this repository and the hash locally, for which record commit command had run.

   2. The --no-commit-collection option, which disables commit collection, since you are doing it in separate record commit invocations

With the above example, the list of eventual invocations should be like this:

launchable record commit --source device/zooma=path/to/zooma
launchable record commit --source external/fft2d=path/to/fft2d
launchable record build --name SOMETHING --no-commit-collection \
  --commit device/zooma=db0fd94 \
  --commit external/fft2d=9fcfcc5 \
  --commit frameworks/av=d3427bd