Replacing static parallel suites with a dynamic parallel subset

Some teams manually split their test suites into several "bins" to run them in parallel. This presents a challenge in adopting Launchable because you don't want to lose the benefit of parallelization.

With split subsets, you can replace your manually selected bins with automatically populated bins from a Launchable subset.

For example, let's say you currently run ~80 minutes of tests split coarsely into four bins and run in parallel across four workers:

Worker 1: ~20 minutes of tests
Worker 2: ~15 minutes of tests
Worker 3: ~20 minutes of tests
Worker 4: ~25 minutes of tests

With a split subset, you can generate a subset of the full 80 minutes of tests and then call Launchable once in each worker to get the bin of tests for that runner.

The high-level flow is:

Request a subset of tests to run from Launchable by running launchable subset with the --split option. Instead of outputting a list of tests, the command will output a subset ID that you should save and pass into each runner.
Start up your parallel test worker, e.g., four runners from the example above
Request the bin of tests that the worker should run. To do this, run launchable split-subset with:
1. the --subset-id option set to the ID you saved earlier, and
2. the --bin value set to bin-number/bin-count.
3. If you're using Zero Input Subsetting, add the --output-exclusion-rules option.
Run the tests on each worker.
After each run finishes in each worker, record test results using launchable record tests with the --subset-id option set to the ID you saved earlier.

In pseudocode:

1# main
2$ launchable record build --name $BUILD_ID --source src=.
3$ launchable subset --split --confidence 90% --build $BUILD_ID bazel .
4subset/12345
5
6...
7
8# worker 1
9$ launchable split-subset --subset-id subset/12345 --bin 1/3 --rest rest.txt bazel > subset.txt
10$ bazel test $(cat subset.txt)
11$ launchable record tests --subset-id subset/12345 bazel .
12
13
14# worker 2
15$ launchable split-subset --subset-id subset/12345 --bin 2/3 --rest rest.txt bazel > subset.txt
16$ bazel test $(cat subset.txt)
17$ launchable record tests --subset-id subset/12345 bazel .
18
19# worker 3
20$ launchable split-subset --subset-id subset/12345 --bin 3/3 --rest rest.txt bazel > subset.txt
21$ bazel test $(cat subset.txt)
22$ launchable record tests --subset-id subset/12345 bazel .

[Beta; Gradle only] Dynamic parallel subset with same bin option

Even though launchable split-subset offers evenly split subset bins, there are some cases where certain tests should not run concurrently. You want to use split-subset to split a subset into multiple bins and want certain tests to belong to the same bin to avoid simultaneous execution.

E.g., TestA and TestB use the same record in a database, and their concurrent access may cause the tests to fail.

We provide the --same-bin <file.txt> option to avoid running these tests simultaneously. By adding --same-bin <file.txt> option to split-subset, the test cases listed in the <file.txt> will be placed in the same bin.

In pseudocode:

1# main
2$ launchable subset --target 90% --build BUILD_ID --split gradle src/test/java
3subset/12345
4Your model is currently in training
5Launchable created subset 12345 for build test (test session 12345) in workspace launchableinc/mothership
6
7|           |   Candidates |   Estimated duration (%) |   Estimated duration (min) |
8|-----------|--------------|--------------------------|----------------------------|
9| Subset    |            7 |                  77.7778 |                0.000116667 |
10| Remainder |            2 |                  22.2222 |                3.33333e-05 |
11|           |              |                          |                            |
12| Total     |            9 |                 100      |                0.00015     |
13
14Run `launchable inspect subset --subset-id 12345` to view full subset details
15
16---
17
18# worker 1
19$ cat same_bin0.txt
20example.DB0Test
21example.DB1Test
22
23$ launchable split-subset \
24    --subset-id subset/12345 \
25    --bin 1/2 \
26    --same-bin same_bin0.txt \
27    gradle
28--tests example.DB0Test --tests example.DB1Test --tests example.MulTest --tests example.DivTest
29
30---
31
32# worker 2
33$ cat same_bin0.txt
34example.DB0Test
35example.DB1Test
36
37$ launchable split-subset \
38    --subset-id subset/12345 \
39    --bin 2/2 \
40    --same-bin same_bin0.txt \
41    gradle
42--tests example.AddTest --tests example.SubTest --tests example.PowTest

Last modified 1 year ago