Integration tests

Integration tests#

Integration tests use the built binaries for the CLI and language runtimes. We have some helpers that allow us to create test scenarios and invoke CLI commands.

  • ProgramTest: this is typically the easiest way to write an integration test. ProgramTest imports a Pulumi program from the testdata directory, and runs a series of CLI commands against it.

  • Environment: this is lower level than ProgramTest. It takes care of creating a temporary directory to run commands in via Environment.RunCommand. Use Environment.ImportDirectory to import a test scenario into the environment.

How and when to use#

Prefer language conformance tests over integration tests where possible. Language conformance tests ensure that we test features across all of our languages in a uniform manner. For functionality that is not exercised during a pulumi operation (preview, up, destroy, …), an integration test is appropriate.

Attention

Integration tests should have a TestMain which calls the testutils.SetupPulumiBinary() method to set an explicit path to the binaries under test to avoid reliance on the $PATH which can cause the wrong binary to be used in tests, resulting in incorrect test results.

You can set PULUMI_INTEGRATION_REBUILD_BINARIES=true in your environment to automatically re-build the binaries locally to your repository and have the integration tests use them automatically.

To test an alternative pulumi binary, set the environment variable PULUMI_INTEGRATION_BINARY_PATH to the absolute path of the binary you want to test.

Organization#

Integration tests are located in tests and loosely organized:

Smoke tests are intended to be small tests that exercise a high level pulumi command, matrixed over our supported languages where applicable.

Performance tests are basic performance tests that run as part of pull request and prevent us from introducing major performance regressions.

The bulk of the tests are in tests/integration, but some pulumi commands have their own top level directories, for example pulumi config is tested in tests/config. Within tests/integration tests are either split by language integration_python_test.go, integration_nodejs_test.go, … or by feature in separate directories, for example tests/integration/transforms.

Build the required binaries#

# From the repository root, build the Pulumi CLI and the Go, Python and Node.js language runtimes.
make build
# You can also build individual SDKs
SDKS="nodejs python" make build
# or just the main Pulumi CLI
SDKS= make build
# The Node.js TypeScript SDK needs to be built separatly
cd sdks/nodejs && make build install

To run a single integration test, run the following command from the repository root.

go test -tags=all github.com/pulumi/pulumi/tests/integration -run ${MY_TEST_TO_RUN}
Contents