Package promotion workflows are a great way to isolate and protect production repositories away from public upstreams, so they only receive clear and vetted packages.
Once you upload a binary to Cloudsmith; you can move it from development -> staging -> production, keeping the integrity of the asset, and a provenance trail, and without the bandwidth cost of downloading from staging and then re-uploading to production.
This post will talk about:
- The differences between Dev, Staging and Production environments.
- Cloudsmith's package promotion instructions- package copy/move actions
- Symbiotic features that play nicely with package promotion include webhooks, quarantining and upstreaming. These can be used together to create workflows and drive actions.
- A Potential workflow for package promotion
- Future improvements to workflows on package promotion with policy management will allow you to set up policies that can be verified as a prerequisite to package promotion.
Differences between Dev, Staging, and production environments
The main three environments in software development are development, stage, and production.
- The Development environment is used for developers and their colleagues early in the software process to see how new features will work, to try out improvements and pull in new dependencies or update existing dependancies. End users should not have access to these packages when they use the software product.
- The Staging environment will be used for testing integration and is a Production-like environment that allows user creation without impacting Production data. It is sometimes accessible with permission from the product team. Only whitelisted IPs have access to Staging.
- The Production environment is what end users are using on a live system.
Package Copy/Move Actions (API/CLI)
Cloudsmith's package promotion workflows are powered by its feature to copy or move packages between repositories, usually for the concept of promoting immutable versions of packages between separate build, test, staging and production pipelines - It means that the package that was verified in build is eventually the same one as in production when it gets promoted to there. Cloudsmith facilitates the automation of copying and moving packages using the CLI.
Let's see an example of moving a package in action:
As you can see, the apache-13pom-bXt package was moved (or "promoted") from the source repository to the dest repository. Copying packages works exactly the same, except it duplicates the package rather than moving it from the source to the destination.
In addition to moving and copying packages, you can also resync packages via the CLI as well, which the CLI will now also utilise to retry packages that have failed (for whatever reason) upto a configurable amount of attempts.
To see the full list of help for actions, run the following commands:
> cloudsmith mv
> cloudsmith cp
> cloudsmith resync
Symbiotic features
Package promotion on its own isn't enough - you need to couple it with other features to create workflows and drive actions. Let's talk about webhooks, multiformat repos, vulnerability scanning and setting up upstream so that all your 3rd party dependencies are also scanned.
Webhooks
Webhooks are one way that apps can send automated messages or information to other apps. Webhooks are great for integrating Cloudsmith with other systems in your build pipeline, by sending data or notifications to other tools in your stack and helping to enable automation across your workflows.
In particular, Cloudsmith has webhooks for vulnerability scanning. Anytime a scan is completed, Cloudsmith can dispatch a corresponding vulnerability webhook to trigger some action like quarantining your package.
Upstreams
Cloudsmith's vulnerability scanner is particularly effective when combined with upstreams. Upstream proxying and caching allow you to upload and use the packages you own, while Cloudsmith fetches and caches your dependencies from public repositories e.g. Maven Central or NuGet.org.
This enables you to use Cloudsmith as a first-class cache and a central source of truth for packages, to protect you from outages of external services.
Any supported package fetched from a caching-enabled upstream will be automatically scanned for vulnerabilities.
Multiformat repos
Multi-format repositories allow you to store artifacts of different formats in the same place. Organize your packages by environment, project, package type, or whatever way you see fit- we are not opinionated about how you organize your packages or containers.
Vulnerability Scanning
Package vulnerability scanning is a key step in securing your software delivery pipeline and reducing the risk of releasing insecure software into production.
Cloudsmith makes it easy for your team to introduce vulnerability scanning by automatically scanning every supported package format pushed to a Cloudsmith repository or fetched from a caching-enabled upstream.
Cloudsmith's vulnerability scanning feature is available for the following package formats:
- Docker
- Ruby
- Python
- Composer
- Maven
- NuGet
- Golang
- Cargo
- npm
Cloudsmith's Security Scanning page shows a breakdown of the number of packages scanned/not scanned, the age of scans, the pass/fail count overall and the maximum severity of those found.
Potential workflow for package promotion
- Setup your Dev repo with upstreams and your Staging repos with no upstreams
- Push your package to Dev repo
- Set up a webhook for scan results
- Automatically push the package to Staging when the results return with no vulnerabilities.
NOTE: you may want to run more tests against this package in real life before promoting it.
I'll assume you have installed the Cloudsmith CLI for the examples below. The CLI interfaces with Cloudsmith's API and allows users, machines and other services to access and integrate smoothly with Cloudsmith without requiring explicit plugins or tools.
Setup your Dev repo with upstreams and your Staging repos with no upstreams
Create 2 repositories Dev and Staging by following the instructions here.
Set up an upstream for the Dev repo by following the instructions here.
Push a package to Dev
A vulnerability scan is automatically triggered for supported packages every time they are pushed. If you set up an upstream, your 3rd party dependencies are also scanned for vulnerabilities.
Cloudsmith supports over 28 different types of packages. Uploading a package/binary/image to a Cloudsmith repository is simple:
- Upload via the package-specific native CLI / tools e.g. using Maven's mvn deploy, Node's npm publish, .NET's nuget push or Ruby's gem push.
- Upload via the API using tools/integrations (such as the official Cloudsmith CLI).
- Upload directly via the website.
To upload a package via the Cloudsmith CLI, use the cloudsmith push command:
cloudsmith push <format> OWNER/REPOSITORY <package_file>Set up a webhook for scan results
Setting up webhooks so Cloudsmith can send automated messages to alert another system when a package had been scanned is great for setting up vulnerability workflows.
Set up a webhook for vulnerability scanning by clicking Webhooks in your repository and then create a new webhook. Then:
- Select the Create Webhook button.
- Populate the Target Endpoint URL with the tool you want to receive your webhook like Jenkins, Zapier, CircleCI, Github Actions or BuildKite.
- Selecting the Package Security Scan Completed checkbox
- You can test out the Webhook by selecting to Ping (test) this webhook.
Promote a package
The webhook is triggered when the scan is finished and will have a payload in the HTTP req with information about the package and any vulnerabilities, including:
"slug_perm":"IB6FYhIvaoAy",
"max_severity":"Medium",
"num_vulnerabilities":140,"
"vulnerability_scan_results_url":"https://api-prd.cloudsmith.io/v1/vulnerabilities/demo/examples-repo/IB6FYhIvaoAy/"This information can drive actions such as promoting a package based on the maximum severity reported.
To promote a package using the Cloudsmith CLI, use the command cloudsmith mv` as follows:
cloudsmith mv OWNER/REPOSITORY/PACKAGE
For example, using the data from the snippet above where IB6FYhIvaoAy is the package identifier or the package slug:
cloudsmith mv demo/examples-repo/IB6FYhIvaoAy
Work to be done
Cloudsmith is working on our policy management feature which can be integrated into your package promotion workflow. This will allow you to set up policies that can be verified before a package can be promoted.
Conclusion
Package promotion used in conjunction with Cloudsmith's features like multiformat repositories, webhooks, vulnerability scanning and upstreams will set you up with a very useful workflow that can isolate and protect your production repositories from upstreams and vulnerabilities.
