Custom EULA

It is simple to add a custom End User Licence Agreement for your package downloads in a Cloudsmith repository.

Dan McKinney

Solutions Engineer


As a vendor, understanding your bandwidth usage is an invaluable insight into how packages are distributed across your user base and how specific users have grown over a timeframe.

As a user base grows from 10s, 100s, and 1000s of users and package downloads are many multiples of your total number of users, it's essential to understand the distribution of your bandwidth utilisation by a user (or more precisely, entitlement token's) to aid in the identification of token's that make up a large percentage of your overall traffic.

When providing an entitlement token to a user under a specific license, you trust an entitlement token will be used to download packages upon the agreed-upon terms. However, an increase in your total bandwidth may start off slow and continuously grow over several months until it's suddenly become a massive increase in your overall bandwidth. Understanding this usage and precisely where this growth originates helps identify where/where change occur and provides options to mitigate runaway bandwidth from specific entitlement tokens by changing how those specific users are managed.

Whenever a user downloads a package from a private repository using an entitlement token, the exact number of bytes transmitted from our servers to the user host is stored for that specific token as an individual log entry. At the end of each day, we calculate the total bandwidth for every user across all of our repositories containing one or more packages and store the result as a daily aggregated value representing bandwidth usage.

The easiest way to get started exploring these metrics is to check out the metrics command within the Cloudsmith CLI. Alternatively, to get more fine-grained metrics, you can implement a programmatic solution using our API or one of the API binding libraries published in various languages.



Using the Cloudsmith CLI you can quickly and easily query the total bandwidth usage for your repository by running the following command with your organisation/repository.

The following screenshot shows an example repository contain a number of active and inactive entitlement tokens alongside the total bandwidth used. The statistics table provides a simple insight into min/max/average token usage.

You can also retrieve usage metrics for one or more specific tokens by providing a comma separated list of Entitlement token (identifiers)

cloudsmith metrics tokens cloudsmith/example --tokens=ZGCV58VqT8Sl

If you wish to drill down further into a specific period of usage for a token, you can supply am epoch timestamp for the 

 parameters to include only usage within that period. In this example, a single token is displayed for all of 2019.

Vendors Rejoice! Analyse your Bandwidth Usage

Using Cloudsmith to store and distribute any type of file

A quick walkthrough of how you can use a private Cloudsmith repository to store and then distribute any type of file.

At Cloudsmith, we want to be your “one central source of truth” for your dependencies and package management needs. And in keeping with this ideal, we are extremely pleased to announce that we have added fully configurable transparent Proxying and Caching support for Debian packages.

Well, in short, it means that you can now use your private Cloudsmith repository for all of your Debian package needs – whether that is your own private packages or packages that you need from public upstream sources. Your private Cloudsmith repository is all that you need to handle both.

If you request a package from your Cloudsmith repository, and that package isn’t present in the repo, then Cloudsmith will automatically check any upstream repos that you have configured and will then fetch (and optionally cache the package for future requests) from the upstream.

This brings you several important benefits:

. Your Cloudsmith repository is the only repository you need to configure on your clients. No more need to configure multiple repos, and handle multiple authentication credentials etc. Configure the upstreams once in Cloudsmith, and that’s it done.

. If you have cached packages and dependencies that you require in your Cloudsmith repository, then if the upstream repo goes down, is otherwise unavailable, or if the packages are removed then you can still access your cached versions. No more breaking of build or deployment process due to an unreliable upstream.

. You can view details on what specific packages were requested from the upstreams. Gain insights into what you have, and what’s missing – or who and what else you are currently relying on.

. Cloudsmith repositories are backed by a performant, global CDN. This means that your own packages and those cached from an upstream are delivered with the same low latencies and speeds. Going further than this, with edge nodes in almost all geographic regions, your users will experience this performance wherever they are located. Distributed teams all benefit equally.

. All of your packages and dependencies in one place means it’s easier for you to implement the controls and security policies that you need. Multiple sources mean multiple management tasks. Keep everything in one place and keep a tighter hold on what you have.

Well, it’s easy. In your Cloudsmith repository, you’ll see a menu item called “Upstream Proxying”. This is where we will configure our upstreams. Simply click the “Create Upstreams” button and select “Debian” to create a new Debian upstream:

You are then presented with the “Edit Debian Upstream” form. This is where we enter the details of the Debian upstream we wish to use.

We add a Name for the upstream, it’s URL and a priority weighting- in cases of multiple upstreams this will determine the order in which they are checked for a package.

We can then choose to fetch and cache any requested package (instead of just fetching them), and to verify the SSL certificates provided by the upstream. In addition, we can choose to enable this upstream for source packages too.

Next, we select the distributions and architectures that we wish to use this upstream for, and finally, we can add optional authentication headers (for private repositories that require authentication) and also optional arbitrary headers, if you wish to send something custom along with your request.

And that’s it, we have now added a new Debian upstream to our Cloudsmith repository.

Behind the scenes, Cloudsmith will now start to index the packages available in the upstream repository. The upstream will be ready for use as soon as the indexing is complete.

So now, for the next request for a package that isn’t present in this repository, Cloudsmith will check the upstream and fetch it if it is available there and also cache it in the Cloudsmith repo (if you enabled caching) for future requests. It’s that simple.

Debian upstream proxying and caching support is just another step on our path to providing you with the centralized controls, security, management and visibility that you need to enable a modern, high-velocity and DevOps-first workflow for your package management needs. It means fewer things to worry about, less exposure to change and most importantly....

Caching and Upstream Proxying for Debian Packages

Today managing your licenses with Cloudsmith has become incredibly simple.

Now, with the help of our License Compliance UI, not only can you update the license associated with a package without needing to modify a package, plus you can also view statistics of how your overall licenses appear across all packages within a repository. Don't believe me? Have a look at the screenshot below:

From the overview section, we can see the breakdown of total packages by format, types of licenses used across all packages in the repository, and the number of unlicensed packages in the repository.

In this example, we want all our licenses to be uploaded with the MIT License; however, several packages are missing licenses, and one package has uploaded with the wrong license. Let's open the edit view for the django-guardian package and update it to use a MIT License.

At Cloudsmith, we endeavor to match the license defined within a package's metadata as accurately as possible. For example, the BSD license defined within this package's metadata is checked against a valid SPDX license.

The SPDX License List is a list of commonly found licenses and exceptions used in free and open source and other collaborative software or documentation. The purpose of the SPDX License List is to enable easy and efficient identification of such licenses and exceptions in an SPDX document, in source files or elsewhere. The SPDX License List includes a standardized short identifier, full name, vetted license text including matching guidelines markup as appropriate, and a canonical permanent URL for each license and exception.

Anytime Cloudsmith matches a license automatically, we will always provide a description of what the license was defined as within the package's metadata, our confidence of how accurate the match is, and the new license that has been applied. You can find the following description on the edit page for any license that has been automatically applied:

> The 

 license provided within this package’s metadata is a 

 SPDX license and was automatically added to this package.

Anytime the match is not accurate to a high percentage, or the license is not supplied, we leave the license empty for you to decide how you want to resolve it.

To change this license or add a new one, select a new license from the autocomplete/dropdown and click the Edit button to save this change.

Once all the packages within a repository have been updated with a license, the overview provides an easy way to confirm all packages are using the same license and all packages are licensed.

Simplify License Compliance

Using GitHub Actions to push packages to Cloudsmith

Dan walks through how to connect up GitHub Actions to push packages to Cloudsmith.

GitLab CI/CD is a tool that is built into GitLab. It allows you to create automated tasks that you can use to form a Continuous Integration and Continuous Delivery / Deployment process.

You configure GitLab CI/CD by adding a yaml file (called `.gitlab-ci.yml`) to your source repository. This file creates a pipeline, which will then run when a code change is pushed to the repository. Pipelines are made up of a series of stages, and each stage can each contain a number of jobs or scripts. The GitLab Runner agent will then run these jobs.

For an on-premise instance of GitLab, you can install the GitLab runner agent on your own instances, and it supports many different operating systems (thereby creating your own fleet of instances to run your pipelines). But to keep things simple in the following examples, we will use gitlab.com and the default hosted GitLab runner environments provided.

So why would you want to use Cloudsmith with your Gitlab CI/CD pipelines? Well, there are a few reasons:

Universality – Cloudsmith supports over 20 package formats, so whatever artifacts your project produces, you can find a home for them in a private Cloudsmith repository

Control – Cloudsmith private repositories provide extensive security and access controls, that have been designed to accommodate workflows such as internal development, deployment or even distribution to external customers. The fine grained permissions system available enables you to craft bespoke access control, and lock down or open up your repository as much or as little as you need.

Automation and Integration – Thanks to the Cloudsmith CLI and also native support for format-specific package managers, a Cloudsmith private repository can fit in seamlessly with other tools in your development or distribution processes. It provides you with a single source of truth across your packages and dependencies.

Performance and Reliability – As a cloud-native platform, Cloudsmith manages the availability and performance of your repositories. You don’t need to worry about managing a fleet of servers, containers or virtual machines. Global performance, backed by our ultra-fast CDN and multi-region infrastructure, ensures that your packages are delivered worldwide. Reliably, quickly and securely.

That’s not all. With features like custom domain support, configurable upstream proxying and caching, configurable edge caching rules, download logs / statistics and more, Cloudsmith aims to provide the best solution for all your package management needs. It really is the ideal tool in a high-velocity CI/CD workflow – precisely the type of workflow that GitLab CI/CD is intended to enable you to create.

OK, so let’s get started with a worked example. The very first thing you will need is some source code in a GitLab repository that you want to build. For this example, we will build Ruby source code into a 

Our project on GitLab has the following structure:

The important thing here, as previously mentioned, is the .gitlab-ci.yml file. This is where we define the GitLab CI/CD pipeline that will run when we push a change to the GitLab repo.

Let’s take a look at the .gitlab-ci.yml file:

The first thing we have specified is the image that will be used when creating the Docker container for the GitLab runner that will execute this pipeline, in this case, Ruby 2.5.

Next, we add an environment variable, RUBYGEMS_HOST. This is where we define the URL of the Cloudsmith package repository that we will push the result of the build to.

We then define two stages in this pipeline, the build stage and the push stage.

This stage is pretty straightforward. We just run the `gem build` command to build our Ruby source as defined in our cloudsmith-ruby-example.gemspec file. Following this, we have defined an `artifact` job, as we need to temporarily store the output of the build so that the push stage next can use it.

This is because different stages in a pipeline will run on a new runner instance, so a subsequent stage wouldn’t have the access to the package built in a previous stage. You could also perform any jobs required to build and push within the same stage, and therefore on the same runner to avoid this, but for more complex pipelines you’ll likely have many stages.

The push stage is a little bit more complex. This is due to the fact that we are going to push our package to a private Cloudsmith repo, which requires authentication. The Ruby package manager, gem, allows you to store your authentication credentials (in our case, the Cloudsmith API Key) in a credentials file, located at ~/.gem/credentials – But of course, we don’t want to check this credentials file into our GitLab repository along with our source!

So we can make use of GitLabs ability to add variables to the source code repository. We can create a file variable called “CLOUDSMITH_API_KEY”, and then as part of the push step, we add a job to move this variable into the required location before we run the `gem push` command.

You add this file variable in your GitLab repository settings:

Now, when our push step runs, the `mkdir` and `mv` jobs will create the required ~/.gem/credentials file (with our Cloudsmith API Key in it), and all without exposing our API Key in any logs on the runner instance, nor checked in with our source code.

The final job in our push step simply runs `gem push` to upload the package we have built to our private Cloudsmith repo, as Cloudsmith repositories offer full native support for the `gem` package manager.

All that is left to do is for us to make a change to our source, and the commit and push the change to our Gitlab repo. Let’s see what happens when we do that.

The Gitlab CI/CD pipeline starts, and the build stage executes:

The build stage runs `gem build` to build the package and then stores it in GitLab’s temporary artifact storage:

The Push stage then starts to execute once the Build stage completes:

The push stage downloads the package from the temporary artifact storage, creates the required `~/gem/credentials` file, and runs `gem push` which uploads the package to our private Cloudsmith repository:

And that’s it, the pipeline is now complete and reports as "Passed":

If we now go and login to Cloudsmith, and check our `examples-repo` repository, we can see that the Ruby gem we just built is present:

Integrating a private Cloudsmith repository with GitLab CI/CD pipeline is easy, whether you are building a package that Cloudsmith provides native tooling support for (like Ruby) or even if you are using the Cloudsmith CLI to push a raw file, or other binary artifact. You can add your own private Cloudsmith repository with just a couple of lines of configuration, and it just works.

Package management can be complex and difficult, and doubly so if you are trying to manage your own package management solution and infrastructure at the same time. Try it for yourself, and see the productivity and efficiency gains that you can get from a centralized, hosted, secure package management service.

Integrating a Cloudsmith repository with a GitLab CI/CD Pipeline

Using GitLab to push packages to Cloudsmith

A short demo of how you can use GitLab CI/CD pipelines to push a package to a private Cloudsmith repository.

A key factor in good package management is organization. How you organize and structure your repositories will help you to unlock the efficiencies and promises of modern DevOps processes.

There are, of course, a multitude of ways you can organize packages. You can group by version numbers, formats, architectures, filetype and more. At Cloudsmith we have supported this by extracting as much of this metadata as possible when you upload a package and making this metadata available for searching/filtering.

Sometimes, however, the metadata just wasn’t granular enough – or it didn’t provide quite the nomenclature that you may have wanted. You wanted more. And we are extremely pleased to say that we have now added a new feature that greatly expands upon this functionality. Presenting:

So, what is new about this? And why does it matter?

Well, in short, we now give you the ability to add ANY custom tags to ANY package or container, either during package upload or after the fact. And you can do this via the Cloudsmith CLI or the Cloudsmith API.

Let’s say for example that you were using Cloudsmith to distribute your packages to end-users, and you have “Free” and “Premium” editions of your package. Well now you can simply add “Free” and “Premium” tags to the respective packages, and then create 

to give your users access to just the packages they should be allowed – without resorting to adding the edition into the filename or creating two separate repositories. You can manage it all from one central location. And of course, as Cloudsmith repositories are fully multi-tenant for package formats, you can apply these tags across all the package formats you use, all in one repository.

Or perhaps you would like to tag a package based on where it is deployed. You could tag a package as “rest-api”, or similar, to differentiate where in your production application the package is used.

Let’s start off by listing the packages in our demo repo. The CLI command for this is:

So, we can see that we have a collection of 

 in this repo, with various different versions. Let’s check one of those packages to see if it has any custom tags attached. The CLI command for this is:

OK, this package does not have any custom tags. Let’s now add one, and the CLI command for adding a tag is:

Great, we have now added a custom tag “free” to this package. We can repeat this for the other packages in the repository, varying the tags to suit our purpose. In addition, we can also specify that tags are “Immutable”, which means they can only be removed or altered by someone with Administrator permissions on the repository, or by the package owner. When we are finished adding tags, we can look at the repository on the Cloudsmith website:

You can see the new custom tags listed for each package alongside the tags that were automatically created from the metadata when the packages were processed after upload. We can now use these custom tags in any searching / filtering we need to do, or add them as a restriction on an Entitlement Tokens that we create for the repository:

The syntax for searching / filtering is the same syntax you use when creating a search-based restriction for an Entitlement Token, so it really is easy to create a set of access tokens that divide the repository up into subsets of packages.

Visibility of the attributes of your packages is important, it will help you not only structure your repositories, but can also help you gain insight on what package is where or what package a group of customers can access. And it’s not just the basic attributes of a package that you need visibility on, it’s anything about a package that matters to your organization and workflows. With universal package tagging you now have the ability to add your own searchable attributes to your packages, so you can define what is of importance.

Universal Package Tagging

Puppet is a Continuous Configuration Automation tool that’s you can use to automate the configuration of your entire infrastructure. You can use it to manage the configuration of anywhere from a few, to thousands of servers or devices.

The Puppet server is where you create and store your configurations and define which nodes specific configurations will be applied to. Puppet uses modules and manifests to describe how a machine should be configured and they are written in a ruby-like syntax that describes the desired state of a node.

The Puppet agent is installed and runs on each of your nodes. The agent pulls configurations from the Puppet server and then applies them to a node to bring the node into the desired state. It can be configured to periodically check the Puppet server for new or updated configurations (which would be the usual configuration in a Puppet deployment), or it can be started manually on a node as required.

The Puppet server must be installed on Linux but the Puppet agents can be installed on, and manage, Unix / Linux like systems, MacOS, or Windows. Installing the Puppet server and agent(s) is beyond the scope of this article, but for further details, you can see the official 

You can use a private Cloudsmith repository in combination with the automation power provided by Puppet to bolster your pipeline with more security and control over your builds and dependencies. Puppet is an ideal tool to automate the configuration of a Cloudsmith repository on nodes in your infrastructure, and to then also install any required packages and dependencies from this repository. The key is control - control over your packages via your Cloudsmith repository, and control over your builds and configurations via Puppet.

For this example, we will use Puppet to set up a Cloudsmith repository for 

 packages, but Puppet also provides official modules to support the most popular repository types, like yum, pip, 

On our example Puppet server, we will create a module to configure a Cloudsmith repo and install a package. I have created a directory for this module, called cloudsmith-repo. In this directory, I have also created a manifests directory, and this is where we create our manifest (called init.pp).

Let’s look at each bit of this manifest and explain what it does.

Our manifest starts with a class called cloudsmith-repo, the same as the module name.

First we add the puppet “apt module” with:

Since our Cloudsmith repository works as a native apt repo, we can use the class and types from the apt module to define the configuration of our repo.

The next thing we do is use the apt:key type to set up the GPG key for our repository. We need to add the 20 Byte Fingerprint of the repository GPG key, which we can get from the signing keys section of our Cloudsmith repo, and the URL for the key:

https://dl.cloudsmith.io/TOKEN/OWNER/REPOSITORY/cfg/gpg/gpg.8-BYTE-FINGERPRINT.key

After this, we then use the apt:source type to set up the repository itself. We give the repository a short description, then add the URL for the repository - which also contains an access token as this is a private repository. In your Cloudsmith repo settings, you can create as many of these access tokens as you like, and add restrictions to each token. Or you can also use HTTP basic authentication instead, we support both types of auth:

comment => 'An Example Cloudsmith Repository',

https://dl.cloudsmith.io/TOKEN/OWNER/REPOSITORY/deb/debian

We will also add our Debian distribution version, and we also specify that this is a Debian source and binary repository, as Cloudsmith repos support both types of Debian package:

Finally, we declare the package that we would like installed, and we use ensure to specify the version of the package we would like - in this case the latest version:

So, putting that all together and adding in details for the repository and owner etc, we would have something that looks like this:

id => '7DF1831D968E392DAFD802A97D4D4CE49534374A',

https://dl.cloudsmith.io/QOH0JBRgKQx5lE7S/demo/examples-repo/cfg/gpg/gpg.7D4D4CE49534374A.key

https://dl.cloudsmith.io/QOH0JBRgKQx5lE7S/demo/examples-repo/deb/debian

~> package { 'cloudsmith-debian-example':

As stated previously, you would normally configure Puppet nodes to periodically check in with the puppet server to pull and apply configurations, but we will just kick off the puppet agent by running it manually with:

The puppet agent will now connect to the puppet server and will retrieve and apply the configuration:

We can then check to see that the Cloudsmith repo has been added as a source for apt in /etc/apt/sources.list.d/ :

And if we check that our cloudsmith-debian-example package is now present (with apt-cache showpkg cloudsmith-debian-example), we can see that it has been installed:

It really is quick and easy to set up a Cloudsmith repository and deploy your packages using Puppet. We try to ensure that things "Just Work", and when using the built-in apt_module support in Puppet (or other modules to support other repository types, like yum or pip) you should be up and running in minutes, not hours.

Using a private Cloudsmith repository allows you to level up the control you have over your packages and software artifacts, and using Puppet allows you to automate and control your infrastructure configurations. It's a great combination.

Deploy Packages From a Cloudsmith Repository With Puppet

In this blog, we will go through an example of how you can use the 

 to provision resources in Cloudsmith, such as repositories and entitlement tokens.

 is an awesome Continuous Configuration Automation tool. It is used to provision, update and manage infrastructure resources such as Cloud instances, containers, physical machines and more.

It is a firm favourite among developers, due to its brilliant community and mix of power and simplicity. At Cloudsmith, we use Terraform to provision and manage our environments (following a rather disastrous outing with an alternative tool), so we are big fans ourselves.

Terraform uses providers to interact with different APIs and expose any required resources. In the case of Cloudsmith, the Cloudsmith Provider allows Terraform to access the Cloudsmith API and manage two resources – Repositories and Entitlement Tokens.

Let’s take a walk through setting up Terraform to use the Cloudsmith Provider and create some resources.

The first thing that we need to do is to add the following to the "required_providers" in our example Terraform module:

This tells Terraform to download and install the Cloudsmith Provider, and which version of the provider to use.

Next, we need to add our Cloudsmith API Key, for use by the Provider:

Of course, you should never add credentials or secrets in plain text like this in a Terraform module. You should add them as environment variables, or better yet – use a proper secrets management tool like HashiCorp Vault. Also, Terraform Cloud also allows you to store these credentials securely for free.

But in the interests of simplicity, and to keep this example brief, we will just add our API Key here for now.

We now just need to run “terraform init” to install the provider automatically.

Then we add our namespace data source. The namespace data source is used to identify the Cloudsmith account that the resources we want will be created under:

The “slug” is the only required argument. It is the user-facing identifier for the Cloudsmith account, which appears in URIs.

And now we get to creating our resources. The first resource we will create is a repository:

namespace - The namespace is determined from our namespace data source that we added previously.

slug - A user-facing identifier for the repository, which appears in URI’s

The second resource that we will create is an entitlement token. Entitlement tokens provide read only access to a repository, and we can add restrictions to them to control what package they are allowed to access within the repository, and for how long:

namespace - The namespace that this token will belong to, which will be determined from the namespace that the repository we create belongs to.

repository - The repository that this token will be created for, which will be determined from our repository resource.

We have also added two optional arguments to this token, and they are two token restrictions.

The first restriction sets a maximum number of downloads that this token can be used for, and the second restriction sets specific packages, tagged “basic”, that this token is allowed to access. For more detailed information on tokens and restrictions, please see our token 

And that’s it. That’s all we need to add to create a repository and an associated entitlement token with restrictions.

Our complete terraform module now looks like this:

If we now just run "terraform plan" we can see the actions that terraform will perform:

The plan confirms that terraform will create two resources, a repository and an entitlement token.

Now, to go ahead and create these resources, we simply run “terraform apply”. We get the same output as we did from the “terraform plan” command, and we are asked to confirm that we want to perform the actions.

When we enter “yes”, Terraform then creates the resources that we have specified:

If we now log into Cloudsmith and check our repositories, we can see our new “terraform_example” repository:

And if we check the entitlement tokens within the repository, we can see the token that we created via Terraform, complete with token restrictions:

Terraform and Cloudsmith is a powerful combination. It allows you to automate the creation of your package repositories and associated access tokens in a controlled, reproducible way. Also, as you have now defined these resources as code, you can version them and treat them as you would any other source code that you have.

You can find the full documentation for the Terraform provider 

You’re one step further down the DevOps path, long may it continue.

Introducing the Cloudsmith Terraform Provider

Getting Started with Alpine Linux and Cloudsmith

How to get set up and started with your own private, secure repository for Alpine Linux Packages on Cloudsmith.

https://help.cloudsmith.io/docs/alpine-repository

Cloudsmith Blog