By submitting this form, you agree to our 

Rust team's announcement of the 1.68.0 version of Rust

, Cloudsmith is happy to announce our support for the new "sparse" registry protocol, which has been stabilized as part of version 1.68.0.

Cargo is the package manager for Rust, a programming language empowering everyone to build reliable and efficient software. Cloudsmith customers who migrate to the new beta "sparse" registry implementation will dramatically improve the speed and reliability of their builds.

What's changed with Cargo's sparse protocol

Cargo's "sparse" registry protocol defines a new mechanism for communication with remote registries such as crates.io and Cloudsmith. The sparse protocol switches to an HTTP-based transport from the previously supported (and still default) Git-based transport. The Git-based protocol works by cloning a repository that includes JSON metadata for 

The Git-based method of serving the registry has several advantages, including:

Crates.io could rely on GitHub to store metadata, offloading some infrastructure requirements

Incremental updates (determining what's changed) are trivial with Git

Git has a well-understood API that allows for simple integration with other tooling

However, because Git can be highly resource intensive and because Git expects repositories to maintain a clean history over time, this approach began to fail for providers as repositories grew in size. Cloudsmith customers saw this impact, as did the main crates.io registry users. To deal with the problems before they got out of control, The Cargo team implemented an HTTP-based protocol that Cargo could use to pull crate metadata from registries (either public or private).

This new "sparse" protocol allows Cargo to fetch package metadata via simple HTTP GET requests that return JSON for 

 the packages that the client requests. This means:

Only metadata for packages that the build uses need to be pulled

Registries no longer need to maintain long-lived state (git repositories) to serve to clients

HTTP caching and other techniques can be used to speed up performance dramatically

 larger sets of packages due to no longer needing to deal with them all at once

Migrating to the new sparse registry implementation

We recommend Cloudsmith customers migrate to the new beta "sparse" registry implementation as soon as possible. Customers with 1000s of packages in their registries will see a dramatic improvement in the speed and reliability of their builds. The only migration requirement is that you're using Rust 1.68 or nightly.

Please have a look at the documentation in your repositories for setup instructions.

Deprecating support for the Git-based method

It is paramount to Cloudsmith that we maintain a performant and reliable platform for our customers. To ensure we do so for Cargo users, we intend to implement a limit on the number of packages a customer can have in a single repository, and to deprecate support for the Git-based protocol later this year. Currently, the sparse protocol is planned to become the default for crates.io in the 1.70.0 release coming in a few months.

Paddy Carey

Principal Engineer

Experience faster, more reliable builds with Cargo's new sparse registry protocol

At Cloudsmith, our mission is to make it as easy and straightforward as possible to get your packages into our system from where you can manage and distribute them to your heart's content.

To this end, we recently spent some time with Atlassian's Bitbucket team at 

 that makes it simple for Bitbucket users to easily turn their source code into distributable packages and deploy to Cloudsmith in just a few lines of YAML.

Pipes build on top of Bitbucket's excellent 

 functionality which allows users to automate their build and deploy processes, whatever form they may take.

Bitbucket Pipes are, at their simplest, the combination of a Docker container and some YAML-formatted configuration. Pipes are designed to remove what would otherwise be repeated or verbose configuration from a user's pipeline. Pipes can be shared across repositories or organisations and make many common tasks almost trivial.

The Cloudsmith pipe is included in Bitbucket's collection of officially maintained pipes and is available by default for any user that wishes to include it in their pipeline.

To use the pipe you'll first need an application or library that can be packaged into one of the formats that Cloudsmith support. You can see examples of Python and Javascript libraries in 

For the purposes of this example we'll assume you're using the Python example from the link above.

First, ensure that you can package your code using a Bitbucket pipeline by editing 

 in your repository. You can do so either via Bitbucket's web UI or using your normal editor and Git workflow.

For the example library, a basic pipeline that builds a package when a new tag is created looks like so:

This pipeline will build a new package each time you push an appropriate tag you your Bitbucket repository. The built package is then discarded as we have not yet added further instructions to tell Bitbucket what to do with it.

Next we'll add publish the package to Cloudsmith using our official pipe. We'll need to provide an API Key that the pipe can use to authenticate with Cloudsmith. You can find the official documentation for pipeline variables in the 

The API Key should be added as a "secure" variable (so it doesn't leak into logs) with the name 

Once authentication is configured, you can add the pipe configuration to your pipeline.

If using the web UI to configure your pipeline, you can select the Cloudsmith pipe right from your browser, from within the list of supported pipes as in the image below:

If using your local editor, you can follow the instructions in the 

Once added, your pipeline YAML should look something like so:

This configuration instructs the pipe to push artifacts to the 

 repository, with the API key we stored earlier. We're uploading a 

And that's it! Your pipe is now configured and ready to run. You should be able to push a new tag to Bitbucket and see your pipe run.

Bitbucket's pipeline UI provides a great overview of the status of your jobs:

Once pushed, you should see your shiny new package in the Cloudsmith UI, ready for download and install using your preferred tools:

Cloudsmith's Bitbucket pipe provides the easiest and simplest way for Bitbucket users to push their assets to Cloudsmith. Our official pipe is maintained by the Cloudsmith team and you can be sure it'll be kept up to date with all changes and features as we release them.

As always, if you have any questions about the pipe (or anything else), we're available at support@cloudsmith.io.

Cloudsmith + Bitbucket

The subject of containers probably doesn’t need much of an introduction. Since the launch of Docker in 2013 containers have become almost ubiquitous, with 

 confirming their organizations used containers in some way during 2019.

That is no surprise. Docker itself calls the container “a standardized unit of software”, which is a nice way of saying that a typical container packages up all the relevant code, system tools, dependencies and libraries within an application, and effectively isolates that application from the rest of the environment and infrastructure.

As a result, containers have one great merit: they will reliably run, and run in the same way, wherever they are deployed. At a stroke, they eliminate the “well it’s working on my machine” problem that has caused endless heartache for both development and operations teams over the decades.

That benefit should not be under-estimated.

 For developers in particular, it is rather wonderful to be able to put everything into a container and throw it over the wall 

 that it is going to deploy, and in turn knowing that it isn’t going to come back over the same wall for round 2.

It’s also - at least in theory - good for the operations team as well, for all of the same sorts of reasons.

What Is A “Unit” Of Software, Really?

Prompted by the Docker definition of container I mentioned above, it might be instructive to ask ourselves whether the container is either the only, or indeed the best, ‘unit’ of software.

If we take a 10,000 ft view of the history of software development it probably looks something like this:

The entirety of the source code is the unit

In the old days of source code you really had to be on top of things. The whole thing stood up or fell over as a whole, and things were pretty painful and expensive if (when) it did the latter.

When the package became the unit, things improved

. Certainly, if we were following good package management and distribution best practices, then in most cases things fell over less, and were easier to fix when they did. I’ll talk more about this later, but let’s also remember that packages also help speed up development full stop as they enable us to more easily re-use code and services.

In the move to containerization, the third stage in our evolution, the ‘unit’ of software has got bigger. As it has done so, it has concealed complexity in order to enable that crucial isolation from the environment it runs.

Essentially, the container runs as a kind of black box. The operations team just need to know what it does. They don’t need more detailed metadata and they don’t need to know what is inside it. After all, it’s all one self-contained package. So the unit of sofware has not only got bigger, it has also become less transparent.

The problem with a large unit size is that within it are likely to be hundreds if not thousands of dependencies.

And the problem with being opaque is that we don’t know what they are.

What happens when something goes wrong? Let’s imagine (and this won’t take too much effort for veterans of 

 or one of any number of dependency related screw-ups) that a vulnerability is discovered within a certain package.

Where is that package currently in use in our ecosystem? We don’t know. Can we quickly roll that package back to a known ‘safe’ version wherever necessary? No.

Containers - as a ‘unit’ of software - live or die as a whole

. If something is wrong, or might be wrong, we have no option other than to spin up a new version and redeploy: not necessarily a simple task. And as we don’t necessarily know which containers are affected, we find ourselves struggling to get on top of an emerging security crisis based on limited data. Not a nice place to be.

My last point is this: that containerization has led to a general decline in diligence and observance of best practices when it comes to handling packages and dependencies. After all, if I am going to throw them all into one container and check if it runs does it really matter where I get packages from?

The longer answer is yes, because if we aren’t going to be sure what is in any given container further down the line, we should be as careful as we possibly can be that nothing we can’t stand over sneaks in now.

Unfortunately, that doesn’t always happen. The path of least resistance is to get dependencies from pretty much anywhere and live with the consequences later. And as the developer cares about speed, and the ops team will be the ones dealing with those consequences, there is all the more reason to cut corners.

As developers, we need to get back to being diligent - and consistently diligent - about questions of security, provenance, reliability and availability when it comes to packages and dependencies

. We need to be as sure as we can be that what we integrate into our projects (whether destined for containers or not) is what it says it is. And we need to know what we used where and when - so that we can fall back to a previous version quickly and easily if necessary.

It’s the least our colleagues in operations, and the users of our end product, deserve.

Did We Lose Something With The Adoption Of Containers? And Can We Get It Back?

At Cloudsmith, our goal is to make it as easy as possible to get your packages into our system from where you can manage and distribute them according to your needs.

CircleCI is a continuous integration and delivery platform that helps teams release quality code, faster. They provide a mechanism for users to share code and libraries which can be reused across pipelines called "orbs".

Cloudsmith provides first-class support for CircleCI with our official 

. Using the orb, users can easily integrate publishing to Cloudsmith with their existing CircleCI workflows.

Full reference documentation for the orb can be found on the 

. This documentation is automatically generated from the orb itself and so is guaranteed to always be up to date with the latest release of the orb.

To use the orb you must first ensure you are using Circle version 2.1. At the top of your 

You'll need to configure authentication credentials for the orb. To do so, you can add an environment variable named 

 within the CircleCI settings page for your project:

The orb (for now) requires that you have already built the package you wish to publish. Assuming you're publishing a Python library (though the same process applies to any package type), you'll want to run 

Once built, we can use the orb to easily publish the package. The orb provides a number of commands to make this process simpler. We'll first ensure the Cloudsmith CLI is configured and installed, then after we've built the package, publish it:

Putting this all together, we end up with a 

Our official orb provides simple integration for the majority of standard CI usecases, but we know that it won't fit every purpose. For additional flexibility users can mix and match commands provided by the orb and/or use the Cloudsmith CLI directly.

For example, to use the orb to install and configure the CLI, but then use the CLI directly to publish to Cloudsmith, your configuration might look like so:

As always, if you have any questions about integration or would like some general advice, please 

Cloudsmith + CircleCI

It's an exciting day at Cloudsmith HQ, today we're launching beta support for hosted private LuaRocks module repositories.

Created in Brazil in the early 90s, Lua is a lightweight language, designed primarily for embedding into other applications. Lua has a focus on speed, portability and extensibility, making it the perfect choice in many situations. Lua works equally well on all major platforms, whether you prefer Linux, Mac or Windows.

LuaRocks is the package manager for Lua. LuaRocks is responsible for downloading, building and installing dependencies, known as Rocks. The primary source of open-source LuaRocks is the 

. LuaRocks.org is a community-run hosting service for Lua modules and provides a valuable resource which is central to the Lua ecosystem. Anyone can upload Rocks for distribution, either in the main repository or in a per-user repository which is provided by the service.

Whilst LuaRocks.org is the main repository for rocks, LuaRocks provides all the necessary tooling and configuration flags to easily allow pointing at another repository if needed. Cloudsmith, as of today, can provide an alternative repository for hosting your rocks as needed.

There are a number of great reasons you may wish to run your own private repository:

To develop Lua modules internally and share them privately to other teams.

To distribute and deploy your own Lua modules in a pipeline at your org.

To distribute Lua modules as vendored software (i.e. maybe commercially).

To make modifications to public Lua modules, without republishing publicly.

To mirror public Lua modules, to isolate from uncontrolled upstream issues.

To capture the exact state of your dependencies at a particular version/release.

To control (whitelist/blacklist) the exact Lua modules allowed for your org.

To keep track of the exact versions/releases of Lua modules you have/use.

If you're interested in the possibilities that hosting your own rocks brings, then Cloudsmith is here for you. Sign up today to begin hosting and distributing your Lua modules at the world's first hosted private Lua modules repository service.

Getting started with Cloudsmith and LuaRocks couldn't be simpler. First, you'll need a Cloudsmith account and a repository to which you can upload your modules.

If you need to install Lua and LuaRocks you can find instructions on the 

Cloudsmith should work with all supported versions of LuaRocks, but we reccomend using at least 

 for the best experience. You can check your local version like so:

Next, you'll need a rock for uploading to Cloudsmith. LuaRocks has several methods for doing so, you can create source, binary or metadata-only rocks. For full documentation on creating your own rocks, see the tutorial on the 

For demonstration purposes we'll use a rock from LuaRocks.org named 

Next, you can push the module to Cloudsmith using the 

. We don't yet support uploading modules with 

, but we'll be adding support in a future release (if you want it, please let us know). For now, use the CLI.

First, ensure you've installed the Cloudsmith CLI and configured authentication:

Then, use the CLI to upload the module, which operates like 

Your repository should now contain the uploaded modules. An example of which is our official Cloudsmith examples repository (this is where we upload our examples to), as shown below:

Installing packages from a custom LuaRocks server couldn't be simpler, we only need to pass a single command-line argument to specify the location of the custom repository:

If required, you can configure the repository centrally in your 

It really is that simple, and now you've sourced a module privately from your very own private Lua module repository on Cloudsmith. Well done!

Cloudsmith provides fully featured Lua module repositories on all plans, flexible enough for use whether you’re hosting public modules for a public or open source project, or private modules for your company’s internal needs. We're extremely proud to be able to support the Lua ecosystem with this tooling.

You can find further, context-specific information, including detailed setup and integration instructions inside each Cloudsmith repository. You can see an example of this documentation in our 

Lua ROCKS!

 hosting, with ultra-fast and secure delivery of your Helm charts; for the ultimate in integration with the Kubernetes (k8s) ecosystem.

How to host your own Helm repository using Cloudsmith

Originally built by Deis (later acquired by Microsoft), Helm is a package manager for Kubernetes that allows development and operations teams to easily manage and deploy these increasingly complex cloud native applications to their Kubernetes clusters. Helm allows you to manage applications on your Kubernetes cluster in much the same way as you’d manage applications on your Linux server with apt or yum.

 (otherwise known as K8s) is an open source system to deploy, scale, and manage containerized applications. Kubernetes has quickly become one of the most popular container orchestration platforms used by development and operations teams all over the world. It was open-sourced by Google in 2014, built based on years of learning deploying containers at scale inside the search giant.

Despite Kubernetes’ rapid rise to popularity, deploying and managing applications can be a complex and daunting task, often requiring the operator to write and configure multiple inter-dependent configuration files for various parts of the systems being deployed. As our applications become more complex, more distributed, the difficulty of managing them grows even further.

A Helm chart is a collection of Kubernetes YAML template files organized into one single directory. Helm packages are also known as “Helm Charts” and are stored in a “Chart Repository”. By default, Helm comes bundled with the “stable” chart repository, hosted for free by Google. Most public charts are hosted here, mostly provided by vendors packaging their own software for use by others.

Through Helm charts, Helm is used to manage Kubernetes applications. It works by packaging up a set of YAML definitions along with the necessary configuration to quickly stand up all components of an application in a repeatable way. A single chart can be as simple or complex as necessary, deploying anything from a single container to a full distributed application. Helm combines these application definitions with user-provided configuration to allow simple overriding of configuration where needed, allowing users to concentrate on shipping software and not on the nitty-gritty of configuring every application they need to run.

The main requirement for using Helm is a working Kubernetes installation. Setup of Kubernetes is outside the scope of this post, but you can get started locally with the installation of 

, you can initialise Helm on your cluster:

, Helm’s server-side component, responsible for interacting with the Kubernetes API and managing state.

You can then run other Helm commands as needed. To view all available charts you can use the following command:

You should see a full listing of the built-in “stable” repository which contains all supported public charts. You can install any of the charts from the 

This instructs Tiller to pull the packaged chart from the stable repository, render all the templates contained within, and apply those to the running cluster. You can inspect the status of the application with regular Kubernetes CLI tooling:

Or check the status of the chart installation itself using the 

If using only public Helm charts to install third-party applications, then the above instructions are enough to get started in most cases. But if you want to install software for which there are no public Helm charts, whether your own or third-party, you can create your own chart and host it using Cloudsmith.

If you’re creating your own chart, you can use one of Helm’s starter templates to quickly scaffold out the directory for you:

 folder containing all the files and folders necessary to build your chart. You can adjust the templates as needed for your application, 

 is a really useful resource when building your own chart.

Once you’ve finished adjusting your templates, you can create a Chart package with the CLI:

Once built you can push the chart to Cloudsmith. First, ensure you’ve installed the Cloudsmith CLI and configured authentication:

Then, use the Cloudsmith CLI to push the package:

Once pushed, you can configure Helm to install packages from your Cloudsmith repository:

And then install your application to Kubernetes:

There you have it! We have also produced a handy video guide on getting started with Helm using Cloudsmith, which you can follow below:

Getting Started with Helm and Cloudsmith

A short video demo of how to get started with Helm and Cloudsmith, including pushing a Helm Chart to private Cloudsmith repository and then pulling and installing the Chart on a Kubernetes Cluster.

Helm is a powerful tool to help manage the ever-growing complexity of deploying and running cloud native applications on Kubernetes. Like other package managers Helm provides a robust suite of tools for installing, configuring and managing applications on its target platform, Kubernetes.

Cloudsmith provides fully featured Helm repositories on all plans, flexible enough for use whether you’re hosting public charts for an open source project, or private charts for your company’s internal applications.

You can find further, context-specific information, including detailed Helm setup and integration instructions inside each Cloudsmith repository. You can also see an example of 

Cloudsmith Blog

Experience faster, more reliable builds with Cargo's new sparse registry protocol

Cloudsmith + Bitbucket

Did We Lose Something With The Adoption Of Containers? And Can We Get It Back?

Cloudsmith + CircleCI

Lua ROCKS!

An Introduction to Helm