--- title: "Lee Skillen on Streamlined Software Packaging | Cloudsmith" description: "Our CTO, Lee Skillen, joins SemaphoreCI to discuss how Cloudsmith solves the complex issue of streamlined software packaging." canonical_url: "https://cloudsmith.com/blog/lee-skillen-from-cloudsmith-on-streamlined-software-packaging-building-startups-and-on-premise" last_updated: "2019-05-15T00:00:00.000Z" --- # Lee Skillen on Streamlined Software Packaging | Cloudsmith ## What is Rust? What is Cargo? What are Crates? Rust lang, or Rust, is a systems programming language, designed at Mozilla research, and intended to be a safer alternative to existing systems programming languages like C and C++, but still just as fast. Rust was originally started as a personal side-project by Mozilla engineer Graydon Hoare, but the company quickly and rightfully recognised its promise and began sponsorship of the project in 2009. Rust quickly became a critical part of Mozilla's efforts to improve the performance and reliability of their browser technology as part of the Servo project. Since then, Rust has only grown in popularity and appeal, year on year, and for the last four years has achieved the coveted top spot for "most loved programming language" on [Stack Overflow's Developer Survey](https://insights.stackoverflow.com/survey/2019#technology-_-most-loved-dreaded-and-wanted-languages) (also a great read). Whilst Rust is maybe best known as a low-level systems language, the same properties that make it appealing for working close to the metal also make it appealing for many other use cases. Rust's memory-safety, ownership semantics, performance, and comprehensive type system (among others) combine to make Rust a very compelling choice in many fields of engineering. Whether you're writing a device driver for an embedded System-on-Chip (SoC), an operating system, a layout engine for a web browser, a web application, a package management service (heh), or a library for use in the WebAssembly ecosystem, Rust probably has you covered. Early on, the Rust community realised that for a modern programming language to be successful, it needs modern tooling to accompany it. To this end, various teams in the Rust community have concentrated on tooling, such that Rust is now widely regarded as having best-in-class tools in areas like build and package management; which is something that we (Cloudsmith) recognise, respect and adore. Rust therefore ships with its own package manager called [Cargo](https://github.com/rust-lang/cargo). Cargo is an all-in-one frontend for building, packaging and configuring your Rust projects. It can build and run your code (providing a developer-friendly interface to `rustc`) and manages resolving and fetching any required dependencies (also known as "crates" in the Rust world). Cargo can install dependencies from a remote registry into your local project for building, using a command like `cargo install library`. Until April of 2019 the only choice you really had was the official, public registry at [crates.io](https://crates.io/). Crates.io is maintained by the Rust team and is the default public registry for the Rust ecosystem. With the [release of Rust 1.34](https://blog.rust-lang.org/2019/04/11/Rust-1.34.0.html), Cargo now officially supports the use of registries other than crates.io, making it possible to run an alternative registry for the hosting of private or internal crates. There are several equally fantastic reasons for running/using your own registry: - To develop Cargo packages internally and share them privately to other teams. - To distribute and deploy your own Cargo packages in a pipeline at your org. - To distribute Cargo packages as vendored software (i.e. maybe commercially). - To make modifications to public Cargo packages, without republishing publicly. - To mirror public Cargo packages, to isolate from uncontrolled registry events. - To capture the exact state of your dependencies at a particular version/release. - To control (whitelist/blacklist) the exact Cargo packages allowed for your org. - To keep track of the exact versions/releases of Cargo packages you have/use. As such, if you're interested in the possibilities that hosting your own private or internal crates brings, then this is incredibly good news for you: Cloudsmith are proud to provide the World's first commercially available public and [private Cargo registry hosting](https://cloudsmith.io/l/cargo-registry/), with ultra-fast and secure delivery of your Rust packages, alongside all of the usual Enterprise-grade features that we provide. ### See also: - Rust: [https://www.rust-lang.org](https://www.rust-lang.org/) - The Rust Programming Language: [https://doc.rust-lang.org/book/index.html](https://doc.rust-lang.org/book/index.html) - the Cargo Book: [https://doc.rust-lang.org/cargo/index.html](https://doc.rust-lang.org/cargo/index.html) ## Setting up a private Cargo registry at Cloudsmith Getting started with Cloudsmith and Cargo couldn't be simpler. First, you'll need a [Cloudsmith account](https://app.cloudsmith.com) and a repository to which you can upload crates. Secondly, you'll need to ensure you're running at least version 1.34 (or later) of Rust. If you need to install Rust you can use [rustup](https://rustup.rs/), the Rust toolchain installer. Rust is available on most commonly used development platforms. You can check your currently installed Rust and Cargo versions like so: ```json { "_key": "3882b9673602", "_type": "code", "code": "$ rustc --version\nrustc 1.34.1 (fc50f328b 2019-04-24)\n$ cargo --version\ncargo 1.34.0 (6789d8a0a 2019-04-01)\n", "filename": null, "language": "text", "markDefs": null } ``` If you see something like the above, you're ready to go! ### Publishing your Cargo crates to Cloudsmith For the purposes of this demonstration, we'll use the minimal crate created by `cargo init`. You can initialise your new Cargo crate like so: ```json { "_key": "3e417b5e4d8a", "_type": "code", "code": "$ cargo init my-crate\n Created binary (application) package\n", "filename": null, "language": "text", "markDefs": null } ``` If you take a look inside the new crate you'll see a few files: ```json { "_key": "f131b9b5cc6f", "_type": "code", "code": "$ tree my-crate/\nmy-crate/\n├── Cargo.toml\n└── src\n └── main.rs\n\n1 directory, 2 files", "filename": null, "language": "text", "markDefs": null } ``` `main.rs` contains a simple "Hello World" program, and `Cargo.toml` contains all configuration and setup data for your crate. For our purposes we can leave both files as-is. We can use `cargo` to build our crate and prepare it for upload to Cloudsmith: ```json { "_key": "6fff614676c4", "_type": "code", "code": "$ cd my-crate/\n$ cargo package --allow-dirty\nwarning: manifest has no description, license, license-file, documentation, homepage or repository.\nSee for more info.\n Packaging my-crate v0.1.0 (/Users/cloudsmith/my-crate)\n Verifying my-crate v0.1.0 (/Users/cloudsmith/my-crate)\n Compiling my-crate v0.1.0 (/Users/cloudsmith/my-crate/target/package/my-crate-0.1.0)\n Finished dev [unoptimized + debuginfo] target(s) in 1.45s\n", "filename": null, "language": "text", "markDefs": null } ``` **NOTE:** We use `--allow-dirty` in the `cargo package` command as we haven't committed the new crate to git. Under normal circumstances you'll just call `cargo package` without the extra flag. Cargo, by default, spits out your packaged crate under the `target/package/` directory. ```json { "_key": "e44a3ca5943f", "_type": "code", "code": "$ ls -l target/package/\ntotal 4\ndrwxr-xr-x 6 cloudsmith cloudsmith 192 Apr 30 16:09 my-crate-0.1.0\n-rw-r--r-- 1 cloudsmith cloudsmith 673 Apr 30 16:09 my-crate-0.1.0.crate\n", "filename": null, "language": "text", "markDefs": null } ``` Once built you can push the crate to Cloudsmith using the [Cloudsmith CLI](https://pypi.org/project/cloudsmith-cli/). We don't yet support pushing crates with `cargo publish`, but we'll be adding support in a future release (if you want it, please let us know). For now, use the CLI. First, ensure you've installed the Cloudsmith CLI and configured authentication: ```json { "_key": "2e8c02b5f83a", "_type": "code", "code": "$ pip install cloudsmith-cli\n$ export CLOUDSMITH_API_KEY=xxxxx\n", "filename": null, "language": "text", "markDefs": null } ``` Then, use the CLI to push the crate, which operates like `cargo publish` but is specific to Cloudsmith: ```json { "_key": "8f7b906be9fb", "_type": "code", "code": "$ cloudsmith push cargo my-org/my-repo target/package/my-crate-0.1.0.crate\n\nChecking cargo package upload parameters ... OK\nChecking my-crate-0.1.0.crate file upload parameters ... OK\nRequesting file upload for my-crate-0.1.0.crate ... OK\nCreating a new cargo package ... OK\nCreated: my-org/my-repo/my-crate-010crate (ffb4n20QxSYM)\n\nSynchronising my-crate-010crate: [####################################] 100% Sync Completed / Fully Synchronised\nPackage synchronised successfully!\n", "filename": null, "language": "text", "markDefs": null } ``` Your repository should now contain the uploaded crates. An example of which is our official Cloudsmith examples repository (this is where we upload [our examples](https://github.com/cloudsmith-io/cloudsmith-examples) from). Once pushed, your crate is ready for installation with Cargo. ### Installing your Cargo crates from Cloudsmith First, we need to tell Cargo how to access the registry. To do so you need to add the following entry in your `.cargo/config`file (replace `TOKEN` with an entitlement token, which is a type of read-only authentication token that we support): ```json { "_key": "e18a57c17e66", "_type": "code", "code": "[registries]\ncloudsmith = { index = \"https://dl.cloudsmith.io/TOKEN/my-org/my-repo/cargo/index.git\" }\n", "filename": null, "language": "text", "markDefs": null } ``` And then install the crate: ```json { "_key": "8429bab1e7bf", "_type": "code", "code": "$ cargo install my-crate --registry cloudsmith\n Updating `https://dl.cloudsmith.io/TOKEN/my-org/my-repo/cargo/index.git` index\n Downloaded my-crate v0.1.0 (registry `https://dl.cloudsmith.io/TOKEN/my-org/my-repo/cargo/index.git`)\n Downloaded 1 crates (673 B) in 0.54s\n Installing my-crate v0.1.0 (registry `https://dl.cloudsmith.io/TOKEN/my-org/my-repo/cargo/index.git`)\n Compiling my-crate v0.1.0 (registry `https://dl.cloudsmith.io/TOKEN/my-org/my-repo/cargo/index.git`)\n Finished release [optimized] target(s) in 2.60s\n Installing /Users/cloudsmith/.cargo/bin/my-crate\n", "filename": null, "language": "text", "markDefs": null } ``` Assuming `~/.cargo/bin/my-crate` is on your `$PATH` then you should be able to run the installed program: ```json { "_key": "954a86242713", "_type": "code", "code": "$ my-crate\nHello, world!\n", "filename": null, "language": "text", "markDefs": null } ``` It really is that simple, and now you've sourced a crate privately from your very own private Cargo registry on Cloudsmith. Well done! ## Conclusion Rust is an exciting new programming language, with many interesting properties that make it perfect for use in a wide range of scenarios. Cargo provides a robust suite of tools for installing, configuring and managing dependencies for your Rust projects. Cloudsmith provides fully featured Cargo registries on all plans, flexible enough for use whether you’re hosting public crates for a public or open source project, or private crates for your company’s internal needs. We're extremely proud to be able to support the Rust ecosystem with this first (non-official) implementation of public and private Cargo registries. You can find further, context-specific information, including detailed setup and integration instructions inside each Cloudsmith repository. You can see an example of this documentation in our [public examples repository](https://cloudsmith.io/~cloudsmith/repos/examples/setup/#formats-cargo). Why wait? Get your public and private Cargo registry hosting at Cloudsmith now. Cloudsmith provides public and [private Helm repository](/product/formats/helm-chart-repository) hosting, with ultra-fast and secure delivery of your Helm charts; for the ultimate in integration with the Kubernetes (k8s) ecosystem. In this article, we first introduce: - What Helm is - What Kubernetes is - What Helm charts are - How to host your own Helm repository using Cloudsmith Let's dive in! ## **What is Helm?** Originally built by Deis (later acquired by Microsoft), Helm is a package manager for Kubernetes that allows development and operations teams to easily manage and deploy these increasingly complex cloud native applications to their Kubernetes clusters. Helm allows you to manage applications on your Kubernetes cluster in much the same way as you’d manage applications on your Linux server with apt or yum. ## What is Kubernetes? [Kubernetes](https://kubernetes.io/) (otherwise known as K8s) is an open source system to deploy, scale, and manage containerized applications. Kubernetes has quickly become one of the most popular container orchestration platforms used by development and operations teams all over the world. It was open-sourced by Google in 2014, built based on years of learning deploying containers at scale inside the search giant. Despite Kubernetes’ rapid rise to popularity, deploying and managing applications can be a complex and daunting task, often requiring the operator to write and configure multiple inter-dependent configuration files for various parts of the systems being deployed. As our applications become more complex, more distributed, the difficulty of managing them grows even further. ## **What is a Helm Chart?** A Helm chart is a collection of Kubernetes YAML template files organized into one single directory. Helm packages are also known as “Helm Charts” and are stored in a “Chart Repository”. By default, Helm comes bundled with the “stable” chart repository, hosted for free by Google. Most public charts are hosted here, mostly provided by vendors packaging their own software for use by others. ## **What is Helm Used For?** Through Helm charts, Helm is used to manage Kubernetes applications. It works by packaging up a set of YAML definitions along with the necessary configuration to quickly stand up all components of an application in a repeatable way. A single chart can be as simple or complex as necessary, deploying anything from a single container to a full distributed application. Helm combines these application definitions with user-provided configuration to allow simple overriding of configuration where needed, allowing users to concentrate on shipping software and not on the nitty-gritty of configuring every application they need to run. ### **See also:** - [Helm Documentation](https://helm.sh/docs) - [Kubernetes Documentation](https://kubernetes.io/docs/) ## Getting Started With Helm The main requirement for using Helm is a working Kubernetes installation. Setup of Kubernetes is outside the scope of this post, but you can get started locally with the installation of [minikube](https://kubernetes.io/docs/setup/minikube/). Once [installed](https://helm.sh/docs/using_helm/#installing-helm), you can initialise Helm on your cluster: ```json { "_key": "663c10f62c8a", "_type": "code", "code": "$ helm init\n", "filename": null, "language": "text", "markDefs": null } ``` This will install [Tiller](https://helm.sh/docs/glossary/#tiller), Helm’s server-side component, responsible for interacting with the Kubernetes API and managing state. You can then run other Helm commands as needed. To view all available charts you can use the following command: ```json { "_key": "907bcfba2ef0", "_type": "code", "code": "$ helm search\nNAME CHART VERSION APP VERSION DESCRIPTION\nstable/acs-engine-autoscaler 2.2.2 2.1.1 DEPRECATED Scales worker nodes within agent pools\nstable/aerospike 0.2.3 v4.5.0.5 A Helm chart for Aerospike in Kubernetes\nstable/airflow 2.4.0 1.10.0 Airflow is a platform to programmatically author, schedul...\nstable/ambassador 2.0.1 0.52.0 A Helm chart for Datawire Ambassador\nstable/anchore-engine 0.12.1 0.3.3 Anchore container analysis and policy evaluation engine s...\nstable/apm-server 1.0.0 6.6.2 The server receives data from the Elastic APM agents and ...\nstable/ark 4.1.0 0.10.1 A Helm chart for ark\n...\n", "filename": null, "language": "text", "markDefs": null } ``` You should see a full listing of the built-in “stable” repository which contains all supported public charts. You can install any of the charts from the `helm` CLI also: ```json { "_key": "9d54cfdf3d2e", "_type": "code", "code": "$ helm install stable/dokuwiki\n", "filename": null, "language": "text", "markDefs": null } ``` This instructs Tiller to pull the packaged chart from the stable repository, render all the templates contained within, and apply those to the running cluster. You can inspect the status of the application with regular Kubernetes CLI tooling: ```json { "_key": "e1beea23d919", "_type": "code", "code": "$ kubectl get pods\nNAME READY STATUS RESTARTS AGE\nlopsided-cheetah-dokuwiki-85c96f777d-gskjn 1/1 Running 0 13d\n\n$ kubectl get deployments\nNAME DESIRED CURRENT UP-TO-DATE AVAILABLE AGE\nlopsided-cheetah-dokuwiki 1 1 1 1 13d\n", "filename": null, "language": "text", "markDefs": null } ``` Or check the status of the chart installation itself using the `helm` CLI: ```json { "_key": "689b8888f5c7", "_type": "code", "code": "$ helm list\nNAME REVISION UPDATED STATUS CHART APP VERSION NAMESPACE\nlopsided-cheetah 1 Mon Mar 25 17:16:56 2019 DEPLOYED dokuwiki-4.2.0 0.20180422.201901061035 default", "filename": null, "language": "text", "markDefs": null } ``` ## Managing Helm Charts With Cloudsmith If using only public Helm charts to install third-party applications, then the above instructions are enough to get started in most cases. But if you want to install software for which there are no public Helm charts, whether your own or third-party, you can create your own chart and host it using Cloudsmith. If you’re creating your own chart, you can use one of Helm’s starter templates to quickly scaffold out the directory for you: ```json { "_key": "8df9d6c43da3", "_type": "code", "code": "$ helm create my-chart-name\n", "filename": null, "language": "text", "markDefs": null } ``` This will create a `my-chart-name` folder containing all the files and folders necessary to build your chart. You can adjust the templates as needed for your application, [The Chart Template Developer’s Guide](https://helm.sh/docs/chart_template_guide/) is a really useful resource when building your own chart. Once you’ve finished adjusting your templates, you can create a Chart package with the CLI: ```json { "_key": "549102ab0d63", "_type": "code", "code": "$ helm package my-chart-name\nSuccessfully packaged chart and saved it to: /home/me/my-chart-name-1.0.tgz\n", "filename": null, "language": "text", "markDefs": null } ``` Once built you can push the chart to Cloudsmith. First, ensure you’ve installed the Cloudsmith CLI and configured authentication: ```json { "_key": "2ee3b7446461", "_type": "code", "code": "$ pip install cloudsmith-cli\n$ export CLOUDSMITH_API_KEY=xxxxx\n", "filename": null, "language": "text", "markDefs": null } ``` Then, use the Cloudsmith CLI to push the package: ```json { "_key": "e8c4c56a788e", "_type": "code", "code": "$ cloudsmith push helm my-org/my-repo my-chart-name-1.0.tgz\nChecking helm package upload parameters ... OK\nChecking my-chart-name-1.0.tgz file upload parameters ... OK\nRequesting file upload for my-chart-name-1.0.tgz ... OK\nCreating a new helm package ... OK\nCreated: my-org/my-repo/my-chart-name-10tgz (...)\n\n\nSynchronising my-chart-name-10tgz: [####################################] 100% Sync Completed / Fully Synchronised\nPackage synchronised successfully!\n", "filename": null, "language": "text", "markDefs": null } ``` Once pushed, you can configure Helm to install packages from your Cloudsmith repository: ```json { "_key": "a05e590f1f42", "_type": "code", "code": "$ helm repo add cloudsmith 'https://dl.cloudsmith.io/TOKEN/my-org/my-repo/helm/charts/'\n$ helm repo update", "filename": null, "language": "text", "markDefs": null } ``` And then install your application to Kubernetes: ```json { "_key": "51616250b65e", "_type": "code", "code": "$ helm install cloudsmith/my-chart-name", "filename": null, "language": "text", "markDefs": null } ``` ```json { "_key": "0d5494cb133a", "_type": "image", "alt": "Example of a Helm repository on the Cloudsmith platform", "asset": { "_createdAt": "2025-06-05T07:57:53Z", "_id": "image-f5c0fe00720052257e51657298b4fd92dbfe6090-1178x914-png", "_rev": "2MVa2LY6RC9Yy6hPJdh4DX", "_type": "sanity.imageAsset", "_updatedAt": "2025-06-05T07:57:53Z", "assetId": "f5c0fe00720052257e51657298b4fd92dbfe6090", "extension": "png", "metadata": { "_type": "sanity.imageMetadata", "blurHash": "V2R3i]0100Y50e00,w~X~W^l004-Iq9sM|00?ZtIx?In", "dimensions": { "_type": "sanity.imageDimensions", "aspectRatio": 1.288840262582057, "height": 914, "width": 1178 }, "hasAlpha": true, "isOpaque": true, "lqip": "data:image/png;base64,iVBORw0KGgoAAAANSUhEUgAAABQAAAAQCAYAAAAWGF8bAAAACXBIWXMAAA7DAAAOwwHHb6hkAAACsUlEQVR4nKWSy4sdRRSH+5+J2SSgO8H/ShcyiIjGYJQhC3EvuHCrZNAkI0hI4iLm5kWY153cx0xu39dUd1fXq9/3k6p7mYSYgOCBj9N0nfqdV0Uf/fAMz4ffP+XS9iPeu9Z7jYdcuNbjwjc9Ln7b49L2Yz64/uSc9zdc3u5x+bu/g4++uDng6m3PkM9/O2Zrp8/WzhFbN/p8euOIj3855JNf+3z2+5Avd8dc/ePknK83fLU74sqtAVd2R0TPXwoOY0F/mnEYpxzEyYaUg2nG3iRlbyo5XhqGScH4LYwSx0g4xokjUtagjMeS5gYhFbk2WFdQlCWlp6qpmpam7Wi7t9O0npZoNJe8mGUMZjmDWcZ4npLmGve6YFlRVmsqL+6p63Wiav3tvY+P7u1P+fN5zP39GXsngniZkcocrTVam+CNNhjzJhZtHWaT2IvWdU3UjwVHLwWjWcYi0+TKkEmJSDNEKkn9CHIVRKy1qJBIo40NY3LG4KzGlZayroik1mGGobW6xjpHmmXEZ2dMRYaQGqUNzrkQ44W9oLIuJLdigVqOWGQxqZNEUvlshqoqqZsmCCZZwmS5JBY5iXJo4yiKgqqqQpU+XtkCKTVmfko22eP0bMjCCCK/UT8PH+wF/cUsz5gnCUtpkKbczKkIFTpnQ7wXzJXFijn5YkCcTJiqlEjIhFSl6EJRNgV147dXYAuHCxtswwa9mE/mOyh8+5sN14WjMBKRS05FTvQwfsqD+DGP5s84VTFlU9KtOlarVcBb27bhyVhXYjetd92rmLZtcWXNmSqIfj76ix8P7vDT4V0ezI9RtVsH8srWl7rwuOumDWJvWtetqNuO6Pb4hJ0XQ26OTthPUmzTsCnsX+Z/v+Po/DyaJJLxImGS5Kiion2X2n+0yDr/WBXGGhpf3f+Sg38AG4WmKixvgSMAAAAASUVORK5CYII=", "palette": { "_type": "sanity.imagePalette", "darkMuted": { "_type": "sanity.imagePaletteSwatch", "background": "#507985", "foreground": "#fff", "population": 0.26, "title": "#fff" }, "darkVibrant": { "_type": "sanity.imagePaletteSwatch", "background": "#1c4474", "foreground": "#fff", "population": 0, "title": "#fff" }, "dominant": { "_type": "sanity.imagePaletteSwatch", "background": "#1b82c9", "foreground": "#fff", "population": 6.95, "title": "#fff" }, "lightMuted": { "_type": "sanity.imagePaletteSwatch", "background": "#d4c3a5", "foreground": "#000", "population": 0.62, "title": "#fff" }, "lightVibrant": { "_type": "sanity.imagePaletteSwatch", "background": "#9bebe7", "foreground": "#000", "population": 0.13, "title": "#000" }, "muted": { "_type": "sanity.imagePaletteSwatch", "background": "#9d5f5e", "foreground": "#fff", "population": 0.06, "title": "#fff" }, "vibrant": { "_type": "sanity.imagePaletteSwatch", "background": "#1b82c9", "foreground": "#fff", "population": 6.95, "title": "#fff" } } }, "mimeType": "image/png", "originalFilename": "helm_screenshot.PNG", "path": "images/rafvlnhi/production/f5c0fe00720052257e51657298b4fd92dbfe6090-1178x914.png", "sha1hash": "f5c0fe00720052257e51657298b4fd92dbfe6090", "size": 79533, "uploadId": "TawxBv1pOsc7nPqYlBEzl34F51Nukk4t", "url": "https://cdn.sanity.io/images/rafvlnhi/production/f5c0fe00720052257e51657298b4fd92dbfe6090-1178x914.png" }, "caption": "After pushing Helm charts to Cloudsmith, you'll see something like this in your repository.", "link": null, "markDefs": null } ``` There you have it! We have also produced a handy video guide on getting started with Helm using Cloudsmith, which you can follow below: ```json { "_key": "2ff61bc8bd7f", "_type": "video", "id": "2yFGdeaYU9Y", "markDefs": null, "thumbnail": { "_type": "image", "asset": { "_ref": "image-015d78904e87f9d6a2a0044c9ce1e21b7cfc3200-480x360-jpg", "_type": "reference" } }, "title": "Getting Started with Helm and Cloudsmith", "url": "https://www.youtube.com/embed/2yFGdeaYU9Y?feature=oembed" } ``` A short video demo of how to get started with Helm and Cloudsmith, including pushing a Helm Chart to private Cloudsmith repository and then pulling and installing the Chart on a Kubernetes Cluster. ## Conclusion Helm is a powerful tool to help manage the ever-growing complexity of deploying and running cloud native applications on Kubernetes. Like other package managers Helm provides a robust suite of tools for installing, configuring and managing applications on its target platform, Kubernetes. Cloudsmith provides fully featured Helm repositories on all plans, flexible enough for use whether you’re hosting public charts for an open source project, or private charts for your company’s internal applications. You can find further, context-specific information, including detailed Helm setup and integration instructions inside each Cloudsmith repository. You can also see an example of [Helm documentation](https://help.cloudsmith.io/docs/helm-chart-repository) in our public examples repository. Why wait? Get your public and [private Helm repository](/product/formats/helm-chart-repository) hosting at Cloudsmith now. Can you believe it's summer-time already? It's hard for us to tell from the cave that we're coding in day and night, but the office thermostat indicates that it is probably sunny way up there, above the ground. It does mark a fantastic occasion to bring you news of some juicy pieces of new functionality to help get you automated, now with even more secure delivery of your software. We're extremely pleased to announce one of our features that we believe brings us full circle in terms of achieving data portability in and out of the service: Webhooks. This isn't a fancy term for a new Spider-Man gadget. Where the RESTful API (and CLI) allow you to easily get data into the system, the webhooks allows us to push data about events to you. Now you can do clever things like react to a package being promoted to a production repository, in order to drive your pipelines to start a deployment. So yes, the circle is now complete for data portability. Which is great, of course, but we know what you're thinking: Hurry up and get NPM / Docker / NuGet / Alpine / Bower completed (delete as appropriate), and you'd be right. We're working hard on it indeed, in between working on raising funds to speed things up, and we'll hopefully have even more exciting announcements next time. But enough about that for now: **To the product updates!** ## Webhooks _TL;DR: You can now have Cloudsmith inform you, your cats and your servers about events that happen, such as new packages being synchronised. Drive your pipelines or ChatOps with it!_ **Where:** The "Webhooks" sub-menu of each repository. Webhooks - What are they good for? Either for driving automation by pushing data to your pipelines, or by integrating with your chat tools to provide slick ChatOps. Our webhooks support events that occur in a repository, such as when packages have been uploaded, are synchronising, have synchronised or have failed. We plan to add more events in the future, and to have them creatable at the the namespace (user/org) level, or site-wide on Enterprise edition. For pipeline automation, you might utilise webhooks by setting up one to tell a CI/CD tool such as Jenkins, or perhaps Spinnaker (if you're being fancy) that it is time to deploy when a synchronised package is moved to your production repository. In this way, you can control the flow from development to production by limiting who (or what) has the authority to move packages to the production repository, and thus, to deploy publicly. For ChatOps, you can utilise webhooks to send a message to a chat tool such as Slack when each type of event occurs. You'll format this in such a way so as to present critical information to your team, such as what the package is, where it is located, who uploaded it, and how to access it, etc. If you're really fancy, you'll have a Slack integration that lets users interact with Cloudsmith via slash commands, for super slick bi-directional DevOps goodness. Creating a webhook is simple and only requires an endpoint to send it to. If you're looking to create a Slack integration, we highly recommend using the **Handlebars Template** payload format. [Handlebars](https://handlebarsjs.com/) is a mini templating language that you'll use to craft the output of the webhook. More help is available on-site, but if you're stuck, just yell. In the near-future we'll offer more concrete documentation, and beyond that we'll add more tightly integrated plugins for things like Slack that do the hard work for you. ## Composer Package Repository _TL;DR: We've added support for Composer packages (in the form of .phar, .tgz, .zip, etc.), so that you can have your PHP packaging with all the usual Cloudsmith goodness._ **Where:** The "Upload Packages" sub-menu of each repository. As defined by Wikipedia: "Composer is an application-level package manager for the PHP programming language that provides a standard format for managing dependencies of PHP software and required libraries". Cloudsmith now provides PHP as a first-class package repository type with support for most of the metadata that a typical composer.json file exports. Get started by uploading a Composer-compatible .phar package built with phar-composer or similar. If you're using the UI to upload packages then Composer support is immediately available (as per image above). If you're uploading via the CLI you'll need to download and install the latest (0.6.1) version of the cloudsmith-cli application. Assuming you've followed the usual instructions for installation, you can do this with a simple invocation of pip to update your CLI: pip install --upgrade cloudsmith-cli ## Social Authentication _TL;DR: You can now signup and login using third party social authentication credentials from providers such as Amazon, GitHub, GitLab, Google and Microsoft._ Link: [https://cloudsmith.io/user/login/](https://cloudsmith.io/user/login/) This one is probably the most obvious because it's hard to miss, but we've added the capability to login via social providers (as you can see below) such as Amazon, GitHub, GitLab, Google and Microsoft. In the future we'll add additional support for more complicated and Enterprise-level authentication integrations, such as being able to use a third party AD service for orgs. For now though, please enjoy the convenience of jumping services. Since you've already got an account, you'll need to login first and then manage your social accounts in your user settings. You will then be able to connect each of the supported social providers to your account so that you can login with them next time. In the future we'll offer slicker association from login by asking you to re-enter your password if you try to login with a social account but you already have a (previously unconnected) Cloudsmith account. ## Two-Factor Authentication _TL;DR: You can now use two-factor authentication to protect your logins, your Organisations, and your packages with an additional level of security._ Link: [https://cloudsmith.io/user/settings/security/](https://cloudsmith.io/user/settings/security/) Two times the security. Twice as secure. Right? Well, that only matters if the base level of security is strong to begin with. At Cloudsmith security is one of our most paramount concerns, and we utilise our collective years across different disciplines such as financial technology and Internet startups to apply this to package management. You can see this in the architectural DNA of the service, such as how we process packages away from the front-end, through to utilisation of front-end security techniques, such as the use of Content Security Policy (CSP), HTTP Strict Transport Security (HSTS), etc. We now have support for two-factor authentication via a TOTP (Time-based One-time Password Algorithm) device, such as Google Authenticator, LastPass Authenticator, etc. Once you've completed enrolment (i.e. registration of your device with us), you will be challenged to authenticate via the device after social or password-based login. You do this by entering in a 6-digit pin that your device presents. If you forget your 6-digit pin, we also offer a recovery service using disposable tokens. If you're the owner of an Organisation (as defined by Cloudsmith) you can also force enforce enrolment of two-factor for everyone in the Org. A flag that denotes 2fa within the members list will tell you if the member has two-factor enabled or not. If you enforce enrolment and a User hasn't yet enroled, they will not be able to access any of the pages for the Organisation (e.g. they can't view or manipulate packages). If you are security conscious, please consider enabling this. ## Emails Management _TL;DR: You can now add and manage multiple email addresses per User account, in addition to managing your subscriptions to the kind of emails we love to send you to keep you in the loop._ **Link:** [https://cloudsmith.io/user/settings/emails/](https://cloudsmith.io/user/settings/emails/) It's often the case that one ends up juggling multiple email accounts and using them at the same time to access different services. This is often the case if you belong to multiple organisations and each provides you with an email address that is specific to it. To help manage and facilitate this, you can now add multiple email addresses to your Cloudsmith User Account. Once an email is verified you'll be able to login with it. You can also specify your primary email address, which is the one that we'll email you with when contacting you. In addition to that, we're proud to say that we're fully compliant with the latest privacy laws in the EU (the GDPR, which you've likely heard about more often than you'd like to). As such, you can now manage your email communication preferences and choose whether you want to opt-in or opt-out for emails. You're reading the newsletter right now, so if you opt-out of that we'll not send you it anymore; but then you'll not know about new features like this. :-) ## Vendor-Friendly Entitlements _TL;DR: You can now limit downloads to specific packages or download paths, by maximum number of downloads or clients, or by start or expiry dates. Oh, and support for user-based metadata._ **Where:** The "Entitlements" sub-menu of private repositories. We have built Cloudsmith around servicing a trifecta of business use-cases, sometimes called the "3 'D's of Package Management": Dependencies for development, Deployment for operations, and Distribution for Vendors. The third use-case, distribution for Vendors, is all about empowering Users to securely distribute their software to the world at high-speed. It doesn't matter whether the software is free, open-source, or proprietary and paid. If you're selling software and distributing it via Cloudsmith, you'll likely have a license that is associated per Customer and which dictates their terms of usage. Associating the license with an entitlement allows you to control and track downloads of the software specifically for that license. For example, you could only allow the Customer to download specific packages, between August 1st 2018 and August 1st 2019, up to a maximum of 10 downloads from one location (i.e. a single client). Each entitlement can have different restrictions, and you can use the freeform (JSON-based) metadata to add your own information into the entitlement. Support is already available via the UI and via the API, but we'll be adding full support for the enhanced functionality to the next release of the CLI. ## Full Boolean Search _TL;DR: You can now search for packages using full boolean queries (with AND/OR/NOT support), along with additional attributes such as distribution, format, status, etc._ **Where:** The search bar on top of the packages list in repositories. You may or may not recall that in a previous newsletter we said that we'd be gradually rolling out more FOPS support across the site - In case you've forgotten or have never heard of it before, FOPS is our endearing term for **F**iltering, **O**rdering, **P**agination and **S**earch support across the website. This is something that we'd like to make universal across the UI, the API and the CLI, and we're gradually rolling out support where we can. We're still not pleased with our current FOPS level, so we're continuing to add this in-between other feature releases. This month's flavour is the addition of Full Boolean Search to the packages search functionality. Previously we only supported an implicit AND query (i.e. all terms must be present). Now you can take advantage of the full boolean query parser to build complex queries with any combination of AND, OR and NOT. You can use parentheses to group terms if required. As part of this functionality, we have also added support for other search terms, such as distributions (e.g. 'el' for enterprise linux), format (e.g. 'deb' for debian packages), status (e.g. 'in_progress' for packages in progress). Help is available by hovering on the question mark. Support is immediately available via the UI, API and the CLI. You might be excited to know that in the near-future we'll be rewriting the CLI actions that target packages to use the search functionality instead of having to use things like auto-generated slugs (which we appreciate is painful and hard to predict). ## Other Minor Changes In addition to the above, the following changes have made it in: - You can now set email communication preferences. - Each repository page now has additional intro/help text. - The Entitlements list now reflects additional info for restrictions. - Downloads and clients are now tracked per entitlement token. - Access logs related to packages now link back to package. - Many fixes and enhancements across the user interface. ## Hit Me With The Full Changelog You can see the full log of changes for the latest releases at: v0.30: [https://help.cloudsmith.io/blog/v030x](https://help.cloudsmith.io/blog/v030x) v0.29: [https://help.cloudsmith.io/blog/v029x](https://help.cloudsmith.io/blog/v029x) v0.28: [https://help.cloudsmith.io/blog/v028x](https://help.cloudsmith.io/blog/v028x) v0.27: [https://help.cloudsmith.io/blog/v027x](https://help.cloudsmith.io/blog/v027x) ## What's Coming Up Next/Soon? We're full steam ahead on improving the service, and we're likely to have a bumper pack month or two coming up. Expect big changes as we on-board more developers. With that said, here are some little slices of awesome from the roadmap (no particular order): - NPM Packages: A fully-fledged NPM repository (soon). - Better Documentation: A manual on how to use this beast. - Upstream Proxying: Utilising us as a package proxy cache. - Package Redeployment: Overwrite (redeploy) your packages. - CLI Upgrades: Adding new entitlements/webhooks support. - Landing Redesign: Better information on what/why we do this. - Multiple Actions: UI for affecting multiple packages at once. - Docker Packages: A fully-fledged Docker repository. ## Ran Out Of Trial? If you've run out of your trial period, but want to try some of the latest premium features, no problem. Reply to let us know and we'd be delighted to offer you an extension. In fact, we offer this to any user who has missed their chance to trial the service properly, for whatever reason. Just ask! ## Company Update You want to see us succeed, right? Of course you do! To do that we have a humble favour to ask - As a startup who is currently in the throes of an investment round, we can do with all the help that we can get to make things more awesome. The more traction that we can build, the stronger and better the service that we'll be able to build for you. If you can help us by promoting us through word-of-mouth, by writing an article or featuring us, or simply by providing a qualified link back to us, we'd be eternally grateful. As you might have experienced, we do our best to support all of our users in any way that we can. If that means helping you write some code to automate or integrate, we're happy to do so. Some of these things would be hard to scale, but transparency and open/honest friendliness is something that we build into our company ideals, and is something we'd like to continue doing as we grow larger. Getting you automated in the easiest and most secure possible way is the reason that we built Cloudsmith. Let's do this together - If you help, please let us know. :-) ## That's A Wrap! Other than that, as always we hope that you enjoy the latest functionality, and if you've got any questions about any of the above, or need help from integration to use-case guidance, please don't hesitate to reply or contact us on the usual channels. So what are you waiting for? [Get started at Cloudsmith](https://app.cloudsmith.com) now. Remember: **Be Awesome** & **Automate Everything**. **TL;DR:** Cloudsmith released Phase II of our API+CLI which gives users greater control over package handling including search, entitlements and package actions. We're very excited by the additional possibilities of automation and integration that these introduce. If we're helping you automate, we're doing our job right. What is FOPS? Balancing the quest for automating everything with a sublime user-experience is never an easy thing, but we've found that if there's one thing that users love, it is FOPS - an affectionate term that we use for **Filtering**, **Ordering**, **Pagination** and **Search** functionality, and the releases this month are delivering it in spades. Note: To take advantage of the API/CLI features in this newsletter you'll need to upgrade your CLI to the latest version (0.5.5 or above), which you can easily do via pip: ```json { "_key": "e299c41a45b2", "_type": "code", "code": "> pip install --upgrade cloudsmith-cli\n", "filename": null, "language": "text", "markDefs": null } ``` Entitlements (API/CLI) **TL;DR:** _You can now list, create, update, delete, refresh and sync entitlements and their tokens via the API and the CLI._ A hugely requested feature is the ability to manage entitlements for a repository via the API and the CLI, and the following is now possible to do so: - List entitlements in a repository (tokens can be hidden/shown). - Create new entitlements for a repository. - Delete existing entitlements from a repository. - Refresh existing entitlements in a repository (update token). - Update existing entitlements in a repository. - Sync all entitlements from source to destination repository. This opens up new possibilities such as integrating automation of access controls or licensing into your own development or sales pipelines. For example, if you're selling software or libraries that you distribute via Cloudsmith, you could programmatically create an entitlement that you associate with a customer. Now the customer has direct access to your software using the entitlement. When the customer stops paying, remove the entitlement and they will no longer have access to new updates for it. Here's a quick example of what this looks like in the CLI: As you can see we deleted an old entitlement from the repository, then created a new one for this newsletter (you can provide a specific token or let the service generate one for you). Tokens are not displayed by default, but you can use the --show-tokens parameter to show them. To see the full list of entitlements commands, run the following: ```json { "_key": "0a9a12b8be50", "_type": "code", "code": "> cloudsmith ents\n", "filename": null, "language": "text", "markDefs": null } ``` Package Copy/Move Actions (API/CLI) **TL;DR:** _You can now copy, move (and resync) packages via the API and the CLI._ Another popular feature is the ability to copy or move packages between repositories, usually for the concept of promoting immutable versions of packages between separate build, test, staging and production pipelines - It means that the package that was verified in build is eventually the exact same one as in production when it gets promoted to there. It is now possible to automate copying and moving of packages using the CLI. Let's see an example of moving a package in action: As you can see, the sl package was moved (or "promoted") from the source repository to the dest repository. Copying packages works exactly the same, except it duplicates the package rather than moving it from the source to the destination. In addition to moving and copying packages, you can also resync packages via the CLI as well, which the CLI will now also utilise to retry packages that have failed (for whatever reason) upto a configurable amount of attempts. To see the full list of help for actions, run the following commands: ```json { "_key": "4c9371654a85", "_type": "code", "code": "> cloudsmith mv\n", "filename": null, "language": "text", "markDefs": null } ``` ```json { "_key": "3d6ee7a7f14f", "_type": "code", "code": "> cloudsmith cp\n", "filename": null, "language": "text", "markDefs": null } ``` ```json { "_key": "b7c669a3d517", "_type": "code", "code": "> cloudsmith resync\n", "filename": null, "language": "text", "markDefs": null } ``` Package Search (API/CLI) **TL;DR:** _You can narrow results when listing packages using the search queries against package attributes (such as filenames)._ A feature that we've been asked for time and time again is the ability search for specific packages in a repository. This is especially important as your repository grows in size and complexity beyond a few pages of results. Now you're able to query packages based on attributes such as name, version, filename, distribution, architecture, and more to come soon. Let's see an example of searching packages in action: As you can see the results have been narrowed from the full list down to packages with libxml2 in their filename, and then again to the specific 2.0.0 version. Who knew searching could be so exciting? It is when you realise what you can do with it, but let's see that in action in the next section. In the future we'll be expanding the CLI functionality so that you can target packages using attributes instead of the slightly opaque slug that we currently use, which we realise isn't that convenient for delightful users such as yourself. So for example, instead of: ```json { "_key": "895dd66862b7", "_type": "code", "code": "> cloudsmith mv lskillen/test/foo30abarrpm-2 test2\n", "filename": null, "language": "text", "markDefs": null } ``` Perhaps you'll be using something like (tenative) this instead: ```json { "_key": "376c6a0a4f9a", "_type": "code", "code": "> cloudsmith mv lskillen/test/name:foo,version:3.0 test2\n", "filename": null, "language": "text", "markDefs": null } ``` Watch this space! JSON Output (CLI) **TL;DR:** _You can now get machiner-readable JSON output for list-based commands (and more later)._ Having the API and the CLI opens up amazing possibilities for automation, because you can programmatically query and effect actions on the service. What isn't great though is having to parse arbitrary output formats (our CLI tables) designed by demented but visionary developers (that's us). As such, we've implemented JSON-based outputting for some of the critical CLI commands. Essentially any command that outputs a table can now be told to output JSON instead. This provides the full data that is received via the API, and means you can write more powerful integrations using that data rather than the limited subset that gets output when pretty printing tables. Let's see an example of that in action, and what you can do with it: As you can see there is far more information that is returned via the JSON output format than the standard pretty table-based output. If you're not a machine but you'd still like JSON output, you can also use the _prettyjson_ output format instead. One exciting use of this is the query for specific packages you need, and then use their attributes to perform actions, such as downloading the package as we've done in this example. More automation! Package Search/Filtering (UI) **TL;DR:** _You can now search and filter packages in the UI using the same syntax as the CLI._ We wouldn't want the delightful user-experience of being able to search be constrained to just the CLI, so in our quest for FOPS you can now also search and filter packages on the UI too. The search offers the same syntax as the CLI-based search, and filtering allows you to group packages based on type and status, as you can see in the screenshots below: Package Grouping (UI) **TL;DR:** _We now have automatic grouping of packages by name._ As with the necessity for search, being able to quickly find the packages that you need within larger and more complex repositories is difficult. As a package management provider we are (of course) primarily focused on packages, and most packages have a name and version. So to help with that management we've introduced a package groups view that automatically groups packages by their name together as a group of versions (if it has one, if not it'll be in the UNGROUPED "group"). Let's see an example of this in action: As you can see the grouping makes it much easier to find packages that are similar to each other. You're also offered a summary view of that group which includes the amount of packages there are (at a specific state), the name of the group, the top (highest) version of the packages, the last (uploaded) version of the packages, the number of downloads across group and the total size of group. Granular Repository Permissions **TL;DR:** _You can now control who can execute copy, delete, move and resync actions for packages at a repository-level._ Permissions (the ability to do something) and privileges (who has the ability get permission to do something) is a critical piece of managing your infrastructure and software. Being able to provide a fast-and-loose repository is great for development, but not so much when you need a locked down repository for production. So in addition to our standard privileges model, we now offer granular permissions to control who can copy, delete, move and resync packages in a repository. An example scenario of where this is useful is allowing uploads to a production repository, but no deletes (or moves). Let's see an example of the configuration: Note: We're working on the aesthetics to make this smaller and easier to read (and use) in an upcoming release. Hit Me With The Full Changelog You can see the full log of changes for the latest releases at: - v0.26: [https://help.cloudsmith.io/blog/v026x](https://help.cloudsmith.io/blog/v026x) - v0.25: [https://help.cloudsmith.io/blog/v0251](https://help.cloudsmith.io/blog/v0251) What's Coming Up Next/Soon? Little slices of awesome from the roadmap (no particular order): - Webhooks / Notifications: Power your pipelines! - Better Documentation: A manual on how to use this beast. - Upstream Proxying: Utilising us as a package proxy cache. - Package Redeployment: Overwrite (redeploy) your packages. - OAuth/Social-based Login: Login via GitHub/Google/etc. - Statistics/Logs API: Access your access logs, using code. - Docker Packages: A fully-fledged Docker repository. - NPM Packages: A fully-fledged NPM repository. If you feel particularly strongly about any of those, or if you've got suggestions of your own, don't forget to vote and submit ideas for the roadmap. You'll need a Trello account to add your thumbs up, but it's free and easy to signup. No excuses! That's A Wrap! As always, we hope that you enjoy the latest functionality, and if you've got any questions about any of the above, or about the service in general, please don't hesitate to reply or contact us on the usual channels. See you out there! **TL;DR:** We released Phase 1 of our API+CLI, marking a milestone in which we can support end-to-end packaging, from uploading artifacts, to inspecting/managing artifacts, to entitling, downloading and installing artifacts. Get it from pip _today_: `pip install cloudsmith-cli` and [get automating with Cloudsmith](https://app.cloudsmith.com). ## Warning! Mild tangent incoming: > "Now witness the power of this fully operational battle station ..." The quote above, as some of you will undoubtedly know, is from _Return of the Jedi_, the third film of the original Star Wars trilogy. Not those parodies that they made that someone claimed were prequels, _ugh_. It's a moment in which the rather evil Emperor Palpatine wants to show that the newly rebuilt Death Star is, despite appearances, very much ready to live up to its name. He then proceeded to have the Death Star blow up a gigantic cruiser ship just to prove the point. _What a guy_. If you can't visualise it, we've got a GIF as per usual to save the day, probably (_insert pew pew pew here_): ## What's this got to do with Cloudsmith? Well, today we _finally_ released Phase 1 of the Cloudsmith Command-Line Interface (_CLI_, for those in the know), powered by the Cloudsmith RESTful API. The reason for the Death Star analogy is that originally we had planned to only release Phase 1 of the API+CLI to those who had opted-in, but the work that we've put into it has resulted in something altogether more functional and dare we say, fully operational. Rather like the Death Star itself. I'll just side-step the fact that the rebels do (_spoiler alert_) actually still blow it up, and this variant of the Death Star seemed even _less_ effectual than the first one. Probably because it was still a Phase 1 Beta, **_cough_**. The same caveats apply to the API+CLI though - This is considered an early access release, and although we've conducted a swath of panel beating internally, there is still plenty to do to make it better. We'd love to hear your feedback, but please avoid targeting our exhaust ports before our shields are ready. :-) In addition to the API powering the CLI, we have designed the API so that it is fully compatible with [Swagger](https://swagger.io/) and [OpenAPI 2.0](https://github.com/OAI/OpenAPI-Specification/blob/master/versions/2.0.md), allowing us to generate bindings in more or less _any programming language_ in existence. If you're not familiar with the concept of [discoverable APIs](http://www.baeldung.com/restful-web-service-discoverability), it is a concept in which the API is self-describing. It enables users or machines to _discover_ what the inputs, processing and outputs of the API are prior to integration, enabling such delights as the capability to programmatically generate libraries for the API, such as with [Swagger Codegen](https://swagger.io/swagger-codegen/). It certainly pushes us towards our goal to Automate Everything. ## Enough blabbing, give me the deets! Understood, sir ... Sorry, _not sorry_, sir! We'll start with the CLI, since it'll be that which will be more interesting for those of you looking to integrate Cloudsmith into your CI/CD processes for automating artifact packaging, and getting your code into the right environment as quickly as possible. We went for Python as the language for this, since it's what we're most familiar with, it uses the generated Python API bindings (also available for anyone else), and is open-source (licensed as Apache 2.0). It enables all sorts of awesome, with more to come soon, including: - Uploading Debian, RedHat, Maven, Python, Ruby or Raw (Generic) packages, programmatic and completely automated, including waiting on synchronisation. - Token-based authentication, so bot/scripting friendly. - Operations such as retrieval and deletion for packages. - Supports argument-based, environment-based or file-based configuration management, for flexibility. For Phase 2/3 we'd like to build the following into the CLI: - Extended validation, to save time on error conditions with better pre-upload/pre-flight checks. - Additional output formats, such as JSON, to help facilitate integration into your own scripting. - Entitlement code management, to help with users who build billing or granular access controls around their deployments. - Package moving/transfers, for automating pipeline promotion (e.g. promote from staging to production). Here's a GIF snippet of the [CLI Demonstration Video](https://youtu.be/8nlstYU9J5M) below. The GIF only shows the first 10 seconds, but you can see that it's a fully-fledged CLI tool, with _colours_ and everything. :-) The full video demonstrates package uploading. We've also got the following resources to help you: - [CLI Documentation](https://help.cloudsmith.io/docs/cli), for additional help. - [CLI GitHub Repository](https://github.com/cloudsmith-io/cloudsmith-cli), to see how it is built. - [CLI Cloudsmith Repository](https://cloudsmith.io/~cloudsmith/repos/cli/packages/), for pre-releases of the CLI. I think the following sums it up: ### Up and running with the CLI in 60 seconds _You can skip this bit if you're not setting up the CLI._ You can install the CLI straight from the command-line using `pip` (because it's available on [PyPi](https://pypi.python.org/pypi/cloudsmith-cli)): ```json { "_key": "32d4a435ec7a", "_type": "code", "code": "pip install cloudsmith-cli\n", "filename": null, "language": "text", "markDefs": null } ``` To get setup you'll need to retrieve your API key/token from the Cloudsmith service and store it somewhere. You can do this directly via the CLI itself, using: ```json { "_key": "a6724b4bf9ae", "_type": "code", "code": "cloudsmith token\n", "filename": null, "language": "text", "markDefs": null } ``` You'll be prompted for your login (email address) and your password (twice), and then it'll print out your API key if it was successful. When you have it create a config file using [this example from GitHub](https://github.com/cloudsmith-io/cloudsmith-cli/blob/master/config/credentials.ini), but replace the API key with your own, and then place the config file in one of the following locations depending on your OS: - **Windows:** `%APP_DATA/cloudsmith/credentials.ini` - **Linux:** `$XDG_HOME/cloudsmith/credentials.ini` - `$XDG_HOME` defaults to `~/.config`. - **Mac OS X:** `~/Library/Application Support/cloudsmith` To check that it worked, run the following to print out your authentication details (just like in the animated GIF above, which displays mine, _no stealing_!): ```json { "_key": "691f21d3c55b", "_type": "code", "code": "cloudsmith whoami\n", "filename": null, "language": "text", "markDefs": null } ``` If this all works, you've just setup the CLI for working with Cloudsmith, so give yourself a pat on the back or three for being especially fantastic. The demonstration video and the CLI documentation show examples of how to push packages to us to get you on the right track to automating your development and deployment pipelines. To get an idea though, it'll be something as simple as: ```json { "_key": "71e045b7ce3e", "_type": "code", "code": "cloudsmith push deb your-org/your-repo/ubuntu/xenial your-awesome-package.deb\n", "filename": null, "language": "text", "markDefs": null } ``` ... And _you're done_! The CLI will now upload your files, create the package and wait for synchronisation. Automatically. May I just say what a fantastic package `your-awesome-package.deb` is? Kudos to you on your choice of package naming. Moving on! ### What about that sweet, sweet, API? Yes, _sweet_, it is! As previously mentioned we're very proud to have a RESTful API that is completely built with Swagger and OpenAPI 2.0 support. The benefits to you as a developer, integrator, or systems admin, or any other related role, is that you can: - Explore and utilise the API via the Swagger UI. - See how the API works in detail prior to integration. - Use it to build your own integrations directly. You can play and explore about it at the following places: - [API Swagger UI](https://api.cloudsmith.io/), for the discoverable bit we mentioned. - [API Reference Documentation](https://help.cloudsmith.io/v1.0/reference), which is fairly extensive. - [API GitHub Repository](https://github.com/cloudsmith-io/cloudsmith-api), to see our language bindings. **Note:** If you're logged in on the main website, you'll be logged in on the API Sandbox as well. This allows you to play about the the API as an authenticated user to see what it does, although please be aware that any operations do indeed affect your account. ### Did you say bacon, err, Maven ... and Jenkins? > "Did you say bacon? The maple kind" ... _Yup_. You heard right. Along with Phase 2/3 of the API+CLI we're also working on a [Jenkins](https://jenkins.io/) integration and a [Maven Wagon](http://maven.apache.org/components/wagon/index.html) integration, _at the same time_. We too like to live dangerously. The Jenkins integration will allow you to more smoothly integrate Cloudsmith into your CI/CD plugin by offering a Pipeline-compatible (i.e. [Jenkins 2.0](https://jenkins.io/2.0/)) interface into our service. We utilise Jenkins for own our automation and pipelines internally, and think this will be awesome. We've also got other CI/CD services such as Travis, Circle and AppVeyor on our roadmap too. Watch this space. The Maven Wagon integration will allow you to deploy artifacts from Maven builds transparently and automatically into Cloudsmith, which bypasses the need to write a wrapper using the CLI. Both are great, of course, but we know that Maven users are desperately looking to reduce complexity in _any_ way possible. If you're not sick of looking at XML, we can only bow before you in awe. ## That's A Wrap! You can see the [release notes for 0.20](https://help.cloudsmith.io/blog/v0200) in which we introduced Phase 1 of the API and CLI. Obviously the API/CLI took the predominent amount of work in the past bucket or two, but there are other goodies in there, such as adding in links to associated package versions, and we improved the generosity of the free/small tiers. Three cheers for better tiers! As usual keep an eye out on the official Cloudsmith Development Roadmap to see how we're getting on, what's coming up next, and what's coming up in the near future. If there's something particularly awesome that you'd like to see sooner, _please_ vote on it to help us prioritise it. > "Now witness the power of this fully operational ... Command-Line Interface." _Said no-one ever_, but the sentiment is there. It's out, and it's more functional than we originally scoped for. We'd also like to think of our API+CLI as being like the Death Star that defeated the rebels. It'll bring a whole new level to our service, and as we continue to build on it, it'll open up the realm of possibilities with what you can achieve with Cloudsmith. Remember, install it now with `pip install cloudsmith` and join us on the quest to **Automate Everything**. So what are you waiting for? [Get automating with Cloudsmith](https://app.cloudsmith.com)! Until next time, _stay awesome_! **TL;DR:** We rewrote our internal tooling to use Terraform instead of Chef Provisioning, with much pain and yet much gain in productivity, and we released 0.12 of Cloudsmith with major new features such as support for Open-Source repositories. There was much rejoicing! Stay A While And Listen In the past few months we've been primarily focused on rebuilding on our [Infrastructure-as-Code](https://en.wikipedia.org/wiki/Infrastructure_as_Code) (IaC) tool chain for infrastructure orchestration, which is one of those delightful tenets of DevOps that allows us to automate the provisioning of our (extensive) server farm. Indeed it states in our [Tao of Cloudsmith](/company/the-tao-of-cloudsmith) that we must strive to **Automate Everything** and we certainly try to. We haven't quite found a way to automate ourselves out of a job just yet, but it's in the pipeline. ;-) For those of you without IaC (if you have it, bear with me), imagine this: Your team are tasked with producing an exact replica of Production, either for replacement or for another environment such as Staging; the Production environment is a 20 strong server farm (in the Cloud) comprising of web servers, database servers, cache servers, asynchronous task worker servers, middleware/queue servers, and storage servers. These have been meticulously built at the platform-level manually or mostly manually (e.g. within AWS) and hook in with another half a dozen platform services (such as Simple Storage Service, EC2 Autoscaling, CloudWatch, Elastic File System, Identity/Access Management, Elastic Load Balancers, Simple Queue Service, etc.) Even with the assistance of Configuration Management (CM) tools like Chef (or Puppet or Ansible, etc.), and Package Management (PM) tools like Cloudsmith (or those other competitors that definitely don't exist), this still seems like an incredibly daunting process, right? Right. Even boiling the problem down to replacing **one** server can be tricky depending on just how manual the process was to construct it in the first place. This is where something like IaC pays in dividends beyond the initial spend on the cognitive load of learning and deploying it. ## Introducing Infrastructure-as-Code (IaC) By using specialised declarative language and describing the exact steps necessary to orchestrate your infrastructure, from the top of the platform level right down to machine configuration, you will be able to take an environment like Production and produce a replica such as Staging just by altering the configuration and telling the IaC tool to orchestrate everything for you. I'm skipping a few steps here, such as completing IaC beyond infrastructure orchestration with aforementioned CM/PM tools, but the bottom-line remains true; it is a **business redefining capability** that enables you to **reduce the need for expensive Humans** (those pesky engineers). So back to the task at hand; this is what our team was required to do recently, and although it took us 8-10 weeks to perform the switch-over to full IaC, we were able to produce an entire Staging stack (including all of those servers above) in 30 minutes. We then rolled out a major Production release 3 weeks ago using the new tooling, and it took two hours. Production inevitably had taken a little longer because we had to bring down the current Production first, including confirming backups, testing the move on Staging, and then orchestrating the new setup. Of course we had some teething problems in the new deployment, but any problems were promptly executed with a deft swoop of the IaC tool. Apply! Wait a second, backtrack there, I hear you say - Doesn't Cloudsmith sing and dance the praises of DevOps? Shouldn't Cloudsmith already have IaC and all the amazing things it brings? Don't you **Automate Everything**? Yes, yes and yeee... mostly! Here's where the blog turns into a discusion on lessons learnt, on "How Not To Do Infrastructure-as-Code". Although as you'll see it's a story of unfortunate decisions and backing the wrong horse; but it shows that technical decisions can have extremely long reaching consequences. How Not To Do Infrastructure-as-Code Back in late 2014 we first started to integrate Chef as our CM tool of choice, and although it was beyond painful (and still is at times), we reaped the reward of being able to build a server programmatically (notice how this thematically matches and complements infrastructure orchestration.) It was then in 2015 that infrastructure orchestration had **really** started to make a decent impact on the DevOps community, and we noticed. We noticed that we didn't have it, and therefore **needed it**. So in late 2015 we begun our journey that would end up resembling Dante's Inferno and the nine levels of Hell. Out of the tools available for infrastructure orchestration we narrowed it down to two, both of which were still cutting their teeth and were early days Alpha-level products, [Chef Provisioning](https://docs.chef.io/provisioning.html) and [Hashicorp Terraform](https://www.terraform.io/); but both were Open-Source and gaining traction. Having built our CM tooling on Chef, it wasn't a natural decision at the time to go with Chef Provisioning - Chef with the capability to configure infrastructure beyond just machines. What's not to like? We **love** Open-Source and contribute when we can, so we even contributed to the project to help it grow and to implement some AWS resources that we needed for our infrastructure, such as the [initial PR for CloudWatch Alarm](https://github.com/chef/chef-provisioning-aws/pull/462) support. However by the following year it was still severely lacking in many areas. It was missing critical resources that we need to build out parts of the infrastructure (e.g. RDS Parameter/Option Groups, SQS Queues, EC2 Autoscaling Notifications, S3 Buckets, IAM Policies, etc.) and we had acquired some serious amount of local hackage (the technical term) required to run it properly. This involved all sorts of duct-taping usuable for targeting different environments. It was a complete and utter mess, some of which was perhaps our fault. The result was an IaC tool chain that was flaky at best, didn't always work, wasn't convergent or fully idempotent, didn't like running in parallel, had no dry-run to plan changes, and had no easy way of storing state remotely (so collaborating with others was **painful**). This isn't to say that it was an awful product, but it was an awful experience for us and one that I regret immeasurably. We had gained automation in one way, and lost our souls worth of development-time in another. By 2016 we had started to see external proof of our of choice in the wrong tool, such as obvious rot that had set in (in which Chef Provisioning was not keeping up with the rate of change to match platforms such as AWS) and Noah Kantrowitz (a thought-leader in the Chef community known as @coderanger) stating that [Chef Provisioning is unlikely to be a good choice for new projects](https://coderanger.net/provisioning/), and that the project had more-or-less been descope. Turns out he was right about it not being a good choice! **Gulp**. Yep, technical leadership guilt-trip time! Queue masses of finger pointing and chants of "Who did it ... Who did it!" - The critical mass really hit though when earlier this year we needed to make critical fixes to Production and Chef Provisioning **completely failed us**. It had stopped working in strange and obscure ways with our newer releases of Chef, and the rot that had set in by not keeping up with AWS meant that there was more and more things that we couldn't deploy. For example, Application Load Balancer (ELB 2.0) was released by AWS, and this was not supported by Chef Provisioning at all, at the time. There was also no Staging environment to assist with testing Production changes because Chef Provisioning wasn't well suited to doing that either. We would have went to red alert, but that would have meant [changing the light bulb](https://www.youtube.com/watch?v=jZn1fhMdrbQ). ## Rewriting History As We Know It So the executive decision was taken to rip out Chef Provisioning and to replace it with what was now a much more mature product. One that has kept up to date with the rapid change of AWS. One that now has an extremely strong backing of the community behind it. The King is Dead, Long Live the King: [Hashicorp Terraform](https://www.terraform.io/). Yes, exactly the same tool we had evaluated 2 years ago and chose Chef over it. To be fair it was much less polished at the time and it was anyone's game then. So here we are now at the end of the IaC journey. With the assistance of Terraform describing most of our infrastructure, we finally got that Staging environment (and more) and the capability for high velocity releases to Production with support for the latest AWS components. We're still not completely there yet for that 100% automation, but we're on the path to it and things are looking much brighter now. The bottom-line is that the path to DevOps bliss can be dark and full of terrors (OK, I stole that one), and that unfortunate decisions can turn into long-term pain and regret. Acruing technical debt to fix bad decisions is only going to lead to more pain, and sometimes it's better to address an issue at its root cause. Although it resulted in a 8-10 week tangent for us, the result is that we can now focus on other more obvious customer-oriented goals, such as implementing additional package formats and features like Package Retention, Geo/IP Restrictions, and a better REST API (with those sweet, sweet, integrations that DevOps love). Cloudsmith 0.12 So that leads neatly on to our latest release, 0.12, which primarily contains: - **Open-Source Repository Support:** With a generous allowance of 10GB storage and 100GB (and more can be requested), for free! This is a bit different from our competitors (who don't exist, probably), in that you don't need to activate an Open-Source plan to use them. Every single plan offers the ability to create Open-Source repositories, and the features available to them match your actual tier. So if you enjoy the use of metrics and statistics, you can have them on your Open-Source tier too. To get an idea of what they are like, you can see a [live example of an Open-Source Repository](https://cloudsmith.io/~cloudsmith/repos/love-open-source/packages/) on the Cloudsmith account. - **Pricing Plan Calculator:** You can see it in action at the bottom of the plans/pricing page, but we now have support for a calculator to help you decide on which plan is the best based on your storage, bandwidth and feature requirements. We don't list other package formats apart from Raw since they're available on all tiers. - **Fixes All The Things (TM):** OK, not actually all the things, but we fixed many things between 0.10 (0.11 was our Terraform-powered release 3 weeks ago) and the current release. This involves lots of page speed improvements, optimisation for user on-boarding and workflow (with more to come) and security fixes. If you want to keep on top of our releases, we maintain a list of [release notes](https://cloudsmith.com/changelog) that detail changes. That's A Wrap That's all until next time (should you wish to hear about it), but I leave you with one final thought: It seemed obvious to us in hindsight, but until you have a fully IaC-capable infrastructure, you don't realise just how much time you waste on building, deploying, rebuilding, redeploying, fighting fires, remembering how to fight fires, etc. ## How Actually To Do Infrastructure-as-Code (IaC) If you're going to build a software-based service (at any scale), and you want to maintain a reasonable level of velocity (and sanity), do your own research and integrate the best tools/practices you can for IaC/DevOps: - Use an Infrastructure Orchestration tool to describe, version and manage your infrastructure programmatically, such as [Hashicorp Terraform](https://www.terraform.io/). - Use a Configuration Management (CM) tool to describe, version and manage your server configurations/contents programmatically, such as [Chef](https://chef.io), [Puppet](https://puppet.com/), [Ansible](https://www.ansible.com/), etc. - Use a Package Management (PM) tool to store, version and distribute your application dependencies and server software, such as [Cloudsmith](https://app.cloudsmith.com). - Use a Continuous Integation (CI) / Continuous deployment (CD) tool to automate these and bring your testing/deployment pipeline together, such as [Jenkins](https://jenkins.io/), [TravisCI](https://travis-ci.org/), [CircleCI](https://circleci.com/), [Chef Automate](https://www.chef.io/automate/), etc. - Follow the [Tao of Cloudsmith](/company/the-tao-of-cloudsmith) and **Automate Everything**. If you haven't given Cloudsmith a chance yet, please endevour to come and try it out. We don't bite and we're always happy to help you through the process of creating repositories and automating your Package Management roundtrip. We're also good for having a chat about DevOps in general, especially if we can learn something new from how you do it. Don't forget to check out the [live example of an Open-Source Repository](https://cloudsmith.io/~cloudsmith/repos/love-open-source/packages/). See you out there!