Simplify repository structure with connected repositories

With legacy artifact managers, every ecosystem that dev teams use requires a separate repo. What happens when teams try to keep their line of work isolated, but they rely on the same ecosystem and packages as other teams? Platform teams have to configure multiple instances of the same repo for each team. This increases the complexity of the organizational repo structure, and means more repos, URLs, configs, and security policies to manage, not to mention greater storage needs. Using Cloudsmith’s connected repositories you can configure multiple upstreams in a multi-format repo, cache those packages once, and deliver them to downstream repos, all with a single URL.

Connected Repositories

Maintaining upstreams for every individual repository comes with an overhead tax. The amount of effort teams have to put into maintaining upstreams can quickly become a burden depending on how they structure their repositories. Legacy artifact managers rely on language-specific repos so teams need to configure an upstream for each repo. Therefore, if you use npm, Python, Rust, and Maven, then each language has its own upstream URL, configuration, and security policy. Now, if you’re working on a small team and those are the only repositories you pull from, it's manageable. But at scale, going from four repos to 40+, things get much more complex.

While maintaining dozens or hundreds of repos is a very real burden, it's the toll all those endpoints take on your security at scale that's easy to overlook.. The need to configure security policy for each individual upstream simultaneously increases the attack surface and the number of gaps within that attack surface. Policy enforcement becomes inconsistent and hard to do across enterprise-scale upstream configurations. A single enforcement point decreases risk by removing many of the gaps within the attack surface.

Connected repositories are a native Cloudsmith feature that links repositories and resolves packages through a single URL. The images below illustrate two different architectural configurations of connected repositories.

Cloudsmith supports multi-format repositories, so users can configure a single repository with multiple upstreams, as seen in configuration B. The OSS Packages Repository has upstreams for Maven, Python, and NuGet, and this is where packages from those ecosystems get cached. The repository, which is where end users pull from, doesn’t perform any caching or duplication of packages in the OSS Packages repo; it inherits the upstreams from the OSS repo and pulls packages straight through. When configuring connected repositories, teams can set priorities for each upstream (see config B). If two upstreams have the same priority, Cloudsmith automatically breaks these ties based on the upstream creation timestamp.

Package resolution is also automatic for connected repositories. You can see in configuration B above that the Enterprise Packages Repository is PRIORITY 1, the Shared Packages Repository is PRIORITY 2, and the OSS Packages Repository is PRIORITY 3 for the main repository. Cloudsmith resolves packages inside Cloudsmith in priority order first, then resolves packages outside Cloudsmith in priority order. Going back to the diagram, any packages in the main repository get resolved first, followed by packages in the Enterprise Packages Repository, then packages in the Shared Packages Repository, and finally packages in OSS Packages Repository. Each of the upstreams in the OSS repo will have a priority status, so those will resolve according to their established priority.

Check out the Cloudsmith documentation for more information.

Repository architecture

Connected repositories can reduce upstream sprawl and provide teams with more flexibility in how they structure their repos within their artifact manager. The repo architecture that’s right for your team will depend on what factors you value. Typically the key tradeoffs come down to isolation, visibility, access controls, and operational overhead. (If you want a detailed breakdown of the architecture configurations below, our team can walk you through the options.)

• Super- or Mono-repo: This configuration is exactly what it sounds like. A single repo for every package. While this is structurally simple to set up and run, there is no isolation between lines of work or development environments. This approach works for small teams, but quickly becomes unwieldy for larger teams.
• Language/format-specific: This approach creates one repo for every language or format a team uses. This is a familiar repo structure used by legacy artifact managers. It offers a single source of truth for each development language and keeping a format-specific approach can make migrating to a new artifact manager easier. However, this structure lacks combined visibility across formats for usage, licenses, vulnerabilities, logs, etc.
• Line of work: This structure has a repo for each team, service, or product. The line of work approach allows teams to own their repos, which means they can set up granular access controls, giving them more accountability over the trust decisions in their development environment. As for tradeoffs, this structure has higher potential to house duplicate packages and to create sprawl, especially if the repos don’t support multiple formats. Teams need strict tagging discipline when using this approach. For example, poor tagging could lead to deleting packages needed for production
• Environment-based: This approach includes one repo for each different production environment, typically build, test, and production. Environment-scoped RBAC and retention rules help to avoid some of the challenges with the line of work approach. This structure also provides a really clean promotion pathway; teams can clearly tell which development stage a package in is based on the repo storing it. The tradeoff is that teams have less visibility into specific lines of work, and need to make sure they follow clear naming conventions to avoid package conflicts.
• Hybrid: This approach combines the line of work and environment-based structures. It consists of one repo for each line of work within each environment-based repo. While it combines the best aspects of those two approaches, it also introduces the potential for repo sprawl, depending on how many lines of work a company has. This structure also requires advanced package promotion logic to ensure the right packages get promoted to the right location. Sharing packages across lines of work can also be challenging, depending on how teams configure their repos.

Connected repositories give you flexibility. The flexibility to pick a repo structure that fits the needs of your organization and the way your teams want to work. You get the flexibility of optimizing your repo structure to accentuate the tradeoffs you want over the ones forced on you from inherited systems. Connected repositories can reduce the URL- and upstream-sprawl that resulted from choosing anything other than a super-repo.

Ready to optimize your repo architecture? Talk to our team to get started.