Thanks so much, Candice for the warm welcome. Hello, everyone. Good afternoon, good morning, good evening, wherever you are in the world. I'm Adil Aghari Solution Architect Manager over here at Cloudsmith and I'm joined by some lovely folks and welcome to our webinar entitled, So We Know We Have to Secure the Software Supply Chain, But Where Do We Start?

Now, this is very much a carry forward from our last webinar we did in partnership with the Linux Foundation, which was everything you wanted to know about securing the software supply chain, but didn't know where to ask. That is, that was from March 10th, and it's available, of course, on the Linux Foundation YouTube site, as well as the Cloudsmith YouTube, of course, as well.

So this is the next step, sort of, we know. We have to secure it now. Now, how do we actually go about the practical first steps to do so? So thank you so much for joining me. I'll walk through the agenda here real quick. We'll do speaker introductions up front and let everybody know who we are and what we do.

We'll talk a little bit about our organizations and what we do as well. Then we'll, we'll touch on a quick summary from last time just to carry forward into that. And again, you, you, it's not homework. It's not required reading. You don't have to go back to the old one. You can, if you want. We'll summarize some of those points here quickly and touch on some of some of some of what got us here and why we are actually tackling these hard issues for everyone.

And in addition to that, then we talk now about what's next, right? What progress has been made in the space since. Since our last discussion in March you know, the wonderful open source projects, six store co sign you know and get signed some of the stuff coming out of the six word project and, and some of the steps that have been taken there, some of the stuff we've taken in as an organization, and that we can recommend that you may want to take as an organization in, in, in, in this process, and then we'll also time permitting also touch on.

Where there are still gaps in this process. Now, of course, it's still early days in some ways that they say for social security and software supply chain, but we've been saying that for a year or so now already. So I think, I feel like taking some steps is good, but it's good to identify where there are still gaps and where there's room for growth as well.

And then if time permitting, of course, we'll have a Q and a at the end as well. So please feel free to drop questions in the Q& A section if you, if you want to again, you know we're, we're happy to take on anything that you, you guys think of that you folks in the community would like to ask in terms of securing the software supply chain.

So with that, without further ado, let's hop in. I'll talk about myself real quick first. So I'm a a sys admin and I'm passionate about automation. My name's Adil Leghari. I'm active in the PowerShell and DevOps automation communities for quite a few years now. I'm a speaker and author and blah, blah, blah, blah, blah.

I don't need to add to that introduction. So let's move on, move on to Luke. Luke over to you.

Oh, yeah. And thank you for having me. It's pretty good to be here. So evidently Luke Hines, I work at Red Hat in the CTO office. And lead a security team, and we're focused on what we term emerging technologies.

So most of our work, we're very lucky we get to work upstream and create interesting new projects. Some fail and some succeed. Had a lot of failures and, you know, the odd one that's got some traction. And outside of that one of the folks that started SIGSTOR, a project that's gathered a fair amount of momentum recently.

But I also do various other things. So I'm a Kubernetes security response team. So I manage the HackerOne bug bounty program that we have there. There's a group of us that manage vulnerabilities in Kubernetes. I'm also on the Technical Advisory Council of the OpenSSF. So I was one of the early OpenSSF folks.

That's the Open Source Security Foundation. It's part of the Linux Foundation. And various other things as well. Confidential computing. There's a confidential computing consortium that I'm involved with. I'm a board member there. And so I keep very busy in the, in the open source world. And I'm really lucky to be in the position that I get to work on open source most of my days.

So

it's a very unique space to be in to be able to work in open source for your day job, Luke. And I will say Red Hat is one of those organizations that supported this from the beginning. So we appreciate you. Yeah, very much. Yeah. Awesome. Over to you, Lee. Yes. Okay. It's

very hard, hard, hard to follow Lick but I'll try it.

So my name is Lee Skillen. I'm the co founder and CTO here at CloudSpot, which you'll learn a little bit more of later on. I'm incredibly passionate about security the open source ecosystem. And in fact, security has always been one of those things that I think has touched every point of my life.

As a developer maybe I'm lucky in that way. But anyway, it's, I think it's, it's it's influenced. I think the company that we're trying to build at Cloudsmith, and certainly whenever we started out, we started to make artifact management easy to use and hard to misuse and the hard to misuse aspect was to make it as simple as possible As simple as possible to configure basically pipelines and delivery without introducing security issues.

So that's

me. Awesome, Lee. Thank you. Over to you, Dan.

Thanks, Adil. So hi, everyone. Yes, I'm Dan McKinney. I'm a technical account manager at Cloudsmith. You can probably see by my Twitter handle, I was formerly developer relations at Cloudsmith. So I spend a lot of time talking to Cloudsmith users and customers, both open source users and commercial customers, and I hear.

They mask a lot about how do they secure their software supply chain. It's a topic that comes up increasingly and there's barely a conversation goes by with our users and customers where we don't discuss this. So I'm here to give a bit of an insight from the user and the customer side.

From the front lines, Dan, from

the front lines of deal.

Absolutely.

And I don't want Dan to sell himself short. So he wears a lot of hats around here at Cloudsmith. And one of them is our resident DJ as well.

So, yes, that's true. I also write our documentation. I should say I'm a formerly a technical writer. So I write the Cloudsmith documentation and tutorial videos as well.

Generally, I try to help our users.

Awesome. I'll just say that the customers love Dan so much, they wanted to keep him as close as possible. That's right.

That's why I became a technical account manager. That's correct. Thank you, Lee.

Many messages from our customers start with, hi, Dan. Yeah, they do. True story. Just awesome. Thanks so much everyone for joining me.

I'm lucky to have such folks to be working with. Okay. So quickly, let's just touch on about cloud spent a little bit about us and what we do. We'll, we'll spend a lot of your time here. We want to focus on a lot of the open source projects out in this space as well. So, so Dan, do you want to lead us off with a little bit?

Yes,

certainly. So I mean, Lee Lee sort of touched on this already. I'll ask again for more input in a moment. But but Cloudsmith is a it's a fully managed cloud native sort of package management as a service. So we offer universal. And repositories hosted repositories for for binaries and artifacts. We support 28 package formats and multi format repositories.

Keep them all in one place. We have global points of presence 410 plus and a big part of what we do is we integrate. With all the tooling that you already use. And we'll talk about this later, especially with the sort of emerging software supply chain tooling as well. So, you know, ease of use is a huge part of what we say, because if you make it, you know, easy to use by default, people are more likely to use it from the get go.

So ease of uses is a big thing that also extends through to the tooling and software supply chain as well. So that's what Cloudsmith is. Fully managed package management as a service. We're also big supporters of open source. Of course, we host and we provide free hosting for open source projects and we host some fairly big open source projects, things like RabbitMQ and Caddy, the web server as well among others.

So big supporters of open source projects and that's the kind of services that we offer. For them. So Lee, would you like to add in what I have failed to mention?

pRobably the only thing I would say is that Cloudsmith was designed for the get go to be a natural fit for the ecosystem. So basically we're not a platform.

It's definitely an ecosystem product integrates well, plays well with other products certainly for the cloud based ecosystem and the clue is in the name. In fact. A lot of people ask us if we have on premises solutions, and the answer is no, because then we would have to change the name. But, but yes end to end I think is what we call in terms of what we're trying to support.

So it's a, it's a tool for your tool set that helps you to help manage that entire pipeline of artifact production right through to production.

Awesome. Thank you and get an introduction on, on the Red Hat CTO Office and SIG Store from Luke. Yes, sure.

So to tie these two together red has. really been focused on supply chain security for quite a good number of years now, much before it was called supply chain security because of obviously being a Linux distributor, having a package system, which was very a high target for attacks.

Okay. With the likes of RHEL because RHEL runs, I think most of the, like the NASDAQ and various stock exchanges all run on RHEL now. And so still the big verticals. Military, public sector, telco. So, so we've, you know, we've been used to being a target for a while. And so we've always looked at build security.

And so I've been taking an interest in this. I've been working on some, another project, which was moved into the CNCF, a project called Keyline and was tasked with looking at how can we improve the provenance of what we're bringing into Red Hat, okay. Supply chain security. And is that something that could be leveraged for.

Red Hat customers as well, typically us being a Kubernetes shop. So I started to, to look at this, this area and I had this idea that we could really benefit from some sort of Oracle of truth. Okay. Some, uh, credible store of provenance and then thinking about how would that provenance be captured. And we could then leverage that internally for our own sort of, uh, improving the view of, of what we're ingesting upstream from open source projects.

So I started to work around, I think this was kind of. pretty much just as we were going into lockdown. So 2020 sort of March time and originally played around with blockchain. So I tried a few different blockchain platforms that are out there and then realized really that the whole aspect of a token, which is very Susceptible to prices going up and down.

It just wasn't the right platform. I mean, you could have like a private blockchain, but then it wasn't a truly decentralized platform as such. So I just couldn't really get a good fit with blockchain. So I heard about transparency logs and how they've been leveraged for certificate transparency and so forth.

So I started to work on a prototype there, and that was originally recall. Okay. And recall was really around having an immutable, observable, transparent source of what's happening in the supply chain. Okay. So we then had that other folks started to get involved. It's kind of a typical open source story from here.

I built something, kind of had it, wanted to take it in a certain direction, but start to share it with other people, get some, get some consensus from others as to is this useful? And then other people started to collaborate. So we had Dan Lawrence come on. He was at Google at the time. Bob Calloway, the other one in the original three as well.

He was he was working in Red Hat at the time. He's at Google now. It started to get involved. And about that time, I realized really this. Was going to be widely utilized. It had to be in a public domain. Okay. It had to be vendor neutral and it had to be like a for the public good. Okay. So the open source projects would start to, to leverage the technology because the more that would leverage the technology that.

The better fingerprint we would have, the wider scope, that picture we would have of what's happening in the open source community. So I won't go into the full history of SIGSTOR, I'll save it a little bit for later. There's another question that's pertinent to that. But that's really how these two connect, you know.

SIGSTOR was originally something that we... been of use to Red Hat, really. I'm not really seeing it much out of that context. And and then obviously we realized that this could be something that could be much bigger and much more widely used. So I managed to speak talk our CTO into signing the project over to the next foundation, and it became an LF project, and now it's an open SSF

project.

Awesome. Thanks so much for that background. Luke really helps to frame our discussion. So as we lead into this now I just want to touch on so quick summary from last time. And what we'll do now is I'm going to go ahead and take the slides off because I want this to very much be a fireside chat and open discussion.

So, so let's keep us up on the screen and, and we'll touch on a little bit of conversations from last time. Just recap, everyone, get everyone up to speed on, on what we discussed. Gusta in March, right? With Dan Lawrence, of course, from Google and Stigstor and FAME as we've talked about with with ChainGuard as well.

So we touched on last time, why, why soft securing the software supply chain was so important. Now, some of the, you know, vulnerabilities that came out over time, some of the hacks, the solar wind hack, dependency confusion attacks and you know, the executive order with the nation's cybersecurity so talking a lot about how software bill of materials and S bomb has has sort of risen in prominence in terms of a way to you know, have a first step to securing the software supply chain, having proving provenance and showing how you need to actually not only be able to show what's included inside of a container image, but also be able to prove that that is actually what's in there.

Ways to sign it and test it in that process to then we touched on, on, on why this is such a hard problem for organizations to solve. And we can even discuss that a little bit again now, but, but I think part of this is, you know, certain organizations in the private sector. Well, you know started moving away from open source software, which we believed was wrong move.

And a lot of us in the open source community do because, you know, 90 percent of the stuff we develop just to. Percentage that I'm pulling out of the air, but probably pretty good is, is you know, based on open source projects and really those, I mean, the argument that Dan Lawrence made last time was that, you know it's actually a lot more of a secure development practice to use open source projects because they have been vetted, they have been developed out in the open and, you know, the code isn't abstracted or obscured away in any way so that you have that you know, ability to, to, to, to see exactly what you're, what you're implementing.

Yeah. In addition to that, you know, obviously the disparate tooling, and I think we can all speak to this a little bit, but in the space of trying to secure software supply chain you know, trying to make this easy and accessible, simple and frictionless has been a challenge, right? Because there's a lot of there's a lot of layers to it you know, being having Luke, as you talked about an immutable infrastructure to be able to verify things and to, to like so changes, you know Are not like stuff is not easily changeable and then manipulatable by dev in some ways, right?

You want to be able to prove things out and you want to you don't want necessarily you don't want You want secure development practices essentially and you want them easy, right? And sometimes historically, you know The public keys, private keys, a lot of the other stuff in that process has been a little bit, you know, every dev has something running on their machine, but it's not really easy to scale that or to maintain that or to secure that, right?

So that's specifically touching on why it was a hard problem and talking about some of that. We also touched on you know, what does secure software supply chain look like now? Some of the projects that are coming up, we talked about the Salsa framework specifically, and of course the SIGSTAR project, which You're one of the co founders of Luke.

And so also from our perspective with Cloudsmith, some of the stuff that we committed to with, with in that space as well in terms of being able to allow users on our platform to prove provenance in an open source manner. as well, right? And so kind of the next logical step from that and I'm moving the slide deck virtually in my head here is you know, what's being done in this space now and where, where are we at, right?

And update sort of thing. So since March one of the things that I'm I'm proud to say that we've done on our end is, is made that available for everybody. So on our platform and as, as. Many vendors come on to this. Now we are supporting those open source projects in, in that if you are using a tool like co sign to sign your container images, as well as sign and test your SBOMs we allowed you to host that alongside your OCI Docker container images.

So we made a commitment to that and we, that is being delivered now. Right, Dan? Yeah. Yeah. You

know, absolutely. Thanks Adeel. Yeah, no, that is, that is something that we have delivered. And I mean, this is, this is a topic that comes up. And I said this at the start increasingly, and I mean, it used to be, and I'm old enough to remember without giving away my age, but it used to be that it was primarily larger enterprises.

In specific industries that were heavily regulated or had, you know, really strict compliance requirements, so healthcare, automotive and those kind of things. They were the types of users, especially in Cloudsmith that I spoke to that were most concerned about about software supply chain security. But what I'm saying now is that even smaller.

Companies, even smaller groups, you know, 10 person startups, they're starting to ask those questions as well, which is why we brought support for co sign support for, as you said, being able to use in total attestations for S bombs. We rolled that into the product and we, you know, we talked a little bit about this beforehand, but we believe that you need to, you know, sort of democratize this stuff.

So it needs to be available to the smallest. Users on Cloudsmith right up to our enterprise users so we don't put it into the enterprise plans. It's sort of very bad karma to sort of make those, you know, chargeable enterprise features because it was a good, good phrase actually that Luke used beforehand and I'm just going to blatantly steal it, which was the tide lifts all boats.

So by bringing these features to Cloudsmith, but at every, at every, at every user, so right away from the smallest to the biggest Cloudsmith. We've started to see more adoption of them. Now, of course, we brought them because people were asking. So, you know, we're driven by what we hear in the community.

And this is something that we just kept hearing over and over. I've been with Cloudsmith for a few years now, and As I say, there was always this sort of discussion and Luke mentioned that the red hat have been looking at this for a long time. I've heard this, but I'm just hearing it more and more and more and more.

And now every time I join a call with a new user of Cloudsmith that wants to get on boarded with the product. And of course, that's my job to help users utilize the product. It comes up on every single call. Now it comes up on every single discussion. So that was really what, what drove our. Our development roadmap in this direction was, was listening to the customers and that's what the community and the customers are telling us.

So yes, it's all there in the product. Now, it isn't in a walled garden. It isn't fenced off and it's following that sort of Cloudsmith ethos. It's easy to use. You know we'll talk about this later. I'd like to mention salsa and things like that. And it's, if you make it easy, then people will integrate it quicker.

Look, we've always been able to sign packages, right? We've always been able to take steps like that, but it wasn't, it was, it was never easy. Right. You know, I mean, people, you could sign packages with GPG keys and RSA keys and things like this for a long time, but because it was, you know, not, not terribly difficult, but it just.

It added some friction to your workflow and when friction appears, people will, will, you know, they'll scoot around it and take the path of least resistance. So those security principles need to be on that path of least resistance in order for people to just sort of, you know, adopt them on mass. So yeah, absolutely.

We, that's, what's changed from March. And I mean, I was on that webinar with, with Dan Lawrence from chain guard and I said all that kind of stuff then as well. And it is nice. To be able to come back now and actually stand over some of those changes that we we have made since then and talk is cheap as you say and development time is not and so it's great that we've been able to bring that out and it's been well received right from our smallest users right up to the biggest enterprises.

And also touching on the same point, then I know Lee brought this up earlier was the fact that I think recently attending open source summit and stuff. There's a lot of pretty much every talk is talking about software supply chains, right? That was a big point. Yeah, I think there was a lot

of topics that were focused.

Obviously, it's very topical at the moment. Software supply chain. It's in vogue, and this is a great thing, right? You know, so it's probably something that we've been familiar with for a long time, but just in terms of like global mind share of like how important it is. Events like that are really about pushing the boundaries in terms of like understanding and awareness for people.

However, one of the things I did notice is that the vendors were there. I do. Cloudsmith was a vendor there. Red Hat was a vendor there. And the result are amazing companies that are developing within this ecosystem software supply chain and things related to it. But whenever we're talking to quite a few of the attendees there and sort of asking about what they're doing, you know, in terms of their, you know, personal lives in terms of contributors to open source ecosystem, or more importantly, what they're doing at the places they work.

I think it's still the, my observation was still very nascent in terms of like the actual usage out there, you know? So I think what was the, this is the. Not only are we not in the ecosystem doing enough in terms of like, you know, all the amazing advancements we're talking about here, like SIGstore, transparency, attestation, adoption of S forms, things like that, but even some of the lower hind of fruit of do you check, you know, do you have processes that make sure that the software that you've downloaded is the software that you intended to download or integrate, or what does your processes look like in terms of like ensuring that Actually, the dependencies that you bring in, that you're building your product with, you know, some, your critical product, your, you know, it's the, it's the IP for your company that you're selling, you know, your lifeblood is that, is that, but the, how much do you think about?

You know, the the risk that we're adopting whenever you're adopting third party dependencies, it's still very, very naive. I think, you know, I think there's a lot of work to be done just in terms of building awareness. And the thing is, is that the people have realized that there's processes are realized that security is important, but I just think it seems like the ecosystem hasn't arrived at a point where we've made it as frictionless as possible.

Adopted into into companies and if we can even just further a little bit by this conversation in terms of awareness and show the path in terms of the tooling that can be utilized, but also just like the simple things that people can do, then we've, you know, we've, we've at least for a little bit, and that's just necessary to keep on trying, you know, so I think.

That's probably a fantastic segue into talking to Luke here and I'll offer it in terms of like, because he, I think has additional stuff to share. I'm very interested in myself to hear some of the insights, Luke, especially like in the difference, and maybe this will be a topic for later in the call, but the difference between the old school way of ours, you know, or checksum and signature generation that would be relied upon the company's own pipeline to something where we arrive at the public transparency chain.

And just, and just sort of the difference between the two of those. You know, because it's not just about the company itself. Not, we're talking about supply chains and the supply chain is the connect between producers out there and consumers. And sometimes the consumers are also producing software for all their parts and it just continues so on and so forth.

So it's a much wider problem than just our private use.

Yeah. Yeah. So yeah, you frame that well now in a lot of ways, so if we look at the Okay. Where somebody would have a long term private key. Typically, most of the time you might get a few fringe security geek developers that will have a private key, you know, and and they will lock it onto a UB key, but they're, they're a very small minority.

Okay, then the other uses you have is your big corporates and enterprises where they can afford a HSM, which is locked in a. A room somewhere and you need to sign a clipboard to get the key and go in. And it's all very, very strict operating environment around access to that. Okay. And then you've got the rest of the world, the kind of the 97 percent that just don't know what to do.

Do you see what I mean? They're like, I could generate a private key. Do I keep it on my home drive or do you know what? CH mod seven, seven, I guess that will do, you know, what happens if I lose my laptop or what happens if somebody steals, you know, what if I want to use it on different machines? Should I put it on a USB key?

Then I can move it around. And, you know, so just it's a minefield for users, you know, for that, for that majority. If you see what I mean. And a lot of this was a problem with the tooling. Okay. And so for SIGstore, this became our sort of call to vision here. Okay. What we seek to replicate. So we look back to about approximately 2014.

Okay. And prior years. The amount of websites that were leveraging HTTPS. So secure socket layer connections was very low. It was around 30%. Okay. And I believe the reason for that is because It was a real painful UX. So what you had to do is let's say, for example, I, I deploy a WordPress site somewhere on some hosting provider.

Okay. And I think, right, I need to, you know, have HTTPS. I'm fairly sort of, you know, security conscious a bit. I'm a bit of a developer. I like hacking with things. So first of all, I have to work out what OpenSSL commands to run. Okay. You know, Google around, what do I run? Okay. Then I need to get a certificate from somebody.

So I need to sign up to somebody, some CA provider. tHey want money. So I need a credit card. Okay. So that there's a big majority of open source developers immediately gone. There's some of them are still living at home. Do you see what I mean? All that, you know, and then you need to go through this kind of thing of proving who you are.

So I don't know, scan your passport or they'll give you a TXT record that you put on your C name or some, you know, and then they would ping you. And eventually that would all be okay. And then they'd email you a zip bundle with some certificates in. And then you'd spend the rest of the day going, how the hell do I get this to work in Nginx?

So you'd just be copying certificates around and restarting it, getting very impatient, chmod 777 everything, you know, just get the thing working. And then you get it working. And then a year later, it would expire and you go through the whole thing again. Okay, so that was the kind of the experience really.

And then what happened was Let's Encrypt came along. They said, let's make it free. Okay, no matter who you are, you get it for free. Okay. No special preferred treatment for anybody. And we'll give you a tool which will automate the whole thing for you. In fact, it'll even set it up so that at the end, it'll spit out a comp file that you can just drop into Nginx.

Restart your server. You're good to go. You're protected. Okay. And then from 2014, for the next few years, whoosh, the graph went right up to 80 odd percent. Okay. Now, at the same time, what happened was the browsers started to kind of circle the wagons around HTTP. So they made it so that when you go to a HTTP site, It kind of feels a bit, you know, am I going to catch something here?

Do you see what I mean? I mean, if you went to a HTTP site and it said register for an account, you're like, no, you're kidding. Do you know what I mean? I'm not doing that. You just, you know, and, and the, and the whole experience is danger. Do you see what I mean? So they managed to shift the paradigm. So now the kind of the, the, the, you know, it's majority TLS everywhere.

Almost. Do you see what I mean? The expectation is. TLS should be everywhere. No matter if you've got a small little website because you like collecting hobby toy cars or you're a big e commerce site, it's easy to automate this and to get free certificates. So right now the software supply chain, maybe not right now, but you know, going back a couple of years.

It was 2014 HTTP. So package managers were putting in stuff on trusted. There's no provenance chain, container images flying around everywhere, unsigned. Okay. And, and the general predominant model was unsigned unverified. No source of provenance, no non repudiation, all of these guarantees that you should have were not there.

Okay. So the, the kind of the call to action for SIGSTAR was to be, and we, I've said this so many times to be to software sign in what lets encrypt was to HTTPS. Okay. So basically try to have this free service that's available to everybody. Okay. So it could be that little. 12 year old dude that builds an npm package that's really popular to a huge fan that's Generating their own stuff that they want ingested by the open source ecosystem.

So the idea was that it had to be widely available to all. Okay. And then we could hopefully shift the paradigm where if you pull in untrusted software. It feels a bit icky. It feels a bit dangerous. It's, it's socially unacceptable, essentially, you know? Yes. And that, that, that was the kind of, that was the goal of SIG store and you know, and, and like I said, I mean it's the, it's working out pretty well.

'cause we've got a lot of very large uh, open source communities that are adopting Sig store now and, and starting to leverage six store. So, you know, that, that, but that was the good sort of focus that we had to really drive forward.

That's an amazing,

yeah, it's an amazing background and I think it's not, it's no coincidence whenever I'm asked about six store, you know, by other people in the community or perhaps people who are less familiar.

It's the exact knowledge that I utilize in terms of the way I see stor is the comparison. That script. But it's also one of the things you said is essentially the emphasis of trying to optimize for secure by default. You know, so making that really, really easy that you don't need to think about it.

Like, don't make me think whenever it comes to security. You should think about it, but at least if you start from the principles of it's secure by default, then you're already a fantastic foundational state. Let's encrypt. And that okay. And the, and the browser shipped as well. And this is linked to the way we think things are going in general with the ecosystem of software and software pipelines.

That at some point. It will have to be secure by default. And if you're not doing these things, you're not going to be the outlier. You haven't secured artifacts. You haven't thought about provenance. You're not doing signatures in the test station. And you're going to be an outlier of somebody who people will not want to do business with.

You know, so we're not there yet, but that's the way that it will become. So the price is out there. You're going to be the consumers of software. We'll expect you to show that you're paying particular attention, due diligence to the process and making it secure by default for the software as well. And that would be a fantastic golden age if we're able to arrive there.

And yeah, I think it's a fantastic analogy.

Yeah, we will very much live or die on the UX. It has to be simple and seamless. I'm a developer and I have an incredibly short amount of patience with. Anything that gets in my way, you know, my wife will hear me and she'll go, what is it now? It's just like, you know, I'm, I'm kind of, you know, I'm, I'm using language that I wouldn't use on a public forum.

And cause I've got, I'm very intolerant towards disrupting me working. Do you see what I mean? So, so you're so right. This Lee, this has to be seamless. The UX is, is really is central to this. Been a success in this area with sorry to just reflect back, but we had a lot of this with SC Linux when we were trying to get that off the ground at Red Hat because developers just disable it.

Yeah. Right. You can code. I disable it. Someone else's problem. You see what I mean? And so there was so much put in to try and make SC Linux more. user friendly like these. Did you ever see the cartoon books that came out? The penguin and the dog from each other's food and all sorts of tools to generate policies.

And, you know, cause you do you, if the developer, if it, if it in any way impedes them or they feel it impedes them. And there's too much of a time cost for them to adopt that technology. It's you're, you're, you're, you're paddling up the wrong way, really.

Yeah. That whole secure by default thing is very, very, very true, rings true in this space and, and making it easy, you know, and making it accessible so that, so that people will be able to enable it by default too.

So I think another couple of points to highlight in this space with what, what progress has been made is, I mean, especially with six door. All these projects, I think, you know, the, the ability to sign a test different formats of packages. I think one of the big ones, of course, that led the space was Docker images and OCI container images in this space that was early to this game sort of thing, and, and that's kind of what, what, you know, a lot of platforms, including ourselves are supporting.

But I think that recently we should call out the fact that, you know, there's been a lot of movement in, in a lot of different package managers and, and. Format. So, so NPM recently with their RFC that came out to adopt six door as, as as a project and a standard for them you know rust as well.

I think you, you mentioned the key line project stuff, but, but also get sign is a really important project that I've been watching closely to wait for, to get the GitHub verified badge eventually, which is, which is which would be great to be able to have keyless signing, right. Of your commits.

That's great. So any, any of those you want to highlight, Luke, you can feel free.

Yeah. I mean, one thing that comes to mind. I think what we've done, right. And I don't want to appear arrogant here because I've done so much wrong in my career. Looking back what what has gone right. I think in a lot of ways.

We didn't wait for talking about six door and I should pivot to other technologies really but but we didn't wait for a specification. Right. Wait for, you know, we, we just started to build, okay. And then discover from there. It's right. It's right. It's right. Okay. Refine, refine, refine. And that then got us in the position that we had all these tools together and these language frameworks that when somebody came along that needed to solve the issue.

We had something that they could use, okay. So for example, with the, the NPM stuff, we'd already been some Java work. Okay. There was a Shopify. We're looking to do something for Ruby gems. We already had a Ruby gems library because we were building all these things and, you know, and they were all there and available.

And and then at the same time, I mean, to, to call out other projects is brilliant work that's been going on at Salsa, you know, around the, the kind of much needed definition of. Of a kind of a roadmap to reach in an optimal, secure build environment that you should have. And there's also many tools coming up around generating S bombs as well, again, starting to do that.

And I think it's, it's, it's an interesting juncture at the moment because there's two years has been a frenzy of innovation, you know, some incredible tooling that's built. And companies such as yourself that are kind of massaging these into a form that somebody can use, a customer can use. And the interesting thing is we're now at the juncture where To touch upon where you, you said earlier, where people are, are asking how do we, we approach this, you know, where, where do we, where do we begin?

And we now have this juncture where uh, public sectors are starting to mandate that these things should be in place. Okay. So now there's a real large, heavyweight leaning on enterprises to, to implement this. Okay. And I think that's where we now have to sort of. Go to that next level of being the educators really, and helping them to understand how to do that.

Interestingly. And at the same time, we still have a lot of work to do as well. You know, there's I mean, it's, I think in a lot of ways, I see a sort of a logical project progression here. With six store, we started to sign things. Okay. And with S bomb, we started to generate S bombs. Okay. There's still a lot to do around the verification of these things.

Okay. And then. Education around the S bomb, where should it be stored? And what is the perception around an S bomb? Because I think one of the dangers is that a lot of enterprises are going to think it has an S bomb, it's secure, which is not the case at all. Do you see what I mean? Or a software project might not have an S bomb, but it might be very secure, it might be coded very well, it might have a very low attack surface.

So there's an S one will never be atomic, you know, because your use of software is very varied. You see, I could be using a piece of software, but I might only be using a single function, you know, 5 percent of the code base that's within that library or that framework. But I, you know, this is not to say there's been some.

incredibly good efforts to address these sorts of things, like there's VEX that's coming up. You can understand your exact exposure. And there's some smart people grappling with these problems, you know, including yourselves and other people that, you know, around the community and, and it's, yeah, it's just a very interesting time because we've now got that, you know, Governments are saying mandatory, you know, you need to, you're not going to be able to use anything unless it has one of these.

And at the same time, we're still kind of working on these. Yeah.

And I think that's an important point. Thanks, Luke, for highlighting there's because I think part of the next sort of slide that I had there was Was around what does this look like in practice for the open source and for enterprise right and I think a couple of the points you touched on are really good like for for sort of an intro one on one basics to coming into this, you know, talking about securing your software supply chain.

You have your S bomb your software bill of materials. Now, of course, there's a lot of talk around whether this is something that you're going to generate at build time when you build your actual artifact, or is it something where, you know, you're, you're using tools like SCA software component analysis to actually generate this after the fact, right?

Now, the good news is there is tooling in both ways to do that, right? If there is ways to generate your SBOM at build time that are out there, open source project, even if you don't have one already, let's say you're using a container image, you can. Use a great tool like like SIFT, like Anchor SIFT to call another open source project that will actually take your image and generate an S BOM for you, listing out a lot of the components in that package.

And once you've generated that, you can generate those S BOMs, software bill of materials in, in two, two large formats is the ones we've seen out there. You know, of course, SPDX is one and Cyclone DX being the other. And so, so a lot of folks getting started. That's where they start as they start generating a software bill of materials.

Now, now, of course, with projects like six door and cosine, what you can do then is you can start signing your container artifact, your image itself. You can start signing or attesting your individual S bombs and hosting those alongside. You know your artifacts on a tool like Cloudsmith, right? You can now now not only are you showing that hey This is actually what's contained in this image But you're actually attesting to the fact that yes I am who I say I am and i'm also verifying that this is the image and here's how you can show here's how you can actually pull that down and the next step I I see a lot in CICD now is Is with CI pipelines is folks It was generating the S bomb and attesting doing a test in total attestations was step one.

And I think the step two part of this is important is now when, when users are pulling down or consumers are pulling down these packets, are they actually using the verified tool, right?

But I think it's prudent to start signing and Producing definitely, you know, even though the verification part is still, we're still kicking the tires.

They're working out the best approach. There's absolutely no reason to not start signing and generating now. So you build that historic picture, you know, you've got that historical context. It's a good habit to get into. Definitely. No, no need to wait until everything is, is perfect and let. Perfect, be the enemy of good, we can, there's lots of good that we can do right away.

Yeah.

Yeah, so start the generation, start generating your SBOM, start signing now. And then eventually, when everybody comes in to verification that that'll be there. The other additional periods that you mentioned real quick that I do want to highlight for you users out there listening is VEX, right, the vulnerability exploitability.

I think it's exchange. Yeah, I can never remember that. Yeah, I can never remember either, but it's, it's, it's around this principle. I think I know Dan Lawrence from Jenga talks about this a lot where the idea is, Hey, it's great because, you know, I mean, a lot of this too, you know, we had the NIST database, the NV and other things like this, like NVD, sorry.

And like other frameworks that tell you when there's a CVE present or something, but exploitable? Are you even using it in your code? So I think being able to understand these CVEs and other things, and to be able to score that system in some way to say, Hey, This is actually fine, you know, the same image is actually fine.

Go ahead. It's

also really, really important to tie the ecosystem together. So Luke's highlighting the fact that standards are important because they enable co integration with different tools. You know, so for example, with VEX or with SPDX. It's more than just one vendor, more than just one solution. We have CI systems, CD systems, and there's a lot of sorts of information that we need to be able to tie together and into the center place, because only then do you have a holistic view and everything you've got in terms of the pipeline, you know, and I hope that the dream is at some point it's like, okay, we've got a lot of.

systems that are cooperating at some point in the future, utilizing the standards to be able to do that because it's machines talking to machines and we want to highlight the value of that in some way to the users via the attestation and the fact that they've actually got this holistic view into the software pipeline that they've built including all the way back.

The every single dependency that's built in the software that you've built. And sort of probably a selfish item from my perspective, obviously just talking to my clients with is that, you know, we were built as a sort of abstraction layer for the ecosystem in terms of artifact management. You know, so we think about things in terms of how can we integrate with different types of products in the ecosystem or services or tools, but offer that in a nice abstract way, regardless of the artifact format.

And that's still a work in progress because, you know, it's not neat and tidy. It's a mucky business is often how people describe it to me. But, you know, I think that we can only do what we're doing based on open source ecosystem and standards as we've talked about. So it is incredibly important.

Leo, I think as you called out, just sorry, Luke, real quick was the the idea that this is Louis Luke mentioned as well, was the UX, you know, the user experience in the UI, that's something we focus on a lot over here, right.

Is, is the ability to be like, Hey, make this accessible for everyone. Make, make it actually built into the UI so they can see it, you know, make it easy to generate, but make it also. So easy to host alongside and, and make it easy to verify, you know, once you have those pieces in place, you know, then you'll see a lot more adoption with users saying, Hey, I actually have this button or toggle I can turn on, this is reasonably easy to do as a developer.

I don't have to generate public private keys that I'll lose somewhere. Those pieces are really important. Go ahead.

Yeah, I was just going to say, so that paper trail that Liam was describing, that's what keeps CISOs awake at night, is what is their exact exposure to any given risk when it turns, when it, when it, when it comes to, to be.

Okay. So a good example is a log for J. Okay, they want to know where am I exposed, you know, what is the paper trail and and, you know, and, and what are the high profile targets, which ones are more exposed, you know, and that, and that's where some sort of expressive. Frameworks such as VEX can really come into its own then, you know, you can actually ascertain your, your exact exposure.

I

think someone was, I'm not going to name names it was someone external was asking me about this recently, relative to SBOM and why it was important. And although there's a lot of work to do, I think the analogy that I had brought up related to what you're speaking about, Luke, is that you're eating food, you want to know, maybe you've got certain types of intolerance, you want to know what goes into the food.

Perhaps if you care more than you want to know where that food was grown, where it comes from, what country the origin was. And that's not always easy to get with food, but at least there's some, there's some standards in order to help with that. Software is exactly the same in terms of the S bomb is really describing about what the makeup of that was, you know, and the relationship between what went into it and something like VEX is, well, what is the impact of what was into, what went into that software.

So if VEX is describing the vulnerabilities as they're pertinent to something like Log4j, And the S volume is showing you that that is part of your pipeline somewhere. You should be worried and you should be looking at remediation and thinking about, you know, how can we prevent this or fix things.

But only with those we have visibility. You know, I don't think this came apparent whenever I asked another CTO at some point in my recent past. Do you know what software is going into the products that you're building, what your teams are utilizing? And the response was, I feel like I really should though, but I don't.

And I think that at that point I realized that, you know, there's still a lot of work to be done in the ecosystem. So going back to the bare minimum of things that people should be doing, you should be thinking about the dependencies that are going in the software. We should be generating S bombs. You should be generating signatures and doing the bare minimum low hanging fruit of just, like, checks on verification, actually checking that the products that you're building are what is perceived and consumed, and yes, building up that paper trail for something where we can actually utilize it and enhance it upon over time, you know, as the ecosystem

evolves.

So Lee, I'm going to take your analogy one step further. Sorry, Dan, I'll let you speak in a second, but I want, I want to add to this because I love this idea of the, you know, sort of the ingredients on the side of the box. Right? So I liken it and sort of to take the next step with that. I liken sort of you know, your SBOM as, as that sort of ingredients list.

And then when people talk about, Vulnerability exploitability with effects with vex. Let's say I think of vex is sort of, you know, the ingredients on the side say sugar. So you freak out initially saying, oh, there's sugar in here, but facts will say, no, this is actually not high fructose corn syrup. This is just fructose, you know, something straight from fruit.

And so that's. Sort of that peace of mind that you have to say, okay, this is an exploitable, you know, this isn't panic because I know we've all run you know, different tools something like even gripe will, will give you that output from incorrect to total call out sort of against an S bomb, they'll be able to generate right.

What, what vulnerabilities exist. For a lot of that stuff. I mean, we've all seen it. There's just like initial panic. You just see the long list of stuff that's vulnerable. And you're like, Oh man, here we go again. I have to go through this whole process to check everything, but it's important to sort of like separate that signal to noise, right.

In terms of that, go ahead, Dan, you were going to say. No,

it was just really to follow up. Both Luke and Lee had said that. You know, especially with log for Jay. I mean, I do work in the trenches in the front line and the support channels lit up that day. Everybody was suddenly concerned. How do we determine our exposure?

We need that visibility. They wanted that visibility on exactly where they had deployed this. And I think it's easy to get caught up as well on. Doing things, you know, absolutely perfectly and getting to that maximum state. If you look at salsa, if you look at salsa level four, where you need, you know, hermetic reproducible bills and things, that's quite a target to hit for a lot of people.

It can be very off putting, but the key is. To do something right. Let salsa level one is not off-putting, it's just an automated documented build process, right? With provenance for your artifacts. And that's a, that is a great improvement. Anybody that was there is in a much better position when log for Shell hit than, than anybody.

That's not on, on the path at all. So it's not about looking. Example, like I say, set level four on the styles of framework and I'm thinking, Oh my goodness, there's so much work to do to get there. And how do we achieve that? Just start somewhere. As Lee said, yes, you know, check, check, check some sign packages, have an automated build process.

You don't need to go to the full extent to actually reap a lot of benefit, a huge amount of benefit, and it really outweighs the effort that you have to put in just to do a little bit. So I did want to just say that that's definitely what I'm hearing.

I think to enlarge upon that. I mean, you, you might have this sort of Fort Citadel build system completely reproducible, every single artifact, every single line, there's provenance for that.

And a developer gets their account compromised, their Gmail account. Okay. And then, you know, and, and that a lot of the time is how attacks happen. Attacks. A lot of the time, they aren't these complex. No. Buffer overflows against a piece of hardware and you know, it's, it's, it's, it's very simple things. It's very simple low hanging fruit where attackers get in and that's, that's what they target.

They look for the simple things. Okay. And so there is so much that you can do, you know, with just, you know, make sure your developers have 2FA switched on. Okay, that's the key one, you know, I can tell you have a couple of really big companies that have been That the the exploit has actually started around us a compromise around single sign on To that effect where developers account has been compromised.

They then used it to Backdoor code to a code repository and they've accessed the JIRA for closing tickets and all sorts of stuff just from a developer account compromise. So there, there is so much that you can do. And I think one of the things that we spoke about, we were just chatting before the webinar is your, your CI environment.

A lot of the time that, I mean, there's, it's, we live in this. Time where developers have a lot of freedom to express themselves. Okay. And this particularly plays out in CI. It's, you know, it's, it's it's a wonderful world of marketplaces and plugins and scripts shared everywhere that you can run, you know, to do all these automate everything, do you see what I mean?

And I think we kind of enjoy that. We kind of. There's some sort of validation that we get from changing and tinkering with things and making things automated and, you know, and but a lot of the times the security can really take a backseat in that process. Do you see what I mean? So there is so much that you can do just to really take a step back and look at your CI.

And think, where exactly are we pulling things from? Do we really need that? Do we really need that bot that's going to say, Hey, new contributor? You know, is it really important that, you know, who's this coming from? Start to look at stuff like that, really. Cause cause, cause that is part of your production chain.

Okay. Whatever you're producing there. Whatever's been ingested there is going to be part of your production workload. And we never used to treat production systems that way. You know, we were very minded around hardening running scanners to make sure services that you don't need a switched off, you know, the permissions would be locked down.

Or nobody accounts would be closed. You'd really harden a system. Do you see what I mean? And, uh, we have quite a disparity now where the CI, the CI environment it can be a bit, well, hey, you know, having lots of fun here, you know, writing little trinkets and scripts to do different things and, you know, and just, you just one click button and you put in a, something from a marketplace and then you've got this new funky thing in your CI and, and so there's so much that you can do there as well.

It really pays to develop with that with production in mind and, and, you know, making that secure by default you know, mantra continue is, is very important, making it easy and accessible for, for developers to do that is the key. Right. And Celsa

also captured this quite succinctly as well, and Dom sort of mentioned this.

I should

quickly mention Celsa is. Yeah. Yeah. I just want to mention that that's supply late supply chain levels for software artifacts. So it's you can check them out at salsa. dev. We, we do want to call it out. Yeah. Yeah. It's, it's,

it's, it's one of the frameworks out there that are really helping sort of advance awareness of why supply chain is important, but also what you can do as a company.

And obviously Cloudsmith as part of what we do, there's some alignment there. In terms of like, we will help get you to a certain point for it. And then, you know, obviously there's things that you need to do to build upon that. What I was going to say is that Salsa succinctly captured an aspect of just because the ecosystem is built upon level zero artifacts and they don't really do a lot, it doesn't mean that you can't achieve some level of compliance with due diligence of your own.

You know, so it says something like, you know, a level four artifact produced might have been built upon a sea of level zero artifacts, but you have to start somewhere, right? You know, so I think the process is in your company. It starts with the company's here to utilize the product, whether that's from adopting tooling like SIGstore and really, really important things in the ecosystem or by utilizing something like Cloudsmith as a central source of trace.

For artifacts within organizations, you know, so it's pretty important to start tying it together with

Ralph and Tilly. Yeah, no, so, so I just want to do a quick time check. I know we're low on time here. We may go a couple of minutes over just for folks if they want to. But yeah, talking about some of the stuff we should call out some of the projects in the space that we've talked about here.

Before I do. I know our, our folks over at Cloudsmith have been busy in the, in the Q and A as well, Kira from DevRel, Kira Carey who recently did talks on, on a lot of open source software supply chain stuff. She has a lot of great content out there, so I'm going to quickly shout out to her because I think she's got some articles on our Blog and she recently did a webinar on actionable as bomb content right being able to, to actually do stuff with it and analyze it and take a look at it, not just generate it.

So I think, feel free to check out the classmates blog for some of our content. She's also done a webinar on that and she's recently done talks as well so. Will include link later, I think Candace is including some links after and as well. I mean, just wanted to call out you know, links foundation cloud native community Computing Foundation.

I mean, a lot of the folks behind efforts like salsa. dev and, you know, like supporting projects like six store and cosine. I think it's really important. So. That's been really great. And, you know, obviously feel free to check out cloudspot. com. We support open source repositories secure by default.

We like to generate your SBOM alongside there as well. So I think the one question I will highlight here real quick from Alessandro is there was a mention here. He said, Hayal, is the SBOM signing paradigm the only way to solve this problem? So I, I'll just quickly preface this by saying, I don't think it's the only way, but.

In terms of you know, formats and open source standards. I do feel like that's the leading thing right now, which, which the community is again, circling the wagons around, like focusing on. So I do feel like it's important to focus on centralized efforts that everybody can use that are accessible, that are not locked behind enterprise platforms and stuff.

So go ahead, anyone. I think,

well, what I will say about it is it's not the technique of SBOM and SININ in itself, it's the reason why you're doing it. So SBOM is visibility, right, to be able to understand what went into the software and why, and then the SININ side of things is, is the verification to say, can I prove That the state that I've read it with is true, you know, so certainly S forms and silent alone are not just the answer, but they're definitely a big part of what's necessary.

Luke, have you got anything in terms of like additional capabilities? I think.

No, yeah, I would agree with you there. It's, it's, it's what it gives us, you know, it's, it's the data sets that will provide for us to then be able to make decisions. But that's one part of this, you know, there are lots of other controls and technologies.

Yeah,

what I would say, sorry, the other thing I would say, obviously, is pick your tool and wisely. And, you know, I think in terms of utilization of the, of things like this, an organization, that's really, really important from the, trying to think about it from a uniformity way, you know, so you really don't want to have outliers in your organization that some people do sign in this way, some people sign in that way, or some people don't do it, or some people store their artifacts in one place and then other teams in another.

The more that you can leverage doing the same things across the organization, well, one, it's more efficient, and two, you've got a sort of common language internally, utilizing the same tooling. So whether it's SIGstore or a different solution, it sort of doesn't matter. The outcome is that you're generating the SBOMs, you've got visibility on what you've got, but you've also got some way of verifying the assets you're building, and you've got a secure place to store and distribute those from.

That's the selfish angle of right that we're talking about. But regardless of what the solution is, you need to take those boxes and you need to be thinking about it, I think, in terms of your process holistically from development, as I said earlier, right through your production. And that's the key to making sure that, hey, this is secure, more secure than perhaps what we started with, secure by default.

And I think that's along the lines of you know, I mean, also to, just to touch on the S bomb piece of it. I mean you know, remember that that's just a, a format like S bomb is just a principle, right? Software bill of materials, there are different formats for S bombs. There's SPDX and Cyclone DX.

And there's many, like Lee said, there's a lot of tooling around this to use these different formats and coming up in different package formats as well ways to generate the different S bomb types. So it's, I think it's the principles. You have to hold on to more. And I mean, the tooling is there and, you know, choose wisely, so to speak, but, but definitely it's the idea of, you know, like getting from, into the mindset of, of securing by default, for sure.

So I think a great way to summarize this whole talk is Kira has a posted a question in, in the chat as well specifically towards Luke. She said, Luke, are there any easy tips for securing your software? So I think that's a great way to end this.

So is there one easy, one easy trick?

I think what you mentioned before in terms of start simple, right? I'm

just trying to stick to the easy bit,

easy tips to secure your software. So as I'm, I'm, I'm assuming if it's. You know, they're saying your software talking about software that they write uh, if it's open, okay, then it's going to have more eyes on it. Certainly. So that's, I would say that's, that's probably a bit of a sneaky get out one, generally like invite people to review your code, you know, leverage peer review.

That's a very good security control, which is easy and free.

It's the best way, right? Keep it open. Also work

with people who are really into security and involved in the process, right? Actually, this has come up a couple of times, like, you know, in the conversation. And I've heard Linus Law, I think, mentioned several times.

And it still holds true to, until today. And it's essentially given enough eyeballs, all books are shallow. And I sort of think there is an evolution to that, you know, it's, it's fantastic and it's a great proponent of the open source ecosystem to say, well, the more people there are to look at things, then the less likely that there are issues.

And that's the reason why open source works really well, you know, and I think that applies internally as well, which is basically what Luke is saying. It does sort of have a jokey extension of giving enough S bombs. All exploits are shallow, right, you know, so you know what's going into it, but the point is, is just, I think, you know,

yeah.

And something else that quickly comes to mind is the OpenSSF. Go to the OpenSSF org. They've got best practices for developers, they've got lots of nice. Guides there that can help you run it secure

somewhere. Yeah. So thanks so much everyone for your time. I'm going to wrap this up by saying thank you to my colleagues, Luke Hines, Lise Killen, and Dan McKinney for joining us today.

I am Adil Ligari from, from Cloudsmith and thank, thanks so much everyone for your time. I'm going to throw it back to Candice now. I hope that this was productive and, and helpful in further furthering this discussion. Thanks. Over to you, Candice.

Thank you so much, Lee, Adil, Dan, and Luke for your time today.

And thank you

everyone for joining us. As a reminder, this recording will be on the Linux

Foundations YouTube page later today. We hope you join us for future webinars. Have a wonderful day.