How Do Mature DevOps Teams Manage Software Security? [On-demand Session]

Oct 6 2022/Webinar/34 min read
We’ve assembled a panel of experts from the mature DevOps teams of Puppet and Shopify to answer some of your biggest software security questions.

Can't see the embedded YouTube link above? Click here.

There is so much information out there about software security. Every day, there seems to be a new news headline, government regulation, or tool promising to “fix it all”. Do you ever wish you could just peek into how some of the industry’s best dev teams are managing this?

We’ve assembled a panel of experts from the mature DevOps teams of Puppet and Shopify to answer some of your biggest questions:

  • What practices are high-performing DevSecOps teams implementing to bake security into their development practices?
  • What are the big security challenges these teams face?
  • What role does team culture play in attempting to improve security while also maintaining DevOps goals?
  • What are the biggest blockers for adoption?
  • And more!

Jacques Chester, Sr. Staff Software Engineer, Shopify
Nigel Kersten, Field CTO, Puppet
Moderated by Ciara Carey, Developer Relations, Cloudsmith

(00:00) hey everybody thanks for tuning in today to Cloudsmith's monthly webinar today's topic is on how do mature devops teams manage security. So before we get started let's go through a few housekeeping notes, we have prizes to give out so we have two free lunches and two free prize packs to give away at the end of the webinar so be sure to watch till the end to win a chat to be at a chance to win we're also streaming on Twitter on LinkedIn on YouTube as well as our webinar platform please post questions to whatever

(00:37) platform you're using the wonderful Hillary will be monitoring those channels and giving them back to me we're going to be holding one or two polls and again you post in the the your platform or tweet or chat and we will be looking for those questions your answers to your polls so we have two really amazing guests today for our talk so let's bring them on stage hey everyone hey Nigel hey hey so uh today we have Nigel Kirsten and Jack Chester. Nigel Kirsten is the field CTO at puppet by perforce and he's the

(01:24) author of Puppet's much loved report on state of the devops and we also have Jacques Chester he's the senior staff software developer at Shopify he's an author of a book K native and action on Building Services applications and he's um the chair of open SF software repository working group and heavily involved in Ruby's open source community so uh thanks for coming today hey and um Nigel so you have released 10.

(02:00) I think that's the full 10 reports every that's a lot on the state of the devops so uh how was that it's been a pretty massive effort over the years and I have to say you know I'm sort of the last person standing so to speak that the people who I if you're going to talk about the history of State develop support who I think had a bigger impact than you also Alana Brown who's since moved on and now works at remote.

(02:23) com um it was her idea in the first place and she was she really drove it for a number of years I was co-author and then when Dr Nicole forsgren came on for for four years I think it was there she really bought a level of statistical rigor and research to the whole project but there's been so many people Gene Kim James Turnbull Jazz humble Michael stunke we've had so many great authors over the years but last year for us was a really big one because it was 10 years and suddenly made me realize how long I'd been messing around this industry I

(02:52) had to look up the term for this the other day of semantic Association you know when you say a word over and over and over again and it stops it loses all meaning I think devops and Deb second yeah you say devops 20 times and it doesn't mean anything anymore and so what's once number 11 going to be as sure and are you still going to be as big a part to us yeah I'm I'm guiding it at the moment we've got a fantastic researcher Ronan cleanan who's taken on sort of the bulk of the work and is

(03:22) working with some research firms for us we're trying to do something a bit different this year because I think I'm looking chat about this this topic goes forever but basically I think devops is now such a big big field it's very difficult in a single report to sort of come up with interesting useful findings you have the folks at the beginning of their Journey the folks who are very much post devops the folks who've moved on who've tried it who it doesn't work who it does work and trying to do all of

(03:46) that in a single report I think you just end up producing a book every year we're just focused on platform engineering yeah I I can't write a book every single year it gets true yeah yeah yeah absolutely and Shaka tell us about how you recently I know you're focusing on open source security and I'm a long time listener I haven't called any of your working group um software repositories and I'm just wondering uh have you got into that and how what was your journey to yeah I've been I've been co-chairing or

(04:28) Deputy chairing I don't know how you want to describe it with Dustin Ingram from uh pipio from the python software Foundation how did I get into it I used to work for a company called pivotal uh which you know I really enjoyed my time there and one of the things I worked on at pivotal was uh what was called pivotal Network it was our distribution point for all our software products which we needed to do legally and some of the products got installed in cages under armed guard uh because they were fairly sensitive

(04:58) sort of operations is this like euphemism or really no this is really a thing that happened like the the software like the USB stick would be walked under arm guard like it was it was that kind of a place okay and I I suddenly thought there are people in the world who would be very interested in getting inside those cages through our software uh and that was one of those uh oh expletive moments um and that that led to my interest that was that was sort of like the lightning bolt that led me down the path to where

(05:30) I am today oh cool um so our topic today is how do you mature devops teams manage software security and so I thought my first question I'll post to Nigel and like like I know we're saying devops means nothing anymore what those devops mean is it just uh um Automation and cloudy stuff it's just tools right yeah I mean this is I think it's a tough one and you talk to folks like Patrick Dubois who's very much coined the term and there was a deliberate it was very deliberate that there wasn't a clear definition here is

(06:08) exactly the reductive definition we have of what we're trying to do here because in many ways if we tried that if you look back at the early days it was basically a bunch of sys admins going how do we actually be agile how do we actually take agile and spirit and apply it to operations oh look we have all of these cultural problems all of these accountability you know ownership doesn't match Authority all of these things and I think we had had such a vibrant interesting exciting space emerge because it wasn't really tightly

(06:35) defined and you turn up to devops days and you could get a talk about just about anything but then we hit the Enterprise and I think the lack of a definition meant that honestly like shyster vendors stepped in and started going we do devops here is devops in a box or Consultants coming along in a similar way to Agile and faith and various you know sort of permutations like that but I don't think are particularly true to the original spirit so as far as what it actually means to me I take a pretty big tent approach it's a

(07:05) loose collection of practices Technical and cultural to get over organizational boundaries inside organizations so that we can ship software with less stress and better like and that sounds really vague and could apply to just about anything that every time I try and narrow it down much more than that I end up cutting out something I think is important yeah it was like I've worked on those teams where it's like every six months I think that's a normal kind of thing you would release something and it would be very

(07:31) stressful and something would go wrong and um then you'd have to roll back and it it was it was a high stress moment in a a big long journey so I think that moving away from that is is a good thing I'm having 183 people on a bridge call over the weekend yes no one ever wants to do those calls um and so how does that um how do you see security is security becoming bring being brought into it more it wasn't like at the start we tried to merge development and operations and now we're well like not just now but now we're

(08:13) like oh security we're kind of still a bit siled let's bring them back into this tent and um is that how you see a Tracker yes uh and unfortunately in just because of the sort of the economics of the situation it's going to be siled for a while in a lot of ways uh there's just not that many cyber security folks to go around uh so a lot of organizations um either deliberately without thinking about it or out of regret wind up with a Central Security team that acts as a gatekeeper which we know from our devops

(08:49) days uh is is an anti-pattern um the other thing I see as an anti-patent is again very much like the experience that devops went through the evolution is the idea that there's a box of software you can install and today you have security and that's not true at all um and it's it's a Pity that we have to go through this Evolution but I'm I'm hopeful that we'll come out the other side with something better and I know it's not a box but is there like this well a bit of a lag up yeah it is important to think carefully

(09:24) about your tooling um Dan lorentz who's the CEO of a company called chain guard uh says a lot and I agree with him I had a similar sort of motto Once Upon a Time which is that uh build is production the the systems where you are building the software are as sensitive and uh you know risk dense as production itself because as I said you know like if somebody gets into the bucket of bits you are in a world of hurt and a lot of the time people historically have underrated that risk uh and so the bucket of bits has been

(09:57) the fastest path into production to attack the build system itself or the uh the artifact system itself as well um so in terms of your software you should think carefully about about those systems and securing them and hardening them and and applying all the security practices you have now to them um but I think there's there's sort of like two great tributaries of risk or two great tributaries of security risk that you can think about flowing into the river as it were and one of them is that build system Upstream dependency

(10:28) you know risks that come from the outside of the organization and then risks that come from the inside and the really big one is uh making sort of you know unintentional errors in your software that lead to a vulnerability that one I think doesn't get as much time as it needs to um because it's hard again to install something or it's hard to have it you know a checklist that says I have now secured myself against security errors yeah actually one of the times where you probably as a developer have the most

(11:01) power over security is when you're bringing in these dependencies like what like what is it that you should consider when you're like considering bringing a brand new dependency like water that you could like what does the checklist should you have a checklist or are you can use or should you be able to test anything on your developer well yes or no so that that's an emerging field right now is is people producing these checklists um there's even a startup called soccer.

(11:33) dev who have have sort of automated the checklist uh for npm at least um yes I would broadly say like take the things you're already doing so is this project Lively you know is it active are people still contributing um do they respond quickly to problems um you also want to look at security practices like do they have MFA enabled on the repository accounts that they use um but also you want to make sure little things like are you accidentally installing a different dependency from the one you thought are you making a

(12:06) typo so double check that you're getting the package you expect to get little things like those can add up to a lot but I think we're in the early days of having having a strong story about how to pick dependencies with a security point of view because I think one of the things that's underpinning all of this is how hard it is because you know software development is a team sport the teams keep getting bigger and bigger and bigger with different roles and it's often really hard as an individual practitioner to

(12:39) actually make a good decision whether you're locally or globally optimizing and I think that's what a lot of this stuff comes down to it's like your your job that you're being measured on is to ship some software or Implement some features resolve some bugs or whatever and if everyone just goes for the shortest possible path to get there you end up in a situation where the environment they're operating in becomes more fragile more aerophone more insecure and yet we're just not very good as human beings working in large

(13:06) groups how do you surface the right kinds of things to make a decision between local and Global optimization yeah they don't have a solution here no if if you have a solution then I urge you to um put your name in for a Nobel prize in economics exactly because that would be a pretty big breakthrough for a life cycle source code yeah the cicd system the artifact suppository the dependencies the external dependencies on public repos uh and then all the toolinges as well you're like you're scripting your environmental

(13:49) variables like it's just there's just a lot there is there is and that's that's one of the hard things about being a software developer is that there's so much to know about so many topics that it's hard to be an expert in everything um I again I wish I wish I had the solution where I could just you know do a sort of uh an Isaac Asimov thing and you play a tape and that puts a memory in your head uh you could you can tell how dated that story is um it was only good 10 years ago yeah exactly you put in the reel to reel

(14:25) and some blinking lights and there you go um but I think there's there's still a lot of value in uh creating a minimal level of awareness of the possible issues you don't have to necessarily know the solutions you just have to know a that might be a problem here and B where you can get help yeah absolutely and I know both of you are are have talked about how um cultural change and how people are actually and focusing on people is but is a great way to get better security um do you want to talk about

(15:02) um cultural change um in devops and how to how to get your devops processes um really nice and secure using culture Nigel or sure so I think there's a there's a bunch of things to I think unpack there one is that you know devops and you know a lot of the most significant Tech movements we've had of how we build software these are all Grassroots movements these won't buy people at the top of the hierarchical pyramid inside organizations these are people who are down at the bottom and so it's easy to sort of go

(15:37) um we have a cultural problem um and one of the things we found out from last year's state level support when we did a bunch of qualitative and quantitative research was that all organizations with lots of what we would call cultural problems talk about culture all the time but organizations that don't have many of those sorts of problems they stopped using the word culture because it's not it's not actionable and it's actually encourages a weird kind of form of helplessness inside organizations like if you're an

(16:05) individual developer and you're like ah well our culture doesn't allow for people to just make those sort of decisions everyone goes huh you know it's like an earthquake what are you going to do about it you know you just sort of wait for it to move on but organizations have actually implemented these sort of changes and had fewer cultural problems somewhat paradoxically don't talk about culture they talk about specific things we have a problem with ownership we have a problem with making decisions quickly where the problem with

(16:29) documenting travel knowledge or incest knowledge around the code base like all of these things are quite actionable yeah and one of the things last year with the team topologies authors Manuel and Matt who if you haven't read tank topologies it's one of the best organizational design books ever around Tech and they they definition they came down to was stop talking about culture talk about what you need to do to be able to ship something quickly with low cognitive load and stress on individuals if you

(17:00) actually look at those things and identify one then they start becoming things people feel like they can do something about um so that was a really long-winded way of saying I think culture is massively important but you've got to go at least one level below and go what is it we're trying to actually achieve here like not let's not just say culture and throw up our hands but let's go what's the problem and how are we going yeah and then you can make like incremental changes and get better and better and

(17:25) better and until you've just and people people can tell if they're making a difference as you say when they're working correctly one of the things I found really frustrating when I worked at Google was there was this ineffable phrase googliness and people go well that's not very googly and you're like I don't actually know exactly what you mean and I'm pretty sure you're just using this as a weapon to get your view across um you need to get that the googleometer exactly yeah I thought metrics are important to

(18:04) improve software security is it like part of improving your um devops like psychiatrics yeah sorry to cut you off no no but to answer the question as I see it metrics are essential they're not enough uh and as we all know uh if you govern purely by the metrics two things happen one anything that's not in the Matrix you will ignore and two if what you're doing is like a control Loop where you have a little controller you think of yourself as a little controller you've got your sensors which are the metrics coming in and then

(18:43) you've got the actuator which is you doing stuff to the system uh it turns out that if you want to improve the the difference between the Target and what's actually happening now the easiest thing to do is to fiddle with the sensor right it is much easier to to gain the metrics than to actually improve the system so you need to be aware of that and the reason that that's important is that if you tie punishment and reward to metrics they will be immediately gamed within an inch of their life um so those would be the two cautions

(19:13) I'd give about metrics yeah that's very human to like change the change the measuring system that's a really good example of this but they try to incentivize all of the talents to getting everyone to open up bank accounts and instead what they end up finding out was that all of these Terrors tellers on mass we're doing the sort of natural optimization there which is just going okay let's open up lots and lots of accounts for people whether it was a good idea or not yeah oh yeah I remember when I was in um I was in

(19:45) curries it's like electronic store and I was a cashier and I had to really get my metrics up on selling insurance I think on products but I didn't see what the product was and I was like this is my this customer is getting my two cents would you like Insurance on your product and she just looked at me and goes like no it was a vacuum cleaner bag yes we're good yeah you know I asked a question yeah I would say use use metrics to sense the environment but as I said Beware tying punishment reward like if it didn't work for the

(20:25) Soviet Union who had unlimited authority to try to make it work an unlimited supply of men and women with guns and dogs to try and make a metrics governing system work then it's not going to work for you right so yeah so use with caution yeah I think there's a good example so to kind of cut you off here it's like because I get asked this a lot about the big four metrics that came out of the work we did with the Dora folks and that they ran with you know the mean time to recovery change failure rate etc etc

(20:55) deployment frequency and it is horrifying what people out there in the world have done with these metrics they're a sane collection of four metrics that pull in different directions so you can't optimize one too much of the cost of the other but you literally get teams inside Enterprises competing on how to improve all of these things and you know exactly as Jacques was saying like you can improve deployment frequency and mean fake change failure rate by deploying more often and not being as good at measuring

(21:24) it but looking for errors and so you'd get these teams optimizing for one percent two percent three percent improvements in these metrics and sort of losing sight of the biggest picture um but to I guess bring this back to a security lens um the thing I often talk to folks when they're trying to do do devsecops inside organizations at the start of this journey is like how quickly can you push a change to production and know that it's actually gone out because if you can't do that quickly if you can't

(21:52) respond to something push out a fix to it or a change of any kind and know whether it worked or not like that is just the 101 instead of substrate and you can spend all this time optimizing all sorts of other policies and processes but if you can't create change in your environment quickly and reliably and be able to see the results of that change like stop caring about devsecos and all this they just fix all those things first yeah the worst time to find out that you can't deploy to production quickly and safely

(22:23) is in the middle of a security incident or an outage absolutely yeah I'm sure people are found ate that recently with uh log for shell and um on log for Shell is like um do you see critical vulnerabilities and updating your software all the dependencies as a in a having a process for that being really important or um what you yeah so actually on that we have a poll so the question is do you pin your bills or do you update to the latest so this is sort of this question comes up with it's mostly around vulnerabilities well that there's loads

(23:14) of good reasons to update but with respect to security when you um if you update to latest you'll get all the fixes but if you um pin your bills you're not going to be um tricked into updating to a bad version um so and so we see here there's there's most it's kind of half and half but most people prefer to update to the latest so 24 I said by 49 update to latest 20 40 say pin my bills and the rest are it's not important to me so and I I don't really feel like this question is um solved

(23:58) yeah so I in Cloudsmith we always say uh we recommend to pin your bills but like if there's a critical vulnerability it'd be great if you're updated as quickly as possible so I totally see the other side so we like to say pin your bills but then use Tulane like depend about or I think renovate Etc to um to give you a a prompt an alert a PR to with an update to the latest and that will kind of Quicken that cycle so what do you guys think on that topic this one's a bit of a honest this I'll

(24:37) let Jack answer this more in more detail but I'd say at a high level the way I feel that is some of it depends on scale if if you're like two developers who own the whole system that you're in like you know in a very small startup the answer is very different too if you're a multinational bank with regulations and you know hundreds and hundreds of teams interacting with each other I do think you know the big problem with auto updating to latest all the time is when are you creating that artifact that are

(25:04) you testing so like are you creating something that's going to be tested in the test environment are you going to be able to reproduce that artifact again I think there's some Nuance here and it's it involves you know probably doing a mixture of both but choosing when in your software delivery life cycle you do each of those activities the the most depressing answer from experts is there's Nuance um it depends well on the one hand and on the other hand um I I'm broadly in the camp that you should pin your dependencies in source

(25:36) code and update them automatically I don't like mystery dependencies showing up in production without warning and without a record that makes me deeply uncomfortable personally but I recognize that it's a hassle um we are sort of like in I don't know like not quite the pre-history but we're definitely at least no further than the Bronze Age in term of dealing with this stuff um we have technology but it goes blunt easily um and causes a lot of a lot of hassle um and we just need to learn to grow the muscle

(26:11) to do it and that's just going to take a lot of time and be sporadic and uneven um but I do agree with Nigel's point that there's sort of minimum standards of hygiene you need to reach first you need to have good testing in CI in place you need to have smooth the road to production from source code changes those are the same capabilities you will need to automate upgrades I would put nesters here about like the trade-off and risks between waiting to upgrade versus upgrading too soon uh and sonotype have released their

(26:43) eighth state of the supply chain report a few days ago it's worth reading um they do fantastic fantastic research their position is that you should hang back a little you know one or two versions behind the pace or maybe some amount of time I think would be a better way to do it on the theory that if you're right at the bleeding edge you will you will get cut um from time to time and that it's not worth the risk um I'm kind of on the fence about that uh I think that the incidence of a vulnerability existing is far higher

(27:16) than the incidence of a supply chain attack being successful so the balance of risks what about General bugs too because this is the one that always gets me like there's nothing I find more frustrating than if you're developing something using a bunch of libraries or Frameworks and you keep beating your head against the wall going why is this not working it should be working and then you upgrade a dependency and you're like ah it was actually impossible wrong yeah I think there's something to be

(27:40) said to for staying relatives generally leads to a better experience um well it's also because upgrading is is not just like a linear function of the number of things you have to upgrade it's exponential right because there are interactions between the dependencies so the longer you lead it you know like the the larger that sort of Cartesian join of Doom gets uh so you wanna you wanna keep close to the edge if you can at Shopify for example um we have the monolith which is the main application that probably largest

(28:10) rails app in the world and we keep that on Rails Edge once a week once a week we upgrade to what is literally in the main repo of rails like we're not waiting to point releases or anything like that we're keeping up with it because we know that the upgrade pane is just too large if we hang back for a year like it would just just be catastrophic uh and I can I can sort of look back at the earlier history of the company through through documents and and and get get commits and I can see that pain and I can see why we did it

(28:44) yeah I I just set up a call with a customer who's still on red hat four and is I'm likely to ever get off because they they let they left it too long and now they have just specialized little bit of history that they have have to work around yeah muted are you on mute no the webinar guards are against us no still muted [Music] hmm no this is how you know it's live everyone yeah so I think one of the interesting things while while Christian's working out her audio is a lot of this conversation around

(29:38) security issues and software Supply chains it often feels kind of one-sided in terms of companies that are getting an awful lot of software sort of for free from volunteer maintainers who have been you know every time one of these vulnerability comes out it's like everyone has the pitchforks out for the meantime like I'm you know I was doing this out of the goodness of my heart and I was maintaining that stupid backwards compatible feature because you all protested against it I think something has to change about the producer

(30:09) consumer relationship with open source like there's a general assumption that it's software of a certain quality everyone should try and write good software but something feels out of kilter in society about the promises and commitments that people expect there's a there's a really fascinating paper that just uh is is currently in preprint on SSR and the social science research Network it's a pre-print server called tragedy of the digital Commons uh which which is like written for a Law Journal but goes into

(30:40) kind of like the economics of it you know like the law in economics kind of situation and she makes exactly the same point which is that large software companies in particular are free writing off the community uh in a big way her argument is that like the sort of the the ambient costs of security risks should be pushed back onto those companies to Bear because they're the ones who are best able to bear it yeah it's I totally agree yeah big Tech loves open source like sharks love fish you know yeah did you see the I saw can you

(31:15) guys hear me now sorry about that yes yes but um my laptop the battery is gone but anyway I saw that legal letter that wanted the log for Jay uh um maintainer is received and it was just like oh for the love of God like he he's like doing this for free and you're like telling him uh giving him a legal letter to update and like from a company that's using his code for free it's it definitely doesn't sit well doesn't seem morally right or something I mean I'm I'm kind of in an interesting

(31:50) position here because I'm I've been one of the champions for introducing MFA requirements for uh for software repositories you know whether where the authors need to have NFA enabled um because their packages are so widely used and in the sense that's imposing a cost you know it's imposing imposing additional effort on the package maintainers who didn't didn't ask for it right and and I do feel bad about that but I I then have to sort of take the utilitarian stance that the end consumers are far more numerous and for

(32:22) them the consequences are far more serious if if there's a compromise um it's it's a tricky it's a tricky thing but I think the difference there is that like the end consumers can just involve other you know random open source developers who who didn't expect something nasty to come down the pipe as well as the companies who can bear the cost and should contribute back yeah yeah I I saw there's um on Pipi they have some stats on who has converted to 2fa it's it's not like super impressive it's like 20 of people

(32:55) that will eventually be asked be forced to have 2fa have turned on 2fa um I is maybe some of them don't know about us or some of them don't want to do it and they'll just wait till they have to and it won't be a big deal that's that's largely what happened in in Ruby I I know some of those authors because they work at Shopify and they said yeah we agree with logic we're just not going to do it until you make us do it because it's just yeah it's just work right it's an additional thing to do

(33:24) yeah yeah and what about like I've seen I saw a list of um things that maybe open source maintainers can do to be more secure but it was like a lot of stuff for someone to do it was like um uh add scorecards to their repo um uh there was like there was just a ton of stuff to oh do a course like I just can't imagine if you're doing this in your spare time that like a lot of people are gonna do it especially when people often got into this because it was fun you know like hey I solved a problem in a fun

(34:02) interesting way and I want to share that with the world and I think I don't know if our software licenses are the ways or some kind of opt-in system but I feel like there's got to be a way to distinguish between hey everyone here's something fun and cool have at it and I am deliberately building something that I would like to be part of a bigger structure and a bigger ecosystem um and I think that's sort of the constant trade-off you don't want to you don't want to stifle people just sharing

(34:27) code that is useful and fun but there's going to be some Declaration of intent from there I think the sort of the the coordination point or choke point depending how you look at it is probably going to be the software repositories because they can set the terms yeah under which they agree to distribute the software um and so if you you don't like those terms you are within your rights to take their software which is open source into running yourself yeah within your rights to just distribute Source uh from a website that

(34:58) you own like there's Alternatives like they're not as convenient right they aren't but you know that's that's the trade-off yeah I think that's yeah and it's similar to all of this reminds me of I was a Debian maintainer back in the day when um you know you were sort of in one of two big Linux camps and I was quite shocked when I sort of moved to that point level of suddenly having all of these security processes enforced on me but it was the right thing to do because that was the Distribution Center you

(35:26) know volunteers oh and like so a lot of these things were Debbie and Community already had a lot of these uh yeah I mean I think you know as as much as you know I hate to you know particularly towards the end up to claim the death of the operating system distribution a lot of these problems I think have been solved in smaller communities before we're just now dealing with them happening faster in a bit bigger scale and you know in in Tech I feel like we love nothing more than to ignore the discoveries of the past

(35:59) yeah and so um what else they say so what do you think there are the biggest challenges in software security or is it like we've been talking about how there's just so many challenges and it's just all of them together but if you're gonna give yourself a top one or two what would be your favorites well that's tough this this goes back to that earlier discussion about culture versus practices uh there's there's this vast amount of latent risk out there and we've just got to sort of chip away at

(36:35) at everything that gives right we're pushing in every direction at once and anything that gives we push harder because we're getting some progress out of it we're retiring some risk from it um building up those layers of security building it up and and you know reducing the net risk for everybody which is which is the sort of the goal you know that there's that problem that uh open source is basically are comments right like it's it's a kind of a resource that you can't exclude people from using but

(37:06) where if lots of people use it then that puts pressure on the maintainers it's Rivals as economists call it and that's Commons and they're difficult to govern they're difficult to manage because you know everybody's an individual they've got different incentives uh to to be selfish and the difficulty is finding those well-positioned parties to be involved so to their credit I know we bashed up big companies but to the credit a lot of them are coming to the table uh or trying through the open source

(37:33) security Foundation which I participate in open ssf so you've got your Googles and your Microsoft's and your Amazons and and a whole bunch of companies uh participating uh contributing money contributing uh folks time trying to sort of attack this on all France um yeah the trick is is going to be like to your point you're like will it just seem like a loud crescendo to open source maintain is like here's a massive list of things that we can offer you where do I start like um I is that the 10 point Mobility
(38:09) plan is is uh part of the open ssf uh um way to secure open source and do you feel like open source is one of the most important things to secure when we're talking about software in general oh yeah yeah it depending who you ask it's it's um is the sky blue only on sunny days um yeah it's it's everywhere now it's in pacemakers it's in nuclear power plants like there there isn't there isn't a single critical or high you know High consequence piece of infrastructure uh where the social or technical that

(38:53) doesn't rely on it um we we have to like it's it's the soft underbelly of of the whole of the uh social economic system at the moment and what do you think about regulation because I know the U.S federal government is bringing in some new rules about s-bombs and even vulnerabilities and do you see that as a way to improve security of a product strictly yes um this this is a good example of that argument from tragedy of the digital Commons article that the cost should be pushed onto the large companies that

(39:33) currently free ride and have the resources to not free ride um and the US government is in a great position because it's the single largest page tourist software in the world to push push those standards down and to make them common and once they become common then other consumers from those companies will say well you already have that capability I demanded also and that creates a sort of a flywheel effect um but in terms of Regulation of Open Source software itself outside of those big companies like your

(40:04) regular maintainer at home on a weekend uh dear God no um that that would that would kill the Golden Goose but not before the goose you know uh defecated all over the bed hey Nigel what do you think well I think what is a rather hairy thread to mix metaphor or something we don't value maintenance enough in society and I think this is sort of part of the problem that you know and this is why I think right to repair movements and all of these things are so important that you know you work in Lots we have a culture in software development that I

(40:42) think reflects Society in general at the moment which is it is considered better to launch new things than to iterate on existing things and the job of maintainers everywhere is to iterate on the existing things and I think the healthiest software engineering environments I've ever worked in have been the ones where they really senior folks are sort of proclaimed you know people lauded look at systems make small incremental changes to them over time keep them going in the right direction and that that's recognized as valuable and I

(41:15) think this this is sort of the whole problem maintainers anywhere near enough and so they feel at the end of a supply chain when really should be going you know no you're a critical part of this whole process you know if I could wait any wand it would be around us valuing the act of Maintenance more so that big companies did want to participate in it so that they you know reached out to maintain projects with respect you know I think Google does a reasonably good job of this like we we've had Google for each

(41:47) other security vulnerability it's something you you ship you know we've seen some of our users have it you know they basically wield a big stick and go if you don't do something about this in 30 days or 60 days or whatever we'll just we'll shout it from the rooftops and they can because they Googled it I think there are ways to do that sort of sort of encourage people to do the right thing but fundamentally we've got to Value the act and process of Maintenance more everywhere and she I wonder if like

(42:16) um government Fontaine could help I know like obviously the 10 point Mobility plan should improve security and that's using money I know um maybe to I there was talk about resetting putting funding towards resetting to fa shark and public repositories um but do you see it do you think that maintainers could get paid for improving security of product obviously selective products like uh that are used in critical systems like do you think that would that's a solution or is that just it's just it's not it's not gonna it's

(42:57) not maintainable in the long run I'm concerned that it goes back to that problem of metrics um that that it will you know the incentive is just to do what what the funder says and that will attract people you know like the the story of um the the British trying to get rid of cobras in India and they pay people to bring in Cobra heads and people to start up breeding cobras uh or something similar where there's gun buybacks and people are just 3D printing guns on mass and bringing in boxes of 3D printed guns and making

(43:28) money that way um I'm concerned about that I think where government has a role in terms of funding at least would be on what you might think of as sustainment activities so things like uh subsidizing or fully funding training right making it freely available to as many people as possible encouraging colleges and universities to pick it up as part of their curricula um things like uh you know shared resources for software repositories uh shared resources for open source projects that need you know Security

(44:05) review a lot of things that the open ssf is already doing um can definitely be scaled up with Government funding what do you think about punitive approaches too like this is something I'm always curious about because it feels like most of the huge companies that suffer data breaches that were honestly pretty derelict in terms of oh yes they just haven't been punished like it's all by governments and yeah why would you invest in security when it doesn't actually matter yeah I I I'm I'm a bit of a you know like I consider

(44:37) myself a Centrist I used to be a Libertarian but I'm about to sound like a raving Looney Lefty because I I think there is far too many things in um corporate malfeasance in which the punishment is a fine whereas it should be criminal time for the executives who authorized or who failed to authorize you know some activity um yeah because that's the only thing that actually gets their attention if you get fined it doesn't fall on the people who made the decision it falls on the shareholders exactly people the

(45:07) people people right Optus is a good one yes like you have a company that literally litigated you know lit pressure lobbied the government to make sure companies weren't accountable in these sort of situations and then now all of these millions of people have surprised including me my my passport number right yes yeah I actually I was listening to the the security weekly podcasts and at the end of it they talked about insurance as a way to um uh to drive companies to to do more to be better at security and it can be a

(45:43) more effective way than um compliance so that like when you when you have a data breach uh and you realize you're not insured and you have to pay a lot of money to maybe for um on on people so suing you or even to get back to where you were if you've lost data that that is a quite an effective way why not why not both um you know we we have punishments for people who you know like if you don't do fire safety in your factory right not only do you mess up your insurance and not only can you face fines but the

(46:21) people who are responsible are criminally malfeasant they can go to jail for neglecting fire safety um you know the consequences of of data breaches uh dire uh the consequences of lackadaisical security are only going to grow worse as time goes on and as as all matter somehow becomes programmable basically then this stuff really matters and I think this argument that like oh but the corporate veil is sacrosanct it's just like the corporate veil is there to deal with um you know questions of who owes debt to whom like who can be who who who is

(46:56) liable for how much it didn't give you like a magical get out of jail car that was never the idea so as I said I sound like you're raving Mooney on this point because I'm so frustrated by companies that walk away with a fine and the executives are still there right they don't get sacked they just go like oh well that's the cost of doing business and that to me is psychotic yeah I mean you know yeah there's not enough accountability at the cover level absolutely with you on that one we could

(47:21) rise up and smash the systems right loads of all countries yeah on that I think we're gonna announce our prize Hillary do you wanna now that we've done the rally yeah no we've gotten to where we were so the prizes are announced they're in the chat we have who gets a free lunch I hope everybody enjoyed um our talk today I loved it I'm so sorry about my uh my speaker issues it happens you guys were such Pros you continued on the conversation another way to put it is that we talk too much [Laughter] and thank you for being such a wonderful guest shark and Nigel it was like really nice to talk to you so it's bye from our

(48:45) guests you guys can say bye all right thanks for having us and it's by from me so thanks everybody for joining we'll see you at the next monthly Cloudsmith webinar

Get our next blog straight to your inbox