Webinar

DevOps Debriefs: How The Financial Times Beat Leaks with OIDC

  • Apr 16 2024
  • 30 mins
  • Securing your organizations pipelines, OIDC authentication

Things you’ll learn

  • Securing your organization's pipelines
  • OIDC authentication
  • Say goodbye to long running tokens

Speakers

Rob Godfrey
Rob Godfrey
Senior Technical ArchitectFinancial Times
Ciara Carey
Ciara Carey
Sales EngineerCloudsmith

Summary

In our first episode of DevOps Debriefs, join Cloudsmith and special guest Rob Godfrey, Senior Technical Architect at the Financial Times (FT) for a discussion on the crucial role of authentication and credential management in ensuring software pipeline security. We’ll discuss: Innovative strategies that empowered the Financial Times team to overcome software supply chain risks in their pipelines. How the team responded to the fallout of the CircleCI breach. Insights into the challenges and triumphs as the Financial Times fortified its pipelines against potential risks. The pivotal role of Cloudsmith in supporting FT's adoption of OIDC and providing comprehensive visibility into their entire software supply chain.

Transcript

  1. 00:00:00
    Ciara Carey
    Hi, I'm Ciara Carey, and welcome to Cloudsmith's webinar on all things supply chain security and package management. Cloudsmith is your cloud native, universal, artifact management platform. We integrate with all your CI/CD, and we support all your formats, from your Docker images to your packages. Rust crates to your NPM today we're going to be talking about how to securely connect to your SaaS build tool in your build pipeline
  2. 00:00:26
    Ciara Carey
    often these connections involve long running API tokens stored in your CI/CD environment, such as your GitHub Actions or your CircleCI. A classic use case might be you have your CI/CD tools, they need access to cloud services, your AWS, your Cloudsmith, to publish artifacts, to deploy software, access resources in their cloud provider.
  3. 00:00:50
    Ciara Carey
    And the classic way to do this is to create a user in your cloud service. To generate a secret access key. And then to have [00:01:00] this as your long lived credentials. These credentials need to be stored in your access from your pipeline. They're probably stored in some secret store on your CI/CD. And the problem with this is they tend to have very extensive permissions.
  4. 00:01:15
    Ciara Carey
    Cause they need to create, your CI/CD needs to create, delete, update resources in your source code, in your infrastructure, in your artifact management accounts. And unfortunately, these credentials can be leaked. It's hard to detect when this happened. And it's a, it's a big process to rotate these keys. The issue with the workflow above is that the API, if the API key is stolen, there's, and then there's a risk of unauthorized access and attacker with these credentials can impersonate a user and execute malicious code.
  5. 00:01:50
    Ciara Carey
    A better alternative is to use. OpenID Connect OIDC and this enables short lived authentication tokens, helping you secure your [00:02:00] pipelines against the exposure of credentials. So providers can so providers can not support this, can improve the security of your supply chain overall. So today we are joined by Rob Godfrey.
  6. 00:02:15
    Ciara Carey
    He's a senior technical architect at Financial Times, and the Financial Times is one of the world's leading business organizations. I'm going to bring Rob on stage now. Hi Rob, how are you?
  7. 00:02:28
    Rob Godfrey
    I'm very good, how are you?
  8. 00:02:29
    Ciara Carey
    Hey do you want to tell us about your role in Financial Times?
  9. 00:02:33
    Rob Godfrey
    Yeah I've been at the Financial Times for a little over a decade now, so I've done a few things there, but my current role is I look after two kind of areas.
  10. 00:02:43
    Rob Godfrey
    So one of my teams that I work with, we look after our cloud platforms, not cloud estate helping developers use AWS and Heroku and other tools effectively and also securely. And then the other team I work with look after our developer tooling. So things like GitHub and CircleCI [00:03:00] and Cloudsmith and a few other things as well.
  11. 00:03:03
    Rob Godfrey
    So yeah.
  12. 00:03:03
    Ciara Carey
    Yeah. So I brought you on today because I know early in 2023, you got like a notice from CircleCI to say, That you're you need to rotate and revoke your keys, which I'm sure was an easy task. Oh, we've
  13. 00:03:18
    Rob Godfrey
    got thunder here.
  14. 00:03:20
    Ciara Carey
    That is something. I hope
  15. 00:03:23
    Rob Godfrey
    the internet lasts.
  16. 00:03:26
    Ciara Carey
    It was like foreboding. But yeah, so you got that action required notice from CircleCI.
  17. 00:03:35
    Ciara Carey
    And what was the first reaction?
  18. 00:03:38
    Rob Godfrey
    Yeah, I think it was, I think my second day back after. New years and yeah waiting in my inbox when I kind of sort of started the day was this email from circle ci saying yeah, yeah an advisory to recommend that we Rotate all our secrets that we stored in circle ci pipelines and it was like Okay, so that's sort of I think my plans for [00:04:00] january kind of Disappeared out the window at that point because we use CircleCI extensively throughout our organization.
  19. 00:04:06
    Rob Godfrey
    We have well over a thousand pipelines running in CircleCI and there's a lot of environment variables that we had to inspect basically in triage and then there were a lot of secrets. That's where we were potentially affected.
  20. 00:04:23
    Ciara Carey
    Can you give us an idea of what is a lot of secrets? Is it like a hundred?
  21. 00:04:29
    Rob Godfrey
    Well, we started, we didn't quite know to begin with. And so with, you know, we kind of used the APIs that CircleCI provide to enumerate all the projects and then enumerate the secrets. And we ended up with well over 14, 000 environment variables
  22. 00:04:45
    Ciara Carey
    that
  23. 00:04:46
    Rob Godfrey
    we were contending with. And then after some triaging where we did a bunch of pattern matching on, you know, we looked for AWS access key and, you know, other things like vault token and, you know, other, yeah, there's a big list of these [00:05:00] things that we went searching for.
  24. 00:05:01
    Rob Godfrey
    And then a bunch of human effort as well. We got that down. We reckon there was well over sort of 5, 000 secrets. that were affected and would need to be rotated. So this was quite a big deal. We had to coordinate a response across, probably dragged in 30 to 40 engineers to work on this problem for several weeks.
  25. 00:05:23
    Ciara Carey
    So it
  26. 00:05:23
    Rob Godfrey
    was a big deal at the time. And yeah, and yeah, it required a lot of coordination, a lot of effort from the teams, and yeah, the teams worked really hard. It was quite interesting work as well. So it took several, you know, we identified the critical and high stuff first. And that they were rotated fairly quickly, but there was quite a long tail of other things that we were less concerned about, but still needed to be rotated.
  27. 00:05:47
    Rob Godfrey
    And so, yeah, it took us, took us several weeks to, to get through that, that sets of, you know, thousands of secrets that needed to be rotated. So big deal.
  28. 00:05:56
    Ciara Carey
    Yeah, no mess. And, and so like the [00:06:00] immediate steps you took, obviously you were talking about them. So identifying. the most critical tokens. Is that the first thing that you did?
  29. 00:06:08
    Rob Godfrey
    Yeah, we had to yeah, do a bunch of, you know, the things we knew. So we were kind of looking for things like, okay, are there secrets that give you access to secret stores? Are there secrets that give you access to data and particularly sensitive data or PII, that sort of thing? Are there things that allow, you know, changes to infrastructure.
  30. 00:06:27
    Rob Godfrey
    So there were certain classes that we cared more about and there were other things like, you know, there were certain, I don't know, maybe sort of static analysis tool, sort of test coverage tools that we were kind of less concerned about. You know, there were kind of hoovering data out of our systems, but you couldn't really do much with that data that we were too worried about.
  31. 00:06:47
    Rob Godfrey
    So there were kind of definitely different categories of secrets. So we had to kind of, yes, assign a severity and then prioritize All the critical and kind of high severity things and get those rotated super quickly. [00:07:00]
  32. 00:07:01
    Ciara Carey
    And so did you use this as like a, I, well, I know the answer, but use this as an opportunity to actually improve the security of your build pipelines and your supply chain security overall?
  33. 00:07:12
    Rob Godfrey
    Yeah definitely. I think, you know, never, never let a good crisis go to waste as it were. So so yeah, we, we kind of looked at a few things. So yeah, once, once you've The dust had settled. We sort of did, you know, sort of retrospective type stuff and kind of focus on a few areas. And so one of them was why, why do we have so many secrets?
  34. 00:07:30
    Rob Godfrey
    You know, that's a big number, you know, with a lot of pipelines, you'd get a lot of secrets. But we were kind of, we'd already kind of started the journey of using OIDC with AWS.
  35. 00:07:39
    Ciara Carey
    Okay.
  36. 00:07:40
    Rob Godfrey
    But we hadn't. use that exclusively everywhere. So focusing on these kind of using short lived ephemeral secrets wherever we can was kind of top of the list because it obviates the need to manage that secret within a pipeline, right?
  37. 00:07:56
    Rob Godfrey
    So, and then you don't need to manage it in a secrets manager and then there's a whole bunch of [00:08:00] complexities sidestepped by just using short lived secrets. We also then Decided that we're rather than having secrets sort of sat in environment variables within pipelines, we'd pull secrets from the secrets manager as part of the pipeline execution.
  38. 00:08:17
    Rob Godfrey
    So they then didn't live for a long time in the CircleCI environment. They were just there during the pipeline. a pipeline's execution, so
  39. 00:08:26
    Ciara Carey
    you've got to
  40. 00:08:26
    Rob Godfrey
    minimize the time those secrets were kind of available within the build environment. And then thirdly, we looked at, you know, can we automate secret rotation wherever possible?
  41. 00:08:38
    Rob Godfrey
    Because, you know, manually rotating secrets is toil and drudge work. And, you know, it actually, you know, it's really hard to do that for weeks on end without making mistakes.
  42. 00:08:50
    Ciara Carey
    Yeah, and it's hard to keep the morale up and for weeks doing that kind of thing.
  43. 00:08:54
    Rob Godfrey
    Yeah, definitely.
  44. 00:08:56
    Ciara Carey
    And does CircleCI provide with like extra tools to [00:09:00] help their customers during this time?
  45. 00:09:02
    Rob Godfrey
    They did actually, they did a fairly good job. So they reacted pretty well. So, you know, I think we kind of realized that, you know, this isn't the first time we've had this sort of, a sort of a secret leak. We've, we've seen similar things with other suppliers as well. And so they provided some tools to help enumerate secrets.
  46. 00:09:20
    Rob Godfrey
    They provided one thing that was missing that would was really useful when they added it was adding timestamps to the last change timestamps for environment variables. So you could tell if someone had changed the value of an environment variable, which kind of gave us an indication that yes, this had been dealt with as a secret had been rotated.
  47. 00:09:41
    Ciara Carey
    Oh, yeah. And so is that in the audit log or is it in the settings or something? Where did you?
  48. 00:09:47
    Rob Godfrey
    Yeah, I can't remember. I think they just, I think initially it was part of an API and then they exposed it maybe through the UI as well. So you could, yeah, so we could get the data that said, yeah, these are the things that haven't been rotated.
  49. 00:09:58
    Rob Godfrey
    Because we already had rotation in place for [00:10:00] quite a lot of things, but there were definitely things that weren't. Weren't in, you know, being rotated as perhaps they should have been according to our internal policies.
  50. 00:10:08
    Ciara Carey
    Oh, yeah Something like a crisis to make you realize this is not a drill And was there any When you were adopting OIDC, where available, I know not all built tools support it was there were there was it difficult or was it challenging at all to get it?
  51. 00:10:28
    Ciara Carey
    Was it just Like to get it All your configurations updated. Was it like a difficult process?
  52. 00:10:35
    Rob Godfrey
    There is a bit of complexity in getting the sort of essentially the trust relationship between In our case, there was CircleCI in AWS and CircleCI in Cloudsmith So there is some configuration to get that set up.
  53. 00:10:46
    Rob Godfrey
    But once that's set up It's then really focused on how do you make the developer? experience really simple. So Cloudsmith you provide an endpoint in your API, but even that we felt was too much. We [00:11:00] wanted a couple of lines in the CircleCI configuration to essentially give us this ephemeral token rather than invoke an API endpoint, get, get some, you know, parse the status code, parse the token out and kind of make it available.
  54. 00:11:14
    Rob Godfrey
    So we wrapped the API call in a, in a what's called a CircleCI orb, a bit of reusable. Pipeline configuration. So basically when an engineer wants to get a Cloudsmith OIDC token, it's sort of two lines in there.
  55. 00:11:31
    Ciara Carey
    Oh, cool.
  56. 00:11:32
    Rob Godfrey
    CircleCI sort of configuration for that pipeline.
  57. 00:11:35
    Ciara Carey
    Oh, that's cool. And so when you're, When you're doing this at scale, is it, is it like actually handy enough once you've already rotated all the tokens, you've tried to minimize the amount of tokens used or access.
  58. 00:11:51
    Ciara Carey
    And then, and then when, when you're adding OIDC, I suppose, It's, you've done all the hard work in doing all [00:12:00] the rotation already, so it's not such a hurdle to add this in, especially when you know this will stop this happening again, this, if a breach does happen, you're like,
  59. 00:12:12
    Rob Godfrey
    well, I think the nice thing about using OIDC is You, it's just a lot simpler, right?
  60. 00:12:17
    Rob Godfrey
    So with a short-lived credential, you a don't need to store it in a Secrets Manager. You then don't need to really track it and say who, who's gonna own this secret and be responsible for it. You then don't need to rotate it ever, because it's only gonna live for an hour. or so, you know, minutes or hours rather than weeks and months.
  61. 00:12:38
    Rob Godfrey
    So you kind of just, you completely sidestep a whole bunch of complexity. So that's why, you know, where OIDC is available to us, we will use that in a heartbeat over kind of long live credentials now.
  62. 00:12:51
    Ciara Carey
    Oh, absolutely. And is there some tools that you kind of wish were supported? OIDC, is there?
  63. 00:12:57
    Rob Godfrey
    Well, if you just think of the moving parts in your [00:13:00] pipeline, yeah, you've kind of, it would be great if, you know, everything, you know, kind of hooked together with OIDC within the pipeline.
  64. 00:13:07
    Rob Godfrey
    So then all those things didn't know. A challenging bit and one thing we're looking at internally is whether internal kind of APIs could be enabled for some sort of OIDC integration, but that's going to be a custom built kind of Yeah, I
  65. 00:13:26
    Ciara Carey
    know OIDC has this idea of the provider and the service. So in that scenario, who would, who's the provider?
  66. 00:13:35
    Ciara Carey
    I suppose you're just using your whatever the Google or AWS credentials. Is that what it is?
  67. 00:13:42
    Rob Godfrey
    Yeah. So you've got the. CircleCI acts as the sort of, it will pass information about the pipeline and the organization and the various bits and pieces of information about the job execution and then it's up to the service essentially to [00:14:00] evaluate whether it's that information looks okay and then issue a short lived credential based on the information it provides and the trust relationship it has.
  68. 00:14:10
    Rob Godfrey
    So yeah, we're kind of looking to see if we can use kind of AWS's OIDC capabilities to essentially provide a shim to other services. Oh brilliant, for
  69. 00:14:21
    Ciara Carey
    internal stuff as well.
  70. 00:14:23
    Rob Godfrey
    Yeah, so you can lean on sort of IAM credentials and their their sort of new short lived role base type things.
  71. 00:14:31
    Ciara Carey
    Yeah, and for for services that don't currently support OIDC, I think you you have a kind of a way to make that a little bit more secure as well and make revoking and rotating API tokens a bit easier.
  72. 00:14:45
    Rob Godfrey
    Yeah, so one of the areas we looked at was how do we do sort of automated key rotation. And we, we, Use a top tool called Doppler and we've brought that tool in because it has integrations with various other bits of our estate so AWS [00:15:00] and Heroic, Q Circle, CI, GitHub and so you can basically Take a secret publish it into Doppler and then Doppler will know where to then push that secret To if that's how you want to set things up So you can say, okay, when I'm rotating a secret, you can basically go into Doppler and say, rotate this secret.
  73. 00:15:19
    Rob Godfrey
    It can go off to AWS, say, generate a new user credential, and then push that out to CircleCI or wherever it needs to go. So it's kind of, it's taking all the, There's complexity and the toil away, I guess, is what it's trying to say. Yeah,
  74. 00:15:35
    Ciara Carey
    is that where you're always, do you find that security and simplicity just go really well together?
  75. 00:15:41
    Ciara Carey
    That when you remove that complexity, that it just generally is more secure?
  76. 00:15:48
    Rob Godfrey
    It doesn't have to be security. If you kind of, if you want to kind of make, you know, nudge an organization in a certain way, yeah, make it really simple. And, you know, so in this case, yes. It applies to security, [00:16:00] but there are lots of other things that we do similar things with cost optimization and things like that.
  77. 00:16:05
    Rob Godfrey
    So kind of making that the sort of preferred choice, the easy choice.
  78. 00:16:10
    Ciara Carey
    Yes.
  79. 00:16:10
    Rob Godfrey
    Yeah.
  80. 00:16:11
    Rob Godfrey
    If it's hard, you're not going to get any traction. That's the reality.
  81. 00:16:15
    Ciara Carey
    Absolutely. Yeah. So and so are there any after this breach and how it. It took so many resources of the financial times for weeks on end. What, was there any extra tooling and infrastructure that you brought in besides Doppler and OIDC?
  82. 00:16:32
    Ciara Carey
    Or training or anything like that? I
  83. 00:16:35
    Rob Godfrey
    think the One thing we did have, we have an internal sort of inventory of our infrastructure estate. And so some of the things we did to augment that was to actually expose metadata about secrets into that as well. So that gave us a way to, and this, this kind of inventory, it's basically a big graph database and it connects together our systems, the people that operate those systems, the [00:17:00] infrastructure components that make a kind of part of that system.
  84. 00:17:02
    Rob Godfrey
    And we didn't have secrets in this time last year, But now we do. So we can basically link together, okay, this, this, if we know this secret's been affected, we can easily work out, okay, well, we need to this team over here about that. So, and that's across all our different platforms. So we, we have little bots or agents that kind of, kind of go into AWS and put out metadata Heroku or, you know, CircleCI or GitHub and kind of metadata about our secrets and also things like SSH keys and certificates.
  85. 00:17:38
    Rob Godfrey
    And then once we have all that, it makes it really easy to sort of start to go. There's, there's issues over here that we need to go and look at.
  86. 00:17:45
    Ciara Carey
    And so do you have any advice to give, give give other companies that experienced a breach?
  87. 00:17:53
    Rob Godfrey
    Yeah, I think it's worth preparing. for it. So at least, at least kind of run a thought experiment where you kind of think, [00:18:00] okay, what would happen if our secrets in some, one of our vendors were, were, were leaked, how would we react?
  88. 00:18:08
    Rob Godfrey
    So, and a part of that, I think what we worked out was we kind of need to know where our secrets are. What secrets do we have? Who's responsible for them? So then you got kind of a, you know, because one of the one of the challenge we had actually was we had secrets when we're trying to rotate that were part of systems that the the responsible person had maybe left the company.
  89. 00:18:29
    Rob Godfrey
    And so we now couldn't get into the You know an external system that issued the secret so we had to start raising support cases in certain Instances to kind of get access to to rotate secrets, which is never a great thing But yeah, that's these sorts of things happen when you know people leave organizations, unfortunately
  90. 00:18:48
    Ciara Carey
    You were talking about this graph Catalog before is that is that how you addressed that issue?
  91. 00:18:55
    Rob Godfrey
    Yeah Well, we can certainly spot things when we know if people have left the [00:19:00] organization for example so you can kind of quite quickly see that Are there kind of areas of our estate where we need to maybe reassign ownership of stuff and things like that? So
  92. 00:19:10
    Ciara Carey
    yeah, and even what you're saying there kind of reminds me of like the problem of like supply chain security where you need to know a big problem is when a vulnerability comes in You need to know actually what's in your software.
  93. 00:19:22
    Ciara Carey
    So it's like the same problem, basically. Knowing what you have is you're, and being able to audit that gives you a step up.
  94. 00:19:32
    Rob Godfrey
    Yeah. And certainly things like knowing how to rotate, like rotation procedures, because you have all sorts of different secrets. And from different sort of source systems, you know, you kind of got to have that documented somewhere and so yeah, and then if you can automate those processes as well.
  95. 00:19:48
    Rob Godfrey
    So yeah, it's yeah, certainly when you kind of got lots of interconnectivity between lots of SaaS providers, it gets quite complicated.
  96. 00:19:56
    Ciara Carey
    Yeah, it's like so simplicity, and knowing what you have and who [00:20:00] owns it is word for word. Would help
  97. 00:20:02
    Rob Godfrey
    Yeah,
  98. 00:20:05
    Ciara Carey
    i'll just see if there's any questions. Oh, there's something in the chat Oh christopher can hear us both.
  99. 00:20:10
    Ciara Carey
    Thank you. Christopher. Maybe it was just me then
  100. 00:20:13
    Ciara Carey
    And does anybody from that's joining us today have any other questions on how to react to a breach or OIDC We'd love to hear from you.
  101. 00:20:23
    Ciara Carey
    Yeah, so so no questions so far, but looking ahead. Is there any other? Strategies that you'd like to adopt in order to improve the Your supply chain security your your build tool security.
  102. 00:20:37
    Rob Godfrey
    Yeah, I mean we're definitely focused on package management So we became a Cloudsmith customer last year. So We recognized that our approach to package management needed some updating should we say?
  103. 00:20:51
    Rob Godfrey
    And yeah, we're kind of in the process of progressively rolling out Cloudsmith is our kind of universal package manager [00:21:00] for different ecosystems. So we've done about five language ecosystems now. We'll do a couple more this quarter and continue rolling them out throughout the organization.
  104. 00:21:09
    Ciara Carey
    So yeah, that's,
  105. 00:21:10
    Rob Godfrey
    that's the main focus from that sort of the build tooling.
  106. 00:21:15
    Rob Godfrey
    Yeah, that's, that's
  107. 00:21:16
    Ciara Carey
    good. And do you think like this, this whole incident, do you think has made you more aware of other issues that were maybe Possibly in your build pipeline to kind of sit up and take note. Not that you didn't do that before, I'm sure.
  108. 00:21:31
    Rob Godfrey
    I think what it has done is basically it's sort of burst the bubble that you kind of, you know, I think when you came, we use lots of SAS vendors, right?
  109. 00:21:41
    Rob Godfrey
    And so you kind of expect them to manage stuff securely. And then I think when these sorts of things happen, and certainly happened to a few of your vendors, You kind of sort of take stock and think, well, it could probably happen to any of our vendors at some point. So we need to be somewhat prepared for that eventuality and [00:22:00] so that we can kind of respond appropriately.
  110. 00:22:02
    Ciara Carey
    Yeah. So having like, yeah, having sort of like a plan to if something happened and not being shocked and kind of left standing still.
  111. 00:22:11
    Rob Godfrey
    Yeah, definitely. And so, yeah, things like getting insight into, as I say, secrets now, we kind of prioritize that. So we kind of at least know where secrets are maintained and managed and that sort of thing really helps or will help, we hope, with anything in the future.
  112. 00:22:27
    Ciara Carey
    Great. So thanks so much. Rob for today. Really appreciate it. And it was great talking to you about this, because I'm sure this can happen to any company, basically. We're nearly well, all the companies we deal with, they have a lot of, we're a SaaS tool that will probably have more than one SaaS tool in their bill pipeline.
  113. 00:22:46
    Ciara Carey
    And this will be front of mind for them. And like, maybe it was it's people use this as a time to reevaluate the tooling they are actually using in their bill pipeline. Cool. So [00:23:00] yeah, thank you so much Rob for today and so our next webinar will be next month and I will let you know the details of that and we'll, we'll talk to everybody soon.
  114. 00:23:10
    Ciara Carey
    So thanks again for coming and I'll talk to you soon. Bye bye. Bye thanks.

Comments