100-Year-Old Fitch Ratings Upgrades to DevSecOps – InApps 2022

When Mir Ali joined Fitch Ratings in 2015, there was already a lot of talk about changing their software processes, driven by the unreliability of their stack.  Fitch Ratings is a financial company that, along with Moody’s and Standard & Poor’s, is one of the three nationally-recognized statistical rating organizations designated by the U.S. Securities and Exchange Commission (SEC). 

Making the change to DevSecOps in a 100-year-old company is not easy, noted Fitch, who is now the director of head of shared services. But the problems could no longer be ignored. Basically, said Ali, they didn’t understand what was going on in production. There were too many outages, there was no traceability, and not a lot of collaboration across the pipeline. When an incident happened, nobody knew why or what to do about it.  Standard procedure was to just reboot the database service.  This was obviously not acceptable in a company monitored by the SEC.

Top Challenges

One challenge was a general lack of security knowledge. Next was the lack of collaboration or understanding of individual responsibility. Last was the inadequate automation.

At first, there were so many things to do it was hard to get prioritized. The teams took a step back and created a long view of what they wanted, and that vision informed all of their subsequent decisions.

They decided to push the security defects to the top of the list. The remaining tasks fell into three areas. They wanted to apply that security throughout the development process, and standardize that process across the entire company. Lastly, they wanted to simplify the process as much as possible.

Simply put, Ali said, DevSecOps is expanding the DevOps collaboration to include Security.

“The purpose and intent of DevSecOps is to build on the mindset that ‘everyone is responsible for security’ with the goal of safely distributing security decisions at speed and scale to those who hold the highest level of context without sacrificing the safety required.” — devsecops.org

Applying Security Throughout the Development Process

It was key, said Ali, to integrate security into the DevOps pipeline so it was an integral part of the entire software development process.

Security Checkpoints in Fitch’s DevSecOps Pipeline

The management team started by seeding security experts onto each of the teams as they broke down the silos across the DevOps pipeline. “Make them feel part of the team,” he said, “take their input, use their skills. Make it part of the automation.” The security experts give you context, provide expertise, what is the latest risk, what do we need to watch out for? Security became everybody’s job.

Next, they put security tools in place. “It’s not all about tools,” said Ali, “but you do have to have tools in place to make security possible.” For example, good dashboarding is key along with automation. The top priorities for the security of the continuous delivery pipeline were proper access control, locking the build system down, and standardizing naming conventions.

They started by going after low-hanging fruit. Yahoo’s data breach was caused because its system didn’t have password protection in place, so someone simply cloned the database. That should not be possible by anyone without the highest security level, he said. So the first step was defaulting access to none. Permissions have to be granted, and are entirely role-based and automated.

“Encrypt everything” became a new motto. Visibility into the pipeline was another critical area. “Make sure that logging entries make sense to humans,” Ali suggested. A lot of logs read like gibberish, which is unhelpful when troubleshooting.

Establish Security Checkpoints

Once basic overall security was in place, they started building automated security checkpoints into the pipeline.  “An engineer will turn things off if it’s not passing, saying ‘I’ll come back to it later’ but they never do,” he said.  By automating the checkpoints, you don’t give them that chance. They use Gauntlt, which allows the user to write automated testing specifically for security.

The Fitch DevSecOps Pipeline

When Ail arrived, engineers had been putting solutions to scaling the data in place, but each team was working independently, so there were a variety of solutions in place and engineers were suggesting tons of different options. It took a while but they picked the best solution for each piece of the pipeline for their needs and made it standard.

Similar to the PagerDuty’s own DevOps transformation, the project has taken about three years.

The Myths of DevSecOps

There are a few myths surrounding DevSecOps that need to be dispelled, said Ali. The first is the myth that if you automate security you will give up control. The truth is that it will make your company more compliant. The automation provides consistency and traceability.

The next myth is that simply adding security tools creates DevSecOps. It’s not about adding new tools, nor adding more developers. It’s not a capability, it’s a mindset, he said. “You can easily do this with the developers you have.” Make security part of their job and part of their performance review.”

Where to Start

Still overwhelmed? Start small. Ali started with email integration.  Slowly they added tools and started breaking down silos and changing expectations. After the email integration, the company started using PagerDuty‘s alerting services, which led to the integration of Atlassian’s Jira bug tracking system and the whole pipeline being centralized and having logging and intelligent monitoring.  

Three years later, they are now capturing things before they fail.

Biggest Challenge for Developers 

Have to put yourself in their shoes. If you tell them, you have to automate, you have to use DevOps tools, and now I’m saying ‘build in security as well’, from their perspective, all of a sudden I have three jobs instead of one.

“Given this, I would hate my job at that point,” said Ali. So that’s where I came and actively provide coaching evangelization and support. He told his team they would provide training, tools and frameworks, and support to do their jobs. 

Seeing that most of the work was actually done, the development team realized that they just had to leverage the automation.  Now, no one checks in their code, everyone is doing it naturally. Deployment is no longer an issue.

“There is still a challenge, but we have created a path for them to start adopting and not staying back,” he said.

Final Tip

“You have to be aware of where you put your data, figure out what’s in it, how you inspect the data, who interacts with the data and how is it protected,” he said.

“Your apps and data are not secure until you ask these questions.”

PagerDuty is a sponsor of InApps.

Images by T.C. Currie.