2019 GoDaddy Data Breach: A Mystery?
"The combination of RSA encryption and an SSH connection offers some protection from attackers, but, the combo isn’t infallible."
Saul Abrams sits at an antique writer’s desk in his study. A typewriter, reel-to-reel, ashtray, coffee mug, an ivory-handled magnifying glass, and telegraph rest on the dark brown wood.
Saul’s study is somewhat of a museum, discarded newspapers liter every table-top, every shelf is filled with old books and if it weren’t for Saul’s insistent smoking, the room would smell of the particular mold that grows on antique tomes. Saul’s smoky study is suspended in twilight, low light is easy on his myopic eyes and allows him to think.
If he were anywhere but his private study, Saul would be an exhibit in Dadaism. Gray and unkempt hair peek out from underneath a tweed Deerstalker cap. He wears an argyle blazer and matching slacks. His old face, a face that like a raisin, didn’t age well, reveals the bewilderment churning within him. A pipe betwixt his thin lips, he is deep in thought and inhaling rhythmically.
As he takes a drag from his pipe, a cloud of smoke surrounds him. The smoke appears blueish in the dimly lit room. He dictates the day’s notes to the reel-to-reel as he repeatedly puffs on the pipe, kindling the plug of tobacco, its orange ember one of the few lights in the study.
Saul finishes the entry, switches off the reel-to-reel and Mary Gonzales, his maidservant, enters the study.
— “May I refresh your coffee?” she says.
— “Please do Mary and book the next train coach for Phoenix Arizona,” Saul replies.
Saul hears the door creak and intuitively knows it’s his assistant, Patrick O’Neil. After all, it must be Noon, and Pat, if anything, is a punctual youth, thinks Saul. And so, he continues to read the paper.
Pat stands in the dimly lit room waiting for Saul to acknowledge him, but Saul neither looks up from behind the paper or offers Pat a seat.
A long silence passes before Saul speaks. “I’m afraid, Pat, I will be leaving Martha’s Vineyard for a while,” says Saul, his face hidden behind the newspaper that he reads, the Daily Gazette, “I will be going to Phoenix Arizona. I believe Mary, that ever helpful child, is packing my bags and booking a coach as we speak” he concludes.
— “But, you detest the desert,” says Pat, as he shells a handful of peanuts and crams them into his mouth, placing the shells in the amber-colored ashtray on the table.
— “It’s even worse than a desolate desert. I could cope with Bedouin and their infernal camels, but these rapscallions...” Saul replied.
— “Besides, so much sun isn’t good for you, Saul,” Pat warned.
— “Indeed, Phoenix is a city of cumberworld IT professionals and relentless sun, but duty calls my young assistant,” says Saul as he packs his pipe with tobacco.
— “I saw it in the paper today too, GoDaddy’s CentOS servers were attacked by a gang of nefarious privacy pirates, they used some kind of defect in SSH to access the server, I presume. I will go with you,” insists Pat.
— “Presumptions help little in solving a case, we are not yet aware of whom, or how, they entered this server, whatever that may be? The case aside, you will stay here and see that neither Mary nor those blue-jays that ride the trains ransack the place,” concludes Saul.
— “As you wish. I will explain servers and SSH, to the best of my ability, to you. A server...”
An Intro to Cryptology
Mrs. Gonzales is everything Saul could’ve wanted in a maidservant, her vibrant eyes bring light into Saul’s dimly lit study and her youthful step counteracts his elderly gate, both metaphorically and literally. But most of all, she is kind, gentle, attentive and obedient. It’s a rare occasion that she contradicts or disobeys Saul.
It’s around 9:00 AM and Mary finds Saul, as usual, in his study.
— “Did you book tickets on a train for Arizona, Mary,” says Saul.
— “I did not!” exclaims Mary.
— “And why not? I’m needed in Arizona?”
— “You are too old for travel, you Don Quixote.”
— “I see you’ve read Cervantes.”
— “I have, and in Spanish. Why don’t you and Pat solve the caper via SSH,” suggests Mary, she has a way of getting through to Saul's stubborn old soul. It's her familiarity, ebony hair and reserved manner that allows her to get through Saul's thick skin.
— “You mean to say, use the same method the criminals did to commit the crime to solve it?”
— “Brilliant! Why didn’t Pat think of this?”
— “I don’t think it occurred to him that...”
The Suspects and an Intro to RSA
It’s past Noon and Pat hasn’t arrived. I shall dictate the day’s notes to my marvelous recording machine, thinks Saul.
It’s noon, and as usual, Pat has arrived to discuss the case with Saul. Saul senses that something is heavy on Pat’s mind, his blue eyes have lost their luster. Saul and Pat sit in the dim study and Mary prepares coffee.
— “Much time has passed,” says Saul.
— “Yes, it has. Too much time.”
— “I have a confession to make,” says Pat nervously.
— “What is it, my boy?”
— “Mary, come here,” yells Pat. Saul packs his pipe and takes a long drag.
— “Would you like the coffee now?” asks Mary.
— “No, I think it’s time to tell Saul our secret,” says Pat, and he runs his hand through his red hair, a twitch.
Mary places the serving dish on the table, fixes her dress, and begins to speak, slowly, her word is calculated:
— “Saul, we, Pat and I, are in love.”
Saul exhales, places his pipe on the dark wood desk, makes his expression blank, just for effect, and breaks a smile:
— “You’re not mad!” says Pat.
— “No, now let’s get back to the case. ”
— “I have received a letter from Markku Rossi, I will fill you in on the details,” says Pat, elated that Saul gave the wedding his blessing.
The house has become a hub of activity as Mary and Pat prepare for their wedding, a photographer, caterer, florist... It has been arranged, the wedding will take place on June 1st.
It’s 1:00 PM, Saul and Pat sit in the study, discussing recent developments.
— “I hope you’re not too distracted to work,” says Saul in a low and serious tone.
— “Don’t be foolish. In fact, I’ve come with some recent developments. Do you think that it could be as simple as customers changing their FTP and SSH passwords to weak ones, something that could be easily guessed by attackers?”
— “No, I don’t think so, GoDaddy requires complex passwords,” says Saul.
— “Has Mary not been well? She hasn’t been timely in bringing my coffee.”
— “She has been under the weather, perhaps it’s the strain of planning the wedding. I’m sure she’ll be fine after this is all out of the way. As I said, I have brought some interesting information, here’s the bug report from the version of OpenSSH we assume that this virtual burglar used.”
The investigation has slowed down. A sudden turn in Mary’s health has caused a heaviness to return to the estate.
— “Come in, Pat,” says Saul. He sits at his desk, and with a worried look, he packs a plug of tobacco into his pipe.
— “I’ve heard that Mary was taken to the hospital.”
— “I would be remiss if I didn’t speak honestly, she is in the hands of a good doctor, but it’s just cold-feet my boy, she will be fine,” says Saul in an attempt to mask his concerns over Mary’s worsening health.
The house feels lifeless as Saul and Pat discuss the details of the crime without enthusiasm, the sounds of Mary singing to herself while washing the dishes no more echo through the halls, leaving the house a dark and tobacco-stained bachelor pad.
Pat is visiting the hospital and Saul sits alone in his study. The tape turns on his reel-to-reel as he records his thoughts.
An Error of IT
The house has become like a tomb, dark and stuffy. Mary has been ill for weeks. The nature of her illness is unknown, but it is now serious. The wedding has been postponed, Both Saul and Pat work with less enthusiasm.
It’s noon, but the study is dark as night. The smell of tobacco lingers in the air as Saul and Pat discuss the case.
— “I have found no leads among the ex-employees. It appears that most employees find GoDaddy a pleasant place to work,” says Pat.
— “As I expected my boy, but this does seem to be the most likely route, they have the easiest access to the servers."
A Case Without Closure
Mary has passed. She will be buried among the estate’s sprawling lawn and the natural beauty of Martha’s Vineyard. The case is cold. Saul and Pat have all but given up on finding the culprit, regrettably, wedding plans have given way to erecting a sepulcher.
“All-in-all, I see three possible suspects:
- The culprit could’ve gained access to the server through a vulnerability in CentOS or software used to communicate with CentOS,
- He could’ve gotten in through brute force, using software, or by guessing weak passwords created by Users;
- And finally, it may have simply been an ex-employee who already had access to the server, no matter which method was used, too much time has passed. We will never know,” says Saul and takes a sip of his coffee."
This is mostly a reiteration of what our protagonist, the detective Saul, deduced. It’s a warning to those who maintain servers that utilize SSH keys or other means of secure connection, like SSL. These keys are prone to attacks, and despite often being encrypted with RSA, they are vulnerable. In fact, software like SSH Attacker was designed to infiltrate such encryption.
Certainly, the combination of RSA encryption and an SSH connection offers some protection from attackers, protection in the form of nifty features: authentication, tamper protection, SH5, and complex encryption ciphers. But, the combo isn’t infallible.
Here's something to keep in mind: the sheer number of SSH keys might allow for an attack, an enterprise, like GoDaddy, may have millions of them on hand. The more there are, the easier it is to crack one. For example, if a server merely possesses one public/private key pairing, a Hacker would be less likely to sniff it out. It’s the difference between guessing a number in a hat of many numbered slips of paper and guessing the one number in this author’s mind, although this is, admittedly, highly oversimplified.
Simply, one is more likely to guess a number combination when there’s a number of available answers, rather than one, it’s the law of odds.
Moreover, in most enterprises, SSH keys are static, they remain the same. And, it’s easy to see why rotating, regularly disposing and recreating, millions of keys may cause panic among the IT Department - not just is it a logistical nightmare - a few mislaid keystrokes could cause a company-wide outage. But, neglecting to rotate keys might result in long-term access to data by nefarious individuals, loss of customer base, and a PR nightmare.
In addition, SSH keys are often shared or replicated by numerous employees, servers, and other infrastructure components. According to Cyberark.com, “as few as five to 20 unique keys can grant access to all machines throughout a [typical] enterprise,” though it’s not possible to know if GoDaddy practiced key rotation.
Indeed, neglecting to rotate keys lightens the IT Department’s workload, in the short-term, but makes attacking a server a much easier task in a number of ways: permanent keys with root privilege allow for network-wide domination if they fall into the hands of disgruntled ex-employees - the lead our detective heavily leaned on.
In fact, “SSH key duplication creates complicated, many-to-many private/public key mappings that significantly reduce security because it is difficult to rotate and revoke a single key without breaking untold other SSH key relationships that share the same key fingerprint.”
Aside from all-out attacks, enterprises, such as GoDaddy, often lose track of credentials when migrating from development to production servers, leaving the digital vault door open to both hackers and ex-employees. These keys must be scrubbed from the servers when they go into production.