I woke up on a Sunday morning, made coffee, and opened my email. The first message was from AWS. It was a billing alert. My monthly spend, normally around forty dollars, had crossed the two hundred dollar threshold with two weeks still left in the cycle. I assumed it was a mistake. I had a small side project running on a single EC2 instance and an RDS database. There was no traffic spike, no new deployment, no reason for the cost to quintuple overnight. I logged into the AWS console, opened the billing dashboard, and felt my stomach drop. The cost explorer showed a massive spike in EC2 usage, but not from my instance. Someone had spun up eight large EC2 instances in a region I had never used, and they had been running for over twenty four hours, mining cryptocurrency. My AWS account had been compromised, and the root cause was a security mistake so basic that I was embarrassed to admit it. This is exactly what happened, how I fixed it, and the safeguards I put in place so it never happens again.
How It Happened: The Access Key I Should Never Have Created
Months earlier, I had been testing a deployment script on my laptop. I needed programmatic access to AWS, so I created an IAM user with administrator permissions and generated an access key and secret key pair. I hardcoded those keys into a configuration file, ran my script, and forgot about them. Later, I pushed the entire project directory to a public GitHub repository. I was in a hurry. I did not check what was in the directory. The configuration file with the plaintext keys went up with everything else. I never noticed, because the repository was a personal experiment and I had no contributors watching. But automated scanners watch all public repositories. They search for AWS access key patterns around the clock. My keys were discovered, extracted, and sold within hours of the push.
The attacker did not do anything destructive. They did not delete my data or deface my application. That would have triggered alarms and prompted me to act faster. Instead, they quietly created EC2 instances in a far away region and ran crypto mining software. The mining operation burned CPU credits and accumulated charges silently. My normal resources were untouched, and my monitoring alerts were scoped only to my primary region. The attack was designed for stealth, and it worked perfectly. By the time the billing alert triggered, the damage was already substantial.
The Discovery and the Panic
I opened the EC2 dashboard and did not recognize half the instances. They had names I did not assign, running in the Seoul region, which I had never visited. The instances were large, expensive types. I immediately terminated all of them. Then I checked the IAM dashboard and found the compromised access key. I revoked it instantly, which cut off the attacker’s access. I also rotated every other access key in the account, even the ones that had not been exposed, because I no longer trusted anything. I enabled multi-factor authentication on the root account, which I should have done years earlier. The immediate threat was contained within about fifteen minutes of opening the email. The financial damage was already done, but the data and the application were safe.
The Damage Assessment
I spent the next hour going through every AWS service, checking for resources I did not recognize. The attacker had created the eight EC2 instances, a few security groups, and a key pair. They had not touched S3, RDS, or IAM beyond the compromised key. The total bill for the unauthorized usage came to just over two hundred dollars. That was the cost of the large instances running for a bit more than a day. I opened a support case with AWS, explained what happened, and asked if there was any way to get the charges waived. I was honest about my mistake: I had exposed the key in a public repository. AWS reviewed the case and, to my surprise, credited the full two hundred dollars back to my account. The support engineer noted that I had responded quickly, secured the account, and this was a first time incident. I was lucky, and I knew it. The credit was not guaranteed, and I should not have been in the situation in the first place.
The Root Cause: More Than Just a Leaked Key
The obvious mistake was pushing secrets to a public repository. But the deeper mistake was granting administrator permissions to an access key that was only needed for a single script. The key could do anything: create instances, delete databases, change billing settings. If I had assigned only the specific permissions required for the deployment script, the attacker’s blast radius would have been much smaller. They might have been able to read from an S3 bucket or publish to an SNS topic, but they could not have spun up a fleet of large EC2 instances. The principle of least privilege is not just security jargon. It is a financial protection mechanism. I learned that lesson at a cost of two hundred dollars and several hours of panicked remediation.
The Fixes I Put in Place Immediately
After the immediate threat was gone and the AWS support case was resolved, I rebuilt my security posture from the ground up. The first change was to never use long lived access keys for anything unless absolutely necessary. For any future automation, I would use IAM roles instead of access keys. Roles provide temporary credentials that rotate automatically and cannot be leaked in the same way. For my local development, I set up AWS SSO so I could get short lived credentials through the CLI without ever storing a secret on disk. The configuration files that had caused the leak were deleted and replaced with environment variables sourced from a file outside the repository, a file that was added to .gitignore.
The second change was to implement a pre-commit hook that scanned for secrets before any commit left my machine. I used a tool called git-secrets that blocks commits containing patterns matching AWS access keys, private keys, and other sensitive strings. The hook runs automatically and refuses to let me commit if it finds a match. It is not foolproof, but it adds a layer of defense that would have caught my mistake before it reached GitHub. I also enabled GitHub’s built-in secret scanning, which sends an alert if a secret is pushed to a repository, even a private one. The scanner would have notified me within minutes of the push, potentially before the attacker found the key.
The third change was to set up AWS Budgets with actual alarms. I had a billing alert before the incident, but it was a single threshold at fifty dollars. I added alerts at twenty five, fifty, one hundred, and two hundred dollars. I also set up a zero spend budget that would fire if any usage appeared in a region I did not use. If the Seoul instances had triggered an alert immediately, I could have responded hours earlier and reduced the damage. The budget alerts now go to both email and a Slack channel that I check more frequently than my inbox.
The Lesson That Changed How I Think About Cloud Security
The breach was not sophisticated. I was not targeted by a nation state or a clever social engineering attack. I made a simple, common mistake: I checked credentials into version control. The attacker used a fully automated script. They did not know who I was or what my account contained. They cast a wide net and caught me because I had left a hole. That realization was humbling. The barrier to being compromised is not high. It is a matter of whether you are an easy target or a hard one. Before the incident, I was easy. Now I am harder, though I know that no defense is perfect.
I also learned to think of cloud credentials as radioactive. Every access key is a loaded weapon. It should live as briefly as possible, have as few permissions as possible, and be treated with the same care as a password. The convenience of a quick access key for a one-off script is not worth the risk. I now invest the extra fifteen minutes to set up a role or a temporary credential. That time is insurance against another Sunday morning panic.
What I Would Do Differently If Starting Over
If I could rewind to the day I created that access key, I would stop myself. I would use an IAM role for the deployment script, or I would use temporary credentials from AWS SSO. If I absolutely had to use an access key, I would scope it to the exact permissions needed for the task, and I would set an expiration policy so the key would deactivate automatically after a few days. I would also use a .env file from the very beginning, with .gitignore configured before the first commit. The fix is trivial, but only if you do it before the secret is exposed. After the exposure, the fix is expensive and stressful.
I would also audit my AWS account regularly. Before the incident, I logged into the console only when I needed to deploy something. Now I have a monthly calendar reminder to review IAM users, access keys, and resources in all regions. The review takes fifteen minutes and catches any lingering resources or unused credentials. It is a boring habit, but boring habits prevent exciting disasters.
Advice for Anyone Running Cloud Infrastructure
If you use AWS, or any cloud provider, assume that you will eventually leak a credential. It might be you, or it might be a teammate. The question is how much damage that leaked credential can do. Limit permissions ruthlessly. Use roles instead of keys. Enable MFA on everything. Set up budget alerts with low thresholds. Scan your commits for secrets. These are not advanced security measures. They are the minimum for anyone who cares about not waking up to a surprise bill. My mistake cost me a few hundred dollars and a stressful morning. It could have been much worse. The attacker could have deleted my database, held my data for ransom, or used my account to attack others. I was lucky, and luck is not a strategy. I have replaced it with a set of habits that make me a less appealing target. I hope you do the same before you learn this lesson the hard way.
