We made 75 private repos public on a timer. The internet noticed in 6 minutes.
(codatus.com)
from peternovakdev@programming.dev to programming@programming.dev on 13 Jun 08:03
https://programming.dev/post/51918758
from peternovakdev@programming.dev to programming@programming.dev on 13 Jun 08:03
https://programming.dev/post/51918758
Live AWS keys in 75 throwaway repos, each made public for one of five windows from 60 seconds to 12 hours, every use logged. The keys were tripwires; the real question was who notices a private repo going public, and what they do once they’re in.
The most useful finding is the dull one: re-hiding the repo does nothing. One busy harvester kept re-validating the captured keys for a day after the repos went private again. Only rotating the key stops it.
This came out of building a monitor for exactly these repo-setting changes.
#programming
threaded - newest
Info for anyone reading, while the read was quite interesting, the whole article turned out to be an ad.
From the post description this made it obvious it’s an ad for something. Otherwise it sounded like someone actually made reasearch on the subject.
Ovaltine? A crummy commercial? Son of a bitch!
Fair to call out. This did come out of me building a product in the space, and I’d rather disclose that than bury it. The method and numbers are real, happy to get into either.
It’s really not surprising that it’s so fast, since you can easily get newly created repos and repos made public from a github API (the “list public events” one at /events). Makes sense that people are polling this and feeding it to TruffleHog.
I guess the rather consistent 6 minutes don’t come from it actually taking so long but rather from some kind of caching that only makes these repos show up after 5 minutes plus 1 minute for fetching and using the api key.
The 6 minutes was the earliest contact, not the typical one. Most first hits came around 8 minutes. I agree there has to be something delaying repos showing up. I was expecting even 60 seconds of exposure to be enough to get caught.
That matches what I saw. One of the actors was a Hetzner host running TruffleHog, and the busiest was a harvester on two OVH IPs doing nothing but GetCallerIdentity checks. So yes, someone is polling the public events feed and scanning whatever shows up. The keys got found the moment the repo was visible to that feed.
There’s hardly any cost to a bot operator, malicious , opportunistic or legitimate, to hit your end-point, so once they found a reason to hit it, hitting it a million more times costs cents.
Operators like Meta seem to make it a sport, trying to hit you with multiple parallel requests from multiple sources, across both IPv4 and IPv6 simultaneously, resulting in an effective DDoS for small and medium end point owners and increasing costs significantly for anyone trying fruitlessly to stay ahead of their onslaught.
The malicious traffic by contrast, attempts to sneak in a request with dynamic rate throttling as part of their attempts to stay hidden.
Between these two extremes are the opportunistic operators who hit the same 404 endpoint day after day, hour after hour, minute by minute, for weeks with specific blocks the only remedy.
There are plenty of legitimate bots that quietly go about their business, hitting you every couple of seconds, leaving you alone for long stretches, incrementally crawling, honouring the robots.txt file and generally acting the way a considerate adult might. They’ve been getting lower and lower in numbers over the years.
Source: I have logs.
There are criminal but professsional groups with million-$ budgets out there.
Hell, there are nation states that have been doing this sort of thing for decades. 15 or so years ago I worked in IT at a university. They bought some servers from IBM and had IBM install them on public IP addresses. It is extremely well known that IBM regularly uses default passwords (or at least used to) like “PASSW0RD” with a zero for the O. I had access to one of these servers about 15 minutes after it was set up, and the first thing I did after changing the password was to check the logs. Sure enough an IP address from China had already logged in as root. I immediately wiped the entire server clean and reinstalled everything.
The real question.
Are people watching private repos go public? Or are people watching for exposed credentials?
Like if I make a Snake game in Python, then make it randomly go public, would anyone notice or care?
This is what I struggled with myself, whether basing the research on people using the key is even relevant. But to get the key, they first have to be watching repos go public. So the watching is the common step, whatever the motive behind it. That’s where the 6 minutes comes from.