Took a while to come back to this project. We have rebuilt from scratch with much more hardened configs. The amount of malicious traffic instantly targeting instances is pretty staggering. But between our hardened deployment and detailed logging, we are confident that everything is now alright!

After successfully integrating logtail with Cloudflare and our backend server, we began digging into the logs to both familiarize ourselves with the new platform as well as to confirm our firewall rules were working as intended. It was quite surprising to find that while the server had only been up for a few days, it already appeared to be compromised.

The log above tipped us off that something wasn't right. A connection originating at our server was attempted FROM port 443 to a random port. Port 443 is not often used as a source port. Time to dig deeper.

In the above log, we see a blocked connection from our servers port 22 to a random high port. Again, this is troubling, as port 22 is most frequently used for SSH connections.

After logging in to logtail to view from a web browser, we find that as is common for all servers exposed to the public internet, our server is being hammered by SSH brute force attempts. In some such connections (as in the one pictured above) they appear to have gained partial access. Our server had both a root password and key certificate. It had not been properly hardened yet as we started from a pre-configured instance to quickly find out if Ghost was going to be a good choice for our content management system.

While our outbound firewall hardening seems to have helped blocked the intruders from accomplishing their goal, the indicators point to the system having been breached. It is unclear whether it was the SSH server or a flaw in the CMS which is at the root of the issue.

At this point we decided to leverage Alienvault OTX as it could quickly be installed and scanned using up to date indicators of compromise. We didn't even wait to let the full scan finish as OTX quickly returned showing that there was a process running without binary on disk. While there are other explanations for such a thing to occur (such as an update that occurred without restarting the process), combined with the indicators above it proved to make it evident the integrity of the system was gone.

We decided since this was still pre-launch and we had backups, we would rather tear down the server and rebuild from scratch. It was an obvious decision not to leave a compromised server running while no users would be impacted by it being shutdown.

It may be some days before we are back up securely but in the spirit of the blog we felt it was important to communicate this transparently with a write-up. There will be no further updates until the blog is back up but feel free to click the contact button if you wish to hear from us sooner.

-FritzTech

Ping IPv6 went down.

Ping IPv4 went down.

TCP IPv4 HTTPS:443 went down.

TCP IPv6 HTTPS:443 went down.

FritzTech Blog went down.