Incidents reported on status page for FritzTech https://status.fritztech.net/ https://d1lppblt9t2x15.cloudfront.net/logos/87e376e1a39311dd4591f0ec266f1afe.jpeg https://status.fritztech.net/ en https://status.fritztech.net/incident/445420 Wed, 16 Oct 2024 06:48:55 -0000 https://status.fritztech.net/incident/445420#92b30e65f199bc1fce1b68e425c7a658f653f8125c28906373706148e66783dc FritzTech Blog recovered. https://status.fritztech.net/incident/445420 Wed, 16 Oct 2024 06:43:10 -0000 https://status.fritztech.net/incident/445420#38bc9e12edf54592c9fec2de85b59441c35996f12340ccebaaee20ca6c25ad37 FritzTech Blog went down. https://status.fritztech.net/incident/418528 Fri, 06 Sep 2024 23:12:43 -0000 https://status.fritztech.net/incident/418528#e38818249811cbbbdf2f99ddd1dc98004e7e923de1958d26bdd0682c0a73ba89 TCP IPv4 HTTPS recovered. https://status.fritztech.net/incident/418528 Fri, 06 Sep 2024 23:12:22 -0000 https://status.fritztech.net/incident/418528#f017b8a1f80e7c189424eeaf3a6f8198ba5e7896b75ab0dde99b7d099e4dd73b FritzTech Blog recovered. https://status.fritztech.net/incident/418528 Fri, 06 Sep 2024 23:12:08 -0000 https://status.fritztech.net/incident/418528#cb386dc99f54ec6edb2a27c2cdebc1f37daf3b7df3728b0fafd2f64e9b0fd1e8 TCP IPv6 HTTPS recovered. https://status.fritztech.net/incident/418528 Sat, 24 Aug 2024 06:54:57 -0000 https://status.fritztech.net/incident/418528#65a7dd6bca25554e39833124a87174075b7c01a840c92cfc546e222c37a4f181 TCP IPv4 HTTPS went down. https://status.fritztech.net/incident/418528 Sat, 24 Aug 2024 06:54:19 -0000 https://status.fritztech.net/incident/418528#aa6bb952c0b7653a78816f0dc7d0872c03c71e1ac7a0e886f30d1f6e9d39bcbb TCP IPv6 HTTPS went down. https://status.fritztech.net/incident/418528 Sat, 24 Aug 2024 06:53:38 -0000 https://status.fritztech.net/incident/418528#e58dac5a3988ba5c9c73e990d0f6793c09369a418cf0c5fbb8581e69af9f7e08 FritzTech Blog went down. https://status.fritztech.net/incident/407182 Thu, 01 Aug 2024 05:38:16 -0000 https://status.fritztech.net/incident/407182#a6be1941f4392670afc5374acf70354ea85538232f1a39c143f15e1fc46b24c8 FritzTech Blog recovered. https://status.fritztech.net/incident/407182 Thu, 01 Aug 2024 05:32:01 -0000 https://status.fritztech.net/incident/407182#474c0fe3ad5e71cebab0edb4d94fb5b527bd7c0bac3b5e7f203e4633f7e60fd2 FritzTech Blog went down. https://status.fritztech.net/incident/397095 Fri, 12 Jul 2024 13:07:27 -0000 https://status.fritztech.net/incident/397095#a3a6ef12b55a63d263ae87af0592c8af84dab781062d421fe5ecc1c7504df95c FritzTech Blog recovered. https://status.fritztech.net/incident/397095 Fri, 12 Jul 2024 13:06:55 -0000 https://status.fritztech.net/incident/397095#f057badaead3e78c3e48514bc1b1b5a5d3a0d508df9923b9ba4c9669d3fd8ecd TCP IPv6 HTTPS recovered. https://status.fritztech.net/incident/397095 Fri, 12 Jul 2024 13:02:09 -0000 https://status.fritztech.net/incident/397095#96fba255c92d1115ccb729650b8a6db733ce7580e0d028de26b6d234374bd15f TCP IPv6 HTTPS went down. https://status.fritztech.net/incident/397095 Fri, 12 Jul 2024 13:01:34 -0000 https://status.fritztech.net/incident/397095#e0624b08fce2aab839714e6ea9b611647cf5db6e2cd7ea1896817905c9c21efa FritzTech Blog went down. https://status.fritztech.net/incident/397095 Fri, 12 Jul 2024 12:52:25 -0000 https://status.fritztech.net/incident/397095#844d4ee8b52dba7db8b033898ee0ec42276b2a6dd51361f7b6568540fb4d1ae3 FritzTech Blog recovered. https://status.fritztech.net/incident/397095 Fri, 12 Jul 2024 12:46:43 -0000 https://status.fritztech.net/incident/397095#e6f8f35e7d711c265460d753543db53d2e7092f0da1fe5dab2e3534eeb32d807 FritzTech Blog went down. https://status.fritztech.net/incident/397095 Fri, 12 Jul 2024 07:06:27 -0000 https://status.fritztech.net/incident/397095#53d41ffeebbf418796b20bf1eb9466835c1d83d46832335b21e03ee28f954c3b FritzTech Blog recovered. https://status.fritztech.net/incident/397095 Fri, 12 Jul 2024 07:05:54 -0000 https://status.fritztech.net/incident/397095#02c652eba565ebb4f7aa03656711ab0dd4f9a746fb05f25f22a95a351426422f TCP IPv6 HTTPS recovered. https://status.fritztech.net/incident/397095 Fri, 12 Jul 2024 07:03:30 -0000 https://status.fritztech.net/incident/397095#54e52195a8c5d85b63e8159f179c5b90708252aa7259f2998fa70eb5711a1b3a TCP IPv4 HTTPS recovered. https://status.fritztech.net/incident/397095 Fri, 12 Jul 2024 06:52:48 -0000 https://status.fritztech.net/incident/397095#536ada7cff8d7054b52dc06519fa86f4fb7ed10e6402b60467cb516f1a5f491a TCP IPv4 HTTPS went down. https://status.fritztech.net/incident/397095 Fri, 12 Jul 2024 06:51:51 -0000 https://status.fritztech.net/incident/397095#27cd58b8e0c8dbe0339012f597a631f9181d19a62d50f00ab67f28f01fb20b25 TCP IPv6 HTTPS went down. https://status.fritztech.net/incident/397095 Fri, 12 Jul 2024 06:51:27 -0000 https://status.fritztech.net/incident/397095#c569299b9140571834c2b4243cdb79f5a1ccd1d4eb515f7985d72260ba871e45 FritzTech Blog went down. https://status.fritztech.net/incident/397095 Fri, 12 Jul 2024 05:45:28 -0000 https://status.fritztech.net/incident/397095#a1f09b4d4e67ba1578f351139e2859d25296aec009ae46bda55012f9bb12b5ed FritzTech Blog and TCP IPv4 HTTPS recovered. https://status.fritztech.net/incident/397095 Fri, 12 Jul 2024 05:45:28 -0000 https://status.fritztech.net/incident/397095#a1f09b4d4e67ba1578f351139e2859d25296aec009ae46bda55012f9bb12b5ed FritzTech Blog and TCP IPv4 HTTPS recovered. https://status.fritztech.net/incident/397095 Fri, 12 Jul 2024 05:44:57 -0000 https://status.fritztech.net/incident/397095#e3a39661e29ae842f3db90a17b49e03b64f738be807ae43097ca7ad90c262f75 TCP IPv6 HTTPS recovered. https://status.fritztech.net/incident/397095 Fri, 12 Jul 2024 05:36:50 -0000 https://status.fritztech.net/incident/397095#6df096238abf1f61527568c9b2ff62a1bc037f81d2c23fc2ea018d0ced94a7a5 TCP IPv6 HTTPS went down. https://status.fritztech.net/incident/397095 Fri, 12 Jul 2024 05:34:27 -0000 https://status.fritztech.net/incident/397095#1d0f79912cf8cda080a1bd339748222fe0d569bc1e4005d7bd87d302906e71a7 TCP IPv4 HTTPS went down. https://status.fritztech.net/incident/397095 Fri, 12 Jul 2024 05:33:25 -0000 https://status.fritztech.net/incident/397095#bf1fa5fe829a6e424032a3f4cb379399bea23f4f442a8b36e0554c5c5ab325ef FritzTech Blog went down. https://status.fritztech.net/incident/397095 Fri, 12 Jul 2024 05:18:25 -0000 https://status.fritztech.net/incident/397095#f4d2b7fa8b8671f5bf4b94f47fe87d8feb0b5a3de231b7026535e2c403064877 FritzTech Blog and TCP IPv4 HTTPS recovered. https://status.fritztech.net/incident/397095 Fri, 12 Jul 2024 05:18:25 -0000 https://status.fritztech.net/incident/397095#f4d2b7fa8b8671f5bf4b94f47fe87d8feb0b5a3de231b7026535e2c403064877 FritzTech Blog and TCP IPv4 HTTPS recovered. https://status.fritztech.net/incident/397095 Fri, 12 Jul 2024 05:17:47 -0000 https://status.fritztech.net/incident/397095#6e1a9ec73df607a682420d7a8b05d11589e4bbc85d373e8edc7429f6c380d218 TCP IPv6 HTTPS recovered. https://status.fritztech.net/incident/397095 Fri, 12 Jul 2024 04:46:25 -0000 https://status.fritztech.net/incident/397095#faa09d99243865eca9e4288ed16018425fac562398d09eae03fd2c07dc04d47f TCP IPv4 HTTPS went down. https://status.fritztech.net/incident/397095 Fri, 12 Jul 2024 04:45:44 -0000 https://status.fritztech.net/incident/397095#6c0731c66944d82d3413463324e2a6c8e6fc8f04569f5f58b1ed51979edc09a9 TCP IPv6 HTTPS went down. https://status.fritztech.net/incident/397095 Fri, 12 Jul 2024 04:45:17 -0000 https://status.fritztech.net/incident/397095#112f6e26de49422f7c57a52b9f254a1f51642a07045ef9b4ceb9e88d02aed4a8 FritzTech Blog went down. https://status.fritztech.net/incident/387445 Fri, 21 Jun 2024 09:15:10 -0000 https://status.fritztech.net/incident/387445#2ecd0618886d1e851cb2b209d4edb384c1f7fb986e2b8365c7473d059256ff7d FritzTech Blog and TCP IPv4 HTTPS recovered. https://status.fritztech.net/incident/387445 Fri, 21 Jun 2024 09:15:10 -0000 https://status.fritztech.net/incident/387445#2ecd0618886d1e851cb2b209d4edb384c1f7fb986e2b8365c7473d059256ff7d FritzTech Blog and TCP IPv4 HTTPS recovered. https://status.fritztech.net/incident/387445 Fri, 21 Jun 2024 09:14:29 -0000 https://status.fritztech.net/incident/387445#ebecf85d4aaed5faae87d4cc286329450d591bf20849e3d63d204a93df2de6bf TCP IPv6 HTTPS recovered. https://status.fritztech.net/incident/387445 Fri, 21 Jun 2024 09:10:06 -0000 https://status.fritztech.net/incident/387445#a03702c3108f0c2825e541dde61b7f1a17d7321271ab87e5cb6886e73cecc778 TCP IPv4 HTTPS went down. https://status.fritztech.net/incident/387445 Fri, 21 Jun 2024 09:09:31 -0000 https://status.fritztech.net/incident/387445#4b20db9236fcb67e1fd02af9b2b7ddb3a933f99d2f7afcfb092fe505ac641c77 TCP IPv6 HTTPS went down. https://status.fritztech.net/incident/387445 Fri, 21 Jun 2024 09:09:11 -0000 https://status.fritztech.net/incident/387445#358a82e1f418d2c2d000885e6ed909e80fa8a509a57cc6b560edc80ee1dd0f0e FritzTech Blog went down. https://status.fritztech.net/incident/375703 Tue, 28 May 2024 09:06:00 -0000 https://status.fritztech.net/incident/375703#b417a20ad607a1b3cf108148824bdf19e0622ce5436653097781c3efc18c4a57 After receiving alerts that the server was responding with 502 status codes from indexers, we disabled caching at Cloudflare which revealed the server was indeed responding with 502 for ALL requests. Investigation revealed that the server's reverse proxy to the Ghost CMS was the culprit as the service had changed ports. We fixed the reverse proxy and believe we have the port hardcoded in the Ghost configuration now. The site is back to full functionality! https://status.fritztech.net/incident/375703 Tue, 28 May 2024 08:47:03 -0000 https://status.fritztech.net/incident/375703#2e9032a77f6215adcfcd72a7e773e9dacd5d95d190dc99bcbcc0c351a4090aba FritzTech Blog recovered. https://status.fritztech.net/incident/375703 Tue, 28 May 2024 08:26:03 -0000 https://status.fritztech.net/incident/375703#f0a61d4e488a353ff6df5eba04536c3ad1d0d3ccc06497eb07cb7a4fa2715092 FritzTech Blog went down. https://status.fritztech.net/incident/375703 Tue, 28 May 2024 07:44:03 -0000 https://status.fritztech.net/incident/375703#a3d5b087f8566264e9d729aaa65cee76a1b46e7cf81ad317a753ee8c06b85529 FritzTech Blog recovered. https://status.fritztech.net/incident/375703 Tue, 28 May 2024 04:31:18 -0000 https://status.fritztech.net/incident/375703#235904ca5bff9027d8c3ee16ee5bf44deb73a5320e4c3bf59e69f3276f5c6795 FritzTech Blog went down. https://status.fritztech.net/incident/170653 Thu, 09 May 2024 22:30:00 -0000 https://status.fritztech.net/incident/170653#75653a26c0731b19a7763c92d976dc4cddc5056070b3936d712f5502e0fb4582 Took a while to come back to this project. We have rebuilt from scratch with much more hardened configs. The amount of malicious traffic instantly targeting instances is pretty staggering. But between our hardened deployment and detailed logging, we are confident that everything is now alright! https://status.fritztech.net/incident/170653 Thu, 09 May 2024 22:30:00 -0000 https://status.fritztech.net/incident/170653#75653a26c0731b19a7763c92d976dc4cddc5056070b3936d712f5502e0fb4582 Took a while to come back to this project. We have rebuilt from scratch with much more hardened configs. The amount of malicious traffic instantly targeting instances is pretty staggering. But between our hardened deployment and detailed logging, we are confident that everything is now alright! https://status.fritztech.net/incident/170653 Thu, 09 May 2024 22:30:00 -0000 https://status.fritztech.net/incident/170653#75653a26c0731b19a7763c92d976dc4cddc5056070b3936d712f5502e0fb4582 Took a while to come back to this project. We have rebuilt from scratch with much more hardened configs. The amount of malicious traffic instantly targeting instances is pretty staggering. But between our hardened deployment and detailed logging, we are confident that everything is now alright! https://status.fritztech.net/incident/170653 Thu, 09 May 2024 22:30:00 -0000 https://status.fritztech.net/incident/170653#75653a26c0731b19a7763c92d976dc4cddc5056070b3936d712f5502e0fb4582 Took a while to come back to this project. We have rebuilt from scratch with much more hardened configs. The amount of malicious traffic instantly targeting instances is pretty staggering. But between our hardened deployment and detailed logging, we are confident that everything is now alright! https://status.fritztech.net/incident/170653 Wed, 01 Feb 2023 22:00:00 -0000 https://status.fritztech.net/incident/170653#8c3e7315fd24fcf7a0ba31b4c624f70f854ebd7517ca1fe61a341303e2051dbb After successfully integrating logtail with Cloudflare and our backend server, we began digging into the logs to both familiarize ourselves with the new platform as well as to confirm our firewall rules were working as intended. It was quite surprising to find that while the server had only been up for a few days, it already appeared to be compromised.  The log above tipped us off that something wasn't right. A connection originating at our server was attempted FROM port 443 to a random port. Port 443 is not often used as a source port. Time to dig deeper.  In the above log, we see a blocked connection from our servers port 22 to a random high port. Again, this is troubling, as port 22 is most frequently used for SSH connections.  After logging in to logtail to view from a web browser, we find that as is common for all servers exposed to the public internet, our server is being hammered by SSH brute force attempts. In some such connections (as in the one pictured above) they appear to have gained partial access. Our server had both a root password and key certificate. It had not been properly hardened yet as we started from a pre-configured instance to quickly find out if Ghost was going to be a good choice for our content management system. While our outbound firewall hardening seems to have helped blocked the intruders from accomplishing their goal, the indicators point to the system having been breached. It is unclear whether it was the SSH server or a flaw in the CMS which is at the root of the issue.  At this point we decided to leverage Alienvault OTX as it could quickly be installed and scanned using up to date indicators of compromise. We didn't even wait to let the full scan finish as OTX quickly returned showing that there was a process running without binary on disk. While there are other explanations for such a thing to occur (such as an update that occurred without restarting the process), combined with the indicators above it proved to make it evident the integrity of the system was gone. We decided since this was still pre-launch and we had backups, we would rather tear down the server and rebuild from scratch. It was an obvious decision not to leave a compromised server running while no users would be impacted by it being shutdown. It may be some days before we are back up securely but in the spirit of the blog we felt it was important to communicate this transparently with a write-up. There will be no further updates until the blog is back up but feel free to click the contact button if you wish to hear from us sooner. **-FritzTech** https://status.fritztech.net/incident/170653 Wed, 01 Feb 2023 22:00:00 -0000 https://status.fritztech.net/incident/170653#8c3e7315fd24fcf7a0ba31b4c624f70f854ebd7517ca1fe61a341303e2051dbb After successfully integrating logtail with Cloudflare and our backend server, we began digging into the logs to both familiarize ourselves with the new platform as well as to confirm our firewall rules were working as intended. It was quite surprising to find that while the server had only been up for a few days, it already appeared to be compromised.  The log above tipped us off that something wasn't right. A connection originating at our server was attempted FROM port 443 to a random port. Port 443 is not often used as a source port. Time to dig deeper.  In the above log, we see a blocked connection from our servers port 22 to a random high port. Again, this is troubling, as port 22 is most frequently used for SSH connections.  After logging in to logtail to view from a web browser, we find that as is common for all servers exposed to the public internet, our server is being hammered by SSH brute force attempts. In some such connections (as in the one pictured above) they appear to have gained partial access. Our server had both a root password and key certificate. It had not been properly hardened yet as we started from a pre-configured instance to quickly find out if Ghost was going to be a good choice for our content management system. While our outbound firewall hardening seems to have helped blocked the intruders from accomplishing their goal, the indicators point to the system having been breached. It is unclear whether it was the SSH server or a flaw in the CMS which is at the root of the issue.  At this point we decided to leverage Alienvault OTX as it could quickly be installed and scanned using up to date indicators of compromise. We didn't even wait to let the full scan finish as OTX quickly returned showing that there was a process running without binary on disk. While there are other explanations for such a thing to occur (such as an update that occurred without restarting the process), combined with the indicators above it proved to make it evident the integrity of the system was gone. We decided since this was still pre-launch and we had backups, we would rather tear down the server and rebuild from scratch. It was an obvious decision not to leave a compromised server running while no users would be impacted by it being shutdown. It may be some days before we are back up securely but in the spirit of the blog we felt it was important to communicate this transparently with a write-up. There will be no further updates until the blog is back up but feel free to click the contact button if you wish to hear from us sooner. **-FritzTech** https://status.fritztech.net/incident/170653 Wed, 01 Feb 2023 22:00:00 -0000 https://status.fritztech.net/incident/170653#8c3e7315fd24fcf7a0ba31b4c624f70f854ebd7517ca1fe61a341303e2051dbb After successfully integrating logtail with Cloudflare and our backend server, we began digging into the logs to both familiarize ourselves with the new platform as well as to confirm our firewall rules were working as intended. It was quite surprising to find that while the server had only been up for a few days, it already appeared to be compromised.  The log above tipped us off that something wasn't right. A connection originating at our server was attempted FROM port 443 to a random port. Port 443 is not often used as a source port. Time to dig deeper.  In the above log, we see a blocked connection from our servers port 22 to a random high port. Again, this is troubling, as port 22 is most frequently used for SSH connections.  After logging in to logtail to view from a web browser, we find that as is common for all servers exposed to the public internet, our server is being hammered by SSH brute force attempts. In some such connections (as in the one pictured above) they appear to have gained partial access. Our server had both a root password and key certificate. It had not been properly hardened yet as we started from a pre-configured instance to quickly find out if Ghost was going to be a good choice for our content management system. While our outbound firewall hardening seems to have helped blocked the intruders from accomplishing their goal, the indicators point to the system having been breached. It is unclear whether it was the SSH server or a flaw in the CMS which is at the root of the issue.  At this point we decided to leverage Alienvault OTX as it could quickly be installed and scanned using up to date indicators of compromise. We didn't even wait to let the full scan finish as OTX quickly returned showing that there was a process running without binary on disk. While there are other explanations for such a thing to occur (such as an update that occurred without restarting the process), combined with the indicators above it proved to make it evident the integrity of the system was gone. We decided since this was still pre-launch and we had backups, we would rather tear down the server and rebuild from scratch. It was an obvious decision not to leave a compromised server running while no users would be impacted by it being shutdown. It may be some days before we are back up securely but in the spirit of the blog we felt it was important to communicate this transparently with a write-up. There will be no further updates until the blog is back up but feel free to click the contact button if you wish to hear from us sooner. **-FritzTech** https://status.fritztech.net/incident/170653 Wed, 01 Feb 2023 22:00:00 -0000 https://status.fritztech.net/incident/170653#8c3e7315fd24fcf7a0ba31b4c624f70f854ebd7517ca1fe61a341303e2051dbb After successfully integrating logtail with Cloudflare and our backend server, we began digging into the logs to both familiarize ourselves with the new platform as well as to confirm our firewall rules were working as intended. It was quite surprising to find that while the server had only been up for a few days, it already appeared to be compromised.  The log above tipped us off that something wasn't right. A connection originating at our server was attempted FROM port 443 to a random port. Port 443 is not often used as a source port. Time to dig deeper.  In the above log, we see a blocked connection from our servers port 22 to a random high port. Again, this is troubling, as port 22 is most frequently used for SSH connections.  After logging in to logtail to view from a web browser, we find that as is common for all servers exposed to the public internet, our server is being hammered by SSH brute force attempts. In some such connections (as in the one pictured above) they appear to have gained partial access. Our server had both a root password and key certificate. It had not been properly hardened yet as we started from a pre-configured instance to quickly find out if Ghost was going to be a good choice for our content management system. While our outbound firewall hardening seems to have helped blocked the intruders from accomplishing their goal, the indicators point to the system having been breached. It is unclear whether it was the SSH server or a flaw in the CMS which is at the root of the issue.  At this point we decided to leverage Alienvault OTX as it could quickly be installed and scanned using up to date indicators of compromise. We didn't even wait to let the full scan finish as OTX quickly returned showing that there was a process running without binary on disk. While there are other explanations for such a thing to occur (such as an update that occurred without restarting the process), combined with the indicators above it proved to make it evident the integrity of the system was gone. We decided since this was still pre-launch and we had backups, we would rather tear down the server and rebuild from scratch. It was an obvious decision not to leave a compromised server running while no users would be impacted by it being shutdown. It may be some days before we are back up securely but in the spirit of the blog we felt it was important to communicate this transparently with a write-up. There will be no further updates until the blog is back up but feel free to click the contact button if you wish to hear from us sooner. **-FritzTech** https://status.fritztech.net/incident/170653 Wed, 01 Feb 2023 20:10:41 -0000 https://status.fritztech.net/incident/170653#c58f9e75d0cafdb449c101e54b85a570c7688380875ac15ca6b7e6a6a201e388 Ping IPv6 went down. https://status.fritztech.net/incident/170653 Wed, 01 Feb 2023 20:09:52 -0000 https://status.fritztech.net/incident/170653#013d06ac3322800244003c3ca6af9d712117877ec1d30f3a0f652cdf16e066b3 TCP IPv4 HTTPS:443 went down. https://status.fritztech.net/incident/170653 Wed, 01 Feb 2023 20:09:40 -0000 https://status.fritztech.net/incident/170653#b87926463cc776338f3b98cbdfe68aaa89b4102081adeedc1d9f51b463dde149 TCP IPv6 HTTPS:443 went down. https://status.fritztech.net/incident/170653 Wed, 01 Feb 2023 20:09:20 -0000 https://status.fritztech.net/incident/170653#ab95cbee525918c0d6cf022b86c8884a214025749ebad5442584b008f5473230 FritzTech Blog went down.