Two days ago, I woke up to find the FAST site completely crashed. In fact, I operate a number of sites--and all of them were crashed. And to make matters worse, I woke up with a pretty bad sore throat.

It's not the first time. Over the last year or two, we've had service outages whenever there was unusually high demand on our site (often on Monday mornings right after our memo's went out) and it was usually a simple matter of having my webhoster reboot the server.

I tried several things to avoid these outages. I purchased a more expensive hosting plan (probably far more than I needed), revamped our software to optimize performance, setup downtime monitoring, purchased server auto restart services, and more. It would go fine for awhile, and then crash unexpectedly.

But this time was different. When I contacted tech support, they said the server reboot didn't work. They were going to have to bump it up to their senior support staff and get back to me. Not good.

I tried contacting them every hour or two, for an update. Each time, they told me my ticket had the highest priority level, and would be processed as soon as possible--in the order it was received.

While I waited, I racked my brain to think of something I might have done to cause it. But I hadn't made any software changes the day before, and I had gone to bed with everything working fine. Even my normal debugging tricks, were completely nonfunctional. The server was dead.

Finally, after more than 7 hours I received a note from tech support saying there was some issue on the server, that required a software upgrade, but they needed my permission to fix it. I quickly replied back, and eagerly waited a response.

Nothing.

I continued to harass them (as kindly as I could), only to get the same message. They would take care of it as soon as possible. It had the highest priority. No timetable available.

I was getting increasingly frustrated. Providentially, I had just read about another webhosting company highly recommended by an expert I respect. So I began looking into it. Their support was fantastic. And by the end of the day I made the decision to do the switch.

Normally, it's a fairly simple process. The new webhoster can usually do the transfer automatically. And as soon as you update the nameservers (which tells computers all around the world where your site is located) it just takes a day or two for that information to get around and users start switching from the old location to the new one without them even noticing. That's how it is supposed to work!

No such luck.

First, because my current server was completely crashed, it was impossible to initiate the automatic web transfer. I had to manually download every single file (tens of thousands of files) and then manually upload them to the new server. And then test and tweak all the configuration settings. That took most of a day.

Second, because nameservers update at different times around the world, I couldn't just instantly point people to the new site. I had to wait another 24 to 48 hours, and people would only be able to get to the new site gradually as their local internet providers updated their information.

I only just now myself got full access back a few moments ago. Others have had access for some time. Others are probably waiting still! And there's no way to accelerate that.

Curiously, moments after my own access to FAST was finally restored, my former webhosting provider contacted me saying before they could upgrade my server, they needed to upgrade one of my scripting languages, and needed my permission for that. Well over 48 hours from my initial request for help!

A bit late, I'd say... :)

Some Important Lessons

Through it all, I learned a number of important lessons.

The Enemy is Real
We should never underestimate the enemy. I personally believe this was a spiritual attack. We have been making major progress on our website, with new features and improvements almost every day. And I know the enemy is not happy about that.

In my devotions that morning, I just happened to be studying Acts 13:6-126 And when they had gone through the isle unto Paphos, they found a certain sorcerer, a false prophet, a Jew, whose name was Barjesus: 7 Which was with the deputy of the country, Sergius Paulus, a prudent man; who called for Barnabas and Saul, and desired to hear the word of God. 8 But Elymas the sorcerer (for so is his name by interpretation) withstood them, seeking to turn away the deputy from the faith. 9 Then Saul, (who also is called Paul,) filled with the Holy Ghost, set his eyes on him. 10 And said, O full of all subtilty and all mischief, thou child of the devil, thou enemy of all righteousness, wilt thou not cease to pervert the right ways of the Lord? 11 And now, behold, the hand of the Lord is upon thee, and thou shalt be blind, not seeing the sun for a season. And immediately there fell on him a mist and a darkness; and he went about seeking some to lead him by the hand. 12 Then the deputy, when he saw what was done, believed, being astonished at the doctrine of the Lord. and the story where Paul encountered a sorcerer named Barjesus opposing his efforts to reach Sergius Paulus, the leader of Cyprus. The enemy hasn't changed a bit.

The Lord is in Control
While this was a major inconvenience for many members, and a big frustration to me personally, I believe the Lord will use it for good. Because I couldn't automatically transfer my site, and had to do it manually, I was able to make dozens of backend changes I would not have gotten around to for a long time.

Those fixes also inspired me to accelerate my timetable on several other big changes. I put together a long list of site goals during ROCKET and scheduled them out over most of 2018. And I've been steadily knocking them out one at a time. But this experience got me thinking that the best way to prevent another attack is to just accelerate my progress. If I do something big every time the enemy interferes, maybe he'll decide it's counterproductive! :)

So I'm doing that. I've completely revised my timetable and will be moving the biggest and most important changes to the front of the line.

Our Community is Awesome
A third thing that stood out to me, is how wonderful our community is. Despite the fact no one could get to our site for 1-3 days, I did not receive a single negative complaint. In fact, I received dozens of supportive messages via email, text, facebook, etc. They reminded me to breathe, they told me they were praying, and they affirmed their support. It was amazing!

One member pointed me to the story of the three Hebrew boys thrown in the fiery furnace, and that the only thing burned was their fetters. At that moment, I was deep in the process of deleting thousands of unnecessary files, so I could focus my attention on just the most important data. It completely changed my perspective: we would come through this stronger, losing nothing but the stuff holding us back.

In Conclusion

It was a harrowing experience indeed. The enemy's attacks are never pleasant, but God is in control, and our community is awesome! Thank you to everyone for your support, prayers, and encouragement these last two days. And in case you are wondering, I'm finally starting to get over my cold. :)

I offer my sincere apologies to everyone who regularly uses our tools and training resources, and will do everything within my power to prevent this from happening again. But I trust, we will come out of the whole thing stronger than ever. And I'm eager to get going on the changes just ahead!

Comments

Have you seen the enemy attack in your life? How important is it to redouble our efforts when we face obstacles? What would you like to see us do at FAST to send a strong message we're only going to accelerate...

Join the Discussion!

Want to read more articles, or leave a comment of your own? Please login. Not yet a member? Why not join our community...

JOIN LOGIN