Aftermath

Adding gateway

The grim march in the days and weeks following launch produced impressive improvements. The CDN’s engineers redeemed themselves for their “sneak preview” error before launch. In one day, they used their edge server scripting to help shield the site from some of the worst offenders. They added a gateway page that served three critical capabilities.

Handling Cookies

First, if the requester did not handle cookies properly, the page redirects the browser to a separate page that explains how to enable cookies.

Setting percentage for allowed sessions

Second, we could set a throttle to determine what percentage of new sessions would be allowed. If we set the throttle to 25 percent, then only 25 percent of requests for this gateway page would serve the real home page. The rest of the requests would receive a politely worded message asking them to come back later. Over the next three weeks, we had an engineer watching the session counts at all times, ready to pull back on the throttle anytime the volume appeared to be getting out of hand. If the servers got completely overloaded, it would take nearly an hour to get back to serving pages, so it was vital to use the throttle to keep them from getting saturated. By the third week, we were able to keep the throttle at 100 percent all day long.

Able to block specific IP

The third critical capability we added was the ability to block specific IP addresses from hitting the site. Whenever we observed one of the shopbots or request floods, we would add them to the blocked list.