[pianotech] OT: About my webhost - NOW if we could only get this kind of info from our own HQ .....

[Duaine Hechler]

Simon Says... (Written by CEO Simon Anderson!)

As Brett has said, January was not a good month for our level of service. And on behalf of all of us here at DreamHost, 
I want to apologize and let you know what the issues were, and what we're doing to fix them.

First up, security. We're now a mid-sized global web hosting company with over 300,000 customers and 1.2 million domains 
under management. That means we get our fair share of hack attempts from around the world. Sometimes it's just kids 
looking to prove themselves, other times it's more malicious than that - people wanting to take down customer blogs or 
websites because they don't agree with the opinions expressed on them; people wanting to steal other folks domains, that 
kind of thing. We put a lot of people and dollars into our security software and systems and protect our customers from 
these attempts every day. But in January a hacker got through our defenses and we took the rapid action of resetting all 
customers' FTP/shell passwords to prevent any malicious activity. In the process we identified a whole slate of 
improvements to our security systems, software and data management that we've already been implementing. Security is one 
of those issues that we can't give too much detail on, but rest assured we're on it and have upped our game 
substantially so that this kind of event is prevented in future. And as outlined in the newsletter below, we're also 
asking you to participate by using strong passwords and other best practices.

The second issue in January was related to our network. Given our unlimited bandwidth policy for shared, VPS, and 
dedicated customers, we have a network that has been designed for speed using tried and tested network equipment. 
However, the complexity of our routing and switching systems has increased as our global customer base has grown. As a 
result, we experienced some unplanned network downtime in January due to network configuration errors, plus the testing 
of some new distributed denial of service (DDoS) equipment. We've already been in the process of a focused effort since 
late last year to upgrade and streamline our network for the next 5 years of customer needs. But these issues have 
accelerated our planned efforts in this area. We'll be investing over 1 million dollars in network upgrades in 2012, 
including a new US east coast data center to better serve our European and east coast customers.

Third, we had an issue with software updates in January that resulted in some service downtime. The issue arose because 
of the way we have approached security updates for the operating system that runs on most of our servers. We've 
typically tested security updates to ensure they install correctly, but our testing of whether they break anything has 
typically been "light" due to the fact that we haven't seen any such issues in the past. Well this has proved to be a 
flawed approach, because one such update in January went badly wrong, and automatically removed some critical software 
on production servers, causing downtime and an understandable amount of customer angst. We've immediately changed our 
software update process to fully test all software going into production, and we'll be acutely focused going on ensuring 
that any necessary software upgrades are performed efficiently and quickly in planned maintenance windows going forward, 
with minimal downtime.

Finally, a quick word about our team. We have very dedicated, smart and super friendly people on our support, systems 
administration, development, engineering and data center operations teams. They work long hours and are absolutely 
passionate about delivering you an awesome and flexible web hosting service at a great price. When there are service 
issues like we've seen in the last month, they pull out all the stops â?? nights, weekends, 24/7 â?? to fix the issues 
and plan out changes to give you a better experience for you in future.

I'm fully committed to ensuring we deliver a reliable, flexible and useful service for you going forward. And I believe 
we're adapting and focusing on the right priorities and investments to continue to deliver a great hosting service and 
experience.

Best,
Simon
@DreamHostSimon


-- 
Duaine Hechler
Piano, Player Piano, Pump Organ
Tuning, Servicing&  Rebuilding
Reed Organ Society Member
Florissant, MO 63034
(314) 838-5587
dahechler at att.net
www.hechlerpianoandorgan.com
--
Home&  Business user of Linux - 11 years