Release 2.0
9
Issue 2.0.9, June 2008 http://r2.oreilly.com
Jesse Robbins, from Velocity, page 03
Jesse Robbins of the O’Reilly Radar team puts it succinctly: “You only make
06: Web Operations and Business
10:
Web Operations and Performance:
Business Principles
17: Case Study: Flickr
20: Case Study: iLike
21: Best Practices
30
:
Conclusion
31:
Appendix: The Technologies
of Web Operations
Jimmy Guterman is editor of Release 2.0 and editorial director of O’Reilly’s Radar group.
In the early years of the commercial web, sites were organized like this:
These sites were neat, clean, manageable. In such an environment, with flat files, a hierarchical structure, and not that many customers, it was relatively easy to keep sites smooth and optimized.
That didn’t last long. Amazon, eBay, and other trailblazers showed that even websites that were neat, clean, and manageable from the point of view of the visitor had to be, in fact, quite complicated on the back end. Flat files couldn’t be the rule, as any large site needed to be built on a database, rather than
More and more, people will
expect pages to load faster,
sites to have higher uptime,
and companies to deliver
more performance with
fewer resources.
HTML pages. And, since you’re asking for passwords and Social Security num-bers, you’d better be rigorous about security. The iTunes Music Store doesn’t look to Apple developers the way it looks to customers.
Anyone running a large-scale web site has at least two great worries: performance and scalability. “Another site is just a click away” has been a web- business cliché since NCSA Mosaic was the browser of choice, but much evidence suggests that the quickest way to get customers off a website is to make it slow and unreliable.
In recent years, as many large sites have launched and some of them have prospered, the art and craft of web operations has become crucial to companies that want thriving digital businesses. Companies don’t just hire “an ops person” to think about such issues. At the leading digital enterprises, web operations is embedded throughout the enterprise. It’s the foundation on which the business runs. More and more, people will expect pages to load faster, sites to have higher uptime, and companies to deliver more performance with fewer resources. The next cool web startup may well be the one that can quickly scale to serve a large and global audience.
In this special issue of Release 2.0, we look at the state of web operations, examine early signals of where it’s going, and present the industry’s best practices and most interesting players. Also available as a stand-alone O’Reilly Radar research report, this issue is a complement to O’Reilly’s inaugural Velocity conference (http://conferences.oreilly.com/velocity) for web performance and operations. Longtime IT analyst and reporter Allan Alter called on the deep experience and hard-won strategic insight of conference co-chairs Jesse Robbins and Steve Souders as he crafted the issue. With this report and conference, and in ongoing coverage on the Radar blog (http://radar.oreilly.com), we offer tools for making sure your site is one of the winners. nn
Release 2.0.9 July 2008 Velocity: Transforming Web Operations from Cost Center to Competitive Advantage
Allan Alter
You only make money when
your website is up.
When people say the Web has changed how we work, they tend to think about how people buy and sell products, collaborate and share information with co-workers, or all the new kinds of businesses that have emerged. What they often overlook is that the Web has changed something else that’s fundamental to every business—execution.
Execution, say Larry Bossidy and Ram Charan in their book by that name, is the “discipline of getting things done…the missing link between aspirations and results.” It’s understanding how to operate a business in an efficient, effective, and reliable way, knowing how to meet the expectations of your customers so your organization can meet the expectations of management and investors. In an online business—and nearly every business is an online business today— execution must include the discipline of operating websites. Only, “include” doesn’t go far enough. An online business must think of the website as one of the most important parts of a company’s operations. It’s just that critical.
Customers don’t care about the operational, behind-the-scenes stuff that goes on, such as how many servers support a site, server automation, or HTML coding. But they do care about whether the site keeps crashing, it takes a long time to download pages, or the features on the site hang up. That’s why smart executives at all Internet companies—in fact, any organization that conducts business on the web—are recognizing that reliable web operations and fast website performance are essential. With the Web now the sales channel most frequently used by U.S. companies, serious money is at stake: Amazon, Google, and Microsoft have found a lag of half a second or less can have a major impact on revenues and the number of searches. As Jesse Robbins, an O’Reilly Radar blogger and website availability expert, says, “You only make money when your website is up. The more available and the faster your website, the more reve-nue-generating pages a customer can view in the same amount of time, and the happier the customer will be.” (Robbins was responsible for website avail-ability at Amazon.com, where his title was “Master of Disaster.“)
Still, most executives don’t fully understand all the potential business benefits of a high performance, high uptime website—and the hit their business can take if they neglect web operations. Nor do they know enough about the basic principles, technologies, and management practices that separate well-run sites from the also-rans. These principles sometimes require new ways of thinking— especially about how to approach website downtime and “failure.” This report, written for business executives and managers at online businesses (or any company with a commercial website), provides a guide to understanding what web operations means, the business opportunities and risks it presents, and the best practices for operating and managing a mission-critical site.
Velocity:
Transforming Web Operations from
Cost Center to Competitive Advantage
What is Web Operations?
“Operations” has long referred to the day-in, day-out processes of a business. Chief operating officers run their organizations’ day-to-day activities; in banking, “operations” refers to running branches and processing checks or transactions. Likewise, the IT profession has been using “operations” for decades to describe running and maintaining mainframes, servers, and data centers. But the phrase “web operations” is much newer. Its earliest use dates back to 2003, when the phrase appeared as part of the name of the Internet Web Ops conference. You still won’t find many companies with a function known as the web operations department. Web operations remains an ill-defined and even controversial term: just as the phrase “classical music” means both music specifically from the era of Haydn and Mozart, and the entire European concert music tradition from Monteverdi to Stockhausen, web operations sometimes refers only to running and maintaining websites, and at other times serves as an umbrella term that also encompasses the field known as “web performance.” In this report, we’ll use the phrase web operations in its broader sense.
Operations = Availability
Theo Schlossnagle, an expert on building scalable, high performing websites and CEO of OmniTI Computer Consulting Inc., is not a fan of the phrase “web operations”; he prefers terms that give a tip of the hat to the technical skills required to run a website, such as “site architects” and “site reliability engineers.” But he does have a clear definition of “web operations” in its narrow sense: it’s “how I put the website in place, how I keep it going, how I meet the demand (if demand rises above capacity), or not eat my shorts in costs (if demand is less than capacity). As business requirements change and mutate, it’s making sure what you have in place still works well.”
Web operations focuses on availability—keeping sites up and running. Availability includes reliability: the capability to consistently download not just web pages, but the features on those pages (e.g., search, video, account informa-tion, online purchasing, chat, etc). To achieve reliability, web operations personnel set up and maintain their sites’ hardware, software, storage, and network infra-structure. They also ensure scalability as demand for the site increases, by designing an infrastructure that can grow and by preparing for the future through capacity planning. Availability also encompasses recovery: the ability to get a website back up should the site, or any individual feature, fail to be available. Part of the job of running a website is setting up redundant systems and even data centers that can take over when equipment fails, and creating plans to get the site back online when it crashes.
Performance = Response
While web operations and availability focuses on the servers, web performance concentrates on what the user sees. “It’s how we deliver our product as quickly as possible and provide a good user experience,” is the succinct definition by Google technical staff member Steve Souders, author of High Performance Websites, creator of the YSlow tool for analyzing web performance, and Yahoo’s former Chief Performance Yahoo!
Web performance focuses primarily on response time: the length of time it take to download a web page. Improving performance involves optimizing the files, instructions, and components that make up a web page. But the definition of performance includes efficiency, too: optimizing hardware, data centers, and networking to serve web pages at maximal speed with minimal resources.
Web Operations and Business
The difference between operations and performance is important to the people who do the work. But for others the distinctions often overlap—at some point, a slow page is equivalent to an unavailable page—and are ultimately not very important to the folks outside the data center. What users care about is their experience on the Web; what mangement cares about most is the bottom line.
“In business,” says Adam Jacob, Senior Partner of HJK Solutions and a specialist in establishing web operations for start-up firms, “ ‘operations’ is usually defined as the things you need to do to extract value from your resources. In the oil industry it’s what you do to extract oil from ground and put it in a barrel. It’s the same with web operations: it’s extracting the value you have in your website and what you are running on it.”
To extract that value, executives don’t need to know the intricacies of web technology. What they absolutely must understand are the three most impor-tant principles for a sound IT infrastructure. And the first of them—design for failure—requires a 180-degree change in how most IT organizations think about downtime.
1. Design for failure. Failure is inevitable: servers break down, bugs and net-work outages occur, people make mistakes. It’s better to assume that failure will happen, and design the infrastructure so it recovers quickly when it does. “A system that can tolerate minor problems and still deliver a great customer experience is better than one that delivers great experience all the time except when the page is blocked out,” says Robbins. As telephone companies and power grids do with their networks, Google, Amazon, and many of the most successful websites design their web infrastructure as a mesh of connected systems. When one system goes down—be it a web server, a database server, or some other piece of hardware—others pick up the load.
John Allspaw, manager of the Operations Engineering Group at Flickr.com, recalls a bizarre request at one of his first jobs that brought home the point for him. “My boss said I want you to take a look at this diagram of networks and servers. In three months I will walk around and randomly unplug things to see if the network still works.” A network that can survive a rambling, destructive force is a perfect allegory for how web operations should work, he says.
In the oil industry, operations
2. Design for scalability. In a scalable architecture, computing resources keep up with demand as it grows. A scalable system allows you to design or modify applications to run on multiple servers, then add servers to the ones already in place (horizontal scalability), or upgrade your current servers with more power-ful and faster ones (vertical scalability). Horizontal scalability is generally cheaper and preferable; vertical scaling makes sense when rewriting code is too costly or—if horizontal scaling is already in place—it makes more sense to replace old servers with fewer, more powerful, less- expensive-to-run servers.
By contrast, an example of an architecture that doesn’t scale is one that keeps a company’s data in a single database that can’t query other database servers. If the database hits its upper limit and it’s necessary for queries to go to multiple database servers, it’s hard to change the code.
For startup companies with a tiny operations staff, the big scalability issue is surviving sudden spikes in traffic and avoiding long periods of downtime. That’s why Sendi Windaja, CTO of startup legal referral website Avvo.com, set up an automated web operations infrastructure from the start—one where the work of adding a new server to a network can be done automatically by the network, rather manually by the operations staff. His one-and-a-half person operations
staff can add a bare metal server, load the operating system and applications, and add it to the network in 30 minutes. Downtime to deploy a new software release is usually a regularly scheduled 10 seconds.
As websites emerge from the startup phase, capacity planning and anticipat-ing the stress points on systems become the main issues. Mainstream retailers and companies have the additional burden of integrating their website’s features with their existing IT infrastructure and systems—still a big challenge for store retailers or catalog firms.
3. Design for the browser. Important as it is, a scalable architecture isn’t sufficient for achieving a fast, bug-free user experience. The reason? Steve Souders’ “Performance Golden Rule:” “Only 10–20% of the end user response time is spent downloading the HTML document. The other 80% to 90% is spent downloading all the components on the page.”
In other words, when you’re waiting for a page to load, the holdup is not the basic HTML frame of the web page, but the time it takes the browser to down-load all of the scripts, images, DNS lookups, stylesheets, and redirects that make up a modern web page.
Souders has set out 14 rules for improving front-end performance in his book High Performance Websites: Essential Knowledge for Frontend Engineers (O’Reilly, 2007). Any website can shed 25 to 50 percent off its download time by following the applicable rules, he claims. A downloadable tool by Souders called “YSlow” (which runs on the Firebug for Firefox tool) analyzes whether a web page is following or breaking these rules, and then grades pages.
Steve Souders’ 14 rules
for high performance websites
1. Make Fewer HTTP Requests.
Reducing the number of components or combining them so fewer requests are needed can reduce response times by as much as 50%.
2. Use a content delivery network.
Bringing the servers closer to users reduces the time to transmit page components over the Internet. (More below)
3. Add an expires header.
For repeat users, using components already in the browser’s cache avoids unnecessary HTTP calls.
4. Gzip your scripts and stylesheets.
Compressing HTTP responses into smaller files reduces transfer times.
5. Put stylesheets at the top of the document.
Placing stylesheets there enables browsers to draw the visible components of a page first as it downloads.
6. Put scripts at the bottom.
Otherwise, some components will be blocked from downloading; also enables components to be downloaded simultaneously.
7. Avoid CSS expressions.
The browser must evaluate them—and usually much more often than developers expect.
8. Put JavaScript and CSS in external files.
This can reduce the size of HTML documents without increasing the number of HTTP requests.
9. Reduce DNS lookups.
Reducing the number of hostnames to look up reduces the parallel downloading that must take place.
10. Minify JavaScript source code.
Removing unnecessary characters from the code makes for smaller, faster-to-download JavaScript files.
11. Avoid redirects.
Rerouting users from one URL to another makes your pages slower.
12. Remove duplicate scripts.
This creates superfluous downloads and evaluations.
13. Remove or reconfigure ETags.
ETags can thwart caching, forcing browswers to make more HTTP requests.
14. Make Ajax cacheable.
Make sure Ajax requests have an Expires header, and follow the other 13 performance guidelines.
Web Operations and Performance:
Business Principles
So how much more value can a company extract from a high-performance, high-availability website than one that’s slow or crash-prone? Can a few extra seconds of download time or minutes of downtime really matter? It turns out that it matters deeply. Web operations has a direct impact on customer satisfaction, revenues, the cost of doing business, and the ability to collect customer data; it also affects Sarbanes-Oxley (SOX) compliance and energy use. In fact, the very survival of a business can even depend on the speed and availability of its web-site. These nine principles highlight how performance and availability impact the business value of a website.
1. Speed keeps customers satisfied. Along with pleasing site design, compel-ling content, and security, site speed is one of the critical customer satisfaction factors for any website. “Without a product that’s fast, it doesn’t matter what features it has, people won’t want to use it. You can’t have satisfied customers if the product is slow,” says Eric Schurman, the senior development lead currently responsible for Microsoft’s Live Search search engine, and formerly responsible for Microsoft’s home page. To retail veteran Thomas Harden, the former COO of Land’s End and executive vice president at L.L. Bean, and the executive who over-saw both companies’ online businesses, “Speed of response time is just table stakes to get into the game. It’s one of the most significant factors causing people to defect from your website.”
Study after study confirms site speed’s importance:
n Experiments by Google found a 500 millisecond delay in the speed with
which results are returned led to a 20% drop in the number of searches. Google also found the use of mobile Gmail on Apple’s iPhones surged after the company tweaked the program to improve its speed.
n Prof. Dennis F. Galletta of University of Pittsburgh, along with other academic
researchers, confirmed delays of as little as two seconds can have a “profound impact” on the way users react to websites. Other academic studies found that sites with download delays are less trusted than faster sites.
n By reducing download times around the world by up to 65%, Cathay Pacific
Airways increased online bookings by more than 100% while saving $1.5 million as more customer booked tickets online instead of the through airline’s call center.
n One third of shoppers using broadband will leave a site after a four second
wait, compared to just 19% of dial-up shoppers, according to a sponsored study of retail websites for Akamai, conducted by Jupiter Research. The same study found 55% of shoppers who spend more than $1,500 a year online considered “pages load quickly” as one of the most influential factors in their decision to return to an online retailer.
Websites and pages don’t have to be equally fast. Users are more demanding with search engines, news, and shopping sites where there is plenty of compe-tition, and more patient on sites that provide tools they use or information they can’t find anywhere else (like their bank account status), according to Allspaw and AOL operations architect Eric Goldsmith. Galletta’s research team found users are more tolerant of slower performance on deep websites, or if they are familiar with the “terminology” on a website.
Speed is a key business metric that just keeps getting more important. Users are more impatient now that they’ve grown accustomed to broadband speeds. Competitors raise the bar by improving their sites’ performance. “Two to five years from now, we will see that speed is a primary differentiator from com petitors,” predicts Souders.
2. Performance boosts customer satisfaction. “The ‘wow!’ factor is a differ-entiator for us,” says Jay Shah, CIO of E-LOAN, Inc., which has been ranked the #1 mortgage industry website by Keystone Systems for five years in a row. When a potential borrower submits their data to apply for a loan, “we can process it, integrate third party data, and make a decision in seconds—and we relay that back to user while they are in the browser session. Then we contact the majority of approved customers within 15 minutes.”
That’s customer satisfaction—and even delight: capturing customers and users by delivering features that exceed expectations, or are amazingly useful or truly enjoyable. Speed can deliver “wow.” But the push to wow customers also poses a performance challenge: how to add functionality without harming performance?
Designers are adding browser-choking features and piling on objects to download. In the past five years, the size of the average web page has more than tripled from 93.7K to over 312K, while the average number of objects per page has doubled from 25.7 to 49.9 objects. Meanwhile, the search for new ways to delight customers continues. In one cutting-edge example, a team
of MIT researchers have come up with a way to change the look and feel of a website to a web users’ cognitive style—impulsive or deliberative, analytic/visual or holistic/verbal, leader or follower, reader or listener. This so-called site “morphing” has a huge potential to increase sales. A BT Group (formerly British Telecom) experimental website using this technology increased purchasing intentions by nearly 20 percent, which could have added about $80 million in revenue to the BT Group’s coffers.
More features need not cause slower performance. You can add more to a page and still make it faster, if you do it the right way. But any company that adds features without instituting sound practices for web performance will find its pages will load more slowly. And as Harden has found, “Customers don’t go backwards. Once they get used to a level of performance, they don’t expect it to get worse.”
Page weight Response time YSlow grade (in Seconds)
The page weight (size of page), response time, and YSlow grade for the home pages of 10 top U.S. websites, circa early 2007 (from High Performance Websites, p. 103) Complex, popular sites tend to have big home pages that don’t load quickly.
Forty-three percent of U.S. households still use dialup, according to an April 2008 recent survey by Website Optimization L.L.C. (Dial-up connections are down: a mere 4.3% of U.S. employees still use dial-up connec-tions at work.)
3. High-performing sites capture more customer information. Companies that can capture data about the activity of customers and visitors can analyze it to improve current products, develop new ones, and uncover ways to generate more income. It’s one reason spending on analytic and business intelligence tools is growing by 13.1% in 2008, faster than any other application area. Strong site performance helps companies do a better job of analysis: People will use a site more often if it’s fast and easy to use. And the more people use it, the more customer information you can collect.
Take Flickr.com: the photography website has two levels—free (which limits users to uploading 200 photos) and pro (which permits unlimited uploading for $24.95 a year and provides other benefits). Flickr constantly examines what convinces users to spend the money, says John Allspaw. “Was it the 200 photo limit? Was there a feature that made them turn? That is an exercise that is con-stantly going on.” Collecting, storing, backing up, and sharing this data is part of the operations team’s job at Flickr, he adds. (For more on web operations at Flickr, see the case study).
This online customer data is also invaluable for capacity planning. “Every time we launch a large feature, we see how the feature is adopted,” Allspaw noted. “The marketing people want to know how quickly it is adopted and its affect on revenue. We use the same metrics to figure out whether we planned for enough capacity for this feature.”
Whether it’s channeling onlne customer data to the marketing, sales, or customer service departments, or using it for capacity planning, the web ops group has to handle this data with care. They need to be sure the data is kept secure, and that policies are in place that make certain only the appropriate people can view it.
4. Speed increases revenue; downtime loses money. The connection between web performance and revenues is very simple: The faster the site is, the more pages you can view in a unit of time. Faster site, more page views, more clicks. Downtime means no clicks at all, and no clicks means no revenue from ads or sales.
There have been plenty of estimates of retail revenues lost from unavailable, sluggish, or buggy websites to back this up. They range from $25 billion a year (Zona Research, 2001) to $877 million for the 2004 holiday season (Tom Kuegler’s Conversion Chronicles blog) and £300 million in a 2007 study of 40 British retail, finance, insurance, and travel websites (SciVisum Ltd.) An academic study of a three-hour-long denial-of-service attack on Yahoo on February 7, 2000 estimated that Yahoo lost 2.22 million visits and $88,854 in click-through revenues while
under attack, and 7.56 million visits and $302,277 during the following 8 weeks. After that time, losses were negligible as visitors returned to Yahoo.
Companies rarely spill the beans on their own winnings or losses, though the information does sometimes sneak out: Greg Linden, a former Amazon.com computer scientist who wrote the site’s recommendation engine, stated in a 2006 speech (and later posted on his website) that Amazon found “every 100 millisecond delay in the time it takes a web page to load costs 1% of sales.”
Microsoft’s Schurman and Thomas Harden (formerly of Lands’ End and L.L. Bean) won’t provide actual numbers, but both confirm Robbins’ argument. “When we make a change to the Microsoft Live Search site that makes it faster or slower, we found it has a direct measurable impact on revenue,” says Schurman. “Will they use us more often if we are faster? Yes. We’ve also found if we make the site faster we’ll even get more ad clickthroughs.” Even a small improvement in site helps. “If you make the site just a tiny bit faster, we do see a tiny change in revenue. Last year we made a very substantial change of performance, on the order of 50 percent, and we saw a really substantial bump in revenue.”
Harden found the connection between site performance and revenues resembles the one between in-stock inventory and retail sales. “When in-stocks are 94% or 96% instead of 100%, the vast majority of customers are not going to feel that. But you reach a point at the high 80’s where things begin to fall apart. It’s almost like you hit a tipping point and things begin to collapse— all the customers are complaining about it. To me, site speed is much the same way. There is a point where boom, you hit a bad point and it drops off like a rock and goes to almost zero very quickly. It’s like this in call centers, too.”
5. Scalability makes or breaks online businesses. Web operations veterans in the Silicon Valley have seen it happen plenty of times: a new web business is written about on Slashdot.org or another popular site, users hit the link in droves, and its servers can’t keep up with the traffic. Some companies, like iLike, survive their brush with popularity; others like Twitter, a target for criticism over its service outages, risk losing users to new competitors.
Sudden spikes in traffic are the most dramatic example of a classic operations issue: scalability. The ability to keep up with demand for the website insures a company’s website can keep pace with growth. Besides lost revenue and customers, failure to scale damages to reputation, and provides openings to
Amazon found “every
competitors. Scalability also matters to mature companies: A website that can’t scale becomes a drag on agility; it cannot execute at the speed the business requires. “Getting the platform right is the basis for better availability, much better information for decision making and risk management, and for making agility sustainable,” says George Westerman, co-author of IT Risk: Turning Business Threats into Competitive Advantage (2007, Harvard Business School Press. $35, 256 pages) and a researcher at the Center for Information Systems Management at MIT.
While managers at larger companies are used to thinking about scale, it’s easily overlooked by developers and managers at start-ups rushing to get their first product out the door. The trick is to design scalability into your architecture from the start.
6. Strong operations practices make Sarbanes-Oxley compliance easier.
Weak web operations practices don’t just set the stage for scaling problems; they set companies up for problems with Sarbanes-Oxley compliance. The Sarbanes-Oxley Act (SOX), passed in 2002, obliges public companies listed on a U.S. stock exchange to certify the accuracy and timeliness of their financial information, and the integrity of the processes which produce it. Since data on clickthroughs, page views, and online sales and order fulfillment are part of the financial information stream, web operations are just as affected by SOX as any other part of IT involved in tracking the flow of information about financial transactions inside a company. Yet, “there’s a misconception that Sarbanes doesn’t apply to websites, even though they are generating a significant vol-ume of dollars,” according to Marios Damianides, a past international president of ISACA and the IT Governance Institute, and a partner with Ernst & Young’s Technology and Security Risk Services (TSRS) group in New York.
Under SOX, companies need to establish operating procedures, document them, show they are repeatable, and document who is involved in the procedures and who has access to the company’s IT Infrastructure. Auditors need to be able to verify all this information. This requirement isn’t necessary just for customer tracking or CRM, but it does cover web operations that are directly linked to the financial chain, just like other IT operations.
Compliance is both a blessing and a curse for web operations. Most of the common practices in a web operations organization must be rethought from a compliance perspective. Everyone must be educated on their responsibilities and underlying reasons behind them.
“The best way to approach compliance is to look at the controls as a set of design requirements.” says Robbins, “Make sure that the compliance team, engi-neering, and operations are working together with a common purpose right from the start. Otherwise you end up with a huge gap between people and process and intense conflict between risk management and web operations.”
7. The web operations cost/benefit equation changes with your business. A big-budget line item always gets attention inside a business. For online businesses, says Flickr’s John Allspaw, web operations is one of the bigger, if not the biggest expense, there is. “Every quarter, the person in finance assigned to us brings a slide with a table on how much money going in and how much is going out. At this point, everyone looks at me. The slide makes interesting read-ing: our capital expenditures is probably the largest driver of the financials every quarter.” Servers, storage, electricity, staff, and (if used) proprietary software are all costs to manage as efficiently as possible; they are part of the equation when deciding whether to use service providers to host websites or host a site internally.
Managing operations expenses while supporting growth is why capacity plan-ning is so important: the central question is how many servers and how much storage to add without buying more capacity than needed. Communi cation between the operations group and finance should be done on an almost con-stant basis, since the group knows when they will need to buy more servers.
8. Efficient websites reduce energy costs. An efficient web operation has the additional benefit of lower energy costs and CO2 emissions. According to a study by Jonathan Koomey, a researcher at the Lawrence Berkeley National Laboratory and Stanford University, 1.2% of electricity generated in the US was consumed by servers and cooling equipment in 2005. That amounted to $2.7 billion in electricity sale, or the equivalent of five 1000 megawatt power plants.
Reducing the number of web servers through virtualization or replacing them with more energy-efficient equipment can help. For example, when Flickr replaced 67 servers with 18 new quad core servers in its web operations center, it reduced power consumption by 70% in that cluster, according to Allspaw. Even a small change can make a difference. According to Steve Souders, if Wikipedia were to make one set of changes—add a far future expires header to the thirteen images on its front page—it would eliminate 52 million image validation requests daily. Those requests require six servers running full time every year, he estimates. “Assume a fully loaded server uses 100W. Six servers, year-round, consume 5,000 kilowatt-hours per year or approximately 500-1000 pounds of CO2 emissions.” For a mid-sized company in Boston, that’s $950 a year.
Case Study:
Flickr
With 26.5 million users storing 2.2 billion photos, Flickr.com, a San Francisco-based subsidiary of Yahoo, is one of the Web’s largest and most successful photo storing and sharing websites. Operating that website is a massive undertaking: it stores 2.1 petabytes of storage space (4.2, if you count storing them twice for redundancy), and operates nine data centers with a total of 1,200 servers using the LAMP stack platform with the MySQL database and PHP.
John Allspaw, manager of the Operations Engineering Group, and his staff are the people who keep Flickr running with rarely a flicker. Allspaw was already a systems operations veteran—he built the web infrastructures for Salon.com, InfoWorld.com, and Friendster.com—when he was recruited to Flickr by founders Stewart Butterfield and Caterina Fake to take over the responsibility of managing web operations from Cal Henderson, the original lead developer who wrote the code for Flickr. Since joining Flickr, his staff has grown from one to eight people, including Allspaw. (The department also includes one database admin-istrator, one search operations adminadmin-istrator, one systems adminadmin-istrator, and four web operations generalists.) IT security monitoring, testing, and auditing for web operations, as for other functions, is handled separately by a group named, aptly, the Paranoid Group. The group’s data centers make operations the largest capital expense at Flickr.
Allspaw’s focus is availability and speed. Flickr has servers in multiple geo-graphic regions to reduce network latency, and its system architecture splits data into multiple database servers for scalability and availability, should a data center go down or traffic routing problem pop up. A firm believer in the princple of design for failure—“the mindset that failure isn’t something you want to avoid, it’s something you expect to happen in unexpected ways and unexpected times,” Allspaw has made sure Flickr’s systems are frequently tested and designed to with-stand failures. “For every feature—and I mean almost every tiny little feature—that developers put into production, there is a hook in the code, an ability to turn that feature off at any moment in time, so we can continue operating the site with a reduced feature set. We can just disable uploads of photos or disable tagging of photos. Why is that important? If a feature depends on a resource that fails, it’s better not to have that feature than throw an error to the user or make things worse.”
But Allspaw’s secret sauce for running a high performing, reliable website isn’t really the technology; it’s the culture within Flickr’s operations group, the way it communicates outside the group, and the management practices it fol-lows. “Successful web operations comes from non-techical practices. Some of it is culture, some are tips or techniques, some are just good habits.”
Take the tense, fingerpointing-prone relationship between software devel-opers who write code, and the operators who manage the servers it runs on. “I am absolutely convinced that the key to a successful web operating culture is that different parts of engineering need to trust each other,” Allspaw says. At Flickr, “One of the reasons it works is that the two groups have an under-standing of what the other person’s job is. I have developers who think like web operations people and web operations people who think like the developers.” Allspaw also stresses interaction between the operations team and Flickr’s product managers, a group of largely non-technical people that manage the features of an application. “Stereotypically, you have product managers who don’t care or have no awareness of operations. You can’t just say we’ll release video. There are a lot of things to considered: storage, technology, resources, time. When product managers don’t talk to the operations group, there are a lot of problems.” Allspaw, along with the development side, is involved early on in product and product feature decisions. And because Yahoo’s technical staff reports separately from the product, design, and marketing groups, technical experts have the independence to make the technical decisions. The result is
Successful web operations
that decisions about launching products and features are made jointly between engineering and product managers.
But the relationship that affects customers most is the unusually close one between operations and Flickr’s customer care group, which interacts directly with customers and handles technical, legal, and other problems. “Traditionally, web companies have customer care or tech support people cordoned off to a different room. At Flickr, engineering operations sits one row of cubes away from the core customer care group in San Francisco.” (Flickr also leverages Yahoo’s own customer care departments on several continents.) There’s an operations staff member whose job is to handle customer-reported technical problems, but other members of the operations engineering group also converse with users about their problems. The result: “Users report one of the reasons they stay with Flickr is Customer Care, and Customer Care is strong because of the close relationship with operations.” Flickr has received praise for how well it keeps users informed about outages; ZDNet’s January12, 2008 story about Flickr’s worst outage in recent years was headlined “Flickr Down, Outage Handled Reasonably.”
The operations group’s communication strategy is key to dealing with tech-nical emergencies. “The worst situation is where the site goes down, and four web operations guys are frantically typing on their laptops and not talking to each other,” says Allspaw. “Communications in an outage is paramount above all else.” When a technical emergency strikes, one person is in charge of conducting the investigation into the problem. The channels of communication are set in advance: IRC (internet relay chat, effectively a never-ending chat room) is used for conversations within the web operations group fixing the problem. One member of operations serves as a single point of communications with the rest of company, and uses instant messaging conferences to give “informative and frequent” updates to non-web-operations personnel who need to know what’s going on. Telephone lists of company personnel are kept up to date and circu-lated to the operations staff. Any changes made during trouble shooting are tracked, so the group knows which change fixed the problem.
Finally, the entire company is in agreement on what key metrics to use in assessing web operations. The important point, says Allspaw, is not that they’ve chosen performance and availability as their metrics, but that Flickr’s C-level executives agree with the operations management staff that those are the right metrics to use.
Operations team: 3 people (manager, senior systems administrator, DBA)
Case Study:
iLike
The experience of iLike.com, a social networking site that helps musicians and music lovers share music, playlists, and favorites with other people with similar tastes, is an extreme example of the importance of scalable operations in an online business. The startup survived a brush with sudden popularity only because it had invested in automating the process of adding new servers to support its website.
Travis Cole, the senior systems administrator for iLike.com, will never forget May 24th, 2007. That was the day iLike’s music widget debuted on Facebook, and the company discovered their four web servers and two database servers were woefully unprepared to handle the traffic coming its way. The site, which had 3.5 million users before the Facebook launch, began adding 50,000 to 100,000 new users a day; within two and a half weeks the company had seven million users. On the first day, bugs and server crashes brought iLike’s widget and website to a standstill.
Day One on Facebook: as traffic soared, iLike’s servers began to crash. Source: iLike. (Note: Vertical axis is hits per minute.)
“If we hadn’t had that
Over that first weekend, the company’s CEO and staff begged, borrowed, and bought over 100 servers, and Cole and his colleagues managed to bring over 70 servers and 10 to 15 databases online—installing the operating system, config-uring the servers, then loading the application code and bringing the servers online. And while they hadn’t anticipated the tremendous rush of server traffic, they had invested in server automation so servers could be loaded and brought online automatically by the network. “If we hadn’t had that automation in place to build the servers quickly, we would have been in big trouble,” says Cole.
iLike has not only survived its initial brush with popularity, but thrived. Today, over 27 million registered users and its web infrastructure is supported by 250 servers. The site’s management also drew several lessons about capacity manage-ment. iLike seeks out vendors that can quickly deliver an order for servers, and puts plenty of spare server capacity in place before launching any new service that could generate a surge in traffic. Better to overbuild and spend extra, says Cole, than not have the site work. The other lesson is consistency—stick with one hardware or server platform as much as possible, because it’s faster and easier to add racks of servers when they are identical.
Best Practices
At first glance, web operations seems to be purely a technical matter—a question of selecting technology platforms, developing systems, and making smart tech-nology choices. Techtech-nology is important, but in reality success in web operations depends on people and organizational culture much more than it depends on tools and programming languages. CIOs and mainstream IT organizations have known this for a long time, but it can still be news for managers and staff who are motivated by technology and have relatively little management experience. They often don’t appreciate the importance of genuine top level executive sup-port, effective leadership by mid-level management, and close attention to user needs and partnership between different kinds of technology professionals— or not until it’s too late.
1. Commit to web operations as a core business advantage. In any organiza-tion, the people at the top—the CEOs, CIOs and CTOs—set the priorities and establish the culture. Web operations are no exception. “You can scale anything. You can make anything fast. The important part of this is it’s a top down way of
thinking. Website performance is a board issue,” says Robbins. When C-level executives neglect or misunderstand operations—a common problem at many web startups—basic procedures like backing up databases are overlooked, and operations only receive attention when things go wrong.
It’s easy to tell when the board or top management falls into the “doesn’t get it” category:
n Management views operations as an expensive cost center that doesn’t
generate revenue. If they can’t see the business value of operations, neither will the technicians on the operations staff.
n The CEO pays attention to operations only when the site goes down and the
company loses money. If CEOs show up when they’re upset, the operations people will sweep problems under the rug.
n Instead of designing for failure, management wants systems that won’t
break, short of a disaster, and regards business continuity as backup when a calamity strikes.
n The corner office types see technology as an enabler, not a partner. When
management takes a “here’s what I want our website to do, now go do it” approach, without understanding the ramifications of their demands, watch out.
Instead, management teams that “get it” do the following:
n The executive team understands web operations is a creator of business
value, and not a drain on it. They also realize the purpose of the ops team is to save the company money and make it more efficient.
n They make web performance a clear corporate goal. That sets the tone for the
n Performance metrics are agreed upon, understood, and followed by top
management and used by the entire company. CIOs should expose other C-level executives to web operations metrics—speed and reliability—by including them in the company’s set of key performance indicators. According to Robbins, that’s how it was at Amazon when he was there. “Bezos looked at my operations dashboard. If there was a crisis or problem anywhere in the company, he had the same view I had.”
n The CEO stays focused on business goals and evaluating her staff’s
perfor-mance, and leaves the technical decisions of which solutions to choose and how to implement them to the people in web ops,.
n The management team has good “translators” working for them, people who
understand what the technical staff needs, but can talk to C-level people in C-level terms about the risks of not investing in operations: falling revenues, less staff retention, inability to penetrate markets.
2. Built a strong web operations culture. While executive management sets the tone from the top, the managers of the operations group must build a strong culture inside their group.
At the startup stage, the developers and operators of the website (often the same person) must be determined to make farsighted technology choices and build the site correctly from the start. The developers need to know how to build fast systems for the long term. When the company grows too large for everyone to know everybody else, operations needs formal processes and com-munication. “When you see stories about a company launching a new feature and it couldn’t handle the traffic, I always assume it’s because one group didn’t talk to another group,” says Allspaw.
But whether in a new company or a century-old retailer, everyone in a strong web ops organizations knows improving the user experience is their #1 job. They are obsessed with viewing availability and performance from the user’s perspective, and know measuring user download times on their own machines doesn’t provide information on what real users are experiencing. Schurman notes how helpful Microsoft’s error reporting service has been, by sending data from users when their software crashes or freezes. “Now we had a great set of information from real users—there are these 5 bugs, but 1 bug causes many times more crashes.”
The managers of high performing web ops groups use metrics to drive the organization. “It’s a great way to get people into the same space,” says Jacob. “If everyone agrees on what the metrics should be and what to do, then everyone up and down the organization is working on the same playbook.” These manag-ers also create a culture of continuous improvement and excellence, a culture where not only the managers but the staff consider substandard work to be unacceptable.
The staffs of the best web organizations have other defining characteristics. They know the company mission and how their work fits in. Not only will they better understand the business value of their work, but they will drop the air of superiority, separateness, and entitlement that so often irritates other functions. Staff members are also willing to go beyond their job description. Because it’s a new field without clearly delineated job descriptions, “a web ops shop requires you to wear many different hats. If you haven’t seen that hat before, you shouldn’t be shy about picking it up and putting it on,” says Schlossnagle. Management must encourage every individual to feel, whatever their job description, that they have the power to make a project successful by going the extra mile, or to bring the website back online by pitching in when something breaks.
3. Understand how to overcome resistance to change. When a website goes down or service level agreements are missed, it’s the web operations staff that usually takes the heat, and works nights and weekends to fix problems. No wonder they’re wary when developers or management urge changes that may put systems and SLAs at risk. Managers worry too: they’re afraid technology-besotted staff members will get too enthusiastic about the latest and greatest. “If you have zero resistance, imagine the anarchy,” says Schlossnagle. “You need adequate resistance, or employees will adopt the technology of today and tomorrow and forget about their job.”
There are two keys to overcoming resistance to a new tool or technology. First, management must be convinced there will be a genuine business benefit. Advocates must understand what their company finds valuable—savings, cus-tomer satisfaction, revenue gains—and convincingly demonstrate the new technology will deliver it. The operations staff need to believe the change will improve the site without setting off a chain reaction of problems. Development people have to act responsibly, partnering with operations to set up a testing
When a website goes down,
process when changes are made in the website. They shouldn’t be allowed to make changes in a production system before heading home late Friday after-noon, and leave it to the operations staff to handle any problems that turn up. And while operations may not have the ultimate stamp of approval on whether something goes into production or not, they have the say on what hoops must be jumped through to get something in production. If new software isn’t ready, the ops staff should have the right to stop the assembly line.
4. Focus on user behavior and interaction with the site. Every website has to find the right balance between speed and functionality—between the mini-malism of Google and craigslist.com and the maximal richness of Myspace. It’s a tough job because user interface design is a moving target; the technologies— Flash, Javascript, Ajax—change very rapidly.
The critical point is understanding the behavior of your users, and how users interact with your site. Search sites, where users are in and out quickly, should optimize for very fast performance the first time the user views the pages. But for sites where you expect the user to stay a long time, it can make sense to optimize for faster performance with later page loads. In a webmail application, for example, downloading a script after log-in that lets users retrieve individual emails more quickly can be a good investment of time, even if downloading that script takes some extra time. Listening and observing is important to under-stand users: large companies like Microsoft and Amazon are continually doing user studies to learn what users want to accomplish on a web page, according to Jacob; Harden says online retailers hold focus-group like interviews, or sit people down at computers to study their dislikes and dislikes.
At E-LOAN, says Shah, “the funnel is a concept we use a lot. The top of the funnel is all website visitors or potential customers. And there are several measurement points and levels before the bottom, which is the transaction itself. We try to manage all aspects of the funnel along the way to maximize throughput.”
Keep in mind too that user experience must be viewed holistically; it includes other factors than availability and site speed, such as reply time to customer inquiries sent through the website, or trust that private information stays private. How well an online business communicates with customers during an outage can make or break its reputation.
5. Clarify your site’s optimization priorities. Since most performance problems lie in the browser side rather than the server side, the question of how to speed up the user experience isn’t how to optimize—Souter’s 14 rules spell that out—but what to optimize first. AOL’s Goldsmith recommends tackling first any steps that get data to the user more quickly, such as making files smaller, using cache, or using content delivery networks. Jacobs recommends running Souders’ Yslow tool, and focus on fixing not the slowest page on the site but the most commonly used slow page. “That’s the one that’s causing problems for the most users.”
As for deciding where to focus your server-side performance efforts, once you get beyond load optimization, virtualization, and capacity planning, opti-mization is primarily a matter of spotting problems before they cause servers to break down. Monitoring and alerting is critical. So is uncovering any signs that are predictors of outages. For example, says Goldsmith, servers often fail because they filled their disks with log files. A simple disk space monitor that sets off an alert when 80 percent of the disk space is filled will help. But with all the predictive measures that can be monitored—network utilization, number of requests—operations staff are best off managing by exception, focusing on metrics that change drastically when there’s a problem.
But ultimately, listening to customers and looking at your competitors are the best guide. As Harden notes, “If I was hearing something from the customers is a real problem, I’d be responsive to that first. And if I were behind in the develop-ment or operations of my site compared to competitors, I’d get to a level playing field with my competitive set before investing in anything new.”
6. Find the right metrics for your needs. There are no standards for web performance metrics. The only agreed-upon principle is that how you measure should be based on what is important to your own organization or the applica-tion. For search engines, the speed of the search results page is critical. For web mail, it’s the time to log into the system. The network graph can also be a useful indicator, since everything done on the Internet consumes bandwidth. If it takes a big dip, something is wrong.
complacent. The changing nature of the business means you have to review metrics every year, introducing new ones and culling out ones which aren’t being looked at.”
A particularly useful kind of metric is a dual business and technology metric: for example, a graph that shows orders per minute against bandwidth availability or site page-load times. That helps the web ops group identify technical problems that will affect the business before the CFO and CEO spots the resulting business problems on their own monitoring systems. Dual metrics also help when it’s time to justify new IT expenditures: “If you have this kind of dual graph, the CFO will think you are his friend. If you have graphs that couple your metrics with capacity planning, and track these graphs to revenue generation, an investment becomes a simple decision,” says Schlossnagle.
Ultimately, a good metric is practical and actionable: it helps capacity planners predict the future, trouble- shooters to understand anomalies, and busi-ness people to connect operational performance to busibusi-ness results. “It’s a metric everyone can understand and stand behind how the business is performing,” says Jacob.
7. Be prepared for incidents. Jesse Robbins, a volunteer firefighter in his spare time, has done a 180° turnaround in how he approaches website outages. “When I began my career I fought failure. I started off as the guy that would hunt down and attack people that cause failures. That’s the big mistake I made.” His approach today is consistent with the design-for-failure credo he has since embraced: treat problems as incidents, not emergencies, and handle them in a calm and well-prepared fashion.
In immature organizations, says Robbins, “the tendency is for people to rush in and get hyper-focused. What winds up happenings is a lot of activity that com-plicates the problem.” More mature organizations have a predictable process for incident management that begins with diagnosis—a controlled response where people test theories and communicate clearly—continues with mitigation and recovery, and concludes with an after-action review. This post mortem “impartially reviews the events leading up to the incident, then analyzes the root cause—not the incident that broke the camel’s back, but why we’re here in the first place and how we can fix things to be better.”
Mature organizations also track both mean time between failure and mean time for recovery (but regard the latter statistic as more important), establish availability or downtime targets such as total outage minutes, and conduct “availability exercises” once a quarter where operations staff respond to random,
progressively more complex breakdowns in the infrastructure. Testing validates the system, but Robbins argues that’s not as important as creating the culture around failure so that when this happens, people know what to do. People need to be comfortable working under stress when it’s 2 AM and they have to react to complex situations with possibly high stakes. The only way to do that in an orga-nization is by regularly conducting very realistic exercises.
8. Hire the right people. It’s a challenge to find good people to fill web operations openings. The list of skills and traits needed is long, the career path is unsettled, training and education is hard to find, and the competition to hire good candidates is growing.
The requirements for web operations and performance experts cover a broad spectrum. Web operations administrators are expected to do things that systems administrators don’t. They need to know databases and how to read and write code. Besides technical skills, they have to understand the business aspects of web operations, risk management, and Sarbanes-Oxley. “Most internet companies have a laundry list that’s unbelievably long. All the things that other people aren’t doing get lumped into web operations,” according to Schlossnagle.” And since the role is still new and ill-defined, the set of skills can’t be learned in schools.
In the meantime, what guidelines should web operations managers follow when deciding whom to hire? There are several qualities that, according to Allspaw, make web operations engineers good ones: “One, of course, is technical skills working in a web ops environment. People who haven’t worked in a web ops environment certainly can learn them, but it’s a hard leap. Two, the ability to admit they don’t know the solution to a problem. A lot of web ops is planning
People need to be
for worst case scenarios. It’s better to say I don’t know what’s going on than mak-ing excuses or pretendmak-ing like I know. They need to be able to admit to mistakes instantly so they can move on to fixing the mistake—for example, misconfiguring a server. Even the best web ops guys sometimes do dumb things.”
“Three, the ability to learn,” Allspaw continued. “Just because they don’t know technology X doesn’t mean they can’t learn it very quickly. If I had a choice between a very hungry and aggressive learner who doesn’t know many tech-nologies vs. a person who is not very aggressive learner but knows a lot, I’ll take the first guy. Four, someone who can act rationally and isn’t too excitable in an emergency or outage situation. Imagine a room full of people in their cubes, and suddenly the website goes down. All eyes are on those people to figure out what’s wrong and fix it. In those situations a web operations person needs to focus, they can’t be distracted.”
9. Hold vendors and outsourcers accountable. In an environment that’s dominated by open source solutions, managing vendors and managers may not rise to the level of a major issue. But once companies sign on to a service provider, proprietary software, or have an outsourcer write code, the usual guidelines for vendor management apply. Among the practices to have proven to be effective in a 2002 CIO Insight survey on vendor management: encourage competitive bids, require that vendors fill out requests for proposals, have skilled negotiators and legal specialists work out a contract, and strike deals at the end of the fiscal year. Once you’ve selected a vendor, use SLAs to ensure vendors deliver what you are expected, and monitor vendor performance against perfor-mance metrics.
The most important point, says Harden, is whenever you outsource, “don’t capitulate responsibility for anything to anyone. You still have to view it as part of your own business. Have the metrics to measure performance.”
Conclusion
If your company hasn’t implemented many of these best practices, there’s no time like today: the pressures to have a fast, scalable, and reliable website will only increase during the next three to five years. Online traffic will grow as more people in the U.S. and around the world get broadband access, and use smart phones to go online. In America alone, there will be 15.8 million servers by 2010, triple the number at the turn of the century.
We expect power costs to rise as the availability of electricity fails to keep pace with rising demand. driving companies to use the smallest amount of power to meet demand. Web ops staff will also have their hands full trying to maintain speedy website performance. Websites will continue to compete by adding more rich content. More of them will contain mashups—single web pages with content from multiple sources. Unless they follow guidelines for fast web pages, mashups are “doomed to be slow pages and bad user experiences,” says Souders. For mainstream retailers selling via stores and call centers with the Web, integrating online customer data with other sources of customer data will remain a challenge; “the only people who are doing it are doing so on a custom basis,” says Harden.
Fortunately, new technologies and tools will also be available to help—at least for managing servers and improving performance. More startups and established vendors are entering the utility computing field, and offering products that will provide better ways to manage virtualization. New storage technologies and methodologies like iSCSI—a way to use Internet protocol with SCSI drives— promise to provide more flexibility. The upcoming versions of Firefox and IE promise to improve download speeds than current ones; the competition between browsers to run Javascript faster is especially strong. More web devel-opers and operations staff are adopting today’s tools, which in turn will improve or be replaced and supplemented with others. “Firebug and YSlow are getting more attention, but they are just the tip of the iceberg,” says Souders. “In a few years we’ll laugh at how infantile they were.” Isn’t that always how it is with tech-nology? Still, the determinant of business success or failure in web operations won’t be the technology, but the discipline and smarts to use them by web ops groups that are properly managed and supported. nn
Appendix: The Technologies
of Web Operations
Executives don’t want to get down and dirty with the nuances of web tech nologies, nor do they need to—that’s best left to the technical staff. But execs can do a better job of providing oversight to their web operations groups, and ensuring the decisions they make will support and grow their business, if they have a basic working knowledge of the technologies and what they do. Ignorance will cause them to push their IT organization to make poor, short-sighted choices. Here, then, is an executive’s overview of web operations technologies, why they matter to their business, and the key issues they need to keep in mind as they and their staff make technology decisions.
Platforms and Platform Architectures
One of the basic decisions that every website makes at the start is “platform” or “platform architecture.” A platform can be defined as:
1. The underlying, fundamental technologies used to run websites. These “stacks” of technologies consist of the operating system, web server, database technology, and programming language. Microsoft’s proprietary platform and the open source “LAMP” stack are the two most commonly used stacks.
Microsoft LAMP (open source)
Operating system Windows/Vista Linux Web server IIS Apache Database SQLServer MySQL Programming language ASP.net Perl/PhP/Python
Release 2.0.9 July 2008 Appendix: The Technologies of Web Operations
2. A service for delivering web applications. These include social networks like Facebook which allow other companies to provide applications on their website. Services like Yahoo Data Database platform, which provides user authentication services to other websites, are also called platforms.
3. A service for hosting websites. An example is “cloud computing” hosting services like Amazon Web Services which allow companies to run their websites off of their servers.
Microsoft and LAMP. Choosing between Microsoft and LAMP isn’t a choice between fast or slow. “Choosing a platform has to do with the cost of doing business, the reliability of the system, who you can find to do the work, etc.” says Schurman.
Microsoft’s strengths, says Schurman, are that its products integrate well with each other and other products that run on that platform. In addition, Microsoft’s broad product line is supported and regularly upgraded by the vendor. Adds Jacob, “there’s a path [for integrating the technologies in the platform] and you buy into the path, and they can tell you what the path is.”
Facebook and other social networks. To attract users, Facebook, MySpace, LinkedIn, and other social networks have enabled programmers to write appli-cations that can run on their website. Months after Facebook opened up its application platform to outsiders in May 2007, over 10,000 applications were written by programmers seeking to exploit Facebook’s social network. The rea-son to choose a social network as a platform is business, not technology: the hope of generating income from the millions of people who use them, usually through advertising. According to eMarketer, 37 percent of U.S. adult Internet users and 70 percent of online teens engage in online social networking every month. However, very few Facebook applications have truly taken off: only the top few have millions of active users.
The cloud as platform. “Cloud” computing (also known as utility computing) is essentially website hosting on a massive host. Businesses like Amazon or IBM, with huge reserves of data center capacity, sell capacity to other companies along with services to run and manage websites. (Inside the cloud, there’s nothing cloudy at all: it’s still servers running software. That’s why technical people often dislike the term.) Startups have begun to use cloud computing in their early stages because it makes economic and technical sense. But at some point the eco-nomic benefits start to dwindle, and you have more control by hosting and managing your website on your own servers.
Languages, Tools and Frameworks
Developer websites are full of debates between partisans of the various compet-ing technologies. We won’t go into all of that here—for more detailed information, turn to other O’Reilly reports—but here’s a quick guide to key technologies and services for a business audience, and their relevance for web operations.
Web programming platforms. These are a mix of programming, develop-ment tools and frameworks for developing internet applications. “They hand you a way to solve all the problems. Need to use a database? Here’s how to do that. Need to do email? Here’s how to do that,” says Jacob. Ruby on Rails is an open source framework for rapid web application development written in the Ruby programming language, while the older Java platform consists of a com-bination of software and specifications that provide a way to develop applica-tions that run across different platforms. Visual Basic performs a similar role in the Microsoft environment.
Programming languages. From a web operations point of view, the main issue when choosing a programming language is the ability to change the code after the application is written—an issue when it comes time to scale or trouble-shoot. Explains Allspaw, one of the reasons people don’t write web apps in C is it’s a difficult code base for making frequent changes. Users of the LAMP stack have a choice of several suitable programming languages, most notably PHP, Perl or Python. They’ve been sprouting Rails-like frameworks to help developers: Python has Jango and Perl has Catalyst. All of them, says Jacob, can build large scale, well-architected applications. The main reason to choose them will be which language your developers are familiar with, and their relative strengths or weak-nesses for building certain types of applications.
Rich internet applications. Developers use different technologies and tech-niques to create the maneuverable images, slideshows, moving advertisements, and other interactive website features known as Rich Internet Applications. Among them are Ajax (a set of techniques using Asynchronous JavaScript and XML), JavaScript, Flash, and Silverlight. These applications can be large and take a long time to download, so websites with RIAs can take a big performance hit and frustrate users if they aren’t designed well. They also complicate the business of measuring website download times, because they start downloading after the rest of the page.
