Release 2.0
12
Issue 2.0.12, April 2009http://r2.oreilly.com
15:
Analytics in Action: A Conversation with LinkedIn’s
Analytic Team
By Ben Lorica & Roger Magoulas
19:
Advanced Meter Infrastructure (AMI)—Smart Grid
and Smart Meters
The “Radar” analogy is a pretty good fit for what we aim to do at O’Reilly—scan for faint signals about new disruptive technologies, and amplify those that matter. In this issue, Ben Lorica and Roger Magoulas, researchers in the O’Reilly Radar group, explore several blips on the Radar that herald interesting and important changes in our world.
First, Ben looks at how mobile phone adoption is driving economic advancement among the millions of “unbanked” people in developing coun-tries where mobiles are the primary Internet access device. In “Mobiles and Money in the Developing World,” he concludes that the new banking services developed for this huge new market not only broaden opportunities for first-time customers to enter the economic mainstream, but they also point to innovative approaches that may help revive the global financial system.
In “Our Search for Intelligent Software Agents,” we evaluate a range of tech-nologies that can enrich customer experience and enhance customer service initiatives, including recommendation engines, rule-based systems, machine-learning tools, and virtual worlds. While we found that many of these promising new technologies were not quite ready for prime time, we did tease out useful principles and best practices for implementing them now and in the future.
We continue to track Big Data, the theme of our last issue. In “Analytics in Action,” we ask the data analytics team at LinkedIn about collecting, analyzing, and getting new product ideas from the data generated by their more than 40 million users. Their responses offer insight and advice for anyone who believes, as we do, that data is a core strategic asset in our Web 2.0 world.
Energy conservation technology and sensors converge in the Advanced Meter Infrastructure (AMI), a system that has the potential to change the way we buy and sell power. Roger provides a short survey of how the AMI works and what we can expect as it gets implemented. Expect more coverage of green technology in future issues.
What’s on your Radar? Send your ideas and requests for topics we should cover to r2@oreilly.com, and keep an eye on the Radar blog at radar.oreilly.com for more on faint signals that are growing stronger. nn
Sara Winge is VP of the Radar group at O’Reilly.
Close to 60% of all mobile phone subscribers reside in the developing world. With the fastest growth in new mobile phone subscribers coming from those countries, the mobile phone is fast becoming the “PC of the developing world”. In developing countries, the mobile network infrastructure is superior to exist-ing fixed-line networks, and users are goexist-ing straight to mobile to get online. The numbers are impressive: at the end of 2007 there were 280 million mobile phone subscribers in Africa; India had just over 360 million at the end of January 2009. Vodafone-controlled mobile provider Safaricom, is the most prof-itable company in East and Central Africa. It’s no surprise that companies and technologists in the developing world are producing innovative mobile appli-cations and business models—they serve the majority of users, and they’re starting from scratch, not building on top of legacy products.
And far from being localized, the mobile applications and enterprises pio-neered in developing countries will be mimicked elsewhere. As American tech-nologist and frequent traveler to Africa, Erik Hersman, is fond of saying: “If it works in Africa, it will work anywhere.”
with bank and debit card accounts. Something as simple as a bank account that allows direct deposit is critical to workers seeking employment with established companies. Large employers reluctant to disburse salaries in cash can deposit mobile airtime or currency directly into the virtual accounts of mobile bank users. A case in point is a recent startup founded by American researcher Nathan Eagle: txteagle matches multinational companies with African workers who have mobile phone access. txteagle workers perform simple tasks and get paid with airtime or deposits into a mobile wallet. In recognition of the poten-tial impact of mobile banking, the Bill and Melinda Gates foundation recently announced $12.5 million in grants for market and regulatory research projects that help expand mobile banking services. The long-term goal of the founda-tion is to encourage savings by providing the unbanked simple mechanisms to start saving. Savings lead to risk protection (“emergency fund”), asset accumu-lation, and funds for education and other basic needs.
Mobile phone users in developing countries
Mobile banking is loosely defined as conducting formal financial transactions with a mobile phone. It’s an extremely competitive space pitting mobile opera-tors against each other and financial services companies. It’s useful to distin-guish between mobile banking—mobile phone access to existing bank customers—and mobile banks—financial institutions that arose with mobile phones. Before describing mobile banks in developing countries, a few factors about mobile phone users in those countries are worth considering. As noted earlier, large numbers of users do not have a bank account and use “pay as you
go” plans to prepay for mobile minutes. Some estimates place the number of prepaid mobile subscribers at more than half of all subscribers in the world. While they pose less of a credit risk to operators, prepaid subscribers also tend to spend less on high-end mobile (data) services. Because of its low cost com-pared to phone minutes, text messaging (particularly SMS) is extremely popular and many mobile banking features introduced in developing countries use SMS to verify transactions. Hannes van Rensburg, CEO of mobile banking technol-ogy provider Fundamo, believes that its instant feedback mechanism and wide-spread use make SMS an ideal entry point for mobile banking. Finally, the econ-omies of many developing countries depend heavily on remittances (both overseas and domestic workers’ transfer of funds to relatives in their home countries). Estimates of the share of workers’ remittance range from a few per-centage points to as much as 10% of the nominal GDP of some developing countries, with an increasing portion flowing through mobile banks. As of mid 2008, researchers from the TowerGroup estimated total global mobile-based, cross-border remittance at $320 billion annually. With these factors in mind, we grouped the services offered by mobile banks into a few categories (see Sidebar: Popular Mobile Banking Services).
Popular Mobile Banking Services
Transferring mobile phone air time
The goal of mobile operators is to make it as convenient as possible for prepaid subscribers to “recharge” or “top-up” the minutes on their phone. Users who want to top-up their own phone typically either send an SMS message or “press a button”. In develop-ing countries, senddevelop-ing mobile phone minutes to friends and family is also popular. Smart Telecom of the Philippines lets SmartMoney account holders send mobile air minutes to other Smart mobile phone subscribers. Available at designated outlets, SmartMoney is a reloadable payment card accessible with a Smart mobile phone that can be used at any outlet that accepts MasterCard. Working with mobile operators in multiple countries, Singapore-based Transfer To takes it a step further by inter-connecting the prepaid services of carriers. Transfer To works with mobile operators in developing countries to tailor services to particular local markets and specifically targets expatriate workers from developing countries who wish to reload minutes on phones back in their home country, a process known as cross-border top-ups.
Financial Inclusion: For the unbanked, the ability to send minutes back and forth turns the mobile phone into an informal savings account and fund transfer service.
Money transfers using mobile phones
cus-tomers access a menu-driven application, built into their SIM card, that allows them to send funds to other mobile phone users. If the receiver is another M-PESA user, the funds get added to his account. Otherwise, the virtual funds can be cashed in with any registered agent using a secret code and an I.D.
At the end of 2008, M-PESA had over 5,000 registered agents in Kenya and 5 million registered users. (In contrast, four major local banks in Kenya had 750 branches and 3 million accounts combined.) In October 2008, over a hundred million dollars was transferred through M-PESA.
Other examples target remittances from expatriate workers. The leading mobile operators in the Philippines have money-transfer services designed to appeal to workers who remit money on a regular basis. Globe Telecom’s GCASH charges a nominal fee to the expatriate money sender, but charges no fee to recipients in the Philippines. GCASH senders go to registered agents and apply for GCASH remit transactions. GCASH recipients are notified via SMS and present an I.D. and a reference number at registered GCASH outlets. Globe Telecom has registered outlets in countries with large numbers of overseas Filipino workers. About one hundred million dollars flows through the GCASH system daily.
Financial Inclusion: Being able to cheaply send, receive, and retain money turns the mobile phone into a zero-interest sav-ings account, although it’s important to note that many services have strict limits on maximum account size (e.g., M-PESA, GCASH, SmartMoney, WIZZIT all have limits). Popular services like M-PESA expand access to money-transfer services by having registered agents in small villages. With traditional money transfer services (and banks) limited to larger towns, users have flocked to M-PESA.
Mobile Wallets and Payment Systems
Regardless of what technology is used, electronic payment systems give the unbanked access to financial transactions nor-mally reserved for bank/debit cardholders. In the Philippines, Smart Telecom’s SmartMoney can be used anywhere MasterCard is accepted. SmartMoney users purchase items by sending an SMS message containing the seller’s merchant number and payment amount. A non-trivial portion of the Zambian GNP flows through the Celpay mobile payment system. After activating their mobile wallets by visiting participating banks, users of eTranzact in Nigeria and Ghana conduct cashless purchases either through SMS or a mobile browser.
Realizing that payment systems can help cut costs and enhance customer satisfaction, companies in developing countries are turning to technology providers who specialize in mobile banking and payment systems. Ten-year old South African com-pany Fundamo is a leading supplier of technology that enables mobile operators to offer financial service products including mobile wallets and mobile banking. Fundamo has deployed their technology over forty times in locations including Africa, the Middle East, and Pakistan. The Celpay system in Zambia uses Fundamo’s technology, as does South African mobile operator MTN, to offer mobile banking services in several African countries. Florida-based Yellowpepper’s platform includes mobile wallets and payment systems aimed at banks, mobile carriers, and companies in Latin America.
Financial Inclusion: The unbanked enjoy benefits usually limited to bank/debit cardholders, including direct deposit, auto-matic bill payment, and cashless purchases. Rather than spend hours in line to pay their bills, consumers pay them instantly. For small merchants, this means not having to carry large amounts of cash when they go to the nearest bank to deposit their earnings.
Banking and Brokerage Services
Mobile phones can provide access to financial information lookups (stocks/bonds, exchange rates) and account and broker-age services. Hello Money, Barclays Bank’s mobile banking service in India, allows customers to perform a wide range of banking transactions: check account balances, transfer funds between accounts, list recent transactions, activate and automate bill pay-ments, and file service requests (e.g., new checks, PIN). In addition, customers can top-up their prepaid mobile phone and pay specific charges on their credit cards. Enabling transactions through mobile phones can result in substantial cost savings for the financial institutions, freeing up funds to target the unbanked. ICICI bank in India needs just five employees to manage 250,000 transactions per day on its online trading platform. Cost savings reaped through the use of mobile technology have allowed ICICI to target previously unbanked customers, resulting in four million new customers in 2008. According to Hannes van Rensburg, some banks have started agile new business divisions charged with establishing mobile banks. Instead of merely pro-viding mobile access to existing customers, these are entirely new entities, created to be more accommodating to unbanked consumers and, because they are “branchless banks,” they are free from some of the business rules that hamper their parent bank.
Challenges
Broadly speaking, the main challenges facing mobile banks are interoperability, cultural issues, regulatory environments, and security and privacy. Mobile usage patterns and behavior vary across countries, and the most successful mobile banking initiatives tend to establish a strong foothold in one country or region before expanding to others. Safaricom’s M-PESA was well-established in Kenya before they announced plans to roll out services in other countries. To ease their entry into new markets, mobile banking technology providers Fundamo and Yellowpepper work with local partners in specific regions.
In some countries, strict rules help regulators keep pace or even stay ahead of the industry. For example, mobile banking in India is subject to stricter con-trols compared to other developing countries. Mobile banking in India is tied to KYC/AML-compliant (see below) bank accounts and credit card accounts. Moreover, since transactions are required to be based in Rupees, overseas remittance services aren’t permitted. In other countries, successful mobile banking services are belatedly subjected to regulatory scrutiny. In late Decem-ber 2008, the Executive Director of the Kenya Bankers Association complained that Safaricom had an unfair advantage in that it behaved like a bank but did not declare itself as one. Around the same period, the Kenyan Finance Minister ordered an audit of M-PESA, which some in the local media attributed to lobby-ists for the banking sector. We often hear people in the mobile banking indus-try declare that they shouldn’t be subject to banking rules “since they don’t award interest on deposits.” What they fail to mention is that banks are closely regulated for another important reason: customers assume that their funds can be withdrawn at a moment’s notice. For this reason, we predict the large amounts deposited in M-PESA and other successful mobile banks will be sub-ject to strict capital controls that ensure adequate reserves. Even when regula-tions governing mobile banks are in place, the enforcement of laws in resource-strapped countries is another matter.
As with other systems that permit money transfers, regulators need assur-ance that mobile banks can’t be readily used for money laundering (among bankers this is referred to as AML or “anti-money laundering” controls). To dis-courage money laundering, banking regulators in each country have specific rules regarding the identity of customers (bankers call this KYC or “know your customer” controls). Critics of mobile banks point out that mobile users who opt to prepay minutes tend to be subjected to fewer identity checks, increasing the possibility that such services will be used for money laundering. In practice, mobile banking services that work closely with bank regulators have KYC
trols that are comparable to traditional banks. In addition, compared to Internet systems than can easily hide behind proxies, mobile money transfers are easier to audit and trace making mobile banking less attractive for money laundering.
In both mobile and online commerce, security requires the right mix of technology and business processes: when properly designed, mobile banking security and privacy is at least comparable to online banking. Mobile phones receive and send information either through SMS, USSD (“SMS with sessions”), or WAP/GPRS. Mobile banking applications are either installed on the device or SIM card, accessed through a mobile web browser, or based on SMS/USSD. A popular option for deploying mobile banking, the SIM card is controlled by mobile operators and contains the user menus and security keys used to encrypt network traffic (STK or SIM toolkit). Aside from PINs, some banking services rely on a secure digital ID stored in either the device, the SIM card, a memory card inside the device, or on an external device. Regulations governing security also vary across countries. Among the most stringent are regulations in India that require mobile banking services have “end-to-end application level encryption” (eliminating the possible use of SMS and USSD). Regardless of the security systems in place, users need to be educated about fraudulent practices as popular scams employ old-fashioned social engineering: smishing, the SMS equivalent of email phishing, is a popular method for stealing credentials and assets.
Mobile Banks and Financial Reform
The potential market is enormous: only about a third of the three billion mobile phone subscribers have a banking relationship. Many researchers expect growth to continue at a steady pace for years to come. Juniper research predicts that mobile banking users will grow tenfold from 2007 to 2011. Berg Insight forecasts that by 2014, five to twenty percent of all international money transfers will be routed through mobile banking services. While we think the global economic crisis may slow down the growth of mobile banks, the trajec-tory and long-term trend is clear: mobile banks will continue to grow steadily in the developing world. Mobile banking transaction costs are low compared to other financial services providers. Popular mobile banking services in the devel-oping world work on simple and cheap handsets, making the cost of entry more manageable. To the unbanked, the convenience and benefits that come with mobile banks are too compelling to ignore. Finally, Hannes van Rensburg believes that, especially in a period of economic crisis, consumers will continue to want easy, real-time access to financial information including current exchange rates, prices, or the balance on their virtual wallets.
Van Rensburg also believes that the success of mobile banking in the devel-oping world is “…teaching the rest of the world how financial services should work.” The global economic crisis has led to numerous calls for reforming the financial services sector in major industrialized countries. The idea that policy makers in the industrialized world should draw inspiration from developing countries may sound audacious, and while financial experts have yet to con-sider it, other thought leaders have. When asked what advice he would give to the CEOs of the world’s largest banks, Harvard University Management Professor Clayton Christensen recently suggested they “…go to the developing world and buy a phone company!” Mobile payment and banking systems have been so disruptive in the developing world, Christensen believes that Western banks would benefit immensely from studying them closely.
At a time when troubled banks are lobbying for minor changes and as little regulation as possible, mobile banks have much to teach our own struggling financial sector. The technology and business processes that power the best mobile payment and savings systems are impressive. Unencumbered by legacy systems and business rules, companies in the developing world have built tech-nology and business systems that leapfrog those in the developed world. While it may be easy to start a small-scale mobile payment system, the most success-ful mobile banks use complex software systems designed to securely handle many more near real-time transactions than traditional banking systems. For
financial services companies seeking growth opportunities, mobile banks are skilled at developing mass-market products for the expanding population of mobile subscribers. Large Philippine banks initially viewed GCASH with suspi-cion, but some eventually realized that partnering with mobile banks gave them access to popular technologies and to consumers in remote areas of the country. The third largest bank in the country developed a service with Globe Telecom, which allows customers to transfer money between their bank and GCASH accounts. More recently, the Rural Bank Association of the Philippines has partnered with GCASH to introduce services for customers of their network of small banks (e.g., Text-A-Deposit, Text-A-Withdrawal, Text-A-Payment).
Mobile operators designed their impressive new financial institutions to accommodate the unbanked. Reviving the global financial system may require a similar emphasis on institutions that cater to the needs of the working class majority. From what we’ve seen in the short history of the industry, products designed to appeal to the unbanked can quickly influence the banking indus-try. The success of GCASH coupled with Globe Telecom’s sustained business development have convinced Philippine banks that partnerships can benefit both banks and telecom companies. With their control of the device (via SIM card) and the network, mobile operators in the developing world are poised to play a prominent role in reinventing global finance. In the meantime, the mobile banking industry remains committed to expanding access and to the development of new products. A recent example involves mobile banks and microfinance outfits in developing countries who are joining forces to give users access to low-interest consumer loans and mobile credit cards. Savvy operators are also starting to look at usage data: Van Rensburg tells the story of how a mobile banking money remittance service within an African country used data mining techniques to further optimize the location of its hubs. nn
Recommended Reading & Viewing
Banking on Mobiles: Why, How, for Whom? (from CGAP.org http://bit.ly/tULM2) Mobile Banking Blog (http://bit.ly/ogWtb) Reserve Bank of India: Mobile Payment in India—Operative Guidelines for Banks (http://bit.ly.gXg5L)
Mobile Phone Banking and Low-Income Customers: Evidence from South Africa (from CGAP.org http://bit.ly/WlY6) GCASH and Rural Banking in the Philippines (http://bit.ly/bPyh4, http://bit.ly/ArAk) Clayton Christensen’s Advice for Jamie Dimon (http://bit.ly/1zzf92)
Release 2.0.12 April 2009 Our Search for Intelligent Software Agents Ben Lorica
Our Search for Intelligent
Software Agents
by Ben Lorica
The O’Reilly Research team recently looked at how software agents could help improve the experience of online customers and increase the productivity of customer care employees. We took a broad view and examined specific technologies such as recommendation engines, rule-based systems, popular machine-learning tools, and emerging platforms including virtual worlds. After talking to a range of experts, including academics, early-stage startups, and established companies, we came to the conclusion that software agents (and related technologies), while intriguing, are still of limited use in this early stage of development. From the interviews and our experience with many related systems, we did, however, glean key principles for successfully implementing agents. We’ve listed these principles below and included a few stories of com-panies and projects that exemplify them.
Domain expertise is important: Effective systems tend to combine domain expertise with clever algorithms. When we looked at customer care systems, those developed in close coordination with frontline employees and users worked best. Training and instructional software developed without consulting pedagogy experts in the specific field covered fared poorly. I’m reminded of a recent story of a financial consultant (working in catastrophe insurance) charged with modeling losses due to hurricanes. Lacking knowledge in basic weather modeling or fluid mechanics, the consultant devised risk models that drew only on techniques from quantitative finance. Using the recent disaster in credit derivatives as a guide, his (hurricane) risk model may work over the short term, but chances are it will eventually lead to large losses.
Integrate into workflow and pay attention to the UI: Currently being used by a third of all U.S. physicians, Epocrates is a medical decision support system available on many handheld platforms. Recognizing that doctors have sporadic computer access because they move around constantly, and that doctor-patient interaction would be adversely affected if a doctor was staring at a computer screen, Epocrates targeted handheld devices. Epocrates software designers also produce intuitive and easy-to-navigate user interfaces for each of the major handheld platforms.
Don’t overlook rule-based systems and checklists: Simple business rules can be used to develop static decision trees that cover many routine situations. Epocrates relies on its staff of MDs and pharmacists to constantly improve its purely rule-based decision support software for physicians. Physicians use Epocrates software to help diagnose and research ailments as well as to research prescription drugs and alternative medicines.
When a rule-based decision support system is too costly to develop, a sim-ple checklist can still make a difference. Peter Pronovost, a critical-care specialist at Johns Hopkins hospital, assembled a checklist to help prevent line infections, a common occurrence in many intensive care units. Pronovost made two key decisions: (1) he decided to create a paper checklist rather than try to cobble together a software system, (2) his checklist was narrowly focused to line infec-tions. After convincing his hospital’s administrators to adopt his checklist, the 10-day line infection rate went from 11% to zero.
Data, data, and more data: Simply put, many algorithmic systems improve as more data becomes available. Recommendation engines require a minimum amount of data to get going, and only after accumulating more data do the sys-tems start generating great recommendations. In the course of developing tools for users (e.g., language translation, image processing), Google research-ers allude to “…a threshold of sufficient data” beyond which their algorithms start producing great results.
Text-mining algorithms are useful, but have known limitations: We are avid users of text-mining tools (e.g., classifiers, topic models, clustering, summa-rization, sentiment analysis) and frequently recommend them to others. Unfortunately, many users don’t appreciate the limitations and metrics used to evaluate popular text mining algorithms (e.g., how much does an algorithm depend on a particular domain, or on linguistic rules and syntax?). We don’t think having a resident NLP expert is necessary, but a basic understanding of their assumptions and weaknesses is important.
Humans are the “last mile:” For best results, combine machine-learning and human editors. Human judgment can nicely complement an automated sys-tem. Algorithms and rules can produce acceptable performance levels, but humans can help turn good systems into great ones. Even when you intend to use pure machine-learning tools, you’ll likely still need humans to help build large training sets. We categorize technical books along several dimensions and use machine-learning and rule-based classifiers to suggest categories. The results are important enough that we use domain experts (O’Reilly editors) to make the final call. Google News is a machine-generated news aggregation ser-vice, while the more influential Techmeme news service uses human editors to monitor and occasionally overrule algorithms. Image tagging (especially porn vs. not porn) is an area where humans are still widely used: in contrast to search engines that rely on algorithms, the more cautious social networks vet every image that gets uploaded onto their sites.
Turks to the rescue: Pioneered by Amazon, mechanical turks are Amazon account holders who agree to perform simple, repetitive tasks for a very small fee. Many sites use turks for image tagging (porn vs. not porn), or text tagging and categorization. Aimed at harnessing East African mobile phone users, txteagle untethers its mechanical turk service from the PC and delivers tasks and payments through the mobile phone. Dolores Labs is a San Francisco- based mechanical turk company that rates their contributing turks and pro-vides companies with an intuitive user interface for defining the tasks they want performed. To improve quality of results, both Dolores Labs and txteagle make it easy to specify the number of turks for each individual task. It turns out that for the simple labeling tasks most associated with mechanical turks, research indicates that accuracy increases rapidly with the number of turks employed: the aggregate work of 4 non-experts closely approximates the work of an expert. Turks from Dolores Labs have helped build training sets for some of our classifiers.
Purpose turns players into mechanical turks who help improve machine-learn-ing algorithms.
Many social networks have been early adopters of the Big Data management tools we described previously (see Issue 2.0.11, February, 2009). Some have begun analyzing their massive data sets. We recently spent a few hours with LinkedIn’s analytics team and were impressed by their willingness to learn and experiment with new tools and techniques, and their increasing role in product development within the company. In a period of relatively high unemployment, sites like LinkedIn can draw even more users if they can facilitate job searches. Examples of how analytic tools have helped job seekers and other LinkedIn users are mentioned in the following interview. Another interesting direction is to aggregate the millions of job histories and look for patterns. LinkedIn’s analytics team is starting to use their data sets to detect macroeconomic trends, an effort that may lead to tools for policy makers.
Unlike many analytic organizations that tend to separate analysts and quants from computer scientists, LinkedIn’s team includes statisticians, text miners, physicists, mathematicians, and computer scientists, who work side- by-side on numerous projects. The size of their data means the team works routinely with Big Data tools including Hadoop, MPP & key-value databases, statistics packages, and visualization toolkits. When suitable tools aren’t avail-able, the team is comfortable building their own. We saw an impressive (but unreleased) interactive visualization tool (complete with transition probabili-ties) designed to help users assess optimal paths to specific job roles. A mem-ber of LinkedIn’s analytics team heads Project Voldemort, a popular open source distributed key-value storage system.
R2: What types of people do you currently have on the analytics team? LinkedIn: We look for the rare combination of technical capability and cre-ativity. Most importantly we look for the ability to analyze a problem and so far we have found these people from all sorts of backgrounds. Our team includes individuals with backgrounds in physics, statistics, math, and computer science.
R2: How did you recruit for this group?
LinkedIn: Luckily at LinkedIn we have a pretty good working knowledge of our product which allows us to reach exactly the people we want to target for job opportunities in analytics. We use the tool just like other users and compa-nies. We post jobs and InMail individuals who appear to have the right back-ground for the areas we want to grow in. Also, we occasionally receive messages from users who inquire about positions in our group. We’ve also started up an internship program for both undergraduate and graduate stu-dents as well as occasional collaborations with academic research groups.
Analytics in Action:
A Conversation with
LinkedIn’s Analytics Team
by Ben Lorica & Roger Magoulas
These research projects include individuals from computer science depart-ments at the forefront of social network analysis and groups from business schools trying to better understand labor flows.
R2: What surprising trends have you uncovered in your data?
LinkedIn: One of the interesting things we have looked at is what new job titles are appearing. We have job history data for many users going back to early in their careers (people list jobs from many years back). Recently we have seen a surge in titles related to the gaming industry (e.g., ‘3D artist’). Using our tools, we can also go back in time and see the advent of software engineering (and the evolution of software engineering titles). Other interesting trends include the recent decline of the real estate industry. We can also use our tools to highlight the rise and decline of the Internet and telecommunications indus-try from 1996-2002. In many ways LinkedIn is getting closer to becoming a good gauge of the world economy, and by looking at the data we can help identify which industries are underserved. In the future, we envision using this data to help guide young people on career paths they should consider pursuing.
R2: What is LinkedIn’s philosophy towards analytics?
LinkedIn: LinkedIn has built up a considerable user base (40 million and growing) and with that, a tremendous amount of data about how users interact with the site. We are using this data to become more user-centric in our design approach and product ideation. For example, by observing the session-level behavior of millions of users, we can identify how to better improve the search experience on LinkedIn and subsequently measure the impact of our efforts. In addition to iterating and improving basic site-wide metrics (primarily visits and engagement), we invest heavily in trying to find insights from our data that can point to new products to develop or areas to focus on. For example, we have extensively studied variations in user behavior based on location and have tried to identify what, if any, approaches we need to rethink as we expand interna-tionally. We can begin to answer questions like: Are users in country X not as viral as users elsewhere because of cultural reasons, features on the site, com-petition, or site speed?
Given the structured profile data we have for each member, we are using analytics to deliver relevant and targeted content to each user. In some cases, this means building upon existing features and finding relevant content to
inspired by analysis. In that way, analytics plays an R&D role within the product group. Our goal is to build new, highly targeted, and data-rich projects quickly. Those that don’t move the needle, we table. Those we see as successful we then graduate to be more heavily invested in by the rest of the organization. We have recently established and built up an engineering team that is focused on delivering data-driven products.
R2: What user-facing features has the analytics team built?
LinkedIn: Our team conceived and built the “people you may know” feature on LinkedIn (and later copied by other social networks like Facebook), which has reconnected millions of members and very often led to better engagement. We’ve also been working on a talent match tool for automatically showing job posters the individuals who best match the openings they’ve posted. We are trying to reverse the results as well, and show the most relevant jobs to our users when they login. We have promoted groups to users based on content in their profile and have built “browse maps,” where users can find similar people or jobs based on the aggregate behavior of millions of users. All of these prod-ucts have demonstrated high engagement rates and have led to longer ses-sions. In addition, there is a viral component in many of these products. For example, if a user joins a group, his or her connections can see that action and take action themselves. So in total, the right targeted content to one individual gets a “multiplier” effect due to the viral nature of our professional network.
R2: How do you see analytics changing Internet businesses and your business?
LinkedIn: Companies like Google, Amazon and Netflix have significant frac-tions of their revenue directly attributed to products that have evolved out of analytics. Identifying what is relevant to a user at the right time is critical to ensuring a click, a purchase, and a good user experience. When LinkedIn launched, it was primarily a product-focused organization with a clear vision of how individuals would manage their career and personal brand. It quickly amassed interesting data on 40 million professionals worldwide, and with that, we see data as a central asset on which we can build relevant content for our users.
Analytics is not simply reporting on the past, but using the insights one finds to improve existing products. Successful Internet companies have recog-nized the power of data, and have invested in building the right infrastructure and recruiting talented individuals to drive the business forward.
R2: Where do you see analytics going?
One of themes on the O’Reilly Radar is the “instrumentation of the physical world.” We’re seeing interesting new technologies that are making it possible for consumers to monitor and manage their own energy consumption and energy suppliers to manage energy supply and demand.
The term “Smart Grid” covers a plethora of these new initiatives, ranging from connecting power lines to new sources of renewable energy to increased grid redundancy to low-loss, super conducting transmission infrastructure to smart meters. Here, we focus on the Advanced Meter Infrastructure (AMI), a system that uses smart meters equipped with two-way communications to help manage the relationship between the electricity grid and electricity con-sumers—providing the opportunity to change how buyers and sellers can work together to address the issues swirling around energy usage.
First, some background. There is clearly change afoot regarding energy policy in the U.S., driven by the threat of climate change and the geo-political ramifications of increased energy independence. We expect increased focus in these areas:
n Energy conservation, including system-wide efforts to reduce energy needs
while maintaining lifestyle requirements
n Developing renewable energy storage, because energy can’t be generated
on demand for many renewable energy sources (i.e., the sun can’t be forced to shine and the wind can’t be forced to blow when energy is needed)
n Building the capacity to handle peak electricity demand. [or Utilities need
to build enough generating capacity to handle peak demand, (most typi-cally hot afternoons when air conditioning usage rises)—capacity that is infrequently used and can be expensive to spin up and shut down.
n Matching consumption and generation. Strategies include:
n Increasing the ability to predict demand, thus improving planning and
energy-generating efficiency
n Implementing time-of-day pricing to smooth and manage demand and
improve energy efficiency
n Using Demand/Response systems that allow utilities to reduce or turn
off power at consumer locations during times of peak demand
AMI and smart meters serve as an important piece of the Smart Grid by helping integrate managed demand and renewables into the energy delivery system. Here’s what they do:
Advanced Meter
Infrastructure (AMI)—
Smart Grid and Smart Meters
by Roger Magoulas
Roger Magoulas is the Director of Research at O’Reilly.
n Provide interval or time-of-use metering via near real-time sensors
n Two-way communications via a Zigbee network
n Smart meters can communicate with the utility for meter reading and
alerts, and can communicate with devices at the consumer side (e.g., air conditioners and refrigerators), to manage efficiency
n Can integrate with home networks for:
n Monitoring usage (smart meters can monitor electricity, gas, and
water usage)
n Integrating with smart appliances
n Demand/Response integration to take advantage of time-of-day tariffs
and peak demand management
n Monitor power quality and detect power outages
n Can mediate between the grid and distributed-energy generation options
like residential solar panels and windmills or electric cars
n Smart meters can help manage delivering electricity back to the grid for
others to use, including managing the tariffs that determine how those generating electricity are compensated*
Increased demand has dropped the price of smart meters from many 100s of dollars to about $100/unit, making smart meters and AMI an easier proposi-tion for utilities.
California Leads the Way
California has been a leader in setting energy policy for decades (well, I guess that whole deregulation debacle doesn’t look so leading in retrospect). Starting in the 70s, the California Energy Commission began to help coordinate energy policy—including decoupling utility profits from energy produced and promot-ing energy efficiency initiatives. These efforts contribute to California havpromot-ing nearly flat per capita energy consumption since the mid 70s—the average Californian consumes 40% less electricity than the national average (see chart). <note: factors other than energy policy help keep per capita electricity low in California, including a mild climate, higher household density, less heavy indus-try and higher energy prices. See “Deciphering California’s Efficiency Successes” by Kate Galbrath, New York Times, April 14, 2009, http://bit.ly/yvUv4>
* FOOTNOTE:
Zigbee alliance promotes a shared RF (radio frequency) protocol for device interoperability Starting in 2000, the dominant Italian
California’s success in managing energy efficiency is likely to influence fed-eral energy policy under the Obama administration. Dr. Stephen Chu left the Lawrence Berkeley National Labs (LBNL) in northern California to run the Federal Department of Energy. Dr. Chu has likely been influenced by the California model—he has specifically referenced California policy in interviews (“Newsmaker: Superweeds, air caves and the future of energy” by Michael Kanellos, CNET News, May 23, 2006, http://bit.ly/1rS7a).
Recently, Southern California Edison and PG&E, the two largest utilities in California, launched significant residential AMI initiatives. The projects are cost-justified by a combination of meter reading savings and rate increases. The meters they’re using allow both time-of-day pricing (i.e., customers pay more during peak load hours) and demand/response systems (customers’ appliances are turned off by the utility during peak load). Southern California Edison already offers demand/response discounts to residential customers willing to have their air conditioners turned off during peak demand hours. Smaller AMI initiatives are underway or planned in other parts of the country.
Implications
Investment in Smart Grid initiatives, including AMI and efforts to reduce peak demand, is included in the recently passed stimulus package and in pro-posed energy legislation. While getting changes to each state’s regulated energy bureaucracy may take time, we expect the use of smart meters to increase significantly in the coming years with the following implications:
California and US per capita electricity consumption (Rosenfeld curve); source: Energy Information Administration (2006)
Energy Usage Monitoring
Near real-time monitoring of household energy usage creates the opportunity for self-competition, goal seeking and other social rewards that help reduce energy usage. Studies point to a 5-15% reduction in energy usage when consumers can track their energy use.
Smart Appliances
While appliance manufacturers have shown whimsical versions of smart appli-ances in the past, there is now a financial incentive for consumers to want smart appliances: coordinating with time-of-day pricing. For example, refrigerators can adjust their compressor cycling to avoid running during peak hours when rates are higher (based on input from the smart meter and predictive algo-rithms); dishwashers and washing machines add delay options so they can be set to run during off-peak hours.
Demand/Response
Commercial electricity users in some states already use Demand/Response to lower rates by automatically having their air conditioning turned off during times of peak demand. Southern California Edison offers discount tariffs to residential customers who have smart meters and are willing to have their A/C turned off by the meters during hot summer afternoons when demand peaks. By helping manage peak demand and encouraging conservation, the AMI helps reduce energy needs and makes integrating renewable energy sources less complex. With explicit government support and consumer interest in energy and climate issues, we see the widespread adoption of smart meters and AMI initiatives helping change how electricity is consumed in the US. nn
References:
Google to Let Users Monitor Their PowerMeters (http://bit.ly/15TnPh) Smart Meter Installations Grow Nearly Fivefold (http://bit.ly/cYmUR) Silver Spring Heads Down Under (http://bit.ly/196bmx)
Can the DOE stimulate the smart grid? (http://bit.ly/rXMmE)
Cars that can be charged from an electric plug, either pure electric cars or plug-in hybrids,* can become an integral part of the smart grid by serving as a distributed battery for the entire electricity network. Electric cars generally are charged at night, using power when energy demand is low. But they can also sup-ply power when it’s most needed—plug them into the grid, and their batteries can provide energy during times of peak demand. In essence, electric cars can become a distributed battery for the entire grid, available to meet spikes in demand.
Electric cars have a number of issues that can limit their adoption, including battery costs, battery disposal, and the lack of a charging infrastructure for the urban settings where electric cars make the most sense. But, in addition to their potential contributions to balancing power supply and demand, electric cars offer the option of clean, no-pollution transportation when the electricity can be gen-erated using clean technologies.
* Currently, only the Tesla and Mini offer small runs of pure electric cars, and there are no plug-in hybrids available from mass-market car makers, although several are planned, including a rumored plug-in Prius and the Chevrolet Volt. The Obama administration has specifically called for incentives for buying plug-in hybrids as part of its proposed energy initiatives.
The Aptera, a three-wheel vehicle expected to be available in the fall in pure electric and plug-in hybrid versions.
Acknowledgements
May 19–21
Where 2.0 (San Jose, CA) http://en.oreilly.com/where2009
Knowing “where” opens up a world of possibilities, from the mundane (finding pizza) to the world-changing (cultural preservation via indigenous maps).
May 26–28
The D Conference (Carlsbad, CA)http://allthingsd.com/d
Walt Mossberg and Kara Swisher grill big-name CEOs at this swanky, sold-out conference.
June 22–24
Velocity: Web Operations and Performance (San Jose, CA)
http://en.oreilly.com/velocity2009
Building successful web sites that are fast, scalable, resilient, and highly available—sounds like a smart thing to do in a downturn. A hit in ‘08, this O’Reilly conference returns for a second year of practical magic.
June 22–25
Enterprise 2.0 (Boston, MA)
http://www.e2conf.com
Web 2.0 meets big business—command-and-control will never be the same.
June 28–July 5
Aspen Ideas Festival (Aspen, CO))
http://www.aifestival.org
Spend your summer vacation in the Rockies, with really smart people (including Tim O’Reilly) at the Aspen Institute’s annual thinkfest.
July 20–24
OSCON (San Jose, CA)
http://en.oreilly.com/oscon2009
July 21–22
Open Government & Innovations Conference (Washington, DC) http://www.opengovinnovations.com
It’s a new day in U.S. government. Transparency, participation, collaboration, and innovation are the goals—this conference focuses on social media as a way to reach them.
August 10–14
Search Engine Strategies (San Jose, CA)
http://www.searchenginestrategies.com/sanjose
Both inspiration and deep nuts-and-bolts information for any business trying to optimize their findability (and isn’t that every business?).
August 20–22
Gnomedex 9.0 (Seattle, WA)
http://www.gnomedex.com
Great sessions and even better networking. The cool kids will all be there— bloggers, social media mavens, and entrepreneurs—figuring out “The Science and Mathematics of our Real Lives.”
September 9–10
Gov 2.0 Summit (Washington, DC)
http://www.gov2summit.com
our website, http://r2.oreilly.com.
name
title company address
city state zip country telephone fax
email (personal email required for electronic access) url
n My colleagues should read Release 2.0, too!
Send me information about multiple copy subscriptions and electronic site licenses.
n check enclosed n charge my (check one) n american express n mastercard n visa card number expiration date cvs code
name and billing address n same as above n see below name
address
city state zip country signature
Please fax this form to 1.818.487.4501 or mail it to Release 2.0, P.O. Box 17046, North Hollywood CA 91615-9588 Payment must be included with this form. Your satisfaction is guaranteed or your money back.
