When computer science professor Avideh Zakhor and graduate student Sen-ching Chung set out to create a video search engine at the University of California, Berkeley, in 1999, they dispatched web crawlers in the wee hours from one of the university’s computer labs to trawl the Internet for video samples. Students arriving the next morning found their desktops teeming with porn. To continue with her work, Ms. Zakhor was obliged to drape the computer screens every night. But the delicate sensibilities of college students didn’t prevent her from ultimately downloading some 45,000 video clips of varying content and length. Not all of it was porn. The randomly assembled collection became the raw material for what, some six years later, remains one of the more innovative experiments in video search.

Adapting techniques borrowed from the far more evolved field of text search, Ms. Zakhor and her grad student invented a method of stamping videos with a unique “fingerprint” that could be subsequently searched and matched against other video samples. The system was designed to search for unauthorized reuse of video content on the Internet without resorting to the text-based search most video search relies on. To be sure, Video Signature, as the pair called their invention, had drawbacks. For one, it was predicated on the idea that all uploaders of video content would use Video Signature, an unlikely proposition in the fractious realm of the web. Still, it demonstrated the feasibility of a certain type of image processing put in the service of video search. Five years after Ms. Zakhor and Mr. Chung published a paper on Video Signature in IEEE Spectrum, Ms. Zakhor says one question lingers: why is video search still so primitive?

IEEE Spectrum

It is a billion-dollar question that also vexes and intrigues search companies, startups, venture capitalists, and consumers. As the computer becomes the television and the television becomes the computer, video will dominate the Internet. The companies that master video search stand to unlock a goldmine. Video search is seen as the virgin frontier awaiting colonization by the likes of Google, Yahoo, Ask Jeeves, MSN, and a host of aspiring search titans.

GoogleAsk Jeeves

Tight-Lipped Interest

In November 2005, 12 million unique visitors from the United States logged into MSN Video, according to comScore Media Matrix. That’s a 20 percent rise from November 2004. “The space is really heating up,” says Kevin Conroy, the COO and executive vice president at AOL for Broadband. Tiny startups like Truveo and Blinkx challenge—and often outpace—large companies. There are no clear numbers for the money pouring into video search. The startups prefer not to talk about how much money they have, and in some cases, like Blinkx, about where it’s coming from. But every major search company is working on video search, and rumors abound about their plans to snap up their fledgling—and often superior—competitors.

Of the video search companies touted as an acquisition target by the large search engines, Truveo stands out. Founded in 2004 in Burlingame, California, Truveo aims to “be the Google of video search,” according to Jeff Clavier, a venture capital consultant who was an angel investor in Truveo. If Mr. Clavier’s boast is right, and Truveo’s algorithms deliver better video search results than competitors’, the company could be worth a lot of money to a Google, a Yahoo, or an IAC. He was right. On Tuesday, January 10, AOL said it bought the company (see AOL Buys Video Search Startup).

Formidable in text search, the big search engines are less so when it comes to video. “Google and Yahoo are very primitive,” says Ramesh Jain, a professor of information and computer science at the University of California, Irvine. “They treat image and video as second-class media.”

Search, as defined by Google and Yahoo, is mostly an ad-driven business. Google brought in revenue in the neighborhood of $5.25 billion in 2005, compared to $3.2 billion the year before; Yahoo generated around $4.83 in 2005. Analysts predict the growth will continue. Search advertising will hit $7.5 billion by 2010, as online ads generate $19 billion, according to forecasts from JupiterResearch. That’s 79 percent more than the $4.2 billion advertisers spent on search ads in 2005. And in the next two years, a significant chunk of ad dollars will move into video search, predicts Gartner analyst Allen Weiner.

A Pillar of the Web

Advertisers spent $225 million in 2005 on online video ads, according to estimates from research firm eMarketer. That figure will grow to $640 million by 2007, and will hit the $1 billion mark in 2008. “Video advertising will be the single most important development over the next 12 to 18 months,” says Jeff Lanctot, vice president and general manager of Avenue A | Razorfish, an interactive advertising agency. “This is where the leaders will stake their ground.” Right now, video ads make up less than 1 percent of the ads Mr. Lanctot’s agency produces. But he expects the market to grow swiftly. Just as text searches bring up text ads, experts expect that video ads can be linked to video searches—if the technology evolves quickly enough.

The migration may be fast, but there is still time for Yahoo and Google to change Mr. Jain’s dim opinion of their video search products. Video search is still in its infancy. In November, 125 million Internet users visited Yahoo. Visitors to Yahoo Video Search accounted for less than 2 percent of those page views. That number is not likely to stay so small. Jeff Karnes, director of media search at Yahoo, says that his company is watching two trends that will make video search a pillar of the web and a big business. First, broadband Internet connections are growing fast in the United States. In 2005, 32 million U.S. households had fast Internet connections compared to 26.7 million in 2004. Second, large media companies are “breaking down barriers” and putting programs on the Internet that once were only on broadcast television and cable. For example, in late December, Yahoo offered two sitcoms from CBS on its web site as part of an ongoing experiment in gauging audience receptivity to watching television online. Both CNN and ABC have also aggressively put programming on the Internet. So-called Internet Protocol TV sites, like Akimbo in San Mateo, California, which provide programming exclusively over the web, received $235 million in venture funding between 2002 and 2005, according to Venture One.

Yet for all the excitement and investment generated by video on the Internet, the search tools consumers have at their disposal to sift through the ever-growing mountain of video need a lot of improvement. Yahoo, like nearly every other search site or video site, largely uses text to direct consumers to content. Mr. Karnes says that Yahoo Video Search receives video content through four avenues: web crawlers; a content acquisition program where companies such as the Discovery Channel, CNN, and other networks provide programming directly to Yahoo; submissions to Yahoo Networks; and the uploading of user-generated content. Mr. Karnes says that Yahoo’s repository of video counts some 15 million clips. The collection is indexed primarily through so-called metadata, the text attached to a video that gives the search engine some inkling of what it is about.

Some of these keywords are affixed by the producers of the videos. For example, the metadata [tag] attached to a sitcom would include the cast, the title, and perhaps a thumbnail summary of the plot. Metadata works just fine if someone is looking for a classic episode of I Love Lucy, and keys in “Lucy, pies.” But relying on metadata to return every video on the Internet with a pie somewhere in it is not possible. Enter tagging.

The Secret is Stickiness

Known to some as “folksonomy” and to others as “social search,” tagging is the process of letting users add searchable reference points. Tagging allows users to provide a much deeper and richer portrait of what is inside a video collection like Yahoo’s. Tagging also permits a web of highly specialized associations, interests, and references to develop around a piece of content. It’s a boon for a company like Yahoo. Not only does tagging bring the company a free source of new, richer metadata, but it promotes the kind of participation that translates into stickiness, keeping users logged into a site. Users become involved and connected to a web site that they help edit. “We are very interested in user-generated content and the power of community,” says Mr. Karnes. That interest was underscored in December, when Yahoo bought del.icio.us, a New York City-based company that facilitates the group tagging of web sites, for an undisclosed sum.

While Google evokes envy, fear, and admiration in companies from Microsoft to Simon and Schuster, its video search product doesn’t garner much attention. In January 2005, Google introduced a video search product that allowed users to upload content to the company’s site, but offered nothing in the way of premium content like television shows or news snippets. “We agree that we have work to be done on the product,” says Peter Chane, the senior business product manager for Google Video. Mr. Chane is tightlipped about what direction that work will take, but says that Google is looking at allowing users to annotate video content. Google will not say how many videos it has in its library, or how many of the searches conducted on its site are for video.

Microsoft

Tagging and better metadata is interesting, but video search will have to go a step further than text-based search if it is to keep up with the tidal wave of video content destined for the Internet, says Mr. Jain. “Somebody has to think about video search as video search, rather than text search applied to video. Most of the products you see are text search applied to video,” says Mr. Jain.

Mr. Jain was chairman of Virage, a company founded in 1994 that is widely regarded as the grandfather of video search. Now owned by Autonomy in the United Kingdom, for a while Virage boasted one of the most advanced video search products on the market. Its list of former employees is a who’s who of video search today.

One company that took a close look at Virage before launching its own video search product is San Francisco-based Blinkx. Suranga Chandratillake, who co-founded Blinkx in 2004,says Virage had first-rate technology but bad luck and a bad business plan. The company made the mistake of pitching its video search product to corporations instead of creating a consumer market, he says. It also arrived on the scene too early, and then had the misfortune of hitting the dot-com bust in 2000.

Blinkx has licensed Virage’s speech recognition software in its bid to become synonymous in consumers’ minds with video search.Mr. Chandratillake says that in addition to speech recognition, which can tag video based on a film’s dialogue, Blinkx uses technology that can identify scene breaks and scene changes, as well as onscreen word recognition programs that can “read” text or closed-caption content that appears in video footage. But what Mr. Chandratillake calls the Holy Grail of video search is still five years away, he says, the sort of onscreen face and object recognition that would allow a searcher to retrieve results with George W. Bush’s face even if the video was not tagged accordingly.

Dinner Dealings

Pursuing the Holy Grail may not be worth the trouble, says Rick Summer, an equity analyst at Morningstar. From telecom to online shopping, the big search companies like Google and Yahoo have been able to muscle their way into many different spheres. Hollywood is not going to be so easy. “It is going to be a challenge for pure search providers to integrate into the already established world [of TV and movies],” says Mr. Summer. Before a Yahoo or a Google can make a mint from video search, the companies have to have content people actually want. Sewing up the primetime, blockbuster, big-name entertainment consumers crave will be about forging relationships, he says. Algorithms won’t get you a table at Spago’s. In the world of entertainment relationships, cable companies such as Comcast have an edge over search companies.

Michael Downing, CEO of GoFish, a San Francisco-based multimedia search startup, agrees. Companies that are protective of their video can just block crawlers trying to get past their walls. But the guarded content is guarded for a reason: it’s worth something. Blinkx, Yahoo, and GoFish are some of the companies that have wooed Hollywood and other content providers. Google has gone a different route. It streams TV shows every now and then, and wants access to the same repositories as everyone else in the business, but for now, it’s settling for the videos its users upload to its site. AOL, which bought Singingfish’s multimedia search engine in 2003, is perhaps the most blessed when it comes down to content. It is part of the Time Warner family, which owns HBO, Warner Brothers, New Line Cinema, and Turner Broadcasting System.

Execs at these companies haven’t realized how big video search can be. Many analysts still haven’t bought into the concept. Morningstar’s Mr. Summer, for one, says companies seeking mega profits might be disappointed. “TV Guide is not a billion-dollar business.” That may explain in part why Ms. Zakhor’s experiment with image processing never left the lab.