DMOZ.org was a website that provided a human-curated directory of many of the most credible websites on the internet. This multilingual resource would find websites in the same niche and group them into categories. A group of volunteers who formed a community that later came to be called the Open Directory Project (ODP) worked on the site. The project closed in March 2017 when its biggest funder, America Online (AOL), indicated that it no longer wished to support it. We traced the history of the project to determine what happened to it.
Why DMOZ was Important to SEOs
When Google launched in 1998 every search engine faced a difficult problem: how can you tell high-quality content from low-quality content?
Some solutions scaled (meaning that the incremental cost of applying that solution to millions of pages was near zero) but were easy to game. One example was to calculate the keyword density of a particular keyword on a page, ranking articles with “natural” keyword densities higher than pages with “unnatural” keyword densities.
Other solutions provided excellent results and were difficult to hack, but didn’t scale at all. One example was to pay humans to read all of the pages that targeted a particular keyword and rank order those results.
Computers are excellent at calculating keyword densities, but keyword density is poorly correlated with content quality. Humans are excellent at identifying quality content, but relative to the cost and computational power of a computer, ridiculously expensive. Google needed a hack that allowed them to harness the power of humans without having to pay them.
Google’s solution, of course, was elegant: when a webmaster chose to link to another resource on the web, why not treat that link as a vote for the quality of that page? The more votes that a particular website acquired, the higher the value of the votes that it cast in favor of other pages.
It’s clear that this one insight, implemented well, gave Google its lead in the search engine race.
But, there were many other problems that humans were better at solving than were the computers and algorithms available in 1998. For example:
- What subject category should a particular website be placed into?
- What was the best summary of a page? (The meta description, which every savvy webmaster tried to optimize?)
- And, most importantly, who could you trust to link to sites based on the merit of the sites, rather than on who was willing to pay the most?
DMOZ was important to Google because it allowed them to answer these questions without having to hire humans to answer them. DMOZ.org was important to SEOs because it was important to Google.
To fully understand DMOZ’s role, you need to understand more about the DMOZ itself.
The Early Days of DMOZ
According to Laisha.com, a website that describes itself as the “insider’s look into what the search engines like and do not like,” DMOZ.org was an attempt to solve the frustrations emanating from a similar directory of the web produced by Yahoo, which was known for having many broken links (Source). So, in June 1998 a new directory called GnuHoo went live. (GnuHoo was a hat tip to both the open-source movement and “Yahoo”).
Initially, the GnuHoo process was disorganized. The system lacked clearly defined policies and proper management. Laisha.com reports that “It was unclear just how sites were to be described or categorized, and general anarchy reigned” (Source). There was also no way of ensuring that unscrupulous editors did not join to promote their businesses.
GnuHoo also met some intellectual property challenges when the GNU project, an established organization which at that time was producing the free UNIXish operating system, protested the use of the name GnuHoo. This forced GnuHoo to rebrand as NewHoo (Source).
Laisha.com reports that within a year of its establishment, NewHoo had around 400 editors, 3,900 categories, and 31,000 websites. Notwithstanding the initial disorganization at GnuHoo, the directory was gaining popularity. The main reason behind its reputation is that it had substantially reduced the amount of time required to index sites. Within the same year, the number of editors would rise to around 8,000. The categories grew to 2,500, and approximately 430,000 sites were listed (Source).
The Middle Years
The moniker DMOZ was an abbreviation of the name: Directory Mozilla. Writing for the British publication, The Guardian, Glyn Moody called DMOZ, “a Wikipedia was written by experts.”
The experts that Moody refers to were volunteers. Their role was to select, evaluate, describe, and organize websites. Each entry went through a painstaking seven-stage process.
According to the site, there were no special requirements for one to become an editor. An archive of an early page from DMOZ.org indicates that all one needed was “an interest or passion and a computer.” However, editors on the site had to ensure that they did not have any conflict of interest (Source).
Because Google trusted DMOZ’s volunteer curation process the search engine would sometimes make use of the DMOZ descriptions as snippets on its results. This would happen in instances were Google felt that the DMOZ depiction did a better job of describing the content of an article than the article’s meta-description or its content (Source).
Acquisition by Netscape
As would be expected from a project that was scaling in the manner that DMOZ.org was doing, the site soon attracted the attention of industry players. In October 1998, Netscape paid $1 million for the directory, promising to maintain the website’s original character of being a non-commercial entity. Soon after the purchase, Netscape changed the name of the project to the Open Directory Project (ODP) (Source).
Soon after taking over the resources of DMOZ.org, Netscape was acquired by America Online (AOL). AOL agreed to respect the Open Directory License. Writing for TranslationDirectory.com, a website that calls itself the “portal for language professionals,” Jim Hedger says that the 1998 purchase could be seen as the “start of the ODP’s rise.” This resulted in it becoming the favored resource where major search engines were getting their data (Source). Andrew Goodman, who writes for the website Traffic.com, reports that the Netscape-owned directory would soon have almost 23,000 editors and 1, 5 million sites in over 230,000 categories (Source).
Allegations of inappropriate conduct
As the ODP grew in popularity, controversy also started to circle it. One of the accusations of improper behavior against the ODP was made in 2007 by the owner of the website Shoemoney.com: Jeremy Schoemaker. Schoemaker says that an individual claiming to be an editor at the OPD had attempted to extort $5,000 from him. In an article posted on his website, Schoemaker says he ignored the email but soon received another one telling him that his site was no longer listed. He claims that the same editor wrote to him again, advising him to rethink not paying him.
What Then Happened to DMOZ?
In early 2017 DMOZ posted the following note: “As of Mar 14, 2017, Dmoz.org will no longer be available.”
Beyond the obvious issue of cost, DMOZ had stopped being relevant. If, in 1998, DMOZ could far outperform computers in terms of categorizing and summarizing sites, 19 years later that was no longer true.
Zvelo.com, a website dealing with content categorization, notes that DMOZ suffered as a result of the arrival of new technology such as artificial intelligence and machine learning, which made data analysis much faster without human intervention. The human-edited directory was simply no longer scalable as a business model. (Source)
Read more about our top blogs here: