Following the historical backdrop of Search engine optimization is similar to attempting to follow the historical backdrop of the handshake. We as a whole realize it exists, and we know it's a significant piece of business. Yet, we don't invest a lot of energy contemplating its beginnings - - we're generally worried about how we use it everyday.
But unlike the handshake, SEO is fairly young, and changes frequently. Quite appropriately, it appears to be a millennial -- its birth is predicted to fall somewhere around 1991.And in its relatively short life, it’s matured and evolved rather quickly -- just look at how many changes Google’s algorithm alone has gone through.
The first idea for creating a common archive for all the world’s data came to fruition in 1945. That July, Dr. Vannevar Bush -- then director of the now-defunct Office of Scientific Research and Development -- published a piece in The Atlantic proposing a “collection of data and observations, the extraction of parallel material from the existing record, and the final insertion of new material into the general body of the common record.” In other words, we believe, today’s Google.
Several decades later, in 1990, McGill University student Alan Emtage created Archie, which some say was the very first search engine -- though that remains up for debate, according to research from Bill Slawski, president and founder of SEO by the Sea. However, Archie was what Slawski called the “best way to find information from other servers around the internet at the time,” and is actually still (very primitive) operation.
The next decade saw several pivotal developments, with the more commercial versions of search engines we might recognize today taking shape.
- February 1993: Six Stanford students create Architext, which would later become the search engine Excite. Some, like Search Engine Land (SEL), say that Excite “revolutionized how information was cataloged,” making it easier to find information “by sorting results based on keywords found within content and backend optimization.”
- June 1993: Matthew Gray debuts World Wide Web Wanderer, which later became known as Wandex.
- October 1993: Martijn Koster introduces ALIWEB, which allows site owners to submit their own pages (unbeknownst, sadly, to many site owners).
- December 1993: At least three “bot-fed” search engines exist -- JumpStation, RBSE spider and World Wide Web Worm -- which likely means they were powered by web robots to crawl both servers and site content to produce results.
- 1994: Alta Vista, Infoseek, Lycos, and Yahoo search engines all come to fruition.
- 1996: Google co-founders Larry Page and Sergey Brin begin building a search engine that they initially call BackRub.
- April 1997: AskJeeves is introduced, later becoming Ask.com.
- September 1997: Google.com is registered as a domain name.
It’s worth noting that nearly twelve years later, in June 2009, Microsoft released Bing -- its previous editions were also known as Live Search, Windows Live Search, and MSN Search.
But here’s where SEO itself comes in. As search engines became more mainstream and widely used, site owners started to get wise. As SEO community Moz puts it, “It was discovered that by taking some rather simple actions, search engine results could be manipulated and money could be made from the internet.”
Those results, though, weren’t exactly quality ones. And that, dear readers, is where the SEO story begins.
A Brief History of Search & SEO
With search engines becoming household names and more families becoming connected to the Internet, finding information came with greater ease. The problem, as noted above, was the quality of that information.
While search engine results matched words from user queries, it was usually limited to just that, as an overwhelming amount of site owners took to keyword stuffing -- repeating keywords over and over again in the text -- to improve rankings (for which there was no criteria), drive traffic to their pages and produce attractive numbers for potential advertisers.
There was also a bit of collusion going on. In addition to the keyword stuffing, people were using excessive and “spammy backlinks,” according to SEL, to improve their authorities. Not only were there no ranking criteria at the time -- but by the time search engines fixed algorithms accordingly, there were already new black hat SEO practices taking place that the fixes didn’t address.
But then, two kids at Stanford got an idea.
When Page and Brin set out to create Google, that was one of the problems they wanted to solve. In 1998, the pair published a paper at Stanford titled “The Anatomy of a Large-Scale Hypertextual Web Search Engine,” where they wrote:
It was in that same paper that Page and Brin first mentioned PageRank, the technology that Google uses to help rank search results based on quality, and not keywords alone. Some might say that thesis cleared the path for SEO as we know it today.
The Early 2000s
The early 2000s saw the beginning of the Google takeover. In the process of making search engine technology less advertising-centric, Google began to provide guidelines for white hat SEO -- the kind that the “good guys” stick to -- to help webmasters rank without any of the common fishy behavior from the 90s.
2000-2002
But according to Moz, the guidelines didn’t yet have an actual impact on ranking, so people didn’t bother following them. That’s partially because PageRank was based on the number of inbound links to a given page -- the more of those, the higher the ranking. But there wasn’t yet a way to measure the authenticity of those links -- for the early part of the 2000s, Marketing Technology Blog says it was still possible to use these backlinking techniques to rank pages that weren't even related to search criteria.
But in 2001, Brin and Page appeared on "Charlie Rose," when the host asked them, "Why does it work so well?" As part of his answer, Brin emphasized that -- at the time -- Google was a search engine and nothing else, and was looking at "the web as a whole, and not just which words occur on each page." It set the tone for some of the initial major algorithm updates that would begin to more closely examine those words. Have a look at the full interview:
2003-2004
This approach to the web being about more than just words really began taking shape in November 2003, with the “Florida” update to Google’s algorithm. Enough sites lost their ranking for Search Engine Watch to call the response to Florida a massive “outcry,” but careful to note that many sites benefitted from the change, too. It was the first major instance of sites receiving penalties for things like keyword stuffing, signaling Google’s emphasis on solving for the user first -- mainly with quality content.In 2004, one of the more primitive versions of Google's voice search existed, in what the New York Times called a half-finished experiment. And while the technology was somewhat infantile at the time -- just check out what the instructions looked like at first -- it was also a signal to the future importance of mobile in SEO. (Stay tuned -- more on that later.)
2005: A big year for SEO
One of the biggest years in the search engine world was 2005. That January, Google united with Yahoo and MSN for the Nofollow Attribute, which was created in part to decrease the amount of spammy links and comments on websites, especially blogs. Then, in June, Google debuted personalized search, which used someone’s search and browsing history to make results more relevant.That November, Google Analytics launched, which is still used today to measure traffic and campaign ROI. Check out its baby photo:
2009: SEO shakeups
In 2009, the search engine world saw a bit of a shakeup. Bing premiered that June, with Microsoft aggressively marketing it as the search engine that would produce noticeably better results than Google. But as SEL predicted, it was no “Google-killer,” nor did its advice for optimizing content significantly contrast Google’s. In fact, according to Search Engine Journal, the only noticeable difference was Bing’s tendency to give priority to keywords in URLs, as well as favoring capitalized words and “pages from large sites.”
That same year, in August, Google provided a preview of the Caffeine algorithm change, requesting the public’s help to test the “next-generation infrastructure” that Moz says was “designed to speed crawling, expand the index, and integrate indexation and ranking in nearly real-time.”
Caffeine wasn’t fully introduced until nearly a year later -- when it also improved the search engine’s speed -- but in December of 2009, a tangible real-time search was released, with Google search results including things like tweets and breaking news. It was a move that confirmed SEO wasn’t just for webmasters anymore -- from that moment forward, journalists, web copywriters and even social community managers would have to optimize content for search engines.
Here's Matt Cutts, Google's head of webspam, discussing Caffeine in August 2009:
2010-Present
When you’re typing in a search query into Google, it’s kind of fun to see what its suggestions are. That’s thanks to the Google Instant technology, which rolled out in September 2010. At first, Moz says, it made SEOs “combust,” until they realized that it didn’t really have any result on ranking.
But Google Instant, along with the evolution of SEO from 2010 on, was just another phase of the search engine’s mission to solve for the user -- despite some controversy along the way around pages whose rankings were actually improved by negative online reviews. The algorithm, Google said, was eventually adjusted to penalize sites using such tactics.
More on Google Instant, circa 2010:
That year also saw a growing importance of social media content in SEO. In December 2010, both Google and Bing added "social signals," which first displayed any written Facebook posts, for example, from your own network that matched your query. But it also began to give PageRank to Twitter profiles that were linked to with some frequency. The importance of Twitter in SEO didn't end there -- stay tuned.
2011: The year of the panda
The trend of punishing sites for unfairly gaming Google’s algorithm would continue. Some of these incidents were more public than others, including one with Overstock.com in 2011. At the time, according to Wall Street Journal, domains ending with .edu generally had a higher authority in Google’s eyes. Overstock used that to its advantage by asking educational institutions to link to its site -- and use keywords like “vacuum cleaners” and “bunk beds” -- offering discounts for students and faculty in return. Those inbound links would improve Overstock’s rankings for queries with such keywords, until Overstock discontinued the practice in 2011 and Google penalizing them soon after.
It was also the year of Panda, which first rolled out that February -- the algorithm update that cracked down on content farms. Those were sites with huge quantities of frequently updated, low-quality content that was written with the sole purpose of driving search engine results. They also tend to have a high ad-to-content ratios, which Panda was trained to sniff out.
Panda itself has undergone several updates -- so many that in its timeline of changes to Google’s algorithm, Moz declined to list any that weren’t major after 2011. Even with that exclusion, the timeline still lists twenty-eight panda updates -- for most of which the impact was difficult to measure -- through July of 2015.
2012: Along came a penguin
In April 2012, Google took what it called “another step to reward high-quality sites” with the first of many Penguin updates -- and, in the process of announcing it, acknowledged Bing’s month-earlier blog post on the changing face of SEO. Penguin targeted sites that more subtly used non-white hat SEO tactics; for example, those with content that might be mostly informative, but was also sprinkled with spammy hyperlinks that had nothing to do with the page’s H1, like in this example:
