RentClicks was so close to complete death last night. As you can see below, RentClicks only has 9 pages listed with Google as of today. Try it here.
What happened was on 6/15 RentClicks, for some odd reason, placed a disallow statement in their robots.txt file. What that means is RentClicks was telling search engines like Google to go away and not take their content. Slowly over the last week the page count on Google for RentClicks dropped from about 30K to almost zero (pretty fun for a competitor like myself to watch!).
Then last night around 10pm my time, I could see that the RentClicks techs were doing some major work to the Rental.com and RentClicks.com sites and corrected the robots file. What they ended up doing was disallowing robots for the rental.com beta site and allowing the spiders to cache RentClicks again so Google does not see a duplication issue–which was smart of them.
So, what does this all mean? I am not completely sure what they are doing over there in “RentClick’s Land”, but my guess is they discovered that they made a huge mistake on the robots and wanted the RentClicks traffic back, which has taken a nose dive the past week.
But does it matter? They are only days away from redirecting RentClicks to Rentals.com.
For a search engine guy like myself, it has been really fun to watch. It was also a good lesson on how Google ranks web sites.
Related Articles
7 users responded in this post
This should be an early warning sign for what a mess things are about to become. It is pretty obvious that there is no business plan and managing all these sites is almost impossible. How you let your search ranking drop on google the main search engine is mind blowing especially when you have already been “black listed” off yahoo.
I’m not sure I agree that having multiple sites is necessarily impossible. In the vacation rental space, HomeAway.com has been doing almost the exact same thing for the past 3 years.
If you pay for sites based on their cash flow driven by organic traffic, almost by necessity you have to keep them running.
What RentClicks is doing makes no sense to me unless they are going to put in place a bunch of 301 Redirects from Rentclicks.com to the counterpart pages Rentals.com. But you don’t block Google and hope they find your content on your new site… you have a very controlled transition using well-timed changes to sitemaps and 301’s as described by the pros at SMX Advanced. If there wasn’t an embargo on discussing specifics of that session, I’d say more.
Jeb,
I was there at that session as well at SMX, I gotta think that no one from CSI was there or they wouldn’t have blocked the spiders from RentClicks.
I will have to disagree with you on HomeAway.com.
Here is a test, type in “Vacation rentals” into Google - notice that only two of HomeAway’s sites appear in the top 10 - before they purchased the top 10 vacation rental sites, all 10 used to be in the top 10. Google seems to only allow 2 sites from the same company in the top 10.
RentalHouses.com has not changed their WhoIs information which I feel is cheating the system. I am guessing that Robert Massey sold them his site but leases the domain name back to them for 1 year until he is cut loose like the RentClicks owners were.
[...] CSI blocked the search engines spiders from retrieving their content. Update: Read more about the Google spider issue here. CSI also eliminated the “Family of Sites” package and re-branded it the [...]
That’s a great analysis of robots.txt and how powerful it can be. So I took a look at how a pro like you puts it out.
You have this line in your robots.txt, Dave.
User-agent: ia_archiver
Disallow: /
There is a rumor that your first site was a scrape of Rentclicks. I didn’t believe it so I went to Archive.org to prove what a clean, honest hard working man in the basement you are. Now there is no proof. So, I have to ask, why would you block the Internet Archive from your site?
The reason I block the Internet Archive is for competitive reasons. With that type of history info you can study how a site grows. For instance with RentClicks you can see which markets they grew from and at what rate - to me that is a gold mine of info.
As for scraping the RentClicks site - sure I studied it along with the other top sites when I first started - I would have been blind not to.
But I did not copy anything from their site including images, text, or code if that is what you call scraping.
Does my site look similar to RentClicks with a map in the middle of the home page sure it does, but so does many rental and real estate sites.
Mr. Curious did you know that RentClicks bought their site from ScriptMate for 149 bucks and then sent their site on a CD ROM to DC from copyright protection?
[...] this month traffic stats, we see that RentClicks.com took a big hit in traffic (due to a robots.txt file blunder) and RentalHouses.com took the #1 [...]
Leave A Reply