Another way to implement a site search facility

Providing kick ass navigation and product search is the key to success for e-commerce sites. Conversion rates highly depend on user friendly UIs which enable the shopper to find the desired product with a context sensitive search in combination with few drill-down clicks on navigational links. Unfortunately, the build-in search as well as navigation and site structure of most shopping carts simply sucks. Every online store is different, hence findability must be customizable and very flexible.

I've seen online shops crawling their product pages with a 3rd party search engine script because the shopping cart's search functionality was totally and utterly useless. Others put fantastic efforts in self made search facilities which perfectly implement real life relations beyond the limitations of the e-commerce software's data model, but need code tweaks for each and every featured product, specials, virtual shops assembling a particular niche from several product lines or whatever. Bugger.

Today I stumbled upon a very interesting approach which could become the holy grail for store owners suffering from crappy software. Progress invited me to discuss a product they've bought recently --EasyAsk-- from a search geek's perspective. Long story short, I was impressed. Without digging deep into the technology or reviewing implementations for weaknesses I think the idea behind that tool is promising.

Unfortunately, the EasyAsk Web site doesn't provide solid technical and architectural information (I admit that I may have missed the tidbits within the promotional chatter), hence I try to explain it from what I've gathered today. Progress EasyAsk is a natural language interface connecting users to data sources. Users are shoppers, and staff. Data sources are (relational) databases, or data access layers (that is a logical tier providing a standardized interface to different data pools like all sorts of databases, (Web) services, an enterprise service bus, flat files, XML documents and whatever).

The shopper can submit natural language queries like "yellow XS tops under 30 bucks". The SRP is a page listing tops and similar garments under 30.00$, size XS, illustrated with thumbnails of pics of yellow tops and bustiers, linked to the product pages. If yellow tops in XS are sold out, EasyAsk recommends beige tops instead of delivering a sorry-page. Now when a search query is submitted from a page listing suits, a search for "black leather belts" lists black leather belts for men. If the result set is too large and exceeds the limitations of one page, EasyAsk delivers drill-down lists of tags, categories and synonyms until the result set is viewable on one page. The context (category/tag tree) changes with each click and can be visualized for example as bread crumb nav link.

Technically spoken, EasyAsk does not deal with the content presentation layer itself. It returns XML which can be used to create a completely new page with a POST/GET request, or it gets invoked as AJAX request whose response just alters DOM objects to visualize the search results (way faster but not exactly search engine friendly - that's not a big deal because SERPs shouldn't be crawlable at all). Performance is not an issue from what I've seen. EasyAsk caches everything so that the server doesn't need to bother the hard disk. All points of failure (WRT performance issues) belong to the implementation, thus developing a well thought out software architecture is a must-have.

Well, that's neat, but where's the USP? EasyAsk comes with a natural language (search) driven admin interface too. That means that product managers can define and retrieve everything (attributes, synonyms, relations, specials, price ranges, groupings ...) using natural language. "Gimme gross sales of leather belts for men II/2007 compared to 2006" delivers a statistic and "top is a synonym for bustier and the other way round" creates a relation. The admin interface runs in the Web browser, definitions can be submitted via forms and all admin functions come with previews. Really neat. That reduces the workload of the IT dept. WRT ad-hoc queries as well as for lots of structural change requests, and saves maintenance costs (Web design / Web development).

I've spotted a few weak points, though. For example in the current version the user has to type in SKUs because there's no selection box. Or meta data are stored in flat files, but that's going to change too. There's no real word stemming, EasyAsk handles singular/plural correctly and interprets "bigger" as "big" or "xx-large" politically correct as "plus", but typos must be collected from the "searches without results" report and defined as synonym. The visualization of concurrent or sequentially applied business rules is just rudimentary on preview pages in the admin interface, so currently it's hard to track down why particular products get downranked respectively highlighted when more than one rule applies. Progress told me that they'll make use of 3rd party tools as well as in house solutions to solve these issues in the near future - the integration of EasyAsk into the Progress landscape has just begun.

The definitions of business language / expected terms used by consumers as well as business rules are painless. EasyAsk has build-in mappings like color codes to common color names and vice versa, understands terms like "best selling" and "overstock", and these definitions are easy to extend to match actual data structures and niche specific everyday language.

Setting up the product needs consultancy (as a consultant I love that!). To get EasyAsk running it must understand the structure of the customer's data sources, respectively the methods provided to fetch data from various structured as well as unstructured sources. Once that's configured, EasyAsk pulls (database) updates on schedule (daily, hourly, minutely or whatever). It caches all information needed to fulfill search requests, but goes back to the data source to fetch real time data when the search query requires knowledge of not (yet) cached details. In the beginning such events must be dealt with, but after a (short) while EasyAsk should run smoothly without requiring much technical interventions (as a consultant I hate that, but the client's IT department will love it).

Full disclosure: Progress didn't pay me for that post. For attending the workshop I got two books ("Enterprise Service Bus" by David A. Chappel and "Getting Started with the SID" by John P. Reilly) and a free meal, travel expenses were not refunded. I did not test the software discussed myself (yet), so perhaps my statements (conclusions) are not accurate.

Stumble It!

Post it to del.icio.us

Follow-up: Erol's patch fixing Google troubles

Erol developers test their first Google-patch for sites hosted on UNIX boxes. You can preview it here: x55.html. When you request the page with a search engine crawler identifier as user-agent name, the JavaScript code redirecting to the frameset erol.html#55x0&& gets replaced with a HTML comment explaining why human visitors are treated different from search engine spiders. The anatomy of this patch is described here, your feedback is welcome.

Erol told me they will be running tests on this site over the coming weeks, as they always do before going live with an update. So stay tuned for the release. When things run smoothly on UNIX hosts, a patch for Windows environments shall follow. On IIS the implementation is a bit trickier, because it needs changes of the server configuration. I'll keep you updated.

Stumble It!

Post it to del.icio.us

Erol to ship a Patch Fixing Google Troubles

Background: read these four posts on Google penalizing respectively deindexing e-commerce sites. Long story short: Recently Google's enhanced algos began to deindex e-commerce sites powered by Erol's shopping cart software. The shopping cart maintains a static HTML file which redirects user agents executing JavaScript to another URL. This happens with each and every page, so it's quite understandable that Ms. Googlebot was not amused. I got involved as a few worried store owners asked for help in Google's Webmaster Forum. After lots of threads and posts on the subject Erol's managing director got in touch with me and we agreed to team up to find a solution to help the store owners suffering from a huge traffic loss. Here's my report of the first technical round.

Understanding how Erol 4.x (and all prior versions) works:

The software generates a HTML page offline, which functions as an XML-like content source (called "x-page", I use that term because all Erol customers are familar with it). The "x-page" gets uploaded to the server and is crawlable, but not really viewable. Requested by a robot it responds with 200-Ok. Requested by a human, it does a JavaScript redirect to a complex frameset, which loads the "x-page" and visualizes its contents. It responds to browsers if directly called, but returns a 404-NotFound error to robots. Example:

"x-page": x999.html
Frameset: erol.html#999x0&&

To view the source of the "x-page" disable JavaScript before you click the link.

Understanding how search engines handle Erol's pages:

There are two major weak points with regard to crawling and indexing. The crawlable page redirects, and the destination does not exist if requested by a crawler. This leads to these scenarios:

1. A search engine ignoring JavaScript on crawled pages fetches the "x-page" and indexes it. That's the default behavior of yesterdays crawlers, and still works this way at several search engines.


2. A search engine not executing JavaScript on crawled pages fetches the "x-page", analyzes the client sided script, and discovers the redirect (please note that a search engine crawler may change its behavior, so this can happen all of a sudden to properly indexed pages!). Possible consequences:
   
   
   • It tries to fetch the destination, gets the 404 response multiple times, and deindexes the "x-page" eventually. That would mean that depending on the crawling frequency and depth per domain the pages disappear quite fast or rather slow until the last page is phased out. Google would keep a copy in the supplemental index for a while, but this listing cannot return to the main index.
   
   
   • It's trained to consider the unconditional JavaScript redirect "sneaky" and flags the URL accordingly. This can result in temporarily and permanent deindexing as well.


3. A search engine executing JavaScript on crawled pages fetches the "x-page", performs the redirect (thus ignores the contents of the "x-page"), and renders the frameset for indexing. Chances are it gives up on the complexity of the nested frames, indexes the noframe-tag of the frameset and perhaps a few snippets from subframes, considers the whole conglomerate thin, hence assignes the lowest possible priority for the query engine and moves on.

Unfortunately the search engine delivering the most traffic began to improve its crawling and indexing, hence many sites formerly receiving a fair amount of Google traffic began to suffer from scenario 2 -- deindexing.

Outlining a possible work around to get the deleted pages back in the search index:

In six months or so Erol will ship version 5 of its shopping cart, and this software dumps frames, JavaScript redirects and ugly stuff like that in favor of clean XHTML and CSS. By the way, Erol has asked me for my input on their new version, so you can bet it will be search engine friendly. So what can we do in the meantime to help legions of store owners running version 4 and below?

We've got the static "x-page" which should not get indexed because it redirects, and which cannot be changed to serve the contents itself. The frameset cannot be indexed because it doesn't exist for robots, and even if a crawler could eat it, we don't consider it easy to digest spider fodder.

Let's look at Google's guidelines, which are the strictest around, thus applicable for other engines as well:

1. Don't [...] present different content to search engines than you display to users, which is commonly referred to as "cloaking."


2. Don't employ cloaking or sneaky redirects.

If we find a way to suppress the JavaScript code on the "x-page" when a crawler requests it, the now more sophisticated crawlers will handle the "x-page" like their predecessors, that is they would fetch the "x-pages" and hand them over to the indexer without vicious remarks. Serving identical content under different URLs to users and crawlers does not contradict the first prescript. And we'd comply to the second rule, because loading a frameset for human vistors but not for crawlers is definitely not sneaky.

Ok, now how to tell the static page that it has to behave dynamically, that is outputting different contents server sided depending on the user agent's name? Well, Erol's desktop software which generates the HTML can easily insert PHP tags too. The browser would not render those on a local machine, but who cares when it works after the upload on the server. Here's the procedure for Apache servers:

In the root's .htaccess file we enable PHP parsing of .html files:
AddType application/x-httpd-php .html

Next we create a PHP include file xinc.php which prevents crawlers from reading the offending JavaScript code:
<?php
$crawlerUAs = array("Googlebot", "Slurp", "MSNbot", "teoma", "Scooter", "Mercator", "FAST");
$isSpider = FALSE;
$userAgent = getenv("HTTP_USER_AGENT");
foreach ($crawlerUAs as $crawlerUA) {
if (stristr($userAgent, $crawlerUA)) $isSpider = TRUE;
}
if (!$isSpider) {
print "<script type=\"text/javascript\"> [a whole bunch of JS code] </script>\n";
}
if ($isSpider) {
print "<!-- Dear search engine staff: we've suppressed the JavaScript code redirecting browsers to "erol.html", that's a frameset serving this page's contents more pleasant for human eyes. -->\n";
}
?>

Erol's HTML generator now puts <?php @include("x.php"); ?> instead of a whole bunch of JavaScript code.

The implementation for other environments is quite similar. If PHP is not available we can do it with SSI and PERL. On Windows we can tell IIS to process all .html extensions as ASP (App Mappings) and use an ASP include. That would give three versions of that patch which should help 99% of all Erol customers until they can upgrade to version 5.

This solution comes with two disadvantages. First, the cached page copies, clickable from the SERPs and toolbars, would render pretty ugly because they lack the JavaScript code. Second, perhaps automated tools searching for deceitful cloaking might red-flag the URLs for a human review. Hopefully the search engine executioner reading the comment in the source code will be fine with it and give it a go. If not, there's still the reinclusion request. I think store owners can live with that when they get their Google traffic back.

Rolling out the patch:

Erol thinks the above said makes sense and there is a chance of implementing it soon. While the developers are at work, please provide feedback if you think we didn't interpret Google's Webmaster Guidelines strict enough. Keep in mind that this is an interim solution and that the new version will handle things more standardized. Thanks.




Paid-Links-Disclosure: I do this pro bono job for the sake of the suffering store owners. Hence the links pointing to Erol and Erol's customers are not nofollow'ed. Not that I'd nofollow them otherwise ;)

Stumble It!

Post it to del.icio.us

Follow-up on "Google penalizes Erol stores"

Background: these three posts on Google penalizing e-commerce sites.

Erol has contacted me and we will discuss the technical issues within the next days or maybe weeks or so. I understand this as a positive signal, especially because previously my impression was that Erol is not willing to listen constructive criticism, regardless Googles shot across the bow (more on that later). We agreed that before we come to the real (SEO) issues it's a good idea to render a few points made in my previous posts more precisely. In the following I quote parts of Erol's emails with permission:

Your blog has made for interesting reading but the first point I would like to raise with you is about the tone of your comments, not necessarily the comments themselves.

Question of personal style, point taken.

Your article entitled 'Why eCommerce Systems Suck', dated March 12th, includes specific reference to EROL and your opinion of its SEO capability. Under such a generic title for an article, readers should expect to read about other shopping cart systems and any opinion you may care to share about them. In particular, the points you raise about other elements of SEO in the same article, ('Google doesn't crawl search results', navigation being 'POST created results not crawlable') are cited as examples of ways other shopping carts work badly in reference to SEO - importantly, this is NOT the way EROL stores work. Yet, because you do not include any other cart references by name or exclude EROL from these specific points, the whole article reads as if it is entirely aimed at EROL software and none others.

Indeed, that's not fair. Navigation solely based on uncrawlable search results without crawler shortcuts or sheer POST results are definitely not issues I've stumbled upon while investigating penalized Erol driven online stores. Google's problem with Erol driven stores is client sided cloaking without malicious intent. I've updated the post to make that clear.

Your comment in another article, 'Beware of the narrow-minded coders' dated 26 March where you state: "I've used the case [EROL] as an example of a nice shopping cart coming with destructive SEO." So by this I understand that your opinion is EROL is actually 'a nice shopping cart' but it's SEO capabilities could be better. Yet your articles read through as EROL is generally bad all round. Your original article should surely be titled "Why eCommerce Systems Suck at SEO" and take a more rounded approach to shopping cart SEO capabilities, not merely "Why eCommerce Systems Suck"? This may seem a trivial point to you, but how it reflects overall on our product and clouds it's capability to perform its main function (provide an online ecommerce solution) is really what concerns me.

Indeed, I meant that Erol is a nice shopping cart lacking SEO capabilities as long as not the major SEO issues get addressed asap. And I mean in the current version, which clearly violates Google's quality guidelines. From what I've read in the meantime, the next version to be released in 6 months or so should eleminate the two major flaws with regard to search engine compatibility. I've changed the post's title, the suggestion makes sense for me too.

I do not enjoy the Google.co.uk traffic from search terms like "Erol sucks" or "Erol is crap" because that's simply not true. As I said before I think that Erol is a well rounded software nicely supporting the business processes its designed for, and the many store owners using Erol I've communicated with recently all tell me that too.

I noted with interest that your original article 'Why eCommerce Systems Suck' was dated 12th March. Coincidentally, this was the date Google began to re-index EROL stores following the Google update, so I presume that your article was originally written following the threads on the Google webmaster forums etc. prior to the 12th March where you had, no doubt, been answering questions for some of our customers about their de-listing during the update. You appear to add extra updates and information in your blogs but, disappointingly, you have not seen fit to include the fact that EROL stores are being re-listed in any update to your blog so, once again, the article reads as though all EROL stores have been de-listed completely, never to be seen again.

With all respect, nope. Google did not reindex Erol driven pages, Google had just lifted a "yellow card" penalty for a few sites. That is not a carte blanque but in the opposite Google's last warning before the site in question gets the "red card", that is a full ban lasting at least a couple of months or even longer. As said before it means absolutely nothing when Google crawls penalized sites or when a couple of pages reappear on the SERPs. Here is the official statement: "Google might also choose to give a site a 'yellow card' so that the site can not be found in the index for a short time. However, if a webmaster ignores this signal, then a 'red card' with a longer-lasting effect might follow."
(Yellow / red cards: soccer terminology, yellow is a warning and red the sending-off.)

I found your comments about our business preferring "a few fast bucks", suggesting we are driven by "greed" and calling our customers "victims" particularly distasteful. Especially the latter, because you infer that we have deliberately set out to create software that is not capable of performing its function and/or not capable of being listed in the search engines and that we have deliberately done this in pursuit of monetary gain at the expense of reputation and our customers. These remarks I really do find offensive and politely ask that they be removed or changed. In your article "Google deindexing Erol driven ecommerce sites" on March 23rd, you actually state that "the standard Erol content presentation is just amateurish, not caused by deceitful intent". So which is it to be - are we deceitful, greedy, victimising capitalists, or just amateurish and without deceitful intent? I support your rights to your opinions on the technical proficiency of our product for SEO, but I certainly do not support your rights to your opinions of our company and its ethics which border on slander and, at the very least, are completely unprofessional from someone who is positioning themselves as just that - an SEO professional.

To summarise, your points of view are not the problem, but the tone and language with which they are presented and I sincerely hope you will see fit to moderate these entries.

C'mon, now you're getting polemic;) In this post I've admitted to be polemic to bring my point home, and in the very first post on the topic I clearly stated that my intention was not slandering Erol. However, since you've agreed to an open discussion of the SEO flaws I think it's no longer suitable to call your customers victims, so I've changed that. Also in my previous post I'll insert a link near "greed" and "fast bucks" pointing to this paragraph to make it absolutely clear that I did not meant what you insinuate when I wrote:

Ignorance is no excuse [...] Well, it seems to me that Erol prefers a few fast bucks over satisfied customers, thus I fear they will not tell their cutomers the truth. Actually, they simply don't get it. However, I don't care whether their intention to prevaricate is greed or ignorance, I really don't know, but all the store operators suffering from Google's penalties deserve the information.

Actually, I still stand by my provoking comments because at this time they perfectly described the impression you've created with your actions respectively lack of fitly activities in the public.

1. Critical customers asking whether the loss of Google traffic might be caused by the way your software handles HTML outputs in your support forums were downtrodden and censored.
2. Your public answers to worried customers were plain wrong, SEO-wise. Instead of "we take your hints seriously and will examine whether JavaScript redirects may cause Google penalties or not" you said that search engines do index cloaking pages just fine, that Googlebot crawling penalized sites is a good sign, and all the mess is kinda Google hiccup. At this point the truth was out long enough, so your most probably unintended disinformation has worried a number of your customers, and gave folks like me the impression that you're not willing to undertake the necessary steps.
3. Offering SEO services yourself as well as forum talks praising Erol's SEO experts don't put you in a "we just make great shopping cart software and are not responsible for search engine weaknesses" position. Frankly that's not conceivable as responsible management of customer expectations. It's great that your next version will dump frames and JavaScript redirects, but that's a bit too late in the eyes of your customers, and way too late from a SEO perspective, because Google never permitted the use of JavaScript redirects and all the disadvantages of frames were public knowledge since the glory days of Altavista, Excite and Infoseek, long before Google overtook search.

To set the record straight: I don't think and never thought that you've greedily or deliberately put your customers at risk in pursuit of monetary gain. You've just ignored Google's guidelines and best practices of Web development too long, but --as the sub-title of my previous post hints-- ignorance is no excuse.

Now that we've handled the public relation stuff, I'll look into the remaining information Erol sent over hoping that I'll be able to provide some reasonable input in the best interest of Erol's customers.



Tags: Search Engine Optimization (SEO) Google Penalties DreamTeam Erol eCommerce Shopping Carts E-commerce Software Web Development

Stumble It!

Post it to del.icio.us

Google deindexing Erol driven ecommerce sites

Follow-up post - see why e-commerce software sucks.

Erol is a shopping cart software invented by DreamTeam, a UK based Web design firm. One of its core features is the on-the-fly conversion of crawlable HTML pages to fancy JS driven pages. Looks great in a JavaScript-enabled browser, and ugly w/o client sided formatting.

Erol, offering not that cheap SEO services itself, claims that it is perfectly OK to show Googlebot a content page without gimmicks, whilst human users get redirected to another URL.

Erol victims suffer from deindexing of all Erol-driven pages, Google just keeps pages in the index which do not contain Erol's JS code. Considering how many online shops make use of Erol software in the UK, this massive traffic drop may have a visible impact on the gross national product ;) ... Ok, sorry, kidding with so many businesses at risk does not amuse the Queen.

Dear "SEO experts" at Erol, could you please read Google's quality guidelines:

· Don't [...] present different content to search engines than you display to users, which is commonly referred to as "cloaking."
· Don't employ cloaking or sneaky redirects.
· If a site doesn't meet our quality guidelines, it may be blocked from the index.

Google did your customers a favour by not banning their whole sites, probably because the standard Erol content presentation technique is (SEO-wise) just amateurish, not caused by deceitful intent. So please stop whining

We are currently still investigating the recent changes Google have made which have caused some drop-off in results for some EROL stores. It is as a result of the changes by Google, rather than a change we have made in the EROL code that some sites have dropped. We are investigating all possible reasons for the changes affecting some EROL stores and we will, of course, feedback any definitive answers and solutions as soon as possible.

and listen to your customers stating

Hey Erol Support
Maybe you should investigate doorway pages with sneaky redirects? I've heard that they might cause "issues" such as full bans.

Tell your victims customers the truth, they deserve it.

Telling your customers that Googlebot crawling their redirecting pages will soon result in reindexing those is plain false by the way. Just because the crawler fetches a questionable page that doesn't mean that the indexing process reinstates its accessibility for the query engine. Googlebot is just checking whether the sneaky JavaScript code was removed or not.

Go back to the whiteboard. See a professional SEO. Apply common sense. Develop a clean user interface pleasing human users and search engine robots as well. Without frames, sneaky respectively superfluous JavaScript redirects, and amateurish BS like that. In the meantime provide help and work arounds (for example a tutorial like "How to build an Erol shopping site without page loading messages which will result in search engine penalties"), otherwise you don't need the revamp because your customer base will shrink to zilch.

Update: It seems that there's a patch available. In Erol's support forum member Craig Bradshaw posts "Erols new patch and instructions clearly tell customers not to use the page loading messages as these are no longer used by the software.".


Tags: Search Engine Optimization (SEO) Erol eCommerce Shopping Carts e-commerce software Web development

Related links:
Matt Cutts August 19, 2005: "If you make lots of pages, don’t put JavaScript redirects on all of them ... of course we’re working on better algorithmic solutions as well. In fact, I’ll issue a small weather report: I would not recommend using sneaky JavaScript redirects. Your domains might get rained on in the near future."
Matt Cutts December 11, 2005: "A sneaky redirect is typically used to show one page to a search engine, but as soon as a user lands on the page, they get a JavaScript or other technique which redirects them to a completely different page."
Matt Cutts September 18, 2005: "If [...] you employ [...] things outside Google’s guidelines, and your site has taken a precipitous drop recently, you may have a spam penalty. A reinclusion request asks Google to remove any potential spam penalty. ... Are there [...] pages that do a JavaScript or some other redirect to a different page? ... Whatever you find that you think may have been against Google’s guidelines, correct or remove those pages. ... I’d recommend giving a short explanation of what happened from your perspective: what actions may have led to any penalties and any corrective action that you’ve taken to prevent any spam in the future."
Matt Cutts July 31, 2006: "I’m talking about JavaScript redirects used in a way to show users and search engines different content. You could also cloak and then use (meta refresh, 301/302) to be sneaky."
Matt Cutts December 27, 2006 and December 28, 2006: "We have written about sneaky redirects in our webmaster guidelines for years. The specific part is 'Don't employ cloaking or sneaky redirects.' We make our webmaster guidelines available in over 10 different languages ... Ultimately, you are responsible for your own site. If a piece of shopping cart code put loads of white text on a white background, you are still responsible for your site. In fact, we've taken action on cases like that in the past. ... If for example I did a search [...] and saw a bunch of pages [...], and when I clicked on one, I immediately got whisked away to a completely different url, that would be setting off alarm bells ringing in my head. ... And personally, I’d be talking to the webshop that set that up (to see why on earth someone would put up pages like that) more than talking to the search engine."

Matt Cutts heads Google's Web spam team and has discussed these issues since the stone age at many places. Look at the dates above, penalties for cloaking / JS redirects are not a new thing. The answer to "It is as a result of the changes by Google, rather than a change we have made in the EROL code that some sites have dropped." (Erol statement) is: Just because you've got away so long that does not mean that JS redirects are fine with Google. The cause of the mess is not a recent change of code, it's the architecture by itself which is considered "cloaking / sneaky redirect" by Google. Google recently has improved its automated detection of client sided redirects, not its guidelines. Considering that both Erol created pages (the crawlable static page and the contents served by the URL invoked by the JS redirect) present similar contents, Google will have sympathy for all reinclusion requests, provided that the sites in question were made squeaky-clean before.

Stumble It!

Post it to del.icio.us

Why eCommerce Systems Suck at SEO

Listening to whiners and disappointed site owners across the boards I guess in a few weeks we'll discuss Google's brand new e-commerce penalties in instances of -30, -900 and -supphell. NOT! A recent algo tweak may have figured out how to identify more crap, but I doubt Google has launched an anti-eCommerce campaign.

One don't need an award-winning mid-range e-commerce shopping cart like Erol to gain the Google death penalty. Thanks to this award winning software sold as "search engine friendly" on the home page, respectively its crappy architecture (sneaky JS redirects as per Google's Webmaster guidelines), many innocent shopping sites from Erol's client list have vanished, or will be deindexed soon. Unbelievable when you read more about their so-called SEO Services. Oh well, so far an actual example. The following comments do not address Erol shopping carts, but e-commerce systems in general.

My usual question when asked to optimize eCommerce sites is "are you willing to dump everything except the core shopping cart module?". Unfortunately, that's the best as well as the cheapest solution in most cases. The technical crux with eCommerce software is, that it's developed by programmers, not Web developers, and software shops don't bother asking for SEO advice. The result is often fancy crap.

Another common problem is, that the UI is optimized for shoppers (that's a subclass of 'surfers', the latter is decently emulated by search engine crawlers). Navigation is mostly shortcut- and search driven (POST created results not crawlable) and relies on variables stored in cookies and whereever (invisible to spiders). All the navigational goodies which make the surfing experience are implemented with client sided technologies, or -if put server sided- served by ugly URLs with nasty session-IDs (ignored by crawlers or at least heavily downranked for various reasons). What's left for the engines? Deep hierarchical structures of thin pages plastered with duplicated text and buy-now links. That's not the sort of spider food Ms. Googlebot and her colleagues love to eat.

Guess why Google doesn't crawl search results. Because search results are an inedible spider fodder not worth indexing. The same goes for badly linked conglomerates of thin product pages. Think of a different approach. Instead of trying to shove thin product pages into search indexes write informative pages on product lines/groups/... and link to the product pages within the text. When these well linked info pages provide enough product details they'll rank for product related search queries. And you'll generate linkworthy content. Don't forget to disallow /shop, /search and /products in your robots.txt.


Disclaimer: I've checked essentialaids.com, Erol's software does JavaScript redirects obfuscating the linked URLs to deliver the content client sided. I've followed this case over a few days watching Google deindexing the whole site page by page. This kind of redirects is considered "sneaky" by Google and Google's spam filters detect it automatically. Although there is no bad intent, Google bans all sites using this technique. Since this is a key feature of the software, how can they advertise it as "search engine friendly"? From their testimonials (most are affiliates) I've looked at irishmusicmail.com and found that Google has indexed only 250 pages from well over 800, it looks like the Erol shopping system was removed. The other non-affiliated testimonial is from heroesforkids.co.uk, a badly framed site which is also not viewable without JavaScript. Due to SE-unfriendliness Google has indexed only 50 out of 190 pages (deindexing the site a few days later). Another reference brambleandwillow.com didn't load at all, Google has no references but I found Erol-styled URLs in Yahoo's index. Next pensdirect.co.uk suffers from the same flawed architecture as heroesforkids.co.uk, although the pages/indexed-URLs ratio is slightly better (15 of 40+). From a quick look at the Erol JS source all pages will get removed from Google's search index. I didn't write that to slander Erol and its inventor Dreamteam UK, however these guys would deserve it. It's just a warning that good looking software which might perfectly support all related business processes can be extremely destructive from a SEO perspective.

Update: Probably it's possible to make Erol driven shops compliant to Google's quality guidelines by creating the pages without a software functionality called "page loading messages". More information is provided by several posts in Erol's support forums.

Tags: Search Engine Optimization (SEO) eCommerce Shopping Carts e-commerce software Web development

Stumble It!

Post it to del.icio.us