content: You didn't misunderstand what they are doing, but your conclusion that they aren't going to make things 'standards compliant' as well is where you're incorrect. If you look at Chrome and Firefox, they are both standards compliant, but they also have a crap load of hax in them for patterns that aren't correct, but websites use all the time.  Recent (but older) versions of IE were standards compliant, but didn't have a lot of those site specific hax that Chrome has.  This left the feeling that IE was improperly rendering things, when actually it was the site that was delivering bad content. Having the crawler go through and find distinct things that sites are using, and aren't necessarily in the standard seems like a good way to go about that.