Traffic measurement #2: why comScore got Facebook wrong, i.e. take all of their data with a grain of salt

On Wednesday, veteran journalist and blogger Om Malik posted a provocative question: Facebook Traffic Tanks - This can't be real?

comScore is about to issue September 2007 user engagement and page views data, and ... there seems to be a 9.3% decline in [Facebook's] unique visitors from 33.75 million in August 2007 to 30.6 million in September 2007. Even their page views are down 3.8% from August 2007. (See chart.)
...
Why would that happen to both Facebook [and Classmates] ahead of back-to-school season? It just doesn’t make any sense!

Andy Beal of Marketing Pilgrim pointed out that it's a seasonal blip that showed up in Hitwise data last year. Great catch, but still doesn't explain why apparent traffic should drop when usage is almost certainly growing.

Yesterday Om filled in the details:

Paul Sutter, co-founder & president of Quantcast explains that the dip we are seeing is because of the panel model adopted by comScore. ComScore has a panel that has a bias toward Internet users who log on from home. The same is true of all measurement panels - Nielsen, Hitwise or Quantcast. As kids go back to school, they vanish from the panel, even though they are still using Facebook, from school dorms.

(Quantcast supplements their panel with direct measurement, which we use.)

As I said earlier today when defending comScore and Reuters:

Every source for data on page views, visits, unique visitors, links, and time spent is flawed. But imperfect data is better than no data.

Gathering and organizing data is part of the central mission of Blogcosm; perhaps I should add the above to a footer that runs on every page. I'm not picking on comScore here, the point is to take all site data with a grain of salt. Please add a comment if you have other specific examples.

Blog profile: GigaOm