{"id":1890,"date":"2016-03-21T23:00:45","date_gmt":"2016-03-22T04:00:45","guid":{"rendered":"http:\/\/blogs.ams.org\/blogonmathblogs\/?p=1890"},"modified":"2016-03-22T01:11:50","modified_gmt":"2016-03-22T06:11:50","slug":"all-the-p-values-fit-to-print","status":"publish","type":"post","link":"https:\/\/blogs.ams.org\/blogonmathblogs\/2016\/03\/21\/all-the-p-values-fit-to-print\/","title":{"rendered":"All the P-values Fit to Print"},"content":{"rendered":"
\"A<\/a>

A bell curve. Image: Cris, via Flickr.<\/p><\/div>\n

I\u00a0feel\u00a0like I\u2019ve seen news stories or blog posts about p-values every day this month. First, Andrew Gelman reported<\/span><\/a> that the editor of the journal Psychological Science, famous to some for publishing dubious findings on the strength of p<0.05, will be getting serious about the replicability crisis<\/a><\/span>. (The editorial he referenced came out last November, but Gelman tends to write posts a few months in advance.) Then the American Statistical Association released a statement about p-values<\/span><\/a>, and a few days later, the reproducibility crisis in psychology led to some back-and-forthing between groups of researchers with different perspectives on the issue.<\/span><\/p>\n

At the heart of much of the controversy is that much-maligned, often misunderstood p-value. The fact that the ASA\u2019s statement<\/span><\/a> exists at all shows how big an issue understanding and using the p-value is. The statement reads, \u201cthis was not a lightly taken step. The ASA has not previously taken positions on specific matters of statistical practice.\u201d Retraction Watch has an interview with Ron Wasserstein<\/span><\/a>, one of the people behind the ASA\u2019s statement.<\/span><\/p>\n

At 538, Christie Aschwanden<\/a><\/span> tries to find an easy definition of p-value. Unfortunately, no such definition seems to exist<\/span><\/a>. \u201cYou can get it right, or you can make it intuitive, but it\u2019s all but impossible to do both,\u201d she writes. Deborah Mayo, \u201cfrequentist in exile,\u201d has two interesting posts about how exactly p-values should be interpreted<\/span><\/a> and whether the \u201cp-value police<\/span><\/a>\u201d always get it right. Mayo<\/span><\/a> and Gelman<\/span><\/a> were also two of the twenty people who contributed supplementary material for the ASA statement on statistics. <\/span><\/p>\n

Misuse and misinterpretation of p-values are part and parcel of the ongoing reproducibility crisis in psychology. (Though some say it isn\u2019t a crisis at all<\/span><\/a>.) Once again, Retraction Watch is on it with a response to a rebuttal of a response<\/span><\/a> (once removed?) about replication studies. The post goes into some depth about a study that failed to replicate, and I found it fascinating to see how the replicating authors decided to try to adjust the original study, which was done in Israel, to make it relevant for the Virginians who were their test subjects. Gelman also has<\/span><\/a> three<\/span><\/a> posts<\/span><\/a> about the replication crisis that I found helpful.<\/span><\/p>\n

One of the underlying issues with replication is something a bit unfamiliar to me as a mathematician: inaccessible data. Not all research is published on the arXiv before showing up in a journal somewhere, so there are still paywalls around some articles. More troubling, though, is the fact that a lot of data never makes it out of the lab where it was gathered. This makes it hard for other researchers to verify computations, and it means a lot of negative results never see the light of day, leading to publication bias. The Neuroskeptic blog reports on a lab that has committed to sharing all its data, good bad and ugly.<\/span><\/a><\/span><\/p>\n

So what\u2019s the bottom line? It\u2019s easy to be pessimistic, but in the end, I agree with another post by Aschwanden: science isn\u2019t broken<\/span><\/a>. We can\u2019t expect one experiment or one number to give us a complete picture of scientific truth. She writes, <\/span><\/p>\n

\n

The uncertainty inherent in science doesn\u2019t mean that we can\u2019t use it to make important policies or decisions. It just means that we should remain cautious and adopt a mindset that\u2019s open to changing courses if new data arises. We should make the best decisions we can with the current evidence and take care not to lose sight of its strength and degree of certainty. It\u2019s no accident that every good paper includes the phrase \u2018more study is needed\u2019 \u2014 there is always more to learn.<\/span><\/p>\n<\/blockquote>\n

<\/div>","protected":false},"excerpt":{"rendered":"

I\u00a0feel\u00a0like I\u2019ve seen news stories or blog posts about p-values every day this month. First, Andrew Gelman reported that the editor of the journal Psychological Science, famous to some for publishing dubious findings on the strength of p<0.05, will be … Continue reading →<\/span><\/a><\/p>\n

<\/div>\n","protected":false},"author":61,"featured_media":0,"comment_status":"open","ping_status":"closed","sticky":false,"template":"","format":"standard","meta":{"_jetpack_memberships_contains_paid_content":false,"footnotes":""},"categories":[28],"tags":[542,198,543,544,545,313,541,463],"class_list":["post-1890","post","type-post","status-publish","format-standard","hentry","category-statistics","tag-american-statistical-association","tag-andrew-gelman","tag-deborah-mayo","tag-ivan-oransky","tag-john-ioannidis","tag-p-values","tag-reproducibility-crisis","tag-statistics"],"jetpack_featured_media_url":"","jetpack_shortlink":"https:\/\/wp.me\/p3tW3N-uu","jetpack_sharing_enabled":true,"_links":{"self":[{"href":"https:\/\/blogs.ams.org\/blogonmathblogs\/wp-json\/wp\/v2\/posts\/1890","targetHints":{"allow":["GET"]}}],"collection":[{"href":"https:\/\/blogs.ams.org\/blogonmathblogs\/wp-json\/wp\/v2\/posts"}],"about":[{"href":"https:\/\/blogs.ams.org\/blogonmathblogs\/wp-json\/wp\/v2\/types\/post"}],"author":[{"embeddable":true,"href":"https:\/\/blogs.ams.org\/blogonmathblogs\/wp-json\/wp\/v2\/users\/61"}],"replies":[{"embeddable":true,"href":"https:\/\/blogs.ams.org\/blogonmathblogs\/wp-json\/wp\/v2\/comments?post=1890"}],"version-history":[{"count":3,"href":"https:\/\/blogs.ams.org\/blogonmathblogs\/wp-json\/wp\/v2\/posts\/1890\/revisions"}],"predecessor-version":[{"id":1894,"href":"https:\/\/blogs.ams.org\/blogonmathblogs\/wp-json\/wp\/v2\/posts\/1890\/revisions\/1894"}],"wp:attachment":[{"href":"https:\/\/blogs.ams.org\/blogonmathblogs\/wp-json\/wp\/v2\/media?parent=1890"}],"wp:term":[{"taxonomy":"category","embeddable":true,"href":"https:\/\/blogs.ams.org\/blogonmathblogs\/wp-json\/wp\/v2\/categories?post=1890"},{"taxonomy":"post_tag","embeddable":true,"href":"https:\/\/blogs.ams.org\/blogonmathblogs\/wp-json\/wp\/v2\/tags?post=1890"}],"curies":[{"name":"wp","href":"https:\/\/api.w.org\/{rel}","templated":true}]}}