Hexbyte  News  Computers This economics journal only publishes results that are no big deal

Hexbyte News Computers This economics journal only publishes results that are no big deal

Hexbyte News Computers

Most new publications, upon their launch, seek to promote their content as novel, surprising, exciting.

A new journal that began publishing this week does … the opposite of that.

Start with the name: Series of Unsurprising Results in Economics (SURE). The journal publishes papers with findings that are, well, really boring — so boring that other journals rejected them just for being boring. Its first paper, published Tuesday, is about an education intervention that was found to have no effects at all on anything.

But before you close this tab, hear me out. SURE is actually far from boring, even if the papers it publishes are guaranteed to be, as the name implies, unsurprising. In fact, it’s a pretty big deal, and a significant step toward fixing a major problem with scientific research.

SURE exists to fight “publication bias,” which affects every research field out there. Publication bias works like this: Let’s say hundreds of scientists are studying a topic. The ones who find counterintuitive, surprising results in their data will publish those surprising results as papers.

The ones who find extremely standard, unsurprising results — say, “This intervention does not have any effects,” or, “There doesn’t seem to be a strong relationship between any of these variables” — will usually get rejected from journals, if they bother turning their disappointing results into a paper at all.

That’s because journals like to publish novel results that change our understanding of the field. Null results (where the researchers didn’t find anything) or boring results (where they confirm something we already know) are much less likely to be published. And efforts to replicate other people’s papers often aren’t published, either, because journals want something new and different.

That makes sense — but it’s terrible for science. This tendency leads researchers to waste time on analyses that other researchers may have already pursued but not publicized; to twist their data for results so they can publish when they initially don’t find anything; and to look for surprising outliers instead of the often mundane reality.

But awareness about this problem is growing. And in response, scientists are trying to build better processes. SURE is one step toward that goal.

How publication bias can mislead us

SURE is an online-only, open-access, no-fee journal. It accepts papers that its independent peer reviewers verify are “high quality” and that were rejected from other economics journals only because their results were statistically insignificant or otherwise unsurprising.

The first paper published, for example, by Nick Huntington-Klein and Andrew M. Gill at California State University, looked at whether informing students about the benefits of taking more credit hours (to improve their odds of graduating) would motivate them to take more classes or finish school sooner. It doesn’t. That’s unfortunate, but now we know, and other researchers can avoid this dead end. The published results will help steer clear of publication bias too.

Publication bias is often cited as a major factor in the so-called replication crisis in research. We’ve started to look back at old results in fields from medicine to psychology and have found that we can’t reproduce many of the claims in those papers — so they may have been wrong. Scientists are realizing that better methodology is needed across the board to avoid publishing research that gets it wrong.

Here’s how publication bias works: Imagine that 200 scientists go to work on an important question, like, say, which early childhood interventions improve test scores in fourth grade. (I picked that question because there’s a good case that the correct answer is “none of them.”) Most of the researchers find no results. They don’t publish those findings, just sadly call it off and move on to a different research project.

But some of them will get results — by pure chance. A common convention is to declare results “statistically significant” if they have a p-value of less than 0.05, which simply means there’s a less than 5 percent chance that the result a study found would have occurred by coincidence if there were no real effect there at all.

That means that if you have hundreds of studies, a dozen of them will find a p-value that’s less than 0.05, just by chance. And because those findings are surprising, a bunch of papers will be published identifying promising interventions that do not, in fact, get results.

That has all kinds of consequences. Using the published research, charities and policymakers might start trying to implement the interventions, and end up wasting money and resources on things that don’t work.

There are more considerations at work here too. Not publishing enough papers can hold back an academic’s career, which makes it hard to just move on when the data comes up empty. Driven by this imperative, some scientists will rerun their numbers, comparing different variables, in search of a statistically significant result that they can then publish. That makes it vastly more likely you’ll get a result you can write a paper about — but it’s intellectually dishonest, and the results will likely be false.

That’s where SURE comes in. If you conducted a rigorous study but journals find your result too boring to publish, SURE will publish it. The hope is that this will fix the incentives for the whole field. More null results will get published, mitigating publication bias. Researchers can get a paper published even if they found null or unexciting results, which should discourage scouring their data for unreliable results.

If SURE works, hopefully it’ll be emulated — economics isn’t the only field that needs it. While exciting results get headlines, it’s the boring results that often do the most to add to our knowledge of the world. Those boring results deserve a journal of their own.


Sign up for the Future Perfect newsletter. Twice a week, you’ll get a roundup of ideas and solutions for tackling our biggest challenges: improving public health, decreasing human and animal suffering, easing catastrophic risks, and — to put it simply — getting better at doing good.

Read More

Hexbyte  Hacker News  Computers SSC Journal Club: Dissolving The Fermi Paradox

Hexbyte Hacker News Computers SSC Journal Club: Dissolving The Fermi Paradox

Hexbyte Hacker News Computers

I’m late to posting this, but it’s important enough to be worth sharing anyway: Sandberg, Drexler, and Ord on Dissolving the Fermi Paradox.

(You may recognize these names: Toby Ord founded the effective altruism movement; Eric Drexler kindled interest in nanotechnology; Anders Sandberg helped pioneer the academic study of x-risk, and wrote what might be my favorite Unsong fanfic)

The Fermi Paradox asks: given the immense number of stars in our galaxy, for even a very tiny chance of aliens per star shouldn’t there should be thousands of nearby alien civilizations? But any alien civilization that arose millions of years ago would have had ample time to colonize the galaxy or do something equally dramatic that would leave no doubt as to its existence. So where are they?

This is sometimes formalized as the Drake Equation: think up all the parameters you would need for an alien civilization to contact us, multiply our best estimates for all of them together, and see how many alien civilizations we predict. So for example if we think there’s a 10% chance of each star having planets, a 10% chance of each planet being habitable to life, and a 10% chance of a life-habitable planet spawning an alien civilization by now, one in a thousand stars should have civilization. The actual Drake Equation is much more complicated, but most people agree that our best-guess values for most parameters suggest a vanishingly small chance of the empty galaxy we observe.

SDO’s contribution is to point out this is the wrong way to think about it. Sniffnoy’s comment on the subreddit helped me understand exactly what was going on, which I think is something like this:

Imagine we knew God flipped a coin. If it came up heads, He made 10 billion alien civilization. If it came up tails, He made none besides Earth. Using our one parameter Drake Equation, we determine that on average there should be 5 billion alien civilizations. Since we see zero, that’s quite the paradox, isn’t it?

No. In this case the mean is meaningless. It’s not at all surprising that we see zero alien civilizations, it just means the coin must have landed tails.

SDO say that relying on the Drake Equation is the same kind of error. We’re not interested in the average number of alien civilizations, we’re interested in the distribution of probability over number of alien civilizations. In particular, what is the probability of few-to-none?

SDO solve this with a “synthetic point estimate” model, where they choose random points from the distribution of possible estimates suggested by the research community, run the simulation a bunch of times, and see how often it returns different values.

According to their calculations, a standard Drake Equation multiplying our best estimates for every parameter together yields a probability of less than one in a million billion billion billion that we’re alone in our galaxy – making such an observation pretty paradoxical. SDO’s own method, taking account parameter uncertainty into account, yields a probability of one in three.

They try their hand at doing a Drake calculation of their own, using their preferred values, and find:


N is the average number of civilizations per galaxy

If this is right – and we can debate exact parameter values forever, but it’s hard to argue with their point-estimate-vs-distribution-logic – then there’s no Fermi Paradox. It’s done, solved, kaput. Their title, “Dissolving The Fermi Paradox”, is a strong claim, but as far as I can tell they totally deserve it.

“Why didn’t anyone think of this before?” is the question I am only slightly embarrassed to ask given that I didn’t think of it before. I don’t know. Maybe people thought of it before, but didn’t publish it, or published it somewhere I don’t know about? Maybe people intuitively figured out what was up (one of the parameters of the Drake Equation must be much lower than our estimate) but stopped there and didn’t bother explaining the formal probability argument. Maybe nobody took the Drake Equation seriously anyway, and it’s just used as a starting point to discuss the probability of life forming?

But any explanation of the “oh, everyone knew this in some sense already” sort has to deal with that a lot of very smart and well-credentialled experts treated the Fermi Paradox very seriously and came up with all sorts of weird explanations. There’s no need for sci-fi theories any more (though you should still read the Dark Forest trilogy). It’s just that there aren’t very many aliens. I think my past speculations on this, though very incomplete and much inferior to the recent paper, come out pretty well here.

(some more discussion here on Less Wrong)

One other highlight hidden in the supplement: in the midst of a long discussion on the various ways intelligent life can fail to form, starting on page 6 the authors speculate on “alternative genetic systems”. If a planet gets life with a slightly different way of encoding genes than our own, it might be too unstable to allow complex life, or too stable to allow a reasonable rate of mutation by natural selection. It may be that abiogenesis can only create very weak genetic codes, and life needs to go through several “genetic-genetic transitions” before it can reach anything capable of complex evolution. If this is path-dependent – ie there are branches that are local improvements but close off access to other better genetic systems – this could permanently arrest the development of life, or freeze it at an evolutionary rate so low that the history of the universe so far is too short a time to see complex organisms.

I don’t claim to understand all of this, but the parts I do understand are fascinating and could easily be their own paper.

Read More