My book manuscript, Bit by Bit: Social Research in the Digital Age, is now in Open Review. That means that while the book manuscript goes through traditional peer review, I also posted it online for a parallel Open Review. During the Open Review everyone—not just traditional peer reviewers—can read the manuscript and help make it better.
Schematic of Open Review.
Screenshot of Open Review interface. Click for full size.
First a bit of background, Bit by Bit: Social Research in the Digital Age is a book for social scientists who want to do more data science, data scientists who want to do more social science, and anyone interested in the future of social research. Here’s how the preface begins:
“For me, this book began in 2005, when I was working on my dissertation. I was running an online experiment, which I’ll tell you all about in Chapter 4, but now I’m going to tell you something that is not in any academic paper. And, it’s something that fundamentally changed how I think about research. One morning, when I checked the web-server, I discovered that overnight about 100 people from Brazil had participated in my experiment. This experience had a profound impact on me. At that time, I had friends who were running traditional lab experiments, and I knew how hard they had to work to recruit, supervise, and pay people to participate in their experiments; if they could run 10 people in a single day, that was good progress. But, with my online experiment, 100 people participated while I was sleeping. Doing your research while you are sleeping might sound too good to be true, but it isn’t. Changes in technology—specifically the transition from the analog age to the digital age—mean that we can now collect and analyze social data in new ways. This book is about doing social research in these new ways.”
When I submitted the manuscript to Princeton University Press for peer review, I also created an Open Review website and posted the entire manuscript online (in a later post, I’ll write more about the build process for the website). One of the reasons that I was so excited about the Open Review website is that it offered me a chance to get feedback from people with different perspectives and intellectual backgrounds. And, I have not been disappointed. So far I’ve received more than 100 annotations from about 10 people. In a future post, I’ll do a more quantitative analysis of the feedback, but in this post I’d like to offer some more general impressions.
First, and most importantly, this feedback has lead to changes that definitely improved the manuscript. The annotations have identified a number of typos and grammatical errors. But, more important than that, the annotations have identified parts of the text that readers found confusing, unconvincing, or exciting. Further, a couple of annotations have pointed me to interesting papers that I had not seen. Thus, so far, the feedback has led to changes that are evolutionary not revolutionary. But, many incremental improvements results in a noticeably clearer, crisper book.
During this month I’ve also learned that the Open Review process itself can be improved. I think the biggest challenge is encouraging readers to offer more complex—and therefore more valuable—feedback. Feedback about typos is helpful, but the book will eventually have a copy editor who should catch these. Therefore, I’d prefer for readers to focus their feedback on more intellectual aspects of the book. I think that part of the problem is that I didn’t do a great job of setting expectations for what Open Review is about and so people thought that it was about typos. Let me be clear: Open Review is not about typos. In order to address this problem, I’ve updated the page where I explain Open Review. Eventually, the norms for this process will become more standardized, and future authors should not have as much difficulty trying to explain the process or set expectations.
I’d like to end this post by answering a number of questions that people have asked me about Open Review.
Who are the people giving feedback?
I would like to thank the people who have offered helpful feedback during the Open Review process: jeschonnek.1, Nick_Adams, DBLarremore, Nicolemarwell, dmf, cc23, efosse, cfelton, jboy, and jugander.
Also, I would like to thank Arvind Narayanan, Betsy Paluck, Chico Bastos, Nick Feamster, and Don Dillman for sending me feedback through email (feedback that was generated by the Open Review process but which is not visible to everyone).
The people offering feedback are a wonderful mix. Some are undergraduates; some are postdocs; and some are professors. Some are social scientists; some are data scientists; and some are computational social scientists. Some live in the US and some live in Europe. And, my guess is that none of them would have been asked by my publisher (Princeton University Press) to do a peer review. In other words, Open Review collects feedback that would not come through traditional peer review.
Can I see the annotations?
Yes, everyone can see them. There are two main ways to see them. First, you can see them on the right side of the page on the Open Review website. And, you can see a stream of annotations from the hypothes.is website.
Why are you letting these people write your book?
I am not letting these people write my book. I promise to read and think about all the feedback received during Open Review, but I won’t make all the changes suggested by the participants. This book represent my view; it is not a wiki book. At this point, I think I’ve made changes based on about half of the annotations.
How did you get brave enough to do Open Review?
I see it totally differently. To me, bravery is writing a book and then having it published without getting feedback from as many people as possible.
Can Open Review replace peer review?
I don’t think so. Open Review—at least so far—seems to lead to short comments on small chunks of the text. I expect that peer review will lead to longer comments on the entire arc and argument of the book. These are two different—and complementary—things. Also, peer review helps the publisher assess the quality of the text in a way that Open Review does not.
How long will Open Review last?
Basically as long as peer review lasts, which I expect to be a few months. Open Review ends ends when I submit my final manuscript to my publisher.
How do manage all of the feedback?
We’re using hypothes.is, an amazing open source annotation system.
Can I participate in Open Review?
Yes. Please visit http://www.bitbybitbook.com and start annotating.
Can I put my book into Open Review?
Yes. In fact, we are about to make an announcement that should make it much easier for everyone to put their book into Open Review.
