Message 00204 [Homepage] [Navigation]
Thread: joxT00000 Message: 123/176 L39 [In date index] [In thread index]
[First in Thread] [Last in Thread] [Date Next] [Date Prev]
[Next in Thread] [Prev in Thread] [Next Thread] [Prev Thread]

Re: [jox] more on open review process



Hi Johan and all!

Last week (8 days ago) Johan Söderberg wrote:
I attach a paper which proposes a method for opening up the process
for reviewing journals. It is written by Toni Prugi. I met him at a
conference this weekend. He was on his way of implementing these
ideas with a couple of other journals, and I suggested to him to
post it here, since we are very much in the process of debating the
same questions. Except for the argument against the defunkt, present
system, which we all know well, the actual proposal starts at page
17. As I read him, the overall advice is to keep things simple, at
least to start with. It is unrealistic that we will get it right the
first time.

Thanks for posting this - and for giving a hint where to start reading
;-) . It is really interesting and I'd like to comment the section I
read.

A Simple Transition: the Linux kernel development process
=========================================================

The above elaboration is perhaps too complex13 to be implemented
straight away, to be the next step in a move from a closed access
journal, to an open-process one. Ideally, we need a simple
transition model.14 A model that will require a minimum amount of
both additional labour and capital investment at the beginning (most
editorial boards are volunteers already stretched to limits), and
that will scale, if required, at a later stage. As Benjamin Geer
correctly suggested, Linux kernel development process is one such
model. It is well tested as it has been working well for over a
decade in software.

I think it is important to learn the lessons from peer production.

Here's how would such model work: the editor has gathered a group of
scholars who have the time and interest to do peer review; Linus
Torvalds, main author of Linux kernel calls them his
"lieutenants". There is an open mailing list, and a web
site that says: "If you want to publish an article in this
journal, you must propose your idea on the mailing list before you
write the article."

People show up on the mailing list and say things like,
"I"m thinking of writing an article explaining X, etc.,
etc." The lieutenants (and the other subscribers) say,
"That won"t work unless you deal with Y somehow. Also,
you"ve assumed that X=Q, which is doubtful. Go and read Z and
think about it some more." Thus they prevent submissions that
are based on ignorance, well-known fallacies, etc. And they do this
much more quickly than traditional peer review, because they
don"t have to read an 8,000-word article to find out that
there"s a serious problem: they can find and fix bugs at the
design stage rather than the implementation stage. As it is well
known, it is much cheaper and quicker to fix bugs at the
design stage.15 Indeed, to improve peer reviewing, some of
Armstrong"s suggestions are very close to ours: "With an
early acceptance procedure, researchers could find out whether it
was worthwhile to do research on a controversial topic before they
invested much time. An additional benefit of such a review is that
they receive suggestions from reviewers before doing the work and
can then improve the design." (Armstrong 1997:17)

I totally agree and exactly for the reasons given.

After the initial discussion, the authors then go away and produce
rough drafts, which can be incomplete, or even just outlines with
implementation notes (data to be gathered later, etc.). He posts the
draft back to the mailing list. Then people on the list say,
"OK, that looks better, but you need to make sure you deal with
A"s argument, and get data on B, etc." Thus by the time an
author submits an actual article, the editor and the peer reviewers
already have a pretty good idea of what"s in it. The author
also has a good idea of how receptive the reviewers are to the
article, and thus how likely it is to be published. This helps
everyone avoid wasting time on submissions that have no chance of
being accepted, and yet, most important, the quality control role of
the peer reviewing process is maintained.

Yes.

The lieutenants do not have to do all the reviewing themselves,
because authors comment on each other"s works in progress on
the list. It"s in their interest to do so, because the tougher
they are on each other, the less likely it is that flawed articles
will slip through the process, and the better the journal"s
reputation will become, thus making it a more prestigious place to
get published. This means less work for the lieutenants. It also
means development of a community of peer reviewers whose interest
becomes to increase the reputation of a journal in which they
publish.16

I'd disagree with this. Mostly because I think the review process
should not be published unless the author wants this to happen.

Instead of publishing issues on a regular basis, the journal can
publish each article electronically whenever it is ready. Articles
get published when the community consensus is that they"re good
enough to publish. At any given time, if there are no finished
articles, the journal does not have to publish anything; thus there
is no pressure to lower standards or to rush the process in order to
meet a deadline.

Yes.

A print issue can be treated as the "Best of", or a
special/themed issue, containing only a selection of what has been
published on-line. This process would make a journal a lively place
of activity, with authors being always kept up to date with what is
going on with their submissions, and with a possibility for any
journal reader to get engaged, on a volunteer basis, through this
open process.

Yes.

Over time, the editor should become more of a coordinator, like
Linus, whose role is mainly to establish a general editorial line
(e.g. it"s a political journal, and not one on culture, yet
papers and issues on culture are welcome if done from angles
productive for political debates and issues) and to arbitrate
between the lieutenants when they disagree.

Yes.

All that is needed for this process to start being used is an open
mailing list. For early stage ideas, author can write emails
directly to the mailing list - reviews can be done by replies. In a
later stage, authors can email Word or Open Office documents, and
reviewers can use commenting features, the system everyone is
familiar with.

Well, in general I'm a big fan of mailing lists. But in this case I
think a web based system would be more useful. I'd suggest to offer
potential authors a place where they can propose an article in the way
outlined above and reviewers can help the author to write a great
article. I think a web page is more useful because it gives every
stakeholder a clear structure where the subject is *one* proposed
article.

There is a ready available slightly more advanced option, a
Wordpress plugin17 that enables online commenting of the text
written in blog pages, where comments appear sideways to the
paragraph being commented on (Fitzpatrick 2007). There are plenty
examples on their website, check the The Iraq Study Group Report
with comments. The fanciest cutomized interface for this extension
is the one used for the McKenzie Wark's 2007 Gamer Theory (Harvard
University Press) book. Pages are shown like a deck of cards, there
are arrows underneath for next/previous navigation, and on the right
hand side is the scrolling box with comments.

For this journal we have a Plone site which offers lots of options in
this regard - though I don't know whether it has such an option.

To make it simple to start with, all that is needed is an open,
archived, easy to backup, mailing list. Other parts of the open
process can be improved later. However, it is important to remember
that Wordpress and Drupal could be a good extension of the mailing
list. Hosting for both is cheaply available, and in the case of
Wordpress, there is point-and-click backup and restore
functionality. For the needs of most academics, blogging software
provides incredibly richly and easily (minimal, often no, technical
knowledge required) extensible cooperation platform. It is a vast
collection of multiple functions providing impact far greater than
several separate pieces of individual software.

See above.

Open-process peer reviewing and citing early drafts
===================================================

There is one significant problem with the processes as open as we
are suggesting here. Although authors might like the more extensive
peer reviewing that is likely to happen on an open mailing list, it
is to expect that most of them would not want to have their work
cited, nor used anywhere, before the final version accepted by the
journal isn't ready. Or, at minimum, before they post a copy
publicly for reviews on their blog, as some authors do. It would be
extremely difficult, if not impossible, to prevent that with
technical solutions. Yet, there is a cultural safeguard parallel
from the Linux kernel development that we can reuse.

If you have a Plone based solution then it is easy to fine-tune
publication of things.

If a Linux kernel is released with a serious bug, people get
annoyed, and the author of the offending code might be publicly
embarrassed. However, if you post buggy code on the Linux kernel
mailing list and someone notices, the worst thing that will happen
to you is that you will have to fix it. Why? Because everyone knows,
it is not safe to download source code
from mailing lists and expect it to work properly. This is a
cultural thing: it is accepted that free-software mailing lists are
for hashing out ideas, not for finished work. Everything about them
screams: "Danger, Construction Work".

I think texts are more complicated than source code. For source code
the final frontier is: Does it work or not. Texts are more complicated
and thus I think a multi-dimensional rating helps users. Of course the
ratings needs to be done to the final version of the article.

Therefore, we think that peer review could be open (also suggested
by Armstrong) if it had the right cultural safeguards. There would
have to be some principle like "respect for peer review",
which meant that citing journal-mailing-list messages and
preliminary drafts in academic articles would be considered a huge
taboo. Academic ethics would have to include the idea that you can
criticise preliminary drafts as much as you want, but only on the
journal-mailing-list. If you want to criticise them anywhere else,
you have to wait until the final version, or a draft version
approved by the author, is published. In this case, we believe,
authors could be made comfortable with proposing preliminary ideas
and subsequent drafts on a mailing list, without having to fear that
they will be attacked while in the middle of writing.

With a web based solution with fine-tuned publication these concerns
would vanish.


						Grüße

						Stefan
______________________________
http://www.oekonux.org/journal



Thread: joxT00000 Message: 123/176 L39 [In date index] [In thread index]
Message 00204 [Homepage] [Navigation]