Message 00238 [Homepage] [Navigation]
Thread: joxT00189 Message: 13/77 L5 [In date index] [In thread index]
[First in Thread] [Last in Thread] [Date Next] [Date Prev]
[Next in Thread] [Prev in Thread] [Next Thread] [Prev Thread]

Re: [jox] Cutting the Knot

[Converted from multipart/alternative]

[1 text/plain]
Hi Ed, all

Thanks for those thoughtful comments. Just a quick response to clear up a couple of points.

- Categories: in the proposed model these only apply to how reviewers characterise submissions, not to the reviewers themselves. I agree with some of your points such as (a) number should be limited (b) universal would be good (c) how does one define "rigour" - for (c) that's why I suggested appreciation/commendations or reservations from reviewers instead of these categories: then it is clear that this is really subjective. OTOH I think that its reasonably obvious whether an author is coming from a more activist or academic perspective - or it may be both...
But all in all I think you have made some useful suggestions here, will digest them at length.

- Concerning the review discussion: it's agreed that these - while they are taking place - will not be public. The question to decide is whether they will occur on a website (reviewers and other authorised people have to visit the discussion) or whether it's on a private list (discussion automatically comes to the reviewers and other authorised people)? Maybe with rsss feeds you can have website content automatically come up to subscribers - but what trace does it leave? The good thing about email is that it is archived.

- A thorny issue is rejection. I dont know if you have read the piece by Whitworth and Friedman I referenced - its all about increasing publication whilst protecting (if that's the right word) journal reputation; if you combine that with community vetting and orientation (through advice provision to authors on an open list - not the same as the review list / website) as suggested in the Prug piece, rejection rates should drop a lot. In fact you will notice that there is no mention of rejection in the model I included at the end of my message... In the previous iteration of this model there was a procedure for rejection with possible votes etc but I took it out here as in my mind if someone is oriented by the list and later rated by reviewers it would be up to them to accept publication with a so-so or not perfect rating, or self-reject by refusing such a publication... So I guess we need to work out whether rejection is explicitly put in the picture or not... Hope this makes sense (and I really mean that!)

Thanks again for your comments,

----- Original Message -----
From: Ed Steinmueller <w.e.steinmueller>
Date: Monday, February 22, 2010 2:33 am
Subject: Re: [jox] Cutting the Knot
To: journal

Dear Mathieu and colleagues:

Regarding the questions that Mathieu poses in this long but very 
interesting e-mail
I suggest the following:

--Regarding Categories--
As a general desiderata:
     The number of categories for a multi-
category rating should be small,
     very clear and universal (every paper 
could be assessed in relation to
     the category).

This is a call to minimise the temptation to expand categories.

Then, more specifically,
1.  Objective should be objective -- what is activist, 
academic, or theoretical
   can be subjective for a particular instance.  
I suggest that 'prospective' (based
   upon developments that have not yet occurred) and 
'formalised' (based upon
   formal logic or mathematical technique) are 
objective.  Native English is
   ok but perhaps should be an author rather than a 
reviewer category.  Empirical
   might be further defined as 'based upon data or 
experience' to avoid confusion.
2.  Subjective should be meaningful for the audience -- 
what constitutes 'rigour'
   other than the 'utilisation of formal logic or 
mathematics' may be so subjective
   as to be useless to the audience while the phrase 
in quotes here is an objective
3.  To my taste, congratulations is too generic and again 
reveals more about the
   reviewer (their tolerance and appreciation of the 
effort required to write a paper
   at all for example) than the paper.  It is a 
category that suggests an additional
   element is needed and would be better labelled 
'commendations' which would
   specifically evoke a listing of what is to be commended.
I do think that objective, subjective, additional notes is a 
useful thematic division
for the grouping of categories.

--Regarding Open or Limited Discussion--
I think that this is a crucial choice for the community.  I 
believe that an open
forum will discourage reviewers for several reasons, the main 
ones are: a) except
for those individuals who routinely express their opinions to 
any who will listen,
the public exposure of messages directed to the author that have 
critical content
will have the effect of watering down the candour of comments or 
discouragingthem altogether,  particularly if this is done 
for papers for which a decision has
not been reached (this is a general problem with any opening of 
the editorial process)
and b) reviewing will risk becoming a forum and hence potentially
indeterminate in outcome (see comments below on editorial process).
My conclusion is that papers that are rejected should not have 
the reviews
published (see below suggestion -- this would apply to papers 
with two rejects
or one reject seconded by a third reviewers) although the author 
would of course
have access to all reviews).  The corollary of this line of 
argument is a web-based
and 'closed' (to all except editorial board, authors, and 
reviewers) review process
prior to publication.

--Regarding Editorial Process--
I do not think that the process as outlined scales very 
well.  If one were speaking
of a limited number of submissions that were initially well-
executed in most respects
with a few missing or contested bits, it might be possible to 
employ this process to
achieve a periodic 'issue' of a journal (time delimited 
publication).  If the aim is to
create an archive which grows incrementally (not time delimited 
publication) then
the 'time in process' may be less severe.  The process as 
outlined seems likely to
sacrifice quality in favour of closure.  Here is why I 
reach that conclusion:

In my role as a reviewer, I rarely see papers that are 
sufficiently polished to be
published without significant revision.  The process as 
outlined suggests that I make
my comments for revision and then no longer have any 
responsibility for the final

Viewing this process (as outlined with a change in reviewers 
after a revise and
resubmit), as a reviewer, I have three choices:
1)  consistently reject all papers that I think need 
significant revision in the
   of the quality of the journal (i.e. the 
preponderance of papers),
2)  make my comments for revision as my contribution to the 
community and ignore
   the subsequent process,
3)  consistently accept papers that I think have some merit 
(even if the execution
   is flawed in various ways). Executing 1) will lead 
to my exclusion as a reviewer in fairly short order or, if 
others are similarly minded (i.e. also believe that most papers 
are under-polished), a thin journal,
2) is problematic for the author because they will then draw 
additional reviewers with
their own (different) choices for revisions and thus discourages 
author submissions.
Thus 3) is the likely outcome of the process as outlined.

Alternatively if revisions must be executed to satisfy the 
initial reviewers (i.e. review of revised manuscripts by initial 
reviewers) the objections raised concerning the 'closed' 
(elitist) tendencies of the peer review process come to the 
fore.  In other words, this solves the problem identified 
for 2) but reproduces the current system with the only 
difference being that the community of reviewers becomes more
like a cadre or community capable of developing collective norms.

It may be that a cadre of reviewers who can effectively co-
ordinate and reach
consensus will emerge.  I suggest that this cadre will 
consist of people who have
quite a bit of time to engage in the process and thus it is an 
open (to be determined
by experience) question whether a) there is a sufficient number 
of such people to
create a reviewer community of adequate size and b) that 
achieving coherence will
reproduce some of the features of 'closed' communities which are 
being contested, i.e. the norms will again be seen as 
discouraging by authors.

The only way out of this box that I can see is that revise and 
resubmit constitutes
a commitment to publish.  In this case, I think reviewers 
should be given one more
round of comments regarding the revised papers.  Adopting 
this proposal would
make it possible to cut out the consensus formation part of the 
process.  I would
1.  Two rejects means reject.
2.  One reject and one revise and resubmit leads to a third 
reviewer being appointed.
   If this reviewer votes reject, then the paper is 
rejected, otherwise the paper is
   published after revisions in response to those 
suggested (by first round reviewers
   -- a reviewer might vote to reject but have ability 
to comment on the basis of
   'if this paper were published it would be desirable 
to...' -- and possibly 
the         third
3.  Two revise and resubmit reviews is a commitment to 
publish after revisions have
   been made with the revised submission subject to 
further published comments
   by the reviewers
4.  This last category would also be applicable to one 
accept and one revise and
   resubmit or two accepts (one might still wish to 
suggest minor corrections)

This streamlines the process and makes it more determinate and 
less costly in terms
of negotiation potentially improving the reviewer recruitment 
process.  It may reduce
the quality of the journal and discourage some reviewers but it 
will allow more
heterodox work to appear.  It is an open (to be determined 
by experience) question
how reviewers (which are not easy to recruit) will view the 
prospects of their reviews
being published for papers that are eventually published.

Where I have not made comments, I agree with the elements of the 
proposalMathieu has made.

Hope this helps...

Best regards, Ed Steinmueller

Mathieu ONeil wrote:
Hi everyone

Well, it's taken a bit of a while, but I finally got around to 
attempt to
square the circle / find the North West Passage / cut the Gordian
knot, in short: address the issue which caused so much heat last
December, our peer review system. I have come up with a solution
which I hope will satisfy most. Let me say from the start that it
involves some kind of post-publication public assessment of
submissions – I think there is no getting around that. Anyone who
is completely against this can decide if they want to continue with
the project or not, but hear me out first, yall.


1-summary of issue
2-proposed solution
3-decision: categories
4-decision: review discussion system
5-time-frame for decisions
7-CSPP peer review process: main stages

1-summary of issue

This is by no means everything that was said, I tried to get to 
the main
nut, apologies if I forgot anyone. 
The central issue we wrestled with was whether to follow the
recommendation laid out in Whitworth and Friedman's paper [1] to
accept significantly more papers, and rate these according to
different categories. There was a nice summary by Andreas 
Wittel who
said it was a question of whether to have “pre-selection”
(through accept / reject) or “post-selection” (through rankings).


Brian Whitworth logically argued for ratings:
"A journal that can't be bothered rating its submissions 
doesn't deserve
to succeed. Equally one that selects the best and leaves the 
rest is
elitist. There is no easy way between these options, so we suggested
both highly selective reviewing and completely open publishing. The
multi-grade system lets anyone publish but all needn't be rated equal
- though there can be multiple criteria. I guess this goes against
the politically correct idea that we are all equal, but 
actually some
of us run faster, others cook better and a few of us can 
actually do
mathematics - so really "equality" is a myth. The real
equality is of opportunity not ability, which is why this approach
lets everyone in who wants to come in."

However a number of people (Athina, Biella, Andreas, Felix) expressed
reservations about ratings because of their perceived overly
hierarchical, discriminatory or “school-like” nature; especially,
some did not like the numerical ranking. [See]>
Many points in favour of ratings were made by StefanMn:

StefanMn then proposed a choice where authors submitting a 
proposal could
indicate whether they want a binary model (publish or reject) 
or a
multi-dimensional rating system:

2-proposed solution

There are two main problems with his proposal. First, I think 
we should be
as consistent as possible in what we present to the world. It would
be weird to have some papers with an appreciation and others without.

Second, if we want to open up the journal selection process and 
provide rewards to those who do
normally invisible work (i.e. reviewers), in line with Toni Prug's
proposal [2] for a community peer review system (through a list 
where proposals are
vetted and reviews are released), then by definition we are rejecting
the publish / don't publish model: vetting and orientation occur
upstream, even before an actual full submission. If we agree to work
with authors by reviewing and discussing their proposal it 
would be
hard to come to the end of that process, and reject. So we will
necessarily publish more than a traditional journal, including papers
that are not written in perfect English (for example), or that are
problematic in some other way. As was said:

Felix Stalder said that “we should publish only papers that we 
agree are fit for publication” and
StefanMn responded: 
“But "fit for publication" is not based on a single reason.
There may be articles which we consider great in many 
dimensions but
they lack some certain feature. Lack of this feature normally would
make them unacceptable but if we c25an express this lack by a rating
then the credibility of our journal is maintained *and* the article
is published.”


This is the essential point, and, upon reflection, I agree with 
it. And
yet: the fact remains that people are uncomfortable with 
ratings. I
think it all depends how we do them. In order to protect the
reputation of the journal, we need to alert readers that we are aware
of flaws, but that _we decided to publish anyway_. Hence the 
need to
“qualify” or “signal” (rather than “rate”) published
submissions. Another thing: I think it is important to distinguish
between appreciations of what a submission clearly “is”
(activist, academic, native-English speaking authored, etc) and
appreciations as to what a submission “is worth” (original,
rigorous, etc): the first appreciations may be described as more
“objective” than the second. 
3-decision: categories

So we could have either a simplified choice of rating (outstanding,
excellent, fair) or an even simpler choice (yes, no) as in: this
paper is a grand grassroots testimony (Activist: yes) but the English
is not perfect (Native English: no) or: this paper is
academic-research oriented (Academic: yes) but it is not based on
empirical evidence (Empirical: no); or: this is a utopian fantasy
about how a peer-produced society would produce gastronomical
delights (Theoretical, Activist: yes); etc.

A-For “objective” categories we could have:
-Native English

B-For “subjective” categories we could have:

we could ask reviewers to indicate whether they want to express:
-Reservations (yes/no) -Congratulations (yes/no) about the 
[Thus indicating clearly that this is a personal opinion].

In all cases these “appreciations” (whether Activism, Originality,
Congratulations) would have to be clearly linked to the reviews which
would be released when the submission is published.

Finally the editor should have the right to intervene and add 
an appreciation
if he or she feels that reviewers have been unduly harsh or

So, this is the first point to decide: what categories do we have?

4-decision: review discussion system

The other point to be decided concerns the process of 
discussion of
submissions: should these be held on a restricted mailing list 
(to be
clear: not the one we are using now, which is open, but one that
would be reserved to reviewers and authors) or on a protected 
part of
the website?

StefanMn argued for the website option:

“Well, in general I'm a big fan of mailing lists. But in this 
case I think a web based system would be more useful. I'd 
suggest to offer potential authors a place where they can 
propose an article in the way outlined above and reviewers can 
help the author to write a great article. I think a web page is 
more useful because it gives every stakeholder a clear structure 
where the subject is *one* proposed article.” [See: lost the 
ref, sorry]

It's true that having one (restricted) mailing list where all 
submissions>are discussed could be messy (though less so if 
people do not
interfere with titles of emails thereby breaking threads). And it
might be easier to create files that can be used later on in the
website when publishing, I don't know. At the same time I see some
problems with setting up discrete pages for articles: a) 
authors and
reviewers might in fact benefit from reading discussions on other
articles; b) not sure about this, but there might be 
complications in
access rights – who can access what article page, etc?: c) finally
the advantage of the list is that you are kept abreast of discussion
as they go along, whether you seek the information or not –
otherwise many people (myself included) might not go to the website
very often: with a list, you have no choice, you get the message.
This is a strong advantage, in my view.

So it would be great to hear people's opinion on this second 
5-time-frame for decisions

Ideally we should be able to formally announce the journal and 
call for
papers at the conference which Athina has kindly organised as our
launch party – VIRT3C in Hull, in a month – woo-hoo! 
[What do you mean, it wasn't organised for that?! Oh well, my 
mistake. ;-)]

In any case decisions should be finalised a couple weeks before the
conference – in order to give ourselves time to set up anything
that needs to be set up (particularly if we go with the website
discussion option).  So I think a
two-week period should be enough - today is Friday 19 February March
2010 – I am invoking Maintainer Mojo to request that final
decisions be reached by Saturday 6 March 2010. No further debate
after that will be accepted as relevant. Tough love, people!

Whitworth B and R Friedman (2009a) "Reinventing academic publishing
online. Part I: Rigor, relevance and practice", First
Monday, Volume 14, Number 8 - 3 August 2009; Whitworth B and R 
Friedman>(2009b) "Reinventing academic publishing online. Part 
socio-technical vision", First Monday, Volume 14, Number 9 - 7 
September 2009.

Toni Prug, “Open-process Academic Publishing”

peer review process: main stages

Prospective authors submit a proposal to the list.

All list members vet this proposal during a reasonable period 
of time (1-2 weeks?): is it appropriate for the journal, are 
arguments or references missing?

Authors write their submission.

Authors submit to the journal.

The submission is posted by the editor to a password-protected 
part of the website [mailing list?] who also alerts the main 
journal list that he has done so.

The editor suggests two expert reviewers (volunteers welcome).

The two expert reviewers read and evaluate the submission 
during a reasonable period of time (3 weeks?). Reviewers are 
encouraged to coordinate their

Reviewers post their reviews and recommendations to a password-
protected part of site [mailing list?] and alert the list that 
they have done so.

The list discusses this during a reasonable period of time (1-2 
During this time consensus emerges: publish, revise and 
resubmit (to two other reviewers, for example?), or

During this time consensus does not emerge: the decision then 
moves to a formal vote on the Governance Board: publish, revise 
and resubmit
(to two other reviewers, for example) or reject.

Submission and review process published.

Readers can comment and rate.

Authors can respond in comments section [and add links in the 
text to relevant comments and responses - no updating of text though].




Dr Mathieu O'Neil
Adjunct Research Fellow
Australian Demographic and Social Research Institute
College of Arts and Social Science
The Australian National University
email: mathieu.oneil[at]

[2 text/html]

Thread: joxT00189 Message: 13/77 L5 [In date index] [In thread index]
Message 00238 [Homepage] [Navigation]