Message 00200 [Homepage] [Navigation]
Thread: joxT00189 Message: 10/77 L2 [In date index] [In thread index]
[First in Thread] [Last in Thread] [Date Next] [Date Prev]
[Next in Thread] [Prev in Thread] [Next Thread] [Prev Thread]

RE: Re: [jox] Multi-rating mode of evaluation (was: Multi-rating mode of evaluation / Updating papers)

[Converted from multipart/mixed]

[1 text/plain]
Dear all, 
sorry to engage with this very interesting discussion so late. I have joined this list only recently and might just repeat stuff that has already been said. 
I would side with those voices that have expressed their concerns about a ranking system. All the arguments against such a system have already been made by others.
I would only like to add one point. I don't necessarily see why an online journal should strive to publish a higher number of publications just because it can. The reputation of a journal will always depend on the quality of its contributions rather than the quantity of publications. One might argue now that this is a political project  and that it is not about status and reputation but about the possibility to give a platform to as many people as possible. The problem with this argument is that the former politics of inclusion and exclusion of certain voices are changing anyway. With worldpress and other platforms everybody can publish every single thought anyway. In my view an online journal can't contribute to a democratisation of 'the public sphere' as this is happening with or without academic journals. Thus a journal will always have to engage with processes of selection, that is what it makes different from facebook. 
This is of course the reason why a ranking system would be needed if there is an abundance of publications. So the alternative is either pre-selection or allowance with post-selection through ranking. I side with the traditional form, pre-selection - at least those articles that are published won't have to be subject to new lists and hierarchies and patronising classifications. A ranking system is not less elitist that traditional forms of pre-selection. After all, nobody would by a washing machine that gets one out of five stars in consumer reviews. So why should an online article that gets one out of five should do much better?


From: owner-journal on behalf of Mathieu O'Neil
Sent: Fri 04/12/2009 16:39
To: journal
Subject: Fwd: Re: [jox] Multi-rating mode of evaluation (was: Multi-rating mode of evaluation / Updating papers)

[Converted from text/html]

Hi Biella, Athina, all

Thanks for your input, points well taken. My only concern is that I
really think we need some structure, a clear system. I don't think you
can avoid some form of classification and hierarchy in any case
(starting with publish / don't publish: unless we publish everything
that is submitted) though of course we would want to make the process
as transparent and fair as possible, without it turning into an
unreasonable time cost. The main advantage of the proposed rating
system, as I saw it, was to increase the number of published papers.
As I said before the paper recently posted by Johan (which I have not
read in detail and whose author has now joined this list, btw) has
some useful suggestions (such as initial feedback on proposals) but in
my view the proposed peer review process is a bit drawn-out and may
complicate the author-reviewer interaction. This could be a good
thing, not sure. It's certainly an interesting proposal. At the same
time the only way to find out if a system works is simply to try it


ps. Hope the mailer issue is resolved?

-------- Original Message --------
From: Gabriella Coleman<biella>

Hi folks,

Athina has laid out most of my concerns so there is not much I will
want to add except the following.

What I "like" about the idea behind the rating system is the
commitment to be more self-conscious about what types of papers we
would like to see published and push against the conservative
tendencies in peer review, which often punish controversial papers.

What I don' think we need is some numberical rating system. Why do we
need a rating system to engender a review process /journal that takes
a different tack toward the paper it accepts? Why can't a
conversation/conscioussness about what we expect, what we would like
to see not take its place? This is where again a conversation between
the reviewers might be the way to do it, even if brief.

I am part of a journal collective that has a pretty explicit process
for paper submissions, peer review (which is not blind) etc. We review
everything ourselves and  there is a bias in many respects that folks
are comfortable with. In many respects, why can't we foster "a bias"
within the group that takes on the challenge of accepting papers that
might otherwise be shut out? Why do we need to cede this ethical/
epistemological commitment to a rating system?

Finally: I think we should be open to different types of articles:
those that are controversial, those that are more essay like and
extremely provocative (and lack "data" see Prickly Paradigm Press
books for what I am thinking about; they are excellent and moreover
excellent in the classroom), for example.  But nonetheless, I think we
should make sure that we have a basic and high standard for
communicability. Otherwise it will just reflect poorly on us and
counteract everything we are trying to do.

All best,

Athina Karatzogianni wrote:
[Converted from multipart/alternative]

[1 text/plain]
Hi All,

My view is indeed that rating is unnecessary for many reasons. These
are my
more important ones but I can think of many more:

  - you are treating the readers as idiots who cant do it themselves
  - you are essentially playing favorites with the texts
  - rating is a very subjective business and disagreeing with
  position irrespective of the quality of the paper might confuse
the two, if
  you disagree but the work is great how do you rate? The mechanism
  describe really doesnt say anything to me
  - especially when on cutting edge research matters, it is
difficult to
  even rate altogether
  - you are creating a HIERARCHY of  papers and  i do find
hierarchies in
  general quite distasteful. For a progressive journal on p2p this
could be
  very problematic.
  - What is a better paper and who decides? Majority decision making
  not work well in real life, why do you want to transfer it onto an
  experimental progressive journal?
  - Also, people on it might feel uncomfortable with the rating and
it is a
  very technocratic thing to do altogether. I mean this is not a
  product and we dont sell DVD players!!
  - Lastly personally trust very few researchers' opinion about
research to
  begin with. Do you trust everyone? Do you trust the majority?

Nevertheless, I told Mathieu that all you can  do is experiment see
how it
goes in my personal communication with him. In which case, please go
and I just register my concerns on the issue. Only trial and error
settle this one.


On Fri, Dec 4, 2009 at 8:56 AM, Mathieu O'Neil <
mathieu.oneil> wrote:

[Converted from text/html]

Hi Brian, Johan, all

I looked quickly at the paper Johan posted - the argument is that
improvement of submissions should be inspired by FLOSS development
adopting mechanisms such as this email list: authors post a
suggestion, reviewers collectively point out possible omissions, the
author goes back to work then submits a more fleshed-out version,
people on the list point out more "bugs", then more fixing up, etc,
until publication. The author argues that there has to be a
(worldwide?) social contract that no-one will quote the discussions
submissions until publication. This last part seems hard to enforce,
plus the process seems to me a bit drawn-out but I really like the
initial stage (get list feedback on abstracts / submissions).
this could this be combined to a rating system?



From: bwhitworth<b.whitworth>

This is an interesting discussion - sorry I cant participate fully
to a current overload.

By our First Monday analysis journals accepting only the best, and
those "open to all", are both paths well travelled in academic
publishing, with known outcomes - the first gives rigor but not
relevance and the second reduces quality and recognition.

Statements like "We should publish only papers that we agree are fit
for publication" or "We should ..." in general assume that we
the journal. Our paper at
that control mentality
to introduce the ideal of democracy in academic publishing, i.e.
government by the people for the people

Reviewers imposing grades on accepted publications only denigrate
authors if the journal makes them publish. If authors choose to
publish how can they be offended? If they were offended they would
just choose not to publish. Reviewing is then just the journal doing
its job.

A journal that cant be bothered rating its submisions doesn't
to succeed. Equally one that selects the best and leaves the rest is
elitist. There is no easy way between these options, so we suggested
both highly selective reviewing and completely open publishing. The
multi-grade system lets anyone publish but all neednt be rated equal
though there can be multiple criteria. I guess this goes against the
politically correct idea that we are all equal, but actually some of
us run faster, others cook better and a few of us can actually do
mathematics - so really "equality" is a myth. The real equality is
opportunity not ability, which is why this approach lets everyone in
who wants to come in.

Likewise the ratings of registered readers, while informal, are not
unexpected nor imposed. The public is always entitled to its
The system need only identify and ban spammes and trolls, as
does. The view of the public should not be a secret, so people can
rate what they read.

The social principles outlined in our paper were fairness, public
good, transparency, freedom and order. Achieving all these in one
is probably impossible, as socio-technical systems are still
struggling to evolve. It is all very compex. Hopefully this journal
can make some advance on what went before, even if we dont get it
right first time.

Regardless of the outcome, thank you for trying!

Brian Whitworth
----- Original Message -----
From: "Felix Stalder" <felix>
To: <journal>
Sent: Friday, December 04, 2009 1:36 AM
Subject: Re: [jox] Multi-rating mode of evaluation (was: Multi-
mode of evaluation / Updating papers)

On Wednesday, 2. December 2009, Stefan Merten wrote:

We should publish only papers that we agree are fit for
But "fit for publication" is not based on a single reason. There
be articles which we consider great in many dimensions but they
some certain feature. Lack of this feature normally would make them
unacceptable but if we can express this lack by a rating then the
credibility of our journal is maintained and the article is
I think multi-rating models are too complicated, and patronizing to
the author and the reader. I mean, if we like the paper enough to
it in our journal, we should do it. Period.

Do we really need to say something like: we give this paper an 'a'
grammar, a 'b+' in originality, an "a-" in methodology and a 'b-
' for
bibliography? Shouldn't the reader be able to figure it our him/

If we think a paper would be great to publish, but lacks some

aspects, we should ask the authors to revise it before publishing. I
see this as censorship or forcing anything upon the author, but
as a
process of critical evaluation that leads to an improvement.



--- <>  ----------------------------- out
*|Mediale Kunst/Media Arts Zurich.13
*|Manuel Castells and the Theory of the Network Society. Polity,
*|Open Cultures and the Nature of Networks. Ed. Futura/Revolver,


No virus found in this incoming message.
Checked by AVG -
Version: 8.5.426 / Virus Database: 270.14.83/2529 - Release Date:
26/09 19:42:00

[2 text/html]

Dr Mathieu O'Neil
Adjunct Research Fellow
Australian Demographic and Social Research Institute
College of Arts and Social Science
The Australian National University
email: mathieu.oneil



Gabriella Coleman, Assistant Professor
Department of Media, Culture, & Communication
New York University
239 Greene St, 7th floor
NY NY 10003

Dr Mathieu O'Neil
Adjunct Research Fellow
Australian Demographic and Social Research Institute
College of Arts and Social Science
The Australian National University
email: mathieu.oneil


This email is intended solely for the addressee.  It may contain private and confidential information.  If you are not the intended addressee, please take no action based on it nor show a copy to anyone.  In this case, please reply to this email to highlight the error.  Opinions and information in this email that do not relate to the official business of Nottingham Trent University shall be understood as neither given nor endorsed by the University.
Nottingham Trent University has taken steps to ensure that this email and any attachments are virus-free, but we do advise that the recipient should check that the email and its attachments are actually virus free.  This is in keeping with good computing practice.

[2 application/ms-tnef]

Thread: joxT00189 Message: 10/77 L2 [In date index] [In thread index]
Message 00200 [Homepage] [Navigation]