Message 00166 [Homepage] [Navigation]
Thread: joxT00000 Message: 118/176 L38 [In date index] [In thread index]
[First in Thread] [Last in Thread] [Date Next] [Date Prev]
[Next in Thread] [Prev in Thread] [Next Thread] [Prev Thread]

Re: [jox] Re: Multi-rating mode of evaluation / Updating papers

[Converted from multipart/alternative]

[1 text/plain]
Hi Christian, all

I think your suggestion that reviewer reports (and possibly ratings if we go that route) could be made accessible to readers is a good one. That would also be a way of recognising the invisible work of reviewers - though should these visible reviews be signed? 

Regarding "controversial" POVs I'm not 100% sure what is being alluded to. I'm assuming that any submitted paper would be rational, and if it is, then it can be rationally evaluated, criticised and discussed, whatever the POV is (this may be an obvious point but rational POVs that are also clearly racist, sexist, etc would not be acceptable in my view).


----- Original Message -----
From: Christian Siefkes <christian>
Date: Monday, November 23, 2009 2:24 am
Subject: Re: [jox] Re: Multi-rating mode of evaluation / Updating papers
To: journal

Hi all,

Stefan Meretz wrote:
The journal might contain strong pov articles, which is good. 
One aim of 
the Journal must be to support a "thinking against the 
mainstream". If 
peer production is really a new thing, then theorzing around 
this topic 
will and has to be new and unfamiliar. -- Will the expert 
committee work 
in this fashion? Or are unfamiliar povs are rated out, because 
the pov 
is not shared? (this does say anything against the persons 
listed which 
I don't know).

if a public rating system is used, POV and other controversial 
articles will
generally end up with an average rating, since some reviewers 
like them,
while others don't. Hence it might indeed be better to stick 
with a binary
"publish / don't publish" decision from the journal editors / 
reviewingcommittee, and let the readers do the ratings. Of 
course, reader ratings
will have the same effect of punishing controversial items, but 
at least
reader ratings don't sound quite as "official" as committee ratings.
(Incidentally, I noted that effect with the IMDB: excellent, but 
unusualmovies often get an average rating -- typically, high 
ratings do indeed
indicate that a movie is good, but average or poor rating don't 
give much
reliable information about the movie.)

Of course, internally the reviewers will probably use some kind 
of rating
system, since peer review usually does, and if we want 
transparency we can't
hide these ratings from the readers -- but at least I wouldn't 
show them by
default, but place them somewhere in the background information 
about the
article (reader has to click on "reviewing process / reviewer 
feedback" or
something like that).

Best regards

|------- Dr. Christian Siefkes ------- christian ----
| Homepage: | Blog:
|    Peer Production 
|---------------------------------- OpenPGP Key ID: 0x346452D8 --
A misleading benchmark test can accomplish in minutes what years 
of good
engineering can never do.
        -- Dilbert

Dr Mathieu O'Neil
Adjunct Research Fellow
Australian Demographic and Social Research Institute
College of Arts and Social Science
The Australian National University

E-mail: mathieu.oneil
Tel.: (61 02) 61 25 38 00
Mail: Coombs Building, 9
Canberra, ACT 0200 - AUSTRALIA

[2 text/html]

Thread: joxT00000 Message: 118/176 L38 [In date index] [In thread index]
Message 00166 [Homepage] [Navigation]