Message 00116 [Homepage] [Navigation]
Thread: joxT00000 Message: 145/176 L23 [In date index] [In thread index]
[First in Thread] [Last in Thread] [Date Next] [Date Prev]
[Next in Thread] [Prev in Thread] [Next Thread] [Prev Thread]

Re: [jox] Re: Peer Review

[Converted from multipart/alternative]

[1 text/plain]
ps. Apologies to StefanMn, he already addressed some of these issues a while ago in his comments on the articles (way down at the bottom of this email)...

----- Original Message -----
From: Mathieu O'Neil <mathieu.oneil>
Date: Saturday, September 26, 2009 5:48 pm
Subject: Re: [jox] Re: Peer Review
To: journal

[Converted from multipart/alternative]

[1 text/plain]
Hi everyone

Sorry for the long delay in resuming the processs.

First I wanted to make a suggestion to address the concern 
raised by StefanMn in relation to new reviewers changing the 
course of the project: there could be two distinct groups. 
One group, "reviewers" (not yet involved in the project), would 
be mostly involved in peer reviewing. 
Another group,"editors" (already involved), would be involved in 
peer reviewing as well as overall project governance. Over time 
if reviewers take part in discussions etc and are accepted they 
could become part of the editorial team? Just a thought.

OK, in terms of the peer review process and the articles by 
Whitworth and Friedman (WF) which I referenced a while ago - 
message below if you want to have a look.
I think these articles offer some very useful ideas. But not all 
ideas need to be put into practice. In any case they all require 
some technological expertise… 

Following are some thoughts and questions.

1_General principles
First here is a summary of what they propose in their new 
system, as opposed to the traditional system

a) higher rating discrimination (a many-point scale, not just 
b) more submissions to be rated (rate all)
c) more people to rate (community involvement)
d) different ways of rating (formal review vs. informal use ratings)

So what we could have is a more transparent process where all 
submitted articles are peer reviewed and then rated from +5 
(excellent) to –1 (don’t publish). 
However in the WF system if authors decide they want to publish 
publicly a – 1 article then it is published anyway. This is of 
course debatable…

WF suggest that + 5 articles could be published as print or 
otherwise "frozen". Graham did mention print-on-demand which 
would be something to think about for the future. Does anyone 
know anything about this? I don't..
Note: If people want to keep commenting on articles (for 
readers) or responding to comments (for authors) in my view that 
is fine but as was discussed previously I don’t think the idea 
of endlessly updating the body of an article itself (as happens 
on Wikipedia) is appropriate for a journal.

3_Anonymous review choice
Allows authors to improve their paper privately. I think for 
research papers they should all be reviewed in this way. Maybe 
we could have a choice between "light" (one reviewer) and 
"heavy" (two-three reviewers) reviews  but this may end up 
complicating the process a bit. So it might be better to just 
have normal two reviews?

4_Reader ratings
WF say these could be the done formally by votes or informally 
with mouse clicks (number of views or downloads). Thoughts? 
Obviously informally needs more technical monitoring.

5_Performance reports 
WF propose that like students who can request grades from 
universities contributors (such as authors) could request 
reports not just of publications but of citations, number of 
comments, of downloads and views generated. Reviewers could ask 
for records of their contributions to be sent directly to institution.
Possible? Desirable?

6_View filters 
This refers to only displaying content rated above "x" like on 
Slashdot I guess.
Should it be done? Can it be done?

7_Same again function 
Automatically helps people find more of the same – ie find 
content rated highly by those who rate like you.
Should it be done? Can it be done?

WF suggest:
Logical flow
Others? Less?

9_Progression of readers
I think most would agree that we should have reader input. If 
there is a rating system generated by readers then they can 
offer a counterpoint to “expert” and / or academic reviewers. WF 
suggest there could be a natural path to associate reviewer, 
reviewer, associate editor, etc

This raises the question of who can be a reader? Do we require 
credentials? Even more fundamentally, this raises the questions 
of identity and privacy for readers and commenters.
WF say (and I agree) that we need contributors to register to 
avoid spam and vandalism. They write that jounals should “check 
the credentials of _who_ is submitting not censor _what_ is 
submitted”. I think they mainly mean authors though. It is not 
entirely clear what they think about readers (a priori "expert" 
reviewers are known in-house so not concerned by this issue). My 
preference would be to encourage or demand real names for 
commenters and raters, on principle. Many of the problems on 
Wikipedia (for example) derive from anonymity and the attendant 
lack of responsibility which almost generates disruption.
Questions about this:
-- Can / should we have an automatic registration system for 
participants or would people need to approach us to be 
registered and given access?
-- Do we check that people who want to register are who they say 
they are? If yes, how?

There are other ideas such as having an automated  system 
for logging and tracking submissions but this could be set up 
later as there is already a system for doing this - me :-) - 
whereas some of the points above have no solution yet.

In short: 
To everyone: what do we want?
To StefanMn and StefanMz: what can we have?


----- Original Message -----
From: Mathieu O'Neil <mathieu.oneil>
Date: Tuesday, September 15, 2009 8:00 pm
Subject: Re: [jox] Re: Peer Review
To: journal

[Converted from multipart/alternative]

[1 text/plain]
Dear all

Apologies for slackness, I've been travelling and am currently 
snowed under. I want to re-read those papers again before 
commenting on Stefans poins. But: It does not make sense to be 
discussing openly something which only a few have seen so I'm 
taking it upon myself to post the references here. I never got 
an answer from the authors to my question as to whether it 
be OK to distribute. In French there is a saying: "Qui ne dit 
mot consent". To potential readers: please remember the second 
part of the First Monday paper is still a draft so do not 
My biggest question is: they suggest publishing everything 
is submitted with ratings ie 5 excellent, 4 good, etc even -1 
(dont publish) if the author wants. I think this is an 
interesting concept but I also think this is potentially going 
towards the "process" option we discussed before. 
articles rated 4 or 5 could periodically be gathered into a 
"stable" release that could be advertised across the weboverse?
Anyway here are the refs:
 Hi Mathieu Thanks for your enquiry and interest. Your 
journal  looks good. The social design of a KES is quite 
complex, but you can see our  first attempt to lay out 
details and spirit of the endeavor at In  
note the section on privacy, where we allow reviewers to 
reveal  themselves after the review is over if they want 
to. Also note it is still draft  until it comes out in 
FM.   Some more detail on Socio-technical design 
in  general is also given in our Handbook of STS Design, 
see  for which mashup is a 
first go at making such stuff available. In particular,  
check out my chapter 1 which gives an idea of some of the 
complexity, see
chapter1.pdf    Note that the full design is 
yet specified - we  intend to develop this working with 
collaborators in a feedback process, so if  you want to 
part of that let us know. I am working with Rob Friedman and
   Michael Browstein on this project, so I copy 
email to them  also.   all the best   
Brian Whitworth  


----- Original Message -----
From: Stefan Merten <smerten>
Date: Monday, September 7, 2009 10:50 pm
Subject: Re: [jox] Re: Peer Review
To: journal
Cc: Stefan Merten <smerten>

Dear Mathieu, CSPP Board Members, all!

Last week (9 days ago) Mathieu O'Neil wrote:
OK, I contacted one of the authors of the FM paper and he kindly
sent me links to his second part as well as another relevant

Mathieu sent the links privately. Indeed a very useful 
paper. I'll
respond to this here because otherwise the communication 
becomes to
confusing for me ;-) .

In his mail he also mentioned that they are seeking 
feedback from
people on these issues. I responded that would be possible 
but that
I was acting as part of a group and queried whether it 
be OK
to circulate the papers to this group (without really 
who was
in it though I had originally sent a link to this list's 

They are welcome. But then we need the right to put the link here
(it's a public link anyway so this should not matter too much).

@Mathieu: I hope you won't mind that I quote the following 
questions> > from your private mail here.

Last week (7 days ago) Mathieu O'Neil wrote:
I guess the main questions are: which features would we want?

In general I think those guys are very much on the same 
as we
:-) . I think the key question to solve I already mentioned 
my post
from `Mon, 24 Aug 2009 12:25:45 +0200`__. To prevent leaking 
too much
information from the paper here I only reply to what is new 
or very
different from my suggestions there or comments from others 
made here
already. I relate to the chapter "DEMOCRATIC ONLINE KNOWLEDGE


* Multi-level ratings

  Similar to my previous suggestion the 
can and 
should be
  given a number of levels reflecting how the 
reviewers / 
editorial  board assesses the submission. That solves 
problem of the binary
  accept / reject scheme. On a website it is also 
easy to flag
  articles appropriately.

  Also using several rating dimensions is a good 
idea. What 
is needed,
  however, is to reduce the many dimensions to a 
number to give
  orientation to readers. May be the scientific 
activist category
  could be one such dimension.

  I think the reviewers should still have the 
responsibility to rate
  an article. May be the reader rating could be 
dimension -
  given that there are enough ratings from the readers.

  Readers need to have an option to tune what 
they consider
  most important. Technically this would mean to 
have different
  reports using a reader's weights. That is 
technically simple.

* Submissions

  I think every submission should be addressed to 
editorial board.
  IMHO this also adds a level of responsibility on 
the part 
of the
  submitters: It's a difference whether you just 
something in an
  electronic system or ask real people to consider 
  This doesn't mean that a submission is withheld 
from the 
website. It
  can be put there immediately if the author 
wishes so.

* Anonymous contributions

  In general I don't like anonymous contributions 
as I don't
  like near-anonymous contributions from obvious 
pseudonyms. I see
  that creation of an account actually feels like 
obstacle - at
  least to me. But nonetheless I don't like 
contributions.  IMHO this at least reduces the spam 
very much.

Which of those
we want can we have without too much hassle?

I think most if not all of their suggestions and our ideas are
technically feasible. IMHO the real work (i.e. human labor) 
reading> submissions, thinking about them and making 
This real work
needs to be done anyway. The rest can and should be left to the

                                        	Grüße> > 

                                        	Stefan> > ______________________________

Dr Mathieu O'Neil
Adjunct Research Fellow
Australian Demographic and Social Research Institute
College of Arts and Social Science
The Australian National University

E-mail: mathieu.oneil
Tel.: (61 02) 61 25 38 00
Mail: Coombs Building, 9
Canberra, ACT 0200 - AUSTRALIA

[2 text/html]

Dr Mathieu O'Neil
Adjunct Research Fellow
Australian Demographic and Social Research Institute
College of Arts and Social Science
The Australian National University

E-mail: mathieu.oneil
Tel.: (61 02) 61 25 38 00
Mail: Coombs Building, 9
Canberra, ACT 0200 - AUSTRALIA

[2 text/html]

Dr Mathieu O'Neil
Adjunct Research Fellow
Australian Demographic and Social Research Institute
College of Arts and Social Science
The Australian National University

E-mail: mathieu.oneil
Tel.: (61 02) 61 25 38 00
Mail: Coombs Building, 9
Canberra, ACT 0200 - AUSTRALIA

[2 text/html]

Thread: joxT00000 Message: 145/176 L23 [In date index] [In thread index]
Message 00116 [Homepage] [Navigation]