Re: [jox] Re: Peer Review
- From: "Mathieu O'Neil" <mathieu.oneil anu.edu.au>
- Date: Sun, 11 Oct 2009 19:06:35 +0200
[Converted from multipart/alternative]
[1 text/plain]
Hi all
Well, here is my response to my message outlining some suggestions and questions to the proposed journal project which was posted a couple of weeks ago. Someone needed to respond, and since no-one did I figured I might as well. There are evidently limits to the long-term validity and viability of such a monologue but for the time being I think the benefits outweigh the disadvantages… In any case StefanMn had already written about some of these issues on August 24 and Sepember 7 so I added his opinion where I thought applicable.
It would be good to start approaching people to act as reviewers. As suggested previously, I intend to post a draft letter to the list for feedback. However I can't do this until some of the website / technical issues have been resolved (in particular: what type of reader feedback do we want? Do we offer to provide stats for authors / reviewers? )
Finally, when do we expect to be able to start putting content up about the project (to start with just a page with project description and participants)?
Cheers
Mathieu
(snip)
First I wanted to make a suggestion to address the concern
raised by StefanMn in relation to new reviewers changing the
course of the project: there could be two distinct groups.
One group, "reviewers" (not yet involved in the project), would
be mostly involved in peer reviewing.
Another group,"editors" (already involved), would be involved in
peer reviewing as well as overall project governance. Over time
if reviewers take part in discussions etc and are accepted they
could become part of the editorial team? Just a thought.
Yes, great idea! ;-)
(snip)
2_Publication
So what we could have is a more transparent process where all
submitted articles are peer reviewed and then rated from +5
(excellent) to –1 (don’t publish).
However in the WF system if authors decide they want to publish
publicly a – 1 article then it is published anyway. This is of
course debatable…
Well, I'm not sure that articles rated -1 should be published... others may disagree.
WF suggest that + 5 articles could be published as print or
otherwise "frozen". Graham did mention print-on-demand which
would be something to think about for the future. Does anyone
know anything about this? I don't..
Note: If people want to keep commenting on articles (for
readers) or responding to comments (for authors) in my view that
is fine but as was discussed previously I don’t think the idea
of endlessly updating the body of an article itself (as happens
on Wikipedia) is appropriate for a journal.
I agree with my opinion
3_Anonymous review choice
Allows authors to improve their paper privately. I think for
research papers they should all be reviewed in this way. Maybe
we could have a choice between "light" (one reviewer) and
"heavy" (two-three reviewers) reviews but this may end up
complicating the process a bit. So it might be better to just
have normal two reviews?
Yes, just one system of expert review would be best, I think.
4_Reader ratings
WF say these could be the done formally by votes or informally
with mouse clicks (number of views or downloads). Thoughts?
Obviously informally needs more technical monitoring.
I would go for votes but I can't really speak about this as this concerns the website maintainers – what do they think?
5_Performance reports
WF propose that like students who can request grades from
universities contributors (such as authors) could request
reports not just of publications but of citations, number of
comments, of downloads and views generated. Reviewers could ask
for records of their contributions to be sent directly to institution.
Possible? Desirable?
I think this could be a good idea but there again the website maintainers need to indicate what they think is desirable / possible?
6_View filters
This refers to only displaying content rated above "x" like on
Slashdot I guess.
Should it be done? Can it be done?
Maybe this is not necessary for now.
7_Same again function
Automatically helps people find more of the same – ie find
content rated highly by those who rate like you.
Should it be done? Can it be done?
Maybe this is not necessary for now.
8_Metrics
WF suggest:
Relevance
Rigour
Writing
Comprehensiveness
Logical flow
Originality
Others? Less?
This seems like a good metric as it could apply to both research papers and position / activist papers.
9_Progression of readers
I think most would agree that we should have reader input. If
there is a rating system generated by readers then they can
offer a counterpoint to “expert” and / or academic reviewers. WF
suggest there could be a natural path to associate reviewer,
reviewer, associate editor, etc
Thoughs...
I think this could be a good idea.
10_Identity
This raises the question of who can be a reader? Do we require
credentials? Even more fundamentally, this raises the questions
of identity and privacy for readers and commenters.
WF say (and I agree) that we need contributors to register to
avoid spam and vandalism. They write that jounals should “check
the credentials of _who_ is submitting not censor _what_ is
submitted”. I think they mainly mean authors though. It is not
entirely clear what they think about readers (a priori "expert"
reviewers are known in-house so not concerned by this issue). My
preference would be to encourage or demand real names for
commenters and raters, on principle. Many of the problems on
Wikipedia (for example) derive from anonymity and the attendant
lack of responsibility which almost generates disruption.
Questions about this:
-- Can / should we have an automatic registration system for
participants or would people need to approach us to b registered and given access?
StefanMn has indicated that his preference would be for authors to contact us rather than just dropping a text in a box. Why not extend this to commenters / readers? That would help to prevent spam in my view.
-- Do we check that people who want to register are who they say
they are? If yes, how?
Maybe we could request a link to a website or weblog or archive that contains a trace of the person's identity?
That’s it!
Cheers
Mathieu
----- Original Message -----
From: Mathieu O'Neil <mathieu.oneil anu.edu.au>
Date: Tuesday, September 15, 2009 8:00 pm
Subject: Re: [jox] Re: Peer Review
To: journal oekonux.org
[Converted from multipart/alternative]
[1 text/plain]
Dear all
Apologies for slackness, I've been travelling and am currently
snowed under. I want to re-read those papers again before
commenting on Stefans poins. But: It does not make sense to be
discussing openly something which only a few have seen so I'm
taking it upon myself to post the references here. I never got
an answer from the authors to my question as to whether it
would
be OK to distribute. In French there is a saying: "Qui ne dit
mot consent". To potential readers: please remember the second
part of the First Monday paper is still a draft so do not
quote.
My biggest question is: they suggest publishing everything
that
is submitted with ratings ie 5 excellent, 4 good, etc even -1
(dont publish) if the author wants. I think this is an
interesting concept but I also think this is potentially going
towards the "process" option we discussed before.
Perhaps
articles rated 4 or 5 could periodically be gathered into a
"stable" release that could be advertised across the weboverse?
Anyway here are the refs:
Hi Mathieu Thanks for your enquiry and interest. Your
journal looks good. The social design of a KES is quite
complex, but you can see our first attempt to lay out
the
details and spirit of the endeavor at
http://brianwhitworth.com/BWRF-FM-Part2.pdf In
particular
note the section on privacy, where we allow reviewers to
reveal themselves after the review is over if they want
to. Also note it is still draft until it comes out in
FM. Some more detail on Socio-technical design
in general is also given in our Handbook of STS Design,
see http://brianwhitworth.com/sts for which mashup is a
first go at making such stuff available. In particular,
check out my chapter 1 which gives an idea of some of the
complexity, see http://brianwhitworth.com/STS/STS-
chapter1.pdf Note that the full design is
not
yet specified - we intend to develop this working with
collaborators in a feedback process, so if you want to
be
part of that let us know. I am working with Rob Friedman and
Michael Browstein on this project, so I copy
this
email to them also. all the best
Brian Whitworth
Cheers
Mathieu
----- Original Message -----
From: Stefan Merten <smerten oekonux.de>
Date: Monday, September 7, 2009 10:50 pm
Subject: Re: [jox] Re: Peer Review
To: journal oekonux.org
Cc: Stefan Merten <smerten oekonux.de>
Dear Mathieu, CSPP Board Members, all!
Last week (9 days ago) Mathieu O'Neil wrote:
OK, I contacted one of the authors of the FM paper and he kindly
sent me links to his second part as well as another relevant
chapter.
Mathieu sent the links privately. Indeed a very useful
paper. I'll
respond to this here because otherwise the communication
becomes to
confusing for me ;-) .
In his mail he also mentioned that they are seeking
feedback from
people on these issues. I responded that would be possible
but that
I was acting as part of a group and queried whether it
would
be OK
to circulate the papers to this group (without really
saying
who was
in it though I had originally sent a link to this list's
archive).
They are welcome. But then we need the right to put the link here
(it's a public link anyway so this should not matter too much).
@Mathieu: I hope you won't mind that I quote the following
questions> > from your private mail here.
Last week (7 days ago) Mathieu O'Neil wrote:
I guess the main questions are: which features would we want?
In general I think those guys are very much on the same
track
as we
:-) . I think the key question to solve I already mentioned
in
my post
from `Mon, 24 Aug 2009 12:25:45 +0200`__. To prevent leaking
too much
information from the paper here I only reply to what is new
or very
different from my suggestions there or comments from others
made here
already. I relate to the chapter "DEMOCRATIC ONLINE KNOWLEDGE
EXCHANGE".
__ http://www.oekonux.org/journal/list/archive/msg00091.html
* Multi-level ratings
Similar to my previous suggestion the
submissions
can and
should be
given a number of levels reflecting how the
reviewers /
editorial board assesses the submission. That solves
the
problem of the binary
accept / reject scheme. On a website it is also
easy to flag
articles appropriately.
Also using several rating dimensions is a good
idea. What
is needed,
however, is to reduce the many dimensions to a
single
number to give
orientation to readers. May be the scientific
vs.
activist category
could be one such dimension.
I think the reviewers should still have the
responsibility to rate
an article. May be the reader rating could be
one
more
dimension -
given that there are enough ratings from the readers.
Readers need to have an option to tune what
dimensions
they consider
most important. Technically this would mean to
have different
reports using a reader's weights. That is
technically simple.
* Submissions
I think every submission should be addressed to
the
editorial board.
IMHO this also adds a level of responsibility on
the part
of the
submitters: It's a difference whether you just
drop
something in an
electronic system or ask real people to consider
your
submission.
This doesn't mean that a submission is withheld
from the
website. It
can be put there immediately if the author
wishes so.
* Anonymous contributions
In general I don't like anonymous contributions
as
well
as I don't
like near-anonymous contributions from obvious
pseudonyms. I see
that creation of an account actually feels like
an
obstacle - at
least to me. But nonetheless I don't like
anonymous
contributions. IMHO this at least reduces the spam
problem
very much.
Which of those
we want can we have without too much hassle?
I think most if not all of their suggestions and our ideas are
technically feasible. IMHO the real work (i.e. human labor)
is
reading> submissions, thinking about them and making
decisions.
This real work
needs to be done anyway. The rest can and should be left to the
machine.
Grüße> >
Stefan> > ______________________________
http://www.oekonux.org/journal
****
Dr Mathieu O'Neil
Adjunct Research Fellow
Australian Demographic and Social Research Institute
College of Arts and Social Science
The Australian National University
E-mail: mathieu.oneil anu.edu.au
Tel.: (61 02) 61 25 38 00
Web: http://adsri.anu.edu.au/people/visitors/mathieu.php
Mail: Coombs Building, 9
Canberra, ACT 0200 - AUSTRALIA
[2 text/html]
______________________________
http://www.oekonux.org/journal
****
Dr Mathieu O'Neil
Adjunct Research Fellow
Australian Demographic and Social Research Institute
College of Arts and Social Science
The Australian National University
E-mail: mathieu.oneil anu.edu.au
Tel.: (61 02) 61 25 38 00
Web: http://adsri.anu.edu.au/people/visitors/mathieu.php
Mail: Coombs Building, 9
Canberra, ACT 0200 - AUSTRALIA
[2 text/html]
______________________________
http://www.oekonux.org/journal
****
----- Original Message -----
From: Mathieu O'Neil <mathieu.oneil anu.edu.au>
Date: Saturday, September 26, 2009 6:58 pm
Subject: Re: [jox] Re: Peer Review
To: journal oekonux.org
[Converted from multipart/alternative]
[1 text/plain]
ps. Apologies to StefanMn, he already addressed some of these
issues a while ago in his comments on the articles (way down at
the bottom of this email)...
----- Original Message -----
From: Mathieu O'Neil <mathieu.oneil anu.edu.au>
Date: Saturday, September 26, 2009 5:48 pm
Subject: Re: [jox] Re: Peer Review
To: journal oekonux.org
[Converted from multipart/alternative]
[1 text/plain]
Hi everyone
Sorry for the long delay in resuming the processs.
First I wanted to make a suggestion to address the concern
raised by StefanMn in relation to new reviewers changing the
course of the project: there could be two distinct groups.
One group, "reviewers" (not yet involved in the project),
would
be mostly involved in peer reviewing.
Another group,"editors" (already involved), would be involved
in
peer reviewing as well as overall project governance. Over
time
if reviewers take part in discussions etc and are accepted
they
could become part of the editorial team? Just a thought.
OK, in terms of the peer review process and the articles by
Whitworth and Friedman (WF) which I referenced a while ago -
message below if you want to have a look.
I think these articles offer some very useful ideas. But not
all
ideas need to be put into practice. In any case they all
require
some technological expertise…
Following are some thoughts and questions.
1_General principles
First here is a summary of what they propose in their new
system, as opposed to the traditional system
a) higher rating discrimination (a many-point scale, not just
accept-reject)
b) more submissions to be rated (rate all)
c) more people to rate (community involvement)
d) different ways of rating (formal review vs. informal use ratings)
2_Publication
So what we could have is a more transparent process where all
submitted articles are peer reviewed and then rated from +5
(excellent) to –1 (don’t publish).
However in the WF system if authors decide they want to
publish
publicly a – 1 article then it is published anyway. This is of
course debatable…
WF suggest that + 5 articles could be published as print or
otherwise "frozen". Graham did mention print-on-demand which
would be something to think about for the future. Does anyone
know anything about this? I don't..
Note: If people want to keep commenting on articles (for
readers) or responding to comments (for authors) in my view
that
is fine but as was discussed previously I don’t think the idea
of endlessly updating the body of an article itself (as
happens
on Wikipedia) is appropriate for a journal.
3_Anonymous review choice
Allows authors to improve their paper privately. I think for
research papers they should all be reviewed in this way. Maybe
we could have a choice between "light" (one reviewer) and
"heavy" (two-three reviewers) reviews but this may end
up
complicating the process a bit. So it might be better to just
have normal two reviews?
4_Reader ratings
WF say these could be the done formally by votes or informally
with mouse clicks (number of views or downloads). Thoughts?
Obviously informally needs more technical monitoring.
5_Performance reports
WF propose that like students who can request grades from
universities contributors (such as authors) could request
reports not just of publications but of citations, number of
comments, of downloads and views generated. Reviewers could
ask
for records of their contributions to be sent directly to
institution.> Possible? Desirable?
6_View filters
This refers to only displaying content rated above "x" like on
Slashdot I guess.
Should it be done? Can it be done?
7_Same again function
Automatically helps people find more of the same – ie find
content rated highly by those who rate like you.
Should it be done? Can it be done?
8_Metrics
WF suggest:
Relevance
Rigour
Writing
Comprehensiveness
Logical flow
Originality
Others? Less?
9_Progression of readers
I think most would agree that we should have reader input. If
there is a rating system generated by readers then they can
offer a counterpoint to “expert” and / or academic reviewers.
WF
suggest there could be a natural path to associate reviewer,
reviewer, associate editor, etc
Thoughs...
10_Identity
This raises the question of who can be a reader? Do we require
credentials? Even more fundamentally, this raises the
questions
of identity and privacy for readers and commenters.
WF say (and I agree) that we need contributors to register to
avoid spam and vandalism. They write that jounals should
“check
the credentials of _who_ is submitting not censor _what_ is
submitted”. I think they mainly mean authors though. It is not
entirely clear what they think about readers (a priori
"expert"
reviewers are known in-house so not concerned by this issue).
My
preference would be to encourage or demand real names for
commenters and raters, on principle. Many of the problems on
Wikipedia (for example) derive from anonymity and the
attendant
lack of responsibility which almost generates disruption.
Questions about this:
-- Can / should we have an automatic registration system for
participants or would people need to approach us to be
registered and given access?
-- Do we check that people who want to register are who they
say
they are? If yes, how?
There are other ideas such as having an automated system
for logging and tracking submissions but this could be set up
later as there is already a system for doing this - me :-) -
whereas some of the points above have no solution yet.
In short:
To everyone: what do we want?
To StefanMn and StefanMz: what can we have?
cheers
mathieu
----- Original Message -----
From: Mathieu O'Neil <mathieu.oneil anu.edu.au>
Date: Tuesday, September 15, 2009 8:00 pm
Subject: Re: [jox] Re: Peer Review
To: journal oekonux.org
[Converted from multipart/alternative]
[1 text/plain]
Dear all
Apologies for slackness, I've been travelling and am
currently
snowed under. I want to re-read those papers again before
commenting on Stefans poins. But: It does not make sense to
be
discussing openly something which only a few have seen so
I'm
taking it upon myself to post the references here. I never
got
an answer from the authors to my question as to whether it
would
be OK to distribute. In French there is a saying: "Qui ne
dit
mot consent". To potential readers: please remember the
second
part of the First Monday paper is still a draft so do not
quote.
My biggest question is: they suggest publishing everything
that
is submitted with ratings ie 5 excellent, 4 good, etc even -
1
(dont publish) if the author wants. I think this is an
interesting concept but I also think this is potentially
going
towards the "process" option we discussed before.
Perhaps
articles rated 4 or 5 could periodically be gathered into a
"stable" release that could be advertised across the weboverse?
Anyway here are the refs:
Hi Mathieu Thanks for your enquiry and interest. Your
journal looks good. The social design of a KES is
quite
complex, but you can see our first attempt to lay out
the
details and spirit of the endeavor at
http://brianwhitworth.com/BWRF-FM-Part2.pdf In
particular
note the section on privacy, where we allow reviewers to
reveal themselves after the review is over if they
want
to. Also note it is still draft until it comes out in
FM. Some more detail on Socio-technical design
in general is also given in our Handbook of STS
Design,
see http://brianwhitworth.com/sts for which mashup is
a
first go at making such stuff available. In
particular,
check out my chapter 1 which gives an idea of some of the
complexity, see http://brianwhitworth.com/STS/STS-
chapter1.pdf Note that the full design is
not
yet specified - we intend to develop this working with
collaborators in a feedback process, so if you want to
be
part of that let us know. I am working with Rob Friedman and
Michael Browstein on this project, so I
copy
this
email to them also. all the
best
Brian Whitworth
Cheers
Mathieu
----- Original Message -----
From: Stefan Merten <smerten oekonux.de>
Date: Monday, September 7, 2009 10:50 pm
Subject: Re: [jox] Re: Peer Review
To: journal oekonux.org
Cc: Stefan Merten <smerten oekonux.de>
Dear Mathieu, CSPP Board Members, all!
Last week (9 days ago) Mathieu O'Neil wrote:
OK, I contacted one of the authors of the FM paper and
he kindly
sent me links to his second part as well as another relevant
chapter.
Mathieu sent the links privately. Indeed a very useful
paper. I'll
respond to this here because otherwise the communication
becomes to
confusing for me ;-) .
In his mail he also mentioned that they are seeking
feedback from
people on these issues. I responded that would be
possible
but that
I was acting as part of a group and queried whether it
would
be OK
to circulate the papers to this group (without really
saying
who was
in it though I had originally sent a link to this list's
archive).
They are welcome. But then we need the right to put the
link here
(it's a public link anyway so this should not matter too much).
@Mathieu: I hope you won't mind that I quote the following
questions> > from your private mail here.
Last week (7 days ago) Mathieu O'Neil wrote:
I guess the main questions are: which features would we want?
In general I think those guys are very much on the same
track
as we
:-) . I think the key question to solve I already
mentioned
in
my post
from `Mon, 24 Aug 2009 12:25:45 +0200`__. To prevent
leaking
too much
information from the paper here I only reply to what is
new
or very
different from my suggestions there or comments from
others
made here
already. I relate to the chapter "DEMOCRATIC ONLINE KNOWLEDGE
EXCHANGE".
__ http://www.oekonux.org/journal/list/archive/msg00091.html
* Multi-level ratings
Similar to my previous suggestion the
submissions
can and
should be
given a number of levels reflecting how the
reviewers /
editorial board assesses the submission. That solves
the
problem of the binary
accept / reject scheme. On a website it is
also
easy to flag
articles appropriately.
Also using several rating dimensions is a good
idea. What
is needed,
however, is to reduce the many dimensions to a
single
number to give
orientation to readers. May be the scientific
vs.
activist category
could be one such dimension.
I think the reviewers should still have the
responsibility to rate
an article. May be the reader rating could be
one
more
dimension -
given that there are enough ratings from the
readers.> > >
Readers need to have an option to tune what
dimensions
they consider
most important. Technically this would mean to
have different
reports using a reader's weights. That is
technically simple.
* Submissions
I think every submission should be addressed
to
the
editorial board.
IMHO this also adds a level of responsibility
on
the part
of the
submitters: It's a difference whether you just
drop
something in an
electronic system or ask real people to
consider
your
submission.
This doesn't mean that a submission is
withheld
from the
website. It
can be put there immediately if the author
wishes so.
* Anonymous contributions
In general I don't like anonymous
contributions
as
well
as I don't
like near-anonymous contributions from obvious
pseudonyms. I see
that creation of an account actually feels
like
an
obstacle - at
least to me. But nonetheless I don't like
anonymous
contributions. IMHO this at least reduces the spam
problem
very much.
Which of those
we want can we have without too much hassle?
I think most if not all of their suggestions and our ideas are
technically feasible. IMHO the real work (i.e. human
labor)
is
reading> submissions, thinking about them and making
decisions.
This real work
needs to be done anyway. The rest can and should be left
to the
machine.
Grüße> >
Stefan> > ______________________________
http://www.oekonux.org/journal
[2 text/html]
______________________________
http://www.oekonux.org/journal