Re: [jox] Re: Peer Review
- From: Stefan Merten <smerten oekonux.de>
- Date: Tue, 20 Oct 2009 08:05:27 +0200
Hi Mathieu and all!
In general I think we have reached a stage of rough consensus. The
recent silence on the list to me is a clear indicator for this. I'd
find it useful if the results of our discussion could be summarized.
Mathieu?
My next most important task is clarifying a final but important detail
in the website setup - see [pox] - and then I think we can start a
"go-live".
I'm giving my opinion on the questions raised by Mathieu. Sorry for
any duplication.
3 weeks (24 days) ago Mathieu O'Neil wrote:
First I wanted to make a suggestion to address the concern raised by StefanMn in relation to new reviewers changing the course of the project: there could be two distinct groups.
One group, "reviewers" (not yet involved in the project), would be mostly involved in peer reviewing.
Another group,"editors" (already involved), would be involved in peer reviewing as well as overall project governance. Over time if reviewers take part in discussions etc and are accepted they could become part of the editorial team? Just a thought.
This matches my idea quite closely. My experience in project setups is
exactly this: Check how people perform in the "normal" community and
if you like it ask them whether they would like to take more
responsibility.
OK, in terms of the peer review process and the articles by Whitworth and Friedman (WF) which I referenced a while ago - message below if you want to have a look.
I think these articles offer some very useful ideas. But not all ideas need to be put into practice. In any case they all require some technological expertise...
I just started to get acquainted with the non-technical details of
Plone_ - which will be the technical basis of the website. So I'm not
yet an expert for using every feature of Plone but I know Plone is a
*very* rich "eco-system" with lots and lots of features. From this I
can draw some conclusions on what is possible probably.
.. _Plone: http://plone.org/
Following are some thoughts and questions.
1_General principles
First here is a summary of what they propose in their new system, as opposed to the traditional system
a) higher rating discrimination (a many-point scale, not just accept-reject)
One key advantage of the web: You have no space limitation and thus
can be more voluminous easily.
b) more submissions to be rated (rate all)
A question of available labor force basically.
c) more people to rate (community involvement)
Another key advantage of the web: Feedback channels.
d) different ways of rating (formal review vs. informal use ratings)
Another advantage of the web where it is quite easy to measure use
ratings.
2_Publication
So what we could have is a more transparent process where all submitted articles are peer reviewed and then rated from +5 (excellent) to -1 (don't publish).
However in the WF system if authors decide they want to publish publicly a - 1 article then it is published anyway. This is of course debatable...
I think the editorial board should be the final decision maker here.
If an article is just rubbish it doesn't need to be published in our
journal - regardless on how much the author wants it.
In the web people are flooded by shit anyway and I think the people we
want to address with the journal are grateful if someone acts as a
filter for them.
WF suggest that + 5 articles could be published as print or otherwise "frozen". Graham did mention print-on-demand which would be something to think about for the future. Does anyone know anything about this? I don't..
Print-on-demand is a model for books. I don't know whether this works
for articles. ChristianS offers his book by a print-on-demand model so
he is an expert.
Note: If people want to keep commenting on articles (for readers) or responding to comments (for authors) in my view that is fine but as was discussed previously I don't think the idea of endlessly updating the body of an article itself (as happens on Wikipedia) is appropriate for a journal.
Yes. May be we can take it this way: Improving an article (beyond
grammar and the like) is the sole responsibility of the original
author.
3_Anonymous review choice
Allows authors to improve their paper privately. I think for research papers they should all be reviewed in this way. Maybe we could have a choice between "light" (one reviewer) and "heavy" (two-three reviewers) reviews but this may end up complicating the process a bit. So it might be better to just have normal two reviews?
I think we should prevent making things too complicated. In practice
we will find some practice which works under the given constraints and
we can formalize this later.
4_Reader ratings
WF say these could be the done formally by votes or informally with mouse clicks (number of views or downloads). Thoughts? Obviously informally needs more technical monitoring.
So far I have no idea on what is possible in monitoring page views
with Plone. Sure we have an Apache in front of Plone which logs hits
but a useful monitoring needs some work - especially because the
caches in front of Plone may hide accesses from Plone. Since this is a
standard problem there might be already a solution to this problem out
there.
5_Performance reports
WF propose that like students who can request grades from universities contributors (such as authors) could request reports not just of publications but of citations, number of comments, of downloads and views generated. Reviewers could ask for records of their contributions to be sent directly to institution.
Possible? Desirable?
For the possibility of numbers technically generated see above.
For the question of availability: If the numbers are available they
could be published right away. In any case I have no general problems
with such use since it creates / indicates more attention for our
journal and that is in the best interest of the journal.
6_View filters
This refers to only displaying content rated above "x" like on Slashdot I guess.
Should it be done? Can it be done?
As I explained I think it could be multi-dimensional. It is certainly
doable but it evenly certainly needs some technical work (which I'm in
general ready to do). If we are lucky there is already a solution to
this problem out there.
7_Same again function
Automatically helps people find more of the same - ie find content rated highly by those who rate like you.
Should it be done? Can it be done?
I'm not sure whether I understand this fully but I guess it is
technically doable.
8_Metrics
WF suggest:
Relevance
Rigour
Writing
Comprehensiveness
Logical flow
Originality
Others? Less?
See my suggestions in another post.
9_Progression of readers
I think most would agree that we should have reader input. If there is a rating system generated by readers then they can offer a counterpoint to 'expert' and / or academic reviewers. WF suggest there could be a natural path to associate reviewer, reviewer, associate editor, etc
Thoughs...
Reader input / rating is fine.
10_Identity
This raises the question of who can be a reader? Do we require credentials? Even more fundamentally, this raises the questions of identity and privacy for readers and commenters.
The general reader should need no credentials in any case. Why limit
read access artificially?
For commenters a registration seems to be a useful thing.
BTW: In general it is easier to create a limitation right from the
start than introducing it later on.
WF say (and I agree) that we need contributors to register to avoid spam and vandalism. They write that jounals should 'check the credentials of _who_ is submitting not censor _what_ is submitted'. I think they mainly mean authors though.
I also think they are talking about authors.
It is not entirely clear what they think about readers (a priori "expert" reviewers are known in-house so not concerned by this issue). My preference would be to encourage or demand real names for commenters and raters, on principle. Many of the problems on Wikipedia (for example) derive from anonymity and the attendant lack of responsibility which almost generates disruption.
Wikipedia is a nice example of how difficult it is to introduce
limitations later - though Wikipedia is not really comparable to a
journal.
Questions about this:
-- Can / should we have an automatic registration system for participants or would people need to approach us to be registered and given access?
Plone has a sophisticated user and rights administration. As a
technical basis this certainly suffices.
For commenters there should be a registration which can be operated by
everyone. For anything else the editorial board needs to make a
decision.
-- Do we check that people who want to register are who they say they are? If yes, how?
This check is practically impossible. If people want to hide their
identity they can. And that's fine with me. It would be good, however,
if they use a consistent alias in this case and don't act under more
than one alias.
For anything more than commenting we need to require people to give
names which are not obvious aliases / nick names - may be with
exceptions for artists. We are doing something serious here and if
people think they need to hide their identity they are in the wrong
place anyway.
There are other ideas such as having an automated system for logging and tracking submissions but this could be set up later as there is already a system for doing this - me :-) - whereas some of the points above have no solution yet.
Plone has also a sophisticated system for supporting a publication
process. AFAICS this can be configured and I think it will be easy.
In short:
To everyone: what do we want?
To StefanMn and StefanMz: what can we have?
My cent.
Grüße
Stefan
______________________________
http://www.oekonux.org/journal