Concerns and suggestions (was: Re: [jox] Multi-rating mode of evaluation)
- From: Stefan Merten <smerten oekonux.de>
- Date: Mon, 21 Dec 2009 13:20:17 +0100
Last week (8 days ago) Mathieu O'Neil wrote:
HI Athina, Stephan, all
Hi Matheiu and all!
I dont think just because one person has their mind set on something
constitutes a sufficient reason to do it, whoever that person may be.
Well, there are more people but in any case I think that we should
decide on the basis of arguments. From my experience consensus comes
when the result is mature. Before that point people feel uncomfortable
and although you have a quick decision you have a long period where
some people are against the decision.
I think its important to keep in mind what people want. A number of
people have expressed serious reservations about rating
indiscriminately.
But are these reservations true to our mission?
In addition: I saw not one point addressing the problems Brian
discussed in his paper. All reservations I saw were very abstract and
most of them didn't tell which real persons really suffer. That also
makes it hard to deal with them - if nobody suffers then there is
nothing you can improve.
At the same time StefanMn has argued for ratings:
but why? If I remember correctly it was mainly to allow papers that
were either a) "controversial" (ie political, radical) or b) written
by non-English speakers, to be published.
My main points are:
* Allow to publish less-than-optimal papers
That was also your point taken from Brian's points BTW.
* Provide help to readers
* Be true to our mission
My suggestion is to compromise: normally published papers are not
rated. They are produced along the lines I suggested in my post called
"Alternative peer review system", inspired by Toni Prug's piece. But
when authors or reviewers feel that the paper are concerned by either
of the categories mentioned above, then authors can ask for and
reviewers can suggest that they are rathed in these categories. That
way papers which would either not be published or take endless
negotiation or fixing up to be published can be.
That is funny because most concerns Athina IMHO would even grow bigger
with such a system...
In any case one of my suggestions also is an option model. However,
I'd suggest to have two modes the submitter can choose from:
* Binary rating
A paper is submitted and it is accepted by the journal responsibles
or not. Papers which are less-than-optimal are not accepted in this
mode.
* Differentiated rating
A submitter can ask for a differentiated rating from the journal
responsibles. Papers which are less-than-optimal can be accepted in
this mode. There is a perspective where readers can see and may be
even use the rating for their own perspectives.
In case there are journal responsibles who for whatever reason
refuse to turn their expert knowledge into a differentiated rating -
that seems to be one point of Athina's concerns - they probably can
not help with such submissions.
If we leave this choice to the submitters then we can also check what
they prefer and thus make an experiment. I'd also think the concerns
of Athina are void then because the submitter chooses herself on how
to be treated and I can see nobody who suffers in this case.
The second suggestion is to create several rubrics where all papers
are sorted into. For instance there could be rubrics for
* Controversial
* Theoretical
* Empirical
* Critical
* Affirmative
* Activist report
* Bad English
This would be similar to a differentiated rating but it reduces the
rating to a single dimension which makes it difficult for a critical
paper based on empirical studies...
Also I'd think it would be useful to have rubrics for the topics we
agreed on:
Topics of interest include, but are not limited to: the political
economy of peer production; peer production and expertise; critical
theory and peer production; peer production and exchange; peer
production and social movements; peer production as an alternative
to capitalism; peer production and capitalist cooptation; governance
in peer projects; peer production and ethics; the peer production of
hardware; peer production and feminism; peer production, industry
and ecology.
-- http://cspp.oekonux.org/call-for-submissions
When I think longer about it all this really calls for tags which are
attached to a submission...
The problem with this approach is that the responsibles need to agree
on a set of tags.
Well, the second suggestion is probably a bad compromise. The first
suggestion is very close to what you, Mathieu, suggested already and
from my POV even addresses some of Athina's concerns better.
What we still need to think about is creating rubrics and/or tags to
sort submissions according to topics.
Grüße
Stefan
______________________________
http://www.oekonux.org/journal