Message 00151 [Homepage] [Navigation]
Thread: joxT00000 Message: 108/176 L35 [In date index] [In thread index]
[First in Thread] [Last in Thread] [Date Next] [Date Prev]
[Next in Thread] [Prev in Thread] [Next Thread] [Prev Thread]

Multi-rating mode of evaluation / Updating papers (was: Re: [jox] Pot-pourri)

[Converted from multipart/alternative]

[1 text/plain]
Hi all

Thanks to everyone who has contributed to the process this week. I just wanted to alert everyone that there is an issue that needs to be resolved: Felix and Athina (see below) have objected to our stated policy of "rating" submissions according to a scale of -1 to [PHONE NUMBER REMOVED] (inspired by Brian Whitworth and Rob Friedman's papers on academic publishing in First Monday). As you know, this would in theory facilitate the publication of a greater number of submissions than is the case in other journals (though if you consider the fact that some academic journals publish 5 or 6 issues each year, that represents a fair number of papers). 

We could perhaps simplify the rating system but this would not change the fact that there would be a situation where some published submissions are given a higher "expert rating" than others before publication (as well as a "reader rating" given by registered users after publication). The question of how people who have been given an inferior rating by reviewers would feel about this situation has not been addressed so far and I think it needs to be. If we do not rate then do we go to (in theory) a lower publication rate where only excellent contributions are published?

I urge everyone to express their opionion about this core mechanism in our peer review process...


ps. Another important issue was raised by Brian (see below) which we had discussed previously on this list: whether authors can update their papers after receiving reader feedback. As I thought more about this I realised that having version numbers (ie v.4.8) might correct the problem of not knowing which version to refer to. So I'm starting to reexamine my position about this. I can think of several online texts which had several versions, Wark's The Hacker Manifesto, Raymond's Cathedral and the Bazaar, and I know that when I wrote an article on Wikipedia for a magazine this year, the English version is more accurate than the original French because I got some feedback from the French - and then I got more feedback on the English version...  I think it might be a good thing as long as its not constant (say you could only create a revised version a limited number of times a year?) Thoughts?...

(Felix Stalder: )
As for the review process, I think having people to be able to
comment is
great. I'm not so sure, if we should adopt a rating of published
from +2 to +5. This would mean that the reviewers tell the
audience which
papers they think are every good, and which are merely good
enough. I think
this is a bit unfair to the published authors and creates biases
in the
audience. After all, wouldn't you start reading papers that are
high, rather than those ranked poorly?

(Athina Karatzogianni:)
in global politics. I agree with Felix, I never thought that the 
rating was
a good idea myself.


On Tue, Nov 17, 2009 at 10:33 PM, Mathieu O'Neil

[Converted from multipart/alternative]

[1 text/plain]
Hi all

=Brian Whitworth=

This is good news, and I am happy to assist in any way I can. 
Our main idea
is to go from a static KES to a dynamic one, e.g. authors can update
versions of what they write based on reader feedback, so 
things evolve
rather than being fixed in stone, plus with attribution. When 
the changes
stabilize, one can still do fixed print. So most current 
publishing manager
software design's wont cut it, like manuscript central, as a 
secret reviewer
feedback system is behind the publication "shop window". You 
can still have
that right at the beginning, by author choice, but the main 
idea is a system
tht is transparent and open. If you can make a move in that 
direction that
works, or implement the design the second part of our paper 
outlined in any
way, I encourage you.


Thanks for your message, and for your offer to help. I will 
put you down as
part of our scientific committee. Regarding the updating of 
content, we
discussed this, and came to the conclusion that while we are 
all for readers
leaving comments to papers, and authors responding to readers 
in comments,
the notion of a permanently mutable text (as is the case for 
example on WP)
did not really suit us in the context of a journal. I can see 
the attraction
in what you suggest below but it does raise a number of issues:
- at what point is an article "stable", if the possibility 
always exists
that some new argument, for or against, comes along and must 
be dealt
- another possible problem with continuously changing the main 
part of a
paper is that it would create uncertainty as to which version 
was being
referred to in subsequent publications.
So we decided that we would rather have people write a new 
paper based on
all the comments and responses, if necessary. At the same time 
this is
something that we can talk about more. I will subscribe you to 
our list and
raise this alongside other issues that come up.

[2 text/html]

Thread: joxT00000 Message: 108/176 L35 [In date index] [In thread index]
Message 00151 [Homepage] [Navigation]