Message 00245 [Homepage] [Navigation]
Thread: joxT00189 Message: 17/77 L4 [In date index] [In thread index]
[First in Thread] [Last in Thread] [Date Next] [Date Prev]
[Next in Thread] [Prev in Thread] [Next Thread] [Prev Thread]

Re: [jox] Cutting the Knot

Hi all,

given the length, here's the summary:

i support StefaMn proposal for authors choosing the qualifying model (binary/ratings)

i extend it with the choices of _early screening_ and peer reviewing models (both open/closed choices).

However, i'm not convinced _rating_ is the most suitable name for signaling the attributes of an article.

i also support use of Plone over email list.

Journal Commons is a new project working on providing advanced tools and organizational techniques for cooperation in knowledge production - we're using Plone, working closely with two journals (in private for now), i will inform the list of our milestones in coming months.



my firm belief is that academic journals should facilitate, assist and improve the production and spread/distribution of new knowledge. Journals do not produce new knowledge. Authors do. This, in my experience, is lost on almost all journals i learned about so far.

Instead, there's rigidity on the side of journals, dictating to authors a set of rules which, given our level of technological development, make little sense today. At the same time, i believe to a large extent because of the opaqueness of peer reviewing system, authors are not being helpful to journal editors either: they regularly send articles with vast deficiencies in quality of argument, novelty, or simply required formatting, which rightly drives editorial boards mad and adds to their workload by what it is often seen as lack of respect on the side of authors.

I think that two moves could help to reduce tensions on both sides and improve the production: journals should be more accommodating/open to the needs of authors, simultaneously with being more strict in social pressures on authors to observe these new, more flexible, and more appropriate to our times, set of rules.

StefanMn then proposed a choice where authors submitting a proposal could
indicate whether they want a binary model (publish or reject) or a
multi-dimensional rating system:

I like this proposal. I see it as a desirable increases in flexibility on the side of journal, which authors, i believe, will appreciate. As to readers, i don't see the problem with some articles being rated, and others not. There could be an icon always displayed next to the article, indicating it is a rated article. I expect that as the time goes, more authors will choose ratings, since they will figure out that it gives them more chance of being published (with problems that article might have noted by the ratings), rather then rely on the binary publish/reject decision.

My view is that binary model is terrible, unscientific, and antagonistic for entirely wrong reasons (i'm a big supporter of antagonism as method, when suitable) and should be gradually replaced. However, new models should not be imposed, but rather offered as alternatives that are monitored, evaluated and improved. We have had too many centuries of binary model to displace it over night, and authors would rightly be skeptical of such move without seeing the benefit of it first in practice.

Hence, i think that StefanMn proposal is an excellent way to win over authors in favor of new systems, in a way which gives them the chance to both choose and observe how the new models work. I much prefer this approach, than having only new systems (ratings, open-process, or otherwise), because i see how it would enable us to win authors over and to demonstrate that we are not claiming that we definitely know what is right and what is wrong. This, position of 'we know best for sure' is the attitude of almost all the journals i interacted with (perhaps i was unlucky, but it seems a pattern), and no wonder authors send all kind of junk - they see journals as arrogant, uncooperative and self-serving gatekeepers to their career advance (i've sent my first ever journal article submission, one on the open process in academic publishing recently to a journal i.e. this is not entirely my own experience, but what i observed from years of being surrounded by academic friends and colleagues + the interactions i had with several journals in the past year). Pushed by the publish-or-perish evaluating model, by the academic rewards model in general, and by the arrogance and uncooperativeness of journals, authors do often behave badly. But it comes partially, perhaps largely, out of desperation.

We can improve this relation a lot by these new models.The result, i hope (convinced by the arguments and analysis), will be a far better mutual respect and relationship authors-journal-readers and it's fitting for a peer production journal to be self-reflective and hence the innovator in own field of production. The general spirit of the two papers referenced (Reinventing academic publishing online + Open Process) is for me the spirit of peer production, of less centralized, yet more structured and more beneficially (for all sides) organized systems - in the context and sphere of knowledge production.

Second, if we want to open up the journal selection process and provide rewards to those who do
normally invisible work (i.e. reviewers), in line with Toni Prug's
proposal [2] for a community peer review system (through a list where proposals are
vetted and reviews are released), then by definition we are rejecting
the publish / don't publish model: vetting and orientation occur
upstream, even before an actual full submission.

Indeed. The open process model relies on discovering, and fixing when possible, problems in the early stage, when the cost of doing so is low on both sides (authors, editors/reviewers). However, in humanities and social sciences, it is often the case that the quality and novelty of the contribution cannot be seen until the whole argument is developed into a longer, more fined grained piece i.e. the details are sometimes all that matters in a piece, and we cannot see it in an early stage. Situation seems even worse for natural sciences, where months, or years, of lab experimental work might not results into a single publishable paper nor new findings (i was told). Yet, it is funders, or heads of research centres, who still make a judgment on the plausibility of the project before it starts, hence acting as a form of early peer review. We can try to act in a similar way, judging the plausibility of a short proposal being developed in something we consider worthy of being publishable. Also, many ideas are visibly worthy of developing in their early stage, within the first thousand words, as a rough proposal.

In short, there are issues with the open process as well, but i also do not see it as a binary either/or. I would like to see us offering to authors the options to choose between open/closed twice:

a) EARLY SCREENING: authors choose whether to submit proposal (let's say up to 1000 words) for a paper; they can do so either in an open OR closed way i.e.their proposals, our comments, their comments back, and our decision (either Yes, please develop and submit; OR No, we don't think it's for this journal) are either visible, or not, publicly, depending on what authors choose.

b) REVIEWING: use open OR closed peer reviewing for full length articles i.e. authors do not have to go through a) at all, they simply follow the traditional model and submit full paper, choosing for peer reviewing to be open or closed.

(c) special cases: There are several possible complications in which the journal will have to make decision. One example is that a series of peer reviewers approached can refuse to peer review the article if their name is used, since they might be intimidated by the importance of the author, or for some other reason. In that case, the journal might approach the author and insist that peer reviews must be done without peer reviewers names used. There are other possible twists and resolutions, we'll learn as we do it.

Later, we could move to a more refined set of early screening and peer reviewing workflows (where some aspect might be open to some groups at some stage in the process, and not in other stages) as we all learn by practices what works well, what not, and how to develop it further.

In order to protect the
reputation of the journal, we need to alert readers that we are aware
of flaws, but that _we decided to publish anyway_. Hence the need to
“qualify” or “signal” (rather than “rate”) published submissions.

Yes, i agree, very well put. I'm not convinced _rating_ is the best term for what we're intending to do either, although i like the procedure and support it. Qualify or signal seem clumsy, though not entirely unusable substitutes. When we say that a paper has been rated, it is intuitive. Not so when we say it has been signaled, or qualified. I'm undecided. Any other ideas how to name this?

So, this is the first point to decide: what categories do we have?

i have to think about this separately.

4-decision: review discussion system

+ email OR web platform for reviewing - the shades of openness

I love email lists, and i find it hard to accept more complex web tools for collaboration, since, as a rule with rare exceptions, i find them less, not more helpful as tools/environments to assist and change work positively. However, i was recently given an introduction to Plone, and i was won over by it. Excuse me for the technical language for a moment: it seems to me that worklows, transitions and fine grained access for groups, including the acquisition mechanism [1], will lend themselves well to variety of degrees of openness and structures of workflows for peer reviewing. Even more so, if each Folder, or Collection object (which contain other objects i.e submitted documents) can have RSS feeds to which we can subscribe too, including a unified feed - this is likely to be not so diffifult to add, even if it currently does not exist in the Plone and its plugins (call products in Plone). For example, see this product, which enables any object within Plone (documents, folder, etc - if i'm not mistaken) to be watched, so that watchers automatically receive emails on any change in the observed object.

For the two of the projects i'm working on with two other journals, we decided to use Plone. We're now working on building worklfows to present it to academics involved in those projects.

I would therefore recommend using Plone (and not just any other web tool), and not an email list, with one important note: to increases the chances of Plone academic users being satisfied with it as a helpful platform, we must keep things simple to start with i.e we must not alienate early users with complex workflows and procedures.

Complexity can be added once the initial usage picks up and once the usual resistance (i have plenty of it too) to new systems is overcome through the satisfaction with the benefits that the platform brings to its users.

When we're done with the projects i'm currently working on, we will share workflows and all the information on our Plone setup, use, and comments we got from academic communities we're working within. So far, journals and communities expressed the wish to do this development process in private, until they reach the decision on whether to use the new models we are presenting - this should be within the next month or two.

Finally, Juan Grigera (who is working with me on this) and i decided the create a project out of this, we named it Journal Commons. We will launch it once the proposals we're working on are decided on. We aim to support journals to implement new processes of cooperation, using advanced web tools and organizational techniques. Given the number of technically skilled people here, and the use of Plone, we have a basis for a potential close cooperation on the Journal Commons as a separate project. We'll keep the list informed with major project milestones in the coming months.


[1] 'Acquisition allows behavior to be distributed hierarchically throughout the system [...] you can change an object’s behavior by changing where it is located in the object hierarchy.'


Thread: joxT00189 Message: 17/77 L4 [In date index] [In thread index]
Message 00245 [Homepage] [Navigation]