At NEJLT, the review process has the following top-level steps.
The process is explained in detail below, as well as how NEJLT manages anonymity.
While during reviewing authors and other reviewers will have their identities concealed from each other, NEJLT adopts an open policy later in the decision process. After reviewing is done, reviewer names may be revealed both to other reviewers and also to authors. This follows indications that malicious reviews are less prevalent when reviewer identity may be revealed.
At NEJLT, we hope to foster constructive dialogue between authors and experts in the field, and remove the shields of anonymity is a step towards improving that dialogue, especially important in a field such as computer science where reviewing can be hypercritical. Revealing reviewer identity serves to improve review tone, while also removing biases during reviewer discussions.
Below are the review forms for the different article types at NEJLT. All reviews also include general comments. These forms can help authors align their work with reviewer expectations, and help inform which article type a manuscript could be submitted as.
This kind of article focuses on study of linguistic aspects of its subjects.
Relevance: Is the subject matter of this paper within the scope of NEJLT?
Readability/clarity: From the way the paper is written, how well can you tell what research question was addressed, what was done and why, and how the results relate to the research question?
Originality: How original and innovative is the research described? Originality could be in the linguistic question being addressed, in the methodology applied to the linguistic question, or in the combination of the two.
Technical correctness/soundness: Is the research described in the paper technically sound and correct? Can one trust the claims of the paper—are they supported by the analysis or experiments and are the results correctly interpreted?
Reproducibility: Is there sufficient detail for someone in the same field to reproduce/replicate the results? [n/a for certain types of theoretical results]
Data/code availability: To what extent is the data/code (as appropriate) used in this submission available to the research community, or is there a compelling reason given why this is not possible?
Generalizability: Does the paper show how the results generalize, either by deepening our understanding of some linguistic system in general or by demonstrating methodology that can be applied to other problems as well? [n/a for certain types of theoretical results]
Meaningful comparison: Does the paper clearly place the described work with respect to existing literature? Is it clear both what is novel in the research presented and how it builds on earlier work?
Substance: How much substance does this paper have? Is it enough for a full-length journal article, or would it benefit from further development?
Relevance: Is the subject matter of this paper within the scope of NEJLT?
Readability/clarity: From the way the paper is written, can you tell what research question was addressed, what was done and why, and how the results relate to the research question?
Clarity (a) - Hypothesis stated: Is it clear what the authors’ hypothesis is?
Clarity (b) - Hypothesis tested: Is it clear how the authors have tested their hypothesis?
Originality: How original and innovative is the research described? Note that originality could involve a new technique or a new task, or it could lie in the careful analysis of what happens when a known technique is applied to a known task (where the pairing is novel) or in the careful analysis of what happens when a known technique is applied to a known task in a new language.
Technical correctness/soundness: Is the research described in the paper technically sound and correct? Can one trust the claims of the paper—are they supported by the analysis or experiments and are the results correctly interpreted?
Soundness (a): Is it clear how the results confirm/refute the hypothesis, or are the results inconclusive?
Soundness (b): Do the authors explain how the results follow from their hypothesis (as opposed to say, other possible confounding factor)?
Soundness (c): Are the datasets used clearly described and are they appropriate for testing the hypothesis as stated?
Reproducibility: To what degree is there sufficient detail for someone in the same field to reproduce/replicate the results?
Data/code availability: To what extent is the data/code (as appropriate) used in this submission available to the research community, or is there a compelling reason given why this is not possible?
Error analysis: How well does the paper provide a thoughtful error analysis, which looks for linguistic patterns in the types of errors made by the system(s) evaluated and sheds light on either avenues for future work or the source of the strengths/weaknesses of the systems?
Meaningful comparison: Does the paper clearly place the described work with respect to existing literature? Is it clear both what is novel in the research presented and how it builds on earlier work?
Substance: How much substance does this paper have? Is it enough for a full-length paper, or would it benefit from further development?
The contribution of a reproduction paper lies in analyses of and in insights into existing methods and problems—plus the added certainty that comes with validating previous results.
Relevance: Is the subject matter of this paper within the scope of NEJLT?
Readability/clarity: Is the paper well-written and well-structured?
Data/code availability: To what extent is the data/code (as appropriate) used in this submission available to the research community, or is there a compelling reason given why this is not possible?
Analysis: If the paper was able to replicate the results of the earlier work, does it clearly lay out what needed to be filled in in order to do so? If it wasn’t able to replicate the results of earlier work, does it clearly identify what information was missing/the likely causes?
Generalizability: Does the paper go beyond replicating the results on the original to explore whether they can be reproduced in another setting? Alternatively, in cases of non-replicability, does the paper discuss the broader implications of that result?
Informativeness: To what extent does the analysis reported in the paper deepen our understanding of the methodology used or the problem approached? Will the information in the paper help practitioners with their choice of technique/resource?
Meaningful comparison: In addition to identifying the experimental results being replicated, does the paper motivate why these particular results are an important target for reproduction and what the future implications are of their having been reproduced or been found to be non-reproducible?
Resource papers present a new language resource. This could be a corpus, but also could be an annotation standard, tool, and so on.
Relevance: Is the subject matter of this paper within the scope of NEJLT?
Readability/clarity: From the way the paper is written, can you tell how the resource was produced, how the quality of annotations (if any) was evaluated, and why the resource should be of interest?
Originality: Does the resource fill a need in the existing collection of accessible resources? Note that originality could be in the choice of language/language variety or genre, in the design of the annotation scheme, in the scale of the resource, or still other parameters.
Resource quality: What kind of quality control was carried out? If appropriate, was inter-annotator agreement measured, and if so, with appropriate metrics? Otherwise, what other evaluation was conducted, and how agreeable were the results?
Resource accessibility: Will it be straightforward for researchers to download or otherwise access the resource in order to use it in their own work? To what extent can work based on this resource be shared?
Metadata: Do the authors make clear whose language use is captured in the resource and to what populations experimental results based on the resource could be generalized to? In case of annotated resources, are the demographics of the annotators also characterized?
Meaningful comparison: Is the new resource situated with respect to existing work in the field, including similar resources it took inspiration from or improves on? Is it clear what is novel about the resource?
A position paper presents a challenge to conventional thinking or a futuristic new vision. It could open up a new area or novel technology, propose changes in existing research, or give a new set of ground rules.
Relevance: Is the subject matter of this paper within the scope of NEJLT?
Readability/clarity: Is it clear what the position is that the paper is arguing for? Are the arguments for it laid out in an understandable way?
Soundness: Are the arguments presented in the paper relevant and coherent? Is the vision well-defined, with success criteria? (Note: It should be possible to give a high score here even if you don’t agree with the position taken by the authors)
Creativity: How novel or bold is the position taken in the paper? Does it represent well-thought through and creative new ground?
Scope: How much scope for new research is opened up by this paper? What effect could it have on existing areas and questions?
Meaningful comparison: Is the paper well-situated with respect to previous work, both position papers (taking the same or opposing side on the same or similar issues) and relevant theoretical or experimental work?
Substance: Does the paper have enough substance for a full-length paper? Is the issue sufficiently important? Are the arguments sufficiently thoughtful and varied?
A survey paper provides a structured overview of the literature to date on a specific topic that helps the reader understand the kinds of questions being asked about the topic, the various approaches that have been applied, how they relate to each other, and what further research areas they open up. A survey paper should be about a sufficiently focused topic that it can do this successfully with in the page limits.
Relevance: Is the subject matter of this paper within the scope of NEJLT?
Readability/clarity: Is the paper generally easy to follow and well structured?
Organization: Does the paper organize the relevant literature in a narrative and identify common strands of inquiry?
Scope: Does the paper identify a reasonably focused area to survey?
Thoroughness: Given the area identified to survey, does the paper cover all of the relevant literature? Is the literature reviewed represented accurately?
Outlook: To what extent does the paper identify areas for future work and/or clearly point out what is not yet handled within the literature surveyed?
Context: Does the paper situate current research appropriately within its historical context? (We don’t expect papers to start with Pāṇini, yet at the same time something that only cites work from this and last year probably doesn’t capture how current work relates to the bigger picture.)
Submissions are assigned a managing editor (ME), who desk-checks and deals with author contact, and allocates an action editor (AE) to make the decision and to find reviewers.
Papers must reach “minor edits” by the end of the 2nd review cycle. Possible cycle outcomes are:
NEJLT maintains a large standing pool of reviewers, drawn from the global NLP community and from reviewers who have previously been responsive and thorough. We expect AEs to be responsive; in return, we limit their load, and will find new AEs if manuscript load increases. The journal should have as much transparency as possible: this means notifying authors as manuscripts move through the pipeline.
The normal estimated reviewing time is seven weeks, and from 2021, the target maximum time from submission at NEJLT to a first decision is twelve weeks.
Prior reviews from some venues may be submitted with a manuscript. These may be considered by AEs for the first decision, and are released to reviewers only after the first round of reviewing. This is to avoid biasing our reviewers’ initial impression and thus compromising review quality. Consent is requested during manuscript submission to verify reviews with the original venue.
Sometimes a manuscript is not eligible for review and has to be rejected before this step. This is called a desk reject. Desk rejects should be quick. These will be issued for manuscripts that are clearly out of scope; that are much below the required quality standard; that are duplicates or plagiarised; or that openly reveal author names. These are generally unfortunate for both authors and editors to deal with, and we would prefer not to send any desk rejects, so please read the guidelines before submitting a paper; contact us if you are in doubt.