ACL Short Papers Review Form proposal

ACL short

START Conference Manager

NAACL HLT 2009 Short Papers Review Form proposal

Review Form

Submission #-1: Your Paper Title

Authors: Your Name

Reviewer: Your Reviewer's Name

Secondary Reviewer (if any):

Ranking

Please evaluate the submission according to the criteria below. Scores for numerical categories are ordered from "bad" to "good." That is, a low score represents a nega= tive evaluation, and a high score represents a positive evaluation.

Evaluation Category

Attributes of short papers (checkboxes)=
Does this paper have particular attributes that make it suita= ble for being a short paper at NAACL HLT 2009? Check any attributes that ap= ply. (You may check more than one or no boxes here. If you check no boxes= , you should normally give a lower rating to the next question.)
A small, focused contributio= n
Work in progress
A negative result
An opinion piece
An interesting applicati= on note

Appropriateness (1-5)
Based on the att= ributes of the paper, does it fit in as a NAACL HLT 2009 short paper?
5 = Very appropriate for an NAACL HLT short paper, addresses an import= ant and/or cross-cutting problem.
4 = Computational linguistics or Human Language Technology, but is of = interest to a small group or not a great fit to the short paper form.
3 = CL or HLT, but ill-suited to being a short paper.
2 = Only marginally relevant or appropriate to a CL or HLT audience.
1 = Inappropriate.

Clarity (1-5)
For the reasonably well-= prepared reader, is it clear what was done and why? Is the paper well-written and well-structured? Does the English or the mathematics need cleaning up? Would the explanation benefit from more examples or pictures?

Is there sufficient detail for an expert to validate the work, i.e., by r= eplicating experiments or filling in theoretical steps?

(Take into account whether any obscurity or minor English errors could be= fixed with relatively little effort, or whether the paper requires more work than is likely to be carried out in the 2.5 weeks available.)
5 = Admirably clear.
4 = Understandable by most readers.
3 = Mostly understandable to me with some effort.
2 = Important questions were hard to resolve even with effort.
1 = Much of the paper is confusing.

Primary contribution of the paper (radioboxe= s)
What is the primary contribution of this paper? What is the dimension on = which it stands out as suitable for publication?

An interesting application should be useful to other researchers in = HLT, and/or an interesting commercial application of HLT.

A good experimental study can have either positive or negative resul= ts.

Defines an interesting new prob= lem
Presents a clever twist on = an old problem
Presents an interesting app= lication
Tells us something interest= ing about language
Presents a novel technique
Presents a technique that i= s new to CL or HLT
Presents/exploits a new dat= a source
Advances the state of the a= rt on a problem
Comprehensive, good experim= ental study
Presents a novel, interesti= ng perspective
Hard to tell
As far as I could see, ther= e wasn't one

Originality / Innovativeness (1-5)
How= original is the approach? Does this paper break new ground in topic, methodology, or content? How exciting and innovative is the research it describes?

(Note that a paper could score high for originality even if the results did not show a convincing benefit; an application note may likely sc= ore low on originality in terms of theoretical techniques.)
5 = Surprising: Noteworthy new problem, technique, methodology, or ins= ight.
4 = Creative: Relatively few people in our community would have put th= ese ideas together.
3 = Somewhat conventional: A number of people could have come up with = this if they thought about it for a while.
2 = Rather boring: Obvious, or a minor improvement on familiar techniq= ues.
1 = Significant portions have actually been done before or done better= .

Soundness / Correctness (1-5)
Is the t= echnical approach sound and well-chosen? Can one trust the claims of the paper? Are the claims supported in an approp= riate way, such as by proper experiments, proofs, or argumentation, relative= to the nature and degree of completeness of the work?
5 = The approach is very apt, and the claims are convincingly supporte= d (or, in the case of work in progress, the existing work points towards a f= inal work that will convincingly support the claims).
4 = Generally solid work, though I have a few suggestions about how to strengthen the technical approach, argumentation, or evaluation= .
3 = Fairly reasonable work. The approach is not bad, and at least the main claims are probably correct (or, in the case of opinion pieces, interes= ting), but I am not entirely ready to accept them (based on the material in = the paper).
2 = Troublesome. There are some ideas worth salvaging here, but the wo= rk should really have been done or evaluated differently (or, in the case of= work in progress, it should be done or justified better).
1 = Fatally flawed.

Meaningful Comparison (1-5)
Does the a= uthor make clear where the problems, methods and ideas sit with respect to existing literature? Are the references adequate? Are any experimental results meaningfully compared with the best prior approaches?
5 = Precise and complete comparison with related work. Good job given the space constraints.
4 = Mostly solid bibliography and comparison, but I have some suggestions.
3 = Bibliography and comparison are somewhat helpful, but it could be hard for a reader to determine exactly how this work relates to previous work.
2 = Only partial awareness and understanding of related work, or a fla= wed empirical comparison.
1 = Little awareness of related work, or lacks necessary empirical com= parison.

Impact of Ideas or Results (1-5)
How s= ignificant are the ideas or work described? If the ideas are novel, will th= ey also be useful or inspirational? If the results are sound, are they also important? If this is a commercial application note, will the app= lication be an important and visible contribution from HLT?
5 = Will affect the field by altering other people's choice of researc= h topics or basic approach.
4 = Some of the ideas or results will substantially help other people'= s ongoing research or development.
3 = Interesting but not too influential. The work will be cited or us= ed, but mainly for comparison or as a source of minor contributions.
2 = Marginally interesting. May or may not be cited.
1 = Will have no impact on the field.

Impact of Resources (1-5)
In addition = to its direct intellectual contributions, does the paper promise to release any new resources, such as an implementation, a toolkit, or new data?

If so, is it clear what will be released and when? If so, will these resources be valuable to others in the form in which they are released? Do they fill an unmet need? Are they at least sufficient to replicate or better understand the research in the paper?

(This question encourages authors to help the field advance, by releas= ing their systems, data, or tools. But many papers will not provide any reso= urces.)

New software that will notic= eably affect the field
Software that will be us= eful to others
Software that allows oth= ers to replicate the work
New data that will notic= eably affect the field
Data that will be useful= to others
Data that allows others = to replicate the work
Minor contribution of so= ftware or data
No new resources provide= d

Recommendation (1-5)
There are many good submissions competing for slots at NAACL HLT short; how important is it to feature this one? Will people learn a lot by reading this paper or seeing it presented?

In deciding on your ultimate recommendation, please think over all your scores above. But remember that no paper is perfect, and remember that we want a conference full of interesting, diverse, and timely work. If a paper has some weaknesses, but you really got a lot out of it, feel free to fight for it. If a paper is solid but you could live without it, let us know that you're ambivalent. Remember also that the author has a couple of weeks to address reviewer comments before the camera-ready deadline.

Should the paper be accepted or rejected?
5 = Exciting: I'd fight to get it accepted
4 = Worthy: I would like to see it accepted
3 = Borderline: I'm ambivalent about this one
2 = Mediocre: I'd rather not see it in the conference
1 = Poor: I'd fight to have it rejected

Presentation Format (1-5)
If this pape= r were to be accepted, what format do you think would best serve the audienc= e? For example, a talk is best for work that needs a bit more time to explai= n. A poster may be a good choice if the key points can be illustrated with= a few figures, described easily in 5-10 minutes, and/or the audience would = benefit from an interactive demo. Please consider the topic and depth of the= paper rather than the quality.

Talk
Poster

Reviewer Confidence (1-5)
5 = Positive = that my evaluation is correct. I read the paper very carefully and am very familiar with related work.
4 = Quite sure. I tried to check the important points carefully, and checked for uncited prior work. It's unlikely, though conceivable, that I missed something that should affect my ratings.
3 = Pretty sure, but there's a chance I missed something. Although I have a good feel for this area in general, I did not carefully check the paper's details, e.g., math, experimental design, novelty.
2 = Willing to defend evaluation, but it is fairly likely that I missed some details, didn't understand some central points, or can't be sure about the novelty of the work.
1 = Not my area, or paper is very hard to understand. My evaluation is just an educated guess.

Evaluation Category
Attributes of short papers `(checkboxes)`= Does this paper have particular attributes that make it suita= ble for being a short paper at NAACL HLT 2009? Check any attributes that ap= ply. (You may check more than one or no boxes here. If you check no boxes= , you should normally give a lower rating to the next question.)	A small, focused contributio= n Work in progress A negative result An opinion piece An interesting applicati= on note
Appropriateness `(1-5)` Based on the att= ributes of the paper, does it fit in as a NAACL HLT 2009 short paper? 5 = Very appropriate for an NAACL HLT short paper, addresses an import= ant and/or cross-cutting problem. 4 = Computational linguistics or Human Language Technology, but is of = interest to a small group or not a great fit to the short paper form. 3 = CL or HLT, but ill-suited to being a short paper. 2 = Only marginally relevant or appropriate to a CL or HLT audience. 1 = Inappropriate.
Clarity `(1-5)` For the reasonably well-= prepared reader, is it clear what was done and why? Is the paper well-written and well-structured? Does the English or the mathematics need cleaning up? Would the explanation benefit from more examples or pictures? Is there sufficient detail for an expert to validate the work, i.e., by r= eplicating experiments or filling in theoretical steps? (Take into account whether any obscurity or minor English errors could be= fixed with relatively little effort, or whether the paper requires more work than is likely to be carried out in the 2.5 weeks available.) 5 = Admirably clear. 4 = Understandable by most readers. 3 = Mostly understandable to me with some effort. 2 = Important questions were hard to resolve even with effort. 1 = Much of the paper is confusing.
Primary contribution of the paper `(radioboxe= s)` What is the primary contribution of this paper? What is the dimension on = which it stands out as suitable for publication? An interesting application should be useful to other researchers in = HLT, and/or an interesting commercial application of HLT. A good experimental study can have either positive or negative resul= ts.	Defines an interesting new prob= lem Presents a clever twist on = an old problem Presents an interesting app= lication Tells us something interest= ing about language Presents a novel technique Presents a technique that i= s new to CL or HLT Presents/exploits a new dat= a source Advances the state of the a= rt on a problem Comprehensive, good experim= ental study Presents a novel, interesti= ng perspective Hard to tell As far as I could see, ther= e wasn't one
Originality / Innovativeness `(1-5)` How= original is the approach? Does this paper break new ground in topic, methodology, or content? How exciting and innovative is the research it describes? (Note that a paper could score high for originality even if the results did not show a convincing benefit; an application note may likely sc= ore low on originality in terms of theoretical techniques.) 5 = Surprising: Noteworthy new problem, technique, methodology, or ins= ight. 4 = Creative: Relatively few people in our community would have put th= ese ideas together. 3 = Somewhat conventional: A number of people could have come up with = this if they thought about it for a while. 2 = Rather boring: Obvious, or a minor improvement on familiar techniq= ues. 1 = Significant portions have actually been done before or done better= .
Soundness / Correctness `(1-5)` Is the t= echnical approach sound and well-chosen? Can one trust the claims of the paper? Are the claims supported in an approp= riate way, such as by proper experiments, proofs, or argumentation, relative= to the nature and degree of completeness of the work? 5 = The approach is very apt, and the claims are convincingly supporte= d (or, in the case of work in progress, the existing work points towards a f= inal work that will convincingly support the claims). 4 = Generally solid work, though I have a few suggestions about how to strengthen the technical approach, argumentation, or evaluation= . 3 = Fairly reasonable work. The approach is not bad, and at least the main claims are probably correct (or, in the case of opinion pieces, interes= ting), but I am not entirely ready to accept them (based on the material in = the paper). 2 = Troublesome. There are some ideas worth salvaging here, but the wo= rk should really have been done or evaluated differently (or, in the case of= work in progress, it should be done or justified better). 1 = Fatally flawed.
Meaningful Comparison `(1-5)` Does the a= uthor make clear where the problems, methods and ideas sit with respect to existing literature? Are the references adequate? Are any experimental results meaningfully compared with the best prior approaches? 5 = Precise and complete comparison with related work. Good job given the space constraints. 4 = Mostly solid bibliography and comparison, but I have some suggestions. 3 = Bibliography and comparison are somewhat helpful, but it could be hard for a reader to determine exactly how this work relates to previous work. 2 = Only partial awareness and understanding of related work, or a fla= wed empirical comparison. 1 = Little awareness of related work, or lacks necessary empirical com= parison.
Impact of Ideas or Results `(1-5)` How s= ignificant are the ideas or work described? If the ideas are novel, will th= ey also be useful or inspirational? If the results are sound, are they also important? If this is a commercial application note, will the app= lication be an important and visible contribution from HLT? 5 = Will affect the field by altering other people's choice of researc= h topics or basic approach. 4 = Some of the ideas or results will substantially help other people'= s ongoing research or development. 3 = Interesting but not too influential. The work will be cited or us= ed, but mainly for comparison or as a source of minor contributions. 2 = Marginally interesting. May or may not be cited. 1 = Will have no impact on the field.
Impact of Resources `(1-5)` In addition = to its direct intellectual contributions, does the paper promise to release any new resources, such as an implementation, a toolkit, or new data? If so, is it clear what will be released and when? If so, will these resources be valuable to others in the form in which they are released? Do they fill an unmet need? Are they at least sufficient to replicate or better understand the research in the paper? (This question encourages authors to help the field advance, by releas= ing their systems, data, or tools. But many papers will not provide any reso= urces.)	New software that will notic= eably affect the field Software that will be us= eful to others Software that allows oth= ers to replicate the work New data that will notic= eably affect the field Data that will be useful= to others Data that allows others = to replicate the work Minor contribution of so= ftware or data No new resources provide= d
Recommendation `(1-5)` There are many good submissions competing for slots at NAACL HLT short; how important is it to feature this one? Will people learn a lot by reading this paper or seeing it presented? In deciding on your ultimate recommendation, please think over all your scores above. But remember that no paper is perfect, and remember that we want a conference full of interesting, diverse, and timely work. If a paper has some weaknesses, but you really got a lot out of it, feel free to fight for it. If a paper is solid but you could live without it, let us know that you're ambivalent. Remember also that the author has a couple of weeks to address reviewer comments before the camera-ready deadline. Should the paper be accepted or rejected? 5 = Exciting: I'd fight to get it accepted 4 = Worthy: I would like to see it accepted 3 = Borderline: I'm ambivalent about this one 2 = Mediocre: I'd rather not see it in the conference 1 = Poor: I'd fight to have it rejected
Presentation Format `(1-5)` If this pape= r were to be accepted, what format do you think would best serve the audienc= e? For example, a talk is best for work that needs a bit more time to explai= n. A poster may be a good choice if the key points can be illustrated with= a few figures, described easily in 5-10 minutes, and/or the audience would = benefit from an interactive demo. Please consider the topic and depth of the= paper rather than the quality.	Talk Poster
Reviewer Confidence `(1-5)` 5 = Positive = that my evaluation is correct. I read the paper very carefully and am very familiar with related work. 4 = Quite sure. I tried to check the important points carefully, and checked for uncited prior work. It's unlikely, though conceivable, that I missed something that should affect my ratings. 3 = Pretty sure, but there's a chance I missed something. Although I have a good feel for this area in general, I did not carefully check the paper's details, e.g., math, experimental design, novelty. 2 = Willing to defend evaluation, but it is fairly likely that I missed some details, didn't understand some central points, or can't be sure about the novelty of the work. 1 = Not my area, or paper is very hard to understand. My evaluation is just an educated guess.

Detailed Comments

Please supply detailed comments to back up your rankings. These comments will be forwarded to the authors of the paper. The comments will help the committee decide the outcome of the paper, and will help justify this decision for the authors. Moreover, if the paper is accepted, the comments should guide the authors in making revisions for a final manuscript. Hence, the more detailed you make your comments, the more useful your review will be - both for the committee and for the authors.

Enter comments here:

Confidential Comments for Committee

You may wish to withhold some comments from the authors, and include them solely for the committee's internal use. For example, you may want to express a very strong (negative) opinion on the paper, which might offend the authors in some way. Or, perhaps you wish to write something which would expose your identity to the authors. If you wish to share comments of this nature with the committee, this is the place to put them.

START Conference Manager (V2.54.5)

NAACL HLT 2009 Short