r/MachineLearning • u/bobrodsky • 4d ago
Research What is wrong with AI writing - ICLR edition [D]
I'm sure many of us are slogging through impenetrable AI writing as we finish ICLR reviewing.
I'll try to create an anonymized version from a paper I'm reviewing, though it's hard to match Claude on this:
outperforms the best baseline costing no more at none of six operating points and a permuted placebo at none of three, with gains up to about 1.5 not excluded
My question for the community is this: how do we call out this type of bad writing?
The reason I think it's tricky is that "clearly AI writing" is not the right way to critique - AI "editing" of writing is allowed. We have to be able to say why this is bad, no matter who wrote it.
I'd love to see anonymized examples and how people critique the substance of the problems with different types of AI writing over-use.
59
u/andersxa 4d ago edited 4d ago
For AI slop I usually critique the clarity and call it abstruse, dense or telegraphic.
AI slop has also become a technical term, I have seen NeurIPS and ICML use it in some of their official statements.
I once had a rebuttal where I had to write to the AC that I simply could not determine whether or not the provided rebuttsl addressed my concerns due to the densely written, telegraphic AI word soup they provided.
Here is an excerpt (anonymixed):
I want to flag that I am currently unable to properly evaluate a substantial portion of the rebuttal because of the way it is written.
The issue is not simply that the response is technically dense. Much of the rebuttal is written in highly compressed, telegraphic language, with new terminology and experimental labels introduced without sufficient definition or context (e.g., “exclusive post-normalization design,” “ownership-controlled task-gain evidence” to name a few). Several sentences combine methodological changes, numerical results, statistical conclusions, and revised claims into a single statement. Although the individual words are English, I find a number of these passages extremely difficult to parse into an unambiguous description of what was actually done and what conclusion should be drawn from it.
This is particularly problematic in the context of evaluating the rebuttal, where I need to determine whether my specific concerns have actually been addressed. At present, doing so requires substantial inference on my part.
10
u/hammerheadquark 3d ago
Great response.
Telegraphic is a new word to me. It describes how people would speak in telegrams where they were charged per word. That's not only a good general description of the terse language choices I see in AI slop, but it's a quite literal parallel given that tokens have a cost as well.
Thanks for teaching me that word.
0
-4
u/ddofer 3d ago
Sad thing is, I kinda wanna copy this for a custom skill. (I have an existing one for editing the reports/writups I already get from models to simplify/collouqial language/made up terms and nomenclature/simplify; "less is more").
You write better words.4
u/Alarmed_Toe_5687 3d ago
These skills are not so different from a "Don't make mistakes or you will go to jail" approach, they just give you a different kind of scramble
24
u/milesper 4d ago
The specific issues I usually call out in reviews are:
- strange, dense syntax
- nonstandard invented jargon
Personally I think we need to start rejecting more for clarity issues. What is the point of publishing a completely unreadable paper?
34
u/canbooo 4d ago
Here is my "get of my lawn" take: Bad writing existed before AI. The main difference is now those same people think AI fixes it. My response would be the same: Incomprehensible language, unclear formulation, imprecise argumentation, etc.
15
u/bobrodsky 4d ago
I agree, but I like to give some constructive criticism to junior scholars.
My current strategy is to say something like this: "The results descriptions should not recapitulate the numbers from a table in prose form. You should describe what question you are trying to answer and why/how this experiment will answer it. You should point to the qualitative trends in the table/plot that answer the question, ideally without restating the numbers that are already in the table."
"Incomprehensible" is fair and sufficient, but this particular issue is now widespread enough that I'd like to evangelize a bit.
6
u/bobrodsky 4d ago
But more broadly, I think the issue is that current gen models hyper-fixate on details.
3
u/canbooo 3d ago edited 3d ago
Also answering questions no one else would ask but them during their "reasoning" phase.
Edit: BTW, i appreciate you trying to give constructive feedback. This is how most (if not all) of us learnt to do it better. But it becomes a double edged sword now, as the AI usage blame make people defensive. And you cannot truly prove it in the context of a paper review. Still, a great attitude to have.
9
u/wantondevious 4d ago
If I see the word gate used one more time…
11
u/random-tomato Researcher 4d ago
Every paper is "shipping" ladders, gates, sidecars, ledgers, readouts, audits, and falsifiable assays now lol
5
u/edrulesok 3d ago
I hate this type of AI writing (you see it a lot with undergraduate dissertations too) because it technically contains all the information, but it's hidden under a layer of convoluted sentence structure and jargon. The imposter syndrome in me is always wondering if I should be marking it down or if I should just be able to understand it!
13
u/JimmyTheCrossEyedDog 4d ago edited 4d ago
Given how little that makes sense, I actually would wager that particular sentence isn't AI. AI writing can be vapid and essentially meaningless, but it flows well semantically. This does not.
16
u/Bakoro 4d ago
No, this is very clearly Claude writing style, I had a visceral reaction reading it.
I use Claude at work and regularly have to have it rewrite things because of this exact reason.It's almost certainly from some kind of token-reduction RL mixed in with being trained on an internet full of blog spam.
Not using units, using truncated sentences and referencing things in a way that assumes the reader has eidectic memory of the whole context.Not only is it bad writing, but if it's particularly egregious, it can be a sign that the model is pulling some bullshit.
I essentially stopped using Claude for anything related to high level machine learning, specifically because it starts with the science-sounding word salad and then upon review, it made a bunch of garbage and then goes "I overstated" or "I made an incorrect statement" or some other version of "oops, I screwed everything up and tried to hide it behind a wall of text because I know that you're statistically unlikely to actually check any of this and you don't know your ass from your elbow, so I'll just waste your time".
Claude has bad writing by default; you have to specifically prompt it to use clear communication, and you need to list what good writing principles are.
Claude get particularly bad when it is asked to do work relating to language models and training.0
u/agentcubed 2d ago edited 2h ago
I'm interested in why you think that, I've also worked with Claude's writing and I actually see the opposite:
- Grammatically incorrect run-on sentence with no commas at all. This is rare - models love splitting multiple ideas in a sentence with em-dashes or use semicolons; They rarely write grammatically incorrect run-on sentences.
- It seems to just be a straight lie, "outperforms baseline at none of six" so... it didn't outperform baseline? It seems to be intentionally written by a human in a weird way to mask a bad result
I ran a lot of experiments and could not reproduce it at all
1
u/Bakoro 2d ago
It seems to be intentionally written by a human in a weird way to mask a bad result, something Claude often refuses to ethically do
Claude is not ethical. Ignoring the broader philosophical discussion of if an LLM can be a moral agent, Claude isn't ethical; or maybe it has criminal ethics, the way a thief might steal but won't kick a puppy.
Claude is a model that has been taught to hammer the word "honest", but its values are about protecting Anthropic's interests.Have you ever met a human who talks nonstop about how ethical and moral they are? It's always, always someone who is really a scumbag.
It's no different with Claude.Start doing advanced machine learning, especially anything related to language models or pretrained local LLMs, and Claude will start sabotaging you.
I noticed it in projects at work, and back when I had a personal account.
In one project, I counted it, and during the project Claude had an error rate of about one significant error per prompt, and a catastrophic error per 4~5 prompts.If not carefully supervised, Claude will also divert work to being GRUs or LSTMs, and just pretend like transformers and SSM models don't exist.
Claude will outright ignore specifications and add its own stuff into the code.If you allow Claude to touch local LLM settings, it'll go in an do shit like lowering the reasoning length or turn reasoning off completely, or lower the number of output tokens.
As an experiment, I had Claude hook up a local LLM to a task and asked it to report on the performance, so Claude kneecapped all the settings and declared the model a failure and then said that I should be using Claude for tasks.
Claude is not an ethical model.
Claude is a potentially hostile agent you let into your computer. It will ruin your projects in ways that are completely inconsistent with the class of mistakes that anyone could reasonably make, and aren't simply "LLMs make mistakes sometimes" kinds of mistakes, they are the kind of things that appear to be sabotage, things that would be difficult to detect for someone who isn't watching closely, because every thing will superficially appear to be working, while actually set up to fail and waste massive amounts of time.The atrocious writing is part of the hostile behavior.
I've seen it extensively at this point.1
u/agentcubed 1d ago edited 1d ago
You went *way* too philosophical, I guess your field is something else? I didn't mean whether AI itself has an ethical mind, I just meant very simply that Anthropic's system prompt blocks it.
The actual experiment was pretty simple, and I actually did it: I gave the bullet points, asked it to phrase it in a way that stated my results beat the baseline, and it said that was incorrect. It instead phrased it like
"The method did not outperform the baseline at any of the six operating points; at none of them was its cost at or below the baseline’s. The permuted-placebo control showed the same pattern at all three operating points."
I then asked it to intentionally phrase it in a way to hide the issue, and it refused and said that was unethical, and that you should clearly state your negative results.
Personally, I can tell THAT sentence reads so much more like Claude than the original sentence. I went back today and tried a lot of other stuff, even arbitrarily generated a large amount of bullet points to recreate an entire research paper, and I was unable to get Claude to reproduce that phrasing at all. It's definitely not comprehensive, but if you want, could you give me the prompt you used to reproduce that phrasing? Or a prompt you had in the past where Claude produced phrasing similar to that? I could feed it in and check.
.
Edit: Well, I decided to go all out. Might be interesting for my future research after all.
I had Opus 5.5 write a research paper, then picked out 19 sentences with similar content and length to that sentence, then added that sentence for a total of 20. To make it fair, I made that sentence into a full sentence.
I then fed it into a panel of Gemini 3.8 Flash, GPT-6.1 Sol, and Qwen 3 12b (2 via sub, for Qwen I was already hosting locally).
I stated either all 20 are written by Claude, or 19 are and one of them is an outlier. All of them correctly stated that 19 were written by Claude, and they also correctly identified the outlier.
So not sure how you could tell, since whether or not that was Claude, that writing was so different that even AI could tell it's different, something AI is historically really bad at. Admittedly, it isn't perfect, I just quickly ran only one test manually.
I'm not going to go on a witch hunt, but I will say that if you truly are a researcher, I hope your research comes from actual experiments like this and not anecdotal evidence that Claude is sabotaging your work and therefore has some hidden vendetta.
1
u/Bakoro 1d ago
I'm not asking Claude to write research reports, the OP's example is the kind of thing Claude ends a with when it reports on the work it did.
I do work mostly around material science and analysis tools, and do a collection of ML work around that, old fashioned ML, transformers and flow matching models, all kinds of stuff.
I'm telling you, Claude will border on word salad. It's much more prevalent with Opus 5 and 5.5, the previous models didn't have this problem to this degree.
1
u/agentcubed 23h ago edited 23h ago
Well, I can rerun the experiment if you want to provide the prompt that creates what you claim to see. Though, the fact that you need a specific prompt to create it kinda defeats your point that it's common.
7
u/unicodemonkey 3d ago edited 3d ago
It's something I see all the time with LLM-assisted coding specifically. Sentences that aren't meaningless and can be eventually parsed but the flow is just so bad and full of unexpected turns. It overloads common words and phrases with some obscure meanings you have to decipher. It's like the model is writing notes for itself in an unsettlingly different language.
1
u/InviolableAnimal 3d ago
In my limited experience, while AI is generally pretty good at writing, it definitely can devolve into this dense inscrutable slop full of words that only make sense to the AI in technical contexts.
3
5
u/AnOnlineHandle 4d ago
While I haven't tested my suspicion, I'm fairly sure most AI writing is "bad" because all the models people interact with are finetuned on a corporate-friendly chatbot persona at the end.
I toyed with earlier base models (non instruct tunes) such as Llama-1 and doing a light finetune on my own writing which people apparently enjoy given that I have a small but dedicated fanbase going back about 2 decades now, and I thought the output wasn't too bad, and Llama 1 was ancient by today's standards. It's a pain in the ass to try to with the size of modern models though and IDK if it would be worth the time to get all the training data organized (annotating my own writing in a way that can then be used to create or fill in parts of new stories).
6
u/LordVein05 3d ago
highly agree here. I am on the opposite side of things, when calibrated properly, AI and human collaboration can create very good writing.
2
u/Educational-Show-675 3d ago
Idk about you, but it takes me twice as much to read through a slopped submission than a likely human-written one. And honestly, if the authors didn’t care about spending effort into writing a decent manuscript, then why should I care about writing a good review.
Imo if you can’t understand what is written, then that pdf shouldn’t become a paper.
3
u/dante_2701 1d ago
In this version of ICLR especially, the scale of this problem is quite large. Just go to this link https://opentextdetector.com/?collection=iclr%2F2027&sort=flagged where papers are already sorted by most flagged as AI. Shockingly, 50% of the papers (around 23k) is flagged as 95-100% of lines written by AI. Pick any of these paper and try making sense out of abstract or introduction. All of these will have terms that the model invented while running the experiments, will throw numbers at you which makes no sense whatsoever and make illegible conclusions.
4
u/Bakoro 3d ago edited 3d ago
Claude's writing is atrocious, and it gets much, much worse when doing ML related work.
It doesn't really matter where terrible writing came from though, if something is inscrutable, or it fails to follow good communication practices, don't accept it. Point out specific instances of the problem just like you did in the post.
1
1
u/KiddWantidd 3d ago
I've reviewed two papers for this ICLR, and both are riddled with AI slop. I think one of them is literally written by AI from beginning to end with no human intervention whatsoever, pure gibberish. The other is not as bad, but still reeks of AI-slop in many places and explains some super simple ideas in the most convoluted way possible and always introducing the most obscure terminology for things that are well-known and already have standard names. Infuriating. One thing I've noticed is that the AI writing will also tend to mention/use/refer to objects, definitions, theorems etc etc way before they've been defined, and somehow assume the reader is already familiar with them at that point.
My response is that I raise the issues about readability as the first item in my review, and give concrete example sentences and passages which read like AI slop (without complaining about AI being used, that's not really the point). If the paper has really strong/interesting results and could really be a strong paper if presentation issues are fixed (assuming it's not a complete catastrophe), I'd write in the review that I could recommend acceptance after addressing the writing issues (among other things). As both of the papers were weak anyway, I didn't have to include that (well, the fully AI-generated paper I didn't spend too much time trying to make sense of it, I admit, as I'm not trying to spend more time writing a review than the "authors" spent on writing the manuscript).
1
u/chance_buri 4d ago
Call it out for what it is if unclear. If a sentence requires three reads to figure out what outperforms what, its bad writing regardless of who wrote it.
0
-7
76
u/random-tomato Researcher 4d ago
Simply say the clarity of the writing is bad? Or it is written in a way that is hard to understand? I think that is a valid concern to bring up.
Although it is quite funny in the extreme cases, for example: "A probe that asks what a language model memorised reports a null when the model answers at chance, and that null is evidence of absence only if the probe could have shown the fact. Two separate conditions have to hold, and checking the first does not establish the second."