idk why are you changing targets for comparison?

it's like:

"does apple cure cancer in monkeys?" vs "does blueberry cure diabetes in pigs?"

> using VADER over something like DistilBERT is NLP malpractice in 2025.

Ouch. Was that necessary?

I used $1000 worth of GPU credits and threw in VADER because it’s basically free both in time and credits.

I usually do this on large dataset out of pure interest in how it correlates with expensive methods on English language text.

I am well aware of how VADER works and its limitations, I am also aware of the limitations of all sentiment analysis.

Price isn't a real issue in almost every imaginable use case either. Even a small open source model would outperform and you're going to get a lot of tokens per dollar with that.

> dictionary mapping each word to a single sentiment weight

That seems to me like it would flat out fail on sarcasm. How is that still considered a usable method today?

To be fair it is still stuck there, so

That is a really good question, I am not sure where to draw the line.

I think it would be safe to say BERT is/was firmly in the non-traditional side of NLP.

A variety of task specific RNN models preceded BERT, and RNN as a concept has been around for quite a long time, with the LSTM being more modern.

Maybe word2vec ushered in the end of traditional NLP and was simultaneously also the beginning of non-traditional NLP? Much like Newton has been said to be both the first scientist and also the last magician.

I find discussing these kind of questions with NLP academics to be awkward.

Yes, my comment focused on NLP research but the importance of university teaching has also taken a hit - not that I fear bullying, but now students can have a dedicated custom teacher with infinite time and patience and that can answer questions at 3 AM, and obviously that reduces the relevance of the professor. While the human interaction in in-person teaching still provides some exclusive value, demand logically should go down. Although don't underestimate the power of inertia - one could also think that small noname universities would go out of business when the likes of MIT began offering their courses to everyone online, and it didn't happen. I do think LLMs bring higher risk than that, and a shrinking will indeed happen, but maybe not so dramatic. Let's see.

Regarding science, if we leave it exclusively to corporations we won't get very far, because most corporations aren't willing to do the basic/foundational science work. The Transformers and most of the relevant followup work that led to LLMs were developed in industry, but they wouldn't have been possible without the academics that kept working on neural networks while that field was actively scorned during the 90s-2000s AI winter. So I think research universities still should have a role to play. Of course, convincing the funders that this is indeed the case might be a different story.

Not totally related but I have wondered how someone who thinks they are an expert in a field may deal with contradictions presented by GPT.

For example, you may consider yourself an expert on some niche philosophy like say Orientalism. A student can now contradict any theories you can come up with using GPT and the scary thing is that they will be sensible contradictions.

I feel like the bar to consider yourself an expert is much higher - you not only have to beat your students but also know enough to beat GPT.

Its not the same at all. Llm is big yes and thats part of it. But llm is small compared to equivalent performance machine with something like ngram statistical models. You'd need the whole universe. Or something prohibitive like that. And itd still be worse. People don't like it but LLMs 'understand' texts in a very real meaning of the word. Because that's the most compressive way to do the task it's trained to. Is it the same as human understanding? Most likely not, but that complain is cheating.

NLP is indeed at its top, NLP professors aren't :)

Imagine if you had stayed in academia and kept working in MT for the last two decades. First of all, now you would see how LLMs render all your work pretty much obsolete. That's already hard for many people. Not so much for me, as I consider myself to be rather adaptable, and maybe not for you either - you can embrace the new thing and start working on it yourself, right?

But the problem is that you can't. The main issue is not lack of explainability, but the sheer resources needed. In academia we can't run, let alone train, anything within even one or two orders of magnitude of ChatGPT. We can work with toy models, knowing that we won't get even remotely near the state of the art (many are now doing this, with the excuse of "green AI", sustainability and such, but that won't even hold much longer). Or we can work with the likes of ChatGPT as plain users, but then we are studying the responses of an "oracle" that we don't even have control of, and it starts looking more like religion than science.

Ten years ago an academic could beat the state of the art in MT and many other NLP tasks, now for many tasks that's just impossible (unless we count coming up with some clever prompt engineering, but again, religion). For those of us who were in this field because we liked it, not only to make a living, research has become quite unfulfilling.

Thanks for the link, just read it, and the Chomsky transcript. Chomsky wanted deep structure, Norvig bet on stats, but maybe Turing saw it coming, kids talk before they know grammar and so did the machines. It turns out we didn’t need to understand language to automate it.

> ever get like ASIC for ml

Is this what you're mentioning?

[0] https://linearmicrosystems.com/using-asic-chips-for-artifici...

In NLP, transformers replaced RNNs. In computer vision, CNNs replaced previous methods (e.g. feature descriptors), and recently got replaced by visual transformers, though modern CNNs are still pretty good.

It's called the AI effect: https://en.wikipedia.org/wiki/AI_effect

> The author Pamela McCorduck writes: "It's part of the history of the field of artificial intelligence that every time somebody figured out how to make a computer do something—play good checkers, solve simple but relatively informal problems—there was a chorus of critics to say, 'that's not thinking'."

It doesn’t mean they tie intelligence to subjective experience. Take digestion. Can a computer simulate digestion, yes. But no computer can “digest” if it’s just silicon in the corner of an office. There are two hurdles. The leap from simulating intelligence to intelligence, and the leap from intelligence to subjective experience. If the computer gets attached to a mechanism that physically breaks down organic material, that’s the first leap. If the computer gains a first person experience of that process, that’s the second.

You can’t just short-circuit from simulates to does to has subjective experience.

And the claim other humans don’t have subjective experience is such non-starter.

This is why I want the field to go straight to building indistinguishable agents- specifically, you should be able to video chat with an avatar that is impossible to tell from a human.

Then we can ask "if this is indistinguishable from a human, how can you be sure that anybody is intelligent?"

Personally I suspect we can make zombies that appear indistinguishable from humans (limited to video chat; making a robot that appears human to a doctor would be hard) but that don't have self-consciousness or any subjective experience.

Humans are basically incapable of recognizing that there’s something that’s more powerful than them

They’re never going to actively collectively admit that that’s the case, because humans collectively are so so systematically arrogant and self possessed that they’re not even open to the possibility of being lower on the intelligence totem pole

The only possible way forward for AI is to create the thing that everybody is so scared of so they can actually realize their place in the universe

I will not be convinced a machine demonstrates intelligence until someone demonstrates a robot that can navigate 3D space as intelligently as, say, a cockroach. AFAICT we are still many years away from this, probably decades. A bunch of human language knowledge and brittle heuristics doesn't convince me at all.

This ad hominem is really irritating. People have complained since Alan Turing that AI research ignores simpler intelligence, instead trying to bedazzle people with fancy tricks that convey the illusion of human intelligence. Still true today: lots of talk about o3's (exaggerated) ability to do fancy math, little talk about its appallingly bad general quantitative reasoning. The idea of "jagged AI" is unscientific horseshit designed to sweep this stuff under the rug.

In the natural world, intelligence requires embodiment. And, depending on your point of view, consciousness. Modern AI exhibits neither of those characteristics.

How do they convince themselves that other people have intelligence too?

It is until proven otherwise because modern science still doesn’t have a consensus or standards or biological tests which can account for it. As in, highly “intelligent” people often lack “common sense” or fall prey to con artists. It’s pompous as shit to assert a black box mimicry constitutes intelligence. Wake me up when it can learn to play a guitar and write something as good as Bob Dylan and Tom Petty. Hint: we’ll both be dead before that happens.

> And are you really trying to present that you have practical experience building comparable tools to an LLM prior to the Transformer paper being written?

I believe (could be wrong) they were talking about their prior GOFAI/NLP experience when referencing scaling systems.

In any case, is it really necessary to be so harsh about over-confidence and then go on to predict the future of solving hallucinations with your formal verification ideas?

Talk is cheap. Show me the code.

[dead]

Over my years in academia, I noticed that the linguistics departments were always the most fiercely ideological. Almost every comment of a talk would be get contested by somebody from the audience.

It was annoying, but as a psych guy I was also jealous of them for having such clearly articulated theoretical frameworks. It really helped them develop cohesive lines of research to delineate the workings of each theory

Maybe they could’ve tried to say something pro-linguistics, but the comment was entirely anti-LLM.

The interesting question is whether just a gigantic set of probabilities somehow captures things about language and cognition that we would not expect ...

> Don't try and say anything pro-linguistics here, (...)

Shit-talking LLMs without providing any basis or substance is not what I would call "pro-linguistics". It just sounds like petty spiteful behavior, lashing out out of frustration for rendering old models obsolete.

This paper is awful. They bizarrely argue the fact that transformers are not very sensitive to word order as a positive of transformers despite the fact that's not how languages work. There's also this absurd passage.

>However, a closer look at the statistical structure of language use reveals that word order contains surprisingly little information over and above lexical information. To see this intuitively, imagine we give you a set of words {dogs, bones, eat} without telling you the original order of the words. You can still reconstruct the meaning based entirely on (1) the meanings of the words in isolation and (2) your knowledge of how the world works—dogs usually eat bones; bones rarely eat dogs. Indeed, many languages show a high level of nondeterminism in word order (Futrell et al., 2015b; Koplenig et al., 2017), and word order cues are often redundant with meaning or case markers (Pijpops and Zehentner, 2022; Mahowald et al., 2023). The fact that word order is relatively uninformative in usage also partly explains why bag-of-words methods dominated NLP tasks until around 2020, consistently outperforming much more sophisticated approaches: it turns out that most of the information in sentences is in fact present in the bag of words.

While it is certainly possible to guess that your interlocutor meant "dogs eat bones", the sentence "bones eat dogs" is entirely possible (if unlikely)! For example, imagine a moving skeleton in a video game or something. The idea that word order isn't vital to meaning is deeply unserious. (Of course there are languages where word order matters less, but there are still important rules about constituent structure, clausal embedding etc, which constrain word order).

It tends to happen for any prompt that calls for generating a piece of output for which there are many valid answers, but one is highly weighted and you want variety. Do you remember that meme a few years ago where people were asked to generate a color and then a hand tool, and most people immediately responded "erqunzzrebengyrnfgbarbsgubfrgjb"? (rot13+padding for those who haven't done this)

This particular example is too small to regularly trip AIs, but as a general rule I do not consider it tricky to try to textually negative-prompt to remove a commonly-valid-but-not-currently-wanted response. (Obviously, if you manually tweak the weights to forbid something rather than using the word "not", this fails.)

From my very rough observations, for models that fit on a local device, it typically starts to happen maybe 10% of the time when the prompt reaches 300 characters or so (specifying other parts of what you want); bigger models just need a bit more input before they fail. Reasoning models might be better, but we can watch them literally burn power running nonsensical variations through the thought pane so they're far from a sure answer.

This happens in any context you can think of: from scratch or extending an existing work; single or list; information lookup, prose generation, code generation (consider "Extend this web app to do lengthy-description-of-some-task. Remember I am not using React you stupid piece of shit AI!").

Just like having employees with experience, I guess.

But tenured researchers are supposed to have some more protection specifically because they do research (and reach conclusions) on topics that people in leadership positions in society might not like.

This looks like the post: https://jdsemrau.substack.com/p/finer-gliner-and-smolagents-...

This is pretty much correct. I'd have to search for it but I remember an article from a couple years back that detailed how LLMs blew up the field of NLP processing overnight.

Although I'd also offer a slightly different lens through which to look at the reaction of other researchers. There's jealousy, sure, but overnight a ton of NLP researchers basically had to come to terms with the fact that their research was useless, at least from a practical perspective.

For example, imagine you just got your PhD in machine translation, which took you 7 years of laboring away in grad/post grad work. Then something comes out that can do machine translation several orders of magnitude better than anything you have proposed. Anyone can argue about what "understanding" means until they're blue in the face, but for machine translation, nobody really cares that much - people just want to get text in another language that means the same thing as the original language, and they don't really care how.

Tha majority of research leads to "dead ends", but most folks understand that's the nature of research, and there is usually still value in discovering "OK, this won't work". Usually, though, this process is pretty incremental. With LLMs all of a sudden you had lots of folks whose life work was pretty useless (again, from a practical perspective), and that'd be tough for anyone to deal with.

I wouldn't say NLP as a field was resistant to probabilistic approaches or even neural networks. From maybe 2000-2018 almost all the papers were about using probabilistic methods to figure out word sense disambiguation or parsing or sentiment analysis or whatever. What changed was that these tasks turned out not to be important for the ultimate goal of making language technologies. We thought things like parsing were going to be important because we thought any system that can understand language would have to do so on the basis of the parse tree. But it turns out a gigantic neural network text generator can do nearly anything we ever wanted from a language technology, without dealing with any of the intermediate tasks that used to get so much attention. It's like the whole field got short-circuited.

The way I have experienced this, starting from circa 2018, it was a bit more incremental. First, LSTMs and then transformers lead to new heights on the old tasks, such as syntactic parsing and semantic role labelling, which was sad for the previous generation, but at least we were playing the same game. But then not only the old tools of NLP, but the research questions themselves became irrelevant because we could just ask a model nicely and get good results on very practical downstream tasks that didn't even exist a short while ago. NLP suddenly turned into general document/information processing field, with a side hustle in conversational assistants. Already GPT2 essentially mastered the grammar of English, and what difficulties remain are super-linguistic and have more to do with general reasoning. I would say that it's not that people are bitter that other people make progress, it's more that there is not much progress to be had in the old haunts at all.

I think you greatly understate the impact as EVERYONE is freaking the fuck out about AI, not just NLP researchers.

AI is obliterating the usefulness of all mental work. Look at the high percentage of HN articles trying to figure out whether LLMs can eliminate software developers. Or professional writers. Or composers. Or artists. Or lawyers.

Focusing on the NLP researchers really understates the scope of the insecurity induced by AI.

All of this matches my understanding. It was interesting taking an NLP class in 2017, the professors said basically listen, this curriculum is all historical and now irrelevant given LLMs, we’ll tell you a little about them but basically it’s all cutting edge sorry.

Even 15-ish years ago when I was in school, the NLP folks viewed probabilistic models with suspicion. NLP treated everyone from our Math department with suspicion and gave them a hard time. It created so many politics that some folks who wanted to do statistical approaches would call themselves CS so that the NLP old guard wouldn't give them a hard time.

Sounds like the bitter lesson is bitter indeed!

The progression reminds me of how brute force won out in the chess AI game long ago with Deep Blue. Custom VLSI and FPGA acceleration and all.

> most effective way to make forward progress

powerful response but.. "fit for what purposes" .. All of human writings are not functionally equivalent. This has been discussed at length. e.g. poetry versus factual reporting or summation..

I agree with criticism of Noam Chomsky as a linguist. I was raised in the typological tradition which has its very own kind of beef with Chomsky due to other reasons (his singular focus on English for constructing his theories amongst other things), but his dislike of statistical methods was of course equally suspect.

Nevertheless there is something to be said for classical linguistic theory in terms of constituent (or dependency) grammars and various other tools. They give us much simpler models that, while incomplete, can still be fairly useful at a fraction of the cost and size of transformer architectures (e.g. 99% of morphology can be modeled with finite state machines). They also let us understand languages better - we can't really peek into a transformer to understand structural patterns in a language or to compare them across different languages.

do transformers not use a symbolic and a probabilistic approach?

It reminds me much more of Paul Feyerabend's even better book "Against Method" https://en.wikipedia.org/wiki/Against_Method

Or Planck's principle - "Science progresses one funeral at a time".

> That the whole field seems to be moving in a direction where you need a lot of resources to do anything. You can have 10 different ideas on how to improve LLMs but unless you have the resources there is barely anything you can do.

I think you're confusing problems, or you're not realizing that improving the efficiency of a class of models is a research area on it's own. Look at any field that involves expensive computational work. Model reduction strategies dominate research.