Noam Chomsky, Large Language Models (LLM) and AI

A retrospectives from the paper "Genuine Explanation and the Strong Minimalist Thesis"

After a long time, Noam Chomsky published a paper, “Genuine Explanation and the Strong Minimalist Thesis” [1]. Beyond the importance of linguistic theory discussed in the paper, the criticism of recent AI developments are very interesting. The note is a quick analysis of Chomsky’s observations.

Large Language Models (LLM)

Chomsky is known for his Linguistic theories and quest to explain the phenomena of Language and Universal Grammar. Within his new paper's purview, he seems pessimistic about the Large Language Model and the value it is expected to deliver. The potential importance of linguists in identifying and generating frameworks for mitigating the hallucinations needs to be addressed. His criticism is very sharp, and to quote the paper, “it does just as well or better with systems that violate base principles of language.” He pulls sustainability arguments to prove his argument, “telling us nothing about language and the fair question is whether it is doing anything at all apart from using a lot of Californians energy.” The massive GPU computes infrastructure used by OpenAI and Google in building LLMs are the subject here. In the second paragraph of his LLM criticism, he calls the LaMDA from Google “Another joke, for the same reasons.”

The long-standing impacts of such criticism by pioneers in Linguistics and Computational Linguistics take us back to the early days of Natural Language Processing (NLP). At this juncture, linguists stay criticism only part with LLM and ignore the harm, and not collaborating and designing frameworks will bring another George Town Experiment impacted era.  

While LLM is not intended to address everything about language (by accepting Chomsky’s criticism), it mimics the very language production. Whether it cares about the I-Language or Faculty of Language(FL) is not a consideration for Artificial intelligence developers and inventors. Accepting the same is high time for a systematic study of hallucination patterns and false authority detections to avoid AI-based language generators going from a crawl to walking and running the state with minor to significant mistakes. To address it, it is essential to have greater direction and collaboration initiation from authorities like Chomsky, but not limited to him alone.   

Artificial Intelligence

Considering the early collaboration and contribution of Chomsky linguistic theories influenced significant advances in computing, including FORTRAN, his view about recent developments in AI is interesting. He is fully cognizant of the contributions and impacts AI brought to humanity. He believes we are “benefitting from clever engineering.” “It is like building a better bulldozer. And that is now pretty much what ai is.” He observes that there is a potential for a GPT-like system to answer some questions about linguistics. But the following comment makes his argument strong.

“say the gpt series, or Google’s LaMDA, the one I speak about first now. I don’t see any point to them at all. But what is the point of a system that can match 45 tera-things in 45 terabytes of data and make you think you are listening to something real. That is a game. But as long as ai is following that direction, it may be useful for linguistics like maybe it can study huge number amounts of data and find properties that were not noticed, something like that. It could be a good tool. But again, it is an engineering result.”

These observations hint at the commercialization of human-centered AI systems and the shrinking of engineering product designs.

“If you have a proposal to improve gpt4 to gpt7, you get paid for it, you get a job. That is contributing nothing except some engineering applications. I myself hope that ai will return to its early days, the days of Alan Turing, Marvin Minsky, Herbert Simon, John McCarthy, (and) others who were interested in scientific questions. But that is mostly been marginalized.”

For the last decade, AI, especially NLP, came from the close circle of academics to practical applications and an accessible branch of knowledge to the larger engineering community. The glory of interdisciplinary subjects and the necessity of collaboration with language and linguistic experts evaporated or pivoted to a new dimension. While Andres NG says about Data Centric AI, the reference goes to collaborations, including linguistic communities, to build quality data for better AI solutions. When Chomsky mentions a ‘marginalized community interested in the scientific question,’ the margin and gap widen, while we claim tokens in Natural Language Generation and Augmentations with Generative AI (GenAI). It is time to look back at what happened in the initial days of Machine Translation research. There was a point where researchers realized a lexicon, rules, and computers would solve Machine Translation. We are still inventing and evolving with neural encoders, decoders, and transformers. Chomsky's observations may be biased from his peers and his scholarly interactions with the pioneers in AI and computing. At the same time, standing in the shoes of an implementor, integrator, and observer community, it has deep elements that require attention to detail.

Conclusion

While Chomsky looks optimistic about the LLM, his observations need more guidance to address the linguistic challenges LLM brings. While LLMS are statistical language models, the minimalism and Universal Grammar (UG) relevance in the LLM hallucination mitigations may have value. Looking at the larger linguistic theory, I decided to take one example and ask ChatGPT. Here is what I found, and Chomsky is correct. :-)

His observations about AI developments hint that the AI Winter is coming. At the same time, we can't ignore the ground-level realities which may fuel his comments.   

[1] Genuine Explanation and the Strong Minimalist Thesis, Noam Chomsky,https://brill.com/view/journals/cose/8/3/article-p347_002.xml?s=09