A Question We Can Measure, but Not Overclaim
Since the release of ChatGPT, many researchers have wondered whether scientific writing has started to change. The question is tempting because it feels observable: certain words seem to appear more often, abstracts sometimes sound more polished, and phrases associated with large language models have become part of everyday discussion about research.
But a feeling is not a measurement. This pilot study asks a narrower question: if we look at a large set of psychology abstracts over time, can we detect measurable changes in their language distributions during the period when LLMs became widely adopted?
The answer is yes, with an important caveat. The corpus shows measurable lexical and coarse structural shifts between 2020-2022 and 2023-2025. But the results do not prove that LLMs caused those shifts, and the largest adjacent-year change does not appear exactly at the 2022-to-2023 boundary.
The Dataset
The analysis uses 112,689 English-language psychology abstracts from OpenAlex, covering publication years 2020 to 2025. Records from 2026 were collected during the broader workflow, but they were excluded from the reported KLD analysis because 2026 was only partially available at the time of writing.
The study focuses on abstracts because they are compact, comparable units of scientific communication. They also carry a lot of information about how a field frames its research: what topics are salient, what methods are foregrounded, and how results are presented.
The corpus was cleaned to remove obvious noise such as empty abstracts, duplicate records, HTML remnants, URLs, very short texts, and suspicious non-abstract content. The goal was not to create a perfect representation of psychology, but to create a reproducible snapshot that can support a first round of descriptive analysis.
What KLD Is Doing Here
The main tool is Kullback-Leibler divergence, or KLD. In plain terms, KLD compares two probability distributions and asks how different they are.
For this project, each year of abstracts can be treated as a kind of language fingerprint. One fingerprint might describe the relative frequency of individual words. Another might describe the relative frequency of simple part-of-speech patterns. KLD then compares one year with another, or one period with another, and gives a number for how much the later distribution differs from the earlier one.
This matters because the analysis does not begin by choosing a few suspicious words and looking for them. Instead, it first asks whether the whole distribution moved, and only then asks which words or patterns contributed most to that movement.
The First Result: Change Exists, but Not as a Simple 2023 Break
The first layer of analysis looks at unigrams, which are individual word forms. This is the most direct layer because it captures changes in topics, common academic vocabulary, and possible style words.
The pre/post comparison between 2020-2022 and 2023-2025 shows a detectable unigram difference, with KLD = 0.0256. That supports the idea that the later period is linguistically different from the earlier period.
But the adjacent-year comparisons complicate the simple story. If ChatGPT had produced an immediate, sharp break in psychology abstract language, we might expect the 2022-to-2023 comparison to be the largest. It is not. In this sample, the largest adjacent-year unigram shift is 2024-to-2025, with KLD = 0.0257.
That does not rule out LLM influence. Scientific publication has delays, adoption takes time, and research communities change gradually. But it does mean the result should be described as a distributional shift during the LLM adoption period, not as proof of a sudden 2023 rupture.
Which Words Contributed?
KLD can be decomposed into feature-level contributions. That lets us ask which words made the later period most different from the earlier period.
The top contributors are not one kind of signal. They are a mixture.
Some words look like topic or corpus signals. The word ai became much more frequent in the later period, rising from 736 occurrences in 2020-2022 to 3,861 in 2023-2025. That likely reflects the growth of AI-related research within psychology. Words such as sleep, emotional, and interventions also look closer to shifts in psychology topics or research areas than to LLM writing style.
Other words are more ambiguous. Terms such as findings, enhance, insights, and significant may be related to changes in academic reporting style. They also overlap with the kind of polished, contribution-oriented language often discussed in relation to LLM-assisted writing. But the study does not treat them as proof. A word can become more common for many reasons: a topic shift, journal conventions, database composition, human style trends, or LLM assistance.
One of the most important contributors is and, a very common function word. That is mathematically meaningful but interpretively delicate. Very frequent words can contribute strongly to distributional differences even when their change is small. They should not be read as direct LLM markers without a separate function-word analysis.
Time Trajectories Matter
A top contributor is easier to interpret when we place it back onto a timeline. If a word spikes in only one year, it may reflect sampling noise or a temporary corpus change. If it rises steadily, it may indicate a more durable trend.
The word ai rises from 0.43 per 10,000 tokens in 2020 to 5.24 in 2025. The word findings rises from 12.09 to 21.18. The word enhance rises from 1.55 to 5.53. Topic-related terms such as emotional also increase.
This is why the study treats the signal as mixed. The later period is different, but the difference contains several overlapping processes: more AI-related research, shifts in psychology topics, possible changes in academic style, and ordinary corpus composition effects.
A Second Layer: POS Trigrams
The second layer uses part-of-speech trigrams. These are short sequences of grammatical labels such as adjective-noun-noun or verb-adjective-noun. They are not full syntax. They cannot tell us exactly how sentences are structured. But they can provide a rough check on whether the change is only lexical or whether broader patterning also moves.
The POS trigram signal is smaller than the unigram signal. The pre/post POS trigram KLD is 0.0092, compared with 0.0256 for unigrams. The strongest adjacent-year POS trigram shift also appears in 2024-to-2025.
Several high-contribution patterns involve adjective and noun-heavy structures, such as VERB ADJ NOUN, ADJ NOUN NOUN, and ADJ ADJ NOUN. These may suggest later abstracts use more descriptive or compressed academic phrasing. But this layer is deliberately treated as secondary evidence, not as proof that scientific grammar changed in a precise way.
What the Study Can and Cannot Say
The strongest conclusion is modest: OpenAlex psychology abstracts show measurable lexical and minimal grammatical distribution shifts between 2020-2022 and 2023-2025.
The strongest adjacent-year changes occur between 2024 and 2025 rather than exactly between 2022 and 2023. The unigram results show a mixture of AI-topic growth, psychology topic shifts, high-frequency function-word sensitivity, and possible academic style-word changes. The POS trigram results provide weaker secondary evidence for broader structural patterning.
What the study cannot say is just as important. It cannot identify which abstracts were written with LLM assistance. It cannot prove that LLMs caused the observed changes. It also cannot fully separate writing style from topic changes without further checks.
The next useful steps are concrete: inspect real abstract contexts for the top contributing words, run robustness checks with function words treated separately, and control for psychology subfields, journals, or article types where possible. Once complete 2026 data is available, the analysis should also be rerun with a full-year sample.
For now, the study is best understood as a reproducible early map. It shows that something measurable is happening in the language of psychology abstracts during the LLM adoption period. The more difficult task is to determine how much of that signal comes from AI writing assistance, how much comes from AI becoming a research topic, and how much comes from the normal evolution of scientific communication.