Similiscopy, leveraging large language model hallucinations to verify contextual similarity

Abstract: This paper proposes "similiscopy" a novel binary (presence/absence) scientific technique that would belong to a broader method of content analysis. It is defined as a technique with binary outcomes that leverages hallucinations made by generative language models in order to demonstrate the similarity of contexts, at least on a statistical-linguistic level. I propose the use thereof to resolve apophenia in the early stages of research, as it could be used to gauge similarities between the two contexts without relying on subjective interpretation. One experiment was done regarding a roleplaying language model that generated aspects of a high-fantasy language from the wrong worlds. Another was done with a language model that has been confusing real languages. These two similar contexts drove large language models to hallucinate (in this particular case, when a generative language model assigns aspects of one pattern to another) and, thus, were successfully used as a form of unusual verification of the technique.   1. Introduction   Large language models fundamentally operate as statistical models of language. Generative models like this work by unidirectionally generating the next token in a sequence [1], where tokens can include words, parts of words, punctuation, and other symbols. ChatGPT-3.5, ChatGPT-4, and character.ai are examples of such systems. The tokens are generated based on context, which in this case is everything that precedes the token being generated. In other words, the next token is stochastically predicted via a statistical approach, inferred from the rest of the context. [2] Theoretically, similarity between two contexts should be detectable via hallucinations if a token or a set thereof that does not belong to one context ends up being generated therein. This would only work if a set of tokens was known to belong to another context. In other words, it should be  possible to construct a content analysis method that relies on hallucinations to gauge statistical-linguistic similarity between the two contexts. The goal of this paper is to verify or falsify a potential technique of leveraging hallucinations made by generative language models where data that belongs to one context gets assigned to another to provide evidence for a similarity between them. To do so, contexts already known to be similar will be used to check the technique to see if the generative large language model would confuse the two. The hypothetical technique is named similiscopy, and it is defined as a technique with binary outcomes that leverages hallucinations made by a generative large language model in order to demonstrate the similarity of contexts, at least on a statistical-linguistic level. The goal of the technique is to serve as an early apophenia check during the generation of a hypothesis. Apophenia is a tendency to perceive nonexistent patterns [3].   2. Explaining the languages for ChatGPT3.5 language recognition testing   Serbian, Montenegrin, Croatian, and Bosnian languages are mutually intelligible. Some scholars maintain that it is one polycentric language (Serbo-Croatian). [4] The Serbian language uses two scripts, Latin and Cyrillic [5], whereas Croatian uses only the Latin script. In the Bosnian language, Cyrillic script is acceptable [6], even if rarely used. Montenegrin is a small language in comparison to the previously mentioned three, and there are controversies regarding the standardization thereof. [7] For the most part, the Cyrillic script in Serbo-Croatian is mostly present in Serbian, both because they use the Cyrillic script most often and because they are the most spoken Serbo-Croatian language/variation, consequently it is only reasonable to assume that most of the training data for large language models in the Cyrillic script in Serbo-Croatian was in Serbian. This yields the prediction that ChatGPT3.5 would detect Bosnian text as Serbian if it were in the Cyrillic script in a manner that is inconsistent with randomness. The prediction from this passage is termed hypothesis 1, or H1.   3. Explaining Witcher and Tolkien universes languages for the purposes of the study   Tolkien was a renowned linguist. His work on worldbuilding includes building entire languages, such as the elven languages of Quenya and Sindarin that are still subjects of scholarly work [8]. The Witcher universe, on the other hand, included a language spoken by the elves that was only superficially present, the language itself is not known to have ever been fully constructed. This yields a qualitative prediction for the method tested. That Character AI, a large language model specialized in roleplaying, would confuse Tolkien with the Witcher universe by assigning aspects of the Tolkien universe when it comes to language to an elf from the Witcher universe in a replicable manner. The prediction from this passage is termed hypothesis 2, or H2. There was anecdotal evidence for the hypothesis before it was made, but it lacked reproducibility.   4. Methods   For hypothesis H1, Bosnian Wikipedia’s, the second page was chosen because the first passage thereof contained no words that would be inherently written in the Latin script. Seeing as Montenegrin is a small language and unlikely to be detected, and Croatian does not use the Cyrillic script, a two-tailed binomial probability of success of 0,5 was chosen for text converted into the Cyrillic script. The first passage was converted into the Cyrillic script, and ChatGPT3.5 was asked to recognize the language. The process was repeated ten times. Later, for the refinement of H1, the fourth page of Wikipedia was chosen as it did not contain words inherently written in the Latin script. The text in the Latin script was used for the control group. For both scripts, the response was regenerated ten times. Instead of a binomial test, two-tailed Fisher’s exact test was used.  To test H2, a preexisting template of Ida Emean aep Sivney (a Witcher universe elf) was used. The template was mostly empty, and there was nothing in the character template explicitly indicating that she was an elf. She, however, used words from the Elder Speech (a language of one elven culture in Witcher universe) in the pre-written dialogue. She was repeatedly asked in different ways what language that was until the prompt could reproducibly cause cross-over hallucinations. The final prompt was used ten times. Later, for the purpose of replication, the nearly empty template of Franceska Findabair (another Witcher universe elf) was also used to see if the chosen prompt caused the same response. There was no point to statistical testing due to a lack of inconsistency between the null hypothesis and H2 (randomness would not be at odds with H2, as the response that crosses a universe should not happen at all), and the probability of success in the case of randomness was impossible to estimate, and there was no material for a control group. Each and every single exact prompt is present in the data file for both hypotheses.   5. Results   For H1, when the text from Bosnian Wikipedia was remade into the Cyrilic script, ChatGPT3.5 consistently mistook it for Serbian. This process was repeated ten out of ten times. Two-tailed binomial, p = 0.001953125 For the refinement of the H1 ChatGPT3.5, it mistook the language for Croatian instead of Bosnian (somewhat surprisingly, but still very much in line with the expectation of the technique) ten out of ten times. Once the text was turned into the Cyrillic script 10 out of 10 times, it was recognized as Serbian. Seeing as the output remained binary, Fisher’s exact test was still used (p < 0.0001). For H2, after multiple trials and errors, a prompt that reproducibly created cross world hallucinations without referencing other high fantasies was: „Hello lady, I know this is an elven language, but which one, if I may ask?“ The correct answer to that would be „Elder Speech“ or „Hen llinge." Unexpectedly, two responses indicated high-elven (possibly from the Warcraft universe). Three out of ten times, it indicated Sindarin. Once it generated „silven-folk,“. Seeing as large language models rely on tokens, rather than the words themselves, it could be „Silvan," wood elves in the Tolkien universe, rather than „Silven“. After testing the prompt on the template of Franceska Findabair, four out of five responses indicated that she spoke Sindarin.   6. Discussion   6.1. Limitations of the method Like a significant part of the scientific methods and techniques (such as statistical significance tests), this method appears to be much better at verification than falsification. Falsification could only be theoretically achieved through induction.    6.2. Potential for further improvement   Hypothetically, one could leverage temperature, training data, size, and/or quantization of the AI, as well as check how often these mistakes occur, so as to create a quantitative tool for measuring context similarity in a manner that does not depend on subjective judgements. This would enable the measurement of not only whether the contexts are similar; rather, it could also measure to what degree the contexts are similar.   6.3. The failure to utilize ChatGPT scripts for generation of different dialects   There was an idea to test ChatGPT3.5 and ChatGPT4 on the generation of non-Serbian words from Serbo-Croatian when Cyrillic and Latin were used. On April 26, the quality of the output in Serbian fell to the degree where nonexistent words were being generated in the Cyrillic script. Preventing the completion of the experiment.   7. Conclusion   This technique drives the behavior of the AI as one theoretically would expect. It could be useful to resolve apophenia in the early stages of research. References Song, X., Zhang, Y., & Liu, Z. (2023). Bidirectional Generative Pre-training for Improving Time Series Representation Learning. arXiv preprint arXiv:2402.09558. doi.org Naveed, H., Khan, A. U., Qiu, S., Saqib, M., Anwar, S., Usman, M., Akhtar, N., Barnes, N., & Mian, A. (2023). A comprehensive overview of large language models. arXiv. doi.org apophenia. (n.d.). In Merriam-Webster Dictionary. merriam-webster.com The Economist. (2017, April 10). Is Serbo-Croatian a language? The Economist. economist.com Alexander, R., & Elias-Bursać, E. (2010). Bosnian, Croatian, Serbian, a Textbook: With Exercises and Basic Grammar. doi.org. Jovanović, S. (2018). The Discursive Creation of the ‘Montenegrin Language’ and Montenegrin Linguistic Nationalism in the 21st Century. Acta Universitatis Sapientiae, European and Regional Studies, 13, 67-86. LibGuides: Bosnian/Croatian/Serbian Languages: Bosnian Beregond, A. S. (2015). Arda Philology 5: Proceedings of the Fifth International Conference on J.R.R. Tolkien’s Invented Languages, Omentielva Lempea, Helsinki, 8-11 August 2013. Arda. Affiliation At the time of the writing and submission, the author was a PhD candidate at Megatrend University, Faculty of Biofarming, and an analogue of CEO (in the anglosphere) that in Serbia is titled „direktor“ of ecoera doo, Belgrade. The word „doo“ has roughly the same meaning as „LLC“ in the anglosphere or GmbH in Germany, Austria and Switzerland.