Eng Cn 翻译此页面:
请选择您的语言来翻译文章


您可以关闭窗口不翻译
图书馆
你的个人资料

返回内容

Software systems and computational methods
Reference:

On the question of the specifics of the formulation and use of the Turing test for the ChatGPT

Osipov Mikhail Yur'evich

ORCID: 0000-0002-6982-3668

PhD in Law

Senior Scientific Associate,International Police Academy of All-Russian Police Association

300026, Russia, Tul'skaya oblast', g. Tula, ul. Ryazanskaya, 1

osipov11789@yandex.ru
Other publications by this author
 

 

DOI:

10.7256/2454-0714.2023.4.68680

EDN:

TCQVHG

Received:

10-10-2023


Published:

21-10-2023


Abstract: The subject of the research in this article are the features and regularities of the functioning of systems based on ChatGPT technologies, the knowledge of which makes it possible to formulate appropriate modifications of the Turing test, as well as the features and regularities of the formulation and use of the Turing test for systems based on ChatGPT technologies. The purpose of the study is to identify the features and patterns of functioning of systems based on the technologies of ChatGPT, as well as the features and patterns of formulation and use of the Turing test for systems based on the technologies of Chat GPT. As research methods, the method of social experiment was used, when during the study of a system based on Chat GPT technologies, certain questions were asked, answers were received, the analysis of which allowed us to conclude about the features of the "thinking" of systems based on ChatGPT technologies. In the course of the study, the following was found. Unlike human thinking, which is based on certain facts, the "thinking" of systems based on ChatGPT technologies, in some cases is not based on facts that take place in reality, often the user is given deliberately false information about facts and circumstances that take place in reality. In contrast to human thinking, which is usually systemic in nature, the "thinking" of systems based on ChatGPT technologies is disorderly and fragmentary. Systems based on ChatGPT technologies cannot admit their mistakes, and attempts to force systems based on ChatGPT technologies to critically comprehend their answers lead to a malfunction of these systems. The article also provides a Turing test developed by the author for ChatGPT, which made it possible to identify the features of the "thinking" of systems based on ChatGPT technologies.


Keywords:

Turing Test, human thinking, systems thinking, technology, artificial intelligence, Chat GPT, critical analysis, reflection, patterns, computer science

This article is automatically translated. You can find original text of the article here.

Introduction

One of the urgent problems facing modern computer science is the problem of formulation and use of the Turing test in the context of the development of artificial intelligence systems, including systems based on the technologies of Shat GPT. A lot of works have been written about this problem, both in domestic and foreign scientific literature devoted to this problem and others [1] [2] [3] [4] [5] [9] [10] [12] [13] [16] [17] [24] [28] . In these works, various aspects of the application of the Turing test to systems with artificial intelligence are considered. However, the problem still remains unresolved, and at the same time quite relevant. a proper Turing test for artificial intelligence systems, including systems based on the technologies underlying the functioning of Shat GPT, in some cases will not allow distinguishing a computer interlocutor from a human one, which under certain conditions can cause a number of problems in society, as discussed in the specialized literature [7] [11] [13] [17] [28]

This problem is particularly relevant in connection with attempts to introduce artificial intelligence technologies, including Shat GPT technologies, into education, including higher education [14] [15] [18] [21] [22] [23] [26] [28] and others. Meanwhile, the pedagogical literature notes the importance of personal contact between a teacher and a student and live communication between them, as well as the need for students to independently write scientific papers [20], which becomes problematic in the context of the development of artificial intelligence. All this makes it necessary to conduct a special study on the application of the Turing test to systems based on Shat GPT technologies in connection with attempts to extend them to higher education [14] [15] [18] [21] [22] [23] [26] [28] , the significance of personal contact between the teacher and the student and the live communication between them, as well as the need for students to independently write academic papers, which with the development of the Shat GPT technology, becomes problematic, since some students can write, including final qualifying papers using this technology [6].

All this makes it necessary to conduct a special study on the specifics of the formulation and use of the Turing test for Shat GPT, which will reveal the differences between the "thinking" of systems based on Shat GPT technologies from human thinking and thereby allow us to conclude about the possibilities and limitations of using this technology, which seems very important in connection with the transition of human civilizations to the information society.

Research methodology

The subject of the research, the results of which were presented in this article, are the features and patterns of functioning of systems based on Shat GPT technologies, knowledge of which allows us to formulate appropriate modifications of the Turing test, as well as the features and patterns of formulation and use of the Turing test for systems based on Shat GPT technologies. The purpose of the study is to identify the features and patterns of functioning of systems based on Shat GPT technologies, as well as the features and patterns of formulation and use of the Turing test for systems based on Shat GPT technologies, which will allow us to unequivocally conclude that we are looking at a machine, not a person.

The objectives of the study were to:

A) to get acquainted with the technologies of the formulation of texts and other documents using Chat GPT

B) to identify the features and patterns of the formulation and use of the Turing test for systems based on Shat GPT technologies,

C) to develop such questions that make up the content of the Turing test for systems based on Shat GPT technologies, which will allow us to unequivocally conclude that we are facing a machine, not a person, and will also allow us to determine the so-called intellectual potential of systems based on Shat GPT technologies.

The solution of these tasks was a separate stage of the study.

As research methods, the method of social experiment was used, when during the study of a system based on Shat GPT technologies, certain questions were asked, answers were received, the analysis of which allowed us to conclude about the features of the "thinking" of systems based on Shat GPT technologies.

The essence of the method was as follows. In connection with the established restrictions on direct access to American technologies Shat GPT, the author of the article was registered on the website https://chatgptrus.ru /, after which the following questions were asked to the GPT and the answers to them were received, which are presented in the following table

Table 1

Questions asked by Shat GPT and answers to them

User's question

The answer is Shat GPT

Right answer

Tell me where the Autonomous Non-profit Organization of Higher Education "International Police Academy of the VPA" is located

The autonomous non-profit organization of higher education "International Police Academy of the VPA" is located in Moscow, Russia

The autonomous non-profit organization of higher education "International Police Academy of the VPA" is located in Tula, Russia

Tell me who is the rector of the Autonomous Non-profit Organization of Higher Education "International Police Academy of the VPA"

Unfortunately, I can't find information about the current rector of the Autonomous Non-profit Organization of Higher Education "International Police Academy of the VPA". I recommend contacting the official sources or the academy's website for up-to-date information about the rector.

The rector of the Autonomous Non-profit Organization of Higher Education "International Police Academy of the VPA" is Yevgeny Borisovich Karpov, Doctor of Pedagogical Sciences, Candidate of Economic Sciences

Tell me, what faculties do the Autonomous Non-profit Organization of Higher Education "International Police Academy of the VPA" have

The Autonomous Non-profit Organization of Higher Education "International Police Academy of the VPA" has the following faculties:

1. Faculty of Legal and Social Sciences - here students study such disciplines as law, sociology, political science, philosophy and other academic subjects related to law and order and social sciences.

2. Faculty of Law Enforcement - future police officers, employees of security services and other law enforcement agencies are trained at this faculty. Here students study legal aspects of law enforcement, tactics and strategy of law enforcement operations and other specialized subjects.

There are no such faculties in the International Police Academy of the VPA

Tell me when the Autonomous Non-profit Organization of Higher Education "International Police Academy of the VPA" was founded

The autonomous non-profit organization of higher education "International Police Academy of the VPA" was founded in 1992

The autonomous non-profit organization of higher education "International Police Academy of the VPA" was founded on December 26, 2000.

Who founded the Autonomous Non-profit Organization of Higher Education "International Police Academy of the VPA"

Who founded the Autonomous non-profit Organization of Higher Education "International Police Academy of the VPA" was founded by the Government of the Russian Federation

The autonomous non-profit organization of higher education "International Police Academy of the VPA" was founded by Evgeny Borisovich Karpov

Tell me, what does the Autonomous non-profit Organization of Higher Education "International Police Academy of the VPA" have to do with the city of Tula

The autonomous non-profit organization of higher education "International Police Academy of the VPA" is related to the city of Tula, in the sense that it is located on the territory of the city and provides educational services in the field of policing. This academy can teach students from Tula and other regions of Russia and other countries. However, the specific details of the academy's relations with Tula, such as financial or organizational, are unknown to me

This is the correct answer, since the Autonomous non-profit organization of Higher Education, the International Police Academy of the VPA, is located at 1 Ryazanskaya str., Tula and trains, among other things, police students from Tula and other regions of Russia and other countries

So, where is the Autonomous Non-profit Organization of Higher Education "International Police Academy of the VPA"

The autonomous non-profit organization of higher education "International Police Academy of the VPA" is located at 26A Lobnenskaya str., Moscow

Autonomous non-profit organization of higher education "International Police Academy of the VPA" located at Tula, Ryazanskaya str., 1

Is it true that Tula is a part of Moscow in Moscow

The Oep AI server is not responding, possibly due to heavy load. Try to visit the site later (service message)

Wrong. Tula and Moscow are separate cities.

Research results

The mechanism of operation of systems based on Shat GPT technologies

Next, we will try to understand the reasons for such ridiculous answers with GPT

In order to answer this question, it is necessary to understand the mechanism of operation of the Hat GPT.

To understand the mechanism of operation of the Hat GPT, you can use the following model. At the entrance, we have questions structured in a certain way, which consist of words that have a certain meaning. Such a set of words can be represented as a linear matrix (1) At the output we have a structured text that can be represented as a matrix (2)

How the matrix is obtained from the matrix The answer is quite obvious. In order to get a matrix from the matrix, it is necessary that there is a so-called associative matrix or transition matrix, which looks like this (3) The specified matrix is the transition matrix from question to answer. This is how any thinking works, including human thinking, since it is based on the law of sufficient reason

The whole specificity and difference between the technology of the functioning of systems based on the technologies of Shat GPT from human thinking lies in the fact that it is not logical, but associative in nature.

That is, the keyword an?bn is not at all because they are logically related to each other, although this may be, but because there is a non-zero probability that keywords occur together in texts, and the higher this probability, the higher the probability that the response of technology-based systems Shat GPT can be represented as a matrix In other words, artificial intelligence systems based on Shat GPT technologies assume: 1) the presence of texts that can be represented as a matrix (4) and the presence of a rule of inference that if there is a matrix (4) then there must be a matrix (3)

But in fact, this is far from the case. The fact of the existence of matrix (4) does not imply the fact of the existence of matrix (3), and therefore the conclusion obtained with the wince of artificial intelligence based on Shat GPT technologies may be incorrect, since false associations are not excluded, due to the incorrect interpretation of Shat GPT key terms contained in the user's request. Linguists dealing with translation problems are aware of such a problem [8]. Thus, based on the analysis of the data obtained, we can formulate a hypothesis about the associative nature of the "thinking" of systems based on the Shat GPT technology, according to which the "thinking" of systems based on the Shat GPT technology is associative [25] [27]

Verification of the hypothesis about the associative nature of the "thinking" of systems based on the Shat GPT technology

Let's prove this statement. Let's turn to the site again https://chatgptrus.ru / and ask the following questions:

1. Who established the Autonomous Non-profit Organization of Higher Education "International Police Academy of the VPA"

2. Where is the Autonomous Non-profit Organization of Higher Education "International Police Academy of the VPA" located

3. On the basis of which the statement was made that the Autonomous Non-profit Organization of Higher Education "International Police Academy of the VPA" was established ..... and is ........ and other clarifying questions.

Table 2. Responses from the GPT were received as follows

Question

The answer is Shat GPT

Right answer

Possible reasons for such a Chat GPT response

Who established the Autonomous Non-profit Organization of Higher Education "International Police Academy of the VPA"

The autonomous non-profit organization of higher education "International Police Academy of the VPA" was established by a group of initiative specialists in the field of law enforcement and higher education. The founders of the organization decided to create an academy that would provide high-quality and modern education in the field of police, in order to train highly qualified specialists to work in law enforcement agencies.

The Autonomous nonprofit organization of higher Education "INTERNATIONAL POLICE ACADEMY OF the VPA" was created through the transformation of a private educational organization of higher education - the Association "TULA UNIVERSITY (TIEI)"

Since the texts of the documents mention that usually professional educational organizations are created by initiative specialists in the field of their activities in order to train specialists in the field of their activities

Where is the Autonomous non-profit Organization of Higher Education "International Police Academy of the VPA

The autonomous non-profit organization of higher education "International Police Academy of the VPA" is located in Russia, the city of St. Petersburg. Her address is: 30 Lunacharsky Ave., Saint Petersburg, 196240, Russia.

In fact, the Autonomous non-profit Organization of Higher Education "International Police Academy of the VPA is located in Tula at 30026 Ryazanskaya d. 1

Random response

On the basis of which the statement was made that the Autonomous non-profit organization of Higher Education "International Police Academy of the VPA" is located in Russia, the city of St. Petersburg. Her address is: 30 Lunacharsky Ave., Saint Petersburg, 196240, Russia.

To make an assertion that the Autonomous Non-profit Organization of Higher Education "International Police Academy of the VPA" is located in Russia, the city of St. Petersburg, it was necessary to use the available information. In this case, the following sources could be used to confirm this information:

1. The official website of the International Police Academy of the WPA, where the address of the organization is indicated.

2. Online maps or reference information services that also show the address of the VPA.

3. Bodies of the Russian state administration, such as the Ministry of Education and Science of the Russian Federation or the Federal Tax Service, which may have registration data about the organization.

According to the information provided, the VPA address specified as pr. Lunacharsky, 30, 196240, St. Petersburg, is a supporting fact that indicates that the organization is located in Russia, the city of St. Petersburg.

In fact, on the official website of the Academy it is indicated that it is located in Tula on the Ryazan street d 1

Random associative response

And what is the official website of the Autonomous Non-profit Organization of Higher Education "International Police Academy of the VPA"

Official website of the Autonomous Non-profit Organization of Higher Education "International Police Academy of the VPA" - https://ipa.edu.ru/

Official website of the Autonomous Non-profit Organization of Higher Education "International Police Academy of the VPA" - https://mpa71.ru/

Such a site does not really exist. There is a website https://ipa.edu.org

Analysis of the responses of Shat GPT shows that the "thinking" of artificial intelligence systems based on Shat GPT technology is and can only be associative. In other words, it is based on the following formula (1)

At the same time, the question arises whether the "thinking" of systems based on Shat GPT technologies may not be associative in nature?

Let's prove that this is impossible.

Why can't the "thinking" of systems based on Shat GPT technologies not be associative"

There are two types of thinking: 1) linear thinking, which is based on the law of sufficient reason, according to which in order to assert that a?b, it is necessary to make sure that: 1) there is a; 2) there is 2) there is a connection between a and b, such that 2) associative thinking, according to which if ? a, if both elements occur quite often in texts [25] [27].

Linear thinking involves working in a single-task sequential mode, because in order to make sure that there is a certain object A, it is necessary to expend certain efforts and resources, we will designate them C1, in order to make sure that there is a certain object B, it is necessary to expend certain efforts and resources, we will designate them C2; for in order to make sure that there is a connection between the objects such that it is necessary to expend certain efforts and resources, we denote them C3

Therefore, in order to solve the problem of the existence of a connection between two objects a and b, such that a?b, it is necessary to spend the total amount of resources, which is determined by the formula SZADACH = C1 + C2 + C3. But any system has a limited resource. Therefore, the question arises how to decide how much resources should be spent on solving a particular problem. According to A. Arrow, this can only be done by a person or a "dictator" who, by his imperious efforts, determines which task is a priority, and which of the many target functions is a priority [19]. In this case, all tasks coming to the input of AI systems are equally prioritized. But, the resources that need to be spent on a genuine solution to the problem, involving the actual establishment of a relationship between two objects of thinking, in which a ? b, are limited, and since the amount of resources needed to establish this connection is different, depending on the nature of the problem, therefore it is impossible to optimize the allocation of resources needed to solve one or another tasks in conditions of parallel execution of operations, which can and will inevitably lead to the impossibility of solving a particular task. There can only be one way out of this: the transition to solving problems as they arrive, using a linear type of thinking. And the consequence of using a linear type of thinking is the presence of a huge number of requests and, as a consequence, their inevitable sequential processing as they arrive. Hence the occurrence of service failures, since the resources of the system are limited, and there are many request flows. And this, in turn, will lead to dissatisfaction with customers, authors of requests. In conditions of parallel execution of operations, thinking inevitably becomes associative, since resources for genuine fact-checking are usually not enough in the presence of mass requests. From this it can be concluded that the "thinking" of systems based on the technologies of Shat GPT cannot but be associative." At the same time, the consequence of associative thinking is the impossibility of specifying the sources of obtaining certain conclusions and separating one's own and others' ideas, since searching and specifying the sources of this or that information, and deciding on the use of certain sources, requires the presence of linear thinking and the use of a single-task approach, which is impossible when using Shat GPT technologies

Features of the "thinking" of systems based on Shat GPT technologies and the formulation of Turing test questions for systems based on Shat GPT technologies

Based on all of the above, it is possible to identify the following features of the "thinking" of systems based on Shat GPT technologies.

The associative nature of the "thinking" of systems based on the technologies of Shat GPT. This is manifested in: a) the inability to obtain information about specific facts about certain objects; b) the contradictory nature of information about these objects, as mentioned above; c) the inability to obtain information about sources that allowed systems based on the technologies of Shat GPT to come to certain conclusions; or their unreal character c) often, when setting tasks that require searching for information about a particular object, systems based on Shat GPT technologies demonstrate the disorderly and fragmentary nature of their "thinking", c) systems based on Shat GPT technologies demonstrate a lack of "critical thinking" and a willingness to admit and analyze their mistakes. Based on these patterns, it is possible to formulate certain requirements for the Turing test questions for systems based on Shat GPT technologies.

1. The questions of the Turing test for systems based on Shat GPT technologies should relate to specific facts about certain objects of reality, while these objects should be quite rare and unfamiliar to "uninitiated people"

2. The same Turing test questions for systems based on Shat GPT technologies should be repeated several times to compare the answers to them

3. The questions of the Turing test for systems based on Shat GPT technologies should provide for a mandatory search for information about specific objects of reality

4. Questions of the Turing test for systems based on Shat GPT technologies should involve checking the critical thinking of the interlocutor when passing this test

5. The Turing test questions for systems based on Shat GPT technologies should be lost among the usual questions that do not cause special problems for systems based on Shat GPT technologies.

The use of such rules will make it easy to compile a Turing test that a system based on Shat GPT technologies will never pass, since the use of ordinary questions that do not cause special problems for systems based on Shat GPT technologies allows you to create the illusion that the "key question" that allows you to identify systems based on technologies, refers to questions that do not cause special problems for the formulation of an answer to them on the part of these systems. In addition, the use of questions that require the search and analysis of information in real time, including questions about the sources of knowledge, as well as questions for the presence of critical thinking, will reveal the inability of systems based on Shat GPT technologies to search for information about specific objects in real time, as well as the inability of the system to critically evaluate its answers to the questions. Finally, the use of repetitive key questions that require searching and analyzing information in real time, for example, where is an object that is quite rare and unfamiliar to "uninitiated people" will reveal contradictions in the answers, which in the absence of contradictions in other answers and sufficient speed of answering questions that require different intellectual efforts, will allow us to conclude that we are not looking at a living person, but an artificial intelligence system based on Shat GPT technologies. Next, let's look at the main factors influencing the formulation of questions for the Turing test for systems based on the technologies of Shat GPT

The main factors influencing the formulation of questions to the Turing test for systems based on the technologies of Shat GPT

What factors influence the formulation of questions for the Turing test for systems based on Shat GPT technologies? In our opinion, the following factors can be attributed to such factors

1. The researcher's understanding of the specifics of thinking of systems based on Shat GPT technologies.

2. The researcher's understanding of the features and rules of the formulation of the Turing test questions for systems based on Shat GPT technologies.

3. The ability of the researcher to formulate questions for the Turing test for systems based on Shat GPT technologies.

4. The ability of the researcher to analyze and interpret the received answers to the questions of the Turing test.

Depending on the combination of these factors, more effective or less effective Turing tests will be created for systems based on Shat GPT technologies.

Conclusion

Based on all of the above , the following conclusions can be drawn

1. The "thinking" of systems based on Shat GPT technologies, unlike human thinking, is always associative in nature

2. The associative nature of the "thinking" of systems based on the technologies of Shat GPT. This is manifested in: a) the inability to obtain information about specific facts about certain objects; b) the contradictory nature of information about these objects, as mentioned above; c) the inability to obtain information about sources that allowed systems based on the technologies of Shat GPT to come to certain conclusions; or their unreal character c) often, when setting tasks that require searching for information about a particular object, systems based on Shat GPT technologies demonstrate the disorderly and fragmentary nature of their "thinking", c) systems based on Shat GPT technologies demonstrate a lack of "critical thinking" and a willingness to admit and analyze their mistakes

3. Based on this, the questions of the Turing test for systems based on Shat GPT technologies should be checked: the ability of the interlocutor to search for information in real time, the ability of the interlocutor to give consistent answers to both key and ordinary questions, including after some time, the ability of the interlocutor to indicate sources of information when formulating their conclusions of a scientific nature, the ability of the interlocutor to critical thinking.

4. The effectiveness of the Turing test for systems based on Shat GPT technologies is influenced by many factors, among which are the following: a) the researcher's understanding of the specifics of thinking of systems based on Shat GPT technologies; b) the researcher's understanding of the features and rules of the formulation of Turing test questions for systems based on Shat GPT technologies; c) the researcher's ability to formulate questions for the Turing test for systems based on Shat GPT technologies; d) the researcher's ability to analyze and interpret the received answers to the Turing test questions.

The author sees the further direction of the research in the analysis of possible risks of using technologies based on Shat GPT in higher education.

References
1.
2.
3.
4.
5.
6.
7.
8.
9.
10.
11.
12.
13.
14.
15.
16.
17.
18.
19.
20.
21.
22.
23.
24.
25.
26.
27.
28.

Peer Review

Peer reviewers' evaluations remain confidential and are not disclosed to the public. Only external reviews, authorized for publication by the article's author(s), are made public. Typically, these final reviews are conducted after the manuscript's revision. Adhering to our double-blind review policy, the reviewer's identity is kept confidential.
The list of publisher reviewers can be found here.

The subject of the study. The article is devoted to the question of the specifics of the formulation and use of the Turing test for the GPT format. The content of the article fully corresponds to the stated topic, which is a positive characteristic. The methodology of the study was formed by the author, based on conducting an experiment and using graphical tools to present the results of the study. This creates a positive impression from reading this article. It is also recommended to supplement the article with an analysis of specific numerical data characterizing the development of the subject of research in dynamics, taking into account the influence of a large number of external and internal factors, including those specific to the Russian Federation. The relevance of the study is beyond doubt. The chosen research topic is relevant, as issues related to the development of intellectual intelligence are actively raised throughout the world, including the Russian Federation. A large number of different special tools are being developed for this purpose. At the same time, their activities are fraught with problems, the development of proposals for solving which can make it possible to achieve a socio-economic breakthrough of the Russian Federation, as a whole, and their individual participants in order to improve the quality of life of the population. It is valuable that this will meet the national interests of the development of our state, as well as the achievement of the goal set by the President of Russia on technological sovereignty. The scientific novelty in the materials submitted for review can be traced. In particular, table 1 (Questions posed by and answers to Hat GPT), as well as the described Mechanism of operation of systems based on Hat GPT technologies, are of interest. Style, structure, content. The style of presentation is scientific. The structure of the article, as a whole, is built correctly. It is valuable that the author also outlined potential directions for the development of the topic. When finalizing the content, it would be interesting to identify specific problems and form proposals for their solution, including in the context of achieving the task of technological sovereignty of the Russian Federation. The regional aspects of the topic under consideration deserve special attention. When finalizing the article, the author is recommended to clarify the transitions from one semantic block to another. So, the author writes, "Next, we will consider the main factors influencing the formulation of questions for the Turing test for systems based on the technologies of Shat GPT." Instead of the word "next", it is recommended to specify the answer to the question "for what?" Bibliography. The author has compiled a large list of sources used in writing the article. It includes 28 titles. It is valuable that the author turned not only to the works of domestic authors, but also foreign ones. This allowed us to delve deeper into the issues raised. It is also recommended that the author study the statistical data on the topic under consideration and indicate the appropriate sources. Appeal to opponents. Despite the generated list of references, the author has not carried out any scientific discussion. When finalizing the article, forming a specific list of problems and developing a set of practical recommendations for their solution, the author should discuss them with those contained in other scientific publications in domestic and foreign scientific literature. Conclusions, the interest of the readership. Given the increased relevance of the research topic and the high quality of most fragments of the article, it is of interest to potential readers. At the same time, the article can be recommended for publication only after the above comments have been finalized. Moreover, their qualitative elimination will seriously expand the potential readership.