Bing cracks IAS Exam

 

Bing cracks IAS Exam

(3 min read)

This is a sequel to my earlier blog of 22nd Feb: ChatGPT fails IAS Exam. If you have read that piece, continue reading; if you haven’t, you may like to read it to get the context.

As you may be aware, Bing by Microsoft is a web search engine like Google, Yahoo, Baidu, DuckDuckGo, and others. But the new Bing is powered by AI (Microsoft is one of the major investors in Sam Altman’s OpenAI which has recently launched for the public its hottest AI tools – ChatGPT and DALL-E).

I signed up for the new Bing on February 19, 2023, and was put on the WAITLIST. On 27^th February I got mail from MS:

You’re in! Welcome to the new Bing! We are excited to give you access to an early preview of the new Bing – your AI-powered answer engine.

How could I wait any longer? I began firing my questions at Bing, and it performed well. Impressive, indeed!

Why not test Bing with a set of questions I had already run with ChatGPT? I tasked Bing to answer questions from Civil Services Preliminary Examination, 2022 conducted by UPSC.

Here is the result of my test:

Bing (B) Scores

General Studies Paper I:

·      Total Questions -100, Total Marks – 200, Time – 2 hours

·      Each correct answer has a value of 2 marks

·      Each incorrect answer has a penalty of (-) 0.66

·      Of the 5 Questions asked, B got 3 correct; it passed 2 Qs since it wasn’t sure of the answer.

·      Score: 6, no penalty marks

·      Total marks – 10, hence score is 60%

General Studies Paper II:

·      Total Questions -80, Total Marks – 200, Time – 2 hours

·      Each correct answer has a value of 2.5 marks

·      Each incorrect answer has a penalty of (-) 0.8

·      Of the 5 Questions asked, B got 3 correct, and 2 incorrect

·      Score: 7.5 – 1.6 = 5.9

·      Total Marks – 12.5, hence score is 47.2%

Result:

·      Cut off is based on GSP I score only. B’s score of  60% in GSP I is well above the cut-off of 45% to 47.5%.

·      B’s score of 47.2% in GSP II was above the minimum qualifying marks of 33%, and hence B passed.

Bing’s Question

After answering my questions, Bing asked me: Is this a question from an examination or a research project? I was candid. I told it that the Qs were from CSEP, and also that I had already run this test with ChatGPT.

Bing: How did ChatGPT fare? Are you happy with my answers? How would you rate Bing vis-à-vis ChatGPT?

Me: Inappropriate to jump to conclusions. I need to work more with both the tools to figure out their strengths and weaknesses.

Bing: That would be interesting. Best wishes for your research.

Which CSP Questions?

In my previous blog, I had mentioned that I had picked up a random sample of 5 Questions each from GSP I & GSP II.

Which questions had I selected? From SET – A (for each paper UPSC has 4 sets – A, B, C & D, to frustrate outside sources to remotely ‘help’ select candidates) for both the papers, the Qs I had picked up are:

GSP I – Q. 10, 31, 50, 71 & 90

GSP II – Q. 10, 31, 50, 71, & 72

If you wish to peruse those questions, you may go to upsc.gov.in where under ‘previous years’ question papers’ you would find downloadable PDF for both these papers for Civil Services (Preliminary) Examination, 2022.

Disclaimer

·      There is nothing called the IAS Exam. Reference is to Civil Services (Preliminary) Examination by UPSC.

·      For the ‘Test,’ random sample questions from CSP 2022 were selected by this author, and the sampling method could be substantially improved.

·      Bing is not designed to write examinations, even though it can solve problems and provide answers. Hence, this ‘Test’ is at best a what-if scenario.

AI World Champion?

Who is the AI World Champion, basis the ‘test’ that I administered to both? On the face of it, Bing cleared the Prelim Exam, and ChatGPT didn’t; but that doesn’t make Bing the winner.

Many users would surely explore the features and capabilities of both these AI tools in the coming days, and media would run several stories on the subject. Based on my quick and brief survey, it would be presumptuous of me to declare a winner. But I hope to use both these tools in the next few months and form my opinion about which one is better for my needs.

Boxers at AI World Championship. Suspense prevails as Referee is yet to announce the Winner.

Image Credit: Prasanna: Human (Concept) & Bing Image Creator: AI (Production)

Bard by Google is not far behind. I would try it when it is released for the public and maybe write a new blog: Bard, Bing, and ChatGPT (BBC, in alphabetical order!).  I hope I & B Ministry will not ban my innocent and innocuous blog since I have not titled it AI: The India Question!

***

Satya Nadella's comment

Dev Vardhan, my batchmate and friend (he had quit the IAS long ago; recently retired after 30 years with McKinsey, USA, and is now a freelance consultant at Chicago) shared my blog with Satya Nadella, CEO, Microsoft. Dev shared with me Satya's response:

“Thanks for sharing Dev.

Super fun! Maybe I can now write the exam and make my Dad proud!”

Mr. B.N. Yugandhar, Satya Nadella's father was a legendary IAS officer, and I had the privilege of meeting him when as Secretary, Rural Development, Government of India, he had visited Kanker and Bastar during 1984-85. I was, then, Additional Collector, Kanker.

Books by the Author

Short story collections:

Tell A Tale and Other Stories

Invisible Poet and Other Stories

The Mysterious Ladies and Other Stories

Fiction

Kathapur Tales

Essays

Pink Diamond and Other Essays

Self-Help

How To Be an Author in 7 Days: A Beginner’s Guide to Self- Publishing

Story books for children:

Cave of Joy: Anand Gufa

Two Tales, Three Tellers: A Fairytale & A Fable

Poetry

RIVER SONG and Other Poems

Songs of Soil: Selected Poems of an Unschooled Bard: Padma Shri Haladhar Nag

O Krishna, O Son! Yashoda’s Sublime Song of Sorrow

***

Note: Print copies of these books, except Pink Diamond and Other Essays, are available at amazon.in, notionpress.com, and flipkart.com; eBooks are available at Amazon Kindle.

***

ChatGPT fails IAS Exam

 ChatGPT fails IAS Exam

(4 min read)

IAS Exam!

The title of this piece is purposely sensational. To entice you to read further. Of course, you know, there is nothing called the IAS exam, but most people nevertheless use that name for the Civil Services Examination (CSE) conducted every year by UPSC for about 1000 Grade I & II posts.

You may have heard the story, possibly apocryphal, of parents of a young man from Bihar who visited the home of a potential bride for their ladla, and the girl’s father asked, ‘Aapke suputra ajkal kya kar rahein hain,’ and the boy’s father replied, with much pride, ‘Shriman Dilli mein IAS kar rahein hain.’ That was truthful, since the young man had joined a coaching institute, and was living in Old Rajinder Nagar in a one room shanty with a shared bathroom for the last several years.

ChatGPT passes tough Tests

ChatGPT, OpenAI’s chatbot, is making news ever since its public release in November 2022. It has bedazzled academics by passing several tough Tests including

·      Wharton Business School’s MBA exam (though with a Grade of B or B-)

·      University of Minnesota’s law exam

·      US Medical Licensing exam, and

·      Google’s interview for a Level 3 engineer for a coding position.

‘ChatGPT on its way to become a virtual doctor, lawyer, or business analyst’ is the title of a piece in a leading business daily.

ChatGPT takes CSE Prelim Exam

It was high time to test the AI tool for the UPSC Civil Services Examination - claimed by the IAS coaching institutes, those who wrote it and succeeded and those who didn’t, as the ‘toughest test’ in the world.

(Don’t believe that tall claim. At the end of this piece is a little Test you may like to take to see how easy it is. Simple, school level math!)

I ran a snap, sample test for the AI chatbot.

Well, ChatGPT didn’t write the Civil Services (Preliminary) Examination 2022 conducted by the UPSC; but I coerced it to take a Test. I picked up five random questions each from the General Studies Paper I and II, and asked ChatGPT to answer.

Why prolong the suspense? Here is the result:

ChatGPT (C) Scores

General Studies Paper I:

·      Total Questions -100, Total Marks – 200, Time – 2 hours

·      Each correct answer has a value of 2 marks

·      Each incorrect answer has a penalty of (-) 0.66

·      Of the 5 Questions asked, C got 4 correct, and 1 incorrect

·      Score: 8 – 0.66 = 7.34

·      Total marks – 10, hence score is 73.4%

General Studies Paper II:

·      Total Questions -80, Total Marks – 200, Time – 2 hours

·      Each correct answer has a value of 2.5 marks

·      Each incorrect answer has a penalty of (-) 0.8

·      Of the 5 Questions asked, C got 2 correct, and 3 incorrect

·      Score: 5 – 2.4 = 2.6

·      Total Marks – 12.5, hence score is 20.8%

Result:

·      Cut off is based on GSP I score only. C scored very high in GSP I, and may be among the highest scorers

·      However, minimum qualifying marks for GSP II is 33%, and hence C failed.

(A painting of ChatGPT robot sweating over a tough question. Painting by Prasanna + DALL-E)

CSE Preliminary 2022- Basic Stats

·      Cut-off: GSP I for General category candidates - 43.5% to 50% in in previous years; likely cut off for CSEP 2022 – 45% to 47.5% (90 to 95 marks in GSP I)

·      Number of candidates who applied: 11.52 lakhs

·      Number of candidates who appeared in Prelim exam: About 8 lakhs

·      Cleared Preliminary Exam: 13090 (13090/8 lakh=1.64%)

·      Cleared CSE Main Exam: 2529 (0.32%)

·      Total Vacancies: 861 (0.1%)

·      Chance of a candidate getting selected for Civil Services: 0.1%

Why did ChatGPT fail?*

·      No match for average human intelligence. Someone has observed that AI at present has the ‘intelligence’ far below that of a cockroach, and he may be right.

·      No strategy. A smart candidate would quickly scan the Qs from beginning to end, answer the Qs where she is sure she is correct, skip the doubtful ones (the golden rule where incorrect answer earns a penalty); GSP I – 50 correct answers out of 100 Qs is enough to clear the exam, GSP II – 27 correct answers out of 80 Qs earn the minimum qualifying marks. No need to try to solve all Qs. Very few candidates have the time to solve all Qs and get it right.

·      No knowledge of previous years’ Qs and cut off, and the criteria of cut off.

·      I have analysed ChatGPT’s incorrect answers, and have a good idea why it goofed up. Should I publish my research like the Stanford Professor, or wait for OpenAI to make me an attractive offer? Let me think over it.

Disclaimers:

1.     My method of random sampling could be problematic. A better method may be to pick up a Question from each category such as Indian Economy, Constitution and governance, Indian history, Environment, Technology, arithmetic, problem solving, comprehension, etc.

2.     A larger sample, say of 20 Qs from each Paper, may throw up a different result.

3.     C may perform better if asked to take the full test (GSP I -100 Qs, GSP II-80 Qs).

4.     I did not tell C about the penalty for an incorrect answer, and the option to pass a question. However, C sometimes says: I don’t know about that or I can’t answer that question. In my test, it answered all the questions readily, and explained the basis of each answer.

5.     C is handling millions of Q.s every minute including the test Q.s I threw at it. Of course, it has access to the neural network of supercomputers, and has been trained on 300 billion pages of text, more than a human can read, and remember in several lifetimes.

6.     Once trained on UPSC Question Papers of previous years, C’s performance may improve.

Take a Test

You are, no doubt, among the brightest and the best. To reassure yourself that your grey cells are still robust and agile, would you like to take a Snap Test of a single question which is not too difficult to solve? Correct answer earns you 2.5 marks, and incorrect answer has a penalty of (-) 0.8; you have 1.5 mins to solve it, and the option to answer or give it a pass, for which there is no penalty.

Your TIME (1.5 mins) starts now.

Question**

The digits 1 to 9 are arranged in three rows in such a way that each row contains three digits, and the number formed in the second row is twice the number formed in the first row; and the number formed in the third row is thrice the number formed in the first row. Repetition of digits is not allowed. If only three of the four digits 2, 3, 7 and 9 are allowed to use in the first row, how many such combinations are possible to be arranged in the three rows?

(a)  4 (b) 3 (c) 2 (d) 1

I guess you solved it, got it right, and within 1.5 mins. Congratulations, you scored 2.5 marks.

**A Question from Civil Services (Preliminary) Examination, 2022, General Studies Paper II, and ChatGPT gave an incorrect answer for this question!

Update*

I published this blog on 22/02/23. The next day, I asked ChatGPT:

Hi, did you fail IAS exam by UPSC, India?

C replied:

I'm sorry, but I cannot take the IAS exam or any other exam as I am an AI language model designed to assist with various tasks such as answering questions, generating text, and providing information on various topics. I have not taken any exams, nor am I capable of doing so. My purpose is to provide helpful responses and assist users in their queries to the best of my ability.

***

Note: 

• Author's profile is at http://amazon.com/author/pkdash
• Books by this author are available on Amazon.in, Kindle eBook, Flipkart, and Notion Press, Chennai.

***