Update (2023-05-25): I've added an additional test with Bard from Google.
ChatGPT is all the rage these days, so I thought I'd do a simple but effective (just kidding) test comparing the two by asking about a bug in Skyrim Anniversary Edition.
Now bear in mind that this test considerably favours ChatGPT as Google is based on vague keywords whereas ChatGPT is able to understand the nuances of the language.
The Question
I asked the exact same following question to ChatGPT, Google, you.com, Bing Chat, and Bard:
In skyrim, my game buggged out at the bloodchill manor quest. The vampire was too powerful and I ran. When I returned later, the door was locked, and now I can't finish the quest and the bloody orc keeps following me. How do I get rid of the orc?
Yes, that is an (unintentional) typo in the question, which I didn't realize at the time.
ChatGPT
ChatGPT gave a very relevant answer and provided not one, but three possible solutions. Out of the three, number 2 seemed like a really plausible solution. Number 3 is not valid because the orc is not killable, at least not when the quest is still active. Number 1 seemed like it could work, but I don't know if it got the prid correct.
So I got into the game, and put the console commands to the test. And lo and behold, the prid was incorrect. So now, I'm wondering if the second solution would work. I put the commands in the console, and voila, the quest ID was also incorrect.
So none of the solutions from ChatGPT worked.
Google picked up on the typo, as it does. And returned only 409 results. This is considered low as it normally has millions of results. Still, we only care about the top 3, because who has time for number 4?
Out of the 3, number 3 is invalid, as it's about game crashing which is not the bug I described.
Number 1 is a reddit post, which describes exactly the same bug. The reddit users provided some solutions, but none of them worked.
Number 2 is a walkthrough of the quest, which doesn't mention the bug I described.
So, none of the solutions from Google worked, either.
You.com
You.com doesn't seem like it knows what it's talking about and just made up a very generic solution.
First, it suggested speaking to the innkeeper and talking to the vampire lord
. I have no idea which quest it's talking about, but there is no innkeeper or vampire lord related to the Bloodchill Manor.
Second, it suggested that not doing these are the reason the quest is bugged. But it's not, the reason is I left without finishing the quest and came back later, as I clearly explained in the question description of the problem.
Third, it suggested that it might not be a bug but rather the orc is a follower. Now, it clearly has no idea what it's talking about and are just making wild guesses. I clearly said in the question it's bugged out, albeit with a typo.
Finally, it suggested reloading a save. Yeah, we are not retards, you.com. We know we can reload a save...
Then it provided Reddit posts related to the bug or the quest. I only looked at the first 3, but none were able to provide a working solution.
Bing Chat
Bing Chat requires that Microsoft Edge must be used. Which is acceptable if it's any good.
Unfortunately, on this particular question, it wasn't able to provide any useful answer.
In addition to the useless answer, it provided 3 links.
1 for Reddit which I assume is the same post that I saw before, which isn't useful.
1 for Steam, and I doubt it has useful information, either.
1 for en.uesp.net which contains a walkthrough of the quest and some useful information, but doesn't solve the bug directly.
Bard
Bard doesn't seem to know exactly which bug I'm talking about, and it generated some generic answers to fix bugs in Skyrim in general.
It also seem to think that bloody
is part of the problem description and not just an emotional expression.
The first suggestion: kill the orc wouldn't work, as the orc is essential (unkillable).
The second suggestion: Wabbajack is an interesting idea, but it doesn't get rid of the orc, though. It just turns him into something else. Also, I'm not sure if Wabbajack works on essential NPCs.
The third suggestion: using the kill console command also wouldn't work because the orc is essential.
The Ultimate Showdown
While neither ChatGPT, Google, you.com, Bing Chat, or Bard was able to provide a satisfactory answer. In my humble opinion, ChatGPT is better in this case, because it allows you to describe your issue in detail like a (somewhat?) normal human being would when asking about an issue. Which is something Google always needed.
It was also able to understand the problem description really well, and provided some very plausible solutions. And while the IDs it provided were pulled out of its exhaust fans, the suggested solutions pointed to the right direction. With the right IDs, those solution could very well have worked.
You.com were also able to seemingly understand that there is a problem with a quest in Skyrim. But it doesn't seem to know exactly what bug or quest I was talking about and just made up some invented reason what caused the bug and made some generic suggestions. It did then suggest Reddit as a place to look at and provided some links.
Google, on the other hand, provided 409 answers. The correct answer could very well be buried in one of them, if you are desperate enough to go through all of them.
Bing Chat simply "couldn't find any information on how to resolve this issue".
Bard, again, seems to understand that there is a bug in Skyrim, but doesn't know exactly which bug or quest I was talking about. So it generated some generic solutions that are applicable to Skyrim.
The Downside of ChatGPT
From what I can see, ChatGPT will only provide the most relevant answers. If the answer it gave is not what you are looking for, you can ask follow up questions, but there is no guarantee it will take previous conversation into consideration.
It assumed I was on PC, but if I wasn't, I wouldn't have access to console commands. But it's my fault, really, for not specifying the platform.
The answer it gave is also more generalized, so it can not pin down the exact IDs.
Google, on the other hand, provides everything it thinks relevant, and allows you to check each one yourself.
Another downside is maybe for website owners. Google essentially drives user traffic to websites. ChatGPT will give answers directly without a link to the website. Cutting out the website that ChatGPT got its training from. This could be solved by providing a citation linked to a list of websites used to train ChatGPT, but the websites must be specific to the question asked. So, not sure how this would be implemented.
Conclusion
ChatGPT seems like a really promising technology being able to understand the human language and provide really relevant answers. With improvements in the future, it could become really powerful.
What are your thoughts on ChatGPT vs Bing Chat vs Bard vs Google vs You.com, or just ChatGPT AI technology in general? Which one do you prefer? Let us know in the comments below!
Bonus Content
I asked ChatGPT and You.com what they think about this website, and here are their responses:
So there you have it, folks.
Customer Testimony:
The blog has some interesting content and appears to be well-written.
The blog appears to be updated regularly, and its content is varied and interesting. It is worth checking out if you are looking for an interesting and informative blog to follow.
Comments
Post a Comment