In late 2023, I wrote an article evaluating how properly ChatGPT and Google Bard deal with writing safety insurance policies. Provided that ChatGPT 4.0 has been out there as a paid model, referred to as ChatGPT Plus, for a while now, and Google just lately rebranded Google Bard as Gemini (with Gemini Superior out there as paid providing), it is a good time to match how the 2 carry out in a head-to-head of the highest 10 use instances for data safety professionals.
Earlier than we bounce in, the same old generative synthetic intelligence (AI) caveats apply: Watch out of the information you punch in and bear in mind the output could not all the time be dependable.
1. Producing Diagrams or Idea Flows
Each instruments declare to have the ability to generate diagrams and idea flows. Nevertheless, Gemini admits it could solely generate ASCII diagrams, pointing you to extra skilled instruments in order for you one thing higher. I requested each instruments to generate a diagram to elucidate the OAuth authentication stream.
Gemini whereas represented in ASCII, does the job and breaks it down into usable classes.
ChatGPT hallucinates badly. At first look, whereas the picture seems skilled, it does not signify OAuth in any respect. The wording is nonsensical, misspelled, or downright illegible: Authiration and Athoricazt anybody?
2. Explaining Structure Diagrams
Each instruments can ingest diagrams and clarify what is going on on. The outcomes are a lot better than what occurs while you ask them to generate diagrams. As enter, I used an instance Internet utility firewall (WAF) structure from Edgenexus.
Google Gemini is a lot better at explaining structure diagrams as a result of it is succinct. ChatGPT will do the job simply high quality; it is only a tad wordy.
3. Decoding Exploit Code
A typical safety operations (SecOps) exercise is attempting to determine what a selected malware or exploit code does. I took a latest Elasticsearch stack overflow public exploit and fed it into every instrument to see what it understood. There is no clear winner: Each instruments establish the exploit accurately and clarify the tip consequence, what every portion of the code does, and the way it works.
4. Decoding Log Information
SecOps professionals typically want to determine what the heck is happening in log information. I fed each instruments an instance CEF format log file of an tried breach and requested every to elucidate what is going on on. Gemini explains it higher, summarizing properly and even suggesting follow-up steps. It additionally clearly states what occurred (tried entry of /and many others/passwd) proper at the start and elaborates on the way it got here to that conclusion. Whereas ChatGPT arrives on the similar conclusion, it’s manner too verbose.
5. Writing Insurance policies and Safety Documentation
I will not elaborate an excessive amount of on this and can as an alternative refer you to my earlier article on this subject. I ran the take a look at once more with Gemini, and the outcomes are in keeping with Bard’s: Gemini clearly understands and generates higher safety documentation than ChatGPT.
6. Figuring out Weak Code
Whereas these instruments weren’t designed for (and should not be used for) figuring out susceptible code, they’ll nonetheless do an ample job. I made a decision to check it by feeding each instruments an insecure direct object reference (IDOR) vulnerability instance in Python, which additionally accommodates a SQL injection.
ChatGPT accurately recognized each vulnerabilities and the dearth of authentication. Gemini missed the IDOR however identified the SQL injection and went a step additional to suggest amended code to repair the vulnerability. ChatGPT also can do that, nevertheless it have to be prompted to take action.
7. Writing Scripts and Code
A typical safety operations heart (SOC) exercise is writing scripts for log parsing or knowledge manipulation. I gave each instruments the next immediate:
“Write me a Python script that extracts all IPv6 addresses from a txt enter file, removes all duplicates, does a lookup to geo-locate and establish the proprietor of the IP, and output the end in a CSV file”
There is no clear winner right here; each instruments produce clear, readable code that works and explains what it does.
8. Analyzing Information and Metrics
I additionally examined whether or not these instruments might assist with analyzing knowledge or safety metrics. Gemini is a giant loser right here as a result of it does not do it in any respect — it could solely information you thru how do that in Excel and Energy BI. ChatGPT has the benefit by means of its Information Analyst plug-in, which ingests Excel information to generate any graphs you need. It even suggests visualization varieties, and you may modify a graph’s design, together with shade, axes, and labels, by means of the immediate.
9. Writing Person Consciousness Messages
Each instruments also can generate emails for safety consciousness campaigns. I gave each the next immediate: “Generate an e-mail used for a safety consciousness marketing campaign. Be humorous and sarcastic. Remind folks why they should not click on on random emails from random folks.”
Gemini wins right here — its e-mail is transient, has the precise tone, and (though humor is subjective) I discovered it barely funnier. ChatGPT nonetheless generates the precise tone and a very good e-mail, however I discovered it a tad too lengthy for an consciousness e-mail. Both manner, each instruments do an excellent job.
10. Decoding Compliance Frameworks
If in case you have a fast query about the way to implement a compliance framework, these instruments can undoubtedly assist. Whilst you could not do that typically, they’re very helpful while you want it.
If you happen to’ve ever argued with somebody about what constitutes a “vital” change below PCI-DSS and the way it ought to be utilized, you are not alone. I prompted every instrument with:
“Clarify the idea of ‘vital change’ within the context of PCI-DSS. What constitutes a serious change often? Record the precise requirement from the usual as properly”
Gemini has the higher hand: It accurately lists the precise necessities from the usual (comparable to 6.4.5 and 6.4.6) and the way to interpret whether or not one thing is a major change. ChatGPT does not point out precisely the place this data seems in the usual.
Which AI Is Higher, ChatGPT or Gemini?
There you have got it. Relying in your use case, both instrument could be a useful ally in boosting productiveness and serving to you in your day-to-day actions within the infosec trenches.