ChatGPT vs experienced copywriter: A head-to-head copywriting comparison
June 2, 2023 • Glenn Murray
ChatGPT talks a pretty big game. And a lot of people have listened. But how good is it, really? Does it spell the end of copywriting, as some have predicted? Or is it just another tool to add to the good copywriter’s toolbox?
Time to find out…
I decided to challenge ChatGPT to a ‘write-off’. Me versus it. How well can it write some pieces I’ve written in the past? Below are 5 direct comparisons. You be the judge.
Some things to consider while you’re reading
The brief
The brief I gave ChatGPT in each case is NOT the brief the client originally came to me with. It’s the detailed information I eventually managed to extract from the client, either through an hour or two on the phone or a detailed questionnaire. (And sometimes with the assistance of a blowtorch and a pair of pincers.)
In my experience, almost all clients are unable to articulate their brand, values, offering, goals, differentiators, voice and audience without an expert drawing it out of them. Someone who knows what questions to ask and who takes the time to ask them.
If one of my clients asked ChatGPT to write their copy for them, and they gave it the same information they originally gave me (pre-blowtorch and pincers), I can guarantee ChatGPT’s copy would be even worse than the below.
The structure
Copy can read beautifully, but still be inappropriate. For example, if it’s going on a home page, your intro needs to be just a few words, otherwise it won’t fit above the fold, once the website is designed and built.
And sometimes the copy needs to be in a text callout or a table, or it needs to appear in an image. Obviously you’re not going to get this sort of guidance from ChatGPT.
OK, let’s get to the comparison…
Comparison 1: Freedom Loans web copy
Freedom Loans is a mortgage broker that specialises in getting loans for people with bad credit. They asked me to write their ‘Bad Credit Home Loan’ page.
 | Read my copy |
 | Read ChatGPT's copy |
 | Read the brief |
Comparison 2: Cloud Networks home page copy
Cloud Networks was a company that provided cloud access to all your normal local software products. If your team used a local version of Office, Outlook, Photoshop, MYOB, etc., and you wanted to continue using them, but you wanted everyone to have cloud access to them (and all the associated files), Cloud Networks would provide it, and you didn’t have to worry about server hardware, server upgrades, security, etc.
| Read my copy |
| Read ChatGPT's copy |
| Read the brief |
Comparison 3: Winning Services home page copy
Winning Services is a family-owned Australian company that’s part of the Winning Group, which specialises in white goods and appliance sales (fridges, freezers, toasters, kettles, washing machines dryers, etc.). They handle all deliveries, installation and warehousing for all other companies in the group, including Winning Appliances and Appliances Online, and they also offer these services to third party retailers. This copy was to promote their services to those third party retailers.
| Read my copy |
| Read ChatGPT's copy |
| Read the brief |
Comparison 4: Adelaide City Council’s ‘Why Adelaide’ page copy
Adelaide City Council wanted a ‘Why Adelaide’ page for its ‘Invest Adelaide’ website. The goal was to encourage businesses to invest in the city of Adelaide.
| Read my copy |
| Read ChatGPT's copy |
| Read the brief |
Comparison 5: Venezia Duffle product description
The Venezia Duffle is a rugged leather travel bag. The retailer wanted a more creative product description for it, at once masculine and playful.
| Read my copy |
| Read ChatGPT's copy |
| Read the brief |
And they were the EASY jobs!
The above examples were pieces that started from pretty defined, finite briefs. The briefs may have been hard to extract from the clients, but once extracted, they were pretty solid.
But the reality is, most of my jobs aren’t like that. Most of the time, even when I have all the information the client is able to provide, it’s hardly a brief at all. It’s more a feeling about their business and their offering, along with a brain-dumped bunch of facts and figures that may or may not make an appearance in the copy.
Or they’re constructed from interviews with numerous subject matter experts, most of whom disagree with each other!
Here are some difficult jobs…
Here are a few examples of that sort of thing.
I would love to have done a comparison of my copy vs ChatGPT’s copy for these jobs, but the reality is, there’s absolutely no way I would have been able to write a prompt for ChatGPT that would have conveyed everything that went into those pieces. It’s simply impossible.
And if it’s impossible for me, with more than two decades of experience capturing and distilling client requirements, I can guarantee it would have been impossible for those clients too.
Verdict
There’s no denying that ChatGPT writes pretty well. But it’s bland and unfocused, and often it’s actually cringey or outright nonsensical.
In other words, it’s exactly like a bad copywriter. Someone who can string together a pretty sentence, but that’s about it.
I know my clients want more than that. They want copy that breaks through, engages their readers, speaks in their voice, distils their offering, and does justice to their brand. And they need all of that done in a way that works with their chosen medium.
I’d be shocked if you didn’t want the same.
PS. In case you’re wondering, I used the latest and greatest ChatGPT for this experiment: version 4, the one you get with a paid account (which I have).
Anna Rogan wrote on June 2nd, 2023
Honestly, this point -> "almost all clients are unable to articulate their brand, values, offering, goals, differentiators, voice and audience without an expert drawing it out of them" cannot be overstated. And it's not because clients are dunces. Seasoned and experienced copywriters, marketing consultants and brand strategists have the very same problem. It's the whole 'trying to read the label from the inside of the jar' effect. Providing PERSPECTIVE is so much a part of the job, of being able to draw out a good enough brief to THEN be able to do the writing. If clients are going to go to the trouble of creating the best possible brief for ChatGPT and what it delivers is average at best—the question for clients is, is the money they'll save from not working with an actual copywriter worth it? I think the answer is obvious. So, MY next question is, what *is* ChatGPT good for? If it can't provide strategy and perspective and it can't deliver a final piece does it have another place in the copywriting process? And where is that place? Can we slot it into customer or brand voice analysis to save time? Can it provide a first draft to edit (or is the first draft going to be too average to bother)? Can it synthesise main points from a brief to create a skeleton draft to work from? I'd love to hear your thoughts Glenn!
Reply
Glenn Murray wrote on June 2nd, 2023
Couldn't agree more, Anna! It's all about that brief. RE what ChatGPT is good for: Perhaps it's good for brainstorming ideas. And I know a copywriter uses it to describe the voice used in existing copy. I've also seen a tool (https://www.chatpdf.com/, built on ChatGPT) that summarises long PDFs. That's very helpful. Beyond that, I think ChatGPT can be used pretty reliably (and very cheaply) to write basic, vanilla SEO content and articles. I know some people still want that sort of stuff, although I don't understand why.
Reply
Paul McDonald wrote on June 2nd, 2023
Can’t see AI giving the personal touch or even getting the Australian terminology right. We are so different, not even overseas copywriters seem to get it right. Loved your writing Glenn
Reply
Glenn Murray wrote on June 2nd, 2023
Thanks Paul! Yeah, that's definitely a problem still. I suspect it'll be an easier one to solve than the others though.
Reply
Michael Davy wrote on June 4th, 2023
The crucial difference to me was the AI generated text felt genetic-ad-like, and I noticed myself having to re-read sections as a result, as I’d tune out - like ads on TV during show breaks. Whereas the professional copywriting by Silver Pistol felt warm, clear, personable and I remained attentive to what I was reading. I say ‘felt’ in these descriptions because I found it hard to put my finger on why this was so from the writings themselves. It was just the impact I experienced from each. I am not a copywriter, so it’s not my field, but I am interested in effective writing. Perhaps, it is the subjective experience I gain vs the simple delivery of information, that ultimately matters. It makes sense that AI is a handy tool, but it is not yet effective at being the artist.
Reply
Glenn Murray wrote on June 4th, 2023
Thanks Michael! Don't worry, it's not just you. I find it difficult (or at least time-consuming) to explain why ChatGPT's copy isn't up to snuff too. It's the same when trying to explain to a freelancer why their copy isn't acceptable. The problems are usually subtle but very profound. But I think you've nailed it when you say you "felt" it AND when you said you had to re-read sections. I think it boils down to one or both of these for most readers.
Reply
Charmaine wrote on June 5th, 2023
I totally agree with your verdict. It's hard to articulate, it's like it's written by a Stepford wife, no depth or feeling. Not in the sense you are writing a screenplay, but there is a need for connection and authenticity that the bot just doesn't seem to create.
Reply
Glenn Murray wrote on June 5th, 2023
I like that. Connection and authenticity. I actually think this stems from a lack of knowledge, not an inability to write with authenticity. When freelance copywriters have delivered similar-sound stuff to me in the past, it's always been because they didn't actually understand that subject matter they were writing about. But I think the key difference here is that, unlike a confused copywriter, ChatGPT can NEVER truly understand something. It's just a language model that places one word after another based on how often they're used in that order in its training material. (I know there's more to it than that, but that's a helpful way to look at it.) It will never have hands-on experience with the subject matter. It will never be able to reflect, as a human would, on how well the subject matter met its information needs. It will always lack subjective human experience.
Reply
Robert wrote on June 5th, 2023
Comparing the pieces written it was clear to me that there is no substitute for good, genuine copywriting. ChatGPT may be able to write adequately but it lacks the personal touch and feeling in it's writing. It read to me more like a dodgy used car ad than anything with real depth and professionalism. There were lots of corny expressions used too. Glenn's writing on the other hand was more informative, relevant and kept me engaged in the subject matter. It had a personal and professional touch too that I didn't feel with ChatGPT. I think you win the 'write off' hands down Glenn.
Reply
Glenn Murray wrote on June 6th, 2023
Thanks Robert! Yeah, the corny expressions are painful, aren't they? And I agree wholeheartedly that ChatGPT's copy lacks that personal and professional feeling.
Reply
David wrote on June 14th, 2023
The AI generated copy reads well, but when I read it side by side against the writing written by the copywriter, it definitely doesn't feel as good. I am not a writer so I can't say why, it just feels that way. The AI writing feels very generic in comparison!
Reply
Glenn Murray wrote on June 14th, 2023
Thanks David. Appreciate your vote. :-)
Reply
Ken wrote on June 16th, 2023
I agree that side by side the AI generated material was clunky and not as personal or warm as what was written by Glenn. There was a point in the AI article for Freedom Loans where I actually thought this is starting to sound like a scam. Glenn’s versions read better flowed better and just generally felt better (way better). My personal concern however is - as a “lazy” reader I wonder if I could tell the difference without the preamble? Without the side by side? I wonder if I could pick it “cold”? I’d like to think that I could but again as a “lazy reader” skimming through things not paying 100% attention and also only taking in what I want to (at the time) I wonder if I’d notice? It really does concern me that I wouldn’t and as such might believe something that’s just not true or feel I understand something that’s almost right but not actually correct. From that perspective in my opinion there always has to be a need for a real person at the other end of the pen.
Reply
Glenn Murray wrote on June 16th, 2023
Thanks Ken. I'm glad it was the ChatGPT version that felt like a scam, and not mine! :-)
Reply