Chat GPT – an AI bot based on GPT-3, with specific training strategies and fine tuning – launched on 30 November 2022 and, like many people across the globe, we made the most of its capabilities by using it to draw ridiculous pictures or getting it to write not-so-flattering haikus of each other.
Yes, some of it was funny and yes, sometimes the output wasn’t quite what we were after, but I think it would be fair to say that it was unadulteratedly incredible. It also underpinned the fearful question that we all started asking each other: are the robots actually here to take our jobs?
Seeing as this is a slightly sci-fi related blog, let’s go back in time. Just over a year before Chat GPT’s launch – in November 2021 – we had an idea.
There was a challenge: 2 blog posts. One blog written by a human named Siobhan. Another blog written by a bot, with prompts given by a man named Dave to OpenAI Davinci (the most capable GPT-3 model available at the time). Which blog would perform best?
Here’s what we found.
Terms and conditions
Naturally, to make this a fair contest, we set a brief with many rules and caveats.
The blogs were to hold roughly the same title, with the starting point as “Is the future of content humanless?”
Both blogs had to meet the following brief: essentially, can content be convincingly written by a bot and will human involvement be removed from content creation in the future?
And then there were our methods of control:
- Both blogs had to have the same word count (give or take a few words)
- Both had the same target keywords
- Both would encounter the same social sharing/newsletter sharing from our marketing team
- Both blogs would receive a strict 10 minute tone of voice/grammar edit from me (as all blogs do that are loaded onto our site)
- Both will be posted on the same day and as near to the same time
- Both will be posted with authors
- Both were to have a note in the top that explained there were two versions of the same blog and that they were part of an evil experiment
The results
For this next part, I’ll happily defer to our Senior SEO Consultant and SEO wizard, Stanley Dunthorne.
Here’s Stan’s analysis:
“It was a remarkably close performance – although, if I had to choose, I’d say the AI blog would probably take the lead. It got a larger number of impressions, ended up ranking for better terms (measured fairly subjectively by me) and ranked for a larger number of terms overall.
But quantity metrics isn’t all we’re focused on of course! Behaviour metrics like time on page and bounce rate were very strong considering it was written by AI, with time on page actually beating the human blog (Note from editor: it’s also worth noting that we positioned this as an experiment from the start, asking our followers to read both and see if they could guess, which in hindsight may have impacted performance.)
Unrelated to this but mildly interesting: both followed a similar pattern in terms of visibility (broadly measured by number of impressions) in that the number of impressions increased for around 6 months following publication before declining and flattening out. The best rankings for both also came within the month or two after publication, perhaps highlighting the importance of content freshness.”
So, are the robots here to take our jobs then?
Well, in classic marketing terms – it depends. Chat GPT is arguably a more superior version of the AI-powered programme we used over a year ago but even Chat GPT needs some pretty specific prompts, including an edit from a human to make sure that your brand’s tone of voice is communicated in the right way. As a head of marketing, it’s my job to know our brand backwards and so I scrutinise everything we publish; however amazing AI gets, I’m still not wholly convinced that a robot would ever do that accurately.
Whilst hitting SEO rankings is always going to be important, there’s also an engagement factor. Personally, I find SEO-driven content tends to answer a question or problem that I’m not sure how to solve (for example, how to replace the battery in my car’s key fob, after moronically ignoring all the warnings and then being late to babysit for my baby niece. I wasn’t popular that day.) Whereas I love reading articles that are based around someone’s experiences or ideas that, in turn, inspire my own ideas and then experiences. This then brings us back to the 2023 marketing buzzword: authenticity.
So, I’d argue that it’s the balance – the sum of all parts, if you will. Like any head of marketing, unsurprisingly I want the Hallam site to have high site traffic, to rank highly for specific keywords and to consistently deliver a high volume of impressions. Surely this is where AI-generated content comes into its own, especially for smaller marketing teams that might otherwise struggle to generate the number of blogs needed to meet a regular posting schedule. However, I also want it to be a place where people come to read the insights from our experts, with content that is somewhat delightfully subjective and also something that they can’t get elsewhere. Who knows, maybe with a few swear words. Authentic ones.
Chat GPT is not at a point where we can remove human involvement completely – and nor should we. In fact, we don’t even have to make the “either/or” choice – we can use both to our advantage, at the same time, in the same article. And, for those of us in content-related marketing roles, we can sleep easily.
For now, at least…👀
