Following ChatGPT’s recent release of their Robots.txt blocking code, 26% of the top 100 websites in the world have blocked it. This shows that there is increasing concern amongst website owners about data scraping and copying by AI.

On September 28 th 2023, Google released the Vertex AI code to block Bard, the Google’s AI extension which is already available in demo mode and is expected to launch officially in the first quarter of 2024.

After the ChatGPT release of the robots.txt code to block their A.I. bot on your site, Bard has followed suite.

You can now prevent Bard a.k.a Vertex AI from accessing your website, or parts of it, by blocking Google-Extended in robots.txt.

 

 

The difference between Bard and VertexAI

Bard is Google’s conversational AI tool.

Vertex AI is Google’s platform for building and deploying generative AI-powered search and chat applications. This cannot be seen publicly yet but it will be launched in early 2024 (March/April) when Google searches will carry AI answers. It’s likely that this will only be shown on results where you ask Google a question. Google’s previews have so far shown that they have no intention of making search or SEO obsolete. Bloggers are responsible for 69% of Google’s revenue, so they will take care of content creators – their livelihood depends on it.

 

Here’s a snippet of the announcement directly from the Google blog:

“Today we’re announcing Google-Extended, a new control that web publishers can use to manage whether their sites help improve Bard and Vertex AI generative APIs, including future generations of models that power those products. By using Google Extended to control access to content on a site, a website administrator can choose whether to help these AI models become more accurate and capable over time.

Making simple and scalable controls, like Google-Extended, available through robots.txt is an important step in providing transparency and control that we believe all providers of AI models should make available.

This strategic approach is seen as a pivotal step towards enabling publishers to manage and navigate the intricate realm of AI applications, particularly as these applications continue to expand and diversify.”

Blocking VertexAI requires adding a simple line of code to the site’s robots.txt file:

 

 

If you want to go ahead and block it you can follow the video below (which shows how to insert the ChatGPT code.
And insert the code provided below.

User-agent: Google-Extended
Disallow: /

 

 

Is it smart to block VertexAI?

Probably not. Google is taking a very different approach to ChatGPT, they have mentioned many times that transparency is one of they most important values and therefore will do their best to give credit where credit is due.

An early peek into Google Bard’s Integration with Search results showed that it is likely that search results that use AI will give credit to the main sites they used the information from. A bit like Google currently shows you featured snippets from website that answer your questions directly for certain queries. (See example below)

This is why I do not recommend blocking VertexAI. It is very likely that Google will credit your website when it uses some of the information in it. (At least that’s the indication we have so far.) So blocking VertexAI might impact the traffic you could potentially get from these AI results near search.

I’ve decided not to block it for now.
I want to see what Google search results integrated with AI look like first.

 

 

What do you think? Will you block Vertex AI? let me know in the comments below

 

———————— About the Author: 

Hi, I’m Stephanie, a Digital Marketing Coach and Business Mentor for Coaches. I’m the creator of the Profitable Content Marketing Method™ and host of the Profitable Online Coaching Podcast. I help coaches generate more leads and clients through organic and paid marketing whilst helping them build a business that makes good money with ease. My clients get fully booked with just a handful of enjoyable marketing hours per week!

I have over 15 years of experience building businesses, growing website traffic, writing blogs, creating social media campaigns, launching group programs and generating leads for my 1:1 practice. As well as ranking articles, videos, and podcasts on the first page of Google. I also managed the content and web dev team in my agency for over 10+ years.

I can teach you how to create content that will convert your visitors into discovery call bookings, every time. Let’s Talk