IndexNow is a protocol developed by Microsoft Bing and adopted by Yandex that enables webmasters and SEO pros to easily notify search engines when a webpage has been updated via an API.
And today, Microsoft announced that it is making the protocol easier to implement by ensuring that submitted URLs are shared between search engines.
Given its positive implications and the promise of a faster indexing experience for publishers, the IndexNow API should be on every SEO professional’s radar.
Using Python for automating URL submission to the IndexNow API or making an API request to the IndexNow API for bulk URL indexing can make managing IndexNow more efficient for you.
In this tutorial, you’ll learn how to do just that, with step-by-step instructions for using the IndexNow API to submit URLs to Microsoft Bing in bulk with Python.
Note: The IndexNow API is similar to Google’s Indexing API with only one difference: the Google Indexing API is only for job advertisements or broadcasting web pages that contain a video object within it.
Google announced that they will test the IndexNow API but hasn’t updated us since.
Bulk Indexing Using IndexNow API with Python: Getting Started
Below are the necessities to understand and implement the IndexNow API tutorial.
Below are the Python packages and libraries that will be used for the Python IndexNow API tutorial.
Before getting started, reading the basics can help you to understand this IndexNow API and Python tutorial better. We will be using an API Key and a .txt file to provide authentication along with specific HTTP Headers.
IndexNow API Usage Steps with Python.1. Import The Python Libraries
To use the necessary Python libraries, we will use the “import” command.
Below, you will find all of the necessary import lines for the IndexNow API tutorial.
2. Extracting The Sitemap URLs With Python
To extract the URLs from a sitemap file, different web scraping methods and libraries can be used such as Requests or Scrapy.
But to keep things simple and efficient, I will use my favorite Python SEO package – Advertools.
With only a single line of code, all of the URLs within a sitemap can be extracted.
The “sitemap_to_df” method of the Advertools can extract all the URLs and other sitemap-related tags such as “lastmod” or “priority.”
Below, you can see the output of the “adv.sitemap_to_df” command.
Sitemap URL Extraction can be done via Advertools’ “sitemap_to_df” method.All of the URLs and dates are specified within the “sitemap_urls” variable.
Since sitemaps are useful sources for search engines and SEOs, Advertools’ sitemap_to_df method can be used for many different tasks including a Sitemap Python Audit.
But that’s a topic for another time.
3. Take The URLs Into A List Object With “to_list()”
Python’s Pandas library has a method for taking a data frame column (data series) into a list object, to_list().
Below is an example usage:
Below, you can see the result:
Pandas’ “to_list” method can be used with Advertools for listing the URLs.All URLs within the sitemap are in a Python list object.
4. Understand The URL Syntax Of IndexNow API Of Microsoft Bing
Let’s take a look at the URL syntax of the IndexNow API.
Here’s an example:
The URL syntax represents the variables and their relations to each other within the RFC 3986 standards.
The “&keyLocation” will bring us to the API Key and its “.txt” version.
5. Gather The API Key For IndexNow And Upload It To The Root
You’ll need a valid key to use the IndexNow API.
Use this link to generate the Microsoft Bing IndexNow API Key.
There is no limit for generating the IndexNow API Key.Clicking the “Generate” button creates an IndexNow API Key.
When you click on the download button, it will download the “.txt” version of the IndexNow API Key.
IndexNow API Key can be generated by Microsoft Bing’s stated address.
Downloaded IndexNow API Key as txt file.The TXT version of the API key will be the file name and as well as within the text file.
IndexNow API Key in TXT File should be the same with the name of the file, and the actual API Key value.The next step is uploading this TXT file to the root of the website’s server.
Since I use FileZilla for my FTP, I have uploaded it easily to my web server’s root.
By putting the .txt file into the web server’s root folder, the IndexNow API setup can be completed.The next step is performing a simple for a loop example for submitting all of the URLs within the sitemap.
6. Submit The URLs Within The Sitemap With Python To IndexNow API
To submit a single URL to the IndexNow, you can use a single “requests.get()” instance. But to make it more useful, we will use a for a loop.
To submit URLs in bulk to the IndexNow API with Python, follow the steps below:
Below, you can see the implementation:
If you’re concerned about sending too many requests to the IndexNow API, you can use the Python time module to make the script wait between every request.
Here you can see the output of the script:
The empty string as the request’s response body represents the success of the IndexNow API request according to Microsoft Bing’s IndexNow documentation.The 200 Status Code means that the request was successful.
With the for a loop, I have submitted 194 URLs to Microsoft Bing.
According to the IndexNow Documentation, the HTTP 200 Response Code signals that the search engine is aware of the change in the content or the new content. But it doesn’t necessarily guarantee indexing.
For instance, I have used the same script for another website. After 120 seconds, Microsoft Bing says that 31 results are found. And conveniently, it shows four pages.
The only problem is that on the first page there are only two results, and it says that the URLs are blocked by Robots.txt even if the blocking was removed before submission.
This can happen if the robots.txt was changed to remove some URLs before using the IndexNow API because it seems that Bing does not check the Robots.txt again.
Thus, if you previously blocked them, they try to index your website but still use the previous version of the robots.txt file.
It shows what will happen if you use IndexNow API by blocking Bingbot via Robots.txt.On the second page, there is only one result:
Microsoft Bing might use a different indexation and pagination method than Google. The second page shows only one among the 31 results.On the third page, there is no result, and it shows the Microsoft Bing Translate for translating the string within the search bar.
It shows sometimes, Microsoft Bing infers the “site” search operator as a part of the query.When I checked Google Analytics, it shows that Bing still hadn’t crawled the website or indexed it. I know this is true as I also checked the log files.
Below, you will see the Bing Webmaster Tool’s report for the example website:
It says that I submitted 38 URLs.
The next step will involve the bulk request with the POST Method and a JSON object.
7. Perform An HTTP Post Request To The IndexNow API
To perform an HTTP post request to the IndexNow API for a set of URLs, a JSON object should be used with specific properties.
Since this is a POST request, the “requests.post” will be used instead of the “requests.get().”
Below, you will find an example of a set of URLs submitted to Microsoft Bing’s IndexNow API.
In the example above, we have performed a POST Request to index a set of URLs.
We have used the “data” object for the “data parameter of requests.post,” and the headers object for the “headers” parameter.
Since we POST a JSON object, the request should have the “content-type: application/json” key and value with the “charset:utf-8.”
After I make the POST request, 135 seconds later, my live logfile analysis dashboard started to show the immediate hits from the Bingbot.
8. Create Custom Function For IndexNow API To Make Time
Creating a custom function for IndexNow API is useful to decrease the time that will be spent on the code preparation.
Thus, I have created two different custom Python functions to use the IndexNow API for bulk requests and individual requests.
Below, you will find an example for only the bulk requests to the IndexNow API.
The custom function for bulk requests is called “submit_url_set.”
Even if you just fill in the parameters, still you will be able to use it properly.
An explanation of this custom function:
I have defined some of the parameters with default values such as “host” for Microsoft Bing. If you want to use it for Yandex, you will need to state it while calling the function.
Below is an example usage:
If you want to extract sitemap URLs with a different method, or if you want to use the IndexNow API for a different URL set, you will need to change “set_” parameter value.
Below, you will see an example of the Custom Python function for the IndexNow API for only individual requests.
Since this is for a loop, you can submit more URLs one by one. The search engine can prioritize these types of requests differently.
Some of the bulk requests will include non-important URLs, the individual requests might be seen as more reasonable.
If you want to include the sitemap URL extraction within the function, you should include Advertools naturally into the functions themselves.
Tips For Using The IndexNow API With Python
An Overview of How The IndexNow API Works, Capabilities & Uses
IndexNow API Documentation and usage tips can be found above.In this IndexNow API tutorial and guideline with Python, we have examined a new search engine technology.
Instead of waiting to be crawled, publishers can notify the search engines to crawl when there is a need.
IndexNow reduces the use of search engine data center resources, and now you know how to use Python to make the process more efficient, too.
More resources:
This content was originally published here.