The problem you are facing with the Yoast SEO plugin in WordPress is related to the robots.txt file generated by the plugin. The block of code you mentioned is a section of rules in the robots.txt file that instructs search engines on how they should behave when crawling and accessing your website’s content.
The main issue is that the Yoast SEO block contains a “Disallow” directive with an empty path for all “User-agent” entries, which means it doesn’t allow search engines to crawl or access any part of your site. This, in turn, can cause your pages and posts to not appear in Google and other search engines’ results, negatively affecting the visibility and traffic of your website.
The solution to this problem is quite simple. You need to modify the Yoast SEO block in the robots.txt file to allow access to important parts of your site that you want search engines to index. To do this, either remove the “Disallow:” line or change it to “Allow: /” to enable search engines to access all areas of your site.
What is the solution?
The corrected block in the robots.txt file should look like this:
# START YOAST BLOCK
# ---------------------------
User-agent: *
Allow: /
Sitemap: https://wordpress.develop-site.com/sitemap_index.xml
# ---------------------------
# END YOAST BLOCKWith this modification, you’ll be allowing search engines to crawl and access your entire site. Once you’ve made this correction, it’s important to verify that the robots.txt file is set up correctly. You can use the robots.txt testing tool in Google Search Console to do this.
If you are using the Yoast SEO plugin, this correction is likely to be sufficient to resolve the problem. However, it’s always recommended to perform an additional check to ensure that other settings or plugins are not interfering with the robots.txt file.
Conclusion
Remember that the robots.txt file is a crucial part of your website’s SEO, so it’s essential to ensure it is properly configured to allow search engines to crawl and index your content. With this correction, your pages and posts should start appearing in search results, thereby improving the visibility and ranking of your website.