# Configure Robots Directives Robots directives control whether documentation crawlers can discover and index your published technical documentation. Use these controls when you need to manage search visibility, restrict crawler access during preparation, or confirm that only approved documentation is discoverable. ## Overview Robots directives are instructions for automated crawlers, such as search-engine bots and documentation indexing services. They help you control which documentation areas can be crawled and whether pages should appear in search results. In Atloria, documentation work is managed from the **Technical Docs** area of a project. Before changing crawler access, make sure the documentation you intend to publish is configured and reviewed. The documentation workflow includes a **CONFIG** stage before the **DOCUMENTATION** stage, so crawler-related decisions should be finalized before you distribute or export your documentation. Use robots controls when you need to: - Prevent unfinished technical documentation from being indexed. - Allow approved technical documentation to be discovered by crawlers. - Recheck crawler access after changing documentation structure or publishing settings. - Coordinate search visibility with your organization’s access-control and security policies. > 📷 _Screenshot pending: The Robots Directives controls showing available crawler-access options for technical documentation._ ## Prerequisites Before you begin, make sure you have: - Access to the applicable project. - Permission to configure project documentation settings. - A project with **Technical Docs** available. - A clear decision from your documentation owner or security team about whether crawlers should be allowed or restricted. - Documentation content that is ready for the appropriate visibility level. - An understanding of whether your organization uses access controls, such as **Role-Based Access Control (RBAC)**, for documentation access. ## Step-by-Step Instructions 1. Open the project that contains the technical documentation you want to manage. 2. Select **Technical Docs**. 3. Select **Configure →** to open the available documentation configuration options. 4. Locate the robots or crawler-access controls in the documentation configuration area. 5. Review the current crawler-access setting before making a change. 6. Select the option that matches your publishing decision. 7. Choose the restrictive option if the documentation is unfinished, internal, or not ready for search discovery. 8. Choose the permissive option only when the documentation is approved for crawler access. 9. Review any related visibility or access-control settings before saving your decision. 10. Save or apply the updated configuration using the confirmation action shown on the screen. 11. Return to **Technical Docs** after the change is saved. 12. Select **View Your Documentation →** to review the documentation as it will be presented to readers. > 📷 _Screenshot pending: The Technical Docs configuration area with robots or crawler-access settings highlighted._ 13. Confirm that the visible documentation matches the intended crawler-access decision. 14. Select **Export Documentation** only after you have confirmed that crawler access and documentation visibility are appropriate for distribution. ## Tips and Best Practices - Configure crawler access before treating documentation as ready to publish. The documentation process includes configuration before documentation generation and delivery, so decisions about visibility should be made early. - Use restrictive crawler settings while documentation is being generated, reviewed, or reorganized. This reduces the chance that incomplete pages are discovered before they are ready. - Review robots directives whenever you add a new documentation area, change the documentation structure, or update access-control requirements. - Do not assume that user access and crawler access are the same. Your organization may use **Access Control** or **Role-Based Access Control (RBAC)** to limit who can open documentation, while robots directives separately communicate instructions to automated crawlers. - Coordinate crawler-access decisions with the owners of application security and access control. This is especially important for technical references that could expose implementation details, internal processes, or non-public configuration guidance. - Verify your documentation presentation after configuration changes. Use **View Your Documentation →** to check that the intended content is available and organized correctly before exporting or sharing it. - If you use documentation templates, review the generated output after changing crawler settings. A template can affect which sections are included, while robots directives affect crawler behavior for the resulting documentation. ## Related Pages - [Configure Technical Documentation](./configure-technical-documentation) - [Generate Documentation](./generate-documentation) - [View Technical Documentation](./view-technical-documentation) - [Export Documentation](./export-documentation) - [Configure CI/CD](./configure-cicd) - [Manage Access Control](./manage-access-control) - [Role-Based Access Control (RBAC)](./role-based-access-control) ## Troubleshooting ### You cannot find robots or crawler-access controls Open **Technical Docs** and select **Configure →** again. If no crawler-related option is available, your organization may manage robots directives outside the project’s documentation configuration screen, or your role may not include the required configuration permission. Contact a project administrator or documentation administrator. ### Your documentation is visible when it should remain restricted Review both the robots directive setting and the project’s access-control settings. Robots directives provide instructions to crawlers, but they do not replace access restrictions. Confirm that the appropriate **Access Control** or **Role-Based Access Control (RBAC)** policy is also applied. ### You changed the setting but are unsure whether the documentation is ready to share Select **View Your Documentation →** and inspect the generated documentation before selecting **Export Documentation**. If content is still incomplete, return to **Configure →** and apply the more restrictive crawler-access option until the documentation is approved.