Setting up auto-crawl for your AI Chat: always up-to-date answers

Auto-crawl is the process by which Wappy automatically visits the pages of your website and processes the content. The AI reads text from product pages, service pages, FAQs, blog articles and other content.

What the crawl picks up:

• Product names, descriptions and prices
• Service pages and offerings
• FAQs and help articles
• Contact information and opening hours
• Blog articles and other editorial content

What the crawl does not pick up: images, videos, files behind a login and dynamic content that only loads after a search (such as search results). Provide that information via uploaded files.

The frequency of the crawl depends on how often your website content changes:

• Manual crawl (Starter plan): you start the crawl yourself from the dashboard. Use this every time you update product prices, add a new FAQ, or update a service page. Expect to do this 1–2 times per month.
• Weekly crawl (Business plan): automatic, every week. Good for businesses that regularly update content: new products, seasonal offers, changed opening hours.
• Daily crawl (Enterprise plan): automatic, every day. Necessary for webshops with daily changing stock, current prices, or seasonal ranges.

Choose a frequency that matches how dynamic your website is — no more than necessary. A too-frequent crawl of a static website adds nothing.

By default, Wappy crawls all publicly accessible pages of your website that are reachable via your sitemap or internal links. You can exclude specific URLs or sections in the dashboard — for example:

• Pages with confidential information that should not be shared via the AI
• Outdated pages you have not yet removed
• Login pages, account pages or order overviews

Tip: do not exclude pages that contain relevant product information. The more relevant content is crawled, the better the AI can answer.

Auto-crawl and uploaded files complement each other. The crawl gives the AI a broad base of your website content. Files give the AI depth — detailed information not on your site.

The best approach:

1. Set up auto-crawl for your entire website
2. Upload additional files for information not on your site: internal price lists, detailed product specifications, conversation guidelines
3. Update files when information changes — otherwise the AI gives outdated answers

More on which files to upload and how to set prompt preferences: Setting up AI Chat: files and prompt preferences.

If the AI answers a question with information that is no longer correct — an old price, an outdated delivery time — the knowledge base has not been updated. The solution:

1. Start a manual crawl directly from the dashboard
2. Check your uploaded files — are they still current?
3. Consider a higher crawl frequency if your content changes regularly

Outdated information is the most common problem with AI Chat. The fix is simple: crawl regularly and keep files up to date. That is a five-minute routine per week.