I am working to standardize our web application scans and one of the options is
"Crawl all links and directories found in:"
What are the Pro/Con of setting the web app to use these or not use them?
These files as I recall intended for different things. The sitemap.xml would "aid" a crawler in finding links or pages that may not normally be presented to a browser.
The robots.txt is usually either helping or trying to stop robots from crawling the site. I don't have any idea how much they are being used anymore. I typically ignore them; but you could run multiple scans of the application in just discovery mode and see if your finding additional pages that your not finding when ignoring these files.
Retrieving data ...