A robots.txt file is one of the primary ways of telling a search engine where it has access on your website. A robots.txt file is a text file following a strict syntax which is read by search engine spiders. These spiders are also called robots, hence the name. The syntax is strict simply because it has to be computer readable. There is no reading between the lines here, something is either 0 or 1.
Your site may have a robots.txt file if it has content that you want blocked from search engines, or are using paid links or advertisements that need special instructions for robots. You may also exclude your website from search engine crawls if you are developing a site that is live and do not want the content public yet.
Robots Exclusion Protocol, the robots.txt file format, is the result of a consensus between early search engine spider developers. Though it is not an official standard by any standards organization, all major search engines adhere to it.
Search engines index the web by spidering pages. They follow links to go from one website to another. Before a search engine spiders any page on a domain it has not encountered before, it opens that domains robots.txt file. The robots.txt file tells the search engine which URLs on that site it is allowed to index. A search engine caches the robots.txt contents and usually refreshes it multiple times a day, so changes are reflected fairly quickly.
Each site has an allowance, as in how many pages a search engine spider will crawl on that site, which SEOs call the crawl budget. By blocking sections of your site from the search engine spiders, you allow your crawl budget to be used for other sections. Especially on sites where a lot of SEO clean-up has to be done, it can be very beneficial to block search engines from crawling certain sections.
You may want to change your robots.txt content is if there are pages on your site that you do not want search engines to crawl and index, or if there are duplicate content issues on your site and you want certain pages to be blocked. If you have two identical pages, you would choose one and add it to the robots.txt file and search engines would not crawl it.