How can I get around the 500 results maximum on the metadata export/search results?
Your export link will look something like this:
That URL will return ICPSR studies in MARC format that were added/updated in the last quarter. It will return the first 500 results. If you change paging.startRow=1 to paging.startRow=501, it will return results 501 through 1000. By doing that, you can snag more than 500 results, but you have to break them into 500-record increments.
You can change paging.rows to a higher value, but it won't let you get around the 500-record limit.
Why doesn't ICPSR just make links for each 500? Or the whole set?
We want to prevent the export functions from taxing the server too much. If we put those links on the site anywhere, then every search engine crawler would hit those links repeatedly, which could cause our server to grind to a halt.