Have you ever wanted to download some dataset from a viewable map? It can also be named as scraping/scrape dataset. If you know that the data is on ArcGIS server REST services (firebug helps), and you have access to it (you can enter to MapServer sub-website) there is a possibility to download it at once to .geojson or .shp file.
All you need is to download and install AGS to Shapefile small software from GitHub:
There are some dependencies to be installed before
You can install via NPM
npm install agsout -g
Or, optionally, download and install from local
npm install . -g
Using the tool step by step
- Create a services.txt to include the services you wish to query
for services.txt – use format [service_endpoint]|[title]|[throttle in ms]. Take note of the PIPE (|) symbol and new line.
//example services.txt file http://test.service/arcigs/rest/flooding/MapServer/0|Flooding_Layer|0 http://test.service/arcigs/rest/flooding/MapServer/1|Earthquake_Layer|5000 http://test.service/arcigs/rest/flooding/MapServer/2|Tornado_Layer|
- Run the script in command line (eg. windows cmd):
agsout --help agsout -s ./services.txt -o ./backupdir -S #-s location of services text file #-o directory to backup services #-S optional shapefile output (requires gdal)
The throttle is helpful for very large extracts where servers may reject too many requests. The throttle number is in milliseconds.
From my experience, it is hard to download large datasets (>1GB) because of connection dropping, either server or yours. What may sometimes help – an option to change a downloaded chunk of data. Default is to save 100 records per chunk. To change that:
In index.js, replace every instance of ‘100’ with ‘1000’. This should break up the features by chunks of 1000 instead of 100.
As a result, data transfer may be more effective.