Have you ever wanted to download some dataset from a viewable map? It can also be named as scraping/scrape dataset. If you know that the data is on ArcGIS server REST services (firebug helps), and you have access to it (you can enter to MapServer sub-website) there is a possibility to download it at once to .geojson or .shp file.

All you need is to download and install AGS to Shapefile small software from GitHub:

https://github.com/tannerjt/AGStoShapefile

There are some dependencies to be installed before

Installation

You can install via NPM

    npm install agsout -g

Or, optionally, download and install from local

    npm install . -g

Using the tool step by step

  • Create a services.txt to include the services you wish to query

for services.txt – use format [service_endpoint]|[title]|[throttle in ms]. Take note of the PIPE (|) symbol and new line.

//example services.txt file
 http://test.service/arcigs/rest/flooding/MapServer/0|Flooding_Layer|0
 http://test.service/arcigs/rest/flooding/MapServer/1|Earthquake_Layer|5000
 http://test.service/arcigs/rest/flooding/MapServer/2|Tornado_Layer|
  • Run the script in command line (eg. windows cmd):
    agsout --help
    agsout -s ./services.txt -o ./backupdir -S
    #-s location of services text file
    #-o directory to backup services
    #-S optional shapefile output (requires gdal)

The throttle is helpful for very large extracts where servers may reject too many requests. The throttle number is in milliseconds.

From my experience, it is hard to download large datasets (>1GB) because of connection dropping, either server or yours. What may sometimes help – an option to change a downloaded chunk of data. Default is to save 100 records per chunk. To change that:
In index.js, replace every instance of ‘100’ with ‘1000’. This should break up the features by chunks of 1000 instead of 100.
As a result, data transfer may be more effective.

Tags: