Jump to content

seeking advice on processing VERY large volumes of vector geodata


Recommended Posts

Hello, comrades!

I have a tough problem to solve :(

Seeking your advice on geoprocessing VERY large volumes of vector geodata. Do not mix geoprecessing (buffer, clip, etc.) with converting between data formats - shp, osm, etc. For example of "large" geodataset you may consider OpenstreetMap Data for entire planet - http://wiki.openstreetmap.org/wiki/Planet.osm

"Planet.osm is the OpenStreetMap data in one file: all the nodes, ways and relations that make up our map. A new version is released every week. It's a big file (XML variant over 400GB uncompressed, 29GB compressed)."

 

On these large datasets I need to utilize standard geoprocessing functions (Clip, Buffer, Erase, Select, Find Centroid, Polygon to Polyline).

I have server-level hardware (both physical and virtual).  

The problem is that I did not find any software that is able to process such large volumes of data. All software I tried simply freezes or quits. Splitting entire dataset into smaller chunks that can be processed is highly undesirable. 

 

The solution that seems to be in Manifold 8 x64. Manifold 8 x64 (no cure can be found :( ) utilizes multi-threading technology, so one can use it to geoprocess very large volumes of geodata. If I am not mistaken, this is the only commercial GIS that almost fully utilizes multithreading. Global Mapper 15 x64 is nice but still not enough. Regarding x64 and multithreading ESRI products seems to be almost total failure for many years. A lot of complaints about lack of multithreading on Esri forums. However, "industry leader" is rather slow in reacting to them :) I suspect that geoprocessing data on ArcGIS Server can be a solution? May be Autodesk or Erdas products can be help too? 

 

I believe, that many other forum participants waste a lot of precious time trying to find solution to process large datasets faster and without the need for  "split-geoprocess-merge" approach. There is a similar need to process very large volumes of raster datasets and any help will be highly appreciated as well!

Regads!

Edited by sv74
Link to comment
Share on other sites

Thanks for responding. However, I need not to convert between different geodata formats, but to geoprocess the data itself (clip, buffer, proximity, etc.).

Link to comment
Share on other sites

fme does has these functions. do read up on it's transformers. it able to work with large data cause it's pipe in.

another solution is QGIS. it's free and read that it's proccesing more efficient than arcgis.

Thank you for your kind clarification about FME Safe transformers. I will definitely try it.

http://fmepedia.safe.com/articles/Samples_and_Demos/Parallel-Processing

 

says:

"For 2012, FME has been updated to take advantage of multiple-core processors, which improve parallelization of computations (doing multiple tasks at once) on modern PCs. It also makes use of hyper-threading, a technology used to make each physical core appear as two logical processors to the host operating system. "

 

I agree that QGIS is very good GIS software. Developer version of 2.2 is already x64.

Regards!

Edited by sv74
Link to comment
Share on other sites

Thank you for your kind clarification about FME Safe transformers. I will definitely try it.

http://fmepedia.safe.com/articles/Samples_and_Demos/Parallel-Processing

 

says:

"For 2012, FME has been updated to take advantage of multiple-core processors, which improve parallelization of computations (doing multiple tasks at once) on modern PCs. It also makes use of hyper-threading, a technology used to make each physical core appear as two logical processors to the host operating system. "

 

I agree that QGIS is very good GIS software. Developer version of 2.2 is already x64.

Regards!

Agree with you concerning QGIS. In various aspect is much better than ArcGIS.

 

;)

 

darksabersan.

  • Like 1
Link to comment
Share on other sites

Join the conversation

You can post now and register later. If you have an account, sign in now to post with your account.

Guest
Reply to this topic...

×   Pasted as rich text.   Paste as plain text instead

  Only 75 emoji are allowed.

×   Your link has been automatically embedded.   Display as a link instead

×   Your previous content has been restored.   Clear editor

×   You cannot paste images directly. Upload or insert images from URL.

×
×
  • Create New...

Important Information

By using this site, you agree to our Terms of Use.

Disable-Adblock.png

 

If you enjoy our contents, support us by Disable ads Blocker or add GIS-area to your ads blocker whitelist