Many idea about Script in Phyton for process data

[3dbu]

Hi community , i have a  doubt , i need made a script in phyton that read a folder with 4800 raster in tif format, each raster is month and i need sum by month and year ,  i made a model builder for iterate just 12 raster x year and sum all, but my problem begin when i want to create a scripts that sum per year all raster and generate a raster for each year that summarized all raster for each month per year, in total are 40 year of data ....

Anyone could give a idea for made that script completely in phyton, all idea are welcome.

reagards .3dbu 

 

 

[jack_stepeny]

U have all the raster in same folder???

Is there any number in the raster names that can be used to identify raster which belongs to Jan which belongs to dec etc. Like MODIS follows naming convention and we can easily identify the which raster belongs to which months..

Please provide more details so that I can help you.

[3dbu]

15 hours ago, jack_stepeny said:

U have all the raster in same folder???

Is there any number in the raster names that can be used to identify raster which belongs to Jan which belongs to dec etc. Like MODIS follows naming convention and we can easily identify the which raster belongs to which months..

Please provide more details so that I can help you.

Thanks, jack_stepeny for your answer, all details are:

1) I have all data (raster file) in the same folder, 

2) Each raster is named "prc_1971_01.tif" .and.."prc_1971_12.tif" for 12 month of the year, and the data is for 40 year ( 40 year x 12 month = 480 raster) fro this variable

3) I need made a raster for each year (summarized all raster of month for each year) and name it like "prc_1971.tif" in other word %prc_1971%, for each year.

And i need made the scripts for do all process, is possible do you have any clue or idea...all idea are welcome.

Thanks in advanced.

 

 

 

[gis.sig.qc]

All these climate data usually have the same extent and the same number of rows/cols, so I suggest you to read your TIF - iteratively - and put them into a numpy array, then do the sum and keep another numpy array with the results. In that way you'll only have 2 variables to deal with (you input_TIF and you sum_TIF), and your memory consumption will be very low, and your speed will increase substantially.

Good luck!

GSQ

 

[jack_stepeny]

If the extent of all rasters are same.. it can be done easily.

1. List all rasters in a folder

2. Loop through list to select rasters from particular year. ( It can be done by matching the year with the name of raster. As u have mentioned name include year)

3. Then sum of all the raster can be done.

If u r comfortable with R I can share code that I have already written to accomplish this task.

[3dbu]

On 2/23/2017 at 7:01 PM, jack_stepeny said:

U have all the raster in same folder???

Is there any number in the raster names that can be used to identify raster which belongs to Jan which belongs to dec etc. Like MODIS follows naming convention and we can easily identify the which raster belongs to which months..

Please provide more details so that I can help you.

Thanks, jack_stepeny for your answer, all details are:

1) I have all data (raster file) in the same folder, 

2) Each raster is named "prc_1971_01.tif" .and.."prc_1971_12.tif" for 12 month of the year, and the data is for 40 year ( 40 year x 12 month = 480 raster) fro this variable

3) I need made a raster for each year (summarized all raster of month for each year) and name it like "prc_1971.tif" in other word %prc_1971%, for each year.

And i need made the scripts for do all process, is possible do you have any clue or idea...all idea are welcome.

Thanks in advanced.

 

I tested my scripts but still need to know how i make for read from a folder all raster and summarized each one month per each year ...

 import arcpy
... from arcpy.sa import *
... jn71 = arcpy.Raster("C:/prc.gdb/prc_1971_01")
... fr71 = arcpy.Raster("C:/prc.gdb/prc_1971_02")
... mr71 = arcpy.Raster("C:/prc.gdb/prc_1971_03")
... ab71 = arcpy.Raster("C:/prc.gdb/prc_1971_04")
... my71 = arcpy.Raster("C:/prc.gdb/prc_1971_05")
... jn71 = arcpy.Raster("C:/prc.gdb/prc_1971_06")
... jl71 = arcpy.Raster("C:/prc.gdb/prc_1971_07")
... ag71 = arcpy.Raster("C:/prc.gdb/prc_1971_08")
... st71 = arcpy.Raster("C:/prc.gdb/prc_1971_09")
... oc71 = arcpy.Raster("C:/prc.gdb/prc_1971_10")
... nv71 = arcpy.Raster("C:/prc.gdb/prc_1971_11")
... dc71 = arcpy.Raster("C:/prc.gdb/prc_1971_12")
... outraster = (jn71 + fr71 + mr71 + ab71 + my71 + jn71 + jl71 + ag71 + st71 + oc71 + nv71 + dc71)
... outraster.save ("C:/prc.gdb/prc_1971")

## Data 1972 process

import arcpy
... from arcpy.sa import *
... jn72 = arcpy.Raster("C:/prc.gdb/prc_1972_01")
... fr72 = arcpy.Raster("C:/prc.gdb/prc_1972_02")
... mr72 = arcpy.Raster("C:/prc.gdb/prc_1972_03")
... ab72 = arcpy.Raster("C:/prc.gdb/prc_1972_04")
... my72 = arcpy.Raster("C:/prc.gdb/prc_1972_05")
... jn72 = arcpy.Raster("C:/prc.gdb/prc_1972_06")
... jl72 = arcpy.Raster("C:/prc.gdb/prc_1972_07")
... ag72 = arcpy.Raster("C:/prc.gdb/prc_1972_08")
... st72 = arcpy.Raster("C:/prc.gdb/prc_1972_09")
... oc72 = arcpy.Raster("C:/prc.gdb/prc_1972_10")
... nv72 = arcpy.Raster("C:/prc.gdb/prc_1972_11")
... dc72 = arcpy.Raster("C:/prc.gdb/prc_1972_12")
... outraster = (jn72 + fr72 + mr72 + ab72 + my72 + jn72 + jl72 + ag72 + st72 + oc72 + nv72 + dc72)
... outraster.save ("C:/prc.gdb/prc_1972")

## Data 1973 process

import arcpy
... from arcpy.sa import *
... jn73 = arcpy.Raster("C:/prc.gdb/prc_1973_01")
... fr73 = arcpy.Raster("C:/prc.gdb/prc_1973_02")
... mr73 = arcpy.Raster("C:/prc.gdb/prc_1973_03")
... ab73 = arcpy.Raster("C:/prc.gdb/prc_1973_04")
... my73 = arcpy.Raster("C:/prc.gdb/prc_1973_05")
... jn73 = arcpy.Raster("C:/prc.gdb/prc_1973_06")
... jl73 = arcpy.Raster("C:/prc.gdb/prc_1973_07")
... ag73 = arcpy.Raster("C:/prc.gdb/prc_1973_08")
... st73 = arcpy.Raster("C:/prc.gdb/prc_1973_09")
... oc73 = arcpy.Raster("C:/prc.gdb/prc_1973_10")
... nv73 = arcpy.Raster("C:/prc.gdb/prc_1973_11")
... dc73 = arcpy.Raster("C:/prc.gdb/prc_1973_12")
... outraster = (jn73 + fr73 + mr73 + ab73 + my73 + jn73 + jl73 + ag73 + st73 + oc73 + nv73 + dc73)
... outraster.save ("C:/prc.gdb/prc_1973")

 

 

So i need made this summarized but still i don't now how make the list raster and read all with made separate like i done, any idea!!! regards and thks in advanced

 

 

[3dbu]

On 2/24/2017 at 8:21 PM, jack_stepeny said:

If the extent of all rasters are same.. it can be done easily.

1. List all rasters in a folder

2. Loop through list to select rasters from particular year. ( It can be done by matching the year with the name of raster. As u have mentioned name include year)

3. Then sum of all the raster can be done.

If u r comfortable with R I can share code that I have already written to accomplish this task.

Cool, nice dude, if you can share the code in R will be welcome, thks in advanced and really bro i appreciate your help, regards.

[jack_stepeny]

12 hours ago, 3dbu said:

Cool, nice dude, if you can share the code in R will be welcome, thks in advanced and really bro i appreciate your help, regards.

Here is code.. Before executing this code make sure raster library is installed in R

library(raster)
#setting working directory(folder path containing all rasters)
setwd("G:\\new_gl_data\\4har")
#listing all raster in a folder
rlist=list.files(getwd(), pattern=".tif$", full.names=F)

#change year here (2001:2015)
for (yr in 2001:2015){
nlist = rlist[grep(yr,rlist)]
stack_ras = stack(nlist)
sum_ras = sum(stack_ras)
sum_brk = brick(sum_ras)
writeRaster(sum_brk, filename=paste("prc_",yr,sep=""), format="HFA",overwrite=TRUE)
rm()
gc()
}

[3dbu]

12 hours ago, jack_stepeny said:

Here is code.. Before executing this code make sure raster library is installed in R

library(raster)
#setting working directory(folder path containing all rasters)
setwd("G:\\new_gl_data\\4har")
#listing all raster in a folder
rlist=list.files(getwd(), pattern=".tif$", full.names=F)

#change year here (2001:2015)
for (yr in 2001:2015){
nlist = rlist[grep(yr,rlist)]
stack_ras = stack(nlist)
sum_ras = sum(stack_ras)
sum_brk = brick(sum_ras)
writeRaster(sum_brk, filename=paste("prc_",yr,sep=""), format="HFA",overwrite=TRUE)
rm()
gc()
}

thanks very much bro jack_stepeny, i 'll run the scripts , i appreciated your time and help for support and help , regards 

[jack_stepeny]

13 minutes ago, 3dbu said:

thanks very much bro jack_stepeny, i 'll run the scripts , i appreciated your time and help for support and help , regards 

Feel free to contact me if u encounter any problem during running the script.

[3dbu]

Nice jack_stepeny, i appreciated your help, and sure if have any question ,i'll contact to you, thank in advanced.