An Example of radar DSP Product Processing

Newfel Mazari Hongjie Xie

What is Geoprocessing ? Why Geoprocessing ?

Geoprocessing tasks can be time intensive since they are

often performed on a number of different datasets or on large datasets with numerous records.
Using ArcGis Tools to process GIS Data

ArcMap ArcCatalog ArcInfo Scripting

Geoprocessing and Scripting

Scripting is an efficient method of automating

geoprocessing tasks. Scripting allows the execution of simple processes (a single tool) or complex processes (piggybacked, multitool tasks with validation). Scripts are recyclable, meaning they can be data nonspecific and used again.
Script: is a set of computing instructions, usually

stored in a file and interpreted at run time.

Tools in Arcgis
All commands (otherwise known as tools) are

maintained in toolsets within the ArcGIS toolboxes.

A toolbox can contain tools, toolsets, and scripts and is

organized according to the collection of geoprocessing commands it contains.

A toolset can contain tools, toolsets, and scripts and is

organized according to the geoprocessing commands it contains.

Types of Tools in Arcgis

System tool

Model Tool

Script Tool

Custom Tool

System Tools
Usually, these tools are installed and registered when you install ArcGIS. Although third-party developers can also create and register system tools. System tools are sometimes called function tools by developers.

Model Tools
These tools are created by user with Model Builder. Some of the tools in the system toolboxes are model tools.

Script Tools
Created by user with a scripting language editor (typically an enhanced text editor). Some of the tools in the system toolboxes are script tools.

Custom tools
Built by system developers and have

their own unique user interface for creating the tool. The ArcGIS Data Interoperability extension contains custom tools.

Scripting for ArcGis

ArcGIS 9 includes scripting support for many of

today's most popular scripting environments, such as:

Python, VBScript, JScript, Perl.

Python
Open Source
Object Oriented Multi-Platform Good Debugging Tools Installed with ArcGis ESRI samples provided

Python
Python is simple to learn because of its clean syntax and simple, clear concepts. Python supports object-oriented programming in an easy-to-understand manner. Documenting Python is easy because it has readable code. Complicated data structures are straightforward to work with in Python. Python is simple to integrate with C++ and Fortran.

Python can be seamlessly integrated with Java.

Python is free from the Web and has a widespread community.

Geoprocessing Concepts for Arcgis

Object: almost anything can be an OBJECT
Attributes Methods

Geoprocessor: component of ArcObjects

manages all the geoprocessing functions available within ArcGIS.
It is an object that provides a single access point and environment

for the execution of any geoprocessing tool in ArcGIS, including extensions.

Using the native arcgis scripting module, depending on version of software version you have.

Resources for Geoprocessing with Pyhton

Arcgis Desktop Help

Arcgis Libraries

www.python.org

How to start: Basic Concepts

Define you geoprocessing problem
Complex Problem

Divide and Conquer Write and draw a chart of required steps Conceptualize the problems and solutions Look at ArcGis Desktop Help Find tools in ArcGis

Concepts Continued
The easiest way to start is to create a model, and export

it to a python script. Look carefully at syntaxes

Concepts again
Each Tool has a scripting help associated with it

Tools Aliases

An Example of Geoprocessing
An Example: DSP data processing

Radar Cells (grids) DSP Example

DSP Example: Radar Cells and 50 Gages

Conceptualize the Problem

Extract radar Values According to Rain Gauge Locations

What are the available tools solutions within ArcGis

Radar Cells: Polygon Feature Class

Gages: Point Feature Class

Shapefiles

Conceptualize the Problem

Gages have - Ids - Coordinates

Radar Attributes

Hundreds of files similar to this one Records from two radars KEWX KDFX

What are the Tools we can USE ?

Analysis

Cartography
Conversion Management

Geostatistical Analyst
Spatial Statistics

Tools We can Use

To make sure that both radar shape file and

gages shape file are on the same Coordinate System

Tools We can Use

Define Projection Tool

Intersect Tool

Export Feature Attributes to ASCII

First Step: Create Your own ToolBox

Add Necessary tools to your own Toolbox

Second Step: Create a Model

My Model

Activate the Model

Test Run the Model

Check the results

Open the ASCII file

Strategy for Work

Divide to 3 Steps

Define Projection

Intersect
Export Attributes to ASCII file

Export to Script: Define Projection

Export to Script: Intersect

Export to Script: Export Attributes to ASCII

Syntax Arguments

Lets Look at the one script: Define Projection

Script Title and General Information

# ----------------------------------------------------------------# DefineProjection.py # Created on: Thu Apr 02 2009 04:48:01 PM # (generated by ArcGIS/ModelBuilder) # -----------------------------------------------------------------With PythonWin 2.5 IDLE

Comments are shown in red color

IDLE: Interactive Development Environment

Import Necessary Modules Comments Line

################ Imports ##################
import arcgisscripting as ARC import sys as SYS import os as OS import locale as LOCALE
ArcObjects

Operating System
File and Folders Path

The locale module opens access to the POSIX locale database and functionality

Create the Geoprocessor Object

# Create the Geoprocessor object GP =ARC.create(9.3)

Load Required Toolboxes

## Load required toolboxes... GP.toolbox = "management"
# Load required toolboxes... GP.AddToolbox( "C:/Program Files/ArcGIS/ArcToolbox/Toolboxes/ Data Management Tools.tbx" )

Script Arguments and Workspace

# script arguments inputWS = "C:\\testFolder\\test1" outputWS = "C:\\testFolder\\test2"
# set the workspace GP.workspace = "C:\\testFolder\\test1

Set the Script to go through a list of files

# Get a list of feature classes in the workspace. fcs = GP.ListFeatureClasses() for fc in fcs: # Process: Define Projection... Statement for Tool .Line 2 of statements .Line 3
List all files in the worksapce folder

Define the Tool Arguments and parameters

# Process: Define Projection... #coordsys = "Coordinate Systems/Geographic Coordinate Systems/North America/North American Datum 1983.prj

coordsys = "Coordinate Systems/Geographic Coordinate Systems/World/WGS 1984.prj

#GP.DefineProjection_management(fcs,"GEOGCS['GCS_WGS_184',DATUM['D_ WGS_1984',SPHEROID['WGS_1984',6378137.0,298.257223563]],PRIMEM['Green wich',0.0],UNIT['Degree',0.0174532925199433]]")

GP.defineprojection(fc, coordsys)

Add Debugging Messages

# Get the resulting messages and print them print fcs print GP.GetMessages()

It is important to print error messages and processes for DEBUGGING purposes

Same Procedure for the Intersect Script

# Load required toolboxes... gp.AddToolbox("C:/Program Files/ArcGis/ArcToolbox/Toolboxes/Analysis Tools.tbx") # Script Arguments inputWS = "C:\\testFolder\\test1" precip_gages_wgs84 = "C:\\testFolder\\test2\\precip_gages_wgs84.shp" outputWS = "C:\\testFolder\\test3" # Set the workspace gp.workspace = "C:\\testFolder\\test1"

Intersect Script: Continued

for fc in fcs:
# set the output name for each feature class to be the same as the input output = outputWS + "\\" + "kewx_" + fc # for each file in the list run the Intersect gp.Intersect_analysis(fc + ";"+ precip_gages_wgs84, output, "ALL", "", "INPUT" ) #get the resulting message and print them print gp.GetMessages()

Dealing with files in Python

Python has built in function to work with files; List of files in a folder Open and read files

Read all file at once Read line by line

Write to files Read lines, search inside lines Read bytes.. Create files for reading and/or writing

Searching for files in folders

Search for files and get a list.
# import required python modules

import sys, string, os, shutil

# Define the path of the serch directory or folder InputFolder = "C:\\testGP\\test1\\" OutputFolder = "C:\\testGP\\test3\\ ListFiles = open(OutputFolder + "\\" + MyList.txt", "w")

listfiles = os.listdir(InputFolder)
for fileNames in listfiles: print fileNames MyList.write(fileNames + "\n") print finished listing files

Reading and Writing in files

Create a new file:
open("C:\\testGP\\output.txt", "w")

Read entire file into a single string: S = input.read() Read next line:

S = input.readline() Read entire file into list of line strings:

L = input.readlines()

Write string S into file:

output .write(S) Write all line strings L into file: output.writelines(L) Manual close of file: output .close()

Export Attributes to ASCII Script

# Load required toolboxes... gp.AddToolbox( "C:/Program Files/ArcGis/ArcToolbox/Toolboxes/Analysis Tools.tbx ) gp.AddToolbox( "C:/Program Files/ArcGis/ArcToolbox/Toolboxes/Spatial Statistics Tools.tbx ) gp.AddToolbox("C:/Program Files/ArcGis/ArcToolbox/Toolboxes/Data Management Tools.tbx" ) # Script Arguments outputWS = "C:\\testFolder\\test4" inputWS = "C:\\testFolder\\test3" outputWS = "C:\\testFolder\\test4" outputASC = "C:\\testFolder\\test4 # Set the workspace gp.workspace = "C:\\testFolder\\test3"

Export Attributes Continued

# create a text file for combining and writing all outputs file1 = open( "C:\\testFolder\\test4\\kewxradar.txt", "w ) # set file2 as the feature class attributes exported to text file # create text files for all outputs with Zero counts and Rain Counts file3 = open( "C:\\testFolder\\test4\\ZeroCountKewx.txt", "w ) file4 = open( "C:\\testFolder\\test4\\RainCountKewx.txt", "w )

for fc in fcs:
# set the output name for each feature class to be the same as the input output = outputWS + "\\" + fc[0:18] + ".txt" # get the count of each intersection file if count>0 then export to ASCII count =gp.GetCount_management(fc)

Attributes Export: Setting the files names, Outputs, Tool parameter

if count >=1:
# set the timestamp for each file timestamp = fc + " " print output + " The count is = " + count # write the timestamp to a text file file1.write(timestamp ) #Process: Export Feature Attribute to ASCII... gp.ExportXYv_stats(fc, "value", "Space", output) #open the ASCII files, read and write its contents to another text file file2 = open (output,"r" ) text = file2.read() file1.write( " " + text ) file2.close()

else:
print output + "count for rainfall radar cells value = " + count + "nothing to export ) print " the script will proceed to the next file " file3.write( output + "\ t" + "Count = 0" + "\n " )

Export Attributes Continued

else:
print output + "count for rainfall radar cells value = " + count + "nothing to export ) print " the script will proceed to the next file " file3.write( output + " \ t" + "Count = 0 " + " \ n " )

file1.close() print gp.GetMessages()

Test your Script Test again

Python is case sensitive for variable names and strings
Indentation is a very sensitive issue in Python Note the difference in the use of forward slash \ for

system paths Double Forward Slashes \\ for workspaces paths

Test Again, run and SAVE your Module

So Lets give it a try!