Recently in Praxes Category

Back in March I led a workshop on geographic mapping and spatial analysis with R. I finally got around to running my notes and syntax through Sweave to create a single document. Click on the image below to access the workshop notes.

Frances Lawrenz is one of my advisors. She knew about my interest in spatially enable evaluation and put me in touch with Bob Tornberg, a graduate student in Educational Policy and Administration who was leading an evaluation for the Bell Museum of Natural History. I had been looking for an opportunity to apply mapping and/or spatial analysis to a micro-level evaluation setting, such as a classroom. Bob wanted to give the primary intended users (PIUs) of the evaluation some information about paths traveled by visitors to the museum's Touch and See Room. Spatially enabled evaluation sounded like a mutually beneficial approach, so we decided to collaborate. I'm glad that Bob involved me before data collection began because I was able to suggest data recording procedures that later facilitated the mapping and analysis. I don't think it would be appropriate to share the statistical results here, but I think it's okay to share one of the maps. I think the maps turned out well, and I'm looking forward to hearing the PIUs' impressions.

My last post demonstrated a structural equation model of complex sample data in . I attempted to replicate Laura Stapleton's example, but my standard errors were larger than expected.

I wrote to Laura, and she graciously reviewed my results. She attributed the standard error discrepancies to a change in IRT score variances. Specifically, the ECLS-K math and reading scores have been re-scaled since the original data set was published, resulting in larger variances in the currently published version. She supplied me with her version of the original public-use data set.

The updated results with the original data set are nearly identical to her published results. I feel confident now about using ECLS-B jackknife replicate weights for SEM in , and I learned a lot from attempting the replication.

I am using data from the Early Childhood Longitudinal Study (ECLS) Birth Cohort for my research assistantship with Judy Temple. I have an analysis in mind that will involve factor analysis and path analysis simultaneously (i.e., a structural equation model).

The ECLS-B data and other large microdata sets represent the population, offer good statistical power, and provide comprehensive measures, making them suitable for structural equation modeling. However, those advantages are often achieved through stratified cluster sampling, which nests participants within primary sampling units in order to ensure adequate representation of strata and hold down data collection costs. Moreover, individuals representing small groups in the population are oversampled, which requires analytically re-weighting those cases downward to reflect population proportions but not down-weighting sample sizes in the standard error calculations. Calculating standard errors under complex sampling conditions is not straightforward compared to simple random sampling.

Is it possible to fit structural equation models of complex sample data in ? Several statistical software programs, including the survey package, can perform standard analyses (e.g., means, generalized linear models) in a manner appropriate for complex sample data. However, hardly any programs offer the ability to fit structural equation models to such data. Using some guidance offered by John Fox, author of the sem package, and an excellent article by Laura Stapleton, I decided to give it a try with R.

Stapleton used data from the ECLS Kindergarten Cohort and two commercial statistical software packages to demonstrate a structural equation model that applies sampling weights and accounts for multistage sampling. Because the ECLS-K is publicly available and is free, I was able to attempt her jackknife example. As hoped, the replication yielded parameter estimates that were comparable to Stapelton's, as well as standard errors that were larger than the naive standard errors. However, my jackknife standard errors were consistently larger than Stapleton's. I don't yet know why they were so large, but it will be good practice for me to find out. It will also be good practice to replicate her example of bootstrapping standard errors. I welcome any feedback about this approach.

One of my research interests has been the geographic distribution of early student achievement at the school district-level. District-level outcomes apply to large areas that are often out of alignment with neighborhoods, cities, and counties where local information could benefit parents and local governments making early education decisions. Additionally, I suspect that learning outcomes vary spatially according to a stochastic process influenced only partially by district boundaries, which in the context of multilevel modeling might appear as autocorrelation among within-group errors or as spillovers resulting from the modifiable areal unit problem. Therefore, I have begun examining sub-district geographic distributions.

School level results go a long way towards providing local-level information, but what about early learning outcomes in between school locations, where children live? As shown in the map at right, school-level results can be used to interpolate outcomes between schools. Interpolation is the process of using known values at a set of locations to estimate unknown values at different locations. When estimating an unknown value, nearby known values are typically given more weight (i.e., made to be more influential) than distant known values. Interpolation may be thought of as a three-dimensional version of locally weighted scatterplot smoothing (LOWESS). LOWESS yields a line, while spatial interpolation yields a surface, both lacking an explicit functional form.

Click on the image to see a larger map created with the help of Finley and Banerjee's Multilevel B-spline Approximation (MBA) package.

Google's statisticians routinely use randomized experiments to improve their products (and profit), but did you know they also conduct quasi-experiments when random assignment is not feasible? I receive the American Statistical Association's (ASA) membership magazine called Amstat News. Daryl Pregibon, a Google statistician (or "engineer" as they are called internally), was invited to write about the company's statistical practices in the May issue. He writes that Google users can be randomly assigned to treatment conditions, but

"it is usually not possible to randomly assign advertisers to treatment groups due to contractual obligations and/or their willingness to be 'experimental units' for a service for which they are paying. In such cases, we ... use statistical methods that try to tease out causal inferences. Propensity score matching, inverse propensity weighting, and double robust estimates are some of the methods established in social and biological sciences currently in use at Google when randomization is not possible."

That approach mirrors best practices in quantitative evaluation. Randomized field trials are considered the gold standard for judging the degree to which a program or its components cause a desired outcome; when random assignment is not feasible, quasi-experiments provide a valuable alternative. Evaluation researchers rarely have as much control over conditions as Google's "engineers." Consequently, evaluators must rely more on quasi-experiments to "tease out causal inferences." Another key difference is that no matter how enormous a program data set may seem and no matter how many parameters a client might want an evaluator to estimate, those amounts will never reach the terabytes of data or the millions of parameter estimates that Pregibon describes as commonplace in life of a Google statistician.

By the way, my master's paper involved applying inverse propensity weighting to account for self selection into a local public school district. Does that mean a career as a Google statistician is in my future?

With spring semester in my rear view mirror, I found some time to use the maptools package to make a proficiency map that can be displayed in Google Earth. It's essentially a choropleth map in Keyhole Markup Language (KML) format. Google Earth takes visualizing educational outcomes to a whole new level. Distributing proficiency maps in KML format would make it easy for parents, school district employees, policy makers, and students themselves to explore their district's test scores and those of nearby districts. Additionally, KML proficiency maps could help evaluators of educational programs involve stakeholders and frame questions.

Try it for yourself. You can click on the image below to explore Minnesota's 3rd grade math proficiency results with Google Earth. After you've gotten your fill of zooming around the map, try the following:

• Click on a district to activate a pop-up window containing the district's name and results.
• Click on the "+" sign next to "2007 Minnesota Comprehensive Assessments (MCA-II)" under the Places sidebar. The district results will expand downward, showing results in tabular format with map links. Double-click a district name in the sidebar to zoom to that location.
• Use the Search -> Find Business section of the sidebar to find an after school tutoring program in a district of your choice.

Please leave a comment if you have any feedback about this approach.

Each year the Department of Educational Psychology holds Graduate Student Research Day to showcase research conducted by its graduate students. About 90 people attended this year. I was one of six students who delivered a presentation. My fellow presenters--Ruth Swartwood, Cengiz Zopluoglu, Alicia Ayodele, Ben Seipel, and Breanne Byiers--all did an excellent job. Many other students shared their research via posters, all of which were outstanding.

My presentation, "A spatial analysis of school district performance in Minnesota, demonstrating spatially enabled evaluation," consolidated some of my earlier posts to this blog and summarized the paper that I am scheduled to present at the American Educational Research Association (AERA) meeting in San Diego.

By looking to other social science disciplines that have become spatially enabled, I identified some ways that applied educational researchers can make better use of geographic mapping and spatial statistics. For example, we could enhance survey data and minimize respondent burden by spatially referencing the primary data and joining secondary data from the Census Bureau. Extending the promising uses, I spatially analyzed school district performance in Minnesota in 2007 and found the following.

• Neighboring school districts influenced performance to a small degree in limited instances (i.e., reading proficiency of third graders in poverty).
• Several school districts were found to have outperformed their neighbors, including Randolph, Medford, Orono, and Royalton.
• Six clusters of adjacent, correlated school districts exhibited low reading proficiency, suggesting they should collaborate to improve preschool and early elementary reading efforts.

You can click on the image to access the full presentation:

If was a band, and if that band had a greatest hits album, and if the album featured R commands instead of songs, then the following commands should be counted among the greatest hits (or the most obscure hits).

• windows(record=T) #place this near the top of your script to record all charts; enables scrolling through and compare charts with page up and page down
• oldpar <- par(no.readonly=T) #assign default graphical parameters to an object, to be called back later with par(oldpar)
• windowsFonts(Garamond = windowsFont("TT Garamond")) #add a new font family to the standard database
• file.choose() #allows you to choose a file interactively instead of specifying the path and file name in quotes
• describe() #a better alternative to summary(); from library(psych)
• qq.plot() #plots a quantile-comparison chart with confidence bands; in library(car)
• prplot() #a function in library(faraway) that produces partial residual plots
• Make.Z() #normalizes all variables in a data frame, available in library(QuantPsyc)
• ci.rc() #calculates a regression coefficient confidence interval, in library(MBESS)
• hccm() #another from library(car); calculates heteroscedasticity-corrected covariance matrices for unweighted linear models
• lmer() #a function in library(lme4) that fits linear mixed models
• identify() #use this function in library(graphics) to label points in a scatter plot

I am the teaching assistant for EPSY 8262 Statistical Methods II: Regression and the General Linear Model, taught by Dr. Jeffrey D. Long. is the statistical software that we use in the course. Installing and using R can be difficult at first, so I've gathered some helpful resources for students and others who are getting started.

• Installing R video tutorials by Milton Mayfield
• Statistics with R video tutorials by Mohammad Ehsanul Karim
• Contributed documentation in many languages
• R Reference Card by Tom Short
• Simple R by John Verzani
• Practical Regression and Anova using R by Julian Faraway
• Using R for Data Analysis and Graphics by John Maindonald
• R-help search engine (highly recommended) by Seyed Reza Jafarzadeh, which searches the mailing list archives.
• For spatial statistics (not pertinent to EPSY 8262), I created an R-sig-Geo search engine to retrieve information from those archives.
• R Commander graphical user interface (GUI) by John Fox
• Data Analysts Captivated by R's Power, a New York Times article by Ashlee Vance