Diego Valle-Jones's Blog

Municipios with missing persons anomalies

2015-03-26T00:00:00+01:00

The Secretaría de Gobernación just updated the missing persons database (RNPED) to include data up to January 2015. The database now contains information on 25,293 missing persons from the *fuero común* (local government) and 443 from the *fuero federal* (federal government).

To make it easier to understand the data and make it actionable, I plotted all the municipios containing outliers during the period from August 2014 (just before the mass kidnapping in Ayotzinapa) to January 2015 (the last date available). The chart includes data from both the fuero federal and común.

I used the AnomalyDetection R package to detect the anomalies.

Not surprisingly, Iguala, which saw the disappearance of 43 students on September 26, 2014 appears first on the list. The Mexican government should pay particular attention to these municipios if it wants to solve the missing persons problem in Mexico.

This is a cartogram of the rates of disappearances at the state level. Obviously, some states like Guerrero and Nayarit (zero disappearances in 2014) under report disappearances.

P.S. Code is available here

The most partisan first names

2015-02-09T00:00:00+01:00

I downloaded data from the Instituto Nacional Electoral to find out which names are the most overrepresented among the three main political parties in Mexico (PRI, PAN, and PRD). As was to be expected —names are markers of class, income, religion, and geographical regions— there are some major differences in which names the members of each political party bear.

The parties differ not only in their politics, but also in their internal structures and regions where they have support. The right wing PAN likes to keep its enrollment small and has 220,568 members or 3% of the total membership of the three major parties, the left wing PRD has 2,590,972 members or 33% of the total, and the PRI has 5,044,528 members or 64% of the total. And obviously, the political support of the parties varies by region, with not that much overlap between the PAN and PRD.

In the following charts I’m showing first names with at least a total count (for all parties) of a hundred and a party memebership of at least 30 (for the party displayed in the chart).

Notice how having more than one name is popular among members of the PAN, also that very few female names are overrepresented. The data you can download from the INE doesn’t include sex, but using an older dataset the membership of the PAN was 52% female a couple of years ago.

Note all the alternate spellings of Anayeli, Isabel, etc and how, unlike the PAN, people with two names are rare among the popular names.

I’ll just use this picture of the Guerrero Chimalli to illustrate the PRI:

In the previous charts I used the full first name as a single name (Luis Miguel is different from Miguel Luis) but we can also analyse the names as single entities:

Female names are more common among PANistas when analyzing each name as a separate entity

Again notice all the alternate spellings (Brayan/Bryan, Jocelyn/Jocelin, etc)

I also created a ternary plot so you can view how common the names are in each political party at the same time (remember the PAN has a very small membership)

I was curious about all the alternate spellings of Anayeli, though in the table below I’m only showing the most common ones —there literally are dozens and dozens of variations— and, well, PAN members just don’t like that name.

Name	Count for All Parties	Party	Count for Party	Expected Percentage	Percent Named Belonging to Party
NAYELI	1730	PAN	39	0.03	0.02
ANALLELY	387	PRD	278	0.33	0.72
ANAYELI	1449	PRD	1015	0.33	0.70
ANALLELI	110	PRD	74	0.33	0.67
ANA YELI	138	PRD	92	0.33	0.67
NAYELLI	134	PRD	88	0.33	0.66
MAYELI	115	PRD	74	0.33	0.64
NAYELI	1730	PRD	1084	0.33	0.63
NALLELY	837	PRD	501	0.33	0.60
YANELI	301	PRD	176	0.33	0.58
NAYELY	217	PRD	124	0.33	0.57
NAYELY	217	PRI	92	0.64	0.42
YANELI	301	PRI	121	0.64	0.40
NALLELY	837	PRI	319	0.64	0.38
NAYELI	1730	PRI	607	0.64	0.35
MAYELI	115	PRI	39	0.64	0.34
ANA YELI	138	PRI	44	0.64	0.32
ANALLELI	110	PRI	35	0.64	0.32
NAYELLI	134	PRI	42	0.64	0.31
ANAYELI	1449	PRI	426	0.64	0.29
ANALLELY	387	PRI	104	0.64	0.27

I figured it would be a good idea to count the percentage of first names that are only present in one party, and to my surprise the PAN has the highest percentage of unique names by far. I also measured the complexity of the names using Levenshtein and Soundex distances, and again the PAN had the most similarly spelled/sounding names.

Here are random samples of some unique by party names:

Names that are only present in the PAN
ADALGOT
BLERIATH
EDONE
EMIDGIO
EROILIA
GIBSON
HABANA
JAUDEL
JOLANI
GLORESBINDA
LIUVITZA
MYLTHON
NUYHIAVI
ORSIRIA
RICCO
VEUDY
YAMICLET
YECIA
YURINTZI
ZAADIA

Names that are only present in the PRD
BASELIA
MUYKEY
ERAMIS
GUILSONI
HAIRA
HEREDINDA
HORANA
IGILVIA
ISBELI
JASSINA
PAARIS
LLARIDEY
MAFAIDA
MARVITA
PRYTSCILIA
SARIDH
YARHELI
YESSMIT
YSAYCHU
ZIRIACO

Names that are only present in the PRI
AAGAR
ADIRAI
BEGAIN
BERENIZA
ELIOTZEL
ELISAMARIA
ENIZAEL
GLENIA
JARASED
MOSAISELA
NARCIDELIA
NAZDRY
POLEN
PONPOM
RUTHYALITH
SELENICO
SHADIRA
SICLALI
WUDRUM
YENIZEN

Hopefully your parents never gave you the misspelled name of a famous Mexican stripper!

Where people have unique first names they, unsurprisingly, also have unique last names

As you probably noticed the names play into some deep seated stereotypes in Mexican culture (some worse than others). Simply perform a search in Google Images for “Brayan” or visit El Deforma to see how they routinely use long names to make fun of the mirreyes. Brayan is in fact more common among members of PRD (even if unique names are more common in the PAN) and long names do seem to be more common among PANistas:

The longest name in the database belongs to GUADALUPE DEL SAGRADO CORAZON ESPINOSA DE LOS MONTEROS Y GOLZARRI, and I bet I don’t even have to tell you what party she belong to (PAN). Well, actually, it seems the name field at the time of entering the data in the INE database had a length limit and some names got clipped off, or abbreviated. I tried to somewhat clean up the data when possible.

Using a logistic regression to predict left wing or right wing party membership (I excluded the PRI since they only care about power and are corrupt, yay stereotypes!) gives exactly the results you would expect:

P.S. Code is available from GitHub

The 74 most violent cities in Mexico (2013)

2015-01-14T00:00:00+01:00

I’ve updated the mxmortalitydb package to include 2013 data. This data only package includes all injury intent deaths (accidents, homicides, suicides, and unspcified intent) that were registered in Mexico from 2004 to 2013. You can use the package to calculate changes and trends in homicide rates in the most violent metro areas (or large municipios) or at the national level.

You can also use the package to explore the deaths whose intent was left unspecified (note the large increase in firearm deaths in 2011, the year Mexico had the highest homicide rate in recent history)

And make pretty plots of the most violent metro areas or large municipios which are not part of a metro area (in Mexico by definition a metro area has to consist of more than one municipio, so it is worthwhile to include single municipios with large populations). I should also note that the database only includes deaths where a death certificate was issued, so if there are clandestine mass graves that haven’t been found (as was the case recently in Iguala) or where the deaths somehow were not registered (as was the case with the Taxco mass grave) then the deaths are not included in the counts.

While violence has decreased in recent years it still has a long way to drop before it reaches the levels seen in 2006.

Note the mixed trend in Michoacán.

Do note that the charts of homicides from 2006 were made with the 2010 population according to the CONAPO, so the homicide rate was underestimated by a little bit. Also, rather than using the raw homicide numbers I adjusted them by classifying deaths of unknown intent as described in the mxmortalitydb package and this post.

The code for generating the charts is available from GitHub

Nobody lives here

2014-12-10T00:00:00+01:00

This is an interactive map of all rural AGEBs that contain no localities and thus are empty of permanent residents. If you compare it to the US version, the total area with no people living inside it is much smaller since the data in Mexico is only available at the AGEB level (kind of like Census Tracts), whereas the data in the US is available at the Census Block level which are much smaller. Since there are no polygons available for rural localities, only points, I counted all rural AGEBs that contained a locality as being inhabited.

With a relief map

With dense vegetation

By type of climate

With protected areas (Área Nacional Protegida)

The code is available from GitHub

HoyodeCrimen.com - Crime information for the Distrito Federal

2014-11-10T00:00:00+01:00

Crime information for the Federal District now has its own website with updated data

https://hoyodecrimen.com/en/ - English version
https://hoyodecrimen.com/ - Spanish version

I’ve also added a trends section where you can look up which cuadrantes experienced a rise in crime

There seems to a be big problem with car robberies near where the new airport will be built

Since the crime data comes from FOIA requests to the SSPDF (Mexico City Police), I’ve added free email announcements to keep you informed of when new data is available:

Notifications in English
Notifications in Spanish

There’s even an API with lots of cools stuff:

Service	Action	URI
Point in Polygon	Given a longitude and latitude return the corresponding cuadrante and sector
Time Series	Crime counts ordered by month of occurrence for a cuadrante or sector
List Cuadrantes or Sectores	Sum of crimes that occurred in a cuadrante or sector for a specified period of time
Top Most Violent	A list of the cuadrantes and sectors with the highest rates (sectores), crime counts (cuadrantes) or change in crime counts (cuadrantes)
DF data	A time series of the sum of all crimes that occurred in the Federal District
Enumerate	Get a list of the names of all cuadrantes, sectores or crimes

P.S. Yes, I know the Spanish translation is not complete, but I’ll finish it someday
P.P.S. The code is available at GitHub. It was built with Python + Flask + PostgreSQL + PotsGIS + Redis + D3 and an unholy combination of jQuery and AngularJS

Important notice regarding crimenmexico

2014-09-25T00:00:00+02:00

The website from which crimenmexico downloads crime information hasn’t been responding for quite a while now. The SESNSP server’s uptime has never been good but it has never stayed down for so long.

which I guess means that Secretariado Ejecutivo del Sistema Nacional de Seguridad Pública has suspended its open data policy of providing crime information. Apparently crime info is now only downloadable from a couple of zip files, but SESNSP made sure to:

Change the names of the crimes to include whitespace at the end (e.g “ROBOS” to “ROBOS ”)
Not include the INEGI code for each state and municipality, thus making it really difficult to merge with other data
Spell state names both with and without accents (YUCATÁN and YUCATAN)
Include subtotals (e.g. total crimes with violence) at the municipality level but not at the state level
Change the file format from csv to xlsx (Excel)

It’s almost as if they didn’t want anyone analyzing the data. It used to be that crimenmexico processed all the information in an automatic manner and made new data available immediately, but with these changes that has become impossible (I’m betting next month there will be more changes). There will now be a delay of a few weeks or so while I figure how to process the data.

At the webpage for downloading the crime information the SESNSP claims that [emphasis added]:

La presentación de los datos sobre incidencia delictiva en este portal electrónico obedece a los principios de acceso a la información y transparencia y a la adopción de las mejores prácticas internacionales en gobierno abierto.

If you disagree you can contact the SESNSP at: cni@secretariadoejecutivo.gob.mx

Crime in Mexico City at the cuadrante delictivo level

2014-05-27T00:00:00+02:00

Update: This post has been superceded by hoyodecrimen.com

Click on the image to visit the interactive visualization

I recently made a FOIA request to the SSPDF requesting information on crime at the cuadrante delictivo level (in-between size a municipio and a manzana), and to my surprise they actually answered with a PDF file containing homicides, robbery to businesses with violence, and car robbery with and without violence, from Jan 2013 to April 2014. I also asked for data on extortion, kidnapping, etc but the SSPDF claimed not to have any records on those crimes.

To be able to display the data on a map I scraped the polygon coordinates of each cuadrante from the find the cop assigned to your quadrant website. The data was supplemented with population figures obtained by using a point-in-polygon algorithm from the 2010 Census at the manzana level.

The data for April 2014 was clearly incomplete and I excluded it from the visualizations. The numbers provided by the SSPDF compare favorably to those provided by the SNSP, though you have to remember these are averiguaciones previas (reports) and that they undercount the number of victims, especially for the crime of homicide.

Crime (2013)	SNSP	SSPDF
Homicidio doloso	749	899
Robo a negocio C/V	4,239	4290
Robo de vehiculo automotor C/V	5,223	5,211
Robo de vehiculo automotor S/V	12,056	12,014

The data at the cuadrante level is visualized as counts because the daytime population of a cuadrante is probably different from the number of people who actually live there.

Quadrants
Sectors (Each sector contains many quadrants)
Website

P.S. The code and data are on GitHub

Analysis of the UNAM’s entrance exam

2014-04-07T00:00:00+02:00

The UNAM is Mexico’s biggest and most important university. To enter it students must either take an exam or graduate from a high school run by the UNAM in less than 4 years with a grade point average of at least 70% (although some majors like medicine require 90% for pase directo). The admission exam is given twice a year, in February and June, and any student from any high school with at least a grade point average of 70% can take it. If the student meets the requirements for entering the UNAM, passing the exam guarantees him admission. The exam has 120 questions.

Depending on the student’s choice of major, the admission exam emphasizes one of four basic areas of study. Let’s say you want to study math: in addition to being tested on the topics every high school student is supposed to know, you’ll get a couple of extra questions about integration by parts; if you want to study biology the extra questions will be about the the Krebs cycle; if you want to study philosophy the exam will probably include extra references to the great works of literature; and if you want to study a social science you’ll be asked about the differences between a grand macchiato and a caffè latte.

Click on the chart to visit the interactive version

Apart from UNAM’s main (and most prestigious) campus of Ciudad Universitaria (CU), the university has several satellite campuses in the Mexico City metro area as well as many others across Mexico. In this post I will only analyse those located in Mexico City, and only for the scholarized system (sistema escolarizado) where the students have to actually sit in a classroom (the UNAM also offers remote TV/Internet classes and under an open system). The admission exam to the UNAM is quite competitive and only a small percentage of those who apply actually get in.

Date	Location	Percentage admitted	Applied	Completed Test	Admitted
2011-06	CU	8.6	30,615	27,515	2,626
2011-06	Not CU	9.1	32,215	29,667	2,916
2012-02	CU	5.2	61,262	55,793	3,192
2012-02	Not CU	6.4	56,012	52,287	3,597
2012-06	CU	5.1	30,944	28,084	1,573
2012-06	Not CU	10.1	32,741	30,354	3,299
2013-02	CU	5.4	63,562	59,425	3,424
2013-02	Not CU	6.7	56,348	53,547	3,800
2013-06	CU	6.3	29,872	26,052	1,868
2013-06	Not CU	13.8	33,744	30,403	4,656

This year Harvard sent out 2,023 offers out of 34,295 applications for an admission rate of 5.9%, which is exactly the same overall admission rate as the UNAM (CU) from June 2011 to June 2013 (though, of course, Harvard has much much lower standards in who it admits).

Admissions by major

Click on the chart to visit the interactive version

Admissions by area

Click on the chart to visit the interactive version

I scrapped all admission results from the UNAM’s website from June 2011 to June 2013. There were some problems with the data at the UNAM’s end since the listings didn’t always match the summary statistics included in the web pages. If you visit the results for Historia del Arte you’ll see that the summary statistics claim that zero students applied to take the test, but the listing includes 3 students (given the test scores of infomatics students I’m not surprised by the mistakes). Anyways, this kind of mismatch was rare and didn’t involve that many students. I took the actual listing to be definitive. In addition the results sometimes include data from students whose result is Cita para aclarar situación escolar, I simply interpreted this as being a missing value.

Physical Sciences, Mathematics and Engineering

Biological Sciences and Health

Social Sciences

Humanities and Arts

Highest scoring majors in each area

Changes in median admission scores from exam to exam (first differences)

Median scores each year

(a standard deviation is 17 points)

One big problem with the admission process at the UNAM is that it doesn’t use Gale-Shapley and thus students have an incentive to not reveal their true preferences:

Imagine a student who is deciding between studying ingeniería mecatrónica and ingeniería mecánica, because mechatronics is very hard to get into, and students only get to apply to one major at each exam, he decides to list mechanical engineering as his choice even when there was a chance (albeit not as big) that he would have been admitted to mechatronics.

Click on the chart to visit the interactive version

Another consequence of not using Gale-Shapley is that some students who do list their true preference are rejected in favor of lower scoring students (this is probably a really big deal).

Click on the chart to visit the interactive version

Imagine a student who lists his or her true preference but who would be willing to entertain studying at an alternative campus with a lower admission score (international relations at CU vs international relations at FES Aragón) or a similar major with lower admittance requirements (international relations vs political science for example) who is rejected from his first choice but obtains a test score higher than the minimum requirement for his alternative.

We can see in the above chart how the percentage of students admitted increases at the lower requirement campuses during the June exam, but there is no such increase at the main CU campus. This is probably a consequence of students who had been rejected after the February exam having another go at being admitted at the lower requirement satellite campuses.

The easiest way to remedy this would be to copy the COMIPEMS exam —which the UNAM helped design— and allow students to list more than one major/campus when applying. Obviously there are a lot of variations and complications the UNAM could use to make its admission process better, for example they could start admitting students from different areas than their first choice major if they had really high test scores. I’m sure the ~~applied math and computing~~ international relations students are up to the task of designing an admittance mechanism that satisfies Gale-Shapley.

There’s also a positive correlation between average test scores and the starting salaries of university graduates (of any university in Mexico City, not just the UNAM) in a poll conducted by Reforma in Mexico City.

P.S. I bet the Escuela Nacional de Artes Plásticas is full of rich kids
P.P.S. Visit the companion website full of interactive charts
P.P.P.S. Source code

Interactive map of the drug war in Mexico - 2012

2014-01-07T00:00:00+01:00

It’s that time of the year again when I update the interactive map of the drug war in Mexico. The map now uses 30 day months to calculate homicide rates and the new CONAPO population estimates. As usual there is also a Spanish version. All deaths registered without a date of occurrence were assumed to have occurred in the same month they were registered and all deaths without a municipio of occurrence were assumed to have taken place where they were registered. Various events are worth checking out:

Zacatecas/Fresnillo
The increase in violence in Nuevo Laredo, which proved to be much worse than thought
San Fernando
The Michoacán/Jalisco border (where a mass grave with over 60 bodies was recently found)
Western border vs Eastern border
More bodies from the Durango mass graves that apparently were not reported
Homicides for the whole sexenio

For the adjusted homicide data I did the following:

I again classified all deaths of unknown intent into accidents, suicides and homicides based on the age, sex of the victim and injury mechanism by which the death occurred. The method I used is similar to the one I used in my Juarez post (with a penalized regression for Sinaloa and knn for the rest of Mexico). For example, if someone told you to guess the intent of the death of a 70 year old woman who died in Merida by motor vehicle, you’d probably guess it was an accident. If you had to guess the intent of the death of a young male in Acapulco who died by firearm, you’d probably think it was an homicide.

I also recoded all deaths by legal intervention as homicides in the adjusted version since it seems legal intervention deaths are not always classified correctly (the 19 deaths from the shootout between the army and the narcos in General Treviño and Mier were recorded as homicides).

The problems with the data in the Federal District and Sinaloa were also corrected.

Since firearm accidents in Baja California went from 6 in 2006 to a 100 in 2007 and again to 6 in 2008 I reclassified all firearm accidents as if their intent were unknown in 2007.

I also added the mass grave in Taxco and the one in San Fernando since they either don’t appear or are incomplete in the dataset from the INEGI.

P.S. You can download a csv with the data from the projects page

Changes in homicide rates

2013-12-18T00:00:00+01:00

More examples of using the mxmortalitydb package! Changes and trends in homicide rates in the most violent metro areas or big municipios. There was a big increase in violence in Nuevo Laredo. For comparison the homicide rate in Chicago (metro area) was 8.2.

library(mxmortalitydb)
library(stringr)
library(plyr)
library(ggplot2)
library(grid)  ## needed for arrow

plotMetro <- function(metro.name, metro.areas) {
    ## Plot the homicide counts in a metro area or municipio metro.name - name of
    ## the metro area to plot metro.areas - data frame containing a list of metro
    ## areas in the same format as the metro.area dataframe from mxmortalitydb

    ## data.frame metro.areas contains the 2010 CONAPO metro areas
    df <- merge(injury.intent, metro.areas, by.x = c("state_reg", "mun_reg"), 
        by.y = c("state_code", "mun_code"))
    ## Yearly homicides in Mexico City, by state of registration
    df2 <- ddply(subset(df, metro_area == metro.name & intent.imputed == "Homicide"), 
        .(year_reg), summarise, count = length(state_reg))

    ggplot(df2, aes(year_reg, count)) + geom_line() + labs(title = str_c("Homicides (plus deaths of unknown intent classified as homicide) in\n", 
        metro.name)) + ylim(0, max(df2$count)) + ylab("homicide count") + xlab("year of registration") + 
        theme_bw()
}

plotChanges <- function(df, metro.areas, country.rate, years) {
    ## Plot of rates and trends df - injury.intent dataframr metro.areas - data
    ## frame containing a list of metro areas in the same format as the
    ## metro.area dataframe from mxmortalitydb country.rate - rate to show as a
    ## gray dotted line years - start and end year to compare changes

    ## Where the municipio where the death occurred is unknown use the municipio
    ## where it was registered as place of occurrance  df[df$mun_occur_death == 999, ]$mun_occur_death <- df[df$mun_occur_death == 
        999, ]$mun_reg

    ## Counts of homicide by state and municipio
    df <- ddply(subset(df, year_reg %in% years & intent.imputed == "Homicide"), 
        .(state_occur_death, mun_occur_death, year_reg), summarise, count = length(state_reg))
    ## Merge the counts with our fake metro areas
    df <- merge(df, metro.areas, by.x = c("state_occur_death", "mun_occur_death"), 
        by.y = c("state_code", "mun_code"))
    ## Now get the counts by metro area (which may contain more than one
    ## municipio)
    df <- ddply(df, .(metro_area, year_reg), summarise, count = sum(count), 
        population = sum(mun_population_2010), rate = count/population * 10^5)
    ## We are only interesed if the metro area at some time had a homicide rate
    ## of at least 15
    df <- subset(df, metro_area %in% subset(df, rate > 15)$metro_area)
    ## Make sure the dataframe is ordered by metro and year
    df <- df[order(df$metro_area, df$year_reg), ]
    ## Order the chart by homicide rate in 2012
    df$metro_area <- reorder(df$metro_area, df$rate, function(x) x[[2]])
    ## Data frame for the arrow structure
    arrows <- ddply(df, .(metro_area), summarise, start = rate[1], end = rate[2], 
        metro_area = metro_area[1], change = ifelse(rate[1] >= rate[2], "decrease", 
            "increase"))

    ggplot(df, aes(rate, metro_area, group = as.factor(year_reg), color = as.factor(year_reg))) + 
        geom_point(aes(size = log(count))) + labs(title = "Homicide (plus deaths of unknown intent classified as homicide) rates and trends") + 
        scale_size("number\nof\nhomicides", breaks = c(log(50), log(500), log(3000)), 
            labels = c(50, 500, 3000)) + geom_segment(data = arrows, aes(x = start, 
        y = metro_area, xend = end, yend = metro_area, group = change, color = change), 
        arrow = arrow(length = unit(0.3, "cm")), alpha = 0.8) + scale_color_manual("year\nand\ntrend", 
        values = c("gray", "black", "blue", "red")) + ylab("metro area or municipio") + 
        xlab("homicide rate") + # scale_x_log10()+
    geom_vline(xintercept = country.rate, linetype = 2, color = "#666666") + 
        annotate("text", y = "Tapachula", x = 25, label = "country\naverage\n2012", 
            hjust = -0.1, size = 4, color = "#666666") + theme_bw()
}

## Let's treat the big municipalities which are not part of a metro area as
## if they were one rename big.municipios to merge with metro.areas
big.municipios2 <- big.municipios
names(big.municipios2) <- c("state_code", "mun_code", "mun_population_2010", 
    "metro_area")
metro.areas.fake <- rbind.fill(metro.areas, big.municipios2)

Changes from 2011 to 2012 and from the year before the drug war was declared to 2012:

plotChanges(injury.intent, metro.areas.fake, 24.5, c(2011, 2012))

plotChanges(injury.intent, metro.areas.fake, 24.5, c(2006, 2012))

Interesting that Tijuana had about the same homicide rate in 2012 as in 2006. The rest of the violent metro areas/large municipios which saw decreases are in Michoacán. Sadly, it doesn’t look like pattern will hold in 2013 (according to crimenmexico Michoacán is experiencing a surge of violence and is at a maximum)

Do note that the charts were made using the 2010 population according to the CONAPO that comes with mxmortalitydb, so homicides in 2012 were overestimated by a little bit and underestimated by a little bit in 2006. Also rather than using the raw homicide numbers I adjusted them by classifying deaths of unknown intent.

ll <- list("Acapulco", "Nuevo Laredo", "La Laguna", "Chihuahua", "Tecomán", 
    "Juárez", "Culiacán", "Victoria", "Hidalgo del Parral", "Zihuatanejo de Azueta", 
    "El Mante", "Ciudad Valles", "Ciudad Valles", "Durango", "Cuernavaca", "Zacatecas-Guadalupe", 
    "Monterrey", "Piedras Negras", "Mazatlán", "Veracruz", "Tijuana", "Guadalajara", 
    "Tepic", "Coatzacoalcos")
names(ll) <- ll  # make lapply print the names of the metro areas
lapply(ll, plotMetro, metro.areas.fake)

## $Acapulco

## 
## $`Nuevo Laredo`

## 
## $`La Laguna`



## 
## $Chihuahua

## 
## $Tecomán

## 
## $Juárez

## 
## $Culiacán

## 
## $Victoria

## 
## $`Hidalgo del Parral`

## 
## $`Zihuatanejo de Azueta`

## 
## $`El Mante`

## 
## $`Ciudad Valles`

## 
## $Durango

## 
## $Cuernavaca



## 
## $`Zacatecas-Guadalupe`

## 
## $Monterrey

## 
## $`Piedras Negras`

## 
## $Mazatlán

## 
## $Veracruz

## 
## $Tijuana

## 
## $Guadalajara

## 
## $Tepic

## 
## $Coatzacoalcos

Check out the source code as an R markdown file.

An R package with all injury intent deaths registered in Mexico 2004-2012

2013-12-17T00:00:00+01:00

This is a data only package containing all injury intent deaths (accidents, suicides, homicides, legal interventions, and deaths of unspecified intent) registered by the SSA/INEGI from 2004 to 2012. The data source for the database is the INEGI. In addition the data was coded with the Injury Mortality Matrix provided by the CDC.

The package is only available from github, to install:

devtools::install_github(“diegovalle/mxmortalitydb”)

be sure to visit its github page to find out more!

Continuously updated crime information

2013-09-30T00:00:00+02:00

I’ve put together a simple website to keep people up to date on the latest crime information. The site polls the SESNSP website many times a day to see if there is new crime information, and if there is, it cleans up the data and makes it available for download.

https://elcri.men/en/csv/ (Engish)
https://elcri.men/es/csv/ (Spanish)

There’s also a simple report based on the data. In the future I hope to also include homicide data from the INEGI, ~~and municipio population from the CONAPO~~ (the data now includes municipio population and information for all state level and federal crimes), so the data is bound to change at some point. If you’d like to see another crime or municipio added to the data or report feel free to contact me.

For major updates or announcements follow me on twitter

P.S. Keep in mind that the numbers refer to police reports and not number of victims.

State population datasets 1990-2030

2013-08-28T00:00:00+02:00

Cleaned up Mexican population estimates by five-year age groups and gender 1990-2030 (mid-year) at the state level:

Original data from the CONAPO website

The best high schools in Mexico City (COMIPEMS analysis)

2013-07-31T00:00:00+02:00

Click on the map to visit the interactive version

In Mexico City all ninth graders who want to enroll in a public high school (grades 10 trough 12 in Mexico) must take an exam called COMIPEMS. In order to ensure students are assigned in a fair way to a school of their choice the following steps are taken:

Before the test students must rank up to 20 schools in order of preference.
All high schools rank the students based solely on the score each students gets in the COMIPEMS exam, preferring students who score higher (no allowance is made for diﬀerent skill categories, well, except for the UNAM and IPN which also requires a GPA of at least 70%).
After the test are graded a computer orders the students by test score and proposes to match each student with his first school choice. If the student has the required GPA and the school he proposed to hasn’t filled up then he is accepted. In a second step each student who has been rejected proposes to their next school choice. Each school that still has free space then considers the students until they reach capacity and so on until all student choices are exhausted.

There isn’t a lot of information on how the COMIPEMS handles ties, ~~but if they use a random algorithm to break ties~~ at least in 2001 they used middle school GPA to break ties, and if we ignore the limited number of schools students are allowed to list, then COMIPEMS would be equivalent to a student-optimal version of Gale-Shapley.

In 2013 out of 310,163 student who registered for the test, 280,096 showed up, out of those, 33,695 (12.0%) were not admitted to any school because of the limited number of options they specified. But they can always apply to one of the low quality schools with excess capacity. It would probably be a good idea to encourage students to submit long lists that contain less preferred but still acceptable options.

In fact, the advice given at the COMIPEMS website is wrong (if indeed the admission process follows Gale-Shapley) since it specifically warns students to limit their options in the FAQ:

What is the maximum allowed number of highly demanded schools a student can list?
There isn’t a maximum number of options; however, it is recommended to choose a maximum a four, this is because highly selective schools can fill up quickly and it is very unlikely that you’ll be assigned a place after your fourth option.
¿Cuál es el número máximo de opciones de alta demanda que se puede seleccionar?No existe un número máximo; sin embargo, se recomienda elegir un máximo de cuatro, debido a que estos planteles llenan rápidamente sus lugares disponibles y es poco probable quedar asignado en una opción de alta demanda que se selecciona después de la cuarta preferencia.

Under Gale-Shapley it is always optimal to list your true order of preferences regardless of whether your options are likely fill up. If Prepa 6 is your true fifth choice then that’s what you should list it as, and if you prefer the CONALEP Neza to not studying high school then that’s what you should list as you twentieth choice.

It would also probably be a good idea to allow schools to rank students based on different skills, some schools specialize in math, other in verbal skills. Perhaps even give priority to students with a sibling already attending one of their preferred options to make life easier for the parents.

For some reason the 2013 scores went up compared to 2011. Even when taking into account that nowadays high school is mandatory and everyone is guaranteed a place no matter how low their scores. With a guaranteed spot I can see how the big discontinuity at score 31 would disappear in 2013 with some students slacking off since they are now guaranteed a place, but there was also an increase in the right side of the curve, so students this year were overall smarter (an improvement of 7 points is a lot).

2013

Min.	1st Qu.	Median	Mean	3rd Qu.	Max.
2.00	55.00	70.00	70.51	86.00	127.00

2011

Min.	1st Qu.	Median	Mean	3rd Qu.	Max.
6.00	49.00	63.00	65.32	80.00	127.00

Both the 2011 and 2013 tests had a standard deviation of 20.4

The best public high schools in Mexico City:

With two exceptions, the top spots are schools run by the UNAM or the IPN. I wonder what the Preparatoria Oficial 99 is doing to attract so many smart students? If you look at the map the rest of the zone it is located in (Coacalco) has quite low scores.

In Mexico, high schools can be of three types: Bachillerato Profesional which is just like regular high school, you take courses in math, literature, and so on; then there is the Bachillerato Tecnológico where in addition to taking the normal high school subjects the students end up taking extra classes to get a certificate as an administrative assistant, computer technician and so on; finally, there is also the Carrera Técnica where no regular high school classes are taken and the students just end up with their certificate in graphic design or whatever. That big bump in the Bachillerato Tecnológico scores is due to the CECyTs run by the IPN which tend to be quite good.

By comparing the histograms of scores of the State of Mexico and the Federal District it looks like the DF is stealing all the smart people from the State of Mexico. Sadly, there just aren’t many good schools in the state of Mexico, and students have to travel all the way to the Federal District to get an education in accordance with their abilities.

By university affiliation

I wonder why the Preparatoria Agrícola Chapingo didn’t take part in COMIPEMS (it is considered quite good)?

We can also investigate whether there is a relationship between ENLACE —a test given to students in their last year of high school that divides them into insufficient, elementary, good and excellent based on their level of knowledge— and COMIPEMS quantiles (the black lines are loess):

Note that the CECyT 9s low percentage in the ‘good’ category is due to its high proportion of ‘excellent’ students as seen in the chart below (don’t forget to hover over the dots):

Bonus points if you guess which CONALEPs are cheating on their ENLACE tests!

When ~~El Diario Oficial de los Mirreys~~ Reforma ran a special section on the best high schools in Mexico City it made sure to exclude public schools from its rankings. Using ENLACE it is pretty easy to see how public schools stack up to private schools:

Cleary the high schools run by IPN are very good, and when we take into account that the high schools run by the UNAM do not take part in ENLACE, it is likely that many of the best students in Mexico City attend public schools.

The top two schools compared:

Since the CECyT 9 is the highest ENLACE scoring school in all of Mexico, the two best schools in the whole country are very likely public! Any ranking that excludes public schools is not based on any measure resembling school quality. Now if you wanted a ranking of the best schools for papalords…

Be sure to read the fine print when a school claims to be the best one! Source: Reforma

The guys at the CECyT 9 are clearly doing something very well and they may indeed be adding value to the education of their pupils rather than simply taking in the best students. In the interactive chart of ‘excellent’ ENLACE students it is clear they are way above what you would predict from their COMIPEMS score. It would be nice if Prepa 6 would participate in ENLACE to be able to compare results. This certainly merits further research to see if the methods they use can be replicated at other schools.

Well, upon further inspection this probably is due to a selection effect since the math scores of Hispanics born in the US are similar to those of Hispanics who weren’t born in the US

The CECyT 9 is also the Mexican school with the most medals in the International Olympiad in Informatics, so it is overall very distinguished.

From all this I conclude it is no big deal if the students from the UNAM-run high schools get automatically admitted to the UNAM’s university system (if they also maintain an average of 70% and finish in at least 4 years). Ok, they could change the requirement to graduate in 3 years, still they are already very likely to be able to pass the admission exam based on their high test scores! But there is in fact one change the UNAM could do to improve dramatically the quality of its student body which will be the topic of my next post.

The code to replicate my analysis is available on GitHub

P.S. Yes, I’m fully aware the students at the top public schools are richer than average, and that in fact the subsidy they receive by attending public school is regressive. But they are not as rich as those attending the top private schools and I get to joke about it.

Shapefiles of Mexico (AGEBs, Manzanas, etc)

2013-06-24T00:00:00+02:00

My post on how to download electoral shapefiles proved to be very popular, so I decided to create another script to download shapefiles of census geographies. This is all data that is freely available from the INEGI website, it’s just that they make it really hard to find, it’s only downloadable as a compressed windows executable, and the download links for the files tend to change every few months.

The shapefiles are encoded as codepage 1252 and were reprojected to WGS84. All data corresponds to the 2010 census.

Now for the ugly bits: You’ll need a recent version of innoextract, one that can work with files created by version 5.5 of Inno Setup (yes, the INEGI distributes shapefiles as windows installers), the one in Ubuntu universe is not recent enough. The version available from this repository should work.

sudo add-apt-repository ppa:arx/release
sudo apt-get update
sudo apt-get install innoextract
chmod +x download-census-shp.sh

~~As of now, this script has not been tested on Windows and Mac systems, only on Ubuntu.~~ This script has been tested on Ubuntu, on a Mac you should try and use ‘brew install wget’ and ‘brew install rename’ before running it, and on Windows use cygwin.

The script will create a directory called shps where all the shapefiles are located, if something goes wrong when downloading be sure to delete it and try again. Once you finish running the script you’ll end up with about 5.5 gigabytes of shapefiles containing all available levels of geographical granularity for the 2010 Mexican census.

Census Geographies:

States -> Municipios (Counties) -> Localidades (Polygon if urban otherwise point) -> AGEB (Census Areas - Urban only) -> Manzana (Census Blocks - Urban only)

Plus, you’ll also get a bunch of shapefiles of parks, rivers, schools, etc.

Municipios (Counties)

Localidades (Urban Polygons and Rural Points)

AGEB (Census Areas)

Manzanas (Blocks)

Ejes viales (Streets)

Servicios A (polygons and lines)

Camellón, Metro Station, Hospitals, Parks, Schools, etc

Servicios L (polygons and points)

Bridges, Railways, Tren Metropolitano, Lindero, etc

Servicios P (points)

Churches, Schools, Hospitals, etc

Here’s the script you need to run to download the shapefiles:

P.S. You can contact the INEGI and tell them what you think of the way they distribute data.
P.P.S. If you need shapefiles of elevation curves, archeological sites, rivers, etc, try this other script

Update: You can also use a docker image to run the script without having to install anything

docker pull diegovalle/scince2010-docker
# shared directory to store the output
mkdir /tmp/scince2010
# run ./download.sh within the container and the files should be ready
# in /tmp/since2010
docker run -v /tmp/scince2010:/shapefiles/shps -i -t diegovalle/scince2010-docker

Maps of Mexico City

2013-04-09T00:00:00+02:00

Just some maps of the Mexico City metro area based on the 2010 census. You can search by address and switch to satellite view. If the maps seem slow I suggest you use chrome.

To calculate the margination index I used the first principal component of the percentage of the population with at least one year of college, households with a dirt floor, households without electricity, etc

When it comes to attracting young women to their neighborhoods the mirrreys beat the hipsters hands down (though the suburbs are probably not the best place to actually meet women).

I also wonder what’s up with all the non religious persons in La Condesa and Roma Norte. The UNAM I can understand, I’m guessing I didn’t get the memo on the hipster heresy.

There’s also a negative relationship between car ownership and population density, though obviously a lot more variables are involved. The rich areas and the outer ring surrounding the urban center are low in population density, while the poor mostly live in high population density areas.

GitHub

Visualising Change in Presidential Votes

2013-03-05T00:00:00+01:00

Click on the image to visit the interactive version (Spanish)

I liked this visualization by the New York Times of the US presidential election so much I decided to recreate it using Mexican data, and in the process added a slope graph. So go visit my new visualization of the change in presidential vote from 2006 to 2012 at the municipality level. I recommend using chrome to view the visualisation, there’s also a Spanish version.

The PRI gained votes in almost all the big municipalities save for Villahermosa (Centro). The home state of AMLO.

Not everyone was happy with the post-election protests of the PRD candidate. Interestingly, he did gain votes in several of the municipalities that make up the Monterrey and Guadalajara metro areas.

People in Villahermosa must really hate the PRI. I was not surprised to see the PAN lose votes in Culiacán, Juárez, Chihuahua, Cuernavaca, Tijuana, Zapopan, Guadalajara, Monterrey…

Scatter plots of votes in 2006 vs votes in 2012:

P.S. The code for the interactive chart is here

Download electoral shapefiles of Mexico (secciones, distritos, and colonias)

2013-02-27T00:00:00+01:00

I’ve created a set of scripts to download and recode shapefiles from the IFE and INEGI. Once you run the scripts you’ll find in the map-out directory:

distritos: Shapefile of the electoral distritos (districts)
secciones-inegi: Shapefile of electoral secciones (precincts) with both the ife and inegi codes for the municipalities each seccion belongs to
estados: Shapefile of the Mexican states according to the INEGI
localidades: Shapefiles of the rural localities and the polygons of the urban ones
municipios: Shapefile of the municipalities of Mexico according to the INEGI
rdata-secciones: serialized secciones (precincts) map as an R object

Since the IFE uses a different coding standard for the municipalities of Mexico than the INEGI, I’ve recoded the municipality codes so that they match the ones the INEGI uses.

Ecatepec, México according to the INEGI is 15 033, while according to the IFE it’s 15 034
Guadalajara, Jalisco according to the INEGI is 14 039, while according to the IFE it’s 14 041

These codes are only available for the secciones electorales (precincts) shapefile and they are contained in the variables:

MUN_INEGI: The inegi municipio codes
MUN_IFE: The original ife municipio codes that came with the file

The codebook for the the census data that comes with the distrito and sección shapefiles is in the FD_SECC_IFE.pdf file and the ife and inegi codes are in the ife.to.inegi.csv file. I also suggest you check out Rob Hidalgo’s repository of public information (from freedom of information requests) which includes a shapefile of the municipios of Mexico according to the IFE (my version comes from the INEGI and they differ by a few municipalities)

P.S. Here’s the code

Update: You can also use a docker image to run the program without installing anything

docker pull diegovalle/download-maps12
# shared directory to store the output
mkdir -p /tmp/download-ine2010 
# once the container is up run make
docker run -v /tmp/download-ine2010:/download-maps12/map-out -i -t diegovalle/download-maps12

Mexican Presidential Election by District

2013-02-11T00:00:00+01:00

Check out the interactive map I made of the results of the presidential election in Mexico. I even used hierarchical clustering to group Mexico into five clusters according to how people vote (obviously the clusters are very rough and subject to statistical error).

These are the median values for the percentage of the vote for each party in the clusters:

Interpretation	Cluster	PRD	PAN	PRI
PRI	1	0.27	0.20	0.46
PRI-PAN	2	0.23	0.34	0.38
PRI-PRD	3	0.40	0.16	0.37
PRD	4	0.54	0.15	0.27
PAN	5	0.22	0.48	0.26

Interactive map of the drug war in Mexico

2013-01-22T00:00:00+01:00

I’ve updated the interactive map I made last year to include the 2011 homicide data the INEGI recently released. Because of bad publicity the government will no longer update the drug war-related homicide database, so that dataset will forever remain stuck in September 2011.

The cool thing about the new version of map is that you can statistically correct some of the errors in the way homicides are recorded in Mexico.

Violent deaths can be classified as suicides, accidents, homicides, legal interventions and those of unspecified intent. There has been an upsurge in deaths of unspecified intent by firearm in Mexico at the same time all homicides increased, so it’s natural to assume that some of those unspecified deaths were really homicides.

To correct the mistakes here is what I did:

Classify all deaths of unknown intent into accidents, suicides and homicides based on the age, sex of the victim and injury mechanism by which the death occurred. The method I used is similar to the one I used in my Juarez post. For example, if someone told you to guess the intent of the death of a 70 year old woman that died in Merida (one of Mexico’s safest cities) by motor vehicle, you’d probably guess it was an accident. If you had to guess the intent of the death of a young male in Ciudad Juárez that died by firearm, you’d probably think it was an homicide.
Recode all deaths by legal intervention as homicides (the 19 deaths from the shootout between the army and the narcos in General Treviño and Mier were recorded as homicides) since it seems legal intervention deaths are not always classified correctly.
Reconcile the problems in the Federal District and Sinaloa
Since firearm accidents went from 6 in 2006 to a 100 in 2007 and again to 6 in 2008 I reclassified all firearm accidents in Baja California in 2007.
I added the mass grave in Taxco and the one in San Fernando since they don’t appear or are incomplete in the dataset from the INEGI.

Here are the results from in terms of sensitivity and specificity from the classification:

Even with the corrections I made there are still some errors left in the database, hopefully in the future I’ll figure out how to fix them.

Visit the map on facebook: https://www.facebook.com/narcomap

P.S. You can download the source from GitHub
P.P.S You can also download the program to clean the mortality database
P.P.P.S If all you need is the data you can download it from the datasets section of my website