Other articles

1. Pandas date parsing performance

   Dates and times provide an unlimited source of hassles for anyone working with them. In this post I'll discuss a potential performance pitfall I encountered parsing dates in pandas. Conclusion: Create DatetimeIndices by parsing data with to_datetime(my_dates, format='my_format').

   ...read more

   There are comments.

2. Analyzing 10 years of digital photography with python and pandas

   I recently switched DSLR camera systems from Canon to Nikon for reasons of marital harmony. That meant choosing which Nikon lenses would replace the four Canon lenses I owned. To make an optimal decision I needed to know my historical usage, so I wrote some python to analyze image metadata from 10 years of digital photography.

   ...read more

   There are comments.

3. Pandas Timedelta: histograms, unit conversion and overflow danger

   The Timedelta object is relatively new to pandas. Recently I worked with Timedeltas but found it wasn't obvious how to do what I wanted. Worse, some operations were seemingly obvious but could easily return the wrong answer (update: this issue was fixed in pandas version 0.17.0). Here I go through a few Timedelta ...read more

   There are comments.

4. Pandas 0.16.0 released

   Pandas 0.16.0 was released this week. Version 0.16.0 includes a new .assign() method of DataFrames, which I'll use on some real world automobile MPG data below (data file).

   ...read more

   There are comments.

5. Saving time and space by working with gzip and bzip2 compressed files in python

   File compression tools like gzip and bzip2 can compress text files into a fraction of their size, often to as little as 20% of the original. Data files often come compressed to save storage space and network bandwidth. A typical workflow is to uncompress the file before analysis, but it can be more convenient to leave the file in its compressed form, especially if the uncompressed file would take up a significant amount of space. In this post I'll show how to work directly with compressed files in python.

   ...read more

   There are comments.

6. Annotating matplotlib plots

   To extend on my post about plotting and reshaping data from the BART API, I worked a bit with the matplotlib annotion interface to add text and arrows to a plot. The meat of this post is in cell #4 below. Download notebook.

   ...read more

   There are comments.

7. Cleaning, reshaping, and plotting BART time series data with pandas

   Introduction¶

   I recently starting collecting data from the BART API, specifically estimated time to departure for trains at the two stations I use most frequently. In this notebook I'll show how I parsed the data from a csv file, reshaped it to fit the questions at hand, and made a few plots. Download notebook ...read more

   There are comments.

8. When joins go wrong, check data types

   Writing and debugging joins can be especially difficult when dealing with data from text files. In some cases there is no resulting data, or (much harder to notice!) a few lines that should be included are dropped. Here I'll go into an example of a failed join in pandas, and …

   ...read more

   There are comments.

9. Working with dates in pandas: a few examples

   pandas has a lot of lifesaving features for dealing with dates. Here's an example timeseries data file, which happens contains a missing date:

   date,temp
   11-1-2014,56
   11-2-2014,56
   11-3-2014,59
   11-5-2014,60
   11-6-2014,55
   

   Loading and plotting the data in pandas gives this result:

   In [0]: %matplotlib inline
   
   In …

   ...read more

   There are comments.

Page 1 / 1