Hamilton Filtering

In this post I present a R function that implements James D. Hamilton's alternative to the Hodrick-Prescott Filter as described in his article "Why You Should Never Use the Hodrick Prescott Filter", forthcoming in the Review of Economics and Statistics. In the paper, Hamilton demonstrates how HP filtering creates artificial correlation between variables and suggests filtering using the residuals of the following regression:
$$ y_{t+h} = \beta_0 + \sum_{l=1}^p \beta_l y_{t+1-l} + \nu_{t+h}, $$
$y_{t+h}$ are the elements of $y$ that can be predicted $h$ periods ahead using its previous $l$ observations. This corresponds to the trend after fluctuations have disappeared after $h$ periods. $h$ should therefore be chosen to be 2 years where we assume macroeconomic shocks to be worn off. $l$ is recommended to include 1 year of data to remove possible seasonal components from the trend. I highly recommend reading Hamilton's blog post on the topic.
I implement this filter in the following R function based on the matlab code accompanying Hamilton's paper.

hamfilter <- function(yt, h=2*frequency(y), p=frequency(y), trend=FALSE){ # Hamilton Filter # # R code to calculate cyclical component based on h-period-ahead #forecast error from linear regression # y(t+h) = b0 + b1*y(t) + ... + bp*y(t+1-p) + e(t+h) # as recommended in # James D. Hamilton, "Why You Should Never Use the Hodrick-Prescott Filter" # Review of Economics and Statistics, forthcoming # Code based on James D. Hamilton's matlab code # inputs: # y = (T x 1) vector or time series, tth element is observation for date t # p = number of lags that should be considered in the regression of y(t+h), # default is the frequency of a time series # outputs: # yreg = (T x 1) vector or time series, tth element is cyclical component for date t # (optional) ytrend = (T x 1) vector or time series, tth element is trend component # # works with any frequency of data # works with leading and trailing missing values (not internal) # preserves class of input data (i.e. time series, vector, etc.) # require(zoo) # create final output matrices to preserve input class and length yreg <- yt yreg[] <- NaN ytrend <- yt ytrend[] <- NaN # prepare data y <- na.omit(yt) T <- length(y) lseq <- 0:-(p-1) X <- cbind(rep(1,T-p-h+1), as.matrix(lag(as.zoo(y),lseq)[((p):(T-h)),])) yh <- y[(p+h):T] # extimate and predict y(t+h) = b0 + b1*y(t) + ... + bp*y(p-1) + e(t+h) b <- solve(t(X) %*% X) %*% (t(X) %*% yh) ytrend[!is.na(yt)] <- c(rep(NaN,p+h-1),X%*%b) yreg[!is.na(yt)] <- y - c(rep(NaN,p+h-1),X%*%b) if(trend==TRUE){ return(cbind(ytrend,yreg)) } else { return(yreg) } }

A Function to Retrieve Time Series by Variable Number from CanSim (Statistics Canada)

In this post, I want to make a function available that allows you to download a time series from Statistics Canada directly into R. The function is a wrapper for the get_cansim_vector function from the 'cansim' package. It determines the frequency of a series and converts it into a time series.

 All CanSim variables start with a "v". To find this vector name, you need to change the view of a table you find on the Statistics Canada website by clicking 'Add/Remove Data' => 'Customize Layout' and then tick the box  'Display Vector identifier and coordinate'

Below is the function that will do the trick. The function works with daily, weekly, monthly, quarterly and yearly data. If the series does not have one of these frequencies, the data is still returned, but not properly converted into a time series. 

ts_cansim <- function(vector, start, end = NA){ # load required packages require(cansim) require(lubridate) # get series if(is.na(end)){ xx <- get_cansim_vector(vector , start_time = start) } else { xx <- get_cansim_vector(vector , start_time = start, end_time = end) } # determine frequency time.diff <- as.Date(xx$REF_DATE[2]) - as.Date(xx$REF_DATE[1]) x <- xx$VALUE n <- nrow(xx) st.date <- as.Date(xx$REF_DATE[1]) end.date <- as.Date(xx$REF_DATE[n]) # daily frequency if (1 <= time.diff & time.diff <= 4){ n.freqency = 365.25 x <- ts(xx$VALUE, start = c(year(st.date), month(st.date), day(st.date)), freq = n.freqency) } # weekly frequency if (time.diff == 7){ n.freqency = 365.25/7 x <- ts(xx$VALUE, start = c(year(st.date), month(st.date), day(st.date)), freq = n.freqency) } # monthly frequceny if (25 < time.diff & time.diff <= 31){ n.freqency = 12 x <- ts(xx$VALUE, start = c(year(st.date), month(st.date)), freq = n.freqency) } # quarterly frequency if (70 < time.diff & time.diff <= 100){ n.freqency = 4 x <- ts(xx$VALUE, start = c(year(st.date), quarter(st.date)), freq = n.freqency) } # yearly frequency if (300 < time.diff & time.diff <= 400){ n.freqency = 1 x <- ts(xx$VALUE, start = year(st.date), freq = n.freqency) } return(x) }

Now we can call the function retrieving the variable from CanSim. 

housing.starts <- ts_cansim('v52300157', start = '1950-01-01') plot(housing.starts)

Let me know if you have any comments or suggestions.

- this post was updated 2020-11-16 - 

Creating a Dataset with Multiple Lags of Each Variable

In many forecasting exercises it is useful to include several lags of a variable as potential predictor. I have not been able to find a nice solution online to create a data set that includes several lags of each variable, so I want to share my solution.
The goal is to create h=12 lags of each variable in a data set.  So if you have three variables in a matrix, say [x1, x2, x3], you want a new matrix of the form [x1_l0, x1_l1, x1_l2,  ... , x1_lh, x2_l0, x2_l1, x2_l2,  ... , x2_lh, x3_l0, x3_l1, x3_l2,  ... , x3_lh, ] , where h is the maximum lag length.

To do this, we'll need the zoo package that allows to create multiple lags of a series and fills up the created matrix of lags with missing values. The forecast package is used to create some example data of time series.

library(forecast) library(zoo)


First, we will create a sample data set of time series using the arima.sim function of the forecast package:

# Create some sample time series data: x0 <- as.zoo(ts(replicate(10,arima.sim(n=100, list(ar = c(runif(1, min=0, max=1), -runif(1, min=0, max=1)), ma = c(-runif(1, min=0, max=.5), runif(1, min=0, max=1))),sd = 1) -runif(1,min=0, max=.5)))) plot(x0)

Now, we will create the lagged data set. The code avoids loops by using the sapply function, thus creating a list with the variables and its lags as elements. The list is then concentrated back into a matrix using the do.call function. The next step is to assign each predictor the variable name combined with its respective lag. This is done by using the the same trick as before, concentrating a list that includes as its elements a combination of the variable name and the respective lag.

# Determine the maximum lag length and derive some parameters n.lag = 12 n.var = dim(x0)[2] seq.l = seq(0,-n.lag) # Creating Lagged Data Set xl <- ts(do.call(cbind,lapply(1:ncol(x0), function(z) lag(x0[,z],seq.l)))) # Name variables in new data set by adding the respective lag colnames(xl) <- as.vector(sapply(colnames(x0), function(z) paste(z,"_l",-seq.l, sep = ""))) # For later you might also want to keep the original variable names groups <- rep(factor(colnames(x0)),each=(n.lag+1))

The Greek Alphabet

Many students and even some professors get sometimes confused about all the Greek letters that are used in economics. Therefore, I made a table of the Greek alphabet for you to print out and put in the back of your binder.

\documentclass{article}
\usepackage[nohead]{geometry}
\geometry{left=0.8in,right=1in,top=1.00in,bottom=1.0in}
\begin{document}
\begin{center}
\section*{The Greek Alphabet}
\begin{LARGE}
\begin{tabular}{|rl|rl|rl|rl|}
 \hline
$A$, $\alpha$                 & Alpha   &                    $H$, $\eta$ & Eta    &                     $N$, $\nu$ & Nu      &              $T$, $\tau$ & Tau     \\
$B$, $\beta$                  & Beta    & $\Theta$, $\theta$ $\vartheta$ & Theta  &                   $\Xi$, $\xi$ & Xi      &   $\Upsilon$, $\upsilon$ & Upsilon \\
$\Gamma$,$\gamma$             & Gamma   &                   $I$, $\iota$ & Iota   &                     $O$,  $\o$ & Omicron & $\Phi$, $\phi$ $\varphi$ & Phi     \\
$\Delta$, $\delta$            & Delta   &                  $K$, $\kappa$ & Kappa  &          $\Pi$, $\pi$ $\varpi$ & Pi      &              $X$, $\chi$ & Chi     \\
$E$, $\epsilon$ $\varepsilon$ & Epsilon &           $\Lambda$, $\lambda$ & Lambda &          $P$, $\rho$ $\varrho$ & Rho     &           $\Psi$, $\psi$ & Psi     \\
$Z$, $\zeta$                  & Zeta    &                     $M$, $\mu$ & Mu     & $\Sigma$, $\sigma$ $\varsigma$ & Sigma   &       $\Omega$, $\omega$ & Omega   \\ \hline
\end{tabular}
\end{LARGE}
\end{center}
\end{document} 

Organizing Your Literature with JabRef

One of the great advantages of LateX is that you don't have to do your bibliography on your own. Of course you can, but most economists like to work 'efficiently'. One might suspect that this includes some degree of laziness, but that is a misconception. Only undergrad students are lazy. Grads and staff are 'busy with other things' or 'prefer to work efficiently'.

Anyways, Jabref and Google Scholar make it pretty easy to organize your literature for your LateX documents. In Google Scholar you can search for literature that is related to your topic by Keywords, similar to e.g. EconLit. A set-back of Google Scholar is that you can not access most of the literature directly. You will have to log in through your university access first. Often links to working paper versions of the actual article are provided. However, you should make the extra effort and go for the published article. The great advantage of Google Scholar is that it provides citations of the literature in TeX format. Unfortunately, this is not the standard option, so you have to adjust the Google Scholar preferences. You can then easily copy and paste the citations in Jabref. Jabref is a useful program that allows you to create and manage a bibliography file that includes all your references. In you LateX document you then ‘link’ to this bibliography file. The LateX document then automatically creates a bibliography for your document with all the literature you have referenced in your document. But let’s go through the set-up process and an example step by step.

I'm assuming you have installed LateX on your computer and you got the first simple example running. Now do the following:

1. Set up Google Scholar

Go to Google and then to Google Scholar.

Now click on ‘Scholar Preferences’ in the top right corner adjust the display of search results to your needs. Important is the last option (Bibliography Manager). Change this option to ‘import citations into BibTeX’.

Now your search results show the option ‘Import into BibTeX’.

If you want, you can also add Google Scholar to your search engines on the top right of your browser.

2. Import citations into JabRef

Search for some example article and click on ‘Import into BibTeX’. Copy whatever comes up in the window. It may look a bit strange for now but it will become clear soon. Now open JabRef. It will be totally blank since your bibliography file doesn’t have any entries yet. Click on the green plus sign on the top to add a new entry. You can either fill in all the information like author, year, journal etc. by hand or you stay with me and go to the last tab, ‘BibTeX source’. Past the TeX code you have copied from Google Scholar.

If you now go to the other tabs, all the relevant fields have been filled automatically. Make sure you adjust any oddities, especially the double brackets around the title name. Important is the ‘BibTeX-Key’ as you will be referring to it in your LateX document. You can change it by hand in the first line of the BibTex source or with the magic wand on the lower left hand side.

If you want, add other pieces of literature, then safe the database (bibliography file) in the appropriate folder. I would recommend to create a new folder for every LateX document. You can then put you bib-file in the superordinate folder and link to it from every document.

3. Citing

Now open WinEdt or whatever editor you are using. If we refer to our bib-file we can cite the papers that it includes. Try the following example. Compared to our first document we have three new more or less self-explaining commands. In the document header, we include ‘bibliographystlye’ that tells LateX how we want our bibliography to look. This document also includes a first ‘comment’. Everything after ‘%’ is not read by LateX and not part of the actual code. Here I’m telling you that you could also use ‘abbrv’ or ‘alpha’ instead of ‘plain’ as the ‘bibliographystlye’ option. The command ‘\cite’ is the actual citation in the text. In the brackets you should put the BibTex key you specified in JabRef. The command \bibliography{'file'} creates your bibliography and tells LateX which bib-file you are using. Here, I use the bibfile all.bib in the superordinate folder (../).

\documentclass{article}

\title{Our second document}

\bibliographystyle{plain}   %alternatively: {abbrv} or {alpha}

\begin{document}

\maketitle This is our second LaTeX document!

\cite{Hryshko2010} talk about consumption.

\bibliography{../all}

\end{document}

If you got the document running, you may not be very happy with the result. You should get a bibliography that is numbered and instead of the author (year) citation that is usually used in economics you get just that number. To solve this problem we must load our first ‘package’.  Packages allow us to use additional commands that are not included in standard LateX. In a future post, I will include a list of useful packages for economics. For now, we load the package ‘natbib’ that allows for author-year citations. The option [round] makes the brackets around the publication year round ( '(1987)' instead of '[1987]').  Packages are loaded by including \usepackage[‘option’]{‘packagename’} in your header. Note that the options for bibliographystyle have changed and we have a new set of citation commands (see table).

\documentclass{article}
\usepackage[round]{natbib}
\title{Our second document}
\bibliographystyle{plainnat}   %alternatively: {abbrvnat} or {unsrtnat}
\begin{document}
\maketitle This is our second LaTeX document!
\citet{Hryshko2010} talk about consumption.
Markets can fail due to asymmetric information \citep{Akerlof1970}.
\bibliography{../all}
\end{document}
 

Sometimes WinEdt doesn’t want to change the bibliographystyle and gives you an error message. When getting such an error message, try pressing enter and run it again. Now it should work.

Installing LateX (on Windows)

If you have decided or are urged to write documents with LateX, first: congratulations! You definitely made the right choice. Documents created with LateX just look beautiful and it certainly is a very useful soft skill to be at least somewhat versatile in programming in LateX. Yes, programming! LateX is a programming language, that needs to be compiled (translated by your computer) and produces a pdf file as output. This means that working with LateX is not WYSIWYG (what you see is what you get) as e.g. Microsoft word. By typing in commands that are a bit similar in structure to html you will rather get what you really mean (WYSIWYM).
Installing LateX is not quite as straight forward as installing other software. You'll have to install several elements separately. First you'll need MiKTeX which acts as a compiler and also provides you with packages. But I will talk about packages in another post.  The second important thing you need is an editor. Since LateX is a programming language, you could basically use the windows notepad, but other editors will provide you with some shortcuts and make life a bit easier. There's a link to an overview of editors on Wikipedia below. I used to use WinEdt which is $40 for students after a trial period. Recently, I have discovered TeXstudio which offers the same (if not more) functionality as WinEdt but is completely free!
Lastly, let me note that this is only one way to work with LateX. If you are very uncomfortable with programming, you might want to work with LyX or maybe Scientific Workplace which are more WYSIWYG.
So, let's go:

1.  Download and install MicTex:  http://miktex.org/
2.  Download and install Ghostscript and Ghostview: http://pages.cs.wisc.edu/~ghost/ 
3.  Download and install WinEdt or other Editor: 
1. TeXstudio: http://texstudio.sourceforge.net/ 
2. Comparison_of_TeX_editors: http://en.wikipedia.org/wiki/Comparison_of_TeX_editors 
   
4. Download Jabref: http://jabref.sourceforge.net/download.php 

Try and run your first document. Every \LateX document has a header (everything above \ begin{document}) and a main part (between \ begin{document} and \ end{document}).

\documentclass{article}
\title{Our first document}
\begin{document}
\maketitle
This is our first LateX document!
\end{document}

Did it work?


Updated: 7/16/16