The first one is that Pittsburgh has a good public transportation system, particularly the bus service provided by the Port Authority of Allegheny County. They have such many bus lines that I can use them for various purposes, including my commute to Carnegie Mellon University (CMU). Although Stanford University provides shuttle service comparable to the similar one by CMU, I didn’t find such convenience in public bus service around Stanford.

The weather in Pittsburgh is also unexpectedly benign. It is colder in Pittsburgh than in California, of course. We do have some snow in winter, but by and large it has been exceptionally warm this winter. I will show you some pictures of Pittsburgh below.

These days I have been gearing up for the Ph.D. program of the Language Technologies Institute at CMU, which I will be enrolling in this August. While making efforts to prepare for the program, I would like to find more goodness and fun here in Pittsburgh for a change.

The Gates-Hillman Centers where I currently work.

The Cathedral of Learning at the University of Pittsburgh.

The Heinz Memorial Chapel close to the Cathedral of Learning.

A spacious and gentle slope on campus at Chatham University.

A scene in the Pittsburgh downtown area.

On the same day, I also attended the Computer Science Departmental diploma ceremony on the lawn of the Gates Building. I received a diploma of Master of Science in computer science. The very moment of the reception was the most impressive to me on that day. I got a little tan with the sunny weather, by the way.

import java.io.BufferedWriter;
import java.io.FileWriter;
import java.io.FileNotFoundException;
import java.io.IOException;

class SomeClass {
    public static void writeFile(String filename) {
        BufferedWriter fout = null;

        try {
            fout = new BufferedWriter(new FileWriter(filename));

            while (/* Some condition */) {
                // Write something.
                StringBuffer lineBuffer = new StringBuffer();
                lineBuffer.append("abc");
                lineBuffer.append("def");

                fout.write(lineBuffer.toString());
                fout.newLine();
            }
        } catch (FileNotFoundException e) {
            e.printStackTrace();
        } catch (IOException e) {
            e.printStackTrace();
        } catch (Exception e) {
            e.printStackTrace();
        } finally {
            try {
                if (fout != null) fout.close();
            } catch (Exception e) {
                e.printStackTrace();
            }
        }
    }

    public static void main(String args[]) {
        writeFile("sample.txt");
    }
}

Unfortunately the weather wasn’t good; it was cloudy and foggy. But still I enjoyed taking a walk around Point Reyes Station and the Lighthouse. Here are several pictures:

There are several shops in Point Reyes Station. A store clerk in the bookstore in the middle of this picture was kind enough to give me a free map for the seashore and tell me the way to my destination, the Lighthouse.

This is Point Reyes Beach that goes along with the seashore. I wish it had been sunny.

You can come across wild animals like this deerlet.

These are trees which I think were probably bent due to the wind from the ocean.

The Lighthouse was encompassed with a fog. If it is sunny, one should be able to see the roundness of the earth with the sea horizon.

I saw some water birds pursuing fish in a marshland on my way back home from the Lighthouse.

Suppose that I need a list of unique words from English sentences described in a text file sentences.txt. Then my initial step is often to use a crude shell command like this:

$ cat sentences.txt | tr ' ' '\n' | sed '/^$/ d' | sort | uniq > unique_words.txt

The list I obtain with this command is neither tokenized nor lemmatized, but it could be sufficient for a quick analysis where I try to get a handle on approximately how many unique words occur in the target document.

If I need a more complicated analysis like extracting a list of unique words with their frequencies, the next step is likely to involve tokenization. A range of tokenization algorithms have so far been proposed according to respective natural languages. As for English, it seems that the simplest way is to build a tokenizer with regular expressions, as mentioned in Jurafsky and Martin 2000. For future reference, I will attach my Java code for English tokenization to this post.

import java.util.ArrayList;
import java.util.Arrays;
import java.util.List;
import java.util.regex.Pattern;
import java.util.regex.Matcher;

/**
 * Tokenizes strings described in English.
 *
 * @author Jun Araki
 */
public class EnglishTokenizer {
    /** A string to be tokenized. */
    private String str;

    /** Tokens. */
    private ArrayList<String> tokenList;

    /** A regular expression for letters and numbers. */
    private static final String regexLetterNumber = "[a-zA-Z0-9]";

    /** A regular expression for non-letters and non-numbers. */
    private static final String regexNotLetterNumber = "[^a-zA-Z0-9]";

    /** A regular expression for separators. */
    private static final String regexSeparator = "[\\?!()\";/\\|`]";

    /** A regular expression for separators. */
    private static final String regexClitics =
        "'|:|-|'S|'D|'M|'LL|'RE|'VE|N'T|'s|'d|'m|'ll|'re|'ve|n't";

    /** Abbreviations. */
    private static final List<String> abbrList =
        Arrays.asList("Co.", "Corp.", "vs.", "e.g.", "etc.", "ex.", "cf.",
            "eg.", "Jan.", "Feb.", "Mar.", "Apr.", "Jun.", "Jul.", "Aug.",
            "Sept.", "Oct.", "Nov.", "Dec.", "jan.", "feb.", "mar.",
            "apr.", "jun.", "jul.", "aug.", "sept.", "oct.", "nov.",
            "dec.", "ed.", "eds.", "repr.", "trans.", "vol.", "vols.",
            "rev.", "est.", "b.", "m.", "bur.", "d.", "r.", "M.", "Dept.",
            "MM.", "U.", "Mr.", "Jr.", "Ms.", "Mme.", "Mrs.", "Dr.",
            "Ph.D.");

    /**
     * Constructs a string to be tokenized and an empty list for tokens.
     *
     * @param  str  a string to be tokenized
     */
    public EnglishTokenizer(String str) {
        this.str = str;
        tokenList = new ArrayList<String>();
    }

    /**
     * Tokenizes a string using the algorithms by Grefenstette (1999) and
     * Palmer (2000).
     */
    public void tokenize() {
        // Changes tabs into spaces.
        str = str.replaceAll("\\t", " ");

        // Puts blanks around unambiguous separators.
        str = str.replaceAll("(" + regexSeparator + ")", " $1 ");

        // Puts blanks around commas that are not inside numbers.
        str = str.replaceAll("([^0-9]),", "$1 , ");
        str = str.replaceAll(",([^0-9])", " , $1");

        // Distinguishes single quotes from apstrophes by segmenting off
        // single quotes not preceded by letters.
        str = str.replaceAll("^(')", "$1 ");
        str = str.replaceAll("(" + regexNotLetterNumber + ")'", "$1 '");

        // Segments off unambiguous word-final clitics and punctuations.
        str = str.replaceAll("(" + regexClitics + ")$", " $1");
        str = str.replaceAll(
                "(" + regexClitics + ")(" + regexNotLetterNumber + ")",
                " $1 $2");

        // Deals with periods.
        String[] words = str.trim().split("\\s+");
        Pattern p1 = Pattern.compile(".*" + regexLetterNumber + "\\.");
        Pattern p2 = Pattern.compile(
            "^([A-Za-z]\\.([A-Za-z]\\.)+|[A-Z][bcdfghj-nptvxz]+\\.)$");
        for (String word : words) {
            Matcher m1 = p1.matcher(word);
            Matcher m2 = p2.matcher(word);
            if (m1.matches() && !abbrList.contains(word) && !m2.matches()) {
                // Segments off the period.
                tokenList.add(word.substring(0, word.length() - 1));
                tokenList.add(word.substring(word.length() - 1));
            } else {
                tokenList.add(word);
            }
        }
    }

    /**
     * Returns tokenized strings.
     *
     * @return  a list of tokenized strings
     */
    public String[] getTokens() {
        String[] tokens = new String[tokenList.size()];
        tokenList.toArray(tokens);
        return tokens;
    }
}

References:

Daniel Jurafsky and James H. Martin. 2000. Speech and Language Processing: An Introduction to Natural Language Processing, Computational Linguistics and Speech Recognition (Prentice Hall Series in Artificial Intelligence). Prentice Hall.

Gregory Grefenstette. 1999. Tokenization. In van Halteren, H. (Ed.), Syntactic Wordclass Tagging. Kluwer.

David D. Palmer. 2000. Tokenisation and sentence segmentation. In Dale, R., Moisl, H., and Somers, H. L. (Eds.), Handbook of Natural Language Processing. Marcel Dekker.

There were several corners for presenting Japanese culture. Out of them, the tea ceremony seemed to be the most popular. Two Japanese women wearing traditional garment, kimonos, demonstrated how to make and have Japanese tea in a formal way. In addition, some Japanese people tried to teach how to write Japanese calligraphy, shodo, and how to do Japanese paper folding, origami. Just being there for about ten minutes, I was struck by a sense of nostalgia.

Two Japanese women demonstrating how to make and have Japanese tea.

Some ornaments for the Boy’s Festival in Japan.

As I wrote in a previous post, some people around Stanford University are very active. I always see several people enjoying their exercise on campus. Some of them are such avid runners that they push a baby carriage with their baby or babies while running, though I think this is a little dangerous.

Once the spring quarter begins, probably I will be very busy again. But I’d like to enjoy myself doing exercise as much as possible. Incidentally, I am somewhat interested in the general relationships between physical activities and our brains. A suggestive (but informal) article on this topic is from PhDs.org: PE for grad students.

import java.io.BufferedReader;
import java.io.FileReader;
import java.io.FileNotFoundException;
import java.io.IOException;

class SomeClass {
    public static void readFromFile(String filename) {
        BufferedReader fin = null;

        try {
            fin = new BufferedReader(new FileReader(filename));
            String line = null;
            while ((line = fin.readLine()) != null) {
                // Do something.
            }
        } catch (FileNotFoundException e) {
            e.printStackTrace();
        } catch (IOException e) {
            e.printStackTrace();
        } catch (Exception e) {
            e.printStackTrace();
        } finally {
            try {
                if (fin != null) fin.close();
            } catch (Exception e) {
                e.printStackTrace();
            }
        }
    }

    public static void main(String args[]) {
        readFromFile("sample.txt");
    }
}

Incidentally I have used BufferedReader rather than BufferedInputStream simply because BufferedReader is more suitable to the text processing that I need to do right now.

#!/usr/bin/python

# csv2html.py
# CSV to HTML Converter

import csv
import sys

table_indent_num = 2
tr_indent_num = 4
td_indent_num = 6
white_space = " "

def main():
    csv_reader = csv.reader(open(sys.argv[1]))
    table_indent = white_space * table_indent_num
    tr_indent = white_space * tr_indent_num
    td_indent = white_space * td_indent_num

    print table_indent + "<table>"
    for i, row in enumerate(csv_reader):
        print tr_indent + "<tr>"

        # Uncomment the following two lines if you don't
        # want a column for indexes
        if i == 0: print td_indent + "<th>#</th>"
        else: print td_indent + "<td>" + str(i) + "</td>"

        # Assume that the first line is a header
        for column in row:
            if i == 0: print td_indent + "<th>" + column + "</th>"
            else: print td_indent + "<td>" + column + "</td>"

        print tr_indent + "</tr>"

    print table_indent + "</table>"

if __name__ == "__main__":
    argn = len(sys.argv)
    if argn != 2:
        print "Usage: python csv2html.py <CSV file>"
        exit(1)

    main()

An example of usage is as follows:

$ more sample.csv
title1,title2,title3
"test11",test12,test13
test21,"test,22",test23
$ python csv2html.py sample.csv > sample.html
$ more sample.html
  <table>
    <tr>
      <th>#</th>
      <th>title1</th>
      <th>title2</th>
      <th>title3</th>
    </tr>
    <tr>
      <td>1</td>
      <td>test11</td>
      <td>test12</td>
      <td>test13</td>
    </tr>
    <tr>
      <td>2</td>
      <td>test21</td>
      <td>test,22</td>
      <td>test23</td>
    </tr>
  </table>