I'm a computer science professor, data scientist, and web geek.

Last weekend Yahoo came to New York for an Open Hack Day, and it was great!

I was invited to speak on a panel on semantic metadata, moderated by Paul Ford (harpers.org) along with Marco Neumann (KONA) and Paul Tarjan (Yahoo/Search Monkey). The panel was a lively discussion, and we got some great questions from the audience.

After the panel, I stayed around to participate in the hack competition. Yahoo! provided a fantastic space, with free-flowing coffee, snacks, comfy chairs and plenty of Yahoo folks and other hackers around to give advice and play foosball with. I teamed up with Diana Eng, Alicia Gibb, and Bill Ward to create the Del.icio.us Cake!

The cake is attached to a laptop via USB. A program running on the laptop accepts a delicious tag and retrieves a list of recent popular sites for that tag from the delicious API. Finally, it iterates through each URL, downloads the page, and computes the sentiment of that page relative to the tag — basically, is the content of the page positive, neutral or negative?

The signal is output to an ardiuno (hidden in the middle of the cake) which turns on the appropriate set of LEDs. There are four sets of LEDs on the cake, one in each quadrant of the delicious logo, one each for positive sentiment, neutral or inconclusive sentiment, and negative sentiment, and, of course, one to let us know that the cake is turned on.

I wrote the sentiment classifiers between around 3am and 6am Saturday morning, so they really were a hack! I trained them on movie reviews data, working with the assumption that 5-star reviews contain positive terms and 1-star reviews contain negative terms. I wouldn’t recommend this approach for a serious attempt at sentiment analysis, but it worked well enough.

We won the food/hardware hack prize, shared with the awesome MakerBot team!

We had a great time creating and presenting the hack. Thanks, Yahoo, and most of all, thanks to Alicia, Bill, and Diana for a really fantastic, silly weekend.

Further coverage:

• Yahoo’s summary of the Open Hack NYC event
• Diana’s writeup for Eyebeam
• CNN.com: Hackers Take Over Times Square

I’ve found myself in need of a name distribution for a few projects recently, so I thought I would post it here so I won’t have to go looking for it again.

The data is available from the US Census Bureau (from 1990 census) here, and I have it here in a friendly MySQL *.sql format (it will create the tables and insert the data). There are three tables: male first names, female first names, and surnames.

I’ve noted several issues in the data that are likely the result of typos, so make sure to do your own validation if your application requires it.

The format is simple:

1. the name
2. frequency (percentage of people in the sampled population with that name)
3. cumulative frequency (as you read down the list, the percentage of total population covered)
4. rank

If you want to use this to generate a random name, you can do so very easily with a query like this:

SELECT name FROM ref_census_surnames n ORDER BY (RAND() * (n.freq + .01)) LIMIT 0,1;

Download it here: census_names.tar.gz

Yesterday, I attended the first Hadoop World NYC conference. Hadoop is a platform for scalable distributed computing. In essence, it makes analyzing large quantities of data much faster, and analyzing very large quantities of data possible.

Cloudera did a great job organizing the conference, and managed to assemble a diverse set of speakers. The sessions covered everything from academic research to fraud detection to bioinformatics and even helping people fall in love (eHarmony uses Hadoop)!

I’m not going to review every session, but I saw several themes emerging from the content and conversations.

Hadoop is Getting Easier

New integrated UIs like Cloudera Desktop and Karmasphere mean that developers will no longer be required to use a command-line interface to configure and execute Hadoop jobs. IBM’s M2 project hides Hadoop behind a spreadsheet metaphor, making the collection, analysis and visualization of data as easy as using Excel.

This doesn’t just speed up development time, it puts the tools for manipulating the data directly in the hands of the people who need the results, without requiring them to talk to a database programmer.

Hadoop is a Utility

The only organizations that talked about building their own Hadoop clusters are those who deal with very sensitive data (VISA) and those who deal with very very large quantities of data (Yahoo, Facebook, eBay). Organizations with more manageable data sets, such as eHarmony and the New York Times, use EC2 and Amazon’s Elastic Map-Reduce. Amazon, Rackspace, and Softlayer have offerings in this area and were all event sponsors.

Yes, you can turn on a cluster of nodes from your living room in your PJs!

Hadoop Can Talk to Your Existing Systems

Hadoop has an ecosystem of supporting products that allow organizations to adapt their existing infrastructure. Cloudera’s Sqoop (which is just fun to say out loud) is a tool for importing data from SQL databases, HBase is a Hadoop database, and Pig lets you talk to the system in a SQL-like language.

I expect we’ll see more information available in the near future to clarify which systems are more appropriate for which kinds of users (an ecosystem decision tree?).

Hadoop is Changing Things

I heard the phrase “an order of magnitude improvement in speed” so many times that I lost count. Speaking from personal experience, the difference you see in productivity between waiting minutes and hours for results and waiting days is immense. When you can see the answer to a question shortly after you ask it you can preserve the context you need to act on that answer immediately without having to spend the time to figure out why you were asking that question in the first place.

Most of the projects were doing fairly simple analysis over data like web user sessions or transactions. I was intrigued by Deepak Singh’s talk on bioinformatics and genome sequencing (slides) and Jake Hofman’s talk on social network analysis (slides). More and more massive datasets are becoming available and will drive techniques for new analysis. I do wish there had been a talk about Mahout, which is a very promising approach to developing machine learning algorithms on the Hadoop platform.

I left the event more excited about the technology and very enthusiastic about the community. Thanks for a great day!

Update: A few other people have written up their notes and impressions from the event:

• Stephen O’grady posted The View from HadoopWorld
• Deepak Singh’s Post-HadoopWorld Thoughts
• HubSpot Dev Blog has two write-ups, by Dan and Steve
• Atbrox has notes from the morning session and the application session
• Alexander Sicular’s Are You New to Hadoop? Settle in…
• Pete Skomoroch posted his slides and thoughts
  

http://jakehofman.com/talks/hadoopworld_20091002.pdf

http://www.slideshare.net/mndoci/hadoop-for-bioinformatics

Dear friends and colleagues,

Do you do research that involves gathering data from human participants? This can be anything from marketing surveys to psychology experiments to medical science. If so, please take a short (5 to 10 minute) survey:

research tool survey

The results of the survey will help us design a new platform for online human research!

I’m very excited about this project and would very much appreciate your input. If you have colleagues who do this kind of work, please pass it on.

Thank you!

I gave a talk at the NYC Python Meetup on July 29 on Practical Data Analysis in Python.

I tend to use my slides for visual representations of the concepts I’m discussing, so there’s a lot of content that was in the presentation that you unfortunately won’t see here.

The talk starts with the immense opportunities for knowledge derived from data. I spent some time showing data systems ‘in the wild’ along with the appropriate algorithmic vocabulary (for example, amazon.com’s ‘books you might like’ feature is a recommender system).

Once we can describe the problems properly, we can look for tools, and Python has many! Finally, in the fun part of the presentation, I demoed working code that uses NLTK to build a Twitter spam filter with 90% accuracy*.

Please let me know if you have questions or comments.

* I’ll post the code and training data shortly

I gave a talk at BarCampNYC4 on Saturday on common data problems and a very light overview of algorithms that address them.

My talk at barcampnyc4 - photo courtesy of dynamist on flickr.

I delivered the majority of the content verbally, by talking through examples of problems and how to solve them, so there’s no guarantee that these slides will make sense, but they might be funny!

Sanford took some excellent notes during the presentation.

There were some very nice comments on twitter.

The discussion was so lively and engaging that I’m planning to expand on this content — I really welcome your suggestions and comments!

Jon Udell hosted Charlie and I on his Interviews with Innovators podcast.

We discussed Path101’s approach to career advice through data, and how the high availability of data is changing the way we make decisions.

Listen here.

I think this might be my very first LSL script, from back in 2005! This script indicates whether your AIM (AOL Instant Messenger) account is online by changing the color of an object. You can configure it to either share your AIM ID publicly, or keep it private.

AIM Indicators in Second LIfe

This script uses the AIM web services API to check your online status — you only need to give it your username, not your password! This is not a proxy service. You can’t send messages through this script, just show your online status in SL.

To use this script, create an object in your favorite shape, create a new script inside of it, paste this code into it and save.

key request_id;
string aim_id;
string av_name;
key data_card;
integer nLine = 0;
integer public = TRUE;
 
default
{
    state_entry()
    {
        llSetText("AIM Indicator... setting up",<1,1,1>,1);
        data_card = llGetNotecardLine("Settings",nLine); // load settings
        llSetTimerEvent(60); // check once per min
    }
 
    // reset when owner touches it
    touch_start(integer total_number) {
        if (llDetectedKey(0) == llGetOwner()) {
            llResetScript();
        }
    }
 
    // read settings from notecard
    dataserver(key query_id, string data) {
        if (query_id == data_card) {
            if (data != EOF) { // not at the end of the notecard
                if (nLine == 0) {
                    av_name = data;
                } if (nLine == 1) {
                    aim_id = data;
                } if (nLine == 2) {
                    if (data == "private") { // if they do not want their ID shared
                        public = FALSE;
                    }
                }
                ++nLine; // increase line count
                data_card = llGetNotecardLine("Settings", nLine); // request next line
            }
        }
    }
 
    timer() {
        string url = "http://api.oscar.aol.com/SOA/key=hi15uD79wuEukQTS/resource-lists/users/*anonymous*/presence/~~/resource-lists/list%5Bname=%22users%22%5D/entry%5B@uri=%22user:"+aim_id+"%22%5D";
        request_id = llHTTPRequest(url,[HTTP_METHOD, "GET"],"");
    }
 
    http_response(key req_id, integer status, list metadata, string body) {
        if (req_id == request_id) {
            list result = llParseString2List(body,["<",">","\n"],[]);
 
            // if user is online
            if (llList2String(result,24) != "http://cdn.digitalcity.com/presence/offline.gif") {
                if (public == FALSE) { 
                    llSetText(av_name+"'s AIM account is online",<1,1,1>,1);
                } else {
                    llSetText("AIM "+av_name+": "+aim_id+" is online",<1,1,1>,1);
                }
                llSetColor(llVecNorm(<160,237,160>),ALL_SIDES);
            } else { // user is offline
                llSetText(av_name+"'s AIM account is offline",<1,1,1>,1);
                llSetColor(llVecNorm(<237,160,160>),ALL_SIDES);
            }
        }
    }
}

This script is released under . Have fun!

Create a notecard called “Settings” with your avatar name on the first line* and your AIM username on the second line. If you do not want your AIM username shared, put the word “private” on the third line. If you do want it shared, change the line to anything else (“public”, or even blank will do). For example:

Ann Enigma
hilm1
private

The script polls the AIM server every 60 seconds, so give it a minute to update. It will reset if the owner clicks on it.

* I’m aware that you can get the avatar name via LSL — this was my first script, be nice!

The April edition of Communications of the ACM has an interesting article on recent advances in active learning by Graeme Stemp-Morlock.

In passive learning (a more traditional approach), you build a large training set of classified data by (often) manually assigning labels. This data is used as the basis of your analysis.

In the real world, we find that generating these large sets of labeled data is often expensive and time consuming. With active learning, you identify the most ambiguous data to label, resulting in a much higher payoff for each label defined (and fewer headaches for your labelers).

The article goes on to mention that active learning is being used in practice with excellent results (for example in music identification, text classification, and even bioinformative), but that the theory lags. This is another example of a gap between the world of the practitioner and the academic work behind it.

Second Life news stand

One of the more popular objects that I created in Second Life is the News Stand. This charming device takes any query term and displays the two latest headlines from Yahoo! News. Clicking the news stand will load the top news story for the topic in a web browser.

I’ve gotten more requests to customize the script than I can possibly keep up with, so I’ve decided to release the code under a creative commons license. I hope you find it useful! Please let me know if you make improvements, so that I can link to them from here.

Unfortunately, parsing XML at all and RSS feeds in particular is extremely messy in LSL. This script doesn’t contain a general RSS feed parser — the parsing code was written specifically for the Yahoo! news feed.

// News Stand script
// by Ann Enigma
 
string url="http://news.search.yahoo.com/news/rss?p=";
key requestid;
string current_url = "http://news.yahoo.com";
integer gLine = 0;
key data_card; // for settings notecard
string query;
 
integer mins = 10; // number of minutes between refreshes
 
get_news() {
    llWhisper(0, "Refreshing the news...");
    requestid = llHTTPRequest(url,[HTTP_METHOD,"GET"],"");       
}
 
default
{
    state_entry()
    {
        data_card = llGetNotecardLine("Settings",gLine); // load settings from notecard
    }
 
    // read data from the notecard
    dataserver(key query_id, string data) {
        if (query_id == data_card) {
            if (data != EOF) {    // not at the end of the notecard
                if (gLine == 0) {
                    query = data;
                } 
            }
        }
        state setup;
    }
    changed(integer change) {
        llResetScript();
    }
}
 
state setup {
    state_entry()
    {
        url += llEscapeURL(query); // set the url
        get_news(); // get the news!
        llSetTimerEvent(60*mins); // every mins # of minutes
    }
 
    touch_start(integer total_number)
    {
       llLoadURL(llDetectedKey(0), "Current Top Story", current_url);
    }
    http_response(key req_id, integer status, list metadata, string body) {
 
        integer count = 0;
        string display;
 
        if (req_id == requestid) { // parse the results (unfortunately messy)
            list result = llParseString2List(body,["<",">","\n","[","]"],[]);
            integer length = llGetListLength(result);
            integer i = 0;
            for (i = 0; i < length; i++) {
                if (llList2String(result,i) == "title" && llList2String(result,i+4) == "/title") {
                    display += llList2String(result,i+3) + "\n";
                    if (count == 1) { 
                        current_url = llList2String(result,i+6);
                    }
                    count++;
                }
            }
            llSetText(display, <1,1,1>, 1.0);
        }
    }
    timer() {
        get_news();
    }
    changed(integer change) {
        llResetScript();
    }
 
}

This script is released under . Have fun!

Create a notecard called “Settings”, with a single line containing your search term. For example:

Educational Technology

The original news stand comes with these instructions, which may come in handy:

To set up the News Stand, simply open the "Settings" notecard, delete the default search term, and type in your own!

This gadget will refresh every 10 minutes, and will display the top two headlines on Yahoo! News for any search term. If you click on the news stand, it will load the top story in a web browser.

If you would like to build your own news stand, you can simply drag the script and "Settings" notecard into any object.

If you want the no-assembly-required version, the News Stand is available for L$350 at Xstreet SL Marketplace or from the ICT Library on Info Island in-world.

The original news stand object was created by (now former) student Kyle Pouliot.