Stop talking, start coding

Posted: April 18, 2010 | Author: | Filed under: blog | 65 Comments »

I read Out of the Loop in Silicon Valley in the NYTimes today, which explores how and why women are under-repesented in tech startups. From the number of retweets I saw and the clicks through bit.ly links (12,579 at the time of this posting), it’s been getting a lot of attention.

There are some very strong, compelling themes in this article. Computer science and engineering to have an “image problem”; the way we teach math to elementary school students is horrible and turns way too many away.

I don’t want to nitpick the article, but there are a few statements that reinforce the very damaging stereotypes that the article sets out to dispel.

“When women take on the challenges of an engineering or computer science education in college, some studies suggest that they struggle against a distinct set of personal, psycho-social issues… Even women who soldier though [sic] demanding computer science and engineering programs in college…”

I’ve been both a computer science student and a computer science professor. I have not seen any evidence that the average undergraduate computer science education is harder than physics, math, chemistry, biology, or many other ‘hard’ disciplines with a much stronger gender balance. Implying that women are unwilling to meet the intellectual challenges of the discipline is bullshit.

“Girls have certain family goals they want to accomplish,” she says. “Working 60 hours a week is difficult because it requires a life sacrifice.”

The men that I know and work with also have wonderful personal lives. Working 60 hour weeks is a sacrifice for them, too.

Please read the whole article. Let me know what you think when you see the material in context.

I’m going to make the assumption that we all believe that having more women in technology is a Good Thingtm.

Many groups have popped up that support women in technology, like Girls in Tech, She’s Geeky, and many others (enumerated in Digiphile’s thoughtful post Why Including women matters for the future of technology and society). More often than not, these groups are the canned food drives of the women in technology movement. They make you feel better, they might do a little good, but they offer no fundamental change to the system that created the problem in the first place.

The Grace Hopper Celebration of Women in Computing does this well. GHC invites women to come to one place, be together, and do science together.

We don’t need affirmative action for women in tech. We need to create experiences that nurture women and men so that more people are inspired to can create beautiful, technical things together.

Art and Technology: Seven on Seven

Posted: March 14, 2010 | Author: | Filed under: blog, Presentations | Tags: , , | 1 Comment »

I’m honored and excited to be participating in Rhizome’s new conference Seven on Seven, where technologists and artists are paired up to create a completely new project in 24-hours.

The formal description:

Seven on Seven will pair seven leading artists with seven game-changing technologists in teams of two, and challenge them to develop something new –be it an application, social media, artwork, product, or whatever they imagine– over the course of a single day. The seven teams will unveil their ideas at a one-day event at the New Museum on April 17th.

I really love this idea because the time constraints and the inherent discomfort of the situation (working in an unfamiliar space with an unfamiliar person) makes it likely that we’ll be able to accomplish something creative and unexpected. Or else it will go completely awry, which will still be amusing for the audience.

I’ve had a lot of fun and been able to work on some interesting projects at hackathons in the past, and I hope this one will be even better.

The event has been covered by an amazing assortment of blogs, including TechCrunch, BoingBoing, Art Fag City, Stowe Boyd and Andrew Parker.

Conference: Search and Social Media 2010

Posted: February 16, 2010 | Author: | Filed under: academics, blog, Presentations | Tags: , , , | 2 Comments »

I recently attended the Third Annual Workshop on Search and Social Media, an academic workshop with very strong industry participation. The workshop was packed, and had some of the most informative and interesting panel discussions I’ve seen (not counting the one I spoke on!).

Daniel Tunkelang did a great job of writing up the specific presentations on his site and on the ACM blog, so I won’t attempt to re-create the presentations line by line at this late date. Rather, I’d like to highlight a few open problems and research questions that came out of the discussions that I hope to see developed in the next year.

Social search consists of a set of problems including (but hardly limited to) search of social content like status updates, real-time search, generating, labeling, and finding user-generated content, ‘long-tail’ events and interests, finding vs re-finding, and trend identification.

What data is available to social search? There are many kinds of social data, from e-mail (private) to blogs (public) and tweets (mostly public) — what is and should be searchable? How do we handle issues of privacy and identity management?

How do we compute relevance, taking into account freshness, accuracy, and degrees of social separation?

Will the architecture of these search engines look like the search engines we’re currently familiar with?

How do we evaluate accuracy and truthiness of social data?

How do we characterize social connections, around concepts like strong vs weak ties, and friend-of-a-friend vs friend-of-a-friend’s-friend? Can we converge on a single social graph representation?

How do we best filter social data to lead to accurate recommendations for content discovery? How do we accommodate the fact that as we move beyond static factual data, two people using the same query may be looking for very different results?

Finally, how do we deal with the chasm between the industry participants (who have LOTS of data) and the academic participants, who suffer from a lack of public (and publishable) data?

Thanks again to the organizers – Eugene Agichtein, Marti Hearst, Ian Soboroff, and Daniel Tunkelang – who put together a fantastic event.

For more on this and a cool demo, check out Gene Golovchinsky‘s look at the SSM2010 twitter coverage.

SMS to e-mail gateway: The SMS doorbell

Posted: January 3, 2010 | Author: | Filed under: blog, projects | Tags: , , , | 5 Comments »

Over at NYC Resistor, it was getting cold, and we needed a doorbell so visitors wouldn’t be stranded outside when the building was locked. A standard wireless model didn’t work reliably (the space is on the fifth floor, just out of range), so various members generally resorted to writing their phone numbers on a sign on the front door when they were expecting guests.

Since almost everyone has a mobile phone already, and SMS-based solution seemed appropriate. In order to implement this we need two things:

  1. An SMS shortcode
  2. A system to notify when the shortcode is triggered

It’s irritating and expensive to acquire your own shortcode, but there are several services that will allow you to use one in exchange for a small fee or advertisements in your messages. TextMarks is my favorite (I used TextMarks for my WhereAmI project). While TextMarks markets their service as a system for mobile mailing lists, they allow you to reserve a keyword and define a behavior (that can include pulling data from a URL!) to occur when that keyword is triggered.

Configuring TextMarks

textmarks_configurationSign up for TextMarks and choose a keyword. Configure the keyword to respond with the “First 120 characters on web page”, and point it at the future home of your script (you can always come back and modify this later).

Note the � as the value of the msg parameter — this instructs TextMarks to send along any additional message contents as the value of that parameter. That means if someone were to text 41411 “doorbell hi this is hilary”, TextMarks would call the script with the param msg=hi this is hilary. This can be quite useful.

The Script

This script is written in Python, but you can use any scripting language you like. This particular script just sends an e-mail to an account when the ‘doorbell’ is rung, but you could have it do pretty much anything up to and including ringing a real bell (which may be coming soon!).

#!/usr/bin/env python
# encoding: utf-8
"""
doorbell.py
Created by Hilary Mason, feel free to use this code in your own projects.
"""

import sys, os
import smtplib
import cgi
import cgitb; cgitb.enable()

class Doorbell(object):
	GMAIL_USERNAME = 'YOURGMAILACCOUNT@gmail.com'
	GMAIL_PASSWORD = 'YOURPASSWORD'

	def __init__(self, msg):
		message = """
From: YOURGMAILACCOUNT@gmail.com
To: YOURGMAILACCOUNT@gmail.com
Subject: KNOCK KNOCK, someone is at the door!

%s
		""" % msg

		server = smtplib.SMTP('smtp.gmail.com:587')
		server.ehlo()
		server.starttls()
		server.ehlo()
		server.login(self.GMAIL_USERNAME, self.GMAIL_PASSWORD)
		server.sendmail('YOURGMAILACCOUNT@gmail.com', ['YOURGMAILACCOUNT@gmail.com'], message)
		server.quit()

		print "You knocked! You can also call us at 347-586-9270. <3, NYC Resistor"

if __name__ == '__main__':
	print "Content-Type: text/plainnn"

	form = cgi.FieldStorage()
	if 'msg' in form:
		w = Doorbell(form['msg'].value)
	else:
		w = Doorbell('There is an anonymous monkey at the door.')

And that's it! Provided you have your keyword configured to point at your script, and the script living at an accessible address, you'll get an e-mail whenever your SMS doorbell is rung and the person who sent the message will get back a cute response confirming their action.

Finally...

This setup can be easily extended such that a message containing 'doorbell hilary' could e-mail only me, or be forwarded to my phone.

I'm curious to see if having a remotely accessible 'doorbell' will encourage pranksters -- we might need to add a password.

IgniteNYC: The video!

Posted: December 24, 2009 | Author: | Filed under: academics, blog | Tags: , | 15 Comments »

The video of my IgniteNYC presentation is up, and has gotten a great response!

I’m working on removing the me-specific bits from the code and I’ll be posting it as open-source very soon!

IgniteNYC: How to Replace Yourself with a Very Small Shell Script

Posted: November 25, 2009 | Author: | Filed under: blog, Presentations | Tags: , , , | 14 Comments »

me glowing at ignite I recently gave a talk at IgniteNYC on How to Replace Yourself with a Very Small Shell Script.

The Ignite events are a fun blend of performance, technology, and speaking skill. Each presenter gives a five minute talk with twenty slides that auto-advance after 15 seconds.

The title of my talk is a classic geek reference (you can get the t-shirt). I’m very interested in developing automated techniques for handling the massive and growing amounts of information that we all have to deal with. I started with e-mail and twitter, both of which are easy to access programmatically (via IMAP and the Twitter API).

In the talk, I went through several of the simple and successful e-mail management scripts that I’ve developed.

I decided to talk about this project because I’m not sure where this should go next, but I got some great feedback and I’m looking forward to future work on the project!

The slides are below, and the full talk will be online soon.

IgniteNYC: How to Replace Yourself With a Very Small Shell Script

View more documents from Hilary Mason.

My code is on TV (and so am I)!

Posted: November 10, 2009 | Author: | Filed under: academics, blog, Media | Tags: , , , | 5 Comments »

FoxNY did a piece featuring me and Diana as hackers who use our technical powers for good, not evil.

There are way too few female technologists on television, and I’m happy to do what I can to show that women kick ass with code! Look for my mischievous I’m-writing-infinite-nested-loops grin in the clip where I’m programming.

If this looks like fun to you, come join us at NYC Resistor (where the segment was filmed!) for Thursday night craft nights or for one of many awesome classes.

Yahoo OpenHackNYC: The Del.icio.us Cake

Posted: October 17, 2009 | Author: | Filed under: blog, projects | Tags: , , , , , | 4 Comments »

Last weekend Yahoo came to New York for an Open Hack Day, and it was great!

I was invited to speak on a panel on semantic metadata, moderated by Paul Ford (harpers.org) along with Marco Neumann (KONA) and Paul Tarjan (Yahoo/Search Monkey). The panel was a lively discussion, and we got some great questions from the audience.

After the panel, I stayed around to participate in the hack competition. Yahoo! provided a fantastic space, with free-flowing coffee, snacks, comfy chairs and plenty of Yahoo folks and other hackers around to give advice and play foosball with. I teamed up with Diana Eng, Alicia Gibb, and Bill Ward to create the Del.icio.us Cake!

The cake is attached to a laptop via USB. A program running on the laptop accepts a delicious tag and retrieves a list of recent popular sites for that tag from the delicious API. Finally, it iterates through each URL, downloads the page, and computes the sentiment of that page relative to the tag — basically, is the content of the page positive, neutral or negative?

The signal is output to an ardiuno (hidden in the middle of the cake) which turns on the appropriate set of LEDs. There are four sets of LEDs on the cake, one in each quadrant of the delicious logo, one each for positive sentiment, neutral or inconclusive sentiment, and negative sentiment, and, of course, one to let us know that the cake is turned on.

I wrote the sentiment classifiers between around 3am and 6am Saturday morning, so they really were a hack! I trained them on movie reviews data, working with the assumption that 5-star reviews contain positive terms and 1-star reviews contain negative terms. I wouldn’t recommend this approach for a serious attempt at sentiment analysis, but it worked well enough.

We won the food/hardware hack prize, shared with the awesome MakerBot team!

We had a great time creating and presenting the hack. Thanks, Yahoo, and most of all, thanks to Alicia, Bill, and Diana for a really fantastic, silly weekend.

Further coverage:

Data: first and last names from the US Census

Posted: October 16, 2009 | Author: | Filed under: blog | Tags: , , , | Leave a comment »

I’ve found myself in need of a name distribution for a few projects recently, so I thought I would post it here so I won’t have to go looking for it again.

The data is available from the US Census Bureau (from 1990 census) here, and I have it here in a friendly MySQL *.sql format (it will create the tables and insert the data). There are three tables: male first names, female first names, and surnames.

I’ve noted several issues in the data that are likely the result of typos, so make sure to do your own validation if your application requires it.

The format is simple:

  1. the name
  2. frequency (percentage of people in the sampled population with that name)
  3. cumulative frequency (as you read down the list, the percentage of total population covered)
  4. rank

If you want to use this to generate a random name, you can do so very easily with a query like this:

SELECT name FROM ref_census_surnames n ORDER BY (RAND() * (n.freq + .01)) LIMIT 0,1;

Download it here: census_names.tar.gz

Hadoop World NYC

Posted: October 3, 2009 | Author: | Filed under: academics, blog | Tags: , , | 9 Comments »

Yesterday, I attended the first Hadoop World NYC conference. Hadoop is a platform for scalable distributed computing. In essence, it makes analyzing large quantities of data much faster, and analyzing very large quantities of data possible.

Cloudera did a great job organizing the conference, and managed to assemble a diverse set of speakers. The sessions covered everything from academic research to fraud detection to bioinformatics and even helping people fall in love (eHarmony uses Hadoop)!

I’m not going to review every session, but I saw several themes emerging from the content and conversations.

Hadoop is Getting Easier

New integrated UIs like Cloudera Desktop and Karmasphere mean that developers will no longer be required to use a command-line interface to configure and execute Hadoop jobs. IBM’s M2 project hides Hadoop behind a spreadsheet metaphor, making the collection, analysis and visualization of data as easy as using Excel.

This doesn’t just speed up development time, it puts the tools for manipulating the data directly in the hands of the people who need the results, without requiring them to talk to a database programmer.

Hadoop is a Utility

The only organizations that talked about building their own Hadoop clusters are those who deal with very sensitive data (VISA) and those who deal with very very large quantities of data (Yahoo, Facebook, eBay). Organizations with more manageable data sets, such as eHarmony and the New York Times, use EC2 and Amazon’s Elastic Map-Reduce. Amazon, Rackspace, and Softlayer have offerings in this area and were all event sponsors.

Yes, you can turn on a cluster of nodes from your living room in your PJs!

Hadoop Can Talk to Your Existing Systems

Hadoop has an ecosystem of supporting products that allow organizations to adapt their existing infrastructure. Cloudera’s Sqoop (which is just fun to say out loud) is a tool for importing data from SQL databases, HBase is a Hadoop database, and Pig lets you talk to the system in a SQL-like language.

I expect we’ll see more information available in the near future to clarify which systems are more appropriate for which kinds of users (an ecosystem decision tree?).

Hadoop is Changing Things

I heard the phrase “an order of magnitude improvement in speed” so many times that I lost count. Speaking from personal experience, the difference you see in productivity between waiting minutes and hours for results and waiting days is immense. When you can see the answer to a question shortly after you ask it you can preserve the context you need to act on that answer immediately without having to spend the time to figure out why you were asking that question in the first place.

Most of the projects were doing fairly simple analysis over data like web user sessions or transactions. I was intrigued by Deepak Singh’s talk on bioinformatics and genome sequencing (slides) and Jake Hofman‘s talk on social network analysis (slides). More and more massive datasets are becoming available and will drive techniques for new analysis. I do wish there had been a talk about Mahout, which is a very promising approach to developing machine learning algorithms on the Hadoop platform.

I left the event more excited about the technology and very enthusiastic about the community. Thanks for a great day!

Update: A few other people have written up their notes and impressions from the event:

http://jakehofman.com/talks/hadoopworld_20091002.pdf

http://www.slideshare.net/mndoci/hadoop-for-bioinformatics

← Older Entries
Newer Entries →