E-mail automation, questions and answers

Welcome! I’ve gotten several hundred e-mails about my e-mail management code. I do want to share it as soon as possible. Here are the answers to the most common questions.

Why separate scripts?

My philosophy is based on the unix command-line tool model; Each script should be simple and useful alone, but when combined together they become extremely powerful.

Why don’t we have the code yet?!

I had no idea the talk would be shared beyond the couple hundred people in the audience or that it would be so popular! I started my position at bit.ly the same day I gave that IgniteNYC presentation, and I also have some other awesome projects that are competing for time.

I have to admit that the trained classifiers are all based on my personal data and were also trained mostly through tweaking in ipython. I need to finish a generic framework for people to train their own filters before I can publish that piece of the system. I promise, I’m working on it.

Keep nagging me — nagging works!

Are you going to commercialize your scripts / can I invest?

I have certainly thought about commercializing the application, but I’m uncomfortable asking people to give me access to their personal e-mail data (even if there are very interesting things to be learned by aggregate analysis).

Just imagine how much more creative, interesting work could be done if we could partially free the world from the e-mail workload… that alone is worth making the code open.

How does it work? What tech are you using?

The scripts run on my gmail account through IMAP (and should work with any IMAP interface, though I’m sure there is debugging to be done). They live on a Linode VPS and run individually via cron jobs.

Most of the scripts are in Python. I use NLTK and libsvm (in addition to my own code) for the data analysis.

I primarily use the gmail web interface (though I’ve flipflopped between Mail.app and Thunderbird for a while), and the only cost is that I have to manually reload the page to see new labels and new drafts appear.

Do your scripts go mad with power and e-mail inappropriately? Are you some kinda robot?

I have all of the scripts deposit suggested responses in the draft folder, and then I use the gmail “multiple inboxes” feature to keep the draft folder up in the UI. It’s very easy to go through and modify or delete responses before they are sent.

Of course, I only thought of that after one of the script DID go a bit mad. I’m still sorry about that, Mom.

I’m not a robot, though of course I would say that anyway! The point of the automation is to remove the stupid parts of e-mail and leave me free to personally address the interesting messages.

If you’ve read this far, there are a few things I would love your feedback on:

What’s a kickass name for this project?

More important, which features/scripts are you most interested in seeing first? The nag script is about ready to go, but I’d like to know where to focus my time.

THANK YOU!


  • Pingback: Tweets that mention E-mail automation, questions and answers -- Topsy.com

  • http://justinspowers.com Justin Powers

    hm, seems irobot is already taken. How about something simple like AutoMail? More encompassing like eOrganize? DistractionKill?

    The nagging script sounds neat, but I think the most early bang per buck would be the importance classifier. After all, even if you have the response suggestor, you are still checking your email constantly to see if there is a message that needs a response. After that, the response suggestor sounds totally kickass. I have a strong personal need for the ‘Finding Patterns’ script, but I don’t see there being as much widespread demand for that one.

    If you need a guinea pig, I would love to alpha test your scripts at any point. I have a few hosted and nonhosted servers, a couple willing (and busy) gmail accounts (gmail and google apps), and coding experience. Let me know if I can be of assistance!

  • http://docgno.me Dave Kerschner

    Awesome! I’m mostly looking forward to reading the classifiers. I’m not really sure that the scripts, apart from the nag script, as far as I understand it would be that useful to me. I’m just not sure I get enough email. I find the subject fascinating though!

  • http://vrillusions.com Todd Eddy

    I’ll go and setup a nag script right now and hope I don’t “accidentally” have it scheduled to run every minute instead of every week :)

    I’m curious on the classifier in just how it was accomplished. I first assumed you just used some bayes based program. Be interesting to see how it’s done in python.

  • http://blog.botfu.com Kevin Marshall

    awesome stuff as usual…my name suggestion is “mailnooty” (and mailnooty.com is currently available, though I like it so much I might grab it myself if you don’t hurry!)

    Oh and if you need/want any help in getting things ready for release let me know…I don’t mind digging around in non-ready-for-prod. code and trying to clean it all up for mass consumption…

    BTW – you should totally switch to disqus for comments…

  • http://babblingdweeb.com Justin Silles

    A video of your lecture passed by my feed today…loved it. Great idea and great use of statistics (a.k.a. artificial AI) with scripting too. Keep it up and we’ll keep nagging you!

    Cheers.

  • Martin

    Nag, nag, nag.

  • http://www.syzygryd.com/ Daniel C. Silverstein

    I’d name it after famous AI’s that hastened or brought about the robocalypse. Skynet, HAL, etc…

  • Tracy

    You NLTK link goes to nltk.com. Don’t you mean nltk.org?

  • http://www.hilarymason.com Hilary Mason

    @Tracy — yes, I did mean ntlk.org. I updated the post, thank you!

  • Levin

    It should be called ScriptNet :)
    (or something else that feels like SkyNet)

  • René

    Hilary! I want your script, oh please please, nag nag… oh, at least your ideas were pretty cool.

  • http://patpack.blogspot.com/ Patrick H

    Call it Emai. Email AI = Emai (eee-my)

    I’m personally interested in the learning aspects and cracking into some code outside of my domain (.Net).

  • Kristina

    I like Patrick’s idea. I think Emai is an interesting, cute name. It make me think of a little japenese duck…lol.

  • Cybe R. Wizard

    HilarEmail
    It can’t miss.

  • http://www.habitationintention.com/ AronSora

    Project Email (Or Inbox) Bouncer

    Your idea just brings the idea of a bouncer for your inbox. Do you have any information package I could send along to my ComSci professor? He needs to see this.

  • http://www.habitationintention.com/ AronSora

    Also, if you could make a way to deal with google alerts. I love google alerts because I get to see everything in my field, but I tend to dump a ton of them looking for a piece of gold. Can you write a program the knows when a google alert is a piece of trash.

  • http://esloveniacorazon.si Carlitos

    IAmNotARobot is a great name if you ask me ;-)

  • fischer

    You should name your project “Unpaid Intern”. Let’s face it – this script is for doing all of the bitch work

  • http://blog.roshambo.org/ Philip Olson

    My guess is it won’t be limited to much (e.g., only email), so let’s see. It removes the ‘over’ in ‘overwhelmed’ but that’s not a great word, although vague images of the concept come to mind. Or continuing down that road, we’d be ‘lesswhelmed’ but hmm… okay, I give up today.

    As for upcoming features, I vote for the “Remember Me?” feature described within the ignite talk. It’d also relate to the bogus unwritten rule of waiting for responses before writing back. The “I wrote you last” issue. Emails get lost, people forget things, it happens. So yeah, that’s my vote :)

  • patrik

    When this gets open sourced, will it support other languages than english?

    Sounds like an awesome thing!

  • http://www.hilarymason.com Hilary Mason

    I love these name suggestions, they’re great!

    @patrik There is currently no support for languages other than English, but for most of the scripts that won’t be too difficult to add.

  • http://www.DavidbBaker.com David Baker

    I would call it Sunny, Sunny the iRobot.

  • Timo Sand

    What I’d like to know is, how does your nag script handle e-mail that doesn’t require an answer? It’s quite often that I e-mail a webclip to someone just because I think they might be interested. Also sending job application’s etc. aren’t e-mails you’d be well off nagging about, atleast in my oppinion.

    The importance analyzer would definitely be the first needed script :)

  • Sean Chitwood

    I look forward to running these scripts.

    Do they use OAuth or do you have to configure them with your password?

  • Tim

    Is it bayesian filtering for the importance classifier?

  • http://twitter.com/offby1 Chris

    Nthing the desire to see the code; maybe throw it up on bitbucket or the like, and let people start hacking at it?

    It sounds like a lot of fun. And useful, which is a great combination.

  • http://www.fifthbiz.com Matt Ledding

    Love the idea of your script, would be super useful to automate a lot of things. You are working on super interesting and super useful stuff, and have my full attention.

    NTLK got me started in python, (am still a noob) because it seemed like a great way of searching word collocation for teaching English, without spending heaps of money on private corpus access by big name universities. Only the Scottish corpus is free here in Europe. (but that is for bairns) and I only see historical English corpuses on a free use basis in the States. A real-time corpus like the internet is far more interesting. If youtube were to open up its speech-rec text for corpus use in TESOL we would have an incredible teaching tool. Anyways, enough drooling on my part about distant possiblities, I should be drooling about the ones you are dangling here.

    I am a graduate of Montreal’s national circus school, but have run away from the circus to join a home… am working with schools in Spain to teach English through shows. (web: http://www.fifthbiz.com) I find the paperwork overwhelming when I am travelling/performing so your scripts could be a godsend.
    I have a google form that lets me calculate distance through the charts api, and make a price estimate for shows. Is there a way with scripts that I could automatically send out a computer generated estimate from that form, and then an automatic “nag” until they check in, or automatically letting them know when there is another show in the area with the reductions in cost that that would imply?

    Anyways… it is really neat to see people make cool things happen. Looking forward to seeing the next step.

    Matt

    That would be useful in automating stages of tour scheduling, and give me more time to study python rather than Spanish geography.

  • http://www.fifthbiz.com Matt Ledding

    Sorry, the last line got moved around. cheers

  • http://www.fifthbiz.com Matt Ledding

    Name suggestions you can throw away: Script yourself, replywood, Premeditated, Lifeguard…

  • http://www.fifthbiz.com Matt Ledding

    …Ghostwriter, Talk to the H.A.N.D (Human Augmenting Net Device) Internal Monologue, Preflections, writeoff, WriteOver, WriteOrWrong, Understudy…

  • http://www.golfnow.com Steve Novotny

    People give me a lot for branding but I love your ideas, so here are a couple good ones for free because you want to do open source.
    BotMail , Mind Mail: ” To ReMind & Read MyMind” , Mat-Mail “The Only Mail That Matters”

  • Chad Marshall

    Nag – I think this is approaching the point where people are going to start to forget about it. Your personal brand is stalled and will fizzle out quickly unless something happens (code released) to regain momentum.

    Why is so much (apparent) effort being spent on what it’s called. When people are hungry they don’t want a debate over what items are called on a menu, they want food.

    (Only said to NAG) – Love your work.

  • http://outoftheblur.com Mike

    How about just demail? For defensive e-mail, or de-e-mail

    Loved the talk and am enjoying learning about your other projects, cheers

  • http://bnbeckwith.com Benjamin Beckwith

    Nag. Wouldn’t early July be a great day to release some code? Maybe while watching some fireworks here in the US?

    I am looking forward to seeing these scripts.

  • Saidi Smeenk

    Nag i want those scrips, and now. :P

  • Saidi Smeenk

    pretty please, scripts scripts

  • Saidi Smeenk

    Focused your time in the frequent ask question and combine it with salesforce. Or i will do that. :)

  • Alex

    Found you via Smarterware.
    I love that you have a script to look up similar e-mails, as that is EXACTLY how I work in my inbox — by saying “how did I respond to the last one of these?”

  • patrik

    nag!

    want scripts!

  • Daniel

    If the scripts are going to be open source why not release them now? As in right now, GO!
    I realize there can be a propensity to keep them private (it’s doesn’t have enough awesome sauce yet) but holding it back is just holding the project back. Many people could find use from this wonderful idea :-) just let it be free!

    I can’t wait to use the scripts!
    Seriously can’t wait :-D

  • Pingback: Jeff Barr’s Blog » Links for Sunday, July 18, 2010

  • The C Man

    Love the idea so much I want to try it out. You can catch me at the Gmail forums if you fancy letting me do that.

  • http://reflectivepixel.com Keenan Brock

    Some name ideas/threads to explore:

    - Hilary Mail, HMail, Hail – *shelter*
    - Spam (and cook it with a *pan*
    - Signal to Noise, Harmonic distortion, *timbre*
    - *nag*

    all the star names seem to have no projects on github and when mixed with email, seem to have a reasonable chance at a good google result.

  • http://reflectivepixel.com Keenan Brock

    I’m curious how you displaying the results.

    1) For email, I’m assuming you set tags on an email message (over IMAP).
    If this is the case, how would you sort your email by the tags? seems gmail is more focused on search than sort.

    2) Do you know if it is possible to extend tweetdeck or another twitter mac client to add filtering / ordering / summary hooks?

  • http://wildapricot.com Frank Goertzen

    nag +1

  • http://www.google.com/profiles/michaelmuse Michael Muse

    I love this concept. Im a little late to the comments party – but it made me think – what if you had a completely public ‘inbox’ – you could connect all your public data to that profile and ANYONE could contact you based on it (advertisers, whoever). Then the whole thing is about optimizing which stuff rises to the top. That could be a pretty cool thing.

  • James Seigel

    Nag +1

  • Steve Gorski

    I was thinking ‘Email Smacker’ then thought ‘SmackMail’ was smoother name. If it was me I’d probably choose ‘SnackMail’ because a love to eat but that’s another story. All the best with your work and projects.

  • Justin

    name suggestion?

    spam can.

    +1 on the nag

  • Sam

    i have some name suggestions.

    Two Moms.

    No Spam!

    Anti-Spam Machine

    The SSS (Super Spam-Stopper)

  • http://zrajm.org/ Zrajm C Akfohg

    Parts of the stuff you talked about in the “Replace yourself with a small shell script” talk can be achieved with Boomerang (http://www.boomeranggmail.com/) — a plugin for GMail. It gives time delayed mail sending (for sending reminders to yourself and others) and have a “boomerang” feature which causes an unanswered mail to re-appear in your mailbox for further actions.

    I don’t think it has the nagging function I remember you talking about, though (haven’t found it yet, if it is there), nor does it has the canned replies feature.

    But something’s better than nothing, right?

  • http://www.unifiedinbox.com Toby

    Hi Hilary – would love to build this concept inside http://www.unifiedinbox.com – our aim is to make the most intelligent inbox management solution to manage today communication overload – wether email, social media or others… Happy to connect again!

  • Wesley Baker

    Hello Ms. Mason, I just heard you talking on Public Radio here in the Columbus, OH area. You have a world of knowledge and I appreciate your outlook on life and technology. In addition, from the sound of your voice I would say that you always have a smile on your face. Keep up the great work. Wes Baker

  • Martin5

    I listened to you on SciFri today and you mentioned a website that would allow me to block my URL information from other servers and websites. I was wondering what that site was.

  • Daniel

    nag +1

  • Daniel

    Yet more incessant nagging.

  • http://www.facebook.com/daniel.l.wilcox Xavier L Jr Wilcox

    we heart open code.

  • Arif

    nudge/nag/ping

  • Brian

    Since I assume that you’re parsing the emails with regular expressions, then may I suggest REmail?

  • Pingback: Call Center et (Tele)marketing Artificiel, et diviser le nombre d’employé du tertiaire par DEUX « singularite

  • San Grail

    Did you ever post this anywhere?
    I mean, I know you probably want to tidy your code up before you show it to anyone (eeep! Someone saw my code!), but, it would honestly be a lot more useful to just see the basic framework that you’ve used, that I could modify myself, rather than… have no idea. :(

  • AJ Hekman

    Any plans to continue work on this? This would be tremendously helpful even in a stripped down form. Perhaps release what you can and the community can contribute?

  • http://www.eduberry.com/eduberryschool_overview/ School Automation Software

    Its supple design facilitates in the Library Automation System that is installed in the school Libraries.

     

  • http://profiles.google.com/sefklon Sef Kloninger

    Hi.  Loved the the talk about this (http://www.youtube.com/watch?v=IoQ4tka1zNk) and was excited to get and play with such scripts myself.  Any thought of when you might publish?  Even in rough form, some pointers to methods would be nice to see.

    Thanks.

    - Sef Kloninger (sefklon@gmail.com, @sefk)

  • http://scriptogr.am/kerma Kerma

    Still no code?