E-mail automation, questions and answers

Welcome! I’ve gotten several hundred e-mails about my e-mail management code. I do want to share it as soon as possible. Here are the answers to the most common questions.

Why separate scripts?

My philosophy is based on the unix command-line tool model; Each script should be simple and useful alone, but when combined together they become extremely powerful.

Why don’t we have the code yet?!

I had no idea the talk would be shared beyond the couple hundred people in the audience or that it would be so popular! I started my position at bit.ly the same day I gave that IgniteNYC presentation, and I also have some other awesome projects that are competing for time.

I have to admit that the trained classifiers are all based on my personal data and were also trained mostly through tweaking in ipython. I need to finish a generic framework for people to train their own filters before I can publish that piece of the system. I promise, I’m working on it.

Keep nagging me — nagging works!

Are you going to commercialize your scripts / can I invest?

I have certainly thought about commercializing the application, but I’m uncomfortable asking people to give me access to their personal e-mail data (even if there are very interesting things to be learned by aggregate analysis).

Just imagine how much more creative, interesting work could be done if we could partially free the world from the e-mail workload… that alone is worth making the code open.

How does it work? What tech are you using?

The scripts run on my gmail account through IMAP (and should work with any IMAP interface, though I’m sure there is debugging to be done). They live on a Linode VPS and run individually via cron jobs.

Most of the scripts are in Python. I use NLTK and libsvm (in addition to my own code) for the data analysis.

I primarily use the gmail web interface (though I’ve flipflopped between Mail.app and Thunderbird for a while), and the only cost is that I have to manually reload the page to see new labels and new drafts appear.

Do your scripts go mad with power and e-mail inappropriately? Are you some kinda robot?

I have all of the scripts deposit suggested responses in the draft folder, and then I use the gmail “multiple inboxes” feature to keep the draft folder up in the UI. It’s very easy to go through and modify or delete responses before they are sent.

Of course, I only thought of that after one of the script DID go a bit mad. I’m still sorry about that, Mom.

I’m not a robot, though of course I would say that anyway! The point of the automation is to remove the stupid parts of e-mail and leave me free to personally address the interesting messages.

If you’ve read this far, there are a few things I would love your feedback on:

What’s a kickass name for this project?

More important, which features/scripts are you most interested in seeing first? The nag script is about ready to go, but I’d like to know where to focus my time.

THANK YOU!


66 Comments on “E-mail automation, questions and answers”

  1. […] This post was mentioned on Twitter by Ruset Zeno, Nitya N. Nitya N said: Loved this. http://bit.ly/aDKUzV RT @hmason My attempt to answer some questions about those e-mail management scripts: http://bit.ly/aXKQPu […]

  2. hm, seems irobot is already taken. How about something simple like AutoMail? More encompassing like eOrganize? DistractionKill?

    The nagging script sounds neat, but I think the most early bang per buck would be the importance classifier. After all, even if you have the response suggestor, you are still checking your email constantly to see if there is a message that needs a response. After that, the response suggestor sounds totally kickass. I have a strong personal need for the ‘Finding Patterns’ script, but I don’t see there being as much widespread demand for that one.

    If you need a guinea pig, I would love to alpha test your scripts at any point. I have a few hosted and nonhosted servers, a couple willing (and busy) gmail accounts (gmail and google apps), and coding experience. Let me know if I can be of assistance!

  3. Awesome! I’m mostly looking forward to reading the classifiers. I’m not really sure that the scripts, apart from the nag script, as far as I understand it would be that useful to me. I’m just not sure I get enough email. I find the subject fascinating though!

  4. Todd Eddy says:

    I’ll go and setup a nag script right now and hope I don’t “accidentally” have it scheduled to run every minute instead of every week :)

    I’m curious on the classifier in just how it was accomplished. I first assumed you just used some bayes based program. Be interesting to see how it’s done in python.

  5. awesome stuff as usual…my name suggestion is “mailnooty” (and mailnooty.com is currently available, though I like it so much I might grab it myself if you don’t hurry!)

    Oh and if you need/want any help in getting things ready for release let me know…I don’t mind digging around in non-ready-for-prod. code and trying to clean it all up for mass consumption…

    BTW – you should totally switch to disqus for comments…

  6. A video of your lecture passed by my feed today…loved it. Great idea and great use of statistics (a.k.a. artificial AI) with scripting too. Keep it up and we’ll keep nagging you!

    Cheers.

  7. Martin says:

    Nag, nag, nag.

  8. I’d name it after famous AI’s that hastened or brought about the robocalypse. Skynet, HAL, etc…

  9. Tracy says:

    You NLTK link goes to nltk.com. Don’t you mean nltk.org?

  10. Hilary Mason says:

    @Tracy — yes, I did mean ntlk.org. I updated the post, thank you!

  11. Levin says:

    It should be called ScriptNet :)
    (or something else that feels like SkyNet)

  12. René says:

    Hilary! I want your script, oh please please, nag nag… oh, at least your ideas were pretty cool.

  13. Patrick H says:

    Call it Emai. Email AI = Emai (eee-my)

    I’m personally interested in the learning aspects and cracking into some code outside of my domain (.Net).

  14. Kristina says:

    I like Patrick’s idea. I think Emai is an interesting, cute name. It make me think of a little japenese duck…lol.

  15. Cybe R. Wizard says:

    HilarEmail
    It can’t miss.

  16. AronSora says:

    Project Email (Or Inbox) Bouncer

    Your idea just brings the idea of a bouncer for your inbox. Do you have any information package I could send along to my ComSci professor? He needs to see this.

  17. AronSora says:

    Also, if you could make a way to deal with google alerts. I love google alerts because I get to see everything in my field, but I tend to dump a ton of them looking for a piece of gold. Can you write a program the knows when a google alert is a piece of trash.

  18. Carlitos says:

    IAmNotARobot is a great name if you ask me ;-)

  19. fischer says:

    You should name your project “Unpaid Intern”. Let’s face it – this script is for doing all of the bitch work

  20. Philip Olson says:

    My guess is it won’t be limited to much (e.g., only email), so let’s see. It removes the ‘over’ in ‘overwhelmed’ but that’s not a great word, although vague images of the concept come to mind. Or continuing down that road, we’d be ‘lesswhelmed’ but hmm… okay, I give up today.

    As for upcoming features, I vote for the “Remember Me?” feature described within the ignite talk. It’d also relate to the bogus unwritten rule of waiting for responses before writing back. The “I wrote you last” issue. Emails get lost, people forget things, it happens. So yeah, that’s my vote :)

  21. patrik says:

    When this gets open sourced, will it support other languages than english?

    Sounds like an awesome thing!

  22. Hilary Mason says:

    I love these name suggestions, they’re great!

    @patrik There is currently no support for languages other than English, but for most of the scripts that won’t be too difficult to add.

  23. David Baker says:

    I would call it Sunny, Sunny the iRobot.

  24. Timo Sand says:

    What I’d like to know is, how does your nag script handle e-mail that doesn’t require an answer? It’s quite often that I e-mail a webclip to someone just because I think they might be interested. Also sending job application’s etc. aren’t e-mails you’d be well off nagging about, atleast in my oppinion.

    The importance analyzer would definitely be the first needed script :)

  25. Sean Chitwood says:

    I look forward to running these scripts.

    Do they use OAuth or do you have to configure them with your password?

  26. Tim says:

    Is it bayesian filtering for the importance classifier?

  27. Chris says:

    Nthing the desire to see the code; maybe throw it up on bitbucket or the like, and let people start hacking at it?

    It sounds like a lot of fun. And useful, which is a great combination.

  28. Matt Ledding says:

    Love the idea of your script, would be super useful to automate a lot of things. You are working on super interesting and super useful stuff, and have my full attention.

    NTLK got me started in python, (am still a noob) because it seemed like a great way of searching word collocation for teaching English, without spending heaps of money on private corpus access by big name universities. Only the Scottish corpus is free here in Europe. (but that is for bairns) and I only see historical English corpuses on a free use basis in the States. A real-time corpus like the internet is far more interesting. If youtube were to open up its speech-rec text for corpus use in TESOL we would have an incredible teaching tool. Anyways, enough drooling on my part about distant possiblities, I should be drooling about the ones you are dangling here.

    I am a graduate of Montreal’s national circus school, but have run away from the circus to join a home… am working with schools in Spain to teach English through shows. (web: http://www.fifthbiz.com) I find the paperwork overwhelming when I am travelling/performing so your scripts could be a godsend.
    I have a google form that lets me calculate distance through the charts api, and make a price estimate for shows. Is there a way with scripts that I could automatically send out a computer generated estimate from that form, and then an automatic “nag” until they check in, or automatically letting them know when there is another show in the area with the reductions in cost that that would imply?

    Anyways… it is really neat to see people make cool things happen. Looking forward to seeing the next step.

    Matt

    That would be useful in automating stages of tour scheduling, and give me more time to study python rather than Spanish geography.

  29. Matt Ledding says:

    Sorry, the last line got moved around. cheers

  30. Matt Ledding says:

    Name suggestions you can throw away: Script yourself, replywood, Premeditated, Lifeguard…

  31. Matt Ledding says:

    …Ghostwriter, Talk to the H.A.N.D (Human Augmenting Net Device) Internal Monologue, Preflections, writeoff, WriteOver, WriteOrWrong, Understudy…

  32. People give me a lot for branding but I love your ideas, so here are a couple good ones for free because you want to do open source.
    BotMail , Mind Mail: ” To ReMind & Read MyMind” , Mat-Mail “The Only Mail That Matters”

  33. Chad Marshall says:

    Nag – I think this is approaching the point where people are going to start to forget about it. Your personal brand is stalled and will fizzle out quickly unless something happens (code released) to regain momentum.

    Why is so much (apparent) effort being spent on what it’s called. When people are hungry they don’t want a debate over what items are called on a menu, they want food.

    (Only said to NAG) – Love your work.

  34. Mike says:

    How about just demail? For defensive e-mail, or de-e-mail

    Loved the talk and am enjoying learning about your other projects, cheers

  35. Nag. Wouldn’t early July be a great day to release some code? Maybe while watching some fireworks here in the US?

    I am looking forward to seeing these scripts.

  36. Saidi Smeenk says:

    Nag i want those scrips, and now. :P

  37. Saidi Smeenk says:

    pretty please, scripts scripts

  38. Saidi Smeenk says:

    Focused your time in the frequent ask question and combine it with salesforce. Or i will do that. :)

  39. Alex says:

    Found you via Smarterware.
    I love that you have a script to look up similar e-mails, as that is EXACTLY how I work in my inbox — by saying “how did I respond to the last one of these?”

  40. patrik says:

    nag!

    want scripts!

  41. Daniel says:

    If the scripts are going to be open source why not release them now? As in right now, GO!
    I realize there can be a propensity to keep them private (it’s doesn’t have enough awesome sauce yet) but holding it back is just holding the project back. Many people could find use from this wonderful idea :-) just let it be free!

    I can’t wait to use the scripts!
    Seriously can’t wait :-D

  42. […] Hilary Mason: Email Automation, Questions and Answers – “Welcome! I’ve gotten several hundred e-mails about my e-mail management code. I do want to share it as soon as possible. Here are the answers to the most common questions.“ […]

  43. The C Man says:

    Love the idea so much I want to try it out. You can catch me at the Gmail forums if you fancy letting me do that.

  44. Keenan Brock says:

    Some name ideas/threads to explore:

    – Hilary Mail, HMail, Hail – *shelter*
    – Spam (and cook it with a *pan*
    – Signal to Noise, Harmonic distortion, *timbre*
    – *nag*

    all the star names seem to have no projects on github and when mixed with email, seem to have a reasonable chance at a good google result.

  45. Keenan Brock says:

    I’m curious how you displaying the results.

    1) For email, I’m assuming you set tags on an email message (over IMAP).
    If this is the case, how would you sort your email by the tags? seems gmail is more focused on search than sort.

    2) Do you know if it is possible to extend tweetdeck or another twitter mac client to add filtering / ordering / summary hooks?

  46. Michael Muse says:

    I love this concept. Im a little late to the comments party – but it made me think – what if you had a completely public ‘inbox’ – you could connect all your public data to that profile and ANYONE could contact you based on it (advertisers, whoever). Then the whole thing is about optimizing which stuff rises to the top. That could be a pretty cool thing.

  47. James Seigel says:

    Nag +1

  48. Steve Gorski says:

    I was thinking ‘Email Smacker’ then thought ‘SmackMail’ was smoother name. If it was me I’d probably choose ‘SnackMail’ because a love to eat but that’s another story. All the best with your work and projects.

  49. Justin says:

    name suggestion?

    spam can.

    +1 on the nag

  50. Sam says:

    i have some name suggestions.

    Two Moms.

    No Spam!

    Anti-Spam Machine

    The SSS (Super Spam-Stopper)

  51. Parts of the stuff you talked about in the “Replace yourself with a small shell script” talk can be achieved with Boomerang (http://www.boomeranggmail.com/) — a plugin for GMail. It gives time delayed mail sending (for sending reminders to yourself and others) and have a “boomerang” feature which causes an unanswered mail to re-appear in your mailbox for further actions.

    I don’t think it has the nagging function I remember you talking about, though (haven’t found it yet, if it is there), nor does it has the canned replies feature.

    But something’s better than nothing, right?

  52. Toby says:

    Hi Hilary – would love to build this concept inside http://www.unifiedinbox.com – our aim is to make the most intelligent inbox management solution to manage today communication overload – wether email, social media or others… Happy to connect again!

  53. Wesley Baker says:

    Hello Ms. Mason, I just heard you talking on Public Radio here in the Columbus, OH area. You have a world of knowledge and I appreciate your outlook on life and technology. In addition, from the sound of your voice I would say that you always have a smile on your face. Keep up the great work. Wes Baker

  54. Martin5 says:

    I listened to you on SciFri today and you mentioned a website that would allow me to block my URL information from other servers and websites. I was wondering what that site was.

  55. Daniel says:

    nag +1

  56. Daniel says:

    Yet more incessant nagging.

  57. Arif says:

    nudge/nag/ping

  58. Brian says:

    Since I assume that you’re parsing the emails with regular expressions, then may I suggest REmail?

  59. San Grail says:

    Did you ever post this anywhere?
    I mean, I know you probably want to tidy your code up before you show it to anyone (eeep! Someone saw my code!), but, it would honestly be a lot more useful to just see the basic framework that you’ve used, that I could modify myself, rather than… have no idea.
    :(

  60. AJ Hekman says:

    Any plans to continue work on this? This would be tremendously helpful even in a stripped down form. Perhaps release what you can and the community can contribute?

  61. Its supple design facilitates in the Library Automation System that is installed in the school Libraries.

     

  62. Hi.  Loved the the talk about this (http://www.youtube.com/watch?v=IoQ4tka1zNk) and was excited to get and play with such scripts myself.  Any thought of when you might publish?  Even in rough form, some pointers to methods would be nice to see.

    Thanks.

    – Sef Kloninger (sefklon@gmail.com, @sefk)

  63. Kerma says:

    Still no code?