The hackathon was a blast, I really love working full-out in an environment filled with people doing the same. It didn't hurt to be surrounded by HackNYers, sponsors, and snacks.
The first project I came up with is called ' 🐤🎶'. However, I couldn't submit that name on ChallengePost, so I gave it the alternate title '(chirp)'. That wasn't the end of the issues caused by the name, however: I thought using it as a subdomain broke ngrok in the middle of the demos (but actually, it was just maintenance. Still, a tense 20 minutes for many a demoer).
🐤🎶 is a tool to reduce the length of potential tweets. It uses WordNet inside Python's NLTK to find synonyms for each word in the provided text. For extra concision, it also suggests emoji which might replace some words, and offers to replace stopwords with the NUL character. You might think of this as similar to the lossy pscyhoacoustic compression used in MP3s, in that it retains the human-interpreted meaning while reducing the amount of data the machine needs to store. In reality, semantic compression is a thing in NLP, but the approach used here is as-simple-as-possible, and aims not to reduce the total number of words needed to represent a corpus, but the total number of characters to represent a single text.1 I've got a few ideas up my sleeve about how to improve it, though.
The process of developing this was surprisingly smooth. I've used NLTK quite a bit in the past, the most difficult part was understanding WordNet's jargon (e.g. what exactly lemmas are). I wanted to add a bunch of configuration options, for example to only replace with more-specific or less-specific terms (hyponyms or hypernyms). That didn't make it into v0.1, but there are two options on the page: one to only suggest emojis, and depth which controls how many levels of synonym-of-synonym to suggest. The backend uses flask, a library I've built so many projects with I can pretty much use it blindfolded.
That said, I did end up using jquery-spellcheck by brandonaaron, which does call the Google API. It was straightforward to make it use my backend, because the library uses a proxy script, which I just replaced with my own. Working out what to serve from the backend was less easy, however. As it turns out, defunct-and-never-quite-official APIs don't have great documentation. I used the example return value in the documentation from this Java library to reverse-engineer the XML return format. For future reference, it looks2 like this:
'charschecked' doesn't actually need to be set, and you can use any number of spaces between the suggested replacement words. If I end up working on 🐤🎶 more, I'm going to have to modify this format, because I'd like to be able to suggest replacments for phrases, not just single words.
I made minor modifications to jquery-spellcheck so that it sends additional URL-parameters on its AJAX call for the depth of synonyms and emoji-only mode.
On another note, I demoed this in Chrome Canary, because the stable version doesn't have emoji support yet... As a side effect of this project, I found to my surprise both in how many places you can use emoji (like some TLDs, although finding a registrar is another question), but also in how many places they don't work. I'm going to continue working on the synonym engine part of this because the frontend is pretty much done (and because for my own use I'd prefer a CLI). One interesting option is that — given that WordNet doesn't just give you synonyms but tells you how words are related — I could replace words with more general (or perhaps more common) ones. This would be more similar to what semantic compression does. It would be cool to see whether replacing words with hypernyms results in something similar to the Getting to Philosophy effect on Wikipedia, where successively clicking the first link on an article will eventually lead you up the hierarchy of more-and-more general topics towards philosophy.3 Stay tuned via GitHub.
Also, thanks to JZ Forde for providing initial feedback and encouragement.
Next up: the winner, GLaPEP8.
Of course, this specification might only apply to the API as implemented in jquery-spellchecker... ↩