David Trejo's Thoughts http://blog.dtrejo.com Most recent posts at David Trejo's Thoughts posterous.com Mon, 02 Jan 2012 14:48:00 -0800 Convert all line endings to LF http://blog.dtrejo.com/convert-all-line-endings-to-lf http://blog.dtrejo.com/convert-all-line-endings-to-lf

Why? So that you can check your node_modules folder into git, whether or not there are mixed line endings in files.

Use dos2unix, which you can get using macports:

sudo port install dos2unix

And then use it:

dos2unix filetoconvert.js

Or if there are many offending files:

find path/to/bad/files/ | xargs dos2unix

And then git add whatever you like and you’ll be good to go!

Permalink | Leave a comment  »

]]> http://files.posterous.com/user_profile_pics/245625/sculptureme.jpg http://posterous.com/users/36EZ12gGuYmJ David Trejo DTrejo David Trejo Fri, 23 Dec 2011 22:38:00 -0800 How to detect that node.js is running on Windows http://blog.dtrejo.com/how-to-detect-that-nodejs-is-running-on-windo http://blog.dtrejo.com/how-to-detect-that-nodejs-is-running-on-windo

Do something like this:

if (process.platform === 'win32') {
  console.log('This is windows');
} else {
  console.log('%s is not windows, I hope!', process.platform);
}

Needed this to update run.js to work on Windows. Still not done making it work, but sometime soon it will happen.

The main reason I need to do this is because fs.watchFile works on Mac/Linux, but only fs.watch works on windows. Once I have it working correctly I'll post an update with code.

Cheers!
David

Permalink | Leave a comment  »

]]> http://files.posterous.com/user_profile_pics/245625/sculptureme.jpg http://posterous.com/users/36EZ12gGuYmJ David Trejo DTrejo David Trejo Fri, 21 Jan 2011 19:35:00 -0800 Scraping Made Easy with jQuery and SelectorGadget http://blog.dtrejo.com/scraping-made-easy-with-jquery-and-selectorga http://blog.dtrejo.com/scraping-made-easy-with-jquery-and-selectorga

_write_less_do_more

A few days ago I was doing a TON of scraping, and as you know, without the right tools, scraping can be a REAL pain. Out of my pain comes your pleasure — here’s a list of scraping tools and resources which will make your life MUCH easier the next time you need some information from a crufty old website. If you’re short on time, skip to the end and read the tl;dr.

Scraping with jQuery and Node.js

Scraping with jQuery is a real pleasure. Here’s some example code to get you started, it gets the top three articles with their point values from Hacker News. The key part:

var window = jsdom(agent.body).createWindow()
    , $ = require('jquery').create(window);
  
  // scrape!
  var titles = $('.title a')
    , points = $('.subtext span');
    
  var printme = $.map(points, function(el, i) {
      if (i < 3) {
        return $(el).text() + '\t' + $(titles[i]).text();
      }
    });
  
  console.log(printme.join('\n'));

To get the example working, make sure you’ve installed node.js and npm). You may find run.js very helpful for auto-rerunning your scraper whenever you make a change.

Test your code live against the page with jqueryify

Now that you can run any jQuery code you’d like, you can start testing your code against the page. I like to use firebug or Google Chrome’s built in console. Not all pages have jQuery included, and sometimes the version of jQuery will be old, so it’s best to overwrite it with your own version using the jqueryify bookmarklet (it’s also a good idea to change the version of jQuery used by the bookmarklet to match the one used by jsdom to avoid any strange bugs).

Find the shortest selector with Selector Gadget

Next up is finding the correct selectors for the information you want to extract. Selector Gadget makes it very easy to find the least complicated selector that still does the job. Of course you can always choose selectors by hand if you don’t fancy selector gadget, but it is super helpful for crufty nasty sites.

Scrape dynamic pages that use javascript to populate information (arg!)

At this point you can easily find selectors and test on the page live, but there is danger ahead if you encounter a site that uses javascript or AJAX to populate the information. Your tests will work just fine, but when you load the page programmatically, the page’s javascript won’t be run. There’s a few ways to get around this:

  1. Use a regex to extract the values from the javascript written to script tags in the page.
  2. Pretend to be the page and Make requests to the AJAX urls to get the information you need
  3. URL hack. There may still be a URL hack you haven’t tried which will give you the information you need. (in this case, request is your friend.)
  4. Run the page’s javascript programmatically, then scrape the information after it’s been slotted into the page. (See jsdom’s documentation.).

Regex: tools to make your life easier

txt2re.com is a regex generator. That’s right, generator, not tester. This means you enter in a string, click the parts you’d like to match, and then copy and paste the regex into your code. For me, this is pure bliss.

regexpal.com helps you quickly test your regular expressions against test data. Protip: only type the regex, omit the / at the front and end which you’d normally use in your javascript.

tl;dr

  • Use jQuery and node.js for your scraping. Example code
  • Test on the live page with the jqueryify bookmarklet
  • Find the best selectors with Selector Gadget
  • For pages with inline script tags containing information, use txt2re.com to generate regular expressions, and regexpal.com to test them.
  • And finally, don’t forget that URL hacking is your friend!

Thanks for reading — if you have any feedback, please leave a comment below!

PS If things are going wrong, here’s the list of common gotchas:

  • use the same version of jQuery to test as the one you use in your scraper
  • make sure the information you need is not loaded by javascript on the page

Permalink | Leave a comment  »

]]> http://files.posterous.com/user_profile_pics/245625/sculptureme.jpg http://posterous.com/users/36EZ12gGuYmJ David Trejo DTrejo David Trejo Tue, 07 Dec 2010 15:47:00 -0800 Touch Tone Tanks featured on TechCrunch TV! http://blog.dtrejo.com/touch-tone-tanks-featured-on-techcrunch-tv http://blog.dtrejo.com/touch-tone-tanks-featured-on-techcrunch-tv

Techcrunch_logo
Our app, Touch Tone Tanks, was featured at 6:45 on TechCrunch TV.

See the video of Touch Tone Tanks

Check out 6:45 of the above video :)

- David Trejo

PS yeah that was pretty repetitive.

Permalink | Leave a comment  »

]]> http://files.posterous.com/user_profile_pics/245625/sculptureme.jpg http://posterous.com/users/36EZ12gGuYmJ David Trejo DTrejo David Trejo Mon, 22 Nov 2010 15:42:00 -0800 Always be Coding http://blog.dtrejo.com/34207327 http://blog.dtrejo.com/34207327

It is very difficult to always be coding. Here is a list that helps me make progress even when I’m presented with a hairy complicated technical problem. This list also helps soldier on even if I’m not feeling particularly motivated, or in the mood.

Mind Tricks

I’ve ordered this list from hardest to easiest in an attempt to trick my brain into doing at least one of these things:

  • Finish the whole project in one day. BAM.
  • Pick an isolated sub-problem and solve that
  • Take a 10 minute break and go outside, then come back and look at the problem again
  • Write pseudocode
  • Write really horrible and ugly and wrong pseudocode, with tons of TODOs and FIXMEs.
  • Find a friend who is doing work, and make them guilt you into working with them [New! A highly effective tip suggested by Charles Yeh]
  • write tests and example cases to help focus your implementation [New! Big hat tip to Jeremy Hoffman]

On a somewhat different axis:

  • Take a pen and paper and sit outside, far away from any screens, and just draw out the problem and how it flows
I actually love drafting with a pen and paper, as you described. Somehow it's easier for me to draw out bullet points, boxes, lines, and bits of pseudocode than it is for me to conceive of it on a screen. A trick I've been doing lately, especially when I have to learn a new system or API with many components, is to put each key function or object on a Post-it note. I stick the Post-Its on my desk, or on a single sheet in my notebook if I want to carry my diagram with me (or draw lines between post-its instead of on them). When I need to change my diagram, I don't have to erase anything, I can just move or replace the applicable Post-Its. High-tech!  — Jeremy Hoffman
  • Write pseudocode with a pen and paper
  • Take nap, but ONLY after you’ve done one of the above things for at least 5 minutes!

If you have any mind tricks, please share!

– David

PS Make sure to use calendaraboutnothing.com to really keep yourself honest (suggested by jettiene of #node.js — thanks!).

Permalink | Leave a comment  »

]]> http://files.posterous.com/user_profile_pics/245625/sculptureme.jpg http://posterous.com/users/36EZ12gGuYmJ David Trejo DTrejo David Trejo Tue, 02 Nov 2010 11:34:00 -0700 Watch me demo my phone-controlled game! 7pm tonight! (#nytm #node.js #twilio #socket.io) http://blog.dtrejo.com/watch-me-demo-my-phone-controlled-game-7pm-to http://blog.dtrejo.com/watch-me-demo-my-phone-controlled-game-7pm-to

Nytm

Update: You can now see a video demo of touch tone tanks! (It turns out we didn't crash and burn)

Good afternoon!

Later today Paul, Justin, and myself will be demoing our phone-controlled tanks game* in front of 900 people at New York Tech Meetup! (I hope we don't crash and burn)

You can watch us live at livestream.com/nytechmeetup at 7pm tonight.

Cheers,

David

 

* In case you're curious about how this works:

First you set up a projector or other large screen, then you bust out your phones, call our phone number, and press the buttons on your keypad to shoot or move around. The tones are sent to the game over all kinds of crazy internet infrastructure (Twilio, node.js, and socket.io if you're curious. We've even open-sourced the communication bits of it so you can build your own phone controlled game or web application. See nodaphone for documentation and source.)

Permalink | Leave a comment  »

]]> http://files.posterous.com/user_profile_pics/245625/sculptureme.jpg http://posterous.com/users/36EZ12gGuYmJ David Trejo DTrejo David Trejo Fri, 03 Sep 2010 13:26:00 -0700 Node.js for Server Newbs http://blog.dtrejo.com/nodejs-for-server-newbs http://blog.dtrejo.com/nodejs-for-server-newbs

Node

This is the story of how a webserver noob and thrifty college student got access to a server on which to run node.js.

This guide is for those who have used node.js locally, but want to run it on a server and show their projects to the world.

This assumes (almost) no familiarity with how hosting works, but does assume some familiarity with node.js and building from source. This guide will help you install node.js on an Ubuntu 10.4, 64-bit machine.

Here we go:

Sign up for web hosting at webbynode. Why did I choose webbynode? It offers hosting for $10/month with root access. Only prgrmr competes†, but they have $4/month worth of support, which is scary for those new to hosting, namely myself. I also shied away from webfaction ($9.50/month), because while they do allow you to install node.js in your home directory, there are all kinds of configuration headaches (I think). In any case, these instructions are specific to webbynode with the Ubuntu setup. Update: to get 99 cents off with webbynode, use davidt at checkout.

 

 

Under deploy, click the bare linux image, then choose ubuntu 10.4, 64-bit. This may take a a minute or so. When it is active, continue on.

On your dashboard you should see the IP of your webby, it looks like this: x.x.x.x

Open up a console* on your local machine, and run these at the command line:

localmachine$ ssh root@x.x.x.x

(Enter your password to log into your webby server)

This is key, as it allows you to do everything else:

webby# apt-get update
webby# apt-get install git-core g++ make curl libssl-dev

(python is already installed)

 

Now to download the node files and build them (make sure to use the latest version from http://nodejs.org/#build):

webby# curl -O http://nodejs.org/dist/node-v0.2.0.tar.gz
webby# tar xvzf node-v0.2.0.tar.gz
webby# cd node-v0.2.0
webby# ./configure
webby# make
webby# make install
webby# node --version
>0.2.0

You're ready to write some JS and serve people up!

Cheers!

-David

 

PS In order to start serving your site from yourdomain.com, you'll need to follow webbynode's instructions for changing the dns setting with whoever you bought your domain from. Here's their guide: http://guides.webbynode.com/articles/webbymanager/dns-setup.html Don't forget to migrate your blog and other things!

Bonus:

Set up security on your machine with http://guides.webbynode.com/articles/security/ubuntu810-ufw.html

Use Tim's guide to restart your node process when it crashes [just pay attention to the upstart parts, if you want to keep moving parts to a minimum]

http://howtonode.org/deploying-node-with-spark

 

† that I know of — let me know if you find other good options!

* If you are using a pc you may need to install something like putty so that you can use ssh.

Permalink | Leave a comment  »

]]> http://files.posterous.com/user_profile_pics/245625/sculptureme.jpg http://posterous.com/users/36EZ12gGuYmJ David Trejo DTrejo David Trejo Sun, 15 Aug 2010 20:57:00 -0700 $1100 clock made with jQuery http://blog.dtrejo.com/1100-clock-made-with-jquery http://blog.dtrejo.com/1100-clock-made-with-jquery
The QlockTwo, made in Germany, is a beautiful device.

I set out to steal some of that beauty and made a version of the QlockTwo using jQuery. You can see it live right here.

Qlocktwo

Building this clock was a ton of fun. It took me a couple of days, and I learned about the relativity of time (ooh sophisticated).

Here are some fancy psychological things that go on when you use the clock:

The first half of the hour creeps along — you don't notice the dots as much as the words, so most of the time it's later than what you see in words. The only way to stay accurate is to pay lots of attention to the dots, which requires more thought and means it takes you longer to read the clock.

The second half of the hour races. The words are always either in synced with or ahead of the real-time. This means it that you'll finish meetings earlier, and be more punctual because you'll give yourself extra time to get places. Just make sure you always schedule your appointments on the hour.

Technical Details

I select the words to be highlighted in a really jenky way. It is very dependent on the structure of the table (yeah I tried to use divs, but I don't have the skill to keep all the words aligned horizontally and vertically and keep everything square too). Here's roughly how the code works:

  • Javascript asks for the time 
  • Time is transformed into words by a for loop which used to be a GIGANTIC composite switch and if else statement. It was a mess.
  • The text in all the table cells is put together into one long string. There is a one-to-one relationship between the table cells and the letters in the string. This means that if I match the word "OCLOCK" in the string, and the O in OCLOCK is the 50th character in the string, then I know I can just light up the 50th table cell and it will contain the "O." That wasn't the most clear way of saying things. Here's how it goes: I want to highlight OCLOCK, the code finds the position of OCLOCK in the string, then highlights the table cells in that position. 
  • Every 400 milliseconds the clock's face is refreshed. It is important to me that the clock change times in a smooth manner, and it is important that the clock light up in a visible way when you load the page. To achieve a smooth fadein, there is a css transition of .12 seconds. This makes things smooth when the clock changes times. The 400 millisecond fadein leaves the page time to load when you first fire it up, so you get to experience the initial glow when the clock lights up. 
  • Now you might be thinking, if the CLOCK REFRESHES EVERY 400 MILLISECONDS THEN WHY ISN'T IT FLASHING LIKE CRAAAAAZYYY?? That's how it was initially, and it was ugly as sin (if you consider sin ugly). Here's the solution: compare the previously lighted letters with the new ones that need to be lighted, unlight the ones not present in the new list, and then light up the new ones. This turns out to be a bit complicated and it turns out Venn diagrams made with water colors are a key tool of the modern-day programmer. 
  • What other cool stuff does the clock have in it? I stole a glow from midtonedesign.com, which adds the halo behind the clock (cheers Jonotan). I also chose a font from the google font directory that mimics the QlockTwo's font quite well. And of course I threw in some text-shadows to make things glow. Mandatory.

Conclusion

 jQuery is tons of fun to work with, and the #jquery channel is full of helpful people. I emailed QlockTwo and apparently my version is a "Nice Application with JQuery and CSS!"

The time distorting aspects of the clock are quite fun, so make sure to put one up in your office so you can wrap up boring meetings more quickly.

Cheers,
David

 

Permalink | Leave a comment  »

]]> http://files.posterous.com/user_profile_pics/245625/sculptureme.jpg http://posterous.com/users/36EZ12gGuYmJ David Trejo DTrejo David Trejo Sat, 14 Aug 2010 16:43:00 -0700 Hotkeys for Hacker News http://blog.dtrejo.com/hotkeys-for-hacker-news http://blog.dtrejo.com/hotkeys-for-hacker-news
I was bored so I made gmail-style hotkeys for Hacker News. You can install the chrome extension here.

Hacker_news_hotkeys_frontpage

I thought I'd add support for Vim and Emacs style hotkeys, but then I didn't.

Hope you like the chrome extension. If you are concerned about privacy, it *only* runs on http://news.ycombinator.com/*

Have a great day,
David
(DTrejo on HN)

PS Here's the source if you fancy seeing it: http://github.com/DTrejo/HN-Hotkeys Most of the action goes on in hnkeys.js

Permalink | Leave a comment  »

]]> http://files.posterous.com/user_profile_pics/245625/sculptureme.jpg http://posterous.com/users/36EZ12gGuYmJ David Trejo DTrejo David Trejo Mon, 02 Aug 2010 10:45:00 -0700 How to Stay Up-to-date with Node.js http://blog.dtrejo.com/how-to-stay-up-to-date-with-nodejs http://blog.dtrejo.com/how-to-stay-up-to-date-with-nodejs

Node

Node.js is changing every day and new releases are very frequent. Here is how I stay up to date with the latest release version of Node.js. 

This guide assumes you already have node.js installed and you are familiar with the command line.

1. Install npm (full npm install guide)

curl http://npmjs.org/install.sh | sh

Or update it if you've already installed it

npm install npm

2. Install nave with npm

npm install nave

3. Use nave to run the version of your choice in a subshell

nave use 0.1.102

4. All done!  Repeat from step 3 whenever a new version of node is released. By the way, npm is great, you can even use it to update itself :)

 

What tools do you use to stay up to date? If there is a better way, please let me know!

Permalink | Leave a comment  »

]]> http://files.posterous.com/user_profile_pics/245625/sculptureme.jpg http://posterous.com/users/36EZ12gGuYmJ David Trejo DTrejo David Trejo Tue, 27 Jul 2010 16:15:00 -0700 Meta Rules http://blog.dtrejo.com/meta-rules http://blog.dtrejo.com/meta-rules

Healthcare_epistemocrat

Check out my guest post to Brent Pottenger's blog, Healthcare Epistemocrat.

The post centers on meta rules and the ways I make my life more enjoyable.

Here's the link:

David Trejo's Meta Rules

 

Permalink | Leave a comment  »

]]> http://files.posterous.com/user_profile_pics/245625/sculptureme.jpg http://posterous.com/users/36EZ12gGuYmJ David Trejo DTrejo David Trejo Thu, 01 Jul 2010 14:41:00 -0700 How to Install Node.JS on Windows http://blog.dtrejo.com/how-to-install-nodejs-on-windows http://blog.dtrejo.com/how-to-install-nodejs-on-windows

Nodejs_logo

Update 11/23/11: Download the official node.exe binary or .msi at http://nodejs.org/#download; install npm using these steps http://npmjs.org/doc/README.html#Installing-on-Windows-Experimental (note that there are plans to include npm in the .msi installer).

Update: Here are the official instructions: github.com/joyent/node/wiki/Building-node.js-on-Cygwin-(Windows)

 

I wanted to install node on my windows machine and I stumbled across this thread.

All you need to do is download the file below and then unzip it.

http://dl.dropbox.com/u/341900/nodejs/nodejs-0.1.99-cygwin-full.zip 

Then run bash.bat and do something of this sort:

 cd ~/Desktop

 nodejs-0.1.99-cygwin-full/node/bin/node.exe example.js

A big thanks to Simon Sturmer for posting the link!

mirror: http://dl.dropbox.com/u/10047/nodejs-0.1.99-cygwin-full.zip

 

Update 11/23: Cygwin is no longer supported. See the top of the page.

The less easy way (will work even if node 0.1.99 gets old)

Go to http://nodejs.org/#download and download the zip file.

Download cygwin at http://www.cygwin.com/ (it is a dependency of node). The download link is at the top right. This could take a while.

Open cygwin via start > Cygwin > Cygwin Bash Shell

Navigate to the location of the zip file, using "ls" and "cd" commands. Mine is on my desktop.

When the zip file is in the current directory, run 

tar zxvf node<tab>

(Press tab after you type node to trigger auto-complete so there are no typos)

Congrats! Now you have an unzipped version of node on your desktop. Now to set up a version of python 2.4 or greater.

Download the latest version of python at http://www.python.org/download/ (it is a dependency of node)

When that finishes, go back to the Cygwin window you had open earlier. cd into the node directory you just unzipped.

Run

./configure
make
make install

You're done! Super congrats since you did it the hard way!

If you feel like running the tests, run 

make test

Hopefully I made the process less painful; please let me know if this could be improved in the comments.

Enjoy!

Permalink | Leave a comment  »

]]> http://files.posterous.com/user_profile_pics/245625/sculptureme.jpg http://posterous.com/users/36EZ12gGuYmJ David Trejo DTrejo David Trejo Sat, 01 May 2010 19:25:00 -0700 Untitled http://blog.dtrejo.com/17928885 http://blog.dtrejo.com/17928885

A while ago I updated dtrejo.com. It looks prettier now, though it could use some more color.

Permalink | Leave a comment  »

]]> http://files.posterous.com/user_profile_pics/245625/sculptureme.jpg http://posterous.com/users/36EZ12gGuYmJ David Trejo DTrejo David Trejo Sat, 17 Apr 2010 18:51:47 -0700 Untitled http://blog.dtrejo.com/16594264 http://blog.dtrejo.com/16594264
Imag0079

Permalink | Leave a comment  »

]]> http://files.posterous.com/user_profile_pics/245625/sculptureme.jpg http://posterous.com/users/36EZ12gGuYmJ David Trejo DTrejo David Trejo Sat, 17 Apr 2010 18:36:44 -0700 Untitled http://blog.dtrejo.com/16593643 http://blog.dtrejo.com/16593643
Imag0078

Permalink | Leave a comment  »

]]> http://files.posterous.com/user_profile_pics/245625/sculptureme.jpg http://posterous.com/users/36EZ12gGuYmJ David Trejo DTrejo David Trejo Fri, 19 Feb 2010 15:17:26 -0800 Untitled http://blog.dtrejo.com/11997059 http://blog.dtrejo.com/11997059
Imag0026

Permalink | Leave a comment  »

]]> http://files.posterous.com/user_profile_pics/245625/sculptureme.jpg http://posterous.com/users/36EZ12gGuYmJ David Trejo DTrejo David Trejo Fri, 19 Feb 2010 15:16:18 -0800 Untitled http://blog.dtrejo.com/11996961 http://blog.dtrejo.com/11996961
Imag0025

Permalink | Leave a comment  »

]]> http://files.posterous.com/user_profile_pics/245625/sculptureme.jpg http://posterous.com/users/36EZ12gGuYmJ David Trejo DTrejo David Trejo Fri, 19 Feb 2010 10:54:44 -0800 Untitled http://blog.dtrejo.com/11985653 http://blog.dtrejo.com/11985653
Imag0024

Permalink | Leave a comment  »

]]> http://files.posterous.com/user_profile_pics/245625/sculptureme.jpg http://posterous.com/users/36EZ12gGuYmJ David Trejo DTrejo David Trejo Fri, 22 Jan 2010 15:12:18 -0800 Untitled http://blog.dtrejo.com/10412648 http://blog.dtrejo.com/10412648
Imag0002

Permalink | Leave a comment  »

]]> http://files.posterous.com/user_profile_pics/245625/sculptureme.jpg http://posterous.com/users/36EZ12gGuYmJ David Trejo DTrejo David Trejo