The problem

From time to time, I have scripts and binaries that I only really need on one system. Since the base dotfiles repo includes a bin directory (for dfm itself), if I just drop files in there, git continually shows me that they are untracked. For instance, if I have a script called only_on_my_mac, then running dfm status shows:

1
2
3
4
5
6

# On branch personal
# Untracked files:
#   (use "git add <file>..." to include in what will be committed)
#
#       bin/only_on_my_mac
nothing added to commit but untracked files present (use "git add" to track)

At the very least, this is annoying. Here, I offer three solutions to this problem.

Solution 1: .gitignore

The first solution is to just drop a .gitignore file in the bin directory and specify each file that needs to be ignored:

1

only_on_my_mac

If you only have a couple scripts, this may work. However, after the third entry, this gets rather tedious.

Solution 2: Create an excluded directory

This solution is a slight modification of the previous one. Instead of ignoring each individual file, create a directory for excluded scripts:

1

mkdir .dotfiles/bin/excluded

Then, ignore all the files inside that directory (except the .gitignore file itself):

1
2

*
!.gitignore

Finally, update .bashrc.load to add the new directory to the path:

1

PATH=$HOME/bin/excluded:$PATH

Now, any scripts that are only for one system can just be dropped into the bin/excluded directory and git/dfm won’t try to track them.

Solution 3: Use dfm recursion

This solution involves modifying the .dfminstall file in the base of the dotfiles repository. Add the following line:

1

bin recurse

Then, when dfm installs, it will symlink any scripts in the dotfiles’ bin directory instead of symlinking the entire directory.

1
2

$ ls -al bin
lrwxr-xr-x  1 jones  staff  13 Oct  4 20:20 bin -> .dotfiles/bin

1
2
3
4
5
6

$ ls -al bin
drwxr-xr-x 3 vagrant vagrant 4096 2012-01-23 01:31 .
drwxr-xr-x 5 vagrant vagrant 4096 2012-01-23 01:31 ..
-rw-r--r-- 1 vagrant vagrant   12 2012-01-23 01:31 only_on_my_mac
drwxr-xr-x 2 vagrant vagrant 4096 2012-01-23 01:31 .backup
lrwxrwxrwx 1 vagrant vagrant   20 2012-01-23 01:31 dfm -> ../.dotfiles/bin/dfm

This solution only started working today. There was an bug in dfm that prevented the bin directory itself from being recurse-able. If you want to use this solution, you’ll need to merge the latest code.

Conclusion

I personally use Solution #2, mostly because I don’t want to keep a list of files to ignore (translation: lazy).

I hope these solutions are helpful. Enjoy.

For the longest time, I was a screen user. Then, a little while ago, I discovered tmux, the next generation terminal multiplexer. Not only is it easier to search for on google, it has a rich and consistent configuration language.

I’ve figured out a rather unique tmux configuration and I wanted to share it.

Background

Originally, I just used tmux on remote servers to control several windows. This made it easy to create new remote windows, but I had to keep multiple Terminal tabs open, one for each remote server. I had long wanted to be able to reconnect with my tabs, much in the same way that I could reconnect with the windows on an individual server. I contemplated just running tmux locally on my mac, but then each new window would have required a new connection, and they wouldn’t be logically grouped.

So I run tmux locally and remotely.

Nested tmux

I manage my nested tmux sessions with three configuration files.

1. .tmux.shared - contains configuration and bindings that are shared between my master and remote sessions
2. .tmux.master - contains configuration unique to my local (master) session
3. .tmux.conf - contains configuration unique to the remote sessions

Shared configuration

Note: bind -n maps a key that works all the time, regular bind maps a key that has to be prefixed with the prefix key.

The shared configuration contains three basic sections:

1. Vim-ish keybindings - I set them whenever I can get them. (-:
2. Misc. configuration - One screen-compatible binding and one I’ll highlight later.
3. Status bar configuration - This I mostly copied from someone else.

Local configuration

The important setting here is updating the prefix to be Ctrl-Alt-b. It took a few days to get used to hitting it, but my left ring finger now drops down to hit the alt key when I want to do local operations.

1

set-option -g prefix M-C-b

One convenience of using Terminal tabs is cruising between them with Shift-Cmd-[ and Shift-Cmd-]. To get a similar facility, I map Ctrl-Alt-h and Ctrl-Alt-l to previous and next:

1
2
3

# window navigation
bind-key -n M-C-h prev
bind-key -n M-C-l next

Remote configuration

My remote configuration file doesn’t make any modifications with regard to nesting tmux sessions. It uses the default Ctrl-b prefix and is named .tmux.conf so that it is the default when tmux is started.

It doesn’t specify next and previous window navigation like the master config because the corresponding choice for keys would be Alt-h and Alt-l, which confuses vim when I need to hit Escape followed by h, which happens rather frequently. I fall back on the normal tmux navigation for next and previous window.

Other nifty settings

Resizing panes

I often split windows into multiple panes. While tmux has some nice default layouts, it is sometimes easier to just move the divisions yourself.

Here’s the configuration section for resizing from my remote configuration:

1
2
3
4
5

# keybindings to make resizing easier
bind -r C-h resize-pane -L
bind -r C-j resize-pane -D
bind -r C-k resize-pane -U
bind -r C-l resize-pane -R

Hitting the sequence Ctrl-b Ctrl-h will make the division between the current pane and the one below it move one line. What makes it usable is the -r flag, which means I can just keep hitting Ctrl-h as many times as I want until the panes look right.

Synchronizing input

Every so often, I want to send the same input to all panes in a particular window. With this configuration, it’s easy to toggle the built in synchronization:

1
2
3

# easily toggle synchronization (mnemonic: e is for echo)
bind e setw synchronize-panes on
bind E setw synchronize-panes off

Conclusion

Using these settings makes it a cinch to reconnect to my entire work environment from anywhere. I have two status bar lines at the bottom of my screen; the lower one is analogous to Terminal tabs and the upper one shows my remote windows.

Enjoy.

Problem

I copy and paste all the time. Most of the time, I copy short pieces of information that are too long to type (I’m lazy) but too short to setup anything more complex (wget, scp, etc.). For a while, this was fine as most of my copy targets were either local to my system or in a terminal window on a remote server. However, as I increased my use of splits in tmux and windows in vim, highlighting remote text with my mouse became horribly cumbersome. I needed a way to copy remote text into my local clipboard.

Partial solutions

Recent versions of Terminal let you select blocks of text when holding down the alt key, but when I copied and pasted, the resulting block of text had extra trailing whitespace.

Another solution I tried was MouseTerm. It’s a SIMBL plugin that sends your mouse events straight through to the remote terminal apps. So, I could “set mouse=a” and then select text in any vim window without overlapping other windows. The only problem was that once the text was selected, I couldn’t copy it back to my local computer.

Then, I found remote-pbcopy. It’s a setup where pbcopy is running in a daemon mode on your local laptop and listening on a specific port. That port is then forwarded to the remote machine with SSH. Finally, a little alias facilitates piping output into that port. The result: remote data ends up in your local clipboard.

This is exactly what I wanted. However, I didn’t like the caveat at the end: there is no security on the listening daemon. This means that any if any malicious (or prank-minded) person can figure out what port you are using, they can smash your local copy buffer.

Remotecopy

My solution to this problem, remotecopy, is an evolution on remote-pbcopy. It uses a secret value, like a password, to authenticate copy requests. To do this, it replaces the client and server with perl equivalents so that a little extra logic can be added.

Here’s the sequence of events.

1. Start remotecopyserver on your local laptop.
2. SSH to a remote host with the following argument: -R 12345:localhost:12345
3. On the remote host, run remotecopy 'test string'
4. Hit cmd-v and enter
5. ‘test string’ is now in your clipboard.

Here’s how it works.

When remotecopy is run, it makes a connection to localhost:12345 (and therefore the remotecopyserver, via SSH). Then, a short handshake is done, followed by the transfer of the copy data.

Before I describe the client side of the interaction, here is how the server operates:

1. On startup, generate a secret string. Listen for connections.
2. When a connection is made, the client will send it’s secret.
3. If the secret matches the local secret, tell the client that it can send the copy data. Read the data and push it into the local clipboard with pbcopy.
4. If the secret is invalid or missing, tell the client so and close the connection. Push the secret string into the local clipboard with pbcopy.

It’s important not to miss the last part of step 4. This makes the secret available later.

Now, back to remotecopy. When remotecopy runs, it doesn’t know the secret from the server. It does the following:

1. Connect to the server and send an empty secret. The server sends back a rejection.
2. Prompt the user for the secret value. (Because the server copied it into the paste buffer, all you need to do is paste (cmd-v) and hit enter)
3. Reconnect to the server, sending the secret and then the copy data.

It’s quite a long description, but the process is very quick. If you already have the secret in your clipboard history, you can pass -s <secret> and remotecopy will only need to make one connection.

Example runs

For each of these examples, after the secret is entered, the data is in the server’s copy buffer.

Copy a simple string.

1
2
3

$ remotecopy foo
Input secret:
rc-b212f4522lle33a689edcca88d6845b8

Copy output of another program.

1
2
3

$ ls | remotecopy
Input secret:
rc-b212f4522lle33a689edcca88d6845b8

Specify secret on command line

Note: no prompt is needed.

1

$ ls | remotecopy -s rc-b212f4520c3e33a689edcca88d6845b8

Using remotecopy with vim

Since I use vim as much as possible, remotecopy includes a vim plugin that enables sending data from remote vim sessions.

To copy the entire file or visual selection, use ,y. To copy a particular buffer, use ,r. When the remotecopy is first attempted, there will be a prompt for the secret. After that, the secret is cached so future copies are quick.

Using remotecopy with dfm

If you’re using dfm to manage your dotfiles, just copy it into your bin directory. Both remotecopy and remotecopyserver are self contained perl scripts that don’t have external module dependencies.

You can also use git subtrees and symlink the vim plugin, like I do here and here.

More information

Each script has full documentation. Just run with the --man option to view it.

The code is available on github.

Enjoy.

As is customary for those who’ve converted from WordPress to Octopress, here’s a quick post about my experience converting this blog.

Getting the blog up and running was a cinch, especially with a good example to examine when I had questions.

Converting old entries

To convert my WordPress entries, I turned to exitwp. It worked pretty well, but I ran into two issues.

The first was that the YAML blob at the top of the converted posts wasn’t formatted correctly.

1
2
3
4
5
6
7
8
9
10
11

---
author: nate
date: '2011-10-29 18:53:01'
layout: post
slug: git-submodules-vs-subtrees-for-vim-plugins-part-2
status: publish
title: Git submodules vs. subtrees for vim plugins, part 2
wordpress_id: '328'
? ''
: - Misc
---

I solved this by switching to chitsaou’s fork.

The second problem was that html2text, which exitwp uses to do the actual conversion, was hard wrapping lines at 78 characters. I fiddled with it for quite a while, hacking the backend code for html2text, but then I remembered that markdown parsers pass HTML straight through (and that I don’t care if old entries are regular HTML). So I just modified the exitwp script to gut the html2fmt method:

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19

$ git di
diff --git a/exitwp.py b/exitwp.py
index ae58d24..d21a8df 100755
--- a/exitwp.py
+++ b/exitwp.py
@@ -37,12 +37,7 @@ item_field_filter = config['item_field_filter']
 date_fmt=config['date_format']

 def html2fmt(html, target_format):
-    html = html.replace("\n\n", '<br>')
-    if target_format=='html':
-        return html
-    else:
-        # This is like very stupid but I was having troubles with unicode encodings and process.POpen
-        return html2text(html, '')
+    return html

 def parse_wp_xml(file):
     ns = {

Old source highlighting plugin

I had used SyntaxHighlighter Evolved in WordPress to handle my syntax highlighting needs, so I needed to convert those to Octopress’ triple backtick format.

For this, I turned to some perl one liners:

1
2
3
4

perl -MHTML::Entities -p0777i -e 's/(\[sourcecode[^]]*\])(.*)(\[\/(sourcecode)\])/$1.decode_entities($2).$3/mse' *.markdown
perl -p -i -e 's/\[sourcecode.*language="([^"] )"[^]]*\]/``` \1\n/' *.markdown
perl -p -i -e 's/\[sourcecode[^]]*\]/```\n/' *.markdown
perl -p -i -e 's/\[\/sourcecode\]/\n```/' *.markdown

That first one uses a little trick to slurp in entire files (seen here) and decode HTML entities.

Finishing touches

And, finally, many urls on my site included the hostname, making the Octopress preview less useful. One more perl one liner:

1

perl -p -i -e 's{http://endot.org/}{/}g' *.markdown

Final verdict

Octopress is awesome.

Enjoy.

WordPress -> Octopress. Details later.

When I talked about submodules vs. subtrees before, one of the things I listed as a benefit for subtrees was the speed of the initial clone.  I’d written a few scripts to help me benchmark the two, and with a little extra time that I have this weekend, I thought I’d share the data.

I generated 2, 4, 6, 8, and 10 plugin repositories for both submodules and subtrees and cloned each one ten times over both a local and a remote connection.  Here is the result:

As you can see, submodules take longer for each one you add and subtrees stay pretty much the same.  Here’s the R code to generate the above graph:

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19

#!/usr/bin/env Rscript

library(ggplot2) # load up the ggplot2 library

# load up the data from the google csv export
smst <- read.csv('data.csv')

# add names to the data
names(smst) <- c('type', 'count', 'time')

# force count to be a factor instead of a continuous variable
smst$count <- factor(smst$count)

# calculate the mean for each type/count group
smst_mean <- aggregate(list(time=smst$time), list(type=smst$type, count=smst$count), mean)

png(filename = "submodule_vs_subtree.png", width=700, height=700)

ggplot(smst_mean, aes(x=count, y=time, group=type, color=type)) + geom_line(size = 2) + ylab("time") + xlab("plugin count") + opts(title = "Submodule vs. Subtree checkout times")

The generation and benchmarking scripts as well as the reported data and code are in my submodule_vs_subtree repo on github.

A couple weeks ago, I had the privilege of crossing the country to attend the jQuery conference in Boston.  It was a unique opportunity for me.  While we use jQuery extensively at work, I haven’t done much with it lately.  However, since we have a few projects coming up that will be web focused, so it was a timely occurrence.  I went with my good friend and coworker David (I’d link to his blog, but he doesn’t have one yet, cough) and another coworker, Erin, who had been to Boston before.

First off, Boston was awesome.  When we arrived on Friday night, we popped over to the conference venue to register before David and I headed into north Boston.  Erin was off to meet some local friends, but not before giving us a quick overview of the city.  We headed up to Hanover Street near North End Park, where we expected to find good Italian food.  We weren’t in the mood for Italian when we reached the area, so we wandered a while before ending up at the Green Dragon Tavern, an establishment that claims to have once been the “Headquarters of the Revolution” for a while.  After starting with a couple beers, we feasted on a Hot Corned Beef (with Mustard Horseradish Sauce) sandwich and some Boston Bangers and Mash.  It was a meal to write home about (or, you know, blog about).  We hadn’t even started the conference and it was already a great trip.  We continued to go out each night after the days festivities, but that first meal was easily the best.

The conference itself was great too.  My lack of recent experience with jQuery meant that almost all the talks were interesting and usually resulted in me adding to my list of things to learn.  Here are some of the highlights (in roughly chronological order):

The jQuery Mobile keynote - Todd Parker & Scott Jehl

I’m quite interested in the mobile web at the moment, and so everything in this talk was fascinating.  They support a whole host of devices (not just iOS) and are very close to releasing 1.0.  Also, there’s an awesome demo of jQuery Mobile’s slide transitions (click here to load the demo directly on your phone).  It’s just regular html with links to images and jQuery Mobile transforms it into a sliding image gallery.

Plugin Authoring Best Practices - Ben Alman

I haven’t created any plugins (yet) for jQuery, but after this talk I not only have the tools and templates to get started quickly, I understand what some of the line noise is when looking at existing plugins.  Also, I learned the term IIFE.

Event Improvements in jQuery 1.7 - Dave Methvin

This was, by far, the most entertaining talk of the weekend. Not only did I learn about the cool new unified event declaration model that’s coming in jQuery 1.7, I got a refresher course on all the memes I have and haven’t seen.

There was quite a bit more from the weekend.  Here are my raw notes, with many many URLs to explore.

All in all, it was a great trip.

In my dfm talk a couple weeks ago, I listed out some low hanging fruit; just a few things that I thought would be easy to add to the system.  Well, this past weekend, I went to the jQuery conference in Boston.  It was a great conference in its own right, and I hope to post on it this week, but for now I want to talk about the improvements I made to dfm while on the plane.

dfm uninstall

The first thing I tackled was the ability to remove dotfiles if they weren’t needed anymore.  Sometimes, you need to log into a shared account (such as root) and you’d like to use your settings, but not leave them behind for the next person.  Now, ‘dfm uninstall’ does just that.

dfm updatemergeandinstall

Up until now, fetching your latest dotfiles changes required two steps: ‘dfm updates’ and ‘dfm mergeandinstall’.  Well, now there’s updatemergeandinstall that does both.  It takes the same flags as either updates or mergeandinstall and works just like it sounds.  Oh, and for the perennially lazy (i.e., me), it has a shortcut named ‘umi’.

The Rest

The test suite covers all the new commands and I added more coverage for the lightly tested mergeandinstall code.  ’dfm install’ now cleans up dangling symlinks, for those times when a file is no longer needed.  And, finally, I refactored the code somewhat to make it easier to work on and to reduce duplication.

To update to the latest, just run these commands:

1
2

$ dfm remote add upstream git://github.com/justone/dotfiles.git
$ dfm pull upstream master

Enjoy.

A couple days ago, I did a presentation on my dotfiles manager (dfm) at the local Perl Mongers meeting.  It was a phenomenally rewarding experience.  The Perl community has always been fun and engaging and this meeting was no different.

Here’s the source for the slides and a PDF rendering for ease of perusing.  It’s a good overview of current usage as well as a laundry list of future features.

Enjoy.

Update: Here are the slides, hosted by speakerdeck.com

I couldn’t easily find information on this, so here’s a summary for those that go after me.

A working example is in the sbt-scalatra-on-dotcloud branch of my fork of my friend’s sbt-scalatra-example project.  If it gets merged in, I’ll update this post.

The sbt-scalatra-example project combines two great technologies.  The first is Scalatra, a super light-weight web framework for Scala that’s modeled after Ruby’s Sinatra.  The second is the simple-build-tool (sbt), a tool for building Scala applications that’s more like rake (config file written in real programming language) than make or ant/maven (config file written in an abstract form).  I was able to run the example, but I wanted to try out deploying to dotcloud, and doing that requires a war file (according to the java service documentation).

The biggest thing I had to figure out is that as of version 0.10 of sbt, web application support is now in the xsbt-web-plugin plugin.  Just knowing that would have saved me an hour this morning. Setting that up and tweaking a few other files means I can now do this:

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15

$ sbt clean package-war
... snip ...
[info] Packaging /home/user/sbt-scalatra-example/target/scala-2.9.1.final/sbt-scalatra-example.war ...
[info] Done packaging.
[success] Total time: 13 s, completed Sep 23, 2011 3:57:00 PM

$ cp target/scala-2.9.1.final/sbt-scalatra-example.war dotcloud/ROOT.war

$ dotcloud create sbtscalatra
Created application "sbtscalatra"

$ cd dotcloud && dotcloud push sbtscalatra
... snip ...
Deployment finished. Your application is available at the following URLs
www: http://sbtscalatra-xxx.dotcloud.com/

Of course, this assumes you’ve installed sbt (brew install sbt on the mac) and signed up and configured Dotcloud.

Have fun.