The Woodshed

Behind here, no one can hear you scream

If You Wish Upon A Cloud...

Keeping Those Config Files Synced

A compadre of mine recently asked how I go about keeping my config files synced. If you are someone who uses customizable programs (IDE/text editors, OS shells, version control software, and the like), and those programs allow storing your settings in some sort of standalone file (whether plain text, XML, binary, or what have you), than this is an issue worth looking at. Reason being, if a program offers customization, and you customize it, you expect it to act the same no matter which computer you're sitting at (or remoted into). And even if the program has a convenient interface for said customizations, you don't really want to spend the time redoing what you've already done.

In fact, I know some (very bright) people who are daunted enough by the task of customization and its maintenance that they prefer to work with programs in their vanilla state, knowing full well that it is suboptimal to do so. (Admittedly these people don't use something like Emacs, which is basically useless without customization.)

What I Use

I will outline my personal scheme for synchronization in this section; alternative schemes will follow.

My method involves a simple directory structure for organizing your config files, the programs Git and Dropbox for grabbing your personalized versions of said files, and some quick setup. The aforementioned tools are free, and clients/front-ends for both are available for all the major OSes. But see the Alternatives section for scenarios when using them might not be straightforward or practical. On with the show...

On your main machine (one-time setup):

  1. Create a folder somewhere, and name it something straightforward (I use "config_files").
  2. Inside that, I like to make a folder for each OS I use, as well as a folder that will contain files common to all platforms. This is purely for organization; you can throw everything in one folder if you want.
  3. Copy all the configuration/customization files/directories that you want to synchronize into this new location.
  4. Assuming you've already installed Dropbox, go to your local Dropbox folder, and within it create another directory. You might want to end its name with ".git", as in config_files.git.
  5. Enter said directory, and use your preferred Git client to init an empty repository there.
  6. Now head back to your config_files directory (where you copied all your stuff). Ensuring that you're in the "top-level" folder of the directories you created, use your Git client again to init another fresh repository.
  7. Still in Git, add and commit all your copied configs. (If you're using the command line, git add . && git commit -m Initial works fine for this. If you're on Windows and have never used Git, a simple guide like this one will get you up and running.)
  8. Finally, use Git to link this repo with the one in your Dropbox. This is called "adding a remote." At the command line: git remote add origin /path/to/your/Dropbox/config_files.git && git push

On your other machines (one-time setup per machine):

  1. Make sure Git and Dropbox are installed (or see Alternatives).
  2. Use Git to clone the repository from your Dropbox into a folder on this machine, e.g.: git clone /my/Dropbox/folder/config_files.git

...And your config files are now on this machine. This seems like a good time for an aside.

Aside: Why Bother With a Repository?

"But Blog Author Guy," I hear you saying, "That Git stuff seemed redundant. Why not just copy the config files into a folder in my Dropbox?"

First of all, the name's Jordan! But yes, I did this myself for about two years. It's fine for maintaining a basic setup on, say, your desktop and your laptop, as long as you're diligent about recopying to and from your Dropbox folder every time you make a change to a file on either machine. But there are some limitations to this approach. For example, you might not be able to install [even the command-line version of] Dropbox on some remote machines. Specifically, the transitory nature of shared hosting screws up Dropbox's authentication scheme. Or you may plain just not want your Dropbox on a computer you don't own.

Using Git opens up the alternative of using a different service to host your config file repository. My personal preference is Bitbucket, which offers free private repositories.

Oh, and of course it never hurts to have the benefits that come with version control anyway—easy to revert a change, and you never have to worry about backups. This is especially true if you tend to change configs on different machines from time to time, and maybe you don't always remember to immediately sync the changes. This can lead to so-called "version conflicts," which programs like Git were designed to handle.

Perhaps most importantly, I've simply found it easier to keep things synced when all I have to do is type git pull. Manually copying files doesn't seem labor-intensive at first glance, but you'd be surprised what a drag it can be.

What I Use, Continued

In fact, I like to eliminate copying files from the procedure altogether. If possible, I like to be free to make modifications to a config file on any machine, and then be able to sync those changes with one line of Git (which I've even made an alias for, heh). Symlinks to the rescue!

Important note for Windows users: Your OS does actually support symlinks! I'll say some more about this in a bit, but tl;dr Don't skip this section.

In any version control software I've used, it's expected that you keep tracked files in the directory structure of the repository. But as you know, programs expect to find their configuration files in various specific places. For example, a common place for one's Emacs init file is ~/.emacs.d/init.el (or in %APPDATA% instead, on Windows). Instead of copying init.el to that location, though, I place a symlink there that points to the file in my repo.

This works great, because the programs don't know the difference, and yet the files are still version controlled and easy to sync.

If you're on Linux or OS X, you probably already know how to symlink a file (just remember the arguments go in the same order as for cp, and be careful with relative paths).

I was surprised to learn that Windows supports symbolic and hard links, since for some stupid reason nobody ever uses them (probably since you can't create them directly in the Explorer GUI; but note you can view and use the links fine from the GUI). The filesystem has had this support since the Win2K days, and then in Vista they added a command-line utility that makes link creation as easy as on *nix. So open up cmd and type:

mklink "C:\path\where\program\looks.whatever" "C:\my\repo\path\file.whatever"

(The only two differences to note from unix ln -s are the reversed argument order, and that if you're linking a directory you should provide the /D switch.)

Alternatives

  • If you don't want to use Dropbox at all, I'd definitely recommend using a (likely private) hosted repository from a service like Bitbucket or GitHub. The process outlined above will work much the same, except the "remote" that you add to your repo (and from which you'll clone on other computers) will be the URL your service provides, instead of a local file path. Note that in this method, every time you sync it will ask you for password confirmation, though (unless you go through the effort to set up public-private key pairs).
  • If you don't want to use any sort of cloud-y service, you can get away with using git archive on your repo, and sending the result to other machines via something like (S)FTP or even email. This will make it a pain to sync any future changes, though. (You can also use this method plus a thumbdrive if you're going to some godforsaken planet that lacks Internet access.)
  • Similarly, if you use Dropbox most of the time, but you can't get it working on a particular machine, you can just use the Dropbox website to get a download link to your repo's folder. (PROTIP: If you want to use wget to fetch a link from Dropbox, append ?dl=1 to said link [which will conveniently skip the dumb confirmation redirect page])
  • You don't have to use Git, of course. In fact, there used to be some advantages to using Mercurial instead (it had better Windows support for a long time, for example). Nowadays they're pretty equivalent from a usability standpoint (same would be true of Darcs or pretty much any other DVCS—you don't need to worry about their differences for a simple usage like this). You could alternatively use Subversion or another centralized server (which obviates Dropbox, of course). If you do, make sure your server will be accessible from the various machines you can expect to be working on (keeping in mind possible blocked ports and firewalls, for example).

Do You Believe In Nomagic?

Command Wrapping in Vim: A Case Study

A writeup on something I finally got around to doing: a custom wrapper for Vim’s :substitute command. Like most editor scripting, it was probably more work than it was worth, but if you use Vim you know that you use :s all the time. So if you have even a minor quibble with it, you get to multiply that quibble times the hundreds or thousands of times you’re going to use that command.

I had 2.5 quibbles with it, to be exact. Let’s get the half-quibble (quib?) out of the way. The default range that :s operates on is the current cursor line, though I tend to use it on the current line quite a bit less than on the visual selection or the whole file. And even the “convenient” way to operate on the whole file makes you type a %, which is clearly a B-teamer in the QWERTY world.

The more significant of my complaints involves its interpretation of characters in the ‘replacement’ pattern. Basically, the default behavior makes just enough sense that you think you understand it, until all of a sudden it bites you. (Specifically, you might think that every character therein is treated literally unless it’s preceded by a backslash [or is a backslash], but you’d only be mostly right.)

The other whole-number quibble is caused by the same underlying “feature” as the previous one: magic. This is Vim’s odd term for which characters in an expression are treated as metacharacters, instead of taken literally. There are 4 levels of “magic”: magic, no magic, very magic, and very no magic (and no, I’m not making that last one up). The option that controls this is global, and thus you are threatened to never change it or you’ll basically never be able to use anyone else’s code ever. The default value is magic, which works more or less okay for quick searches, but tends to require an ugly amount of escaping when you’re using :s or :g (this being my last bone to pick).

You’re allowed to override the magic setting of a particular pattern without changing the global value. This is done by prefixing the pattern with either \m, \M, \v, or \V in accordance with the levels mentioned above. Handy, but not pretty, and ain’t nobody got time for typing \v all over the place. Also, amazingly, you can’t affect the “magicness” of :s’s replacement pattern with these, and yet this pattern depends on the magic setting (which, remember, you aren’t allowed to change).

So now the stage is set for my wrapper around :substitute, which consists of a custom function and a couple of command aliases (cabbrevs).

Most of the coding time, however, was spent on getting a cabbrev to work how a rational person would want it to. Let’s say you pick ss for your abbreviation, like I did. Well, you only want certain sses that you type in a command line (or search, because cabbrevs apply there too for some reason) to be expanded to your command. (This is also why it’s never feasible to use cmap, which would overapply mappings ruthlessly).

So I wrote a couple auxiliary functions. One defines your cabbrev in such a way that, whenever it would be expanded, it runs a “hook” to control what’s actually outputted. The other is that hook—it looks at the current contents of the command you’d typed thus far to see if the abbreviation’s expansion should be outputted.

Anyhoo, here’s the code. You can just put it somewhere in your .vimrc. By default, you’ll get two new “commands”—:ss and :ssl. The first essentially makes your pattern very magic and your substitution no magic. The second makes the pattern very no magic and the substitution no magic. They both assume a range of the whole file unless you specify your own range when you use it (just like :g). So:

" Replace all foofoo in file with foo&bar
:ss/(foo)\1/\1&bar/g
" Replace the first (*_*) on the current line with (^v^)
:.ssl/(*_*)/(^v^)/

I’m still doing a bit of testing, but so far things seem to work!

Gee, Mail

Configuring Gmail for use with other mail clients

I’ve used Gmail since it was in beta (wait, is it still in beta? Well, since it was invite-only), and stuck with the web interface for pretty much the entire time. As far as web interfaces go, it’s always been decent, but the real motivator was because other mail clients notoriously haven’t had a very good track record of exposing Gmail’s functionality.

Sure, it’s not all their fault. From what I understand, Gmail has an odd IMAP implementation, and it certainly approaches email as a whole in a non-traditional way. For example, it doesn’t have “folders”—instead, individual messages have one or more labels that you can use to categorize/filter/search by. (Your “inbox” really just consists of all messages that have an inbox label.) Gmail has also grouped emails by “conversations” since its inception, as opposed to a strict sort by date or some other property of individual messages.

The most recent wrinkle is automatic categorization of new mail that arrives in your inbox—and it’s damn good categorization. I was a bit leery of it at first, but I think it’s been over a year now, and the messages it has auto-categorized have been spot on (I have the Social and Promotions categories enabled). It’s saved me the trouble of having to manually create filters to do the same thing.

Unfortunately, its categorization is apparently also done in a way that standalone mail clients can’t find an equivalency for. So that’s kept me using the web interface (and the Gmail mobile app on my gadgetry).

But Mavericks’ Mail.app looked pretty slick, so today I figured I’d try my hand at getting categories working with it (it already supports conversation grouping, server-side drafts, and sort-of handles labels too).

The first obstacle was message count. This notebook has a small SSD in it, so I definitely didn’t want my entire history downloaded locally. Oddly, the app didn’t let me set a limit client-side.

Fortunately, Gmail lets you do this server-side:

Log in to the web interface, click the Gear icon and go to Settings, and click on “Forwarding and POP/IMAP.”

On this page, you can set a limit on how many messages will be downloaded to an IMAP folder, keeping in mind that every label will likely be represented by an IMAP folder in your client. Due to this, you should probably also configure which labels will be exposed to IMAP clients:

Go to the settings category for Labels, and check/uncheck “Show in IMAP” as appropriate.

I left Sent Mail, Drafts, and All Mail checked, as well as a label named “Personal” that I’d created. Note that depending on your mail client, it may or may not be desirable to have “All Mail” checked. For example, in past versions of Apple Mail, it was recommended to uncheck it—but the Mavericks version has a better understanding of Gmail’s structure, and uses “All Mail” as a sort of canonical index.

I also saved some space by adjusting a Mail.app client option:

Settings > Accounts > Gmail IMAP > Advanced > Uncheck “Automatically download attachments”

On to category support. I won’t waste any more of your time by telling you what I tried that didn’t work. Here’s what I ended up going with:

  • Run the following search in the Gmail web interface: to:(your_address_here@gmail.com) AND -category:social AND -category:promotions (note the dashes before the category specifiers).
  • The returned results should look equivalent to your “Primary” inbox view. If so, click More > Create filter and make sure the above query is showing in the “Has words:” textbox. Click Create filter with this search.
  • In the next dialog, tell Gmail to Apply a label (I used a label called “Personal,” but you can name it whatever). Check the “Also apply filter to matching conversations” box, and then hit Create Filter.
  • With that filter created, all messages in your “Primary” inbox tab will now be tagged with the label you chose. Similarly, when you receive a new email, it will automatically be thusly labeled if Gmail didn’t categorize it as Social or Promo.
  • Finally, you can go back to your desktop mail client and tell it to use that label/IMAP folder as your “default” mailbox. I used Mail’s “Smart Mailbox” feature for this, but I’m sure equivalent functionality is available in any halfway decent client.

Woooooooo

Prompting Expletives 2

Electric Boogaloo

Alright, picking up where I left off previously

After abbreviating long hostnames, I thought it’d be neat to add some color to my prompt. I’d never used the standard mechanism for setting font properties in a console before; basically how it works is, you type an escape sequence representing a certain non-printing character, and any text following that gets the chosen attribute. Unsurprisingly, the attribute codes consist of dense, unreadable gobbledygook. So, some variables:

norm_color='\033[33m'   # The "33" before the "m" represents the color yellow
abbr_color='\033[1;32m' # "1;32" means "bold text, green color"
no_color='\033[0m'      # "0" undoes the effect of previous attributes

The above took some work to get correct. These sequences are called “ANSI escape codes.” ANSI stands for “A Needed Standard, Ignored”, I assume, since support for these “standardized” sequences varies from terminal to terminal. Indeed, if you do a bit of Googling, you’ll see various ways of representing the Escape character (\e, \E, \ESC x1B, \033 were a few examples), different meanings assigned to the numbers separated by the semicolon, different colors represented by the same numbers, etc. In the original code I adapted which used ANSI color setting, it used \e which worked exactly half the time on my machine (it was properly understood by Bash when embedded in the PS1 string, but not understood by my shell’s echo builtin).

Fun fact: if you want to test these out on your terminal, you’ll probably have to pass an argument (usually -e) to echo to have them be understood at all.

Funner fact: if you want to use non-printing characters (such as the color sequences) in your PS (prompt string) definition, you have to enclose them in square brackets. Square brackets with backslashes. So:

norm_color='\[\033[33m\]'
abbr_color='\[\033[1;32m\]'
no_color='\[\033[0m\]'

I wanted directory separators in my path to always be the normal color, and didn’t feel like typing that out every time:

dir_sep="$norm_color/"

(Just in case you are super new to (ba)sh, note that the $ sigil announces to the shell that the word following is a variable whose value should be substituted. Don’t make the assumption that it works like Perl/PHP; if you use that sigil in an assignment statement like $foo=bar, the shell will substitute the current value (or lack thereof) of foo before interpreting the rest of the expression. So what actually gets executed might be something like 42=bar—probably an unrecognized command.)

(Also note that everything you type is interpreted as a string—you don’t need to surround strings with quotes to identify them as such to the interpreter. [The quotes have a different meaning.] The above could equivalently have been written dir_sep="$norm_color"/)

The code I adapted also had a cute line for changing the color of your username based on whether or not you are the superuser:

[ "$UID" -eq 0 ] && user_color='\[\033[1;31m\]' || user_color='\[\033[1;34m\]'

Ultimately, I ended up abandoning this line. (Not that I didn’t like the concept, but because in practice I had a hell of a time getting my prompt [and specifically, the custom function I call to set it] to properly carry over through the various combinations of sudo and su I was using to test it out. Sometimes I set arbitrary constraints on things, though [such as in this case, I made it a “requirement” that I wouldn’t edit any system-wide files like /etc/bashrc or /etc/sudoers to accommodate this playing around]. I think it’s a mental condition.)

However, I left the line in this blog post because it’s pretty Bashy. It works similarly to the “ternary operator” in C, and looks strikingly similar to an idiom used in Lua and other languages. As you’ve gleaned by now, looks are deceiving, and it actually works differently (which I think is the “theme” of these Bash blogs). I won’t go into detail, but know that it’s not using traditional Boolean logic or operator precedence to do what it’s doing.

Oh, and the way the original author got it working is by putting all of his code into a quoted literal string which he embedded in the PS1 prompt string variable; thus when the shell would eval $PS1, his code would get executed (even in superuser context, assuming PS1 is exported and he switches users in a way that keeps that environment variable).

I imagine that what constitutes “eval abuse” in Bash is much hazier than in other dynamic languages that allow the practice. Still, that seemed pretty ugly to me, and I wanted to keep things in a function. (Doing so is a bit more efficient [not that shell scripts are an area where you probably care about efficiency much] in addition to being much cleaner organization-wise.)

So with that in mind, I can start pasting from my actual code:

COLOR_ABBR_PS () {
    local norm_color='\[\033[33m\]'
    local abbr_color='\[\033[1;32m\]'
    local user_color='\[\033[1;34m\]'
    local no_color='\[\033[0m\]'
    local dir_sep="$norm_color/"
# ...
}

I found function declarations in Bash to be quite amusing. For one thing, the () that looks like it’s there for specifying arguments actually isn’t for arguments at all—those parens are merely there to indicate to the interpreter that COLOR_ABBR_PS is a function definition. (Alternatively, you can use the keyword function and omit the parens, but apparently that is “less portable” to other shells.) Indeed, the way you actually refer to function arguments is (surprise!) more retarded: Bash defines some global-ish variables for you during script execution named $1, $2 etc. that refer to the arguments passed to that script. Then for some reason, in a function definition, it replaces those “positional parameter variables,” making them refer instead to that function’s arguments. Oh, and I thought this remark in the manual was funny too: “When a positional parameter consisting of more than a single digit is expanded, it must be enclosed in braces.” Hooray for arbitrary rules!

It came as a surprise to me that given the aforementioned parameter silliness, as well as general absence of namespacing and scoping rules in Bash, that function-local variables are actually supported (sort-of; it’s not true lexical scoping). Within a function definition you can use the local built-in command when you declare a variable, and then that variable’s name won’t hang around after the function returns.

Here I am, writing too many words again. Weirder stuff awaits, I promise.