Duplicate Files

I’ve been hoarding data for more than 20 years. For backups, I used to burn a CD periodically, but I long since ran over those limits. Today, my backups are hard drives. One reason is that I’ve moved between computers several times during that period, and when I do, I find stuff I don’t know what to do with. So I copy all that data into a new folder, typically called something like temp/backup/that-system-name/tmp/old/save/keep/t.files/save.d.

After 20 years, that starts to add up. So I’ve been looking at programs to help me find and get rid of duplicates. (I’ve been using rsync -n, and occasionally diff -qr, to compare folders. But the problem is deciding what folders, at what places in the directory structure, to compare.)

So I’ve been looking to see what kind of tools are available to help. At this point, I looked at duff, jdupes, and dupd.

So far, I’ve focused on dupd. It does what I was thinking needed to be done: crawl the entire hierarchy and save the result as a database.

What I did after installing Linux

I had a hard drive fail on my laptop, so I put the best parts of several broken machines together into a sum-greater-than-the-whole new machine. Right now, I’m installing Ubuntu Linux 19.04. And, since it’s been a long time since I blogged what I do afterward, here is the mid-2019 edition.

References: the usual “what to do” blog posts, e.g., this, this, and this.

Start by installing updates:

$ sudo apt update && sudo apt dist-upgrade

Then gnome tweaks, to make your GUI act less lobotomized:

$ sudo apt install gnome-tweaks

The specific tweaks I want are left-side buttons, static workspaces, desktop icons for home but not trash, the size of the monospace font, and (since this is a 13″ laptop) the scaling factor. I also add a percentage to the battery indicator and weekday to the clock. While I’m monkeying around with my settings, I set up night mode. And then I make sure that Alt-Tab behaves correctly. (See here.)

Then I install the usual assortment of web browsers:

$ sudo apt install chromium-browser

(I also install Google Chrome from a .deb I download from their website.)

Then a whole bunch of things I need:

$ sudo apt install caffeine
$ sudo apt install vim

Then, before I forget, I install libreadline for other things to use:

$ sudo apt install libreadline-dev

Next, I finish installing the usual software subjects:

$ sudo apt install imagemagick colordiff jhead wv pandoc abiword antiword eyed3

Stuff from other systems

Then I’m ready to start pulling things from other machines. The easiest way to do that is to go over there and rsync them to me. So:

$ sudo apt install openssh-server
$ ssh-keygen  # hitting ENTER at each prompt
$ ssh-copy-id {whatever the other machines are}

Then I pop over to those machines and send the appropriate contents back here.


Then I install ruby so I can use all the ruby tools I’ve developed over the years:

$ sudo apt install libssl-dev zlib1g-dev
$ git clone git://github.com/sstephenson/rbenv.git ~/.rbenv
$ git clone git://github.com/sstephenson/ruby-build.git ~/.rbenv/plugins/ruby-build
$ cd ~/.rbenv/plugins/ruby-build
$ hash -r  # might not be necessary
$ rbenv install ( --list | some-version )
$ gem install coderay csv fileutils kramdown mini_magick optimist pericope pry tty-color tty-command tty-screen tty-table zxcvbn-ruby

More Software

Also Telegram, if this machine will be used for your private messaging.

And Dropbox.

installing eyeD3

I keep thinking I should learn the basics of Python programming. But I never seem to get around to it.

Today I needed to know how to install a program written in Python, because Homebrew used to include eyeD3 and today it didn’t. Here’s what I ended up doing:

$ sudo easy_install pip   # because pip isn’t installed

$ sudo pip install eyeD3

then it says that won’t work because libmagic isn’t installed. But fortunately, Homebrew provides that (whatever it is). So try again:

$ brew install libmagic

$ sudo pip install eyeD3

P.S. eyeD3 is the best command-line ID3 tag editor I’ve found. It is the only thing I’ve found that allows you to install album art from the command line. (See my earlier post.)


iMac configuration

I mentioned previously how I’ve rehabilitated the 10-year old iMac. Today, I’ve been setting up the software environment.

First, I transferred my iTunes library over to the new computer.

Then I installed some non-Safari browsers: Chrome, Firefox, and Brave.

Then I set up Homebrew, and after that, ruby:

$ brew install openssl # various messages because Apple doesn’t like openssl
$ brew install sqlite pcre
$ git clone git://github.com/sstephenson/rbenv.git ~/.rbenv
$ git clone git://github.com/sstephenson/ruby-build.git ~/.rbenv/plugins/ruby-build
$ rbenv install -l
$ rbenv install 2.4.1 # (as of May 19, 2017)
$ hash -r # rehash your environment

Setting Up Homebrew on my New Old Mac

I set up Homebrew on my iMac. (If you go to their site, the Homebrew project has documentation there, which explains the whole not-use-sudo philosophy and using /usr/local now that Apple makes it hard not to, but I don’t understand it, so it seems like an invitation to having your machine rooted. But I’m a greybeard and I quit understanding internet security back when we used uucp because we didn’t have a real internet. Hopefully someone has thought it through.)

Because I’m too lazy and ignorant (and possibly stupid) to RTFM, I found a set of color-by-number instructions handy, although I didn’t use anything after XCode (a mere 4GB download) and Homebrew.

$ brew doctor
$ brew update
$ brew doctor
$ brew install {lots of good stuff}

At some point I ran into keg-only and Cellar jabber jabber messages, but (see above) I don’t understand any of that. I just take whatever advice brew doctor gives me and move on from there.

Some of the things I always want on a new Mac (or even one that’s 10 years old) include:

$ brew install imagemagick --with-fontconfig
$ brew install colordiff pwgen tree wdiff pstree
$ brew install pngquant
$ brew install jhead antiword poppler
$ brew install sox lame eye-D3
$ brew install gnupg
$ brew install sqlite

Moving iTunes from one mac to another

The biggest single reason I wanted to fix my old iMac was to put a honkin’ big HD in it, and the reason for that was so I could have an iTunes library on the main hard drive instead of an external (USB 2.0) drive. My iTunes Media folder looks like this:

32 Books
60 Tones
740 Home Videos
6886 Mobile Applications
10791 TV Shows
53041 Movies
84845 Podcasts
95847 Music

Why iTunes doesn’t let us manage our storage intelligently is a mystery (especially given the ridiculously small storage options that Apple provides its Macs, and all their other devices for that matter). But now I’ve got plenty of room to store 250 GB of media in the iTunes folder as well as the different places I keep it for real. So there.

The only problem was getting it into the iTunes folder, in a way that iTunes understands. But this is a known problem, and Apple has a documented procedure.

Then I had to Authorize my computer (the “new” iMac) and Deauthorize the old one (the Mac Mini). I don’t know how Apple keeps track of what computers have been authorized, but they decided this one was already Authorized, from back when it was our main computer. I don’t know if I forgot to Deauthorize it or what, but now I’ve only got 2 of 5 Authorized computers.

Ubuntu 16.10 and Zeroconf

I don’t do enough system admin to enjoy it when I do. (Does anyone?)

When you upgrade Ubuntu, you lose a lot of the software you’d installed. There’s a list of system admin tips that I try to remember here. (Which reminds me, I should find a better home for it for when Twitter dies.)

I couldn’t resolve names of Macs in my local network. I should point out what a shame it is that Mac addresses aren’t the same thing as MAC addresses, since Google as my primary system admin resource.

That’s the work of avahi-daemon and friends (a/k/a Zeroconf and Bonjour). For awhile I thought that something had broken there. But eventually, I figured out that was working, i.e., the problem lay elsewhere.

I wondered if they’d added a firewall to 16.10. They did (or, for all I know, they’ve always had one) called ufw (more here), but it’s (still) not enabled by default. If that ever changes, I can learn what to do about it here.

After about an hour, I found out there is something called nss-resolve. Which is actually a pretty clever idea. Except it wasn’t working. Its configuration file is well documented here. Good luck figuring that out.

Finally, I just compared my 16.10 /etc/nss-switch.conf file with the one from a working installation of 16.04.1. HAH!

Not only was there a difference, it gave me a string I could Google. That brought me to this: http://askubuntu.com/questions/837982/how-to-configure-local-dns-lookup-in-ubuntu-16-10

It also brought me to this: https://github.com/systemd/systemd/issues/4157, which (if I read it correctly) says that I should go ahead and use the string from 16.04.1 that works instead of the one from 16.10 that doesn’t.

System Admin Notes

Windows 10 is, in some ways, worse(!) than 8.1 was. (Really!)

Since I don’t have time to keep sinking into Windows, here’s what I’ve been doing:

Once you’ve done all that, remember to install an ssh server:

sudo apt-get install openssh-server

Also, install rbenv (and ruby-build) and a ruby or two. Plus your favorite gems.