Portfolio projects maintenance

I dug through emails archives, my backup drives, and all my repositories, and decided to organize everything into a common structure.

Project folder structure

For each project, I use following folders:

project/
  backups/  
  code/
  docs and resources/
  screenshots/

Whenever possible, I use date stamps, so that I know what timeframe to align the project to. Each project also has a date in its folder name.

• backups/ contains mostly sql exports, or fragments of data, that I used while working on the project
• code/ is a git repository with project codebase
• docs and resources/ includes designs, wireframes, static assets, text copy, or emails communication with clients. Everything I used to build the project, or that served as materials for improvements and fixes
• screenshots/ provides a peak into the project look and feel. In most cases, this also documents extend of the work, and in some cases, evolution of the project

Code folder preparation

Most of my freelancing and contracting work happened during my studies, before I started working fulltime. At those days, I didn't worry about source control that much. And if I did, it was SVN, the most popular one in those days (I'm talking 2003 - 2011). Also, it was always just me working on code, so no sharing was required.

My backups from those times looked similar to this:

01 backup/
02 backup/
03 backup-live/
04 backup/
...

Yup, very basic. I wanted to move this into git repository, while keeping the dates relevant. I set up a plan consiting of following steps:

• find out date of the latest change in each backup
• make sure there's no unnecessary duplication in whole backups
• commit backups into git repository, one by one

Find latest change date in the backup folder

First, I removed all meta files I was not interested in. */

// rm -r **/.DS_Store

/* Then, I prepared latest.sh script, to scan current directory and echo a file with the latest modified date and time. This took me some time to search and modify a script based on Emerson Farrugia's answer on SO.

#!/bin/bash

find . -type f -print0 |
  xargs -0 stat --format="%y %n" |
  sort -rn | head -1 | 
  cut -f1,2,4 -d" " |
  sed 's/.000000000//'

Some details:

• find . -type f -print0 returns all the files within current directory and its directories, and prints it out. print0 flag is useful for proper filename termination (by 0x0 character), that is than leveraged by xargs. This should prevent any issues with special characters in the file names.
• xargs -0 stat --format="%y %n" runs stat command on each of the arguments (file names) passed in, and for stat, I want to know time of last modification %y and filename %n
• sort -rn, head -1 sorts all the stat information in reverse order (9..0) and by numeric sort - each item starts with last modification date/time, and only then it is followed by name
• cut -f1,2,4 -d" " removes fields I don't care about from each row, using space as a delimiter
• sed 's/.000000000//' removes time precision that I don't need

Checking for duplicate backups

Because I had the backups from different computers and environments, it was not rare that I had duplicate backups, named in different ways:

24-backup/
20071021-backup-desktop/

Once I knew the latest modifications dates, I could easily compare directories that looked suspicious. I didn't automate this, as I thought it's not worth it. Also, some of the duplication could be caught by git itself, when I would prepare commits later.

To get a basic comparison of two folders, I used following command:

diff -qr dir_1 dir_2

Which provided differences in both structure and contents. This was good enough to serve the purpose.

Commit backups

Once I had project's history organized by date, I used vim to quickly create a script for easier git commits. In couple of projects, I had different subprojects over the time, and I decided to put everything together. Eg. for Czechwealth, I created system for automated sending of training materials in 2005, as well as improved their main site to include this system information:

git commit --date="2005-01-17 10:00:00" -m "main site"
git commit --date="2005-01-31 10:00:00" -m "clients section"
git commit --date="2005-02-01 10:00:00" -m "clients section"
git commit --date="2005-02-02 10:00:00" -m "main site"
git commit --date="2005-02-02 10:05:00" -m "main site"

Notice that I didn't care about time that much, and I only used it for situations when I had two backups from the same day.

Now, I just copied each folder into new git repo, added and committed the files using one of the lines from the prepared script. I didn't go over board with automation, as I still wanted to have some control over what gets committed. During this phase, it was nice to see if a commit is worth it, so I ended up discarding some of the older folder backups.

Screenshots

Gathering the screenshots was not that difficult. I already had several versions of my portfolio website, each showing some of the UIs.

What I didn't have were screenshots of CMS and administration interfaces, that I've been creating from scratch since my first project. For those, I had to get the old project up and running, and make screenshots manually. This was challenging mostly due to the age of some of those projects tech stack. I installed Xubuntu in my VirtualBox and spin up good old Apache with whatever stack I needed. Thanks to backups, I've been able to get up almost all projects up and running.

Once I had all the screenshots I wanted, I had to do some adjustments. The most common one was removing of white spaces. I used ImageMagick's mogrify tool for this:

# start at 0,98 and cut 1280x900 rectangle
mogrify -crop 1280x900+0+98 -path dest *.png

# remove 200 pixels from left and right side
mogrify -shave 200x0 -path dest *.png

Both are batch commands, working on each png file in current directory, and saving the results into dest/ directory under the original filename.

Conclusion

It is a nice practice in shell scripting, vim macros, batch processing and in making tech stacks from 2005 to work in current environments. There's been challenges with PHP versions, SQL imports, system libraries, etc.

Some of the practical benefits:

• reduced size of backups
• better navigation through projects history and evolution
• tracking the date of projects development and of my own progression
• all materials and resources well organized

As a sample of reduced size:

Project 1

• before: 151 MB, 12 540 files
• after: 90 MB, 3 911 files

Project 2

• before: 1 232 MB, 37 372 files
• after: 250 MB, 4 085 files