Blog

  • Keeping Multiple Development Machines Synchronized with Homeboy

    Most of us developer types have at least two machines we use routinely, and managing that can be a chore. Specifically, I usually want to do the following every time I sit down at a machine to hack on something:

    • Keep config files like .bash_profile and .gitconfig  synchronized. (This requires scripting since Dropbox ignores hidden files.)
    • Patch all OS-level libraries using the native package management system on OSX, Ubuntu and CentOS.
    • Update my database daemons and other system services.
    • Upgrade development libraries.
    • Merge in ‘git pull origin master’ project source code for all my clones.

    Doing all this every time I hop to a different machines was chore, so I wrote a few BASH scripts to help. More importantly, I recently packaged them into a public GitHub and released it! Homeboy is a set of small, plain BASH scripts. After following the simply installation instructions, you just run homeboy every morning:

    $ homeboy

    This will run the updates you specify, though I’ve only included the stuff I use regularly: namely brew (OSX), rvm (OSX and Linux), apt (Ubuntu), yum (Red Hat/CentOS) etc. I ask that you submit pull requests to add support for updating Perl, Python, MacPorts etc. The synchronization mechanism works by zipping the specified list of files into a .zip in a synchronized directory managed by Dropbox, SugarSync etc. “Pushing” your current set of files to Dropbox is done via:

    $ homeboy-push

    After pushing, the next time `homeboy’ is run on any configured machine, the .zip file will be unzipped into your home directory. It’s really not complicated, but saves time by having to make the same change a bunch of times across different machines and platforms, and having subtle differences.

    When using the git options, homeboy assumes you have a single directory where all your clones are kept, such as ~/Developer/git. Every subdirectory that looks like a git clone will have ‘git pull origin master’ run inside it.

    Pretty silly stuff, right? But hey, all I do is run `homeboy’ every morning I plan on doing development work on a machine, and sip on a cup coffee while everything is brought up to date. 🙂

    Please help test and submit pull requests!

  • Ubuntu 12.10 to 13.04 Server Upgrade Error

    Are you Googl’ing around trying to figure out why you’re getting this error when trying to `do-release-upgrade’ your Ubuntu 12.10 system, even though you’re pretty much up to date?

    root@mia:~# do-release-upgrade
    Checking for a new Ubuntu release
    Traceback (most recent call last):
    File “/usr/bin/do-release-upgrade”, line 145, in <module>
    fetcher.run_options += [“–mode=%s” % options.mode,
    AttributeError: type object ‘DistUpgradeFetcherCore’ has no attribute ‘run_options’
    Error in sys.excepthook:
    Traceback (most recent call last):
    File “/usr/lib/python3/dist-packages/apport_python_hook.py”, line 137, in apport_excepthook
    os.O_WRONLY | os.O_CREAT | os.O_EXCL, 0o640), ‘wb’) as f:
    OSError: [Errno 2] No such file or directory: ‘/var/crash/_usr_bin_do-release-upgrade.0.crash’

    Original exception was:
    Traceback (most recent call last):
    File “/usr/bin/do-release-upgrade”, line 145, in <module>
    fetcher.run_options += [“–mode=%s” % options.mode,
    AttributeError: type object ‘DistUpgradeFetcherCore’ has no attribute ‘run_options’
    root@mia:~#
    deb http://gb.archive.ubuntu.com/ubuntu/ quantal-updates main restricted
    deb-src http://gb.archive.ubuntu.com/ubuntu/ quantal-updates main restricted

    I just burned a couple hours trying to figure out why I only got this error on one specific server. It turns out that it’s a bug in the version of  the ubuntu-release-upgrader-core package, and you must be pulling updates from a “quantal-updates” repository to get the fixed package when you `apt-get upgrade’. To fix this, edit your /etc/apt/sources.list and make sure you have the following two lines:

    deb http://gb.archive.ubuntu.com/ubuntu/ quantal-updates main restricted
    deb-src http://gb.archive.ubuntu.com/ubuntu/ quantal-updates main restricted

    After making sure you have these lines, get yourself updated again and re-try the release upgrade script:

    apt-get update
    apt-get upgrade
    do-release-upgrade

    …and you should be rollin’ towards Raring!

  • Proxy Authentication Against Devise-Based Ruby Applications

    (Quick GitHub link!)

    My team at TGen has a Rails application using a typical devise + cancan installation for authentication and authorization, respectively, and a related Apache HTTPD proxy server we needed to authenticate against active accounts in the webapp before passing traffic to their destination, which is a single host. We tried mod_authn_dbd for a while, but the combination of Devise’s default adapive bcrypt password encryption that is not supported by mod_authn_dbd by default, a complex set of SQL statements to process the authorization directly, and that none of the existing modules seem to be able to do what we want, made it a messy ordeal.

    Instead, we created devise-proxy: a simple proxy server written as Rack Middleware that you can deploy using Passenger within Apache or nginx. You simply provide devise-proxy with a config.yml file defining the hostname and port of the devise webapp to use as an authentication web service, and hostname and port of a single server to use as a forwarding target.

    Clients connecting to the proxy use the email/password for their proxy username/password credentials. All authenticated clients will be forwarded to your provided host/port, but using the path in the client’s original HTTP request. Clients failing to authenticate will receive a 403 and the forwarding host will never be hit.

    gem install devise-proxy # to install

    git clone git://github.com/preston/devise-proxy.git # to clone!

    devise-proxy is released under a BSD license via the GitHub project, here. Good luck!

  • How To Custom Brand The OpenStack “Horizon” Dashboard

    I’m deploying OpenStack “Essex” on Ubuntu Server 12.04, and have the openstack-dashboard package installed to provide the web-based “Horizon” GUI component newly added for the Essex release. Canonical also provides an openstack-dashboard-ubuntu-theme package that brands the Python-based Django GUI. Despite that the last major Canonical-maintained packages based on the OpenStack “Diablo” release in Ubuntu 11.10 did not include an administrative GUI, Horizon — as a standalone component — has been very stable for a mainstream debut. In the future, though, I’d like to see a quick and easy way to change the default branding to use your own logo, colors, and titles using only the GUI’s administrative screens.

    The horizon documents briefly mention branding customization to give you a head start, but you probably want more specific steps. Here’s my custom-branded Horizon dashboard with custom colors, logo, and site title:

    Once you know where to make the appropriate changes, it’s super simple. Step-by-step:

    1. Create a graphical logo with a transparent background. The text “TGen Cloud” in this example is actually rendered via .png files of multiple sizes I created with a graphics program. I used a 200×27 for the logged-in banner graphic, and 365×50 for the login screen graphic.
    2. Set the HTML title (shown at the top of the browser window) by adding the following line to /etc/openstack-dashboard/local_settings.py
      SITE_BRANDING = "Example, Inc. Cloud"
    3. Upload your new graphic files to:
      /usr/share/openstack-dashboard/openstack_dashboard/static/dashboard/img/
    4. Create a new CSS stylesheet — we’ll call ours custom.css — in the directory:
      /usr/share/openstack-dashboard/openstack_dashboard/static/dashboard/css/
    5. Edit your CSS file using the following as a starting point for customization, which simply overrides the Ubuntu customizations made in the ubuntu.css file. Change the colors and image file names as appropriate, though the relative directory paths should be the same.
      /*
      * New theme colors for dashboard that override the defaults:
      *  dark blue: #355796 / rgb(53, 87, 150)
      *  light blue: #BAD3E1 / rgb(186, 211, 225)
      *
      * By Preston Lee <plee@tgen.org>
      */
      h1.brand {
      background: #355796 repeat-x top left;
      border-bottom: 2px solid #BAD3E1;
      }
      
      h1.brand a {
      background: url(../img/my_cloud_logo_small.png) top left no-repeat;
      }
      
      #splash .login {
      background: #355796 url(../img/my_cloud_logo_medium.png) no-repeat center 35px;
      }
      
      #splash .login .modal-header {
      border-top: 1px solid #BAD3E1;
      }
      
      .btn-primary {
      background-image: none !important;
      background-color: #355796 !important;
      border: none !important;
      box-shadow: none;
      }
      
      .btn-primary:hover,
      .btn-primary:active {
      border: none;
      box-shadow: none;
      background-color: #BAD3E1 !important;
      text-decoration: none;
      }
    6. Open the following HTML template in an editor:
      /usr/share/openstack-dashboard/openstack_dashboard/templates/_stylesheets.html
    7. Add a line to include your new stylesheet: (I’ve highlighted the new line in bold.)
      ...
       <link href='{{ STATIC_URL }}bootstrap/css/bootstrap.min.css' media='screen' rel='stylesheet' />
       <link href='{{ STATIC_URL }}dashboard/css/{% choose_css %}' media='screen' rel='stylesheet' />
       <link href='{{ STATIC_URL }}dashboard/css/custom.css' media='screen' rel='stylesheet' />
      ...
    8. Restart apache just for good measure:
       sudo service apache2 restart
    9. Reload the dashboard in your browser and fine tune your CSS appropriate.

    You’re done!

    [ezcc]

  • OpenStack Nova Essex MySQL Database Schema Diagram and SQL

    I’m in the process of setting up an OpenStack Essex installation on a small cluster of Ubuntu 12.04 servers using the Canonical-supplied and supported packages. I innocently and unfortunately ended up with a state corruption issue causing nova-compute to crash on startup, and found it necessary to dive into the schema as a troubleshooting aid. I used MySQL workbench to reverse-engineer the schema into an EER diagram as well as raw SQL. I’m sure others will find this useful as the OpenStack community slowly upgrades to Essex. Hope it helps!

    [Download diagram, SQL, and MySQL workbench file]

  • OpenStack Diablo on Ubuntu 11.10 w/MySQL

    If you’re installing the OpenStack cloud computing platform “Diablo” release on Ubuntu 11.10 server using the official guide and are using MySQL, don’t forget to:

    sudo apt-get install python-mysqldb

    The “glance” package (and probably others) are configured to use SQLite by default and do not automatically install this dependency by default. Unfortunately, the November 2011 version of the manual also fails to mention this and the log file (/var/log/glance/registry.log) prints a variety of “INFO [sqlalchemy.engine.base.Engine…” messages but no warnings or errors. I assume a corresponding comment applies to PostgreSQL as well but have not confirmed.

  • Quick FASTQ File Parsing Via Memory Mapping In C/C++

    I recently had a need to speedily parse through 8GiB+ .fastq text files to calculate a simple statistic of genomic data. My initial “pfastqcount” implementation in Ruby worked fine, but with many files to process took longer than I had hoped in addition to consuming an alarming amount of CPU. I ended up reimplementing the pfastqcount command-line program in C, which takes one or more .fastq files, memory maps them, and creates the statistic. Simply dropping my algorithm down to raw C significantly sped up the process and reduced CPU usage, especially coming from an interpreted language. If any of you bioinformaticians find the need to implement a FASTQ data processing algorithm in C, I encourage you to fork the project and use it as a template. The project is Apache 2.0 licensed for your convenience and publicly available on GitHub.

  • Hacking Your Own Genome

    Many thanks to everyone that participated in my Hacking Your Own Genome session at today’s Desert Code Camp 2011.2 event in Chandler, Arizona! I’m very passionate about the topic area, and hope the session was both entertaining and useful. Here are the presentation materials, source code for the “youandme” application, and a few other links you might find useful. Thanks again for the opportunity and don’t hesitate to reach out and stay connected!

    Presentation:

    Additional Links:

    Have fun!

     

  • Amazon Finally Adds Lending Library

    Late is better than never! (From the press release. Customer details here.)

    With an Amazon Prime membership, Kindle owners can now choose from thousands of books to borrow for free - including over 100 current and former New York Times Bestsellers - as frequently as a book a month, with no due dates
    Books can be borrowed and read on all Kindle E Ink devices and Kindle Fire

    Not exactly the most giving of terms, but it’s a start. Thanks to the Kindle team!

  • 10 Universal Weight Loss Tips For Men

    I’ve spent the majority of my adult life in the “obese” clinical strata, and have only in recent years taken to caring about maintaining a reasonable weight. I am an American male, 5’10”, with Koren-Caucation ancestry and have ranged between 149 and 172 pounds in the past year. My bodies individual natural comfort zone seems to be in the 150-155 range. At my heaviest I capped out in the 200-210 range. In other words, my physical stature and default dietary habits are spectacularly unspectacular for an American, and I consider myself fairly representative of the “average American male”. I lost most of those excess pounds (180+) in a fairly short amount of time. Everyone is unique in many ways, but from my own research and personal experimentation I believe these points to be largely universal for adult men.

    Weight And Health

    Weight loss does not necessarily correlate to health gain. It’s possible to lose weight on a diet of Twinkies, but you would be seriously lacking in dietary components despite being lighter, and most likely put yourself at higher risk of heart disease and diabetes. Assuming that part of your motivation for weight loss comes from a desire for better health and longevity, remember to see the forest through the trees. It’s great to look healthy, and better to be healthy.

    As a general guidelines, stick to eating actual foods. (Edible substances like high fructose corn syrup should not be considered “food”.) If you couldn’t produce the ingredient if you really wanted to, you probably don’t want to eat it. You have tons of zero-calorie sugar replacements–Splenda, Nutrasweet etc.–but these are not magic bullets and generally should be avoided as “crutch” substances. See Michael Pollan’s excellent Food Rules for guidelines.

    10 Tips

    1. Weight yourself daily at a consistent time with no excuses. It’s especially important to continue weighting yourself when you’re struggling to hold yourself accountable and to prevent prolonged lapses of judgement.
    2. Treat weight management as a lifestyle, not a program. Programs are things you do for a short period of time before going back to the status quo. Lifestyle changes are long-term investments made for the benefit of yourself and those you love.
    3. Drink water and tea when you are thirsty. Have other tasty beverages for enjoyment, not to quench thirst. Beer and other alcoholic drinks are unfortunately high in calories, as are many sodas and even fruit juices. Water first.
    4. Shop when you’re full. Plowing through the aisles on an omg-I-have-nothing-to-eat rampage is going to result in a cabinet full of snacks. You body evolved to crave certain foods to compensate for natural rarity. When you’re hungry, reason goes out the door, and satisfying cravings for those foods that are now readily available becomes the easiest fix.
    5. Visit only upper-tier merchants such as Whole Foods and Trader Joe’s when at all possible. In addition to higher quality foods, they do a much better job than conventional grocery stores of not barraging you with excess junk. Fruits and vegetables are also of notably higher quality and tastiness.
    6. Maintain the lifestyle because “nothing tastes as good as fit feels”, not to punish or deprive yourself.
    7. Talk about solutions with others doing the same. Being around others taking action is extremely encouraging and motivating. Keep in mind the exact opposite also applies.
    8. Focus more on diet than exercise. Both are necessary, but you’d be better served with a good diet and only 30 minutes of exercise per week than horrible diet and 4 hours of exercise per week. Many weight loss systems prescribe disciplined physical regiments, but remember that diet matters more.
    9. Weight train for weight loss. Additional muscle mass allows you to burn calories faster, even when you’re not exercising. Cardiovascular exercise is great for your heart and blood pressure, but doesn’t build the calorie burning, protein-consuming muscle like weight training does. Also remember that you cannot control where you lose weight: only where you build muscle. No one is going to see your rock hard abdominal muscles if your mouth can’t trade in the cheese sandwiches.
    10. Know when to break the rules. If you use a formal system such as Paleo or Atkins you may have strict guidelines. At some point, however, most foods are going to be ok in moderation so long as you can control yourself. It’s ok to not be perfect!

     

     

     

     

     

     

     

    Good living and good luck. 🙂