I just implemented a sample pairwise sequence aligner in JavaScript for a course. You can demo it at http://alignment.prestonlee.com, and grab the source code under an Apache 2 license at https://github.com/preston/alignment.
Tag: software
The Translational Genomics Research Institute (TGen) is a non-profit 501(c)(3) organization focused on developing clinically relevant medical diagnostics and smarter treatments related to genomic profiling. I’ve had the pleasure of working with the institute for over a year on a project related to breast cancer, and thought I’d share a widget I wrote keep you up to date on TGen’s latest news items.
The TGen News: Dashboard Widget for Mac OS X v1.1 gives you an always up-to-date set of headlines, pulled straight from the TGen news feed.
If you enjoy this widget, please donate a few dollars to the TGen Foundation and do your part to support the many causes that TGen pursues. Enjoy!
From the TGen website:
At TGen, investigators are pushing the limits of cutting-edge research and technology to discover the genetic cause of disease. Experiments that were impossible and impractical only a few years ago are now conducted every day.
Discovery fuels TGen’s translational research and lies at the heart of our scientific investigations. TGen’s research divisions are designed to foster a wide range of genetic discoveries. These divisions draw heavily upon TGen’s scientific platforms to expedite findings. TGen’s labs are staffed by teams of researchers focused on making genomic discoveries in common diseases and disorders in the areas of oncology, neurogenomics and metabolic disease.
Note: This free product is provided by Preston Lee, and is neither officially endorsed nor supported by TGen.
For all you web application developers, I’d like to briefly note the following stack trace I received from the American Express website while clicking around:
And here is the same trace in plain-text format:
JSPG0036E: Failed to find resource /cardcenter/common/interstitial.jsp
java.io.FileNotFoundException: JSPG0036E: Failed to find resource /cardcenter/common/interstitial.jsp at com.ibm.ws.jsp.webcontainerext.AbstractJSPExtensionProcessor.findWrapper(AbstractJSPExtensionProcessor.java:322) at com.ibm.ws.jsp.webcontainerext.AbstractJSPExtensionProcessor.handleRequest(AbstractJSPExtensionProcessor.java:284) at com.ibm.ws.webcontainer.webapp.WebApp.handleRequest(WebApp.java:3548) at com.ibm.ws.webcontainer.webapp.WebGroup.handleRequest(WebGroup.java:269) at com.ibm.ws.webcontainer.WebContainer.handleRequest(WebContainer.java:818) at com.ibm.ws.wswebcontainer.WebContainer.handleRequest(WebContainer.java:1478) at com.ibm.ws.webcontainer.channel.WCChannelLink.ready(WCChannelLink.java:126) at com.ibm.ws.http.channel.inbound.impl.HttpInboundLink.handleDiscrimination(HttpInboundLink.java:458) at com.ibm.ws.http.channel.inbound.impl.HttpInboundLink.handleNewInformation(HttpInboundLink.java:387) at com.ibm.ws.http.channel.inbound.impl.HttpICLReadCallback.complete(HttpICLReadCallback.java:102) at com.ibm.ws.tcp.channel.impl.AioReadCompletionListener.futureCompleted(AioReadCompletionListener.java:165) at com.ibm.io.async.AbstractAsyncFuture.invokeCallback(AbstractAsyncFuture.java:217) at com.ibm.io.async.AsyncChannelFuture.fireCompletionActions(AsyncChannelFuture.java:161) at com.ibm.io.async.AsyncFuture.completed(AsyncFuture.java:136) at com.ibm.io.async.ResultHandler.complete(ResultHandler.java:196) at com.ibm.io.async.ResultHandler.runEventProcessingLoop(ResultHandler.java:751) at com.ibm.io.async.ResultHandler$2.run(ResultHandler.java:881) at com.ibm.ws.util.ThreadPool$Worker.run(ThreadPool.java:1497)
Most developers might say, “So what? You just clicked on a broken link somewhere. Go back and go about your business.”
And that’s exactly what I did, since I just wanted to pay a credit card bill. But what if “my business” had been to gather forensic information about potential American Express website vulnerabilities? A stack trace may look innocuous, but we can be almost certain that this HTTP request was handled by a Java servlet. It uses Java Server Pages (JSPs) for view templating, and the application appears to be running on IBM WebSphere. Not only this, but we have line numbers for each stack frame, and the CSS appears to be some form of default styling.
Large institutions such as AmEx tend to not upgrade to the latest versions of development libraries until a future interal release cycle where the new version(s) can be internally vetted, tested, QA’d and put through a due diligence process. (Or for older apps, dependencies might not be upgraded at all.) This makes stack traces extremely valuable to an attacker. Why? Because I can obtain all historical versions of technologies such as WebSphere, and map every specific call in the stack trace back to the code until I find the software versions where the line numbers match up. The attacker should at least be able to narrow down the software version numbers to a range, if not the specific release version. After gathering this information, the attacker may choose to fixate on the older packages, and consult databases of vulnerabilities in legacy software until a vulnerability is discovered.
Long story short: turn off the rendering of stack traces in public-facing production applications. They’re convenient for validation of production errors, yes, but you don’t know what they may reveal.
Got a Mac? Like games? Great news! Value’s “Steam” software finally launched for Mac, allowing you to purchase some great gaming titles on your OS X machine. When a game is available on multiple platforms, a purchased license appear to be portable as well. And as a launch promotion, Value is giving away free copies of “Portal” (an awesome title in its own right) through May 24th, 2010. Grab it!
I had a great conversation with remi Taylor of OpenRain Software today on how to handle fuzzy requirements when the customer isn’t available. You’ve been there… you get an issue tracking ticket like “Add Foozle Support” only to find no explanation of what this means other than the issue title. The customer/decision maker decided to add this ticket as your number one priority 2 minutes before getting on a space shuttle to Jupiter and is unavailable for clarification clear through next month. The customer says it’s the top priority, though, so you’ve got to do something.
The problem is an unaddressed gap in requirements between concept and action. In the customers mind, the concept may be clear. They may have a mental concept of what a Foozle is, what it means to the business, and how it will interact with the rest of the system. They may even have a GUI mockup laid out in mental Photoshop securely filed into the right hemisphere of their gray matter. As the developer, however, you don’t know any of this, and cannot act until you’ve figured out what the fooz you’re supposed to do.
The temptation in this situation is to pick up the keyboard and start coding according to your best guess of what the customer should have explicitly specified, since it’s not possible to reach him/her and you have to deliver something before they return. So you add a “foozles” table to the database, create some forms, tie it into the search algorithm and write a robust suite of regression tests. You are now the de facto project expert on Foozle and Foozle-related issues, and are pretty freakin’ proud of your work. The milestones completes, Foozle support hits production and it’s all good.
The customer returns from Jupiter delighted to find their feature request ticket resolved as successfully completed. They check out the work, and gaze wide-eyed upon the screen. Squinty eyes and an emoticon-like frowny face ensue, and all hell breaks loose. Sound familiar?
Yes: there has been a clear and obvious failure in process and communication. You–the developer–probably even muttered this when the issue hit your inbox, but let’s just assume that this particular real-world situation meant that the “proper” process wasn’t followed for legitimate reasons and you certainly weren’t about to twiddle your thumbs for two months waiting to meet with the customer upon return. Fail? Yes. The customers fault for hit-and-run management? Probably. But could you have handled the situation better? Yes! Let’s look at what could have happened to make an unfortunate situation a little sunnier…
When you first started thinking about the issue, you made the same mistake as the customer: you jumped from concept to a mental action plan without communicating the assumptions under which you made it. You formed your own mental concept of what a Foozle is, what it means to the business, how it will interact with the rest of the system, and created your own mental mockup. Unfortunately, all your concepts and conclusions ended up being completely different from the customers (since you couldn’t discuss them), and none of these discrepancies were discovered until delivery.
For our purposes, the interesting part about the situation is not that the lack of clear requirements prevented delivery of the “correct” system (according to the customer), but that you did not state your mental assumptions about Foozles because your instincts told you it was a pointless exercise. The customer was on Jupiter and couldn’t possibly respond on time, so why bother? …right? Quite the contrary!
It is critically important to document assumptions on unclear requirement–especially when the customer is absent–because it places accountability of correct feature development back on the customer. What if your first reaction to the situation was to add a comment to the issue like this?…
I’m going to accept and attempt this issue, but I can’t find any detailed information on what needs to happen and REALLY need some customer time (~1-2 hours) so I know I’m going in the right direction. Not having this information leaves me with a LOT of fuzziness on what the expectation are, so here’s what I’m going to do…
[Explanation of your approach to the problem, explicitly making and stating assumptions as necessary in lieu of clear customer requirements. If you need a definitive answer that you can’t get, explicitly define one as an assumption so the reader knows how you’re getting from high-level concepts to lower-level action.]
If any of this is incorrect, PLEASE ping me IMMEDIATELY. I’m going to start development soon and want to definitively understand the intent of this ticket to avoid wasting time!
Your assumptions could be wrong. Really wrong. They could be so wrong that you’ve wasted 100% of your time because the customer made a typo and meant “Woozle”, not “Foozle”. The key difference is that you’ve been proactive in specifying missing requirements and gave the customer a non-confrontational opportunity to clarify their intention so you can Get It Done Right. Your clear, wrong assumption statements on Foozle concepts should–if the client was fulfilling their obligation–have triggered an immediately phone call from the customer. It didn’t, and that sucks, but given a tricky situation you’ve done pretty much everything you can do to assure the system is correct (short of a high-speed intergalactic chase to flag down the customer), and of that you can be proud.
In the best case, someone familiar with Foolze semantics will chime in to represent the customer until they return. In the worst case, the angry customer call will at least go much better since you can point to your desperate pleas for clarification on specific items, timestamped in a timeline manner before any of the “wrong” work was done.
For HR and legal purposes, most development companies classify Software Engineers into ranks from I to IV (or V). The higher the rank, the higher the responsibilities, expectations, independence and pay grade. To cut it as an interviewer and manager, you’ll need to classify people accurately with a minimum amount of direct personal exposure: a non-ideal but practical requirement of most hiring processes.
While we don’t regularly use titles at OpenRain, we nevertheless have to distinguish senior talent. The core issue is, “How do you objectively identify ‘senior’ engineering qualities?” Today we’ll focus on several key factors always present in quality engineers, independant of language and platform.
Instinct
He/She has developed extraordinary technical relevance filtering to the point of being able to scroll through a never-before-seen 500 line file in a language they don’t know, and tell you..
- how complicated the code is.
- where potential bugs are.
Even with no formal knowledge of code smells or design patterns, a senior developer can sense ugly code and architecture from a mile away even if they don’t yet know exactly why.
Foresight
Long-term implications are always on the mind of the Senior Software Engineer. They’ve been through the end-to-end development process (from requirements gathering to product maintenance and end-of-life) numerous times, know what issues are going to arise and will point out a suitable solution long before the symptoms start to appear. (This quality thus becomes most apparent after delivery when work is bombarded with never-before-seen use cases.) The truly elite developer is often hard to identify because they’re solving the important issues before anyone else notices the problem. (Ben is a primary example of this extraordinary perceptiveness.)
Results Focus
Knowledge without application leads to arrogance without insight. Senior developers are always focused on results which stand the test of time and can easily see through posers who fluff their way through status meetings.
Communication
New developers seldom understand the required differences of communication between different types of stakeholders. Newbies tend to treat all stakeholders as authoritative figures, and are quick to lose direction when exposed to people with differing incentives. The criticality of non-verbal developer communication is also apparent to the senior engineer. For example, a green engineer may see issue tracking as micro-management, automated testing as an ideological obsession, and project planning as administrative overhead, but these are all monumentally important aspects required to keep all developers and stakeholders in a real-time communications loop, since many do not directly interact. A senior engineer will see these concepts as empowering and often get grouchy when not present, because not having clear priorities and documentation introduces roadblocks to results.
Time & Priority Management
A senior engineer can more-or-less tell you what their schedule looks like a week out, even if it’s not written down, and won’t be hesitant to express any issues with workload ahead of time.
Estimation
New software engineers seem to invariably produce time estimates magnitudes off from reasonable numbers. The issue largely appears to be one of Foresight since accurate estimates are oft best produced by benchmarking current requirements against past similar project experiences: a task more easily accomplished with experience. This issue is an arguing point against the “Customer is Always Available” aspect of eXtreme Programming since a green developer is generally more likely to over-commit to a workload than a senior.
Categories
Sufficiency In Software Testing
When I was finishing up my B.S. I took a class in embedded software testing. The big assignment was to write the software that controls a single elevator, test the software to our satisfaction and deliver the whole shebang at the end of the semester. The critical lesson I learned from the course was not that the elevator software was difficult to write, but that there are an infinite number of odd and unfortunate events that could happen to any component involved, at any time, and there is no way to declare with 100% confidence that you have accounted for all possible defects.
So most software is not about perfection, but sufficiency. Everyones wants ultra-high quality, defect free wares, but at some point you must put down the keyboard and declare the product “sufficient” for release. Key problems: “How do you know when you’ve done enough testing?” And just as important, “When is the right time to test?”
This topic has been a open talking point at OpenRain. Marc is a strong proponent of many TDD/BDD principles and goes knife-throwing-freak-show when stuff isn’t well covered. (Ed. note: possible slight exaggeration… maybe.) I am also highly concerned with sufficient tests, but prefer a incremental approach and am wary to invest too much effort in automated tests up front for several key reasons.
- While development is underway, you incur unnecessary overhead to maintain tests developed before design stabilization. This overhead is inevitable during long-term maintenance, but the last thing I want to do on the project I started yesterday is refactor all my tests because I dropped a single column from the “users” table.
- When inexperienced developers write tests too early, they oft end up testing the dummy data and underlying framework, not your design. It is not our job as application-level developers to write test cases for all underlying dependencies, but since that’s all you have at the beginning of a project, it’s easy to waste time here.
- The benefits of writing tests first to flush out design details is diminished in dynamic languages. In Java, writing a quick block of pseudo-code to use your interface is a great way to explore your design from an “external” perspective. Once you’ve achieved design clarity, you can easily use your compiler errors to create correct interfaces. Dynamic languages such as Ruby, however, do not offer this compile-time help, lowering the benefit of the technique.
- There’s no freaking way we’re checking in code that doesn’t compile. Sorry, but if I’m writing a Java unit test, there’s no way I’m putting up with 800 compiler errors (and no autocomplete) over the next day while I generate all my stubs. I don’t care if TDD says otherwise; it’s a stupid practice for statically typed languages.
Granted, if any of our systems crash, we probably aren’t going to irreparably harm anything except for my phone that goes flying across the room for ringing at 5AM, but we still have the issue of “sufficiency”. For OpenRain‘s Rails-based applications, I’ve been using the following philosophies on a personal level.
- Models tests should be implemented first and as soon as possible. Validation logic and other constraints should be verified up front, as key bugs here will likely effect other code. Add sample data as necessary.
- Only functional/integration tests for core use cases should be done early. Adding too many upfront tests to the yet-to-stabilize design tends to add maintenance liability before it’s able to pay itself off.
- Tests for non-core features should be tested shortly after a brief “breathing” period, wherein others can comment on the design/code before you’re fully committed to it. Don’t waste your time with a massive test suite until people stop telling you it sucks.
- Avoid complex methods of testing. Multi-threaded and singleton-based designs have inherent testing complexities, and should be designed out if possible.
- Aim for 100% coverage in dynamic languages. Otherwise you won’t catch retarded bugs like syntax errors.
- Have all known, likely and anticipated issues resulting in a significantly negative state covered by an automated case. This is, perhaps, the crux of my “sufficiency” perspective. You must have some mental benchmark that determines when you are “done”. This does not imply that all issues are resolved, only that they are tracked and, hopefully, all the significant ones are fixed.
I’d love to hear your thoughts on practical testing philosophy. Please let me know what you think!
Categories
Software Engineering Curse Words
Here lie terms frequently used in software development which I don’t particularly care for.
Programmer
Commercial software is as much about programming as building bridges is about installing steel I-beams. Writing actual code is only part of the engineering effort. When I see a job posting entitled “Java Programmer” I usually suspect that this is either (1) a low-level monkey position and/or (2) the person behind the post doesn’t really understand the scope of typical developer work.
Developers are required–much unlike the mechanical nature of an assembly line worker–to make decisions and assumptions about the external purpose and internal nature of their work, often part of a seemingly ingrokable ecosystem. Unless you have a micro managing boss or a heavy-weight process itemizing every last byte of work, you must personally exercise critical thinking, time management and interpersonal skills to balance your never ending stream of unclear and incompletely stated priorities. Being a successful programmer thus requires much more than programming knowledge.
The Point: The term “programmer” in an inaccurate trivialization of the real job. I prefer “Software Engineer” formally and “developer” in colloquial usage.
Senior
For HR purposes, “Senior” is a nice way of labeling someone as having a bit more knowledge, responsibility, general weight, and more income than a non-senior person. The problem is that both senior and non-senior developers tend to have very similar job duties; so aside from income, the criteria of senior personnel are inherently qualitative, subjective, relative to a particular domain (read: not necessarily guaranteed to transfer being projects), and/or effectively indistinguishable from non-senior status.
The effect is that, in a matrix organization, a new project may end up with n00bs who are senior, experts who are junior, and a pay structure which reflects an old project now completely irrelevant to the current situation. Senior and non-senior developers often work together as peers, and everybody quickly figures out who the real leaders are. And that’s frequently very different from the formal structure and correlating pay grade.
The Point: “Senior” tells me that you’re expecting to make more and are probably good at something, which may or may not be relevant to me. It’s not a global implication of elevated wisdom.
Architect
Most “software architects” I’ve met do far more operational and project management than architectural design work. This isn’t to say that they don’t or aren’t capable of making significant design contributions to the project, but that all the overhead of email and meetings between business/team/customer/whomever members sucks up so much time that lower level engineers have to either make the design decisions for the architect or block indefinitely as the architect plays Inbox-fu.
The Point: If you’re an “architect” who doesn’t have time to sit down with the engineers and talk about design, you’re really a technical manager who needs to officially delegate the design work to avoid becoming a bottleneck for the team.
Resource
I shudder whenever I hear or use this word, usually in a managerial, Mythical Man Monthian context trying to quantize everyone into tiny cube shaped units. I find it so important to account for individual character when planning and estimating that I consciously use the word “people” instead of “resources”; it’s a simple trick to force yourself into remembering the undeniable human individuality of the worker bee.
The Point: People aren’t Legos, so let’s not pretend they are.
Categories
Singletons Cause Cancer
It’s been said before. I’ll say it again. The singleton pattern sucks. From a pragmatic point of view, it has two primary drawbacks: reuse and testability.
Reuse
A public static getInstance() method is, by definition, statically bound at compile time. Since you can’t override static methods, reusing singleton code via inheritance means you’ll need to create a new getInstance2() method. No matter how creative you get with this method name, you have to accept that users of your code will periodically call the parent types public getInstance() method instead of your spiffy new getInstance2(). Working off an interface largely becomes a moot point since the developer must know the exact type of singleton they want to use at compile time in order to invoke the correct getInstance() method.
How do you configure a singleton without a parameter to getInstance(), which would not be consistent with the intentions of the pattern? Since the instance is constructed internally using a non-publically-accessible constructor, there isn’t a convenient way of introducing configuration information before it’s created.. unless the singleton is aware of a configuration source at compile time with yet more static binding. This makes the code very inflexible, as developers intending to reuse it will be at the mercy of your pre-chosen configuration mechanism, which may not be appropriate for their circumstances, or even unit testing.
Testability
Unit tests generally require control over the lifecycle of the class under test to fully validate proper state transition and contractual validity. Since you, the master of the known universe, are writing the software, you’ll certainly write negative scenarios into your unit tests to assert proper failure handling. If intentionally introducing a negative test results in an irrecoverable state, how do you throw out the singleton and start the next case with a new one? You can’t. What if your test case is creating a tricky concurrency scenario emulating multiple systems within the sandbox of a single JVM? You can’t (trivially). What happens when you discover you need multiple instances of the singleton within your application? You can’t. Time to refactor.
Additionally, unit testing of code using static singleton dependencies has a high potential of awkwardness due to an inability to swap out implementations for mock objects. Under the principle of designing for testability, quality and maintainability, hackishness is not a quality to aspire to.
Conclusion
Singletons can be hazardous to your health, seriously jeopardize your family’s safety, and have been classified as ‘terrorist patterns’ by the U.S. government. The fact that an application only needs one instance of something does not mean the object should be designed that way, and there aren’t very many scenarios where singletons are appropriate. Do as the Jedi do and use them with consideration and responsibly.