Saturday, June 21, 2008

the perils of dirty, hetereogenous data sources

The company I work for is building a gi-normous database of everything you ever wanted to know about any product, company, or ingredient you've ever heard of. As it turns out, getting lots of data isn't hard, but combining it all together is pretty hard, and making sense of it is very hard. There's a lot of data in the world, no perfect universal product identifier (UPC isn't specific enough for our purposes, and most data sets we get don't include it), and the data is very dirty. In the process of building smarter data ingestion systems and massaging the data, we often come across some hilarious bits of it.

Some things that have turned up on ingredient lists (aside from the usual cryptic 12 syllable chemicals):
  • dust mite
  • panther fur (in a bath soap, of course)
  • air
  • tobacco dust
  • bone oil
  • thistle
  • DNA
  • "explosives" (no further elaboration)
  • care (but, surprisingly, not love)
There are also some hilarious mispellings, often of "deionized." My favorite was "demonized water." You gotta watch out for that stuff.

And there are also the hazard lists. Hilarity ensues when bored government workers have to make official lists. Workplace hazards include "shift work that involves circadian disruption" and "adding machines."

Tuesday, May 13, 2008

There is a cereal and milk bistro in Berkeley!

There is a new 'Cereal and Milk Bistro' near Telegraph and Channing in Berkeley, and while I was skeptical about the concept after just seeing the Coming Soon signs, after a taste test I must say I am impressed.

The bistro carries a full range of cereals, from Fruit Loops to Muesli, and in your bowl you can get two scoops of cereals (preferably mixing healthy and junky). Next comes toppings- these also range elaborately from healthy to junky, from fruit to chocolate chips. They had more toppings than your average frozen yogurt joint.


And finally, you get your choice of milks- skim, 2%, whole (no chocolate yet, but I hear it's often requested). Your bowl thoughtfully comes with a straw, so you can sip up all the remaining milk when you finish your cereal.

Overall, it's a fun idea, though definitely one that will be dependent on business from all the nearby college carb-o-tarians.

Recycling

All I can say is, my pepper plants are happy. Right to left we have some habanero peppers, a tomatilla, and a poblano pepper. Putting the California sunshine to work.

Wednesday, April 23, 2008

Graffiti in Berkeley














I've noticed some interesting graffiti on the Berkeley streets lately, mostly spray-stenciled, and generally on the uplifting side.
















Here's an anonymous love note. Much sweeter than craigslist 'i saw u.'




















Plant a tree. I also saw "The trees are our brothers" carved into wet cement. A friend of mine claims he always looks up at the trees when he sees police officers (or "peace officers" as they are officially called in the DMV handbook) because of those persistent tree sit protesters.

Thursday, April 17, 2008

April is album showing off month

February was album writing month, March was being a workaholic at the new job month, and now April is album showing off month! The RPM challenge posted all the albums they received and made an online streaming jukebox out of them! Check out our album here, and all 793 albums here. Hooray for random yahoos making music in their garages!

Thursday, April 10, 2008

Big-O for business

Most software engineers have found themselves in situations where they are asked by non-software people why some fundamental engineering principle is true. First you bug your eyes out and balk, then resist the urge to say "It's obvious!" and storm away, and then ponder that if it really is indeed so obvious, shouldn't the explanation also be obvious? I found myself in two such situations this week, and in both cases was able to use Big-O algorithm analysis (otherwise known as basic complexity theory) to explain my point. Which was also an excellent excuse to draw graphs on the whiteboard.

1. Why write unit tests?
Who likes to manually test software by hand? Not me. It's boring, repetitive, and time-consuming. There's a time and place for repetitive, time-consuming tasks and that's called knitting. So, how do you minimize your manual testing burden? By writing unit and functional tests. Why is it such a time saver? If you didn't write tests, then every time you changed a single piece of logic in your system, to be sure this change didn't cause any unintended side-effects, you'd have to go through and test the whole system. And of course most people don't have time to do that, so they only test everything before each release. So, if you make n changes or improvements to your system, and you manually test everything before every release, this is order-n work to make all your changes, then another order-n work before each release to test the system. In total you have to do n*num_releases testing work. So the amount of testing you need to do scales, at minimum, linearly with the number of changes you make to your system. If you were very virtuous (which you never have time for) and tested the whole system after every change you made, then the cost of each change would be n-squared work instead of n.

If you write a test for each piece of functionality you create (preferably before you create it), you only need to simply run all your automated tests after you make a change to see if you broke something. So let's say the cost of running the existing tests is negligible, and writing a test is about as much work as writing the functionality (though usually it's much easier). So if you make n changes to your system, for each change you write a test. Let's say this is 2*n total work for n changes made. You need to do no additional manual testing when you make a new change, you just run your existing tests. And before each release, you can just run all your existing tests. So the total amount of work you need to do is a constant factor of the amount of functionality you build: 2*n, and the amount of manual testing before each release is some constant. This scales way better than doing order-n manual testing work before each release (or not doing enough testing and having buggy releases). It's a slightly bigger initial investment, but pays off with less future maintenance work.

2. Why worry about limiting dependencies between people or teams? (aka, why you should leave the software engineers alone so they can just write code)

Let's say you have n entities in your corporate system. In a small company these entities may be individual people, and in a large company these entities may be groups or teams. Entities can have dependencies on each other: like engineers getting approval from management before releasing a new feature, or a software group needing a framework group to fix a bug in a framework before they can fix their performance problems. Let's think about these entities as nodes in a graph, and dependencies as edges between nodes.

A node with many edges is a process bottleneck. An example is a person who must personally oversee all changes before they are released on a website, or a framework that all projects depend on, or a scarce resource (like an Oracle DBA). If you don't limit dependencies in this graph, worst case you have a fully connected graph, which has order n-squared edges. Dependencies translate into added work, and lost time waiting on a blocking task or person. So let's say the number of edges in the graph correspond to the amount of work that needs to get done. If you don't explicitly limit dependencies between entities, you can easily get into situations where there are order-n entities (workers) and order-n-squared edges (work to be done). Not only is this more work than your workers can possibly accomplish even if they work twice as hard (n squared is still > 2n), at every time step the gap between work created and work finished grows bigger. This is bad- it means the farther you get into a project the more you get behind schedule. This can be avoided by explicitly limiting dependencies to a constant number of edges per node. This will ensure that you have order-n edges in your graph of n nodes, and your organization isn't creating more work for itself than it can handle. Ways to do this? Let people report to only one manager instead of several, don't page the whole company over a network issue- just page the networking team, let the software engineers just write code instead of additionally acting as tech support and project managers and deployment engineers all at the same time.

Saturday, March 22, 2008

evaluating the new hotness, Ruby On Rails

I've been getting up to speed in Ruby on Rails over the last 3 weeks, and I'm definitely impressed by the developer productivity you get with this framework. You can literally get a working data-driven website up in under 20 minutes. You can also slap a new front-end or look and feel onto an existing website in about 20 minutes. I like Rails' "convention over configuration" philosphy, having spent countless annoying hours with other web application frameworks and ORM layers tracking complicated bugs down to incorrect configuration. Ruby is easy to learn, and yields very clean, readable code. I'm not sure yet how excited I am about dynamic typing, but you can do some really clever polymorphic tricks. Overall, I like that RoR makes a handful of good software development practices the path of least resistance: test driven development, MVC component separation, etc.

If you're interested in learning RoR, I highly recommend Agile Web Development with Rails.

The biggest criticism I've heard about RoR is that it's slow. It is a bit slow, but it's also easy to make your code faster. We were recently optimizing a backfill process, and found that we could get a 10x speedup by replacing ActiveRecord.find with ActiveRecord.find_by_sql, forcing rails to not sanitize sql strings, and preloading queries that were called often but always yielded the same result. We used ruby-prof to figure this out.

My biggest criticism so far is the lack of a great IDE with RoR support. Lots of RoR people love TextMate, but it doesn't come with a debugger. I can only imagine living in a world without debuggers if I only ever worked on code that I wrote. You don't usually get this luxury in enterprise software. Ruby-debug works great, but by itself is a command line app that spits context to a console window, which is a bit awkward; and every time you start it up you have to setup your breakpoints again, which is time-consuming. I've heard you can hook it into emacs, but haven't tried that yet.

Eclipse has the Aptana plugin, but after using it for a few days I found it quite clunky. Auto-complete is really slow and error prone, search is awkward and not inline, navigation is awkward and really slow (this is more Eclipse's fault than Aptana's though), and most importantly the debugger is so slow that it takes 4 minutes to start itself up, and half the time fails to start up at all. This is unusable.

Next I tried Netbeans, and this is my favorite so far- it's fast and easy to navigate within, the search is inline, autocomplete is fast and actually works, the default shortcuts are intuitive, and the debugger actually works most of the time. Specifically, it works great for debugging the application and the unit tests, but gets confused and hangs on functional tests. I'll be curious to see if this is improved in Netbeans 6.1.