Web Respects

lighttpd unixsocket is too long in error

2009-07-23T08:33:00.000-07:00

I recently found this error occurring in my lighttpd error log.


2009-07-23 08:23:54: (mod_fastcgi.c.1290) unixsocket is too long in: fastcgi.server = ( .fcgi  => ( 127.0.0.1  ( ...

Google did not prove useful, but its was pretty easy to figure out. The string being assigned to socket in the FCGI config is too large. The string needs to be shortened.

switching from programming to responding to emails

2009-06-04T12:37:00.000-07:00

I have the hardest time swithcing from programming to responding to 'quick' emails. My mind moves faster than my hands. I wish there was a syntax check for English, where are they on that? I saw Spelly on Google Wave, that's what I need.

experience with learning Google App Engine and Python

2009-02-09T11:19:00.001-08:00

In my endless aim to learn about the web technologies of the web, I finally came across to using Google App Engine. My motivation to start using it was its promise of development environment where I didn't have to worry about the details of deploying my application.

App Engine provides a integrated environment that allows you to build an application on your local dev box as it would work exactly on the live site. Some would find this constricting, but I just looked at as new platform where I didn't have to worry about the database setup, massive configurations files, and finding hosting for final project.

I've been working on a side project for awhile, which has gone through many iterations of development (in short I was over thinking it). One of the iterations started with me developing on Google App Engine where I got to chance to build using Python. Google uses Python internally for many things (apparently), but they also choose it because it allowed to easily create a sandbox environment.

Learning Python from a Perl and Ruby background has shown me the true meaning of there is more than one way to do it. In dynamic languages, you usually have control of how the program flows, Python follows this same convention, but doesn't provide so many avenues to do so. For example, in Ruby there are about 5 different types of eval, as far as I can find Python has one. These conventions (not restrictions) of Python make you think of your program in a different order of logical steps. Its been kind of PITA because somethings that come intuitive in Ruby just didn't come to me while working in Python.

The sandbox environment of Google App Engine is built the same on the desktop and the deployment environments. It follows the brand of WYSIWYG from a programming stance. The limitations to the sandbox is pretty much that the Python cannot run C-based libraries and access the OS for more advance I/O features.

Because only pure Python applications can run in the sandbox I had to choose a web framework that would work within it. Google App Engine comes with its own lite web framework, but it was just bits and pieces of Django patched together. Web frameworks are plentiful in the Python realm, and webpy seems to be the simplest and easiest to pick up, which instantly attracted me to it.

After working 2-3 weeks learning Python and building my application, I came to the conclusion that ran into too many limitations of my knowledge of Python. I would always wanted to do something that would always bring back to "I can do that in Rails easily".

Mid-entry UPDATE: It appears that Google is planning on supporting Java on Google App Engine. There are even early reports coming in of early Rails development using JRuby on Rails. This is some exciting stuff, which means I will certainly be revisiting App Engine in the near future.

Sinatra speaks a different language, Perl

2008-11-16T10:54:00.000-08:00

So, if it has become obvious to some, I have a thing for web frameworks. I like to understand how and why they work, and for me that usually involves more than just reading the source code. My first attempt was different variations of Puddy -- Rails/Merb like framework in Perl. The second iteration has led me down the path of the Ruby Sinatra framework.

Sinatra is a minimalist framework for creating web applications. Its scope really only extends into realm of controllers. It does support views, but it is still far from the enormous support that Rails would provide in options. The controller aspect allows you to define the route in method call of the action -- often causing your entire application to be just one file. I have used it to experiment with building APIs for some of my projects.

In short, its a light, quick, and not at all a memory hog, so of course I wanted to dissect it. This lead me to just reading the source code, but I wanted to do more to understand how it worked. I should inform you that I do indeed know Ruby. I've used it on a daily basis for the past 2 years for my job(s) and have used well beyond just the Rails environment. Now, I have also been a big supporter of Perl, mainly because it was my first language beyond good old Q-BASIC.

With that in mind, I would like introduce a project that I have been working on Sinatra for Perl. Yes, it is a work in progress, but I have put a lot of effort into making sure that this code base was some what solid before I released it to the world. In the repository, a working example can be found and run on your local machine of the feature set.

I like the ease of of extending Sinatra in either Ruby and now Perl. I have added simple extensions to support page caching, running tasks from command line, and a simple background job server. Many of these are lacking optimizations, but I just wanted to show the ease of extending the framework.

The simplest example of a Sinatra app:


#!/usr/bin/env perl
require 'lib/sinatra.pl';

use strict;

get('', {}, sub{
 return 'Hello World';
});

get(':name', {}, sub{
 my $r = shift;
 return 'Hello, ' . $r->params->{name};
});

A simple DSL that defines what action to perform on a route. LOVE IT!

This has been an educational experience (albeit a nerdy one) and I hope to continue on this project. I think I will probably have to go through a name change in the near feature as not to confuse people on the Ruby version.

rails plugin to white label CDNs

2008-11-10T13:44:00.000-08:00

This is a continuation of my previous post talking about using S3 as a CDN. This post covers some of the issues that were faced with making Rails work nice with a CDN.

Rails is made to do great things, but as many before have me said, handling concurrent connections is not one of them. Once a request comes into Rails, that process (Mongrel, FastCGI, etc.) is blocked till the request is done. Actions like sending emails, transferring files, large calculations need to be pushed away from the user request into another process. There are many solutions such as BackgrounDRb, Starling, etc, which allow you to load long running tasks from blocking Rails.

The task of handling files on a remote server is always a tricky one. Each CDN has there own interface on how to interact with the files -- delete, update, move, etc. This proved to be a problem when trying to test which one would work cleanly with our setup -- widgets stored in the database, which are updated immediately to all users on the Chumby network.

I took a top down approach to the problem. Designing how I wanted the widgets to move from our servers to a another server by building the API of methods. These methods were just a skeleton and did nothing. It just allowed to me to write the code I expected instead of working around another CDN or modules API. The methods could then be filled with the appropriate code to make it work with the CDN.

Originally, the CDN of choice was S3 -- not a true CDN, but suited our purposed of unloading our servers of loading dynamic content from our servers. The necessary API calls were filled to support the functionality of S3. Its then I realized that with the top down approach that the work I had could be easily modified to work with any CDN method. I've written extensions for rsync, ssh, and just S3, but it has supported our needs to be able to test multiple environments and services quickly.

This plugin data_fu is currently a work in progress, but I hope to modify for wide spread use and finalize it.

NOTE: At the time of writing this software that there are many solutions of CDNs that handle transparent proxy and caching of content. The reason these solutions were not used because the content of widgets need to be updated instantly. Since the solution was to use vanilla CDN mechanism so we could change CDN solutions if one went down and switch over was needed immediately.

project euler repo

2008-09-15T20:02:00.000-07:00

I have recently gotten back into the Project Euler problems. The website provides a bunch of math and logic problems that can be solved any way -- paper and pen or programming. I like the challenge every once and awhile, so I thought I might share some of my results of the code I have written to help solve the problems. There are comments in most of these. They are just quick dirty hacks to help me get the answer. Sometimes the output will not be the answer and you might need to go look for it, but the logic is there.

new job and location

2008-09-13T18:44:00.000-07:00

I have neglected my duties as the maintainer of this blog for a few months. Its has been for the better though. About a month ago, I left my position at Chumby and moved to San Francisco to start working at [context]. I had a great time and experience working at Chumby, but I felt that San Francisco is where I needed to be for both work, but also experience. I grew up in one of the largest (and best) cities in the world, and I missed the lifestyle that came with it. San Diego has great weather, but I hated driving everywhere.

Now I am in San Francisco. Trying to sell my car. And enjoying my new job, people, and experience.

If you are in the area, please contact me, nice to meet some readers. :)

Flash on S3

2008-09-13T18:35:00.000-07:00

This is a continuation of my previous post talking about using S3 as a CDN. This post discusses some of the issues that occurred with hosting Flash content on S3, and the solutions for them.

Problems started to occur once Flash SWFs were loaded from the Chumby device and the Chumby website. SWFs have a built in security policy known as cross domain policies that allow the owner of a domain to specify what domains have access to the domain. Think of it as a robots.txt for Flash SWFs.

With S3 there are two ways to access content from a bucket -- AWS based URL or a CNAME from your domain that points to AWS (Amazon Web Services). When the Flash content is on S3, the Flash player looks for the crossdomain going through the AWS URL path. We setup a CNAME 'swf.chumby.com' and placed a crossdomain.xml that could be accessed via http://swf.chumby.com/crossdomain.xml and also one on the top level http://chumby.com/crossdomain.xml. This allowed to control what SWF movies could load the widgets.

Playing the SWF as a stand alone Flash movie never showed any problems. When it was loaded via http://swf.chumby.com the SWF would claim its domain to be swf.chumby.com instead of an AWS domain. From the chumby website there is a way to preview the content that will appear on your Chumby -- the Virtual Chumby. The SWF for the Virtual Chumby exists on the main chumby.com website. With the crossdomain, it was able to load the widgets from swf.chumby.com no problem, but when it wanted to send parameters to the widget a problem occurred.

Flash apparently has various sandbox models for the SWF files. This is good because allows SWF to maintain a state security and ensures your data is protected. This bit us in the ass though. Since a SWF can grant only certain (sub)domains to ability to send it parameters we had 1000s of widget that we could play, but they didn't have access to any information that made it work well within the Virtual Chumby. There were two possible solutions. Change every widget to have the code allowDomain, which would take weeks to contact 3rd party developers, countless resources, etc. The second solution is even tougher it would require moving the Virtual Chumby SWF over to the swf.chumby.com domain and updating the links to it on our website. :)

s3 as a CDN

2008-09-13T18:33:00.000-07:00

I worked for a company that provides widgets as a primary resource for our product the chumby. These widgets are purely static content in the form of Flash SWF files and an associated jpeg thumbnail. This content is provided from our servers from both dynamic (database) and static (file servers) resources. These resources are ready to scale to certain calculated amount before we have to worry about more servers, bandwidth, etc... We try to stay ahead of the curve with growth.

The scaling numbers show that we can do one of two things -- expand our servers and utilize more bandwidth or use a CDN to provide our content utilizing caching. In short, the most cost effective solution is S3. Our content, widgets that can change instantaneously when someone uploads a new one, needs to be provided to all users with in a reasonable time. A normal CDN could take minutes-hours to propagate and take time for integration. Expanding our servers would mean more time and maintenance on our end.

The architecture we have decided is to have a two tier distribution, which will provide with redundancy for widgets. The widgets will exists on our servers in the database and on S3. Our database server is used to hold the widgets because its easy to backup, restore and replicate. With our current system, when a user uploads/updates a widget, it is saved in the database directly, so the newest version can be pushed to users as soon as it gets approved.

Transferring files to S3 has proven to be quite simple to implement. The main problem has been adjusting our architecture to adapt to external URLs. Frontend (website) facing, obviously changing URLs is pretty trivial and all browsers support cross domain loading of content.

Pushing widgets to the database is easy. A simple create/update with ActiveRecord and you're done. When a user uploads a widget, in the same POST request the file is saved to the database, so there is no delay and problems and errors with the file are reported in real time. A blocking operation for Rails, but with size limits imposed on the database, model, and web server it shouldn't be too slow.

To transfer the widgets to S3 from our database in 'real time' is a tricky part. This is a blocking that depends on factors beyond our control. The S3 servers could be done, our bandwidth pipe could be saturated with web hits so upload to outside server is slow, etc. This is a blocking operation no matter what, but one we don't want the the user to have to wait for when they upload a new widget. The solution was to push the transfer of a widget to S3 to a job server, whose main purpose is to queue long running tasks. The job server was built using BackgroundRB that integrates well with Ruby On Rails.

This post is to be continued in follow up posts. There is still so much more to cover with the problems we had with Flash and the framework built to white label CDNs.

very accurate description

2008-08-28T00:38:00.001-07:00

Its not to often that I read something that describes a certain type of people. This blog posting describes a particular type of programmer personality. Programmers come in many shapes, sizes, and mentalities. I am not saying this is a perfectly accurate description of who I am, but there is one paragraph that I read and was like woah! I am not going to quote the paragraph because I really think this is a blog posting all should read about me. :)

s3 logs with webalizer

2008-05-26T12:08:00.001-07:00

I recently saw the site s3stat.com, which is a simple service that takes your s3 logs and pushes them through webalizer to get the nice graphics and stats. The service is $2/month. I thought to myself that this can surely be done for free.

An hour or so later, I think I have something that's pretty comparable in features to s3stat. I am in no way trying to put them out of business. They maintain their website and are improving the s3stat with more features. I purely wanted and needed a way to view s3 stats, and didn't want to pay for it.

This script requires that a bucket on s3 has logging enabled. Please do so before using this script, or you will just get an error that logging is not turned on. The rubygem AWS::S3 is required to run this script. Please look at the options hash for required parameters.


#!/usr/bin/env ruby
require 'rubygems'
require 'aws/s3'
require 'getoptlong'
require 'tempfile'
require 'date'

#something wrong in the s3 gem
Date::ABBR_MONTHS = Date::Format::ABBR_MONTHS

#default arguments
options = {
  :access_key=>'', #the Amazon access key
  :secret_key=>'', #the Amazon secret key
  :bucket_name=>'', #bucket name to pull logs from
  :folder_name=>'webalizer' #foldername for webalizer output
  #:clear_webalizer_folder => true #delete local webalizer data
}

#establish connection the s3
AWS::S3::Base.establish_connection!(
  :access_key_id     => options[:access_key],
  :secret_access_key => options[:secret_key]
)

#find the bucket specifying the log files
puts "Checking for logging for bucket #{options[:bucket_name]}"
if AWS::S3::Bucket.logging_enabled_for?(options[:bucket_name])
  new_log = File.new('bucket_log.log', 'w+')
  log_status = AWS::S3::Bucket.logging_status_for(options[:bucket_name])
  puts "Processing log files"
  AWS::S3::Bucket.logs(options[:bucket_name]).each do |log|
    #convert the lines of amazon s3 log to CLF (Common Log Format)
    log.lines.each do |line|
      new_log << "#{line.remote_ip} - - [#{line.time.strftime("%d/%B/%Y:%H:%M:%S %z")}] \"#{line.request_uri}\" #{line.http_status || '-'} #{line.bytes_sent || '-'} \"#{line.referrer}\" \"#{line.user_agent}\"\n"
    end
  end
  new_log.close()
  #make sure webalizer folder_name exists
  if options[:clear_webazlier_folder] && File.exists?(options[:folder_name])
    Dir["#{options[:folder_name]}/*"].each{|f| puts f; File.delete(f)}
    Dir.delete(options[:folder_name])
  end
  Dir.mkdir(options[:folder_name])
  #run webalizer on current log file
  webalizer_output = `webalizer -o webalizer/ -D dns.db -N 5 -F clf bucket_log.log`
  puts "output from webalizer:"
  puts webalizer_output
  #update webalizer bucket with newest info
  puts "updating webalizer to s3 bucket #{log_status.target_bucket}"
  Dir["#{options[:folder_name]}/*"].each do |filename|
    puts "uploading file #{filename}"
    AWS::S3::S3Object.store("#{filename}",open(filename),log_status.target_bucket,{:access=>:public_read})
  end
end

If you have any improvements please let me know in the comments.

Rails top 100

2008-05-20T11:25:00.000-07:00

This wiki maintains a list of the top 100 websites (by Alexa ranking) that use Rails. Chumby is ranked number 67 on the list.

NOTE: This is a shameless plug because I work for Chumby doing the RoR development. :)

criteria for a recruiter

2008-04-29T01:17:00.000-07:00

The most dangerous part of finding a job online is having to put your contact information out there. When I first graduated college, I posted my resume on every job website that existed. This is what the (useless) career center told me to do. The next day my inbox was flooded with so many job descriptions from companies and recruiters. To a college graduate this is the most misleading part of a job search. I thought I would have my choice of the perfect job.

Each email was an automated message sent base on the criteria that I had an email address and on certain keywords in my resume. They described competitive salaries, with great benefits, awesome work environment, and a great company. Obviously no human has ever read my resume because they would have made the immediate observation that I had just graduated. So my amazement from the emails turned to disappointment, which now has turned to annoyance.

There have been a few exceptional recruiters that have emailed me. Most recruiters follow the same tactics of trying to lure you in, which are the spamming recruiters. I would like to point out some guidelines for recruiters that would make both our lives easier.

Read my resume
It obvious to me when you haven't. When the job description has nothing to do with my previous work, experience, skills, or interests.

Be Personal
If you want to use me as a possible commission be a little more personal and address me -- not the masses. I tend to ignore the recruiters that send me and email with my name obviously copied and pasted in the Dear field. The biggest give away towards those emails is when at the end of the message ask for updated resume, desired salary, newest contact information, current employer, etc. Asking for updated information just seems really tacky to me -- especially when you have no idea what I am looking for.

Describe the Job
Tell me more than the requirements of the job. I can see they want PHP/Perl/Java/C++ experience, but what does the job entail. Will projects becoming my way willy nilly and I have to be able to organize my time? Will I be working with someone else on the same project? What is their development style (extreme|agile|pair)? A job description should describe what a typical day might be for the employee.

Describe the company
Tell me something positive about the company. Please don't copy and paste their description from the corporate profile. If I was a potential investor that might mean something, but I am potential employee. Tell me they have pick up games of basketball, allow flexible time, provide lunch, allow employees to be open about their ideas, have a turnover rate of 10 years, etc. I want to know what the employee culture is like for the company.

Understand the different titles
I have learned that an engineer, developer, programmer, and architect are totally different things. Then there are different levels from junior-senior and I-V. Understand that I might be an engineer, with experience of building a software system abstractly based off an idea someone had. More often than not I have a job description that read it was for someone who picked up PHP in 24 hours -- with no previous computer experience ever. I am a software engineer. I enjoy the challenge of creating something new and putting the puzzle together.

Two years later, after removing my resume and contact information from said job sites, I still get the same emails. My email and resume are being passed around (like wine coolers at a middle school party) to recruiter contact database everywhere. I have learned to ask to be unsubscribe from these lists.

Don't get to discouraged. There are good recruiters out there. I had one excellent recruiter ask me to fill out a questionnaire asking geeky questions of what I have done, when I started, and what I mess around with on computers plus a few logic problems. Usually that is something that you get in an interview, but thats what the recruiter should be doing interviewing you to help you find the best possible job and candidate for that job. They will either find you or you will find them.

dirty fields in ActiveRecord

2008-03-17T15:10:00.000-07:00

Managing databases from a programming environment is always an ordeal. Rails has an ORM (ActiveRecord) that transparently provides a programming environment that maps directly to the database and generates all SQL. ActiveRecord sacrifices efficiency over ease of use in some cases. I believe one of the most neglected ones is how ActiveRecord handles updating records in a table.

When ActiveRecord updates a record in a table, it updates all fields whether or not those fields have actually been changed. In most cases this is fine, which a record is so small that setting the fields again takes a trivial amount of time. I happen to have one those rare cases where this is not a valid solution. I happen to use my database for file storage with files from 10KB to 1MB, so changing the name of the file causes the large chunk of data to be set again.

Only the field that has changed (a dirty field) should be updated. ActiveRecord has no flags for what fields have changed or not. I took the time to extend ActiveRecord to support dirty flags for fields by modifying write_attribute method to mark a hash that an attribute of a record has changed.

With the knowledge of dirty fields, generating the appropriate UPDATE statement needed to be done. The UPDATE statement is currently generated by converting all fields and their values in the an SQL assignment by the function attribute_with_quotes. I just extended this function to check that an attribute had been dirtied before it was added to the assignment statement. Since an UPDATE is syncing the database with the current model all the dirty fields are flagged as no longer being dirty.

Show code:


module DirtyAttributes
 def self.included(base)
   base.class_eval do
     alias_method_chain :write_attribute, :dirty
     alias_method_chain :attributes_with_quotes, :dirty
     after_update :reset_dirty
   end
 end
 def reset_dirty
   @dirtied_attrs = nil
 end
 def write_attribute_with_dirty(attr_name, value)
   dirtied(attr_name)
   write_attribute_without_dirty(attr_name,value)
 end
 def dirtied(attr_name = nil)
   @dirtied_attrs ||= {}
   #logger.info "dirtied #{attr_name}"
   @dirtied_attrs[attr_name.to_s] = true if attr_name
 end
 def dirty?(attr_name = nil)
   @dirtied_attrs ||= {}
   #logger.info "dirty? #{attr_name}"
   return (attr_name.nil? && !@dirtied_attrs.empty?) || @dirtied_attrs.has_key?(attr_name.to_s)
 end
 def attributes_with_quotes_with_dirty(*args)
   quoted = attributes_with_quotes_without_dirty(*args)
   quoted.delete_if {|key,value| !dirty?(key) } unless (self.new_record?)
   return quoted
 end
end

ActiveRecord::Base.send(:include, DirtyAttributes)

I would just like to emphasize how easy this was to do. No more than 30 minutes. This is why I love Rails and Ruby. Yet, what gets me is why this wasn't done before. Sorry to say, but this is really one of those DUH! things that should have been implemented from start. If there is a reason that there wasn't please let me know because I am really curious.

NOTE: I have not made this a plugin yet. This code has not been tested on a production environment either.

UPDATE #1: Because of the way ActiveRecord handles method chaining with alias_method_chain there are some edge cases that need to be solved. Mainly I need to find a way to override the original define update function ActiveRecord::Base. You think it would be easy, but alias_method_chain renames functions, and since it is used to add support for 'update_at/updated_on' timestamps, it has proven to be difficult. I have found ways to make it work, but the potential for other plugins doing alias_method_chain to update could potentially cause problems. I think I will just submit a patch for ActiveRecord.

UPDATE #2: I have been able to edit the code to work correctly in all 1.2.3/2.0 version of ActiveRecord successfully. I had to learn a little more about the internals, but the final code shows that it is possible and pretty easy to do.

June 5, 2008 - It looks like Rails 2.1 is supporting dirty fields. Hurray, but its weird, I had it first. ;)

Puddy framework ideology

2008-01-10T22:08:00.001-08:00

In case you have no idea what I am talking about, familiarize yourselves here. I have been working on my own little mini framework for awhile. By observing some of the things that I have seen in mainly on Ruby on Rails and other frameworks and incorporating them.

The thing that I have realized about many frameworks is the integration they with so much of the web application. I am not complaining at all. Having so many different options available for you is great, but have always gotten me is the overhead of such framework. Having the kitchen sink scenario is nice, but a minimalist framework would work just as well, which allow you to pick and choose just the features that you need. Rails 2.0 have gone in this direction, and I have seen smaller Ruby frameworks like Camping and Sinatra, which are great examples of minimal work.

Yet, what I have always wanted was access to each individual module of a framework and be able to pick and choose which get used. I believe that each module should not wholly depend one another. This can always be the case, but the dependency should be minimal.

I have tried to keep this mentality with Puddy. If anyone has actually kept up with it, the modules are each individually coded, so one doesn't require functionality from another. I guess I am doing an MVC for a MVC framework.

Anyways, a quick updated on the framework. I am making progress on it. It has been a learning experience. There are some parts that I have completely written myself and there are some that I have copied from Rails verbatim, but only because they did such a good job with it. I hope to be able to create an example app real soon to actually demonstrate the framework. I really just want to show something convenient, lite, and fast.

Update: I forget to mention Merb, which seems to be a ncie framework and follows this ideology too. Just when I think I am being innovative -- someone else has already done it. ;)

deep eager loading with ActiveRecord

2007-12-14T15:29:00.000-08:00

I was going through Rails today cleaning up some code and found and interesting thing with ActiveRecord. I have always known it was capable of eager loading of associations. For example, a blog post can have many comments.

Example:


@posts = Post.find(:all,:include=>[:comments])

Well a comment is associated is written by a user and say you want to eager load the user information too. Well it appears that ActiveRecord supports eager loading of associations with associations. Meaning, I can load the user information for each comment associated with a post.

Example:


@posts = Post.find(:all,:include=>[:comments=>[:user])

This is amazing. I had not known about this feature, but decided to give it a try. This saves in time with alot optimizations that I tried with eager loading and caching of associations' associations.

Update: It has been brought to my attention that this actually documented. This either proves I need to read the entire document or it needs to be better organized. :)

bringing back reality into social networking

2007-12-09T21:50:00.000-08:00

I found recently that social networking is no longer limited to the virtual world any more. In an amazing turn of events, people are meeting new people and making professional relationships by meeting face to face. This is all in respect to technology based fields. The days of Facebook, Myspace, and LinkedIn are numbered.

A (real life) social group I found called Jelly is nothing innovated, but something I think I have been missing in my work life. Its a simple concept. Biweekly someone opens up their place to strangers, offers free wifi, and works from home. Its great because people stop by to show what they are working on, from the trivial to the mediocre, but the most important thing is that you meet like minded people. How many people you know wants to leave their place open to complete strangers?

I have slowly learned since graduating college that the working world lacks the inspiration of personal creativity. I have heard the stories of Google that allow you to work on your own projects. I currently work at a start up that allows me work on my own schedule, so I can make time at those points in the day to work on my 'side projects'. Unlike my time in college though, I don't find the environment of people constantly coming up with new ideas and talk to people to throw ideas around. Am I missing something?

The idea behind Jelly is simple and I know its nothing new. It seems to be the simplest thing I've seen in real social networking. There is no one inviting heavy weight VCs, or the rockstar programmer, or the requirement of some sort of meal. One person opens up there place and invited (future) friends over, but with the intention of working on your own stuff. I'm going to have to give this a try. If anyone is interested in the San Diego area.

response to 'Hiring a buzzword Developer ... '

2007-11-30T09:08:00.000-08:00

There are a lot of blog postings out there with the subject of to hire the perfect developer. The most recent article the sparked my interest was '11 Tips on Hiring a Rails Developer'. That describes a great or *shudder* rockstar Rails developer as a hard to come by. There are many articles like this in the Rails world. Describing how to get the Rails developer that goes to the RubyConf, goes to the local Ruby meetup, and dreams in Ruby.

In short, these articles often leave out the most vital argument of most programmers, engineers, scientists, etc. that have the willingness to learn and expand their expertise. At my current job, before I became the Rails developer/Software Engineer, my experience in Rails was next to nil. Mainly it was me tinkering around with the famous screencast showing how to make a blog. I was hired on as QA to test the currently developing website, but sometimes I would go in and tinker with the code, to make fixes that the other developer didn't have time for. Eventually I became the goto guy for the Rails development and I switched roles within the company.

My point is that there are people out there who are passionate about Python, but can pick up Ruby. There is a Java developer out there learning the potential of languages with Groovy. If you look to hard for the person whose main experience has been everything in one concentrated field, what happens when the next buzzworthy thing comes out? Will they be able to learn it, or will they stick to what they know because it can do everything they need it to do?

My belief if the true rockstar is able to adapt to anything.

*PS - In response to the article that sparked this annoyance, You can hire people with a degree too. I have a Masters in Computer Engineering with concentration on data mining.

capistrano deployment error 'Errno::EACCES: Permission denied - /.ssh'

2007-10-26T22:15:00.000-07:00

If you are using a Mac with Tiger and are having an error with capistrano deployment such as:

[deploy:update_code] exception while rolling back: Capistrano::ConnectionError, connection failed for: localhost (Errno::EACCES: Permission denied - /.ssh)

Set your HOME environment variable. i.e. export HOME=/Users/monkeyboy

An hour later, I realized this. Hopefully someone else can find it here.

issues with Flash content within lightbox

2007-10-22T10:32:00.000-07:00

The lightbox concept in webpages started around 2 years ago (as far as I can remember). It allows certain content to be focused in on a webpage, so that nothing else can be distraction. Its been used for photo galleries, ajax content, [new form] popup advertisement, etc. In my case, it needed to show Flash content for configurations of content providers (widgets).

Example screenshot (will be provided once I can get the image to upload)

The problem that was occurring was that the Redbox plugin (for Rails) could not provide the correct support for the Flash content. Redbox is still a nice plugin.

The following is a list of issues that were occuring, so that others may learn from my testing with Flash content in lightboxes:

Redbox uses CSS transparency to creat the grayed out portion of the lightbox -- known as the background overlay. There were problems that occurred with it across the browsers with flash content. Mainly if the user tried to scroll the browser the Flash movie would disappear. The transparency problems were resolved by creating a alpha-PNG to be used for all browsers except IE6 that uses the CSS filter property.
Some control was not provided by Redbox to handle the width and height of the lightbox. This was needed because the Redbox plugin sets the div that displays the lightbox to auto height and width, which is great for content such as images and text. Unfortunately, the width and height of the Flash movie do not get considered into this calculation (a DOM problem I assume) of the width and height of the div containing the lightbox. This would cause the lightbox display to be much smaller than it was supposed to be. The width and height need to be set manually.
On the page where the lightbox was being displayed there are scroll bars that are part of a large select box. In some browsers, *cough* IE, like to overlay the scroll bars over all content, completely ignoring z-index. They need to be hidden while the lightbox is being displayed. It also needs to be taken into consideration that the scroll boxes within the lightbox are not removed.
In Safari (and maybe some other browsers), content loaded from an Ajax request into lightbox needs to be removed from the lightbox even when the lightbox 'innerHTML' is set to empty string. This problem would occur if the same lightbox with Ajax request was performed again the content from the previous request would be there again. This is solved by removing the child HTML elements from the lightbox when the lightbox is no longer needed.
Proper positioning of the lightbox to be centered with respect to the current browser windows scrolled at position, not with respect to just the top of the content. This one was pretty trivial to solve as it was a professional design choice for our content. I label it as an issue because it seems that lightbox should work with respect to the user's view and not force the user to scroll to where the lightbox appears in the browser window.
In Mozilla browsers (FF2) on Mac, the Flash movie likes to disappear every once and awhile depending on the condition that there were two flash movies playing in the same window -- one in a lightbox and one in the background. The other condition is that the flash movie in the background had a flashvar 'wmode' set to 'opaque'. Set this 'wmode' to a different value and the problem no longer occurs.

I make no claims to knowing why some of these issues occurred either browser issues, users issues, etc. I just came up with the best solutions I could to solve the problem from not occurring again in the browsers.

With these issues, I took some time and built a Lightbox script that handles these conditions and supports Flash content plus any other content pretty well. I slapped it into a plugin a while back. I tried my best to make it backwards compatible with Redbox as much as possible, so we did not drastically need to change our code, so mostly its updated Javascript to handle the above conditions plus some extensions to the Helpers for width and height.

Note: I would just like to point out that this code is kind of old. Its still in use at my current employer's website. It works well and has proven to be quite stable. At the time of its creation there were no nice Rails plugins with lightbox support except Redbox, which did not work well with the Flash content.

timed out fragment caching in Rails

2007-10-10T22:28:00.000-07:00

As you can probably tell I am a little into optimizations. In Rails, caching is one of those things that is made pretty easy, but can also be difficult when you want to do complex tasks with the caching -- such as expiring it. There are two ways with expire_fragment within the logic of your controllers or using an Observers. The Observers are useful, but require that much little effort to set things up, which personally I don't want to maintain. The expire fragment was my next target, but yet again a struggle because I had to write logic for it in certain places.

Hence the creation of my next caching mechanism. Rails supports many types of caching like file, memory, memcache, etc. The problem I have found is that memcache is the only that supports auto timeouts of the cache data. Tell me why I cannot have the same functionality built into Rails for the other caching mechanisms? I want to cache fragments in local memory of the server, which require quick access, and are small enough where they don't need to be shared like memcache.

For example I want to cache a fragment for 5 minutes on a page:


This is last 5 minutes of posts from our forums:
<%cache_timeout('forum_listing',5.minutes.from_now) do %>
<ul>
   <% for post in @posts %>
   <li><b><%=post.title%></b> - by <%=post.author.username%> at <%=post.created_at></p></li>
   <%end%>
</ul>
<%end%>

This fragment will timeout after the page that contains it is loaded after the 5 minute mark and then will be reloaded. As you will noticed @posts are loaded, so is_cache_expired? method is provided in the controller and view to make sure if the fragment hasn't expired. This allows data to loaded only when needed.

The following is the code written for the timeout caching. Check back soon for plugin location.


module ActionController
  module Cache
    module TimedCache
      #used to store the associated timeout time of cache key
      @@cache_timeout_values = {}
      #handles standard ERB fragments used in RHTML
      def cache_timeout(name={}, expire = 10.minutes.from_now, &block)
        unless perform_caching then block.call; return end
        key = fragment_cache_key(name)
        if is_cache_expired?(key,true)
          expire_timeout_fragment(key)
          @@cache_timeout_values[key] = expire
        end
        cache_erb_fragment(block,name)
      end
      #handles the expiration of timeout fragment
      def expire_timeout_fragment(key)
        @@cache_timeout_value[key] = nil
        expire_fragment(key)
      end
      #checks to see if a cache has fully expired
      def is_cache_expired?(name, is_key = false)
        key = is_key ? fragment_cache_key(name) : name
        return (!@@cache_timeout_values.has_key?(key)) || (@@cache_timeout_values[key] < Time.now)
      end
    end
  end
end


module ActionView
  module Helpers
    module TimedCacheHelper
      def is_cache_expired?(name = nil)
        return false if name.nil?
        key = fragment_cache_key(name)
        return @controller.send('is_cache_expired?', key)
      end
      def cache_timeout(name,expire=10.minutes.from_now, &block)
        @controller.cache_timeout(name,expire,&block)
      end
    end
  end
end

#add to the respective controllers
ActionView::Base.send(:include, ActionView::Helpers::TimedCacheHelper)
ActionController::Base.send(:include, ActionController::Cache::TimedCache)

If anyone has any suggestions or comments please feel free to leave them in the comments.

Update: This is the link to the plugin.

Puddy -- YAWFP

2007-10-07T13:33:00.000-07:00

Yet another web framework for Perl, but I like to call it Puddy. A work in progress.

Short Description:

A project based on wanting a web framework built for speed and integration with standards. Currently modeling Rails in the area of application design, but improving areas with modules that are known to be faster and optimized (because I say so).

Model uses Rose::DB::Object for ORM support. It will handle to the association with the database that is well designed -- meaning the database handles the relationships not the ORM.

View uses a custom built XML::Builder and EPB that will handle templating that make incorporating Perl code into document easily. There is no language better than the langauge the application is built to use for the View. I am basing off the fact that out of my personal experience I have never met a 'designer' that just use a simple templating language.

Controller... well a work in progrees.

If anyone is interested in helping please message away at /[jtarchie][at][gmail][dot][com]/.

XML fragment caching in Rails

2007-10-04T07:18:00.000-07:00

Rails has been nice to use. The most powerful part of it is the templating engine for RHTML and RXML. A problem that exists is that they do not share some of the same functionality. Both can execute Ruby code, and output specific content, access helpers, etc. Yet, RXML lacks the ability for fragment caching.

The problem exists for fragment caching because it was especially designed for ERB fragments. ERB allows Ruby code to be use as a templating language -- the same way PHP does for HTML. The nice thing out ERB is that everything evaluated and rendered is plain text, but the caching relies on simple tricks with Ruby's eval to get the output of a block of ERB. The caching method cache (for views) even uses a function called cache_erb_fragment.

I use RXML alot, so the benefits of fragment caching should be available as they are for RHTML. Especially with the slow rendering time of the RXML documents (a future post). RXML relies on overloading of a method using method_missing to help create a DSL for an XML document, in short, Ruby code gets outputted to XML.

An Example RXML document:


directors = ['George Lukas','Steven Speilberg','Michael Bay']
xml.instruct!
  xml.directors do
     directors.each do |name|
     xml.director :name=>name
  end
end

With the output as:


<directors>
  <director name="George Lukas"/>
  <director name="Steven Speilberg"/>
  <director name="Michael Bay"/>
</directors>

A very simple example, but lets pretend the each director is dynamically loaded and that there is more information like age, location, and the last film they made. Information that will not change much till a movie is made, they get older, or they move. Never know when a director's already render XML might be needed again.

It has actually proven to be quite a task to do XML caching. It appears to be a subject neglected in the Rails community -- any idea why? I have posted on forums here and here about the topic. I had some preliminary code with those posts too, but I was having problems.

I would like to give back my final results of XML caching. Something that I hope is useful to some.

The following is the code use for the xml caching:


module ActionView
  module Helpers
    module CacheHelper
      def cache_xml(name ={}, &block)
        @controller.cache_xml_fragment(block, name)
      end
    end
  end
end

module ActionController
  module Caching
    module Fragments
      def cache_xml_fragment(block, name = {}, options = nil)
        unless perform_caching then block.call; return end
        buffer = eval("xml.target!",block.binding)
        if cache = read_fragment(name, options)
          buffer.concat(cache)
        else
          pos = buffer.length
          block.call
          write_fragment(name,buffer[pos..-1], options)
        end
      end
    end
  end
end

It works just like normal caching in RHTML documents. This is an example that can be used with the directors XML provided above.

Example Usage:


directors = ['George Lukas','Steven Speilberg','Michael Bay']
xml.instruct!
xml.directors do
directors.each do |name|
  cache_xml("director/#{name}") do
    xml.director :name=>name
  end
end
end

I hope to make this into a plugin very soon. Its easy to copy and paste for right now.

UPDATE: The plugin can be found here.

Experience with the web (Part 1?)

2007-09-29T21:55:00.000-07:00

I have been using the web since '94 in the 1900s. I wasn't apart of its original inception because I was too little and naive to know it even existed. Except for BBSs, those rocked, it still amazes me what an ANSI artist can do with TheDraw.

Currently, I am working for a company using Ruby on Rails as my primary development environment. Everything for the company is being written RoR. Its been interesting to use a framework to this extensive magnitude because you truly realize where its faults and advantages. With the changes, limitations, and additions I have worked into our implementation of RoR I thought that it would be nice to share in the case others might find it useful.

I claim to be no expert to programming. Anyone who make a claim to be an expert at anything better be ready to back it up. I will admit that I have experience though whether it be professional, amateur, or just plain tinkering. A lot of my interest in computers has always been the web. I like the power it has to be accessed by so many people from so many different places.

My first real website was way back in middle school. It was my play ground. Where everything from rants, drawings, hosting of others webpages, etc... were presented. It was my play ground for many year up till high school. Where I unsuccessfully tried starting my own cult. Unfortunately, a one person cult, well, is just lame.

From this website though, I picked up my first CGI scripts from Script Archive. This guy in my opinion really created the interest of CGI programs to provide web pages with interactivity and also was kind enough to lend his source code for everyone to use. This is what sparked my interest in Perl.

My most successful website to date incorporated alot what I had learned, but albeit still very crude. It was a fan site of South Park called killkenny. It had some of the copied content from many other fan sites, but also some original. I setup a weekly quiz, created with a friend of mine, about strange things we found in the episodes. This proved to be very popular because to date we had 100,000 unique hits / month. Keep in mind this is before Google Ads, so we made no money. It was for fun. Eventually it became, like other things, to difficult to maintain and was retired. From this I did the web design, content, hosting management, and interactivity B.A. (before ajax), during the year 1997 or 9th grade.

Well I am going to take a breather because this is much more than I intended to write. I might continue writing about this at a later time, but I would really like to write about issues I have faced with respects to web programming and show my solutions or questions.