Exceptions were occasionally occurring in Padrino: Encoding::CompatibilityError - incompatible character encodings: UTF-8 and ASCII-8BIT:. This is with Ruby 1.9.2. Googling the error message turned up plenty of hits, none of them helpful. Trying the same thing in the Padrino Framework Google Group was more helpful, but a key piece of information was missing. Or since the posts were a year and a half ago, things changed. The mysql gem encodes all strings as 8BIT ASCII! Even if the database is UTF-8. Switching to the msyql2 gem and setting the adapter to mysql2 in config/database.yml solved the problem. If your don’t set the adapter correctly, Padrino complains about stack too deep. I encountered the same stack overflow with ruby-mysql.
Archive for the ‘Developing Amethyst’ Category
Character Encoding w/ Padrino, AR, MySQL, and Ruby 1.9
Friday, January 13th, 2012Setting DB Timezone in Padrino
Wednesday, January 11th, 2012All timestamps in Amethyst’s database are in UTC. Out of the box, Padrino and ActiveRecord assume they are in the local timezone. There may be a better way (i.e., ORM agnostic), but this works for ActiveRecord. In config/boot.rb:
##
# Add your before load hooks here
#
Padrino.before_load do
I18n.locale = :en
ActiveRecord::Base.time_zone_aware_attributes = true
ActiveRecord::Base.default_timezone = :utc
end
The first assignment sets the default locale (found this in the Padrino Localization guide). The next two are ActiveRecord specific. I dug into the Rails lib/initializer.rb code to find them.
Investigating Padrino
Wednesday, January 11th, 2012Rails 3 has outgrown my needs. I spent a day and a half trying to migrate Amethyst from Rails 2.3 to Rails 3.0 and didn’t appear to be any closer to being done. And I encountered a number of Rails3 bugs and/or incompatibilities that had no obvious fix.
Watching Rails 3 development, I see it creeping down the slope of being all things to all Web developers with lots of new, shiny thrown in for good measure. There are a lot of moving parts. (I need to install a Javascript interpreter? Really?) Rails3 was supposed to be much more modularized and configurable, cherry picking just the pieces I need, but I may have to install everything and then try and back out unneeded pieces.
Rails was originally developed by someone writing lots of Websites. They climb the initial, steep learning curve once and then can crank out product at a ferocious rate. The Website part of Amethyst is relatively straight forward. It’s the scoring and the backend (i.e., non-Rails parts) that are the secret sauce. And much of the Rails2 and Rails3 magic costs too much in terms of user response times (i.e. the routing magic – I’ve reverted to string interpolation for URLs).
I’m investigating migrating to Padrino, a Web framework built on top of Sinatra. Padrino claims to be 3-5 times faster than Rails. My first cut using the simplest usable piece of Amethyst (viewing the articles in a whitelisted subset of the RSS feeds subscribed to) is appropriately twice as fast. But it doesn’t yet have pagination or caching. I expect the former to slightly slow it down and the former to significantly speed it up. I’ll be blogging on my progress, the problems I encounted (and solutions when I find them), and how satisfactory I find Padrino.
Fast XML Parsing in Ruby
Tuesday, November 15th, 2011Dr. Dobb’s Journal just published my article on Fast XML Parsing in Ruby It is based on techniques I developed to speed up refreshing RSS feeds for Amethyst. In the years since I first wrote the code and benchmarked the competition, some of them have improved their speed, but my code is still the fastest (though more specialized).
Google Reader
Thursday, November 3rd, 2011Google Reader is Amethyst’s most visible competition. A new release has been rolled out recently that is getting a lot of mostly negative notice. Chris Wetherell was the original developer of Google Reader. He has posted some of his thoughts about the direction Google Reader is going on Google Plus here.
I think his statement, “Reader is (was?) for information junkies; not just tech nerds. This market totally exists and is weirdly under-served (and is possibly affluent).”, is correct, but I haven’t found how to reach those people. In fact, I am seriously considering shutting down Amethyst on the cloud and keeping it as my secret weapon. There are still some things for me to learn running in the cloud, so for the moment it is going to stay up.
Clever Hack, Smelly Code
Wednesday, October 19th, 2011A common operation in Amethyst is to retrieve the feed information from the database, and the number of unread articles in the feed. Operating in pure SQL I could do something like this SELECT feeds.*, COUNT(articles.id) AS count_articles FROM feeds JOIN articles ON feeds.id = articles.feed_id WHERE feeds.id = 1234 (note: not tested). If is possible in Rails 2 with ActiveRecord to do something similar with feed = Feed.find(1234, :select => 'feeds.*, COUNT(articles.id) AS count_articles', :joins => :articles). And it will work. The article count will be in feed.attributes['count_articles']. But, if the find() becomes more complex, it may not. At some point, ActiveRecord will generate it’s own select and the COUNT(articles.id) will fall on the floor.
Clever, but brittle as all get out. A slight change to my calling code or any update to Rails could break, a definite code smell. So I’ve ripped out all places where I did this. Yeah it takes an additional trip to the database, but all or almost all of the necessary data is already sitting in the DB’s cache, so it’s in the low single digit millisecond range. Not worth brittle code.
Conditional GET Requests
Tuesday, September 13th, 2011Amethyst fetches updates to all subscribed RSS feeds every hour. And most of the time, there is nothing new. This is a waste of bandwidth and CPU usage, both Amethyst’s and the RSS feed server. I briefly looked into Conditional GET support (only get the data if it has changed). I didn’t dig very deep and everything I found was about supporting responses to Conditional GETs in Rails, not making requests. Finally it annoyed me enough to dig a little deeper.
The most useful resource I found was a link to RESTful Web Services in Google Books. It isn’t specific to Rails, but the examples are in Ruby. I’m using EventMachine HTTP Request to read RSS feeds and it has last_modified and etag accessors. Just store them in your response handling code, add the stored values to the headers:
request = HttpRequest.new(channel)
headers = {}
headers['If-Modified-Since'] = channel.last_modified if channel.last_modified
headers['If-None-Match'] = channel.etag if channel.etag
http = request.get :head => headers
and handle a 304 (Not Modified) response code the same as a successful response with no new posts.
This cut the CPU by 80%, i.e. CPU load is 20% of what it was. This may have been premature optimization/scaling, but it was very satisfying and it will reduce my Amazon Web Services bill slightly. The developer(s) is one of the stakeholders you need to keep happy too.
OPML Parser Up on Github
Thursday, August 4th, 2011There are several reasons to spin stable parts of a Rails application off into Gems:
- Reduced test times – the tests for the gem only need to be run when the gem changes, not every time the application changes
- Reuse
- Sharing and collaboration
RSS feeds are typically moved from one reader to another in an OPML (Outline Processor Markup Language). Amethyst allows importing your feeds from Google Reader or most other RSS Readers that can export an OPML file. I’ve spun the code to do so off into a Ruby Gem and created a repository for it on github. I’ve probably commited all kinds of faux pas, but hey, it’s my first github repository and correctable.
Disappearing Checked Off Items
Tuesday, May 31st, 2011My workload is largely task driven (as opposed to appointment driven like a family doctor). So I use a number of TODO lists – on my smartphone and on my computer. One thing that bugs me is tasks that disappear when checked off. Poof, all my work disappears and is unacknowledged. Bad for us workaholics trying to stay in recovery. And given the vagueness of gestures on touchscreen devices, not nice. Several times I have had to go digging through the completed items to uncheck a wrong selection.
One of the weaknesses of Amethyst’s UI is short titled articles may lead to errors on clicking the correct delete box. So now delete boxes on articles don’t immediately banish the article. It will be lined out. Clicking the box again will undelete it.
And I’m working on a better solution to matching up short titles with their delete box besides alternating color bars.