UTF-8 Good, Windows-1252 Bad
So I’m using sidekiq, with sidekiq-scheduler at work in production to handle background tasks. Well, the other night around 10pm, just as I was settling down to watch the last episode of Mad Men, all hell broke lose.
After digging through the logs to find out what happened, all I could find was this
2012-05-07T01:59:10Z 24340 TID-fmbc0 ERROR: Manager#assign died 2012-05-07T01:59:10Z 24340 TID-fmbc0 ERROR: "\xC3" on US-ASCII 2012-05-07T01:59:10Z 24340 TID-fmbc0 ERROR: /home/web_apps/dream/shared/bundle/ruby/1.9.1/gems/json-1.6.6/lib/json/common.rb:148:in `encode'
None of the stack trace bubbled up to my app, so I knew I had no chance at rescuing from an exception. That left me with one thing to do, try and figure out what the heck caused this. Fortunately, I was able to lean on a few more seasoned colleagues of mine for advice.
After some discussion, we landed on our app being sent a windows-1252 encoded character, and trying to parse it as UTF-8.
So, to combat such things, we came up with this:
# This method exists because it has been shown that we cannot # trust data coming from facebook # def clean_fb_str(str) ic = Iconv.new('UTF-8//IGNORE', 'UTF-8') if str == ic.iconv(str) # valid UTF-8 string should match its iconv version str else # UTF-8 conversion changed string, force windows-1252 into utf-8 windows_ic = Iconv.new('UTF-8//IGNORE//TRANSLIT', 'WINDOWS-1252') windows_ic.iconv(str) end end
Seems to do the job, as we haven’t yet had a reoccurrence of the issue.
NOTE: After deploying this on ruby-1.9.3, I’ve noticed we are getting deprecation warnings. I suppose we should be using String#encode instead.