Troubleshooting Chinese Character Sets in MySQL

A while back, I picked up a bug where when a customer tried to save certain kinds of data using Chinese characters, we were replacing the Chinese characters like 平仮名 with a series of ?.

This will be a quick dive through how I figured out what the problem was, and then validated said problem, and then what the actual fix is.

This isn’t just a story about character encoding, though. It’s a story about debugging. Debugging skills are valuable, and just like any other skill, they can be sharpened. I have built some competence in this domain, and I wanted to record some of the process, in the hopes that it might help others.

I perceive good debugging to go hand-in-hand with asking good questions. I wrote Asking experts, and gaining more than just answers about how asking good questions is hard, but done well makes it easy for someone else to help you, and leads to deeper understanding of the space.

Mark Dalrymple wrote Thoughts on Debugging, Part 1 and said:

Debugging, to me, is just a skill. It’s an analytical skill, but fundamentally it is a skill that can be learned and developed through practice: you identify problems, analyze the system to figure out what is causing that problem, and then figure out the changes necessary to correct the problem.

Finding the cause leads to the solution. The fix might not be practical from a business perspective. If fixing this bug requires overhauling the entire app, it might make better sense to leave it than to pay what it would take to fix.

Good debugging is also a particularly useful skill a developer could bring to their job, and you don’t have to be an experienced developer to be good at debugging. You could argue good debugging is based more on attitude than technical know-how, so if you’re just getting your start in the software development industry, making sure you’re good at debugging things and then telling potential employers this could be to your advantage.

Here’s some more thoughts on this, from other developers (quoted anonymously from the DenverDevs Slack channel):

For what it’s worth, I had most of this blog post written before even seeing these quotes.

dev_1:

There’s the tiresome old joke about “I’m not better at computers/coding/etc. than you, I’m just better at google!” Ha. So funny.

But a truly under-appreciated skillset that I think doesn’t get its fair share of mention from experts/teachers/celebrity-bloggers or attention by some developers is debugging.

There’s often so much mental headspace put into the idea of writing code, good code, functional code, that “what do I do when I don’t write working code” is rarely addressed.

Even well seasoned (yum) devs are constantly surprised by features of browser devtools, that console.trace() exists alongside console.log(), and more.

I think it’s often seen as a side effect of what we do so much that we learn the tools and techniques as we need them instead of as a crucial skill.

I think making mistakes in your code and figuring out why the error/mistake is occurring is a great way to further comprehension of a language, don’t get me wrong. But so many people are never told where to look first when they don’t know why or how an error is occurring. They just feel like they’re going to a doctor and saying “it hurts” they don’t know where, they don’t know how, just, ouch.

VCR's debug_logger and `git diff`

I recently added the vcr gem to one of our repositories, and was adding tests for an external API.

One of my tests was passing, and I wanted to commit the VCR cassette, along with the test/code that went with it.

I had thought I’d rebuilt the VCR cassette a few minutes before, but when I did git status, none of the expected files showed up. VCR cassettes record API calls and store them as big .yml files, which should show up in git status:

This did not map to my expectation - I expected to see the VCR cassette.

Two tools to help find the missing cassettes

First, I had to make sure the cassettes actually existed and were being used in my tests. Second, assuming the cassettes existed, I had to find them.

I checked the test_helper.rb, to see where the cassettes were supposed to be stashed:

VCR.configure do |config|
  config.cassette_library_dir = "fixtures/vcr_cassettes"
  config.hook_into :webmock
end

Climbing in Cuba, 2019

A few weeks ago, I had the opportunity to go climbing in Cuba.

Mark and Dave, walking back from climbing outside Viñales

Locals crag, called “The roof of the world”. Stunning routes.

because it was so hot, we spent a lot of time in this cave.

Some Lessons Learned While Preparing for Two Technical Talks

A few weeks ago, I gave two talks about Ruby and Rails:

1. An 8-minute “lightning talk” about using .count vs .size in ActiveRecord query methods
2. A 30-minute talk at the Boulder Ruby Group arguing that developers should embrace working with non-development business functions, and the constraints therein. I illustrated this via a story about finding slow SQL queries, and using .count vs .size in ActiveRecord query methods.

Things that went well

• I enjoyed actually giving the talks
• I heard positive feedback after-the-fact
• I learned a lot from the process, and next time the prep will be much less anxiety-inducing

Things that went poorly

• I felt quite anxious in the ten days or so leading up to the talks, thought it was because I was procrastinating.
• I felt stressed and shameful about not having the talks prepared.
• I did not finalize either talk until few minutes before leaving to give it, and was up late at night the night before each talk, doing the lions share of the preparation, therefore I was sleep deprived.

HTTParty and to_json

I was having some trouble debugging an HTTParty POST request.

A few tools that were useful to me:

1. post DEBUG info to STDOUT
2. netcat to listen to HTTP requests locally

I had this code:

options = {
  headers: {
    "Content-Type": "application/json",
    authorization: "Bearer #{our_token}",
  },
  query: { data: true },
  body: { token: their_token },
  debug_output: STDOUT
}

And when I posted it:

HTTParty.post("#{BASE_URL}/endpoint", options)

I kept getting something like this:

opening connection to externalservice.net:443...
opened
starting SSL for externalservice.net:443...
SSL established
<- "POST /endpoint?data=true HTTP/1.1\r\nContent-Type: application/json\r\nAuthorization: Bearer alksdjflkajsdf\r\nHost: externalservice.net\r\nContent-Length: 1234\r\n\r\n"
<- "token=aslsdjfhasiudyfkajn"
-> "HTTP/1.1 400 Bad Request\r\n"
-> "Date: Mon, 11 Mar 2019 16:58:13 GMT\r\n"
-> "Content-Type: text/plain\r\n"
-> "Content-Length: 28\r\n"
-> "\r\n"
reading 28 bytes...
-> "Invalid json line 1 column 7"
read 28 bytes
Conn keep-alive
=> "Invalid json line 1 column 7"