How Much Code Coverage is Enough?

Margaret Williford, Former Developer

Article Category: #Code

Posted on

Measuring code coverage allows you to see exactly what percentage of your codebase you're testing. What should we be aiming for: 90%, 100%, or something else entirely?

We value automated testing at Viget, but we don't have a specific rule on code coverage from client to client. Between personal projects and Viget clients, my testing habits vary widely from scrappy MVP code with minimal tests, to robust end-to-end test suites with 100% coverage. The latter got me thinking: how much does 100% coverage really tell us? I'd argue that, like most things in life, what we should really be striving for here is quality and not quantity. I should note that I'm writing particularly about using Rails, RSpec, and SimpleCov here.

Pros of Testing #

  1. Well written tests communicate intent to fellow developers.
    • Well written describe, context, and it blocks lay out a map for other developers to come in and understand how the code is expected to work.
  2. Testing gives you confidence in edge cases and error handling.
    • Errors will happen. While you can't realistically account for every possible edge case, you can write tests for sad paths to be confident they're handled appropriately.
  3. Integration tests make refactoring less scary.
    • With thorough integration tests, you can rip apart a large feature X and your test suite will ensure there aren't any unintended side effects on feature Y or Z.
  4. Unit tests makes refactoring less scary.
    • Writing that it should have_many :photos spec may seem trivial, but you'll be happy to have it when you change the photos table name to images. You'll catch that the relationship name needs to change.

Cons of Testing #

  1. They take 5ever to write.
    • The more you're in the habit of writing integration specs, the faster you'll become at writing them. Even so, it's always going to be a long, sometimes tedious task to test. However, the hours spent writing tests before launch feel a lot better than the hours spent trying to hot fix a bug that's gone to production.
  2. If you wish to make an apple pie from scratch, you must first create the universe.
    • Let's say you need to test the order confirmation page. You need an order. To create an order, you need a user and some products. You need a way to mock payment processing. The setup bloat can get very big, very fast.
    • Tools like FactoryBot as well as spec helper files that break out common flows will make your life easier. Taking the time to create methods like add_tickets_to_cart and submit_successful_payment that are accessible by multiple specs will make your life (and a future developers life!) easier.
  3. API calls are tedious.
    • You're writing an integration test that requires making an API call to an external dependency. You don't want to hit your rate limit or be charged just for running tests.
    • VCR is your friend here. Make the call once and store the request/response to use every time the test runs.

100%? #

100% coverage does not mean your code is flawless. Firstly, it's easy for that 100% number to become meaningless quickly. You could theoretically put #:no-cov#s all over the place and claim 100% coverage without having even 50% coverage (don't do this!!).

On the other hand, you can "cover" a line of code without testing anything meaningful at all. Consider a method

def display_text
  "Your order has been received."
end

You could write a unit test showing that the display_text method matches a string. That method would be 100% covered. A more meaningful test would be to write an integration test that this display_text shows up on the order confirmation page. 100% covered doesn't necessarily mean a codebase is error-proof.

My Two Cents #

Use a tool like SimpleCov and set a coverage goal, somewhere between 90-100%. You'll want to consider the level of QA engagement, development time available to get a feature out, and how risk averse you are. Mostly, don't get too caught up in that specific number. High coverage != well tested. If you're focused on the number, you could end up with perfunctory tests that don't provide much value. Instead, focus on writing a solid mix of unit tests and integration tests that account for happy and sad paths, and the coverage will come.

Related Articles