Company Updates

Today we’re excited to announce that Code Climate has raised $4.5M in Series A funding from Union Square Ventures (USV) and existing investors NextView Ventures, Lerer Hippeau Ventures, Trinity Ventures and Fuel Capital. Albert Wenger, Managing Partner at USV, is joining our board, and we’re excited to join USV’s portfolio of companies, which includes MongoDB, Twilio, Etsy, and others.

This is a major milestone for us, and we’re excited about what it means for the over 80,000 developers who use Code Climate to build and ship better software. We have a big vision for the future:

Empowering every software developer to improve the quality of their code and the outcomes of their projects with the most advanced, open and extensible platform for source code analytics.

Since our seed funding round in summer of 2014, we’ve introduced an open and extensible static analysis platform, shipped a major new edition of our on-premises product, and launched a powerful browser extension that brings code quality and test coverage information directly into the GitHub user interface.

We’ve grown to provide analysis for more than 90,000 repositories across 20+ programming languages and frameworks, and analyze two billion lines of code each day – making us the largest provider of cloud-based static analysis.

Here’s what our new investors had to say:

There is a common mantra that you can have any two in software: fast time to market, low cost of development, or high quality, but never all three. This, of course, is also what people used to believe about manufacturing before the rise of techniques such as lean manufacturing and continuous improvement. When you lead with quality in manufacturing you can, in fact, have quality, speed, and low cost. The same will be true for code, which makes assessing and managing the quality of code a key challenge for the coming years. - Albert Wenger, Union Square Ventures

We completely agree, and we’re thrilled to have the resources to accelerate our work on this problem for years to come. To that end, we’d like to share a little about the next big thing in the works…

Coming soon: A new, open source edition of Code Climate

We’re working on an open source, standalone edition of the Code Climate application, which you’ll be able to download and run anywhere – including behind your firewall – completely free. We’re calling it Code Climate Community Edition (CE) for now and expect to ship an alpha version within a month.

It should come as no surprise that we’re huge fans of open source. We’ve long provided free service to open source projects, and last year we open sourced our static analysis engines and a command line interface (CLI) when we launched our open, extensible platform. We see providing a workflow-integrated code analytics experience as an open source package as the next big step in making Code Climate a ubiquitous part of every software developer’s toolbox, and hope you’re as excited about it as we are.

Great companies like GitLab and Travis CI have built thriving business around open source (or “open core”) offerings, and we thank them for their inspiration and innovation in these areas.

Since it will be possible to run the community edition of Code Climate for free, you may wonder what our plans are for revenue. In short, they don’t change much. We’ll continue to provide our cloud-based product at CodeClimate.com (still free for open source projects!) where we host and manage everything for you and keep things running smoothly. We’ll also continue to offer an “enterprise” edition of Code Climate for on-premises installs, with exclusive features and services focused on larger teams. We’ll have more to share when we release the Community Edition alpha.

We believe that every developer and every project can benefit from Code Climate from day one, and are happy to make that even easier for more organizations, regardless of their requirements for where they store their source code.

Today we’re excited to announce that the Code Climate Quality platform is out of beta. When we launched the first open, extensible platform for static analysis back in June of last year, we knew it would take some time to perfect. Since then, we’ve been working hard to add features, fix bugs, support community contributors, and upgrade infrastructure to create a superior experience for developers everywhere.

Our engines-based analysis now offers the key features you’ve come to expect from our “classic” offering, and a lot more, including:

• Style checks for Ruby, PHP and Python
• Support for more programming languages like Go, Haskell, JSX, CoffeeScript and SCSS; and frameworks like Ember.js, React and RubyMotion
• Dependency audits for potential vulnerabilities in your RubyGems and NPM packages
• Atom editor integration and a CLI to get feedback before you push
• Improved configurability and tuning options support in .codeclimate.yml(and per-engine config files)
• Ability to create your own analysis engines using our open source spec and make them available through CodeClimate.com

Already, Code Climate serves tens of thousands of developers. We’re proud to be working with a community of over 100 contributors and members of our Developer Program to provide over 1,800 checks across 20 static analysis engines on our new platform.

"Code Climate allows us to incrementally improve and maintain good code hygiene, decreasing the number of defects in our products, and leading to better overall product quality. The Code Climate open platform is a game changer and allows us to build and improve static analysis engines to support our highly complex environment." — Andy Blyler, Senior Director of Engineering, Barracuda Networks

For those who gave our engines-based analysis a try soon after launch, in addition to bug fixes you’ll notice a number of key improvements, including:

• Visibility into the analysis process via our new Builds pages
• More accurate ratings and GPAs
• Improved Issues browsing and filtering
• Better performance for faster results
• Improved stability from infrastructure upgrades we’ve made behind the scenes

Want to see it for yourself? Take a look!

Start using the Code Climate Quality platform today

Sign up today to get started with Code Climate Quality for your projects.

For existing customers, just follow these instructions for moving your repositories over to our new engines-based analysis. We’re confident that you’ll find it’s a superior experience to our “classic” offering.

Finally, we recognize that Code Climate is most valuable when you can use it wherever your work. For customers whose ability to host code with third-party services is limited, we recommend Code Climate Enterprise, our on-premise service.

With the new foundation that the Code Climate Quality platform provides, we’re looking forward to the new opportunities we’ll have to bring you even more functionality. Stay tuned for more big announcements (you can follow us on Twitter here).

Recently, we announced the release of the Code Climate Quality platform, which lets anyone create and deploy static analysis tools to an audience of over 50,000 developers. These Open Source static analysis tools are called “engines" or "plugins," and in this post I’ll show you how to create one from scratch.

We’ll create a plugin that greps through your source code and looks for instances of problematic words like FIXME, TODO, and BUG. This plugin conforms to the Code Climate plugin specification, and all of the code for this plugin is available on GitHub. When we get up and running, you’ll see results like this on your command line:

Plugins that you create can be tested with the Code Climate command line tool. Once you’ve got an Open Source plugins that works, we’d love to chat with you about making it available on our cloud platform, so that your whole community can have access to it!

For more information join our Developer Program.

What’s a plugin made of?

Instead of asking you to dive into the Code Climate plugin specification, to learn what a plugin is, I’ll give you a brief overview here.

A Code Climate plugin is a containerized program which analyzes source code and prints issues in JSON to STDOUT.

Sound simple? We really think it is! Hopefully this blog post will illustrate what we mean.

More concretely, the FIXME plugin we’re going to create contains three important files:

• A Dockerfile which specifies the Docker image
• A bin/fixme executable wrapper script that runs the plugin
• The index.js file, which contains the plugin source code

There are other requirements in the specification regarding resource allocation, timing, and the shape of the output data (which we’ll see more of below), but that’s really all there is to it.

A little bit of setup

Before we write any code, you need a few things running locally to test your plugin, so you might as well get that out of the way now. You’ll need the following things running locally:

• A Docker environment (we recommend Docker For Mac for OSX development.
• The Code Climate CLI tool (you can brew tap codeclimate/formulae && brew install codeclimate on OSX)

Run codeclimate -v when you’re done. If it prints a version number, you should be ready to go!

FIXME

The idea for FIXME came to us when we were brainstorming new plugin ideas which were both high value and easy to implement. We wanted to release a sort of Hello, world plugin, but didn’t want it to be one that did something totally pointless. Thus, FIXME was born.

The FIXME plugin looks for (case-insensitive, whole word) instances of the following strings in your project’s files:

• TODO
• FIXME
• HACK
• BUG
• XXX

This is not a novel idea. It’s well known that instances of these phrases in your code are lurking problems, waiting to manifest themselves when you least expect it. We also felt it worth implementing because running a FIXME plugin in your workflow has the following benefits:

• Existing FIXMEs hacks will be more visible to you and your team
• New FIXMEs will bubble up and can even fail your pull requests if you configure them properly on codeclimate.com

Pretty nifty for around 75 lines of code.

To achieve this, the plugin performs a case insensitive grep command on all of the files you specify, and emits Code Climate issues wherever it finds one.

Implementing an plugin in JavaScript

The meat of the actual plugin is in the index.js file, which contains around 50 lines of JavaScript. The entirety of the file can be found here. I’ll highlight a few important sections of the code for the plugin below, but if you have any questions, please open an issue on the GitHub repo and I’ll try my best to answer promptly!

On to the code. After requiring our dependencies and typing out the module boilerplate, we put the phrases we want to find in grep pattern format:

var fixmeStrings = "'(FIXME|TODO|HACK|XXX|BUG)'";


This will be used in a case insensitive search against all of the files the plugin we’ll analyze.

Next, we create a function that we will use to print issues to STDOUT according to the issue data type specification in the plugin spec. The printIssue function accepts a file name, a line number, and the issue string,

var printIssue = function(fileName, lineNum, matchedString){
 var issue = {
   "type": "issue",
   "check_name": "FIXME found",
   "description": matchedString + " found",
   "categories": ["Bug Risk"],
   "location":{
     "path": fileName,
     "lines": {
       "begin": lineNum,
       "end": lineNum
     }
   }
 };

 // Issues must be followed by a null byte
 var issueString = JSON.stringify(matchedString)+"\0";
 console.log(issueString);
}


This data format contains information about the location, category, and description of each issue your plugin emits. It’s at the heart of our plugin specification and massaging data from an existing tool to conform to this format is typically straightforward.

The data in the JSON your plugin prints will be consumed by the CLI and if you join our Developer Program and work with us, it can also be made available to all users of Quality. We’ll work with you to ensure your plugin is spec compliant and meets our security and performance standards, and get your work in front of a lot of people!

The actual code that greps each file isn’t super interesting, but you should check it out on GitHub and open an issue on the repo if you have a question.

Because it’s a requirement of plugins to respect the file exclusion rules passed to it by the CLI or our cloud services, though, I’ll show a bit of how that works:

// Uses glob to traverse code directory and find files to analyze,
// excluding files passed in with by CLI config
var fileWalk = function(excludePaths){
 var analysisFiles = [];
 var allFiles = glob.sync("/code/**/**", {});

 allFiles.forEach(function(file, i, a){
   if(excludePaths.indexOf(file.split("/code/")[1]) < 0) {
     if(!fs.lstatSync(file).isDirectory()){
       analysisFiles.push(file);
     }
   }
 });

 return analysisFiles;
}


Here I am using the NPM glob module to iterate over all of the files starting at /code recursively. This location also comes from the plugin specification. The fileWalk function takes an array of excludePaths, which it extracts from /config.json (this will be made available to your plugin after the CLI parses a project’s .codeclimate.yml file). This all happens in the main function of the plugin, runEngine:

FixMe.prototype.runEngine = function(){
 // Check for existence of config.json, parse exclude paths if it exists
 if (fs.existsSync("/config.json")) {
   var engineConfig = JSON.parse(fs.readFileSync("/config.json"));
   var excludePaths = engineConfig.exclude_paths;
 } else {
   var excludePaths = [];
 }

 // Walk /code/ path and find files to analyze
 var analysisFiles = fileWalk(excludePaths);

 // Execute main loop and find fixmes in valid files
 analysisFiles.forEach(function(f, i, a){
   findFixmes(f);
 });
}


This main function gives hopefully gives you a clear picture of what this plugin does:

• It parses a JSON file and extracts an array of files to exclude from analysis
• It passes this list of files to a function that walks all files available to the plugin, and produces a list of files to be analyzed
• It passes the list of analyzable files to the findFixmes function, which greps individual files and prints them to STDOUT

Packaging it up

How plugins are packaged as Docker containers is important: it has its own section of the plugin specification. The Dockerfile for FIXME is pretty typical:

FROM node

MAINTAINER Michael R. Bernstein

RUN useradd -u 9000 -r -s /bin/false app

RUN npm install glob

WORKDIR /code
COPY . /usr/src/app

USER app
VOLUME /code

CMD ["/usr/src/app/bin/fixme"]


Here’s a breakdown of each line (for more information about each directive, see the official Docker documentation):

1. The official node Docker container is the basis for this plugin container. It has nodeand npm installed, and generally makes our lives easier.
2. Declare a maintainer for the container.
3. Create the app user to run the command as specified.
4. Install packages with npm install glob so that the external dependency is available when the plugin runes.
5. Set the WORKDIR to /code, where the source to be analyzed will be mounted.
6. Copy the plugin code to /usr/src/app.
7. Use the app user that we created earlier.
8. Mount /code as a VOLUME per the spec
9. Our plugin specification says that the plugin should launch and run immediately, so we use CMD to achieve this. In the case of FIXME, the executable wrapper script instantiates the plugin we wrote in JavaScript above, and runs it. Check it out:

#!/usr/bin/env node

var FixMe = require('../index');
var fixMe = new FixMe();

fixMe.runEngine();


We now have all of the pieces in places. Let’s test it out.

Testing your plugin locally

If you want to test the code for this plugin locally, you can clone the codeclimate-fixme repository locally, and follow these steps:

• Build the docker image with docker build -t codeclimate/codeclimate-fixme . (You must be inside the project directory to do this)
• Make sure the plugin is enabled in the .codeclimate.yml file of the project you want to analyze:

 engines:
   fixme:
     enabled: true


• Test the plugin against the plugin code itself (whoooah) with codeclimate analyze --dev

And you should see some results from test/test.js! Pretty cool, right?

Note that if you want to test modifications you are making to this plugin, you should build the image with a different image name, e.g. codeclimate/codeclimate-fixme-YOURNAME. You would then add fixme-YOURNAME to your .codeclimate.yml file as well.

If you get stuck during development, invoke codeclimate console and run:

Analyze.new(['-e', 'my-engine', '--dev']).run


And you should be able to see what’s going on under the hood.

What will you build?

Hopefully seeing how straightforward an plugin can be will give you lots of great ideas for plugins you can implement on your own. If tools for your language don’t exist, contact us, and maybe we can help you out!

Simple ideas like FIXME have a lot of power when your entire team has access to them. Wire up the codeclimate CLI tool in your build process, push your repositories to Code Climate, and keep pursuing healthy code. We can’t wait to see what you’ll build.

Update: See our CEO and Co-Founder Bryan Helmkamp introducing the Code Climate Quality Platform!

Today, we’re thrilled to launch the Code Climate Quality Platform − the first open, extensible platform for all types of static analysis.

We’ve come a long way since we started building a static analysis tool for Rubyists. Today, we’re fortunate enough to help 50,000 developers analyze about 700BN lines of code every weekday. As we’ve grown, we’ve seen that clear, actionable static analysis leads to healthier code and happier developers. Our new platform brings those benefits to every team, regardless of the technologies they use, so that they can focus on shipping quality software.

What does this mean exactly? First, we’re open sourcing our analysis tools, including the engines and algorithms we use to evaluate code. If you have a running Docker host, using Code Climate on your command line is as easy as:

boot2docker up && `boot2docker shellinit`  brew tap codeclimate/formulae  brew install codeclimate  

We’re also enabling anyone to write static analysis engines that run on our servers by following a simple specification. No longer will you have to wait for us to provide support for your programming languages, frameworks and libraries of choice. Finally, using our new Code Climate CLI, you can now run any Code Climate-compatible static analysis on your laptop – for free.

Let’s look at each of these in turn…

Open Source

We’re releasing the static analysis engines that power the new Code Climate Quality Platform, and going forward, all of our static analysis code will be published under Open Source licenses. Code Climate has always provided free analysis to Open Source projects, and this continues to deepen our commitment to, and participation in, the OSS community.

Code Climate has always stood on the shoulders of giants in depending on Open Source libraries for the bulk of our static analysis algorithms. We would not exist today if not for the great work of people like Ryan Davis (Flog, Flay) and Justin Collins (Brakeman) that demonstrated the value of static analysis and allowed us to quickly bring it to developers.

Open Source static analysis means you can dig in and understand exactly how your code is being evaluated. And customizing static analysis algorithms becomes as simple as clicking the “Fork” button on GitHub…

Extensible

With the release of our new Code Climate Engine Specification, anyone can write static analysis engines that run on our servers. Code Climate’s set of officially supported languages can be augmented with community supported static analysis engines so that you can get confidence in your code, regardless of your choice of technology. For years, our most common type of feature request has been, “Can you add support for X?”, where X is a programming language, version of a language, or sometimes a framework. We’ve always wanted Code Climate to work for all developers, but in the past we were limited by the effort required to add new languages.

We believe that you shouldn’t have to wait for a vendor to add support for the languages you love, so are finally removing ourselves as the bottleneck to new static analysis features. Anyone who is interested in using Code Climate with their favorite languages, frameworks and libraries is free to build an engine to do so. Results from engines retain all the benefits of the Code Climate product, including automatic analysis of every Pull Request, comparison views, ratings/GPAs, and notifications. Of course, there’s already a vibrant community of OSS static analysis tools, and we’re excited to see what people are able to build and integrate. It’s really astounding how simple it is to build a Code Climate engine, as we’ve watched developers build functioning analysis from scratch in a matter of a couple hours.

If you want to give it a try, join our new Developer Program and we’ll be there to guide you along the way.

Run Anywhere

In addition to the spec, we’re also releasing the Code Climate CLI, which makes it easy to get static analysis results (both from Code Climate’s official engines and community-supported engines) in one clear, unified report right on your laptop. When you’re ready, you can load your repository into codeclimate.com and we’ll automatically apply the exact same configuration you use locally to analyze every commit and Pull Request, making the results available to your entire team.

To make static analysis truly ubiquitous, we realized it was not enough to support a wide variety of tools, we need to make it trivial to run these tools anywhere the developer is working. You shouldn’t have to wait to push your source code to a remote server to get clear, actionable static analysis results. Now it’s possible to easily run the same static analysis we run on our servers on your command line.

New Static Analysis Engines

We’re fortunate to be partnering with creators of two prominent Open Source projects who understand the value of static analysis in ensuring healthy code, Tom Dale from Ember.js and Laurent Sansonetti from RubyMotion. Here’s what they have to say about the Code Climate Quality Platform:

“The Ember community takes good tooling seriously. I’m excited to partner with Code Climate and bring their service to our users because better static analysis means better Ember apps.”

– Tom Dale from Ember.js

“HipByte is excited to partner with CodeClimate to provide clear and reliable code reviews for RubyMotion projects. RubyMotion allows Ruby developers to write cross-platform apps for iOS and Android by leveraging the native platform APIs, and a tool for static analysis means fewer crashes and more reliable applications.”

– Laurent Sansonetti from RubyMotion

Accordingly, we’re releasing an Ember.js engine that brings the Ember Watson tool to all Code Climate users, and a set of custom checks for RubyMotion that will help ensure that your application code is reliable.

In addition, we’re proud to release eight new static analysis engines that you can start using with Code Climate today:

• Gofmt, Govet and Golint – The official style, bug and lint checkers from the Go team
• CoffeeLint – Style checking for the CoffeeScript dialect of JavaScript
• CSSLint – Style checking for all CSS stylesheets
• Rubocop – Style and quality checks for Ruby code (including support for RRuby 2.2+)
• ESLint – Linting and style checking for your modern EcmaScript/JavaScript code
• Bundler Audit – Find security vulnerabilities in your Ruby dependencies
• PHP CodeSniffer – Style checking for PHP

Healthy Code Ships Faster

In many ways, our new platform is a culmination of the experience we’ve gained over the past four years building and operating the most popular static analysis app. We’re excited to bring this to you, and look forward to continuing to bring you the best tools to ship better code, faster. If you want to try out the new Code Climate, download our CLI to get started.

In exciting programming language news (how often do you get to say that?) Apple recently released Swift, a compiled language meant to augment Objective-C for development on Desktop and Mobile iOS platforms. While Swift’s primary audience at this (very early) point is iOS developers, the language contains some interesting functionality for program design that will appeal to a wider audience, including Ruby, JavaScript, and PHP developers.

In this post I’ll take a look at three of the program design constructs that Swift provides: Classes, Structures, and Protocols. I’ll discuss how they can contribute to well-designed Object-Oriented (OO) programs in a way that might be interesting to dynamic language programmers, and show how aspects of their design reflect concerns of the platform for which they were developed.

Classes and Structures

The best source of information about Swift so far is The Swift Programming Language, a free e-book provided by Apple. It contains a first pass introduction to the language, a language reference, and a good deal of example code. After introducing the basic facilities of the language – its philosophy, the basic available types and data structures, control flow, closures, and so on – we get to the section on Classes and Structures.

“Classes and structures are general-purpose, flexible constructs that become the building blocks of your program’s code.” [1]

This sounds similar to the functionality of a Class or Module in Ruby. From this list of the features of Classes and Structures, we see even more similarities. Classes and Structures can both:

• Define properties to store values
• Define methods to provide functionality
• Define initializers to set up their initial state
• Be extended to expand their functionality beyond a default implementation

This means that Swift provides similar functionality to familiar OO Programming Languages. We can model problems using objects which can encapsulate data and functionality, and build complex relationships between these objects in a modular fashion. Curious readers might have the same reaction to this as I did – if it’s the case that both classes and structures can do the above, which is a reasonable set of functionality to expect from classes, why are there two constructs? What are the differences between them, and why does Swift as a programming language need both?

First, let’s take a look at what Classes can do that Structures cannot. Classes provide functionality for two crucial pieces of OO functionality that Ruby developers, for instance, tend to rely on somewhat heavily:

• Inheritance, where one class can inherit the characteristics of another
• Type casting, where you can check and interpret the type of a class instance at runtime

Additionally, classes provide the following facilities which are stated in terms familiar to those with experience managing resources and memory manually:

• Deinitializers, to enable an instance of a class to free up any resources it has assigned
• Reference counting, to allow for more than one reference to a class instance

Stated in other terms, inheritance, type casting, deinitializers, and reference counting make it possible to expressively create designs which employ the full compliment of OO techniques.

From my perspective, Classes and Structures have just the right amount of overlap in functionality, leaving enough room for developers to make reasoned decisions about which construct to employ according to the purpose it may be used for. The differences hinge on how instances of Classes and Structures are represented in memory after they are initialized – structure instances are always passed by value and class instances are always passed by reference. This distinction is something that dynamic language developers typically do not have to spend much time thinking about – on a mobile platform, however, this kind of thinking becomes very important.

Different memory semantics is not the only advantage of having these distinct types, however. Because structures are simpler than classes, and cannot be as heavily modified after declaration, they provide an opportunity to create value objects which represent pieces of data independent from their behavior. This is very useful.

While Classes and Structures cover much of the familiar ground with respect to OO functionality in Ruby or Python, there is one more construct that might not be so familiar that I’d like to point out before we draw any conclusions on Swift’s OO design capabilities.

Protocols

Protocols are an interesting addition to the already rich world of classes and structures. They provide a way to define behavior separately from the classes which implement them:

“A protocol defines a blueprint of methods, properties, and other requirements that suit a particular task or piece of functionality”

Similar constructs exist in Java, Go, and other languages, but many dynamic languages do not embrace this design style. Protocols embrace flexible design by encapsulating the necessary data and behavior for a domain idea outside the scope of a Class or Structure definition. This means that a concept can be represented separately from its implementation, allowing for more creative reuse, composition, and more.

To illustrate the usefulness of Protocols, consider the development of a small networked process to record some information from a system over time. You might want to have the system write to a Key-Value store, or to disk, or to STDOUT, depending on a variety of circumstances. In this case, you would simply define a Protocol:

protocol BackEnd {      func get(Int) -> Int      func set(Int) -> Bool  }  

This protocol requires that anything which implements them must contain at least two instance methods – get, which accepts an Integer and returns an Integer, and set, which accepts an Integer and returns a Boolean value. A class that implements this Protocol might look something like the following:

class RedisBackEnd : BackEnd {      func get(id: Int) -> Int {          // Get 'val' from Redis based on 'id'          let val = getFromRedis(id)          return val      }        func set(val: Int) -> Bool {          // Store 'val' in Redis here          let boolFromRedis = storeInRedis(val)          return boolFromRedis      }  }  

You could imagine similar classes for the other backends I mentioned above. This is a very convenient solution for swapping out backends when you need to run your process in different locations and under different circumstances – simply implement new ones as needed, and pass them in wherever a Backend is needed.

Defining a Protocol and implementing it is really the tip of the iceberg, however – things can get very subtle and complex. Protocols can be treated as types, and intricate hierarchies can be created with them. While it is interesting to look through the Swift book to see these examples, I don’t believe they are the real selling points of protocols. Used simply and in combination with classes and structures, protocols provide a missing piece of the design puzzle that often vexes programmers in dynamic languages.

Conclusion

Swift is a very interesting programming language that at first blush appears to have all of the necessary components to build expressive OO programs. In addition to classes, structures, and protocols, Ruby developers might also want to look into Extensions which offer mixin-like capabilities.

Whether you prefer inheritance, composition, or even a more esoteric SmallTalk style flavor of OO, it seems that Swift will be able to support you.

Works Cited

[1] Apple Inc. “The Swift Programming Language.” iBooks.

Service-Oriented Architecture has a well-deserved reputation amongst Ruby and Rails developers as a solid approach to easing painful growth by extracting concerns from large applications. These new, smaller services typically still use Rails or Sinatra, and use JSON to communicate over HTTP. Though JSON has many obvious advantages as a data interchange format – it is human readable, well understood, and typically performs well – it also has its issues.

Where browsers and JavaScript are not consuming the data directly – particularly in the case of internal services – it’s my opinion that structured formats, such as Google’s Protocol Buffers, are a better choice than JSON for encoding data. If you’ve never seen Protocol Buffers before, you can check out some more information here, but don’t worry – I’ll give you a brief introduction to using them in Ruby before listing the reasons why you should consider choosing Protocol Buffers over JSON for your next service.

A Brief Introduction to Protocol Buffers

First of all, what are Protocol Buffers? The docs say:

“Protocol Buffers are a way of encoding structured data in an efficient yet extensible format.”

Google developed Protocol Buffers for use in their internal services. It is a binary encoding format that allows you to specify a schema for your data using a specification language, like so:

message Person {    required int32 id = 1;    required string name = 2;    optional string email = 3;  }  

You can package messages within namespaces or declare them at the top level as above. The snippet defines the schema for a Person data type that has three fields: id, name, and email. In addition to naming a field, you can provide a type that will determine how the data is encoded and sent over the wire – above we see an int32 type and a string type. Keywords for validation and structure are also provided (required and optional above), and fields are numbered, which aids in backward compatibility, which I’ll cover in more detail below.

The Protocol Buffers specification is implemented in various languages: Java, C, Go, etc. are all supported, and most modern languages have an implementation if you look around. Ruby is no exception and there are a few different Gems that can be used to encode and decode data using Protocol Buffers. What this means is that one spec can be used to transfer data between systems regardless of their implementation language.

For example, installing the ruby-protocol-buffers Ruby Gem installs a binary called ruby-protoc that can be used in combination with the main Protocol Buffers library (brew install protobuf on OSX) to automatically generate stub class files that are used to encode and decode your data for you. Running the binary against the proto file above yields the following Ruby class:

#!/usr/bin/env ruby  # Generated by the protocol buffer compiler. DO NOT EDIT!  require 'protocol_buffers'    # forward declarations  class Person < ::ProtocolBuffers::Message; end    class Person < ::ProtocolBuffers::Message    set_fully_qualified_name "Person"      required :int32, :id, 1    required :string, :name, 2    optional :string, :email, 3  end  

As you can see, by providing a schema, we now automatically get a class that can be used to encode and decode messages into Protocol Buffer format (inspect the code of the ProtocolBuffers::Message base class in the Gem for more details). Now that we’ve seen a bit of an overview, let’s dive in to the specifics a bit more as I try to convince you to consider taking a look at Protocol Buffers – here are five reasons to start.

Reason #1: Schemas Are Awesome

There is a certain painful irony to the fact that we carefully craft our data models inside our databases, maintain layers of code to keep these data models in check, and then allow all of that forethought to fly out the window when we want to send that data over the wire to another service. All too often we rely on inconsistent code at the boundaries between our systems that don’t enforce the structural components of our data that are so important. Encoding the semantics of your business objects once, in proto format, is enough to help ensure that the signal doesn’t get lost between applications, and that the boundaries you create enforce your business rules.

Reason #2: Backward Compatibility For Free

Numbered fields in proto definitions obviate the need for version checks which is one of the explicitly stated motivations for the design and implementation of Protocol Buffers. As the developer documentation states, the protocol was designed in part to avoid “ugly code” like this for checking protocol versions:

if (version == 3) {    ...  } else if (version > 4) {    if (version == 5) {      ...    }    ...  }  

With numbered fields, you never have to change the behavior of code going forward to maintain backward compatibility with older versions. As the documentation states, once Protocol Buffers were introduced:

“New fields could be easily introduced, and intermediate servers that didn’t need to inspect the data could simply parse it and pass through the data without needing to know about all the fields.”

Having deployed multiple JSON services that have suffered from problems relating to evolving schemas and backward compatibility, I am now a big believer in how numbered fields can prevent errors and make rolling out new features and services simpler.

Reason #3: Less Boilerplate Code

In addition to explicit version checks and the lack of backward compatibility, JSON endpoints in HTTP-based services typically rely on hand-written ad-hoc boilerplate code to handle the encoding and decoding of Ruby objects to and from JSON. Parser and Presenter classes often contain hidden business logic and expose the fragile nature of hand parsing each new data type when a stub class as generated by Protocol Buffers (that you generally never have to touch) can provide much of the same functionality without all of the headaches. As your schema evolves so too will your proto generated classes (once you regenerate them, admittedly), leaving more room for you to focus on the challenges of keeping your application going and building your product.

Reason #4: Validations and Extensibility

The required, optional, and repeated keywords in Protocol Buffers definitions are extremely powerful. They allow you to encode, at the schema level, the shape of your data structure, and the implementation details of how classes work in each language are handled for you. The Ruby protocol_buffers library will raise exceptions, for example, if you try to encode an object instance which does not have the required fields filled in. You can also always change a field from being required to being optional or vice-versa by simply rolling to a new numbered field for that value. Having this kind of flexibility encoded into the semantics of the serialization format is incredibly powerful.

Since you can also embed proto definitions inside others, you can also have generic Request and Response structures which allow for the transport of other data structures over the wire, creating an opportunity for truly flexible and safe data transfer between services. Database systems like Riak use protocol buffers to great effect – I recommend checking out their interface for some inspiration.

Reason #5: Easy Language Interoperability

Because Protocol Buffers are implemented in a variety of languages, they make interoperability between polyglot applications in your architecture that much simpler. If you’re introducing a new service with one in Java or Go, or even communicating with a backend written in Node, or Clojure, or Scala, you simply have to hand the proto file to the code generator written in the target language and you have some nice guarantees about the safety and interoperability between those architectures. The finer points of platform specific data types should be handled for you in the target language implementation, and you can get back to focusing on the hard parts of your problem instead of matching up fields and data types in your ad hoc JSON encoding and decoding schemes.

When Is JSON A Better Fit?

There do remain times when JSON is a better fit than something like Protocol Buffers, including situations where:

• You need or want data to be human readable
• Data from the service is directly consumed by a web browser
• Your server side application is written in JavaScript
• You aren’t prepared to tie the data model to a schema
• You don’t have the bandwidth to add another tool to your arsenal
• The operational burden of running a different kind of network service is too great

And probably lots more. In the end, as always, it’s very important to keep tradeoffs in mind and blindly choosing one technology over another won’t get you anywhere.

Conclusion

Protocol Buffers offer several compelling advantages over JSON for sending data over the wire between internal services. While not a wholesale replacement for JSON, especially for services which are directly consumed by a web browser, Protocol Buffers offers very real advantages not only in the ways outlined above, but also typically in terms of speed of encoding and decoding, size of the data on the wire, and more.

Editor’s Note: Our post today is from Peter Bell. Peter Bell is Founder and CTO of Speak Geek, a contract member of the GitHub training team, and trains and consults regularly on everything from JavaScript and Ruby development to devOps and NoSQL data stores.

When you start a new project, automated tests are a wonderful thing. You can run your comprehensive test suite in a couple of minutes and have real confidence when refactoring, knowing that your code has really good test coverage.

However, as you add more tests over time, the test suite invariably slows. And as it slows, it actually becomes less valuable — not more. Sure, it’s great to have good test coverage, but if your tests take more than about 5 minutes to run, your developers either won’t run them often, or will waste lots of time waiting for them to complete. By the time tests hit fifteen minutes, most devs will probably just rely on a CI server to let them know if they’ve broken the build. If your test suite exceeds half an hour, you’re probably going to have to break out your tests into levels and run them sequentially based on risk – making it more complex to manage and maintain, and substantially increasing the time between creating and noticing bugs, hampering flow for your developers and increasing debugging costs.

The question then is how to speed up your test suite. There are a several ways to approach the problem. A good starting point is to give your test suite a spring clean. Reduce the number of tests by rewriting those specific to particular stories as “user journeys.” A complex, multi-page registration feature might be broken down into a bunch of smaller user stories while being developed, but once it’s done you should be able to remove lots of the story-specific acceptance tests, replacing them with a handful of high level smoke tests for the entire registration flow, adding in some faster unit tests where required to keep the comprehensiveness of the test suite.

In general it’s also worth looking at your acceptance tests and seeing how many of them could be tested at a lower level without having to spin up the entire app, including the user interface and the database.

Consider breaking out your model logic and treating your active record models as lightweight Data Access Objects. One of my original concerns when moving to Rails was the coupling of data access and model logic and it’s nice to see a trend towards separating logic from database access. A great side effect is a huge improvement in the speed of your “unit” tests as, instead of being integration tests which depend on the database, they really will just test the functionality in the methods you’re writing.

It’s also worth thinking more generally about exactly what is being spun up every time you run a particular test. Do you really need to connect to an internal API or could you just stub or mock it out? Do you really need to create a complex hairball of properly configured objects to test a method or could you make your methods more functional, passing more information in explicitly rather than depending so heavily on local state? Moving to a more functional style of coding can simplify and speed up your tests while also making it easier to reason about your code and to refactor it over time.

Finally, it’s also worth looking for quick wins that allow you to run the tests you have more quickly. Spin up a bigger instance on EC2 or buy a faster test box and make sure to parallelize your tests so they can leverage multiple cores on developers laptops and, if necessary, run across multiple machines for CI.

If you want to ensure your tests are run frequently, you’ve got to keep them easy and fast to run. Hopefully, by using some of the practices above, you’ll be able to keep your tests fast enough that there’s no reason for your dev team not to run them regularly.

Code Climate is proud to announce a revolution in how software is built. To date, we’ve been providing developers actionable insights to help write maintainable code. The feedback, while useful, still requires care and collaboration to incorporate.

Today, we’re significantly streamlining the development process with our new Automated, 1-Click Refactoring feature.

Compared to the legacy approach of refactoring by hand, Automated Refactoring provides many advantages:

1. Saves lots of time by replacing nuanced discussion with a definitive solution
2. Appeases your CTO who has imposed an arbitrary, ill-defined requirement that all features must be “refactored” before they are shipped
3. Works when you’re too lazy to go talk to that guy who worked on that thing one time
4. Boosts your Git commit stats to get a bigger bonus at the end of the year
5. As a substitute for head desk-inducing conversations with difficult team members

Here’s what David Heinemeier Hansson (DHH) had to say after testing it out:

“When most programmers refactor, they produce shit. Code Climate’s automated refactoring tool produces code that is decidedly not shit.”

Automated Refactoring is in public beta for all Code Climate customers, and select Open Source repos. To see it in action, navigate to a class you’ve been struggling to refactor and click the magic wand:

Automated Refactoring is free during our public beta period. Pricing is yet to be determined, but will be significant.

Editor’s Note: Automated Refactoring is no longer available. After April 1st, we determined that the feature was too revolutionary, and it was taken down. For posterity, the revolution was captured on YouTube.

Today we’re proud to announce that we’re bringing Code Climate Quality more in line with how many of you work — you can now see how each change affects the quality and security of your code before it’s merged into main. You no longer have to wait until a security vulnerability is in main before eradicating it from your code base. If you use feature branches, you can ensure the quality is up to your team’s standards before merging. When reviewing a refactoring, you can easily visualize the impact to the codebase on a single page.

In short, you can merge with confidence. Create a Code Climate account start getting analysis of your Pull Requests right now.

There are three new, key features that work together to make this possible:

• GitHub Pull Request integration
• Compare view
• Branch analysis

Let’s look at each of them.

GitHub Pull Request integration

We’re happy to announce that Code Climate will automatically analyze your GitHub Pull Requests. Simply open a Pull Request and Code Climate will inform GitHub when our analysis is ready for you (if you’ve used or seen Travis CI’s build status on GitHub, we use the same exact mechanism):

Branch testing and Pull Request support is included in all our currently offered plans at the Team level and above. We’ll be rolling this feature out incrementally over the next few days, and we will let you know when it’s activated for your account.

Note: Due to some technical considerations around analyzing cross-repo PRs, supporting Pull Requests for our free-for-OSS repos will take some extra time but is high on our list.

Compare view

Once we’ve finished analyzing your pull request/branch, you’ll be able to see the results in our new Compare view. It’s a focused representation of the important changes in your branch. You’ll see the overall change to your repo’s GPA, how your classes or files grades have changed and where code smells have been fixed, introduced or gotten worse:

Branch analysis

Even if you don’t use GitHub Pull Requests, you can start getting feedback on your branches immediately. Start by clicking on the new “Branches” tab for each of your repositories:

Push the “Analyze” button for any branches you care about, briefly sit tight, and within a few minutes the “View Comparison” button will be available to send you to a Compare view.

First-class Pull Request support has been our most requested feature over the last year, and we’re thrilled to be delivering it to you. We’ve been using these new features while we’ve been building them (I know, very meta) and we’re really enjoying the quicker feedback loop.

We hope you enjoy it as much as we have. We’d love to hear what you think!