SE-Radio Episode 261: David Heinemeier Hansson on the State of Rails, Monoliths, and More

Filed in Episodes by on June 28, 2016 1 Comment

hansson100x125David Heinemeier Hansson, creator of the Ruby on Rails framework and a partner at the software development company Basecamp, talks to Stefan Tilkov about the state of Ruby on Rails and its suitability for long-term development. He addresses some of its common criticisms, such as perceived usefulness for only simple problems, claimed lack of scalability, and increasing complexity. David also talks about the downsides of building JavaScript-centric, “sophisticated” web UIs, and why he prefers well-structured, “majestic” monoliths to microservices.

Venue: Internet

Related Links

David’s Twitter: https://twitter.com/dhh

David’s website, http://david.heinemeierhansson.com/
Ruby on Rails website, http://rubyonrails.org/
Basecamp website, https://basecamp.com/
Rails 5.0 Beta announcement, http://weblog.rubyonrails.org/2015/12/18/Rails-5-0-beta1/
ActionCable (Websocket support for Rails), https://github.com/rails/rails/tree/master/actioncable
TurboLinks 5 application, https://github.com/turbolinks/turbolinks
The Majestic Monolith blog post, https://m.signalvnoise.com/the-majestic-monolith-29166d022228?gi=fd9f11cc91c9

 

 View Transcript

Transcript brought to you by innoQ

This is Software Engineering Radio, the podcast for professional developers, on the web at SE-Radio.net. SE-Radio brings you relevant and detailed discussions of software engineering topics at least once a month. SE-Radio is brought to you by IEEE Software Magazine, online at computer.org/software.

*   *   *

Stefan Tilkov:                        [00:00:36.10] Welcome, listeners, to a new episode of Software Engineering Radio. Today it’s my great pleasure to have David Heinemeier Hanson on the show. David is well-know to the community as the creator of Ruby on Rails. He’s also a partner at a company called Basecamp, (formally called 37signals) which is also very well-known.

Today we’re going to talk about Ruby on Rails a bit, but not only; we’ll try to get some of your opinions on other things out to the world, as well. Let’s jump right in! Welcome to the show, David.

David Heinemeier:           [00:01:09.04] Thank you very much for having me.

Stefan Tilkov:                        [00:01:12.09] I’ll assume that everybody in the world knows Ruby on Rails, but maybe I’m wrong. Why don’t you give us the up-to-date, one-paragraph introduction to what Rails actually is and what the current state of it is?

David Heinemeier:           [00:01:27.09] Ruby on Rails is a full-stack web development framework. That means that it includes just about everything you need to make great applications like GitHub, Basecamp, AirBnB, or many of the other applications that have been made over the years with Ruby on Rails. We are just about to release version five. The framework is about 12-13 years old. It got started in 2003 as an extraction from Basecamp, and I’ve been working on it ever since.

Stefan Tilkov:                        [00:01:58.19] That’s one of the more interesting facts about Ruby on Rails. It was extracted from a working application, as opposed to having been created and then looking for something you can build with it.

David Heinemeier:           [00:02:09.20] Absolutely. That’s why it has resonated so well. It’s not an invention out of thin air, it’s an extraction out from real problems that I was facing trying to use Ruby to build a web application. We have carried this ethos forward ever since – Rails is not backed by a large group of consultants who just travel around from shop to shop to help with problems and then try to extract generic patterns off that. The majority of people who work on Ruby on Rails are working on it from the vantage point of a specific application that they’re working on.

[00:02:53.09] Someone might be working at GitHub and they’re extracting the problems that they hit there into generic solutions we can put into Rails. Of course, I continue to work on Basecamp, so whenever I face new challenges, as has been the case with the most recent version of Basecamp version three, where we leveled up on real-time and went deep with websockets.

[00:03:14.05] Presto – that’s a new framework for Rails Action Cable and that ships as part of Rails 5. The reason that’s important is when you’re trying to solve other people’s problems, it’s very easy to go wrong, to imagine that they have certain problems in a certain way, when that’s simply not so. When you’re dealing with your own problems, at the very least you know you’re solving your own problems, so at the very least you’re making a solution that’s good for an audience of one. That is infinitely better than making something that’s created in a vacuum of that audience, a vacuum of direct experience.

[00:03:53.03] What has turned out is when I extract problems I hit at Basecamp, well, I’m not a beautiful, unique snowflake; most people, most of the time, are hitting the same problems I’m hitting when I’m creating a web application.

Stefan Tilkov:                        [00:04:09.17] One of the interesting things about Rails is that it has had a strong influence in many other frameworks. A lot of things that started out in Rails – at least in my experience – were seen appear in other frameworks, as well. What do you think is unique about Rails? What makes it different from all the others?

David Heinemeier:           [00:04:31.20] There are a lot of ideas that we’ve been honing over the years that have taken root in other frameworks, which is awesome. When I got started with Ruby on Rails, it was common that people would spend many hours or even days setting up the skeleton of their application. This was certainly true in the Java world, where you basically needed to do two days of XML setups before you even had the skeleton going to say “Hello, world.”

[00:04:57.20] Even in PHP or other more immediate frameworks, there was a lot of initial exercise. Rails came and said, “That’s silly.” When you start a new application, you should be able to run two commands and configure one database, and off to the races. Then you’re off to actually implement application logic, not plumbing. That was one of those early advances we got from Rails that really took hold.

Many of the other ideas from Rails that are less about just a single technique and more about the philosophy behind it have not been cloned. That’s why Rails continues to be so controversial for a lot of people. The pillars of the Rails doctrine, as I recently wrote up, includes a number of philosophical points where programmers to this day continue to disagree.

[00:05:59.10] The primary one that we’ve held dear in the Ruby on Rails community is this notion of focusing on programmer happiness. It’s not enough for an application to merely work; it’s not enough for it to be fast, it’s not enough to do all the other technical, objective parameters that we’ve been measuring software on for a long time. They also have to make the creators happy, because this has all sorts of wonderful spillover effects, most importantly on motivation. Motivated programmers are simply more proactive programmers. Having productive programmers is one of the key challenges of the whole software movement. Fighting back and pushing back against needing ever bigger teams requires an ever-increasing amount of productivity from a single programmer.

[00:06:49.15] We need to constantly make the frameworks and the libraries in the Ruby on Rails world better and more productive, in order to counteract the spiraling ambitions that people have for, “Oh, an application is supposed to do all these things nowadays.” The cornerstone of that is to ensure that a programmer can sit in front of a computer for eight hours a days – even if there’s not eight hours of pure productivity – and be happy most of the time. That happiness is influenced strongly by tools that he/she chooses to use.

[00:07:28.04] Ruby on Rails is focused on what for a lot of people is considered silly things. Esthetics is one thing. How the code looks, and coming up with subtle, nuanced variations of methods in order to make the code prettier, clearer, shorter, more concise. One of the early revelations of this for me was the keyword ‘unless’. When I first hit Ruby, that was one of those things where it just went, “Wow!” and blew my mind. ‘Unless’ is the same as ‘if not’. It’s just a more concise and prettier way of saying the same thing. So is the fact that you can place ‘unless’ either before the statement or after the statement in Ruby.

You can have these subtle shifts in intonation, where you put the pressure in a sentence of Ruby code, that it’s just an utter joy to work with. It’s this expressiveness that remains both so dear to many Rubyists’ hearts, and at the same time it’s continuously critiqued by other languages and communities.

[00:08:52.01] A clear counter is the Python model, of preferably one way and only one way to do things. Ruby, in many ways, is a complete opposite. Preferably in many ways, as many ways as possible. In many of the same ways that the English language is not 500 different words, it’s in excess of 200,000 words. That is a lot of its beauty. A lot of its beauty is its expressiveness in all these different shades.

This is one of the things that I think about, particularly because I’m a Dane. The Danish language is only used by five million people, and perhaps by consequence it’s a relatively small language, compared to English. It has about half the words, if you don’t count all the put-together words that you have in Danish.

[00:09:46.24] One of the examples I often use is the word ‘light’, or ‘candle’. In Danish, that’s the same word, ‘lys’. Just one word, for both ‘candle’ and ‘light’. That to me is an example of not a very expressive language. English is simply a better language, because it has this way of distinguishing these subtle differences. They don’t rely merely just on context to derive what the true meaning is; you have more words to do it with.

That’s a really long way of saying that in Ruby on Rails we continue to optimize for programmer happiness to the point that we are willing to trade off other things. That is where the rubber really hits the road.

[00:10:39.14] A lot of other communities may say that they’re also about programmer happiness, but they’re not willing to give up other valuable things to have it. They’re not willing, for example, to give up performance, or they’re not willing to give up type safety, or other impediments to programmer happiness in terms of being drags on programmer productivity. In Ruby on Rails, we’re willing to trade those things. I will make this program go slower, such that when I look at the code, I am happier. That remains a deeply controversial statement that I think a lot of programmers would not sign up to. That’s just one example.

[00:11:22.19] We have several examples of major philosophical planks in the Ruby on Rails community that was part of the foundation why I made the framework in the first place, because I had these ideas that were different from the ideas that were being expressed at the time, and continue to be expressed in other environments. At the same time, they remain controversial such that Rails and Ruby remain unique; even though people might be capable of cloning and following these things, we just simply disagree and end up on different paths, which is wonderful. That is one of the beauties of programming – especially programming for the web – that we can disagree and we can have this diversity of implementation and of language. We don’t all have to speak one language. That is the travesty that’s reserved for developing on native platforms.

Stefan Tilkov:                        [00:12:16.17] Let me try to face you with a few of the criticisms that I typically hear when people talk about Ruby on Rails. Interestingly enough, when preparing this podcast I had some very controversial input from my co-hosts, who wanted me to ask you a lot of things. Let me address some of them.

One of them you already mentioned, which is the speed and performance, and maybe related to the scalability of the whole thing. In your opinion obviously it’s worth it, but is it maybe only worth sacrificing those things if you’re not building serious things? As soon as you’ve built something that you really want to make money with or turn into a sustainable business, don’t you just have to switch to something different and give up a little of the happiness, aesthetics and the other qualities, in turn for having something that’s more efficient, faster and scales better?

David Heinemeier:           [00:13:06.18] I’ll give you the summary first. No. Now I’ll give you the longer answer. When I started working with Ruby 12-13 years ago, do you think Ruby was faster or slower than it is today? Do you think computing power was more expensive or less expensive than it is today? Do you think that programmer talent was more or less expensive than it is today?

Stefan Tilkov:                        [00:13:30.23] Do I have to answer that, or…?

David Heinemeier:           [00:13:30.27] I’ll give you the answers for that. Ruby was much slower 12 years ago; computing power was significantly more expensive, and programmers were much cheaper. Even under those parameters, Basecamp – the app that I developed with Ruby on Rails – has been tremendously profitable and successful, even though it’s served millions and millions of customers and even though it’s built on Ruby on Rails. If Basecamp could be a success 12 years ago, given those differences in the environment, think about how much better things are today. Think about how many more environments were the cost of Ruby on Rails in terms of slower computing is dramatically offset by the happiness and productivity that flows into the programmers, the fact that you need fewer of them, the fact that the ones that you have are so much happier, the fact that you simply get more stuff done quicker.

[00:14:43.01] That is not to say that there aren’t domains where Ruby simply isn’t fast enough; there absolutely is. At certain points, for example if you’re trying to process — I don’t know how many tweets they do a second right now… Let’s say they do 100,000 tweets a second at peak – I don’t know, that could be off by a factor of ten, but if you’re processing simply incoming pieces of 140 characters and you need a pipeline for handling that and shoving that into a database, yes, at that point Ruby might very well not be the best choice there. You’re simply at a scale where the economics do start to matter.
Although, what’s funny is if you look at the balance sheet of quarterly reports from even a Twitter – what do you think they’re spending most of their money on? Do you think they’re spending most of the money on hardware, or on San Francisco programmer salaries? I’ll spoil the surprise and say it’s the latter.

[00:15:41.16] Even in that case, the whole purpose of exploratory software – which Twitter certainly falls under – is that first you have to figure out what it is that you want to build. While you’re figuring out what it is that you want to build and how you want to build it, doing so in clay is a lot easier than doing so in iron. Iron is a very inflexible material, that has to be treated in all sorts of interesting ways, with dangerous materials and techniques to bend it. Clay – you just bend it with your hands.

[00:16:18.06] Ruby is like clay; many other languages that have higher performance are more like iron. That is not to say that at some you want to replace the clay, once you figured out what it is that you want to build with an iron structure. What I find interesting is that it is vastly, hilariously overrated how often that conversion needs to happen. There is just a small handful of businesses that truly need to make that conversion. The number of massive businesses (from Shopify, to AirBnB, to GitHub, to Basecamp and whatever else) that continue to work and thrive by building their applications in clay is just overwhelming, to the point that the argument at this point is ridiculous. Even posing the question comes off as slightly ignorant.

[00:17:17.10] It was much easier to pose this question in 2004, when we didn’t have this legion of examples of billion-dollar businesses (even though that’s in my book the lone qualification of success). To the people who ask questions like this, that is the kind of arguments that they need to hear back. You could ask this question in 2004 and legitimately say, “Okay, let’s have a discussion about this.” Asking that question in 2016 is simply just ignorant. It’s ignorant in the face of overwhelming evidence to the contrary.

[00:17:55.01] So yes, Ruby remains low in performance in many of the same ways that you could say Java is low in performance to Assembly. You could write web applications in Assembly; it is technically possible to do that, but it would be incredibly stupid because the benefits you get back simply are not worth it.

When you look at performance, for example, you get to a point where things are simply good enough. For most web applications, if you can just get your server-side response down to around 100 milliseconds as an average, you’re far into the land of good enough, that the responsiveness of your application is no longer governed by how quickly your server-side can respond, it’s governed by all the other factors of how the internet works, how HTTP works, how caching works, the way you use assets, whether you use web fonts, how much Javascript you have and so on. If you look at the size of a whole request, the server-side part of that becomes a tiny little vanishing part.

[00:19:04.07] The benefit of moving from a one-second server-side response time to 300 milliseconds server-side response time – that is significant. There is benefit there absolutely, and you should try to get to that point. Once you hit that point, the benefit of moving from 300 milliseconds to 30 milliseconds is orders of magnitude less. It’s the same kind of parallel I make when I talk about business and success. The difference in someone’s life between having zero dollars and one million dollars – phenomenal; very large difference. It does all sorts of good things to your daily life. The difference between having one million and two million dollars? Vanishingly small. Many, many orders of magnitude less. Even though it’s the same absolute improvement – one million dollars one way, one million dollars the other way. The same is true for performance.

[00:20:04.28] Once you get to a state of good enough, and for web applications I’d generally say below 300 milliseconds, and if you can hit 100 milliseconds – which is where we are for a lot of requests for Basecamp 3 – you’re in the golden land. Spending valuable, precious programming time to suffer in iron-core environments just to make that a little better is such an utter waste of time. That is where the true broken economics exist.

Stefan Tilkov:                        [00:20:35.24] We could argue that it’s not only performance, but also the amount of resources that you need to get a certain [unintelligible 00:20:40.07]. That is very different. If you can get by with a tenth of the hardware, then you do save money. Admittedly, that may not be the major point in your cost structure, because developer time may be way more expensive (and very often is), but there is a difference in having to use ten servers versus a hundred servers, isn’t there?

David Heinemeier:           [00:21:06.19] There is. That is fair; at some scales, for some companies, as the example we just talked about, like processing the hundred thousand tweets, the economics can’t flip where it is worth making things in iron. But for the 99,9%, it’s simply not so. For the 99%, every single time you can trade a more inefficient programming language to get more productivity back, it’s worth it. Programming salaries are only going one way – up. Computing costs are only going one way – down. If you want to be on the right side of trends in history, then that’s where you have to place your bets.

[00:21:47.21] It’s sometimes hard for people that have been in the industry for a long time to recognize that the sands have shifted. In 1995 this was not true. You could not just use Ruby in the same ways that you’re using Ruby today. It would simply be too expensive. The ratio between cost of servers versus cost of programmers was completely different in 1995. Now we’re 20 years later and it’s very different. The problem is that the people who were around in 1995, those economics are imprinted on their retina, they’re imprinted in their skull, in many of the same ways that the grandparents that lived through major economic crises, either the World War II, depressions and so on, sometimes it’s the silliest things that are just imprinted into their skull.

[00:22:37.28] For my grandparents it was this notion that if you wanted to have a really luxurious breakfast, you would just layer on the butter. Because you couldn’t get butter in 1944 in Denmark. So butter became this symbol of luxury, this symbol of “Now we’re really doing well. Let’s really have a feast and let’s layer up these croissants” which is so much butter that you’re going to gag. They didn’t realize that the economic model had changed, that butter wasn’t a precious resource in 1986 when I was forced to eat fully buttered-up croissants. The exact same thing is true for programmers who have lived through the depression of computing, [unintelligible 00:23:22.28] early ages were. It’s incredibly hard to break out of that. Those habits lay deep in our operating system as humans, and rewriting that code – not a small task.

Stefan Tilkov:                        [00:23:37.27] Let me ask one final question regarding Rails, because I do want to talk about other topics, as well. This is a criticism that I hear from people who do use Rails, not from those who have never used it, which is that it has become more and more complicated over the years. It has accumulated more and more things that are included in [unintelligible 00:23:57.09], so it has become a lot more complex than it was in the beginning, it has become way harder to understand, so that many people who use it understand but a fraction of it, and they don’t really know what they’re doing with this very powerful thing. Do you see that as a problem, or do you think that’s an unjustified, unsubstantiated criticism?

David Heinemeier:           [00:24:21.15] I think it’s a feeling. But let me unpack the feeling, because I think the feeling has root in some factors that are actually not that correlated. First, yes, Rails is much larger now than it was ten years ago, but if you look at trying to do the same thing – let’s say creating a simple controller that iterates over a table view of items in a database. When you look at a single use case like that, that’s my measure of success. Is Rails more complicated at solving that use case than it was ten years ago? The answer is absolutely not. Far less code is needed. The code that is needed is far clearer. The helpers that we have to do it are much better.

[00:25:07.22] If you compare use case to use case, I think Rails has jumped leaps and bounds. The difference is that the number of use cases Rails was willing to and able to solve in 2005 is far smaller than the number of use cases it’s willing to solve in 2016. In 2005, if you wanted to protect yourself against cross-site scripting attacks or anything like that, you’re on your own. You research that problem from scratch. You learn everything there is to learn about it and then you build your own solution.

[00:25:44.20] Rails, in that sense, is simple – it simply doesn’t solve the problem. That’s very simple, right? If you just say, “Hey, you have 20 problems you need to solve making your web application. We are the simple framework – we solve two of those. Good luck with the other 18.” That is a simple framework, and there are many frameworks that take that approach.

That is the problem with the word ‘simplicity’. It’s such a devious term, because people embow it with this notion of goodness, when in my book, if I have to solve 20 problems and my toolbox only gives me assistance in solving two of them, that’s not simpler. My final application is far more complex, because I have to research and figure out all the 18 other problems that I needed to solve in-depth.

[00:26:34.13] Let’s say Rails 2005 solved five of your 20 problems. Rails today solves 18 of your 20 problems. That means that we have 13 elements of Rails that now need to be built, and you could say ‘understood’ if you want to understand the whole thing. That’s where I go, “Why do you need to understand the whole thing?”  Most of the time, the purpose of building things into a framework or toolbox is to alleviate the programmer from having to understand problems from first principle. Because if you have to understand every single problem of a modern web application from first principle before you can do anything – well, off to school for you for the next six years. No one is willing to do that. No one is willing to make that trade.

[00:27:23.13] We all rely on the giants on the giants that went before us, and we all built on the things that we have, and we’re better off before it, but yes, it does mean that there are more things. To solve 18 out of 20 use cases, you simply need more tools in your box. The individual tools need not be any more complex. In fact, they might very well be a lot better, and I contend that they most certainly are.

[00:27:47.20] If you look at the reasoned major expansions in scope and breadth of the Rails framework, they come from – let’s take just the two most recent frameworks – Active Job, which provides you with an abstraction and an AC setup for using asynchronous job servers and running logic asynchronously, right? The reason why I know that it’s so much better than what went before it is because obviously I did Jobs before I had Active Job. The amount of code I was able to rip out of Basecamp when we plugged in Active Job – substantial. The amount of complexity in setup and decisions about, “Oh, should I pick this thing? Should I put the other thing that we alleviated by building a framework?” Awesome!

[00:28:32.22] The most recent example – Action Cable… Has anyone tried to do websockets from scratch, from first principles, trying to understand the specs that are involved and then try to do it? HTML and HTTP/1 and so forth – you can understand those things easily from first principle in not a very long time. Understand websockets from first principles and digging through all the underlying specs and incompatibilities and so forth – major task, which is why most people didn’t know how to use it.

[00:29:08.12] Again, they have simplicity. If I don’t choose to take advantage of the real-time web through websockets, my application is simpler, my framework can be simpler. That’s not the notion that we’re interested in with Ruby on Rails. I am not afraid of expanding the scope and solving more out of the 20 use cases that most people need to solve for most applications, most of the time. That was always the governing principle for Ruby on Rails: “Hey, let’s share all the solutions that most of us run into most of the time.” As the web has become more complicated in some senses, there are more of those solutions. The fact that Rails facilitates the sharing of those is exactly why we’re still where we are today, that that is incredibly valuable.

[00:29:54.12] It’s also, again, one of those controversial things. It goes counter to a lot of near and dear held beliefs about how software should be made by most programmers, which is subscribing to things like the UNIX philosophy. “Well, don’t make frameworks. Just make small, pointed tools and the users can figure out how to put that together themselves.” Rails went in the opposite direction and said, “No, we’re going to make an integrated system where all the parts are designed not to be used alone, in isolation, but in fact to be used together, in combination.” Designing tools with the knowledge of the other tools that it’s going to be used with enables us to go in a much higher level of integration and smoothness of the use and all these other benefits that we get from it.

[00:30:42.28] It’s still a deeply controversial aspect, and even within people who do otherwise like Ruby on Rails, this pull towards the (in my opinion) shallow definition of simplicity as simply less stuff, solving less use cases, remains a powerful one. It’s not until you unpack that emotion — because I can have that emotion at times too, where I look at the Rails codebase and I go, “Wow, that’s a lot of things.” Then I usually rifle through the things and see, “Hey, can I find some stuff here that I could kick out?” Sometimes we can. We do occasionally kick things out that at one point were a part of problems that most people need to solve most of the time but they no longer are, so we kick them out. XML processing used to be a bigger deal, now it’s not, so Rails does less to accommodate that and more to accommodate JSON processing, for example.

[00:31:35.10] Things do evolve over time, and we’ll not make any excuses for solving more of the 20 use cases that most people need to do most of the time.

 

[00:31:47.01] Commercial Break [00:32:24.23]

 

Stefan Tilkov:                        [00:32:24.23] You mentioned that you add stuff to Rails and that you embrace modern things. In some aspects you remain quite conservative, some people would say. One of those examples is that Rails is still very much focused on the classical, service-side HTML rendering model, as opposed to how many people nowadays believe – I’m showing my bias here, I probably shouldn’t be doing that – that these days the server-side should be simple and there should be more logic in the client, which is why a lot of single page application frameworks built in Javascript are appearing. Some people use Rails, I’m aware of that, to just build a server-side API, but I believe that’s not something you would recommend people to do. Can you elaborate on your view on that?

David Heinemeier:           [00:33:16.13] Sure. One of the features we’re putting in Rails 5 is actually an API mode, where the Rails stack gets paired down just for the purpose of implementing the main model and using all the wonderful tools we have from Active Record and Active Job and all the other backing to build a great domain model, but then exposing that domain model purely through JSON replies.

We’re accommodating that approach in Rails 5, and that’s one of those areas where I’ve come to realize that Rails today is such a big tent that we can very well accommodate and collaborate with people who have distinctly different ideas of how the final application should be built in most cases, and benefit from it.

[00:34:00.00] If you want to use Amber or React or whatever else you have by the time this podcast comes out – there’s probably four new Javascript frameworks that you’ve had to learn in the interim, and if you want to use any of them with Rails, Rails will welcome you with open arms and it will be an awesome experience for building your domain model.

One of the fallacies that often go into this is that if I’m building most of my user interface on the client side, then I don’t need something like Rails, and I’d go like, “Really?” If you look at the Rails set of use cases that it solves, how many of those use cases are related to HTML? Very little of it. Very little code in Rails is focused around HTML.

[00:34:46.11] We have helpers, we have active view and we do have some stuff around it, but the vast majority are about all the other elements of connecting things to the internet and building up at the main model. So all the benefit of that is still present, whether you choose to build your HTML on the server side or you choose to do it on the client side.

All that being said, I think this current trend and chase of the single page application built through client-side MVC for a lot of applications a lot of the time is a blind alley. It’s a blind alley in the same way that this industry has changed other blind alleys in the past. Everything from Flash, to Silverlight, to Java… I understand that these things are different, and I also appreciate and sympathize far more with people who want to use the native constructs of the web (Javascript, HTML and so on) and the browser from those early attempts, but I consider it still to be reminiscent…

[00:35:47.27] Some of that reminiscence comes from the arguments that are being used to justify the client-side MVC move. A lot of those are things like ‘sophisticated’, ‘modern’, ‘ambitious’. These labels that we apply to the UI and the UX and then we infer that those things are good… In fact, a lot of the times when I encounter UI’s that can be described as ‘sophisticated’, I go “This is a piece of crap.” Things don’t work properly, or it’s too ornated and too complicated for its own good. So much of the information systems that we build today are far better off when we’re constricted and restricted from using all this ‘sophistication’.

[00:36:40.20] It reminds me of the same arguments that we used for Flash, where you’d go like, “Well, Flash doesn’t have these native elements that HTML has for form inputs, but look at all this other stuff we could do! You could make your own! You are not constrained in any way from these shackles of HTML”, and I go, “I want the shackles. Put on the handcuffs. Please!” HTML and the restricted expressiveness is its greatest benefit, its greatest gift to the user interfaces of the world, constricting people from people being able to just dream in Photoshop.

[00:37:13.13] Again, Client MVC is not all that, I’m just saying that there’s a similarity between the arguments used in the two camps, that I find both striking and disconcerting. Most of the time you do not need the sophistication, and giving up the sophistication means gaining something else – productivity. Building these elaborate Client MVC structures is complicated work. I don’t think even the most ardent proponents in that camp would question that. Getting this stuff set up is complicated.

Stefan Tilkov:                        [00:37:51.01] I believe they would say it’s more productive. I don’t agree with them, but I believe it’s a common feeling that it’s more productive if you have something like, let’s say, Angular; it’s easier to build something that is, as you say, fancy and sophisticated if you do it on the client side than if you have to do it on the server side. You obviously don’t agree with that, right?

David Heinemeier:           [00:38:08.18] That’s perhaps the argument of where you place the sophistication. Yes, if you take as a given that the user interface has to be this “sophisticated monstrosity”, then yes, perhaps you could make the argument that it is simpler to build the monstrosity this way. But you would also just say, “Hey, let’s not build a monstrosity”, and if you reconfigure the problem definition that the monstrosity is not a given, then I think the productivity argument quickly deflates and comes back to earth.

[00:38:37.05] As in, “Hey, can we solve this underlying problem in a far simpler way, where we don’t need such a fancy, sophisticated user interface for it?” If that’s the scope of your problem definition, then it becomes far simpler. I don’t think there are any people in the client-side MVC camp who will look at a very plain web page that maybe has a form that can be submitted and say, “Oh, that would be much simpler if we added 15 layers or Javascript gunk on top of it.”

At some root level there is an agreement that the client-side MVC complicates matters, but you could say the disagreement is whether that’s worth it or not, whether we want the same things. In many cases I’d just say we don’t want the same things.

[00:39:27.20] The user interfaces that we’re building at Basecamp are far better suited in the majority of cases from not being sophisticated. I usually look at the things that we do build that are sophisticated and say, “I wish we hadn’t gone so sophisticated.”

Again, this is always a matter of specifics if you look at some of the things we’ve done at Basecamp, for example, we’ve recently spent a whole year building a client-side MVC model for a brand new text editor called Trix. In that case, absolutely that’s the right thing to do. You have to build a very complicated, sophisticated model on the client side to solve that problem, because the fidelity you need to have responsiveness at the key press is very different than the responsiveness you need at the click-a-link press.

[00:40:20.27] If you’re typing things and your text editor can’t keep up with you, it’s not going to be a text editor most people are going to be happy to use. But if you’re clicking a link and that link takes 150 milliseconds to render, things are fine.

You have to look at the individual problems you’re trying to solve and partition things accordingly. What I’m saying is that most applications most of the time when it comes to information systems, the vast majority of that application does not need the sophistication and complexity of the client-side MVC. Reserve that for when you truly need it, which is much the same argument we’re making on native.

[00:40:59.03] We have native apps for four different platforms. If you just take the native mobile apps – take the iOS app. The iOS app is probably 95% web-based (running on Turbolinks 5) and then we have 5% where the interaction speed and the fidelity really does matter, to the point where when we level up to native, that’s a big win. Then we take that win, and you say, “Oh, for this 5% of the application let’s do it native”, which again boils back to the earliest discussion we had, that Ruby is not fast enough. If you hit cases where Ruby is not fast enough, it’s usually at the 5%, or more likely the 1%. Just take that 1% and rewrite in something faster. Reap the benefits of productivity for the other 95% or 99%. That’s a completely valid way of going for it.

[00:41:53.08] The previous version of Basecamp that we had had a very sophisticated client-side MVC-driven calendar system. There you had the same thing – there was a very high level of fidelity needed to make this field good, so we took that one part of the application and made it client-side MVC, and we continued on with the productivity gains of the rest of the application using far simpler techniques.

In Danish we have this saying of, “Don’t shoot sparrows with cannons.” I don’t know if that’s also an English saying…

Stefan Tilkov:                        [00:42:23.27] It’s available in German, at least…

David Heinemeier:           [00:42:25.12] Oh, it is? Okay. I love that saying. Yes, you could absolutely shoot a sparrow with a cannon, but it’s just not a good use of cannon’s time, and it’s overkill for the sparrow. There are simpler ways of getting the solution that you want, which is no sparrows on your front porch. You don’t have to bring out a cannon to deal with that problem. In fact, it’s more likely to just put holes everywhere. That is the analogy I will choose to go with for this argument.

Stefan Tilkov:                        [00:42:53.07] You briefly mentioned Turbolinks, and that plays an important role in why you were able to stick with a server-side HTML model for much of your mobile applications. Can you go into a little technical detail on that?

David Heinemeier:           [00:43:07.04] Yes, absolutely. When we started working on the second version of Basecamp, which was in 2012, the client-side development zone was already taking off and people were enamored by the benefits of not having to reload your whole thing between each page request and the benefits that that gave in terms of rendering time, responsiveness and so on. Ajax came before that, but then got truly on steroids when we said, “Hey, what if the whole thing was just Ajax?”

I looked at all that and went, “Hm… I wonder if there’s a way where we can get the vast majority of the benefit, this feeling of responsiveness, without giving up the simplicity of the request-response model and of simply generating the entire HTML document on the server side, because there’s such great value in that?”

[00:43:56.21] I started exploring ways, and Pjax was an early approach to doing something like this. This is something pioneered at GitHub, and they went down on a little bit different path where they were trying to just pick out some partials and went “Oh, if you just had to replace this one part, the page should be faster.” I think the Turbolink’s realization was, “Hey, that’s actually not the problem.” Generating the whole HTML document on server-side is not where the speed benefit is going to be hidden, especially not if you’re using traditional use of caching through [00:44:25.01] strategies, and so forth.

Turbolinks gave us this notion that we could turn what was essentially a CTI model, where every click would throw away the entire client-side UI and require it to be compiled again fresh, and go to a persistent process model. If your listeners are old enough to remember, this is also something we went through on the server side.

[00:44:53.11] In the old days, CTI had this model. It would spin off an entire process, compile the code – usually to Perl – process the one request and throw the whole damn process away, then do the next thing again the next time. That works fine if you have a very small core of things you want to say. If you just want to say “Hello, world!” or put in the current time, that’s fine; you can throw away the whole process. The more sophisticated you get, the larger your structures get and the larger your processes get, the worse that works. It didn’t take that long for us to figure out that, “Hey, maybe we shouldn’t be throwing away the entire process between every request. What if we just reuse the process? Then we don’t have to compile the whole system again, and there are all these other benefits from it.”

[00:45:42.02] Why is that a controversial notion on the client-side? Because that’s basically the model that we’ve just taken straight over. The controversial point comes in in that people want it both ways, or either way… Either they want – or at least think they want – to do a full-on single page app with a heavy client-side MVC framework and then they accept that they have everything that way, or they want the jQuery soup, where they can just put in any jQuery plugin and everything will just work automatically. Well, the jQuery soup was designed for a plate of CTI, for a plate of disposable processes.

With Turbolinks we’re saying, “We want the persistent processes, we want a persistent Javascript and CSS engine running between requests – and there is a large performance benefit from that. It does mean we have to make some concessions in how we write the rest of the Javascript [unintelligible 00:46:40.22]”, and I think that that would trip people up, at least in the beginning.

[00:46:45.16] I see more and more people coming around to it. The benefits of the Turbolink model, the notion that you can retain full HTML generation on the client side, you can retain the full request-response model has such an appeal on grounds of simplicity when you combine it with things like the native apps that you otherwise would have to build. This is going to get more interest and less knee-jerk reactions once people realize, “Hey, wait a minute… If I have to build a modern app today, do I have to build both a web application API, something on the client side? One native application, from scratch, 100% on Android, one native application 100% on iPhone? Oh, what do I do about the Mac? What do I do about the Windows? All of a sudden, I need 50 people just to make an MVP.” The productivity benefits are simply enormous, and the fidelity that you give up in most cases is negligible, and it’s not something that users cherish.

[00:47:42.29] If you look at the scores, just for the Basecamp apps that are built on Turbolinks 5 with this model I think we have a 4.5 rating and a 4 rating, or something like that – better ratings than lots and lots of fully native apps. So this is not an impediment – at least in a wide variety of information systems – to giving a great user experience. You’re giving up something that most users most of the time wouldn’t miss.

Stefan Tilkov:                        [00:48:09.28] One of the things that has become a recurring theme, so that we have to talk about it now on every SE-Radio episode, is microservices. Microservices and monoliths is a topic that you have a strong opinion on, as you have with most things. Can you elaborate a bit on your views on that and tell us how Basecamp is built?

David Heinemeier:           [00:48:30.09] Yes. I recently posted a full treatise on this, coming off the last keynote I gave at the RailsConf, which was centered around the notion of “The majestic monolith.” Picking the monolithic pattern with pride and intention. Not stumbling into it, not accidentally falling into its pits, but putting down the drawbridge and walking in with your head held high. That is what Basecamp is. Basecamp is a majestic monolith. We have explicitly and intentionally chosen to implement the vast majority of the application within one large application. Then we have these satellites that revolve around it and interface with it. That’s what we’ve just talked about. Even the native apps are 95% just displaying what the monolith sent down the wire. Then you have the 5% of the native specialization on top of that, but it goes back to the same thing.

[00:49:32.15] It goes back to this notion of simplicity, avoiding over-specialization, avoiding patterns that just make things harder for you at a time where you do not need those things. Microservices and service-oriented architectures in general can be very valid and decent pattern to follow once you have no other choice. If you have a massive organization like Amazon, it makes total sense to treat one department as though it was just another place on the internet from another department. But it has all sorts of costs.

In Amazon’s case those costs are worth it, because they don’t feel like they can manage having, let’s say, 5,000 programmers working on the same codebase at the same time. A little asterisk on that is that what’s actually interesting is that both Google and Facebook are big proponents of the monolithic pattern too, and have these truly humongous repos and codebases that they work on.

[00:50:33.29] I’ll put that aside a little bit, because I haven’t explored it fully. I don’t know whether they’ve just stumbled into that or they’ve chosen it by design. I have a tendency to think that there are enough smart people at both places that if they’ve ended up here and stayed here, perhaps it’s very well by design and not just through accident.

Let’s put that aside for a bit and focus on the fact that if you are a smaller team where it is possible for every programmer to understand the vast majority of your application and keep it in their head at the same time, then my services are generally a terrible idea. They are just ways of making things more complicated and more convoluted, and introducing different failure states and failure modes where things can break down.

[00:51:16.04] The difference between making a method call and making an HTTP is huge. You need all sorts of provisions around how that collaboration happens that you simply do not need when you’re just making a method call. This goes back to the Martin Fowler law of distributed computing, which I basically paraphrase as “The first rule of distributed computing is don’t distributed your computing.” You don’t distribute your computing until you absolutely have to, because not doing so is so much simpler. It is much the same argument that we’ve just had about client-side MVC. Client-side MVC is basically distributed computing, or you should say “more distributed computing”, where the client takes a bigger slice of the domain from the server. What I’m advocating is as much as we can get away with, go for the monolithic system where you have a single system that’s responsible for everything. The benefits are enormous.

[00:52:15.28] Some of those benefits are possible because of using a specific tool set. Basecamp 3 is currently 25,000 lines of code. If I had to write that same application in Java, perhaps it would be 250,000 lines of code. Can anyone keep 250,000 lines of code in their head at the same time and make reasonable progress on it? Perhaps not. Perhaps in that case you do have to chop it off. When you’re using highly productive and highly expressive environments, the time at which you do have to do that chop-off comes so much later, and it comes at such a larger scale.

That’s what really gets my goat, when someone is like, “We’re a team of three people and we have these four microservices…” I just go like, “Oh, Jesus Christ! You have the wrong model, kiddo. Your role model is the Amazons in the world, when you should be looking at what are the other companies that have three people doing.” Don’t inflict yourself with this extra pain just because that makes you look big and sophisticated.

[00:53:21.22] Unfortunately, that’s a siren song for programmers. The more complicated and the more sophisticated you can sound talking about it, that’s very often correlated to how smart you think you are.

Stefan Tilkov:                        [00:53:33.12] Let me challenge that a bit for Basecamp itself. Maybe Basecamp is so simple that it fits in a monolith. I know this is not true, but I have to ask you anyway – how complicated is Basecamp?

David Heinemeier:           [00:53:45.27] That’s a great question, because it’s often brought up, and we’ve had this argument since the beginning [unintelligible 00:53:50.19] Basecamp is just a glorified to-do list. Yes, perhaps in some ways it is, and by the way, what’s wrong with that? Basecamp has over 200 different screens. We have almost 1,000 different controller actions. If you look at it in those terms, perhaps Basecamp is not so small after all.

I just ran Rails Stats here… We have 205 controllers, we have just about 200 models as well. No, this is not the biggest system in the world; there are tons of systems that are far bigger, but it’s big enough. It’s big enough that it drives this organization of 50 people and millions of workers. Tons and tons of applications are far smaller than this. Far smaller applications than Basecamp have been chopped up into little microservices and have been worse off for it.

Stefan Tilkov:                        [00:54:42.04] One of the things that you mentioned that seems to be a recurring theme is this notion of keeping things simple, as simple as they can possibly be. I believe that extends to your views on business and business strategy, as well. How much of a similarity do you see between the way you look at code and the way you look at the business side of things?

David Heinemeier:           [00:55:04.23] That’s a great point, because I think there are tons of parallels to draw here. The reason I usually don’t grab the word simplicity is that you can have something that’s simple when it’s solving 2 out of 20 use cases; I would argue that it’s simpler when you solve 18 out of 20 use cases. There’s a fundamental disagreement on the word, which makes the word not very helpful, because everyone will use it to describe what they do. I have never talked to a programmer who said, “Oh, I write complicated code.” No one says that. Everyone says, “I write simple code.” Even people writing things in Vanilla Java [unintelligible 00:55:43.15] “Oh, this is really simple”, and I just go like, “We do not share a common sense of what that word means, obviously.”

[00:55:56.10] Many of the ideas of how to build and how to keep solving a wide array of use cases with the least amount novel effort goes very well into the business. I consider Basecamp as a business, as a system that we should continue to refactor and hack on, and that we continue to have to deal with tipping points. Basecamp today at close to 50 people is a different company than it was at 15 people. Some things just have tipping points that need to be addressed in a variety of ways. That’s the same thing with code.

If we had started out designing Basecamp for how the company should run when it was 50 people, it would not have been nearly as nice of a company when it was five people. You have to pick something that’s appropriate for your scale, and that’s where I reject a lot of the business advice that’s out there. It’s drawn from these hyper cases. They’re drawn from, “Hey, what’s Amazon doing? What’s Facebook doing? What’s Google doing?” and just going like, “That could not be more irrelevant, in many cases, to these small companies that are trying to learn from giants.” Often times the patterns that you need to apply at 10,000 people are 180 degrees the opposite pattern of what you should apply at 10-15 people.

[00:57:23.11] That lesson of context which, funny enough, was part of the pattern language and the way patterns normally are described in the technical community is very often lost. I’m guilty of that at times, too. I assume that the context that I speak of is the things that I know of. When someone reads something I write, they have to just implicitly say, “This is a context that probably applies for a company between 5 and 50 people”, because that is what I have personal experience with. This might very well apply differently for a company of 10,000, or it might also very well not, but at least there’s more grains of salt to be thrown at the problem there.

Stefan Tilkov:                        [00:58:06.21] How much do you think people need to look into the future? Is it a valid strategy to just solve today’s problems and don’t care about what’s going to happen in the future and just take the problems when they arise? Specifically talking about technology and architecture here – how much do you have to plan in advance in your view to make sure that you can handle, for example, an explosion in the number of users that you have? Can you imagine that some people might choose a different approach because they expect that they will be – maybe not the size of Google or Amazon – a lot bigger?

David Heinemeier:           [00:58:39.09] This is the delusion of the lottery coupon liar. “Oh, I’m going to buy a lottery coupon now, so I’d better get that mortgage on the huge mansion lined up, and I’d better get my favorite sports car picked out, and let’s see which riviera we should go to for the summer vacation.” It’s a little premature. The number one issue that most people are facing is “How do we get to the place where we can be so fortunate as to have scaling problems?” It is not to figure out “How can I pre-empt all these scaling problems, because that doesn’t work anyway.

[00:59:15.06] If Facebook had sat down, “Oh, let’s design a system that works really well for 1,3 billion users.” Do you think that system would ever have launched? No, it wouldn’t. No one designs systems upfront that can work for a billion users. At least not if that system does anything novel or moderately complicated. Perhaps it’s possible to design a WhatsApp in advance for a billion users, but I even doubt that. The vast majority of scaling stories that we have from all the survivors who made it to this giant scale was, “Oh, we threw the system out like three times.” Some people look at that as, “Oh, if I was just smarter, I could not have to do that. I could save these three discarded systems on the way to get there”, and I would say “Absolutely not!”

[01:00:04.20] The whole point of how they got there, how they allowed themselves to get there was to not worry about what happens at a billion people. Having to make those considerations upfront is just an utter waste of time, and vastly increases the odds that you are not going to make it, because you are spending time on the wrong thing.

As we were talking about earlier, the difference between clay and iron. Before you figure out what you’re trying to build, using iron to build it is a frustrating experience, and not a very productive one either.

[01:00:34.28] First figure out what it is that you want to build. Figure out a way to get, as they say, ‘product market fit’, selling something to people willing to buy it. Then, once you figure that out, and once you’ve gotten to a billion users, worry about how to deal with that. There’s virtually no examples in recorded history.

The one I keep hearing about, which I’d love to dig into at one point, is Friendster. But almost no online business has ever truly failed because they did not anticipate how the system was going to operate at a billion users. It just doesn’t happen. It especially doesn’t happen because the fundamentals that we build on today, shared nothing architectures and the stack approaches that make sense even at the low end – they can scale incredibly high.

[01:01:21.07] The architecture that we’re using for Basecamp now at millions of users is very similar to the architecture that we started with, because the fundamentals are sound. It’s not like we needed that much of a different pattern or strategy to get there.

As a short conclusion, do not worry about tomorrow. Worry about today. Today has plenty of issues for you to deal with, and then worry about tomorrow tomorrow.

Stefan Tilkov:                        [01:01:48.21] David, those were perfect last words. Thank you so much for spending time with us, it was very, very interesting. Opinionated and fascinating, as was to be expected. Thanks a lot. We’ll put a lot of links in the show notes to stuff you’ve written, and further links on many of the topics we’ve talked about. With that, thank you again, and thanks to the listeners for listening.

David Heinemeier:           [01:02:09.22] Thank you very much.

Facebooktwittergoogle_pluslinkedin

Tags: , , , , , ,