#43 - Unveiling Dark: Ellen Chisa & Paul Biggar

09/24/2019

Last Monday, Ellen Chisa and Paul Biggar unveiled Dark, a new web-based programming environment for creating backend web services. In these conversations, first with Ellen and then with Paul, we discuss how they met, conceived of the idea, iterated on the product, and their long-term vision for Dark.

Dark is a web-based, structured editor with a data store built-in. It’s code has a functional programming feel to it, but it also embraces what they call “functional/imperative”. For example, their “error rail” allows programmers to defer handling nil-cases, much like a dynamically-typed language, but still keeps track of their existence in a monadic structure, like a statically-typed language, but without users having to learn anything about monads!

Paul often brings the discussion of Dark back to Fred Brook’s distinction in No Silver Bullet between essential and accidental complexity. I had fun in this interview diving into the Aristotelian roots of that distinction. We also debated the meaning of the terms “no-code” and “low-code”, and whether either could be applied to Dark.

Dark removes accidental complexity around infrastructure and deployment. There is no separate step to deploy code in Dark. It’s “deployless”. Every single change to a Dark codebase is instantly (in 50ms, the time it takes to get your incremental change to the server) deployed to production servers. Of course this doesn’t mean that every change you make is instantly deployed to users, but simply put on production servers behind a feature flag ready to be rolled out at your discretion. Deployment, getting code running locally to run in production, is eliminated because all code is running on Dark’s platform at all times. What remains is simply choosing when to release that code to users.

One of my favorite parts of Dark is how readable its editor makes functional programming, which I typically find intimidating and difficult to parse. The Dark editor saves all past HTTP requests to all routes, and then uses those values to provide “live data” for every intermediate expression in that route. A dense section of code becomes totally comprehensible by clicking through each expression and seeing actual past values that have inhabited that expression. It combines of the best parts of a debugger and sprinkled console.log statements, but without the downsides of either.

I’m glad that we had the opportunity in this conversation to dwell on some of the trade-offs of using Dark. Paul and Ellen are well aware of the risks customers face by moving their applications onto the Dark platform, and hope to alleviate those risks as much as possible. For example, they are looking into creating a legal structure that will make Dark open-source in the event that Dark shuts down.

Paul Biggar is best-known in the Valley for co-founding CircleCI, a tool for continuous integration and deployment. At heart, he’s a compilers nerd: he got a PhD in compilers, worked on the JavaScript compiler at Mozilla, built CircleCI which is a compiler for deployment, and is now building Dark, a programming language, environment, and infrastructure compiler.

Ellen Chisa is passionate about helping people make things. She worked at Microsoft on Office Mobile, at Kickstarter, and started a company that built tools for travel agents, Lola.

Transcript

Transcript sponsored by Repl.it

Corrections to this transcript are much appreciated!

Steve:

Welcome to the Future of Coding. This is Steve Krouse. So, there are a few exciting episodes to look forward to in the docket. Amjad Masad of Repl.it, the sponsor of this podcast, that episode has been recorded for many months now, but it's finally going to be released soon. I also, a couple of months ago, recorded an episode with Jennifer Jacobs who used to be at the MIT Media Lab and now is an assistant professor at UCSB. We also have Jonathan Aldrich coming on the podcast. You may have heard of his name because he is the adviser, or the adviser of the adviser, of many of my other guests who have already been on the podcast. So, it is about time to have his voice represented here as well. So look forward to all of those episodes.

Steve:

And you also have some solo podcasts of me talking about my own research to look forward to. I have been very delinquent over the past year about publishing those episodes, but I have plans over the next couple of weeks to release one and then maybe a few months later to release another one. So hopefully that happens and you can hear more about what I've been up to. You can always find what I've been up to at futureofcoding.org/log but if you prefer to get it in audio form you'll have to wait a bit.

Steve:

So today's episode is really exciting. It's been planned for a long time, and it's particularly topical right now because they just launched this past week. So I'm interviewing Paul Biggar and Ellen Chisa of Dark today. So beyond their launch, which you probably saw because it was at the top of Hacker News, and all over Twitter. You may have heard of them because I've been working with them over the past two years on a really long and exciting research project, doing a survey of the space of Future of Coding tools, languages and interfaces. And we in the past couple of months packaged up that research to be published and we launched that two weeks ago as the Whole Code Catalog, which you also may have seen on Hacker News and Twitter.

Steve:

So I was really excited to get that out the door because that took a lot more time than you would've expected to put together all those videos and reviews of these tools. I had to actually figure out how to get them to run, and then spend time building things in them so I could report back on how they felt and what it was like to use them. So I encourage you to check out this Whole Code Catalog, which I talk a bit about with Ellen in this episode. So a bit about how this episode is structured. It's a bit different, I guess maybe a little awkward, than the way episodes are normally structured. We were thinking about publishing this episode in a different way, but the way that we decided to go forward is there are two interviews, one with Ellen and then one with Paul in series following this.

Steve:

So this episode is kind of like two interviews smooshed together. The one with Ellen is quite a bit shorter than the one with Paul. With Ellen, we talk about the Whole Code Catalog, we talk about her background and how she met Paul originally, and also we talk about her experience hiring their first customer at Dark, which is not intuitive but a really compelling idea when she explains it.

Steve:

And then with Paul we talk more about the programming languages side of things because he's a compiler expert. He studied compilers in college. He worked on the JavaScript script compiler at Mozilla, and then started CircleCI, and now he's doing Dark. So just to whet your appetite for the conversation, we talk about Dark, which is this web based tool for building web HTTP back-ends. It's known for being a functional programming, structured editor that features live data that you can play with as you are editing your code.

Steve:

So Paul talks a lot about accidental versus essential complexity. And that's something that I've been thinking a lot about in the past couple of months, particularly because essential verses accidental is an Aristotelian idea. And so Paul and I get into that semantic debate a bit. We also get into the debate about what the phrase "no-code" means and if it would apply to something like Dark. And then we get into some of the bigger themes with Dark, including the fact that it's a holistic editor. They've combined many of the features and many different tools and languages into a single holistic tool that does a lot less than, I guess all those tools individually. But because it's been combined into a single tool, it doesn't need to do as much. There's a whole lot of affordances that could be taken advantage of by this holistic nature. But of course there are trade offs and so we get into the positives and the downsides.

Steve:

And the main downside that many people have been pointing out is the fear of lock-in. Because we've all seen what happens when you move all of your infrastructure onto a startup's platform and then that startup goes away. So it's a scary thing, but they have some plans for how to deal with it. And yeah, we talk about a whole bunch of things. This is a pretty long podcast, so I won't recap the whole thing here. You'll just have to listen to get to it. But it's a really fun conversation. And it's exciting to get a little bit more perspective about what they're trying to do in addition to all the information they released on Monday about their launch. So now just a quick message from our sponsor.

Steve:

Repl.it is an online REPL for over 30 languages. It started out as a code playground but now scales up to a full development environment where you can do everything from deploying web servers to training ML models, all driven by the REPL. They are a small startup in San Francisco, but they reach millions of programmers, students and teachers. They're looking for hackers interested in a future of coding and making software tools more enjoyable and accessible email jobs@repl.it If you're interested in learning more.

Steve:

Okay. Now I'm going to introduce Ellen and we'll hear from her for about 30 minutes and then I will introduce Paul and the rest of the episodes will be from him, and without any further ado, Ellen Chisa.

Steve:

Welcome Ellen.

Ellen:

Thanks for having me.

Steve:

Yeah, I'm really excited to have this conversation on the podcast.

Ellen:

Yeah, we've been talking for so long about these tools, but it's exciting to do it for the podcast.

Steve:

So before we start talking about Dark, I wanted to get a bit about your background before Dark and then how you met Paul and got involved in this strange world of improving programming.

Ellen:

Yeah, it actually makes a lot of sense. So for undergrad I went to Franklin W. Olin College of Engineering, and the focus Olin had was always on dramatically improving engineering education. And so this was a while back. So at the time that was things like having project-based education, not necessarily just doing a problem set, like doing a lot of the things we think of as being like really normal in the workplace. But for education it was a different approach. And so I spent a bunch of time there and really enjoyed it. But one of the things that frustrated me was that I did find that a lot of the time I spent programming felt not as rewarding as I wished it would, either because there'd be some syntax error or two tools wouldn't work in isolate together. And that was what led me to go into product management jobs instead of software development jobs, was I just felt like I was making better use of my time.

Ellen:

And so from there I did it a bunch at different consumer companies. I worked at Microsoft for a while, and then I was at Kickstarter, which I loved because I really like helping other people to make things. I went and did the first half of an MBA at Harvard Business School and then I was working at a travel company and I realized that every time I pitched the travel company I kept being like, and then people will travel more so then they will make more things. And I realized I need to get closer to helping people make things again.

Steve:

Hmm. That makes sense. What product did you work on at Microsoft?

Ellen:

I worked on Office Mobile. And so I was there in 2010 which was in the era where the world's smartphone market share went from being dominated by Symbian to being completely iPhone and Android.

Steve:

That's a really interesting time to be at a big tech company.

Ellen:

It was super interesting, and the organization was set up such that we worked a lot between the different groups. So we spent a lot of time with Office and with Windows Phone and with Windows Live, SkyDrive, which I guess is called OneDrive now, and with SharePoint. So we got to see a lot of the different pieces of Microsoft.

Steve:

So how'd you get connected to Paul?

Ellen:

It was funny, so originally Thompson, who started Frame Data sent me an email and asked if I wanted to be introduced to Paul to work in this space. And it just wasn't a particularly good time for me personally. I liked my travel job aside from continually pitching it as being about helping people make things. And my husband and I just bought a house in Boston and it was right before we got married and so it didn't seem like the best timing. And then two weeks after that Paul cold emailed me. I happened to be on a trip to San Francisco anyway and I agreed to have coffee with him.

Steve:

Wow. Yeah, that's a quite a disruptive change. So is that why your MBA is only partially because you left to start Dark?

Ellen:

I actually left when we were starting Lola, I'd been an EIR in Blade, which was a consumer tech incubator in Boston at the time, which then became Lola, and I'd originally intended to go back at some point after Lola. And then I met Paul and I was like, okay, I would much rather work on solving this problem. This is everything I've cared about solving for a long time.

Steve:

Got it. Yeah. Okay. So it sounds like when you first met him, the message really resonated based on your experiences programming in college you definitely saw the problem.

Ellen:

Yeah. And not just programming in college. Also, all of the experiences I'd had in the workspace with like at Microsoft, even just getting the emulator set up for the tooling we were using to build for Symbian was an hour long endeavor. And then inevitably you would have a day's worth of meetings and then by the time you got back to your desk, something else would be broken and you'd be spending another hour trying to get this emulator to work again. And then like at Kickstarter a bunch of the time, at the time I was there, they were using a different payment solution and it was really hard to test anything with the payments flow in your staging environment. And so every time you deployed something related to that you were a little bit nervous. And there's this whole thing around, we had like a campfire room, this was before Slack, to coordinate when we were going to deploy. There was a bunch of really interesting work going on there, but there were all these things that were accidental complexity about software that were getting in the way and like similar problems at Lola for sure.

Steve:

Hm. Right on the front of Dark's website is this claim that you're trying to make programming 100x easier, which is a pretty big number. And I'm curious if in like that first meeting Paul said like that's what he wanted to do and if you believed him, or if you took a bunch of convincing to, or like immediately you were like, yeah, yeah. It feels like this could be made 100x easier.

Ellen:

I was pretty much immediately on board, and Paul and I were actually talking about this last weekend because most people at the time weren't. And a lot of the things we've been working on with Dark for the last couple of years now seem a little bit more tractable. But at the time it seemed off the wall. It didn't seem like something you would be able to do.

Steve:

So before we keep talking around it, I wanted to paint a picture of what Dark looks like today so people have this mental picture in their head of what it is that we're talking about. And I thought maybe you could take that one, like just describe the editor experience kind of roughly like what it looks like and how you interact with it.

Ellen:

Yeah, absolutely. So Dark, like we've said on our website and other places, is a way of trying to remove accidental complexity from software. And so sort of the underlying thing behind that has been with a lot of our developer tooling over time we've been incrementally making things we've always used better. So editors get a little bit better, C/I gets a little bit better, version control gets a little bit better. And while they're getting better, they don't necessarily become easier to use or easier to connect to other parts of the tool chain. And so the original motivation with Dark was how can you pull out all of the things you don't want to be doing to be focusing on your code? And when we started thinking about that, we thought about, well we don't necessarily want to think about setting up our infrastructure. Like the wow moment is you get something working locally and then you're like, "Oh wait, now I have to like go get this somewhere else." So like how can that not be a problem?

Ellen:

And then, same thing, every time you make a change and then you wait and then you deploy again and then you see and you go in this loop, but it's pretty slow because you're waiting so long for your deployment. How can we make that not a thing? And then similarly my old problem from when I was an undergrad, you make a syntax error and then four hours later you find your one syntax error and you are very upset about having wasted four hours. And so we started looking at how to make those not a thing. What we came to was you could make a holistic solution that had the language, the editor and the infrastructure together. And by doing that you would be able to have each of them informing each of the others and do a bunch of work at the boundaries between them that you wouldn't have been able to without integrating them into a holistic platform.

Ellen:

But when you think about how Dark actually works, a lot of the pieces, the language itself for instance, feels a lot like a language. People always think it's going to be way out there. Completely different. But programming your Dark, you use a lot of the same things like they're still ints, they're still strings. You're still using listMap as a function. Like a lot of the core concepts for writing code itself feel the same. Our editor is structured. And so the easiest way to kind of think about what it is to have a structured editor is to think about tools like InteliJ or like what Kite was doing, where you have this much more robust set of auto complete tools. And so we definitely did not invent having auto-complete in an editor, we take a lot of inspiration from other tools that have solved these problems well, but so in Dark when you're adding something, you don't just have the auto complete as a suggestion, you actually can't add something that's syntactically invalid. You can't type "listMa" instead of "listMap" and then get an error for that because as soon as you type "listMa", your only option is going to be using the listMap functionality.

Ellen:

One other third thing that's interesting about Dark, which is probably the more different part, is that we are really focused on giving people the ability to look at the overall architecture of their code base. And I'm sure when you talk to Paul he'll get into this more, but the idea being that when you start a new job, someone takes you to the whiteboard and draws out how your system works and they don't just give you a list of files that you're editing. They tell you about how your system is actually operating. And so the way Dark is organized, we're moving towards, yes, you're looking at what looks like code when you're writing a new digital function or when you're writing some logic, but you still want to be able to see that high level view of how the different things within your code base connect to each other.

Steve:

So how, how does that look?

Ellen:

Yeah.

Steve:

In the editor, how do you connect?

Ellen:

We're only taking our first steps towards this. So one thing we have, it's actually pretty similar, if you saw the work that GitHub came out recently with references, to see how things call each other. But so when you're looking at say an API end point in Dark, if it's writing to a database, you'll see what data store it is writing to. If it is emitting to a worker, you'll see what background worker that it's emitting to and you'll be able to easily get more information about which different pieces of your code base you're connected to when you're editing that specific handler.

Steve:

And another thing just to add on, I don't know, you may have said this already, but an important thing to note about Dark is that it's for creating backend web services. Is that, am I, or am I limiting the scope of what Dark is useful for?

Ellen:

Yeah, that is absolutely correct. So right now Dark is very focused on building back-end web services. We think a lot of the concepts in Dark will apply to other types of programming as well. But when you're already at the point of making a programming language, and the editor, and infrastructure, you have to narrow scope somewhere. And so we started with building back-end web services.

Steve:

I may forget a few, but the sorts of things you can like create at the top level in Dark, whereas you can create a HTTP handler, you can create a cron jon...

Ellen:

A background worker creating a data store. Yep.

Steve:

Yeah. A database...

Ellen:

You can extract any code that you want to reuse into functions that you can rerun. So not like serverless functions, like more classic functions as you would think about them when coding. And then there's a couple of other interesting things. You're able to host static assets. So if you have compiled JavaScript that you want to host on the front-end, or if you have images you're serving to your end users that's how to handle this.

Steve:

So I thought you got a lot of attention, last year I think it was, for hiring your first customer. So I'd be curious where you come up with that idea, and exactly what the deal was for the person who got that, and how it went, what you got out of it, and yeah, if you'd recommend it. That sort of thing.

Ellen:

Yeah. So like we were talking about earlier, you're making a pretty big bet when you're deciding to build on this new infrastructure that you also have to learn a new language for. And so at the very beginning we also hadn't built that many things on top of it ourselves, and we knew that it was really painful, and that we were going to spend basically all of our time having to support the first person. And so we were thinking about how can we incentivize someone to take this big risk with us and how can we find out if that's something people are interested in and making sure we're building for real use cases, rather than just our friends side projects or our side projects.

Ellen:

And so we decided that the classic way to incentivize people is that in addition to giving them cool new technology that would hopefully be better than what they were using, we would also pay them. And so we had Daniel from Dabblefox in for three months and we paid him a stipend of $3,000 a month, because the idea was this was someone who is working on their business. They weren't someone who needed a full time job, like $3,000 a month is not going to support you for very long in the Bay area. But it would give him the incentive to just come in, show up every day, and help us like push through all of the early problems that came up as he was building on Dark.

Steve:

Hmm. And was it worth $3,000 a month? Did you learn a lot?

Ellen:

Oh yeah, it was great. And when you think about that $3,000 a month actually compared to what some companies will do, if they say hire someone, or have one of their founders spend all of their time trying to land one enterprise proof of contract to deal, it's a lot of money, but it's not as much money as it might sound like. So I think it was a really good investment compared to lots of things we've done over time.

Steve:

Hmm. And have you repeated it, or you just needed it that one time?

Ellen:

So we definitely needed it the most the first time. It definitely, it's funny, I was thinking about this recently when Daniel first started, it was pretty much one person was on Daniel duty, and one person was building all of the big projects that came up that Daniel needed. So it was sort of, we like either were immediately triaging or we were working on longer term investments. And so from that regard, a lot of those practices continue today. We still have all of our alpha users in a Slack channel and we try to be as responsive to people as possible and get them bug fixes as soon as is necessary. And then we've actually continued with having people in longer term as well. So we no longer pay people, but now we treat it as more of a Developer in Residence.

Ellen:

And so the idea being we're still getting all of this real time feedback. We get to have someone else from the community in and working with us and spending time with us and they get a desk that they're able to set at, we have some extra space in our office right now. So right now we have Costis in, who's a developer who's working on building an application he calls Song Battles. So people can put up two songs while they're at a party and the attendees of the party can vote on which song they want to hear next to the DJ.

Steve:

Cool. So you're in private alpha now, so you have how many people using the product and is it for businesses, their businesses or is it more just for side products? You're not supposed to use it in the private alpha for a business.

Ellen:

So what we focused on with the private alpha is doing a mix of both. So we kind of think of them as having slightly different use cases. The advantage of having people building real businesses on top of Dark is we start to get that feedback of what people need as they grow, as they have more customers, as they are trying to solve specific business problems. And so from that regard we've had people build complete back-ends for their business on top of Dark. So a great example of this is Chase Olivieri at Altitude, usealtitude.com, and so he built an entire flight deal alerting system, which has by far the best flight deals I have ever seen. And so in Dark like that has his integrations to Stripe so people can pay him, it has his integrations to Mailgun so he is able to send out emails about the deals as they happen, and things like that and that full back-end runs in Dark. That's been important to us and then Pix-

Steve:

That's very cool.

Ellen:

Yeah. And then we also recently had Pixelkeet, which is a design firm that does like a lot of landing page design. We're actually working with them on something else, and they had an internal project tracker that they were using with their clients, and what they did was they built that back-end on Dark so they'd be able to scale and open it up to other design firms to be able to use as well.

Steve:

Hmm. So how many people are in private alpha at this point?

Ellen:

Yeah, I would say at this point we have about a hundred developers who we've let into the private alpha. The number can be a little bit misleading. So that's a mix. So those, obviously there's examples of people who've built entire businesses. There are people who've done really interesting side projects. There was Tokimeki Unfollow, which was originally built on Glitch. It's a really nice website that allows you to individually see everyone you follows tweets and then decide if you want to keep following them or not based on if their tweets sparked joy. Kind of going off of the Konmari trend. And so all of the data store for that runs on Dark now. It got a little big for Glitch. So Julie has moved over the data store and so we have that. And then we also just have friends that we let on super early.

Ellen:

Like I remember Peter van Hardenberg, who was early at Heroku, at the beginning I had him come over and was like, you have to try this new language. It can finally write FizzBuzz, and so like I'm hovering behind him, forcing him to write FizzBuzz in this like absolutely terrible tooling. And so he still has an account from that and sometimes plays with it, but like, so it's a mix of companies, people with side projects that were really interesting and needed Dark for a specific reason. And then just friends who helped us out along the way. Like how you have one for instance.

Steve:

Yep. I, in talking to people about Dark, which happens pretty often given that we work together, people are very eager to see it and play with it themselves. So maybe you can give people some sort of a timeline on when they'll be able to see it and play with it.

Ellen:

Yeah. So we're excited. People are going to be able to see it starting on September 16th with two exceptions. If you are coming to Strange Loop, we are having a party that Friday evening, the 13th and we'll be showing it there, as well as all of our work on the Whole Code Catalog with you. And then we're also going to be showing it very briefly at Full Stack Fest in Barcelona the first week in September. But after that you'll be able to see the video on our website, be able to see everything on the 16th of September.

Steve:

Cool. And can people play with it themselves yet, or we're not there yet?

Ellen:

We're not there yet. I think that's one of the challenges with infrastructure is, if you're opening up a bunch of infrastructure, you want to make sure people are building things that you can support well. So for our case that's back-end web services. One of the things we've also seen with Dark as we've put people into the private alpha, is right now we're very optimized around someone coming in and knowing what they need to build and then building it. And so I know a lot of developers, like you play with a project, be like kicking the tires, and be like, Oh, I'm just going to make some random stuff. Dark definitely works better when you sit down and you're like, this is the API I need, or this is the data store I need, or this is the API and data store I need. But when you come in and know what you need to build, the experience is much better.

Steve:

Hmm. So Dark right now isn't very good for like explorative programming.

Ellen:

I would say, yeah. It's not as good for explorative programming yet. I think it's also by nature of the onboarding. While we've been in private alpha, I've been onboarding people by pairing with them, which is really enjoyable for me. I learn a ton about the product, people have that face time and can get to know us as a company a little bit, in addition to just seeing the product. But we haven't really put that into something that is like an easy to learn onboarding tutorial, which you experienced once recently. And so I think that will help a lot with that too.

Steve:

I'm a little bit confused why you think Dark wouldn't be good for explorative coding because it's like all about.. It has a very tight feedback loop. Like you make a change, you see the result. It seems like that's one of the most necessary ingredients for exploratory coding.

Ellen:

Yeah, I think, actually that's a really good point. I think the reason for that tension is we kind of, we've been calling it the Dark flow recently, but Dark works best if you either already have a front-end you're passing data from, or if you're connecting to an external API that's going to be giving you a response. We very much go from what is the end user going to see to what is the interface the end user is looking at, and then you talk to Dark for building out your back-end to support that, which is a little bit different than how people classically think about a back-end, where they're like, okay, I'm going to get all my models set up, but in Dark, getting all of your data stores set up, you're going to be like, okay, I built a bunch of data stores in five minutes and like cool. Like it's not an exciting experience just because it's so easy, which is great when you're just trying to get it shipped. But like as a fun thing to do, it's not like super fun to type out a schema.

Steve:

Hmm. Oh that's interesting. Well I feel like then the way to phrase it is that yeah, back-end people I guess won't have that much fun, I guess, or phrasing it the other way, front-end people would I guess have a lot of fun building exploratory things with it.

Ellen:

Yeah. And I think back-end people would too, but they just have to flip like, and not all back-end people are this way, but if you start from the perspective that I'm going to start with my front-end, get my front-end going and then use that to play with Dark, then you would have a really good exploratory time.

Steve:

Hmm. That's fun. I guess that kind of makes... I feel like that's what your first tutorial will probably look like, right? You'll supply people with the front-end code, and then work with them through the tutorial to how to write the back-end code.

Ellen:

Yeah, exactly. And that's how we do onboarding now too.

Steve:

So I wanted to, I guess explain to the audience how you and I started working together. I think it started when you reached out almost two years ago.

Ellen:

I think one of the reasons we were reached out to you was like you seem to like the obvious person to talk to for the thing we were curious about where we wanted to see like lots of interesting best in class things from lots of developer tools that were happening and specifically around how they felt and what felt good and what felt bad to serve as inspiration for what we were doing.

Steve:

Yeah. So I think there are like two interesting things about that idea you had to survey the space. I guess the first one is, is the idea of survey surveying the space at all. I, maybe I'm wrong, but I don't think many companies do that. I guess particularly from the developer tools perspective. I don't know any, because that's the space I know best, I don't know any other tools that have done like a survey thoroughly the way we have. So yeah. Where'd you come up with that idea and-

Ellen:

Yeah.

Steve:

Yeah, why?

Ellen:

Yeah. I think this is actually, one of the ways I learn is I like to have lots of inputs to kind of build out a system, and a mental model for how things work. And so I think that's partially like a personal learning choice. And so for me that's like, okay, well let's see all of the things that we've tried before, because we don't want to just go either make the same mistakes or we don't want to do exactly the same thing. We just want to be well informed about what has happened and why, and it was important to me, especially having not been working in developer tools recently.

Ellen:

I think another piece of it is a lot of times doing research or doing surveys like that gets a bad rap because people think of it as, Oh, you're just like making a competitive landscape diagram. Like that's such an MBA thing, and I think that's not the interesting part about looking at other tools at all. I think it's much more you want to look at them for that qualitative sense of what are the trends for where we're going? What are things that people are really enjoying? Why do they really enjoy these things and much more from like this human understanding part of how we're interacting with our tools.

Ellen:

I think in developer tools it was also in particular because we see innovation across so many spaces. So you have academics, like Cyrus working on Hazel who are thinking about it from that perspective. You have people who are building very pragmatic things, or people who are building like a tool that was originally just intended for themselves and it turns out that everyone uses it. You have people who are trying to build companies around developer tools and you obviously like have the big cloud players doing a bunch of things as well and so it's much more disparate than you see in a lot of other spaces.

Steve:

Hmm. Yeah, that's a really good point. I find it's kind of a beautiful thing. It's like hobbyists and academics and start-ups and big companies.

Ellen:

Yeah.

Steve:

That's cool.

Ellen:

I was also really interested in, it was like one of the things that's different about developer tools, is you're building tools for people who are experts at something already. And so I knew I'd already taken a bunch of inspiration from what we did at Lola because we were building tools for travel agents who are experts at booking travel. And so I wanted to also be looking at expert tools across the board, not just necessarily for developers, where when you think about like tools like Origami, which is really focused on designers rather than developers, or things like Photoshop, or Sketch, or Figma, or all of kind of those spaces as well. Those can also inform what you do in the developer space.

Steve:

So I guess maybe if you had to pick one or two of your favorite tools that we reviewed that you'd recommend for people to look at when when the Whole Code Catalog is public, what would you recommend?

Ellen:

Oh, this is so hard. I like them all. I would definitely recommend looking at Coda. I think it's really interesting to think about the contrasting approaches between trying to add more programming technologies into office related tools, compared to starting with a programming tool and then figuring out how to make it more accessible over time. I think I'm also biased there just because I worked on Microsoft Office before, like that space has always been a really interesting space for me. So I would say Coda is definitely one in that regard.

Ellen:

I guess since I've also already mentioned it, Hazel is a good one if you want to see more of an academic approach. And then I think I would also say Eve just because that's sort of the classic one, I really admired how much they were willing to try and experiment, share everything they learned so publicly, like pretty much the opposite approach of what we did at the very beginning, was they were working all the time in public and showing everyone everything they learned as they went as opposed to us kind of spending a bunch of time learning. And now we're trying to share as much as we can. And so I think there's is really interesting as well.

Ellen:

They're all so interesting for different reasons, and if you want to get into the history, like we did HyperCard, and I loved HyperCard so much in elementary school, and I think that was one of those things, where I think actually, the reason I wanted to work on Microsoft Office, was for me, PowerPoint was the next best thing to HyperCard when HyperCard went away. I'm not really sure why I went the PowerPoint direction instead of a programming language direction, but I spent a lot of time, and the PowerPoint object model, like scripting how to move around between different slides with buttons, basically emulating HyperCard. So even from the historical standpoint, HyperCard is really interesting.

Paul:

I think from the community standpoint, Glitch is really interesting. I think from the structured editor perspective, Hazel is really interesting. I think from the, how do you take something that feels a little bit more like Scratch and get it in a browser where people can try it, experience Repl.it, It is really interesting. I think that was your comment on it as well, as like, it's like Scratch for people who are slightly older and don't want to be using this tool that was designed for children. I like Lego robots. I like many programming tools that were designed for children. I think they're all really cool, but, I think Repl.it, It does, up a level that a little bit. I guess again, if you're more on the academic side, Smalltalk is interesting. They're all interesting for different reasons, and so, I guess, I would say when you're looking at the Whole Code Catalog and figuring out what to dig into, I would go a space that you're interested in. I think that's when I get the most out of it, is when I start from other concepts I'm already thinking about.

Steve:

So, maybe talk about why you decided to make all this research public, and put together the Catalog. I was very excited when I heard that was what was going to happen.

Ellen:

I think, actually, I think part of it was we kept doing it, and I was excited and I would talk about our work with people who visited our office all the time, or other people who are interested in this space, and then it became like, "Oh, I'm occasionally copying and pasting individual things and sending them to people, or talking to people about individuals tools that we've worked on," but we assembled the research more as background for ourselves, and because we wanted to know state of the art, but, we want everyone else to have that as well.

Ellen:

And be able to work from that, and kind of see where we're getting to as an entire industry of people who are trying to make the experience of coding better. And so, obviously, our approach at Dark for making coding better is having a full language and then figuring out how to make that not have as much accidental complexity when you're using it and really getting down into that essence of what it is to code. Other people are looking at it from, how can we, kind of, sneak code in through our office tools that we're already using. But, yeah, I think all of us are trying to give more people the ability to compute things in the way that they want to and be able to build their own tooling, and I think having access to that is good.

Steve:

Before we sign off, other places on the Internet that we could link to you or anything, and if there's anything that you wanted people to reach out for, like if you, maybe this residency you mentioned or if you're hiring, or other things.

Paul:

Yeah, absolutely. So we're, our dark website is darklang.com and we are @darklang on Twitter, and then, I personally am @EllenChisa on Twitter. I'm in Steve's Slack group, you're welcome to talk to me there. I feel like the conversation moves very quickly, but you can DM me and I will respond. And then I'm just ellen@darklang.com.

Ellen:

We are definitely hiring, obviously, lots of this stuff, like Steve said is in alpha so there are many bugs to fix. There are many features that are basically stubs that we want to keep working on. There are many exciting problems still to solve, so that's great. In particular, right now, we're looking for more engineers, developer evangelist, possibly a VP of engineering, if anyone happens to be listening who's excited about that, so, yeah, definitely reach out to me for any of those.

Steve:

Great. Well, thanks for talking. This is a lot of fun.

Ellen:

Yeah. Thank you for doing all this work with us.

Steve:

Welcome, Paul.

Paul:

Thanks Steve. Happy to be here.

Steve:

Yeah, I'm really excited to finally have you on the podcast.

Paul:

Yeah, I've been meaning to do this podcast for ages. I feel there's a email in my inbox about this for like a year now.

Steve:

Yeah, well, you guys have been keeping things under wraps while you're working out some of the details, so. I'm excited now finally to share some things about Dark.

Paul:

Yeah, I'm excited to be sharing them as well.

Steve:

So, right, before we get into the details of dark, I wanted to just get a bit about your background because you've been-

Paul:

Mm-hmm (affirmative).

Steve:

-in developer tools for awhile, but, I want to start maybe before, just tell us a bit about you and then what brought you to Circle, and then Dark.

Paul:

Mm-hmm (affirmative). Yeah, so my background, I did an undergrad in CS and I discovered that I really loved compilers. So then, I did a PhD in compilers and then went to work for Mozilla on their JavaScript compiler team. After I was there about a year, I got tired of a particular tool that we were using at Mozilla, which was essentially their CI tool. So, I started CircleCI in about 2011 to do CI. And I did that. I stayed there for about four years. Then, I took a break and started thinking about like, what am I doing next? And, eventually, decided to work on fixing everything, which became Dark.

Steve:

Cool. Yeah, I didn't realize that the idea from Circle came from the experiences at Mozilla. That's interesting.

Paul:

Yeah. Well, I had done a YCombinator startup before, between Mozilla and the PhD, and one of the things that was said to me at the time, was like, "Why don't you do compilers as a service?" And, I kind of had no idea what that was. Then, after spending about a year, the fact that I was frustrated with this tool was like, "Oh, this compilers as a service kind of makes sense."

Steve:

Hmm. Yeah. I guess that's related to Dark. Dark is also a bit of a competitor as a service.

Paul:

Yeah, yeah. I mean, essentially my career is getting people to give me millions of dollars to build fun compiler tools.

Steve:

Hmm. Okay, very interesting. Very sneaky.

Paul:

Mm-hmm (affirmative). Yeah, they don't know I enjoy it. They think it's about the money.

Steve:

Hmm. You're just an academic at heart and just wanting to build compilers.

Paul:

Right, exactly, yeah. And, tenure's a bad place to do it, but industry is a good place to do it.

Steve:

I think a lot of the people who listen to this podcast, a lot of programmers, know what circle CI is.

Paul:

Mm-hmm (affirmative).

Steve:

And based on a few of the essays you wrote while you were at CircleCI, like lambasting all the complexity in normal deployment.

Paul:

Mm-hmm (affirmative).

Steve:

I feel like those were kind of where the seeds of Dark came from. Is that accurate?

Paul:

Yes. A lot of it's the, it's funny. At the time, so, the blog posts that I think you're referring to is, "It's the Future".

Steve:

Mm-hmm (affirmative).

Paul:

And, at the time, I was actually a believer. I was like, "I see why this Docker thing is useful. I understand orchestration." I honestly, you know, was in the camp that this was the future. I wrote a follow-up to it. It wasn't until a couple of years later that I realized, "Oh shit, you know, this is way too much, this is way too much complexity." It combined with a couple of other ideas that I'd had about how hard it was to build internal tools and flaws and of sort of existing program paradigms and how we're trying to build cloud things with languages that are, that is designed for single machines, and sort of that dot impedance mismatch and lack of abstraction. So all of those combined into one and became Dark.

Steve:

Cool. Yeah. So, I didn't realize that the essay... It wasn't actually sarcastic, originally?

Paul:

So no, it was sarcastic, but in a tongue-in-cheek, you know, we're all friends here sort of tone, but I wrote a follow-up that's called, "It really is the future," where I talked about like why containerization is important and orchestration is important, and how I actually believe that this set of tools is the future, and that the problems that they solve are real. And the problems that they solve are real, it just turns out that the solution is too complex.

Steve:

Okay, so, I think that's a perfect transition to my next question. I want to talk about complexity.

Paul:

Mm-hmm (affirmative).

Steve:

Because it's something that a lot of people, a lot of programs talk about in general, but particularly, us folks trying to make programming better.

Paul:

Mm-hmm (affirmative).

Steve:

We talk a lot about this distinction between essential complexity, accidental complexity-

Paul:

Yeah.

Steve:

That comes from Fred, Fred Brooks' essay, which he-

Paul:

Yep.

Steve:

- in turn, borrows, although, back from Aristotle, this distinction between essence and an incident or accident. And part of-

Paul:

Interesting, I didn't know that connection.

Steve:

Yeah, yeah, yeah. And, I didn't realize that connection either, but I was talking to a friend who's a philosopher about this sort of work-

Paul:

Mm-hmm (affirmative).

Steve:

And he was like, "Do you even know what essence is, like Aristotelian essence?" And I was like, "That's not what he's talking about." But then I went back to the Fred Brooks essay, and there's the word Aristotle right there. He says like, he specifically he's, yeah, Fred Brooks is, I think, part of the reason we still reference him is because he's so well read-

Paul:

Uh-huh (affirmative).

Steve:

And I can draw on all those big ideas from, anyways, philosophy. So anyways, I become to, want to drill a little bit deeper into, what is-

Paul:

Mm-hmm (affirmative).

Steve:

the difference between essential and accidental complexity? Because-

Paul:

Mm-hmm (affirmative).

Steve:

As you were just saying, the problems that Docker and orchestration solved are real problems.

Paul:

Mm-hmm (affirmative).

Steve:

So, it's kind of hard to say that they're not part of essential complexity-

Paul:

Mm-hmm (affirmative). Yeah.

Steve:

But then, I guess, if you use a tool like Dark, clearly, they're not part of the essence of Dark, so,

Paul:

Right, right.

Steve:

Yeah. So how do you-

Paul:

I think one of the things that makes Dark what it is, is that we draw an extremely wide bound around accidental complexity, or maybe an extremely narrow one around essential complexity. So, if you have someone that tells you that, you know, architecture or infrastructure is complexity, or sorry, is essential complexity, I'm extremely skeptical.

Paul:

The thing we keep coming back to is that the thing that you want to do with a backend -- so Dark is for back ends -- you want to receive data, you want to send data, you want to process data, and you want to store or query data. And, you know, there isn't a, orchestrating the services that do that, in that description of what you're trying to do. If you ask someone to describe their app, it's going to be an app that talks to an iPhone and saves some data and, you know, sends out some notifications or something like that. And, there's going to be no mention of Kafka, or, Postgres or something along that.

Steve:

Well, so, I guess to push, to use your own definition against you.

Paul:

Mm-hmm (affirmative).

Steve:

If someone was designing an iPhone app, I don't think there'd be any mention of HTTP handlers or databases, either.

Paul:

For sure, for sure.

Steve:

They talk to them directly about, you know, saving data somewhere, you know?

Paul:

Yep, yep. That's absolutely right. And then I think there's a difference for Dark between what we're shipping today and what we think of as of accidental complexity.

Steve:

Hmm.

Paul:

I think the question sort of points out that you've seen Dark, whereas most of your listeners won't, but the, in Dark, we certainly have certain HTTP concepts and that kind of thing, which, as Dark expands over the next, I don't know, decade, or so, you know, may go away if we can.

Steve:

Oh, that's very exciting. Cool.

Paul:

Well, so I mean, there's what we're launching and, you know, what Dark will be in the short term, and then there's the vision of getting rid of accidental complexity. And, I think getting rid of all accidental complexity, and we talk about making Dark, or Dark making coding 100x better. It's very much a computer on every desktop kind of vision. It's not a claim that we have achieved this, but rather that we are going to do that and our current work is in service of it.

Steve:

Mm, Okay. Yeah, that's a very important distinction to make, because, I think-

Paul:

Mm-hmm (affirmative).

Steve:

I even saw people in hacker news comments, you know, saying things like, you know, "There's no way they've done this." And, so it's important to be upfront that you haven't.

Paul:

Yeah, yeah, and that is accurate. I would say we're currently in a, you know, depending on, you know, someone's level of expertise, we're maybe a 2x, 3x, 4x better, which, honestly, is kind of phenomenal. People make companies that make things 10% better and they're successful. But then, they also, we are currently also bringing our own complexity, which tampers that down a little bit.

Steve:

Mm-hmm (affirmative). Oh, interesting. Yeah. Yeah. You remove a lot of complexity from other tools, but there's Dark-specific, new complexity that you're adding to this.

Paul:

Right, right. And most of that is, is things that you can't achieve yet. Things that need to be simplified more, you know, hacks and shortcuts to get to launch, et cetera.

Steve:

Mm-hmm (affirmative). So, I think now's a good time, before we get too much into the weeds, to describe Dark, I guess, conceptually, the things you can do in it now, what it's built for.

Paul:

Mm-hmm (affirmative).

Steve:

And then visually, just describe kind of what the screen looks like and how you can interact with it.

Paul:

Mm-hmm (affirmative). So, taking it from accidental complexity, our purpose with Dark was to make back ends really easily. And, the way that we determined that we can remove all this complexity is by having a holistic editor, programming language, and infrastructure. So Dark, fundamentally, is an editor, which you access via your browser, in which you write the dark programming language and which is instantly deployed to Dark infrastructure.

Paul:

So, how that looks. When you go to the editor, you'll go to darklang.com slash, you know, your app, and you'll be presented with an editor, which is just a black screen and a sidebar. Our current metaphor, which is not a long-term one but is the one today, is a canvas. So, there's little bits of code on the screen and you add new bits of code. The bits of code are specifically handlers and databases.

Paul:

So, handlers, let's say that there's a bunch of HTTP handlers, and they have a specification for what they are, the URL and the method at the top, and then code in the body. And it's visually, everything's very dark, obviously, you know, some shade of black for the background, some other shade of black for the code. And then the code is-

Steve:

Would there be a light mode?

Paul:

You know, we were joking about, like, maybe we should make light mode for everything. And then whenever anyone asks about, why is it called Dark, and, you know, when it's actually light, and just like, feign ignorance. But yeah, there will, of course, be a light mode.

Steve:

Okay.

Paul:

So what you're writing in these handlers, like, the code that you're writing looks a lot like code. There's the, it looks a lot like modern code. You're, you know, you're typing, let X equals five, let Y equals 10, you know, X does Y-

Steve:

And, does it look most similar to OCaml, Haskell...? I don't know which one.

Paul:

So, yeah, it looks a bit like both. Little bit of, tidied up the OCaml syntax. There's, it's definitely based on functional languages, but where we're currently leaning in a functional / imperative direction, which means that we're stepping a little bit back from pure functional. And, I mean, we're really trying to simplify how people code and there's a lot of things in functional languages that don't simplify. I have people code, so we're trying to, you know, separate the two.

Steve:

Got it. So, I thought one way to kind of explore Dark is to talk about some of the big pieces or big ideas behind it.

Paul:

Mm-hmm (affirmative).

Steve:

So, maybe the word "deployless" is a good place to start.

Paul:

Mm-hmm (affirmative). Yep. So, I always bring it back to, you know, the big picture. So the big picture with Dark is, what specific accidental complexity is it that we're addressing today. The two major parts of accidental complexity that Dark in its current form deals with is the deployment complexity and the infrastructure complexity. There's two other major parts. One is API complexity and the other is, sort of, the tooling complexity that we're not solving as much today as the other two.

Paul:

So, getting rid of that deployment complexity is what we're calling deployless, and it's deployless in the way that with serverless there's still servers, but you don't think about servers the same way. With deployless, I mean, there's still a deploy step, but that deploy step is cut to the smallest possible thing. The blog post I wrote about this was how "Dark deploys in 50 milliseconds", and 50 milliseconds is literally, you know, the round trip time to the server. So it's the smallest amount of time it could possibly be. That's what we were going for. I think that's where the name deployless, which we didn't actually coin, came from.

Steve:

Cool. Yeah, I think that kind of leads us well into another big technology that enables the deployless, the structured editor-

Paul:

Mm-hmm (affirmative).

Steve:

And why and why you chose-

Paul:

Yeah.

Steve:

That piece of technology.

Paul:

So, we chose the structured editor for two reasons. One, that deployless was what we were going for, and we knew it couldn't be deployless if we were taking the current way of writing code or the way of deploying code. So today you, you know, take a bunch of code, you push it to GitHub, it gets put in a package by your CI. That package is maybe a Docker container, and it gets sent to a registry and then gets pulled in by your orchestration. You can deploy other ways, you know, you can do a Heroku deploy or DigitalOcean thing, right?

Steve:

Yeah.

Paul:

There's a bunch of different stuff, but they all end up in the thing of: take all of your code, put it in a box, replace the old box with the new box.

Paul:

And, we said, all right, what is the smallest possible unit of, of deployment? And the smallest possible unit is, you know, the thing that you just typed, the keystroke that you typed. And that's what we're deploying. In order to do that, if you were to use today's technology, every keystroke you type, you'd still have to package up the whole thing, put it in a box, put the box on the server and stuff, so then we're like, "Okay, how can we make it so that people can safely write code?" And that's the tie of the structural editor.

Paul:

The structural editor makes it such that, you never have your code in some sort of structurally invalid way. So, there's no syntax. It is not possible to have a syntax error type of parse error. And so, you can't just, you know, take this working code and then, mash the keys and end up in a bad place. If you mashed the keys in Dark, and, you know, assuming you don't hit backspace very much, you'll probably end up with still a working program.

Steve:

That's funny. So, of course, the immediate question that people ask is, every programmer is deploying to production all the time.

Paul:

Mm-hmm (affirmative).

Steve:

This must be chaos.

Paul:

Yeah. Yeah. I've started trying to define what deployless means, and I realized that this was an important thing to bring in to deploylwaa. Because it's not just like, it's not just deployless to rsync to your server. A key part of it is that the tooling that you're using allows you to do this safely, and by safely, I mean that you don't, you know, take down the site. I was going to say frequently, but actually I mean at all, that there's a structure that prevents you from taking down the site, and Dark mostly uses feature flags. We have a bunch of tools around this, you know, function versioning, feature flags, language versioning, database migrations. Those are kind of the main things. But, the point is, that we're integrating the tooling that has the essential complexity of a deployment and that has almost none of the accidental complexity.

Steve:

Okay. So, I think, maybe, I just want to spell that out a little bit-

Paul:

Mm-hmm (affirmative). Yeah.

Steve:

In terms of the feature flags. So future, you've taken a feature, sorry, you've taken a feature, I guess feature flags themselves are a feature of a programming tool chain-

Paul:

Mm-hmm (affirmative).

Steve:

And you're using them in, you're overloading them, you're using them almost like branches?

Paul:

Yes. So, where we're kind of starting from here is the realization that over the last 10 years, for anyone who's doing proper, continuous delivery or continuous deployment, whichever word you prefer, there's a separation of getting the code onto the server from enabling that code for users. The realization comes that, once you have that full separation, which typically uses feature flags, then the actual getting the code onto the server is not important. It's, I mean it's, you know, has to get there. It still has to be done, but it provides risk and it's not an essential part of how people think of their interaction with their users.

Paul:

The way people think of their interactions with their users when they deploy things is, you know, is this going to break users? Is this change going to be okay for users? Is it going to cause them down time? How do we make sure that the right people see it at the start and that we have a slow rollout and that we're able to be sure that making this change does not take the site down, and so on.

Steve:

Got it, got it.

Paul:

So, feature flags, they're sort of a glorified if statement, and it's an if statement that, that you can modify from outside the code, in a certain sense. And, yeah, that's the tool. That's the main tool that we use. So, if you have a piece of code in production, you wrap it in the feature flag and doing so is atomic, because of our structured editor and also because of our language. There's no stage that, which you go through, where the code doesn't work. So, you add a feature flag, you add the conditions for the feature flag, you add the new code. Maybe you just like, cutting and pasting and changing, you know, a character or two. You can do it fairly fine-grained and then, you know, you enable it, maybe just for yourself, maybe for your team, for your QA team. Or maybe, it's such low-risk that you're enabling it for everyone at once, and then you mark it as done and it is in production.

Steve:

Yeah. So, I think there's a part about Dark feature flags in production that I didn't quite get until you just explained it.

Paul:

Mm-hmm (affirmative).

Steve:

So, when you, when you start using Dark, the tool, every change you make, unless you put it behind a feature flag, is in production, which is exactly what you want when you're starting a project and nobody's using it.

Paul:

Yep.

Steve:

But then, the confusing bit is that we all, anyone who's worked on a production tool knows that, once you have people using it, you want the reverse workflow. Any change you make should-

Paul:

Yeah.

Steve:

By default, be behind a feature flag.

Paul:

Yeah, exactly.

Steve:

But that isn't how Dark defaults today?

Paul:

So, there's two phases, and any piece of code will go through those phases. They aren't company life cycle phases. They're, they're almost like a piece of code phases. The first phase is, no one is using it. It's a new feature, it's a new something. So, you can basically do whatever you like in that case, because you're the only user. So you're starting from a blank canvas for this new page, let's say, or this new API. And, you type it all out and, you know, iterate, and then you can completely delete it. No one cares because there's no users.

Paul:

The second phase is, you've got this thing that hundreds or thousands or millions of people are using, and you have to be extremely careful with how you roll changes out to that. And this is the part of coding today that people have almost no support for, where we write all of our code today as if it was the first one. In Dark, there's two separate modes for that. When you have users, all of your handlers lock, and by lock, I mean that you can't make changes to them.

Steve:

Hmm.

Paul:

And then, you must make changes by the feature flag, and then, opening a feature flag gives you the first state again, you're in this fresh paradise because no one's using it. Once people start using that, you know, again, it goes into that lock state.

Steve:

Got it. Oh, okay. I didn't, I guess I've never built a Dark thing with users, so, it never triggered that for me. That's cool.

Paul:

I'm not being particularly careful here, but Dark-the-vision and Dark-

Steve:

Mm-hmm (affirmative).

Paul:

the current state today.

Steve:

I see.

Paul:

So, we don't automatically lock today, which is probably why you didn't see it.

Steve:

I see. So, your customer will tell you, "I have users now." And then you'll switch the mode for them.

Paul:

So, they can lock it themselves.

Steve:

Oh, okay.

Paul:

Yeah, and the issue today is that we're not 100% happy with our feature flag implementation.

Steve:

Hmm.

Paul:

And so, we don't want to impose it on everyone. So, people can use it, but they don't use it as much as we would like because there's still some product and tech that needs to be done to make that a really, really good experience. But that's the plan.

Steve:

Okay. That makes a lot of sense. So, the thing that I had begun to realize is that, the distinction, the incidental complexity you've removed is that, normally you would get the code working on your own physical computer-

Paul:

Mm-hmm (affirmative).

Steve:

And then you would figure out a way to get it working in some container, and then you have to like actually deploy it to some other service and then orchestrate it. So it was like this whole rigamarole to get it working from-

Paul:

Mm-hmm (affirmative).

Steve:

Your local physical computer to some-

Paul:

Yep, yep.

Steve:

Some other place. And so, the real distinction Dark makes, is that there's no physical computer involved, so it's always working on other people's computers. And then, whenever it is that you want to turn the switch to get it shown to users or not shown to users-

Paul:

Yeah.

Steve:

That's up to you. It's like, that's unrelated.

Paul:

Right.

Steve:

But the actual problem you solved is that, we're always working in the Cloud.

Paul:

Yes, yes. So, we're always working, and then these are tools that are built for the problem that you actually want. The essential complexity of building features is, when do we give this to users? Not, how do we get this to that machine over there.

Steve:

Totally. Yeah, okay. That makes a lot of sense. That's very cool.

Steve:

Totally. Yes. Okay. That makes a lot of sense. That's very compelling. So then I guess a follow-up question is: So you explained earlier that part of the reason you wanted structured editors, is to enable this smallest unit of compilation thing. But given that most of the time people are going to be adding new features behind feature flags, couldn't you get rid of the structured editor? And then whatever little bit of text they've put behind a feature flag, just send that little bit of text over? Only if it parses, right?

Paul:

Yes and no. I think that someone could do something roughly like that. It would take a great deal of work to make that effective. The main things that break with that: So one, you've got this parser, and now you've an AST, and in fact you've got two ASTs. So AST is the abstract syntax tree, or, just the representation of the program. So you've got, you've got the representation of the program that's in production, and you've got the representation of the program on the user's thing that was just parsed.

Paul:

What is the distinction between these two things? Do they have a mapping to the other? I mean, they should. Right? Because you've only made one change, but they might not. So if you've got this pure textual thing, it's hard to sync up the old text to the new text. If you've wrapped something in an If statement, for example, might be a good example of that. And they're... The textual mapping algorithms are, around that, are challenging and not particularly advanced. And, so-

Steve:

So it's keeping track of the identifiers of each part of the AST syntax-

Paul:

Yes. I mean in Dark, everything has an identifier. So you make a change, you're making a change to this identifier, and you don't see the identifiers. Their IDs. They're not displayed in the code. And I'm not saying that you couldn't do it without that, but it certainly simplifies it. And there's a couple of other benefits that it brings. Notably the fact that the editor, that you can't have a syntax error. That it's impossible to have a syntax error.

Paul:

But the thing that was interesting to us is, I thought the structured editor, by itself, would provide this, but it actually turns out that we need a ton of language features as well. So feature flags are not really an editor feature, so much as they are a language feature, with editor support. So there's a semantics to every incomplete program, and that semantics of incomplete programs, the fact that Dark represents incomplete programs, turns out to be really important for this sort of editing. Because otherwise you have a program in an incomplete state, and you're working with text to get it to a complete state. And it might be hours between those two. With Dark, the language, and editor, I guess, there is an incomplete state, but all of those are fully representable and have full semantics.

Steve:

I see. So it's the automatic hole insertion from Cyrus's work.

Paul:

They're very similar to the Hazel stuff. Yes.

Steve:

Cool. Yes. Because I think that's what he talks about. The reason you need a structured editor, if you want to solve what he called the gap problem of never having a incomplete state, is that you have to automatically insert the holes. So if you, like "5 +", in our mind there's a hole there, but the computer doesn't realized that there's a hole there. So, you could type a question mark there, an under bar there, but a structured editor would do that for you automatically. Is that what you were saying? Or is there something else?

Paul:

So, it's that, but there's a lot more than that. Let's say that you're in a text editor today, and you're writing an if-statement. You're going to write "if", space, some condition. And you're going to have no tooling support while you're writing that condition, because it hasn't filled out the "then", the hole for the "then", and the "else", and the hole for the "else".

Paul:

So, it's going to be in an incomplete parse state, until you get the braces in place, and the new lines in place, and all that sort of thing. In Dark, when you type "if", you know that that creates a fully structured if-statement, similar to IntelliJ, or something, might, and the semantic analysis keeps running as you're writing that code, as you're writing the condition. So we're literally running your code to show you the results of what that execution would be, and to show you things like type errors, and that kind of thing. And you need that... Like you need your engine to understand what it means to be in this incomplete state. And, if you look at something like languages that have specs, they have this undefined behavior, where sometimes for even complete programs, they don't know what the meaning of a particular program is. And in Dark every program has a meaning. And I'm pretty sure that that's essential to having, at the very least, a good experience here.

Steve:

Hm. Okay. I think I've been convinced why, in order to solve the problems you're trying to solve, you need a structured editor, but it's a hard thing to build. So maybe... because it hasn't really succeeded yet. Like there's no... I guess maybe the most successful one that I could think of is Scratch the, the block-spaced one.

Paul:

Well, Excel.

Steve:

Well Excel isn't, in my opinion, structured, because you're typing the formulas all in text.

Paul:

Mm-hmm (affirmative). Yes, that's fair.

Steve:

Yes, I guess-

Paul:

A lot of the program structure is structured, but the final snippet of code is... I mean, I guess, or I guess you-

Steve:

Because like would you call-

Paul:

You have text editing for an expression and then each cell is an expression-

Steve:

... would you call Jupyter Notebooks, or Observable Notebooks, structured editors, because they have cells that you typed big blocks of code in?

Paul:

I guess I wouldn't.

Steve:

But I, I'm with you that like-

Paul:

Yes, I see what you're saying there. If you were to use Jupyter Notebooks to write very small blocks of code, and there's a little bit of text sitting in those blocks, I'd be willing to call that a structured editor.

Steve:

I see. Okay. So I guess it's a bit of a spectrumy thing. How, structured is the editor?

Paul:

Right. Right. Right. Right. Right. The first structure that I used, and that I really loved was ParEdit, which is a mode for Emacs for writing Lisp in. And that... I mean, partially this is because Lisp is so... it just doesn't have very much structure, but it amazed me how little it did, and how amazing that was. Like, how much you could do with so little. And essentially what it does is, whenever you insert a parens, it inserts the closing parens, and it puts your cursor in the right place. Then it has commands for like, pull the next expression into the parens. It made sure that your program was always well-formed, which for Lisp was really important. It completely took away all the downsides of Lisp, or all the parentheses-related downsides of Lisp. And I was really impressed with this.

Steve:

So, question. How does it know where to put the closing paren, in ambiguous cases, where it could go in a few different spots?

Paul:

Well, so it's... your choice is, and we're talking about ParEdit here. Right?

Steve:

Yes.

Paul:

Yes. So, your choice is, you insert an open parens and it will insert the close parens right here. Then your cursor's inside that, and you use the command to pull more things inside it.

Steve:

Got it. Got it. Cool.

Paul:

And, or, the other thing is you can select a bunch of text, and it will figure out what expression that text is in, and wrap up the whole thing with it. That's another command that you have, or you can unwrap as well. And I think that that works, because Lisp is so simple that there's no more syntax than that.

Steve:

But when you're building a structured editor for a language with, I guess, more complicated syntax, like Dark or Haskell, it seems, based on the amount of effort I've seen people put behind it and yet how less-than-ideal the outcome is, it seems like it's a really hard thing to do. And when I tried, myself, building a JavaScript structured editor, it was like, "Wow, this is way harder than I thought."

Paul:

Yes. So, we're on, probably, the, maybe it's the second or the third iteration, and I think you haven't used our new "fluid" editor. Right? You've only used the existing editor.

Paul:

The default editor.

Steve:

... I spent, maybe 20 minutes, in the new editor.

Paul:

Okay. Our progression was that we started with this thing that was totally new, which was like a graph-based editing, and that sort of went nowhere. We spent a couple of months on that, and then then we had this regular-looking AST-based editor, where you were filling in holes and, and that's the the default editor today.

Paul:

We realized that there's a whole bunch of problems with filling in holes. Mostly, that you actually want to make larger-scale changes to things, and you don't necessarily want to go to an editor command to add a let above. Most common feature we had is how do I insert code above the, above the current line? We realized that we were doing it totally wrong, and we had this vision of, "All right, what if it's still structured, but it feels like code?" And that's why we call it "fluid". You know, feels fluid. So "fluid" is still... It's not quite done. It's currently about 4,000 lines. So 4,000 lines of a OCaml.

Paul:

In terms of your question of like, "Is it hard?" Yes, it's absolutely hard. There's so many edge cases and there's, so many, like, if I'm here and I press this character, what is the thing that I'm trying to do? The result for us has been really, really good. It feels really good. It's interesting that when we look at what some other people in this space are doing... I saw someone come up with this thing called Tofu.js. It was an experiment, that this guy, Gregor did. It's really nice. At least the demo is really nice. I think it doesn't do all that much, but it turns out it does basically the same as us, and has a lot of the same design principles as us. It's really cool.

Steve:

Oh, interesting. And is Tofu.js a projection editor for JavaScript or for his own language?

Paul:

I mean, I think it's a demo of how one could be. I don't think it's a full editor.

Steve:

Got it. Because part of... One question I wanted to ask is: Given that it's so hard, is there, do you see any way for it to be done in a language-agnostic way and then everyone could just use it, kind of like CodeMirror, for example?"

Paul:

Could it be done in a language-agnostic way? I think it would be hard. I mean, in the same way that Language Server Protocol is hard. Doing it for one language is one thing. Doing it as a generalized thing... What we have is like a giant state machine of, when you're in this state, and you press this input, what do you do? Most of those are probably not generalizable. There is the default case of, I press a character and it gets inserted where the cursor is, is probably generalizable, and the 500 edge cases, probably less so. And they differ for every language, and all the language, the different syntaxes I would not enjoy that project. This sounds like a not-fun project.

Steve:

The compiler guy is calling quits. Dangerous.

Paul:

Yes. Well, so one of the things I always hate about compilers is parsers, and I mean, essentially, what I just described, I feel a lot of what this "fluid" editor is, is sort of a reverse parser. You're generating text from the AST, and keeping enough state to know what state you're in. It feels very like parsing, and I feel that we're going to get into the generalizable world here. We're going to be in some sort of parser generators and that kind of thing.

Steve:

Hm. So when I was using Dark, especially the "fluid" editor... I guess both of them, they're both in very early stages where things are morphing, but there were a bunch of bugs, like trying to get in the stage, states, and whatnot. I've used other structure editors, and they also are either very restrictive, and not fluid, or they're buggy. Part of me wonders if there's a way to ever be confident in having caught all the edge cases, and in such a way, that when, instead of 4,000, maybe it'd be 100,000 lines of code, and if someone wants to change a part of it, how will they be sure that it doesn't all break? I guess this is a question about, I guess all programming, but this-

Paul:

Yes, I mean it's, it sounds like you're asking the question like "How does one software engineer?"

Steve:

Okay. I guess that's fair. In particular, structure editors is the kind of software that people, listening to this podcast, I think are disproportionately likely to build themselves. That exact same piece of software. So if you have any software engineering advice specifically for structured editors, that's kind of what I'm getting at.

Paul:

I think the thing that we started doing very early, was unit tests. Far, far sooner than I would typically write tests, and then we're, we're using this statically-typed functional language. So testing is sort of a thing we do when we find bugs, as opposed to when we write code initially, but with the "fluid" editor, we've hundreds of tests now. There's more tests in "fluid" than in the entire rest of the application. The first thing we did was, we made a test harness. We made a test harness that made it super, super cheap to write this sort of thing. I mean that's a general, good testing practice. Thing that we're, that we're going to add soon as fuzzing. I think fuzzing is an essential part of finding all those edge cases. And I think once you run the fuzzer for long enough, and in a certain sense that this is similar to parsing as well, if you run your parser overnight, or if you run your fuzzer in your parser for overnight, or days, you're eventually going to realize that that, yes, it's caught all the bugs. So yes, we have a pretty, immense test suite.

Paul:

I mean one of the things, and this isn't a great answer for most people, is that we invest a lot into it. So there's multiple people on the team here, who are building and solving those edge cases. We're spending quite a lot of money on this. And then there's users who are trying it out, and finding the bugs, which is not a wonderful answer for people who are, maybe, solo projecting it. But that's how we got there.

Steve:

Yes, that makes sense. Let's talk about the Dark data store. Could you compare it to... Well, I guess, maybe first tell me how proud of it you are, because there were a number of things that are in-progress decisions. So if it's something that you're not proud of yet, we don't have to talk about it, but if you're proud of it-

Paul:

I would describe ourselves as not proud of it, yet. I think we're fine with it right now. It's one of those things where, we're on step two of six, and I think we'll all start getting proud of it around four.

Paul:

So, the way it works at the moment is that, when you have a data store, data stores are key value stores. You can store a value in it. Store it by a key, and look it up by a key, and there's, also, you can query it. So it feels-

Steve:

You can query fields by things that aren't the key, like by other values?

Paul:

By exact matches of field names.

Steve:

Okay.

Paul:

So if you use Mongo or Redis, and I know it's not the great, those... Mongo especially isn't the greatest tool to be comparing to, but you save values in a database. So it feels a little bit like document stores. What we're unhappy with is that we don't use real types of them. That's kind of our major thing.

Paul:

Then the next one is database migrations. So handling data migrations, not just given migrations between... when you're, going from one type to the other. And all this is spec'd, but not implemented. And then I think we'll be relatively happy at that point.

Paul:

I think the next thing that it will take for me to be really happy with this, is having more than than just storing records in it, but being able to store anything. And then after that we'll have more persistent data structures. Not just key value stores, but values, lists, et cetera. And at some point we want to add back relational aspects to it. So the initial version of of Dark was relational, and we realized that the data model didn't match a relational... data model of the language didn't match a data model of the code, sorry, of a relational database model. So we tore that out, went to persistent key value stores.

Paul:

The thing that... I sound a little bit negative on my own product here. I guess the important thing is, it works. Where we're going with it? One of the particular challenges is, when you look at something like Mongo, and how it moved its query language... What Mongo and Redis and a bunch of others said was, "SQL is bad, we're going to come up with something different." And the thing that they came up with wasn't great. It's the sort of language that sort of hacked into key names and that kind of thing, and then ends up being really weird. So, we're not happy with that direction.

Paul:

What we want is to be able to do a query that has any arbitrary code in it, and for us to compile that to a really fast indexed query. And that's where we're going with this. So rather than looking things up by specific fields in the record, you would look things up probably by a function, or a lambda, that says if this field is this and has the prefix of this and, you know, all this other stuff, and all this complex logic.

Steve:

Hmm. That's really, yes, that's really cool that you can unify the query language and the programming language that way.

Paul:

Right. And people have done this before. So those LINQ into the C# world. That's, actually, the only one I'm really familiar with it that does this well. But one of the problems that leads to all this accidental complexity is that different domains, you can't use the same language for. So if you're writing something in Rails, then there comes a point where you have to drop down to SQL and now you're in a completely different world, and a world that you probably have no familiarity with.

Paul:

So we want you to write the exact query that you want, in the Dark language. And this part of the accidental complexity that we remove, is figuring out what your infrastructure is. So one of the major things aside from the solving deployment, is solving infrastructure. We want you to just write the application, and we will compile dots to the right distributed systems for your service. And part of that is this query optimizer.

Steve:

I liked your post about getting the benefits of both static and dynamic typing, That's something that I find myself saying a lot, but I think it's kind of rare. I think you and I, we both had this conviction that there is a way to get the best of both worlds with the editor support.

Paul:

Right. Right. If you add tooling in... Current languages do everything in the language, because that's the domain that they have, and that's what they're able to work on. But some things are better done in an editor than they are in the language.

Steve:

Yes. That's well said. I feel like all the flame wars between static versus dynamic, we can just, we can literally end those wars because we can, you can have something that feels as dynamic as you want, but then on the back end, somehow, it could warn you that certain choices you made while prototyping, under certain conditions, that they could specify for you, will lead to errors.

Paul:

Yes. I mean the, there's kind of two aspects to this static/dynamic war. And so one of them is, how much can you guarantee is not going to... how many bugs can you guarantee cannot happen? And then the other side of it is like, what constraints are you going to apply to make that the case?

Paul:

And one of the major constraints that is applied by all these statically-typed languages is, you can only have one type of a thing in a certain place. So, in Python, a list can be anything; which has benefits, and has downsides. And in Haskell, a list is of a particular thing. Now it might be of of a particular thing that's quite general. With type classes or something like that. But it's of a particular thing.

Paul:

I think a lot of the backlash to C and C++ came, because the cost was high and the benefit wasn't all that. And I think when you look at the statically typed functional languages, the things that they move into the type system, in particular, exceptions and null, or the absence of null, I think makes a really, really big difference, and provides a really solid move forward from the other static languages.

Paul:

And this other movement that you have at the moment is gradual typing and bringing in things like TypeScript, and mypy, and static typing into all these dynamic languages. My impression from that is, that they're bringing in this very classic Java/C++ model into languages that aren't that at all. So when I was trying to mypy up the initial version of Dark, which was written in Python, and I discovered I did all this, all this dynamic things, and it just... They weren't represented in type system at all. So I think there's a reason that people are adding these types to these systems, because it's kind of hard. But I think the other way might be the way to go, rather than starting something that's super dynamic, and start to introduce all these constraints. What if you started with the super constrained thing, and then relaxed a bunch of them via tooling.

Paul:

That's essentially what we're doing with Dark. We're, we're starting with something that's roughly like OCaml or Haskell and saying, "All right, what can we relax here? In particular, what can we relax that makes for a better coding experience during different phases of it?"

Steve:

Mm-hmm (affirmative). Yes, I personally think of it as scaffolding. So I think you, like the... I guess to highlight, I think you do particularly well in Dark is, automatic unwrapping of, Maybe, or Option types. Because when you're prototyping, when you're building just for yourself, you don't want to deal with error states. Because like that's just... you know, you're like doing something... Like you just don't want to. It's not part of the thing you're building. You just want to deal with the success cases and then maybe, if this code survives more than five minutes, you'll deal with the error cases. And so yes, you do a great job of that in Dark.

Paul:

Right, exactly.

Paul:

Yes. And that's the really important thing. And the real downside that I feel that you get inside of functional languages is that you pay the cost for code that might not survive. And you don't do this in Python, right? In Python, if you're prototyping something, maybe it'd be for a unit test, if you're tests are in development, maybe you pay the cost up front. But, certainly, when I write Python I prototype it, I get it roughly in shape, and then I add the stuff to make it work.

Paul:

That option isn't really available in OCaml, in Elm You are forced to pay the cost up front, and then discover that, actually, your algorithm was wrong. Well you've paid all this static typing costs, and it would've been much better to not pay the cost, to discover that the algorithm is wrong, and then when you get the algorithm into the right shape, now is the time to add that cost back in. We also have the ability to tell you, "Did the thing that you're worried about actually happen? What is the priority of this?" So you can ship the thing that hasn't had all the edge cases taken care of, and we'll tell you which ones actually happened in practice.

Steve:

Yes. Let's talk about how that looks in the editor. I think you used the term railway programming. Railway?

Paul:

Yes. Railway Oriented Programming is this concept by Scott Wlaschin, I think is how you say his name, who is a an F# consultant and he has a blog post or he has a webpage all about F# and functional programming, which we should link to in the show notes. And whose name I can't remember. I think it's something like F# for Fun and Profit. So he has this concept as a way of teaching monads. So, it's a way of teaching this dual system of real values and error values, and he teaches it with the concept of a railway.

Paul:

And so a friend pointed this out to me a couple of years ago and I went through all the stuff and I was like, this is a really good model. Can we actually put something like this in the editor? And so how this appears is on the right of the editor, so on the right margin. That's where the rail is. And values can be on the rail or off the rail. Being off the rail means that you're handling them with a match statement, or with some of the monadic constructs like you know, maps and option ".then" that kind of thing.

Paul:

But if it's on the rail, we just unwrap it automatically. And then this is where we envisioned putting our built in exception tracker. So the thing that tells you in production all this went wrong and we'll bring you back to that particular trace. And then you can see in this trace, there was this data and it led to this thing being wrong here.

Steve:

Yeah, it's really cool. And I don't think this is in the product today. Well actually it might be. If I do like a search operation in the list, I guess. It's called like find maybe in JavaScript.

Paul:

Yeah.

Steve:

Will that automatically go on the rail? So it would assume that I found the thing.

Paul:

Yeah.

Steve:

Yeah. Okay.

Paul:

Yeah, so anything that returns an Option or a Result automatically goes on the rail. And you can pull it off if you choose to. So if you find that, or if you think in this case, you know, it's always going to be there and it's essentially an assertion violation that it's not, then leave it on the rail. And that's a case where you want to be notified if it's actually on the rail because you misunderstood something.

Steve:

And so if a user hits a rail, a condition that's on the rail is false for some reason. What happens to the request?

Paul:

So it will either 500 or 404. So for options it will 404. For results, it will 500 and show the error. So we're going to add more structured things. So it's like, oh, this particular error shouldn't be a 500. Or this particular error should have this error message or something like that.

Steve:

I see. I think we hadn't defined the terms. In option is kind of like a what Haskell, or Result is kind of-

Paul:

It's a Maybe in Haskell.

Steve:

Okay. Option is Maybe, and Results are?

Paul:

I think people in Haskell use the term Result. But if they don't they use Either.

Steve:

Got it.

Paul:

So a Result is an Either where, which uses the constructors Okay and Error instead of Left and Right.

Steve:

Got it. Cool. All right. Yeah, maybe it does have Results. I haven't used Haskell for web programming since college. So I don't know the state of the art of the type classes.

Steve:

So you mentioned briefly that you used to have what I call a node-and-wire editor, data-flow diagram.

Paul:

Yeah.

Steve:

Maybe I shouldn't, but I call it a phase that all of us kind of have to go through.

Paul:

Right, right, right. It's like grunge music.

Steve:

Yeah, exactly. It's like we all start with regular programming. Then we're like, we see the first one, or somehow we think about visual programming and that's kind of how we think of it. And so we're seduced by it and then we start making it or using existing ones. And then we're like, wait a second, this isn't all that I thought it was going to be. And then you switch back maybe to a structured editor or maybe you switch back to text. You're just done with it.

Paul:

Yeah.

Steve:

And so I normally advise people to just, either just go for it if they really are into it, or just skip that phase, but people don't really listen to me. Would you also advise people to skip that phase, or do you think that there's something there that could be useful? And that maybe if someone really worked on the node-and-wire thing they could figure something out?

Paul:

I was reading something over the weekend that talked about the act of software engineering as the act of like learning the problem space. So the artifact is partially the code, but it's partially what the team has learned from doing it. So I'm not sure I'd advise people to skip it. I think we certainly gained a lot from doing it. So our graph phase was around three months. And I think now we could actually go back and achieve a lot more than we did, because we figured out how to solve a lot of the problems that we had with the graphs, that we could probably back port.

Paul:

But what we did at the end of the graphs was we said, okay, what things are really good about this? And what things are really bad about this? And are the things that are really good about this, are they achievable in another way? And the things that are really bad about this, are they inherent flaws in how one does graphs? Or are they like things that we could figure out new features? And the problem that we had the whole way through the graph thing is that we kept putting off what we said was a bug was actually in some cases like a core usability problem that couldn't be solved. So one of the really annoying things about the graph editor was that we actually had to make perfect outcomes to discover that the perfect outcome still had bad usability.

Steve:

Sorry. What do you mean?

Paul:

So layout as an example. So we would lay out to the graph editor, and sometimes something would end up in a really bad place or be confusing. And we were like, oh, that's a bug. You know, sorry about that, user-who's-in-for-user-testing. You know, it's just a bug. And there's a lot of things where we were just like, "oh that's a bug." And eventually realized maybe this isn't a bug, maybe this is a fundamental flaw in how we do it. So we fixed all the bugs, and then we had users do the user testing. And they still came up with these programs that they were trying to write where the graph was unusable for it. Even though it was doing its job perfectly. We realized that we actually just didn't know how to solve some of these things.

Paul:

So there's a simple example. This sort of relates back to the Excel thing that we were discussing earlier. How do you write if statements with complex conditions? And when I say complex conditions, we were just trying to get people to write fizzbuzz, so not even that complex. So in fizzbuzz you've got, you know, if value mod five equals zero. And so it's two bin ops with a specific precedence. And we discovered that it was just really, really hard to write that because each bin op would form its own node in the graph. And so we were experimenting, well maybe for conditional expressions we can have a text-based thing. We actually wrote a whole parser for like single conditional expressions.

Paul:

And so we eventually got it to the point where users could write it, but then when they went back to edit it, we would expand it out to the whole thing. And then they were like, what the fuck is this? So it just like, it wasn't... Not only was it not easy, but new problems kept coming up all the time. And we didn't have really any answers to a lot of them because the model was just so different. And I mean fundamentally people were confused all the time. And we were confused as well. Like there was a lot of times that we would write things and we were like, yeah, I kind of don't know what I would expect to do here.

Paul:

So what we got from the phase was, there's some specific things that we're trying to do. So we're trying to represent what we call the architectural view. So we are trying to represent how data flows between different components in your programs, but that doesn't have to be done via a graph.

Steve:

Yeah, I don't know about other listeners, but to me it sounds like a pretty strong case to not do the node-and-wire phase. But I guess, I think a lot of people come from tools like Pure Data or other artistic tools that have a data flow graph that they had a lot of success with. And they didn't even a keyboard, you know, they were like clicking and dragging. And so they want to translate that same experience to full coding. But my hunch is that the reason it's never really worked is because full coding is arbitrarily complex in a way that I guess... I don't know, songs are pretty complicated, but I don't know if it has the same kind of Boolean logic and if-statements.

Paul:

Yeah, I mean, the things that we were trying to represent were like lambdas and requests that have data coming in and data coming out, and they sort of are in the same place. The input and the output are sort of in the same place. And we were trying to show what we call live values, which are sort of LightTable-esque data to the side of what data is actually flowing through this particular part of the program.

Paul:

And I think the initial case is like, it looks extremely promising. And the initial demo I made for Dark before we raised money, this was all we had but it was a two week demo of like, can you write code with this? And I was like, yes. So I understand what this is. Each node in the graph is a function, each line is an input. Very similar to Luna if you've seen Luna. But yeah, we couldn't make it feel good.

Steve:

Okay. I want to spend some time on the full vision for Dark. I don't know how much you can approximate it, but one day will it be... Like how far will you go? Will it one day be for writing the front end and the backend? Will you use it to write your apps for mobile devices? And then I guess maybe on the outer edge of what your vision might be. Will you one day have something that looks kind of like Coda in the sense that it'll blur the line between using software and making software? Or are you going to stop well before that and it'll still be, someone writes software and other people use that software?

Paul:

So I guess I should talk about them separately, but the answer to both is yes. So let me start with the with the Coda one. We don't plan to go into the low code, no code side of things. We don't really believe in that except for a couple of things like layout and design, and that kind of thing where we feel that no code is actually a good metaphor for it. But our intent is definitely that you don't necessarily need to make a tool to make the transformations to your state. So what I mean by that, we write handlers in Dark that don't take inputs. We called them REPLs, which is a slight cheat on what a REPL actually is. But where you can just write code and execute it.

Paul:

And then we save them and just leave them in the code. So an example of one of these might be adding a user. So a way to add a user in Dark is to open up your Dark program called the addUser function. And then that triggers all of the, you know, send them the email and put them into the account and all that sort of thing. And that's a way to do it, and its sort of, we want to build up more of that. So when you think of the admin dashboards that people build, like in Django or something like that, that's the sort of thing that we're talking about. And that may eventually become something where perhaps an individual, this is an example that we use in that sort of distant future, where a recruiter might have built their applicant tracking system entirely in Dark. And where they have a bunch of tools that they've written in Dark to do modifications to that. So it's not going to feel like the document-y metaphor, I think. But we definitely want to blur the line between how you write software and how you use software.

Steve:

Cool.

Paul:

So to the other question, does Dark go into front-end and mobile apps? So to a certain extent. I don't think that we're replacing Swift or React. We don't intend to. In fact, when we look at other tools in this space, and Meteor is one example. One of the things that led to huge problems for Meteor was that they were trying to be both frontend and backend at a time where there was a huge explosion and excitement in front end stuff, and that you couldn't use it in the initial version days. But they later let you allow support React in Meteor. So we don't intend to have Dark compiled to JavaScript that replaces React.

Paul:

What what we would like is this GraphQL stuff that you're writing, maybe we can make that easier. Maybe we can add an SDK to React or to Swift or whatever, which does a lot of maybe the transport layer. Maybe we can automatically optimize transport, maybe keep the types the same in your backend at your frontend, that kind of thing. And beyond that, really no idea where that goes. I don't imagine people writing single page apps in Dark though. At least not into some long distance in the future. And if that does happen, it will likely come from the community rather than that's something that we're planning to build.

Steve:

Just to be clear, I think what you mean is that people will be able to build a single page app with Dark as the backend, but you won't be able to build it-

Paul:

Sorry. Yes, yes. I meant, we don't expect that people will replace writing React or writing Vue, or Svelte, or whatever with writing Dark.

Steve:

Yeah.

Paul:

But they will use Dark for the backend. That is what Dark is really good for. It's really good for being the backend to a single webpage app.

Steve:

Yeah, yeah, exactly. I was pretty sure that you weren't-

Paul:

Yeah, yeah.

Steve:

Yeah. You just, you push your static assets to Dark or some other static place and then-

Paul:

Yeah.

Steve:

And then they just talk to Dark. Yeah.

Paul:

Yeah, yeah, exactly.

Steve:

So I want to push back on your claim that you aren't into low-code or no-code because I think that Dark is no code in the same way that Salesforce is anti-software, not software.

Paul:

Oh, you take that back.

Steve:

Well in the same way that Salesforce is not software. Because Salesforce clearly is software, it's just not the annoying parts of software that people use to think associate with software. It's like when a business person thought of software 20 years ago they thought of these massive contracts and this two-year-install period, and trainings.

Paul:

Isn't that what Salesforce has today?

Steve:

No, Salesforce today, well I guess it is kind of, yeah, that's actually true. There is a lot of that today. But the promise is that you just go to a browser and you sign up, and then you just started using it. And the software itself kind of disappears because you're just dealing with your customer data. That's like the dream, that you don't realize you're using software, you're just dealing with your customers. And so I feel like Dark is no-code in that way.

Paul:

What they had as... You know, what they were talking about when Salesforce said no software is like, you know, it's in the cloud. There's no installation. And you know that is sort of the same. And I think that's where the similarities end. I think no software was a very bad metaphor. But I think it was very good for their customers. With developers, developers largely do not want... They like code. They want to write code, and they want tools that make coding better, as opposed to tools that take away that coding.

Paul:

And so when you think about, so coding is kind of like symbolic reasoning. No code is write software without symbolic reasoning. So without the thing that we have spent years getting good at. And without the thing that is super powerful.

Steve:

Well, so-

Paul:

Low code is sort of like writing code with different tools that don't feel code-y. We very much intend to build simpler ways of writing code that feel like code, and that contain the standard metaphors of code.

Steve:

Well I think this is obviously a semantic, like discussion we're having.

Paul:

It is, absolutely.

Steve:

But I think it's a fun one because when I think about the essence of code, I agree that it's like symbolic reasoning of some sort. You know, like the idea of encoding your ideas into Boolean algebra seems pretty core to coding. If you're not doing that, then you're doing something else. But if you look at Webflow, which has put itself at the center of the no code or low code movement with its no code conference. They're like a CSS compiler. And they have animations that you can code in there. And they have a whole CRM. You know, it's very clear to me that Webflow is, it's programming, just with like a direct manipulation interface.

Paul:

So from that description I... So I think the word no code is becoming to mean, even though it's phrased no code, it doesn't actually mean no code. It means none of the annoying parts of coding. I think it means like no syntax errors and things like that. Because I imagine if I put someone who's never programmed-

Paul:

That isn't how I perceive the semantic... Or the meaning of no-code.

Steve:

I know that that's not how the word sounds, no code. But I just imagine putting my brother or my girlfriend, people who could definitely use Dark even though they could definitely not use any other backend tool. Like not even close. I feel like, yeah, so many of my friends could use Dark. And I think they might not realize that their programming. I feel like they might be like, oh, this isn't code. Because they don't know what code is, they just know that coding is a lot harder and they are a lot more error states. Like they've had some negative experiences with coding that they're not having here. This is like using a regular piece of software.

Paul:

I mean it's got a black background, so it's coding.

Steve:

Okay.

Paul:

No, but when I think of what's hard about coding, you're totally right. Like this setup, you know, the git and setting up NPM and bash and terminals and you know Vim. There's a lot of things that add to just huge complexity that you have to get over in order to write your first line, your first working application. And you can go through a tutorial. And you can end up at the other end of the tutorial with an app that does something and you have no idea what you did. That was just the stuff that you had to do to get there.

Steve:

But I think, sorry to interrupt, but just quickly. I think that what sticks with people, in like the medium is the message sense, is all that annoying stuff they had to do, that was the coding. And a little bit of code they wrote, like to get the hello world to show up, that was just for fun.

Paul:

Right. But that's what the coding is. When we talk about symbolic reasoning, it is the print statements, and everything that's around that print statement is crap. It's accidental complexity.

Steve:

So I feel like what I'm getting at is, if that's your stance, then the no-code movement, like there's something wrong with it. Because the no-code movement in theory is getting rid of all of the cruft. And also it's getting rid of the code too?

Paul:

It is, I mean, I think that's fundamentally it. That no-code is about, you know, they came to the conclusion that coding is hard.

Steve:

Oh, I see. Because they saw that the other stuff was hard. They didn't realize you could remove that stuff.

Paul:

Right, exactly. And so if you remove that stuff but kept the coding bit, and kept the ability to write during complete programs that can do anything. And you can see this, I mean, I saw when you did Coda for what I guess is in the Whole Code Catalog. But when we saw the preliminary version of that, there was a bunch of stuff that you couldn't do a lot of programs, you couldn't write, a lot of sort of logic that couldn't be represented. And I'm willing to stipulate that Excel is coding. Like it's sort of c low code. But underlying Excel, there's a lot of code. And there's a lot of formulas that you write. And it's stripped away all of the NPM install, the git init part of coding. Most of the no code tools I've looked at... I do not believe that they fundamentally allow coding under it. Which, clue's in the name.

Steve:

So where would you put Webflow in a spectrum? Because they-

Paul:

So Webflow I think is in the category of things where a visual editor is really a good idea.

Steve:

How come?

Paul:

So I think what no code is really good for is problems that are design related and problems that maybe we shouldn't have been doing in code in the first place. I remember when we were in college, we had this project and we were creating a Windows app. So someone wrote all the code to do it and it was buggy and it was hard to get right. And then instead we used a visual studio built-in, drag and drop thing. And it created a ton of code, and all the code was unreadable and so on. But it worked really well.

Paul:

And you see this kind of in Swift as well. It's hard to write that code. And there are problems that actually maybe aren't really intended to be solved by code anyway. That a visual editor is ideal for things like, how do you lay things out? So I think of Webflow as a great solution for that particular problem. So the particular problem of, how do we build websites? I don't think of it as a particular great problem for, how do we architect front end applications? I mean, which is a problem that it isn't trying to solve, but when you've been writing React or single page apps for awhile, you start to understand that the architecture of this application is super important and it's hard to do. And there isn't something about inherent to low code or to a specific visual paradigm of editing that helps with that problem.

Steve:

All right, that makes sense. I wanted to ask you to talk about some of the downsides of Dark. I liked how you listed that was one of the blog posts you're going to write, like explicitly calling out all the downsides. And you listed open-source, sel- hosting and potential lock in. I think those are some of the biggest ones.

Paul:

Yeah. So the biggest thing is probably that one. So let me start with a smaller one first. So people are going to have to do things a little bit differently in Dark. And specifically they're going to have to leave their favorite language and they're going to have to leave their favorite editor. And I don't think anyone really has a favorite architecture, but-

Steve:

People like GitHub.

Paul:

Yeah. And they're not using the git flow workflow or the GitHub pull request workflow. They're not writing code on their machine. And coding in Dark feels a little bit different. On your first day you're going to feel a little bit uneasy because everything's live and you don't have this thing where you don't have a commit that has the "this thing is done" kind of moment.

Paul:

So yeah, things feel a little bit different, and some of that you'll get over. And some of that is, if you've got this super customized Vim or Emacs or VS Code or whatever set up, you know, Dark isn't going to support that at the start. And we may support similar things at the end, but we're definitely not going to do it at the start. Especially if you're a Vim user and you're used to all the Vim-y things. We may eventually have a Vim mode. We certainly will not on day one.

Steve:

Maybe we'll start with the Emacs mode on day one?

Paul:

I mean, actually, if you're used to Emacs things, things like Control A, Control E, you know, work nicely and the read line shortcuts, and there's things like Alt X to bring up the refactoring commands. So it does feel a little Emacs-y.

Steve:

Oh wow, I was joking.

Paul:

Oh, yeah. And the intent actually is for Dark, for you to be able to write editor extensions in Dark, which is also very Emacs-y. So yeah, so that's going to be one thing. People are going to have to leave some tools, and that's a thing that people on an individual basis can decide to do, or not to do.

Paul:

The more major one is of course, are you locked into Dark? And what are the options around that? And fundamentally for us, it's not a good thing for you to be locked into Dark. Like your product should stand on their own merits. And if people want to leave for whatever reason, and we are aware that we are definitely going to churn customers. You know, that just happens.

Paul:

At Circle I remember that our first customer who got to spending a million a year churned. On our second customer who got two spending a million a year churned. And then we finally had figured out and solved all the problems. And now customers who get to a million a year don't churn. But you know, that is a life cycle that most companies go through where the first people who need a certain thing, if they really need it, that you may not be able to solve their problem and they may need to leave for that. And one of those is scale. So I have no doubt that people will churn. And we want people to be able to churn, mostly because it lowers the risk of signing up to Dark. We don't want Dark to be only for people who are backed into a corner, who don't have the resources, and Dark is the only way to achieve what they want to achieve.

Paul:

We want it to be our conscious choice. You're looking at Dark for its advantages and disadvantages, and you're comparing it to Go, or Node, or whatever on whatever stacks they come with.

Paul:

So, we have looked at various different ways of solving this. The one that people ask specifically about is open-sourcing Dark. When we look at the advantages of Dark and what Dark is, the main thing about Dark is that we run it ourselves. That we host it, that we run the infrastructure, and we've talked a bit about compiling queries, and optimizing that infrastructure stuff. That's a very intentional part of Dark and there's many tools today that don't do that you can run yourself.

Paul:

So, the other factor with this is the current movements in open-source around, especially with open-source databases, around who gets the value out of the ecosystem, and increasingly the answer is Amazon. It's super important to us, especially from the perspective of this lock-in question, Dark needs to be sustainable and it has to be something that builds on itself, and that the company or the people behind building Dark needs to be a sustainable ecosystem for this to be usable for people. We saw what happened with Parse as an example of something that wasn't sustainable. It got acquired by someone who didn't have a strategic value of it. And for people who build on Dark, having Dark be sustainable is existential risk.

Paul:

So, we are not planning to open-source Dark at the moment. Which would be one way of dealing with that lock-in. The way that we intend to deal with that lock-in in the short term is dealing with... So there's two kinds of cases, there is Dark dies, right? Dark isn't able to raise money to some cataclysmic events, and Dark has to shut down. So what do people do then? So in that case we're looking at creating some legal framework that says that we will open source Dark if it shuts down, and maybe even setting aside money or creating a fund to support us during that phase to give people the ability to start a hosting thing, or to run it themselves, or just have some smoothness as they move off it.

Steve:

I like the idea of like social security. When you pay for Dark, 10% of what you're paying for is just in case Dark shuts down. Like getting off of Dark.

Paul:

Yeah. So, absolutely not 10% but yeah. I mean we're looking at it more as a "how many people could we keep this in maintenance mode with, and we need to pay their salaries" and, well I guess I'd be doing it for free. But yeah, there's other people whose salaries we'll need to pay.

Paul:

And at the start when users signed up and we had our first user our second user our handshake agreements with them was if this doesn't work out, we will rewrite the code for you in Node or whatever, and we'll help you get it onto AWS Lambda or something like that. And we want a variation of that. We can't hand write it, but we do want something that if you decide to move off Dark, there's a way to get it onto Lambda or some other framework like that. So this is the situation, not so much Dark shuts down situation, but Dark succeeds but you can't use it for whatever reason.

Steve:

Got it.

Paul:

And so compiling it out into a Node app or a Go app is one option that we've looked at. Compiling it out maybe into a binary so you have to start the code again, but you have the ability to keep running the current thing. All these solve a subset of problems, they're not a complete solution. So we're interested in hearing from people like which of these solutions are actually interesting, and from people that would consider them. I know there's a lot of people who just wouldn't consider them, but I guess that's interesting information as well.

Steve:

Cool. Yeah. I really appreciate how honest and up-front you are with this choice and the trade-offs.

Paul:

Fundamentally lock-in sucks for for us, because it prevents people from adopting. So having a solution to it matters to us.

Steve:

It's funny because that goes against the common sense or common wisdom that lock-in is good for companies.

Paul:

I don't think that lock-in is good for companies. I think people start to resent the companies, and it spends whatever consumer goodwill when they have lock-in. You want lock in from being such a good product that people just can't use anything else. My philosophy of businesses is that lock-in from a monopolist standpoint sucks. So many people hate Facebook and use it everyday, because there's no alternative or look at people's feelings about Verizon or Comcast. There's a certain point at which companies turn to rent seeking, and lock-in is used for rent seeking. And that's the point at which innovation has long died at the company. And yeah, we absolutely don't want that to be what Dark is.

Steve:

But at the same time you fear that if you allay all lock-in fears and go open source, you won't be able to be a sustainable business, because people will just not pay for it. They'll just self-host the open source version.

Paul:

I mean the fear that we're looking at is that Amazon will host it.

Steve:

I see.

Paul:

And this happened with Elastic, this happened with Mongo, and it's not even that Amazon is bad. No doubt they're doing the things that their customers want them to do. But we have a responsibility to our customers to make this sustainable. So open sourcing it-

Steve:

I just wanted to-

Paul:

Ironically isn't necessarily the most sustainable approach.

Steve:

I'm wondering if an open source license, a particular open source license might, you'll just tell me why it doesn't work. Could you open source it in such a way that if I'm using it as an application... Like if I build something in Dark and I want to self host it, I can do that, but if I want to sell it as a platform in the way that Amazon Web Services would, that's not allowed?

Paul:

Yeah, so this is what a lot of these database companies are experimenting with. So they're source-available licenses, they're not technically open-source, because open source has has a specific meaning, and I think it is not possible to be technically open-source in adding that provision. But I think that we're going to see a shift to this source-available thing. There are certain expectations that people have for something being open source, like their ability to contribute to the ecosystem. And that may or may not exist in source-available licenses there.

Paul:

There's a certain thing about open source that it's much more than the legal definition, and it's how you treat your community, and how you build software and that sort of thing. And there is an element to that, that will existing in Dark already. So there'll be a package manager in Dark and they'll be the ability to create and share libraries. So I believe that we will have many of the advantages that people expect to see from open source, but that running the platform themselves is, is not necessarily... Or I don't think there's anyone who wants to run the platform themselves, including people who get too close or get too big for the ecosystem, or people who get too big for our hosting thing. If we can't make it work, it is unlikely that just hosting it themselves is going to solve this problem, except for things like on-prem and that kind of thing. And there's other ways to handle those.

Steve:

Oh yeah. How was the on-prem, like when for legal reasons you need your servers to be on-prem. Do you have a solution for that? I guess Salesforce...

Paul:

On-prem has had many changes in definition over the last decade or so. So it used to be on-prem meant like physically on the premises. Nowadays on-prem is running in an Amazon VPC that you control. So that when when you shut it down, you get to shut it down, and the company can no longer access within it. I think we're probably going to stick with more of the public cloud thing, but perhaps the lines will become so blurred that there's a thing where we can address your issues in a way that, is manageable within the model that we're looking to build.

Steve:

Got it. So one day will you... Is the future of Dark, you're going to compete with AWS directly, and run your own data centers, or you're going to always be on top of some other cloud service?

Paul:

I think that we will always have some amount of things running on the cloud. So in terms of business model, we're absolutely competing with AWS. We think AWS and it's 200 services are, not all of them, but a large majority are accidental complexity and that people shouldn't have to do that in order to write code or in order to receive data, and process it, and store it, and send it. Should be much, much simpler than what exists, and what we're building is essentially a competitor to AWS. The question of when and how we build our own data centers. There's so much like that are in these public that people kind of need. So building all the data centers that currently exist is hard.

Paul:

But even if you were to build them, a lot of what people need is: I need this app to sit next to my existing app, which is in US East 2 or something like that. So it isn't just a case of you can build data centers and replace this. I think there's a hybrid world. I think that we will start looking at optimizations including cost optimizations at some point. So that will move a certain amount of traffic or of computer storage off of AWS. I mean we're not using AWS. We're actually using Google Cloud, but move it off those things or move between those things or allow you to specify constraints where perhaps you don't care about the service at all, but you'd like it to be cheap.

Paul:

It doesn't matter how available it is. Well as long as it's somewhat available. But like you can tolerate a certain amount of down time if it can be run for a third of the price. And in that case why wouldn't we run it on Hetzner or one of the budgets infrastructures rather than paying for the full availability that you get from AWS or GCP instead. So we think that like lots of these are options in the future, but how we think about Dark is a compiler for your infrastructure or compiling your app to the ideal distributed system for what you want. And some of those constraints are cost and some of those will be eventually, yeah, we're, we're going to move off. And I think that we're also going to get a couple of really nice economy's of scale from that optimization. Even staying on the cloud, we might know that there's a certain kind of constraints that you're looking for here mean that we should use this service rather than this service, and this one is a lot cheaper.

Paul:

I think that at some point we'll get to individually compiling databases for your workload and rather than running it on Postgres, which is what we do now. And that we'll realize, orders of magnitude of performance improvements, and cost savings from doing that.

Steve:

Cool. Yeah, it sounds pretty exciting. And now I'm wondering if someone really buys into this vision, which I think a lot of your early customers might. Have you thought about running some customer-fundraise where they could somehow get a piece of the upside if you were able to win?

Paul:

I would be interested in doing that. We'll be fundraising in the near future. It is very challenging to raise money from non-accredited investors.

Steve:

Legally?

Paul:

Legally. It creates a lot of challenges for your company. And so I'm not sure whether we're going to do that. We have a lot of people who are interested in giving some money. So it's not a thing that we need to do, but it's a thing that I like doing from that upside thing. But it might be legally too challenging.

Steve:

Yeah. Yeah, that makes sense. I don't expect you're going to have trouble raising funds from very smart investors, but I feel like you could get some amount of customer loyalty, extra points.

Paul:

Right, right. And I believe in that. I want to do that with Circle. To a certain extent I want to do that for Circle customers today: invest in Paul's next thing. Yeah. It's unfortunate that the legalities are annoying there.

Steve:

Yeah.

Paul:

Yeah.

Steve:

Okay. Two more questions. Hopefully these ones will be quick. I wanted to quickly talk about the tech stack, because as a compiler / programming languages person, I imagine you've put a ton of thought into it. Before you talk about this tech stack, I want to talk about the tech stack at Circle simply because that was how you and I originally met. I don't know if you remember, but I did some search on Hacker News for just the word Clojure, and to see if there were any companies that were hiring for Clojure developers. And that's how I found you initially. And so I'd be curious to hear about why you chose Clojure, and why you didn't choose Clojure and ClojureScript, and why you switched to the OCaml ecosystem. All your thinking. And your experiences with Elm and moving off of Elm, all that, that whole story. I guess it's not a quick question.

Paul:

So initially when Allen and I started Circle -- that's Allen, not Ellen, who's my Dark co-founder. I was working on it nights and weekends, and he was working on it full time. And so we wrote it in Clojure, which was his favorite language. And I was initially against that, but after about six weeks of getting to know Clojure and Emacs, which I learned together, I came to realize that Clojure is this beautiful elegant language, and that Clojure is great. And so our initial front end was written in Backbone and JavaScript, and we converted that into ClojureScript using a model that's very similar to the Elm architecture. That was a couple of years later in 2014, and yeah Dark at a certain point was a Clojure monolith and at some point it forked various things off.

Paul:

And the problem that I found with Clojure, aside from things like, having to use a REPL, because the spin-up time is annoying for the JVM and that kind thing. That's all, tolerable, but if irritating. The major problem that I had with Clojures is the lack of static typing. And this guy, Ambrose did a lot of work on Typed Clojure, Core.typed it was called which Circle sponsored, or Circled sponsored in an amount of, and I think he published it. He published his PhD on it late last year I think or maybe earlier this year. And it's adding gradual types to Clojure. Never ended up being good enough in the experiments that we did. And we ended up with a lot of problems where there was just a nil somewhere and the nil propagated for miles and miles and miles because that's what Clojure does to nil. If you, do things with nil you end up with nil, and eventually you end up with nil, and where did this nil come from?

Paul:

And yeah that was really annoying. And I had feelings that static typing was the solution to this. And so I played around with Haskell for awhile and I'd written a bunch of Haskell years before during my PhD. And I really liked this feeling that once it compiles, it doesn't necessarily work, but like it often works or is pretty close to working you. You never have to deal with a thing like in Clojure or Python where you got the arguments wrong ,and you made a change to something, and now there's this other path just doesn't work at all. And so you have to write tons and tons of tests.

Paul:

But I never was really satisfied with Haskell as a language. I felt that Haskell brought a ton of its own accidental complexity. And so I wasn't happy with it. I'd written a bunch of Elm and Elm I was super happy with. Elm is just such a beautiful language. It's so well designed, and looking at Elm and at how productive I could be with it. Looking at Haskell, I was like this, "There's really something to this statically typed functional language." And so the one that we hadn't tried was OCaml. So when we started Dark, Dark was about 800 lines of Python and we tried to add mypie to it, and just didn't get good results out of that. So rewrote it in OCaml. There was a very high learning curve for OCaml. There's a lot of problems with OCaml, but we approached the tech stack for Dark with the idea of: none of this is going to survive.

Paul:

So we are solving today's problems, and tomorrow we will solve tomorrow's problems because that was the phase, that was the early phase of the company that we were at. We're now switching into a different phase of company, where we think a little more into the future, possibly a lot more in the future. But at the time it was like we have no idea how much of this code is even going to survive at all. So let's write it in a tool that allows us to write things as quickly as possible. And a particular advantage of OCaml for early stage projects is how easily it lets you refactor things. So it's not at all unusual that I'd write 2000 lines of refactor, and that I had the compiler working with me together working, and that at the other end it mostly worked, or 99% worked and there was a couple of bugs leftover.

Paul:

So, OCaml has been great for that stuff. It has not been great for operationalizing. In particular, it's concurrency story: it's very hard to write concurrent code in what you think of a standard OCaml. You have to write this cooperative, it's called the LWT. You use this cooperative Async libraries that completely change what your code looked like and how it feels to write code.

Paul:

So yeah, I don't like where we are with OCaml. Our current view is, that we'll be rewriting it in Rust in a relatively near future. And there's a couple of different plans that are some amount of OCaml left to no OCaml left at all. But we haven't made that decision yet.

Steve:

And what about on the front end?

Paul:

So the front end, we started with Elm and then we switched to BuckleScript, which is a kind of OCaml that compiles to JavaScript. Where we're quite happy with that. It has solved a lot of the problems that we had with Elm, in particularly the lack of native code. I think that there's a possibility that that we move out of OCaml at some point to something along the lines of TypeScript. But we haven't felt that need yet at all.

Steve:

Got it. That makes sense. And then in terms of where you're rendering your Dark canvas, is it on the HTML canvas or is it HTML Elements?

Paul:

Right now it's HTML Elements. Talking to people who have built large scale editors it feels like, there's a future that's WebGL, but we haven't done any experiments with that.

Steve:

Cool.

Paul:

Right now it's just pure HTML with virtual DOM that handles most of the performance.

Steve:

Great. And I guess that just leaves me with the final question: maybe list some of the places that you could be reached online, and then what sorts of things that you want people to reach out to you for, if you're hiring, or other things you want to let people know about.

Paul:

Yeah, so we're hiring. We're hiring a lot of roles at the moment, not all of which are online, but they will by the time this blog or this podcast comes out. We are looking for users, so in particular people who have a deadline for an app that they need to write. So rather than kicking the tires. We're have lots of people offering to kick the tires, and we're really focusing on people who have an app they need to build right now, and that they need to ship soon.

Paul:

So places that we can be reached online: darklang.com is the obvious one, and signing up for our mailing list, signing up for our beta, and applying for jobs. A good place to do that. By the time this comes out, we should have a lot more information on the website about Dark, but we have a bunch of blog posts that are currently on medium.com/darklang. And then on Twitter we are @darklang. I am @PaulBiggar and Ellen is at @EllenChisa. Those are the ones I can think of.

Steve:

Great. Well this was really wonderful. I'm glad we finally found the time to do it.

Paul:

Same, same, same. Yeah, lovely having this chat with you. I really enjoyed it.

Steve:

Great. Talk to you later.

Steve:

If you enjoy these conversations, I bet you'd fit right into our online community. You can join our Slack group at futureofcoding.org/community, where we chat about these topics, share links, feedback, and organize in-person meetup groups in various cities, New York, London, San Francisco, Boston, and others. If you'd like to support these efforts, there are a few ways that you can be helpful. I would really appreciate any reviews that you could leave for the podcast, wherever it is that you listened to it. If any episode in particular speaks to you, I would encourage you to share it with your friends or on social media. Please tag me if you're sharing it on Twitter at @SteveKrouse so I can join the conversation. I also accept support directly on Patreon at patreon.com/stevekrouse. And as always, I'm so thankful for your constructive criticism and feedback, so please don't be shy with your advice. Thank you so much for listening. I will catch you next time.