49 • Ravi Chugh • Sketch-n-Sketch

2021-03-09

Listen in your podcast player by searching for Future of Coding, or via Apple Podcasts | Overcast | RSS

Ravi Chugh is a (recently-tenured 🎉) prof at the University of Chicago. He’s famous for leading the Sketch-n-Sketch project, an output-directed, bidirectional programming tool that lets you seamlessly jump back and forth between coding and directly manipulating your program’s output. The tool gives you two different projected editing interfaces for the same underlying program, so that you can leverage the different strengths of each. In the interview we talk about the principles of bidirectional editing, the team and history behind the Sketch-n-Sketch project, benchmarks and values that can be used to assess these sorts of novel programming interfaces, possible future directions for Sketch-n-Sketch and the field more broadly, and a bunch more. It’s a long one — almost two and a half hours — but it’s packed with thought and charm.

Now, this episode is a bit of an odd one, though if you’re just listening to the audio you might not notice anything usual. In fact, it was recorded over a year ago, before the pandemic hit Canada (where I live). I’d planned to release it in the spring, but hit some snags, and then got burnt out, and then the next thing you know it’s March of 2021! An entire year has just… disappeared. Weird, hey? So the episode is coming out, finally, though in a slightly different form than previous episodes. In the past, I’d spend something like 20 hours meticulously editing the transcript to fix errors and make up for the loss of inflection and subtlety of speech, building up a handy list of links to all the things mentioned, and otherwise making this episode page stand alone as a resource independent of the audio. But due to the aforementioned burn out, I won’t be doing much of that anymore. The thing I’m passionate about is the audio, and the thing that’d keep me from releasing episodes at all is the effort it takes to make these episode pages. So if you find yourself reading this, and if you feel up for helping, I warmly invite you make edits here.

On that note, I need to extend a heartfelt thanks to Kartik Agaram for putting in a ton of work to clean up the transcript over the past few days, and to Ravi for a huge drop of edits and fixes.

Links

Ravi Chugh is here to bake cookies and talk about Sketch-n-Sketch, and I ate all the cookie dough. You can find Ravi on Twitter.

Here is Ravi’s brief research statement that was drafted for a tenure review, and a slightly longer slide deck that covers some of the ideas behind Sketch-n-Sketch (and includes a lot of swanky graphics).

There are lots of other folks in Ravi’s group working on Sketch-n-Sketch and related projects who will be familiar to active members of our community:

Brian Hempel
Mikaël Mayer
Justin Lubin
Nick Collins
Cyrus Omar (who also appeared on the podcast)

Here’s a fantastic presentation of Sketch-n-Sketch by Brian Hempel at UIST 2019.

And here’s the matching UIST paper.

Also fantastic is the 2016 Strange Loop talk by Ravi:

For valuable benchmarks and thoughts about programming by demonstration, check out the book Watch What I Do by Allen Cypher. Benchmarks were also discussed at LIVE 2018, after Brian presented this video.

Transcript

Transcript sponsored by Repl.it

Corrections to this transcript are much appreciated!

Ivan

Welcome back to The Future of Coding. I’m Ivan Reese, and my guest today is Ravi Chugh. Ravi is an associate professor at the University of Chicago. He leads the group working on Sketch-n-Sketch, a programming environment that fuses direct manipulation with text-based code in a paradigm called bidirectional editing. You have a text editor on the left, and a graphical canvas on the right. When you draw on the canvas, it generates code. When you edit the code, it updates the drawing. When you manipulate the drawing, it updates the code. You have two representations of the same information, both equal in importance, but different in how you can work with them. Sketch-n-Sketch is starting out life as an environment for working with SVG graphics and HTML webpages, but there’s a lot of promise to this idea expanding into other domains in the future. I’ll let Ravi take it from here.

Ravi

What Sketch-n-Sketch looks like, what it currently looks like at least, is a pretty traditional two-pane interactive development environment. On the left half of the window, on the left pane, is just an ordinary code editor. A text editor. Really not much different from what you would normally expect. The right side, the right pane, is a completely ordinary canvas on which the output of the program is rendered.

Currently, the Sketch-n-Sketch interactive programming system is tailored to the domain of generating SVG graphics, or also HTML pages. But in the program and the text editor on the left, at the end of the day, the main expression has type SVG or type HTML. And after the user runs that program on the left, the output value — the main SVG or main HTML value — is rendered on the right pane on the canvas graphically, as it would in any other direct manipulation or drawing system.

In that sense, it’s a completely traditional, ordinary programming environment. You write a program, you run it, and it shows the output. The main new features that we’re exploring revolve around making the output of the program editable — manipulable — as opposed to just being the final value that the program spits out, and then it’s inert or disconnected from the program that generated it.

The main goal in Sketch-n-Sketch is to allow the user to actually interact. To change, to drag around things on the output to make changes, and for the system to infer changes to the program to match what interactions the user has performed.

One way to think about it, one way that I like describing it is, in addition to the normal, forward evaluation process in any normal programming language, Sketch-n-Sketch is trying to provide this backwards connection. Mapping changes back from the output, back to the program that generated it.

Just to talk about other interaction modes a little bit, to overview the system: in addition to being a normal programming language in the sense that you start with a program and run it, and then after you’ve viewed the output you can start interacting with it, we try to also provide this backward connection in a new and intuitive way — the system provides the ability to actually add new things, new output values to the output of the program, that were not even in the output before.

Let’s say you had a blank program to start, and then initially there’s actually nothing on your canvas. The Sketch-n-Sketch editor also provides a drawing toolbox of parameters that you might expect in an SVG drawing tool, where you can add new shapes to the canvas, and the system will auto-generate or insert definitions into your code. Which, again, if you would run them in the forward direction, would produce hopefully the values or the shapes that you’ve just added in the output.

The goal is to allow both directions, of both programmatically generating but also using direct manipulation drawing tools, to move back and forth between these two directions of authoring.

Ivan

In the Strange Loop talk that you gave back in 2016, you did a really good job of justifying why this tool needs to exist, and why this bidirectional editing needs to happen. That is: in a normal drawing tool, when you make your drawing, if your drawing has something systematic about it — you give an example of a Ferris wheel with a number of cars around a circle — if you want to change the number of cars, you have to manually go in and move everything around. Whereas if you’re doing procedural graphics, it’s very easy to say, “Oh, I have a number that represents how many cars are around the circle.” But it’s hard to do quick tweaking and adjusting, getting things to look the way you want, if you’re having to use code. You have to make a change and then rerun it, and a change and rerun it. Or you’re iterating your way towards what you want, rather than just doing it.

Yeah, absolutely. That’s one of the really fun parts of this project, I think. It’s got this really wide range of techniques and challenges that all need to be solved, in order to really provide this long-term goal of mixing in a fine-grained way, the best of what GUIs and direct manipulation offers and what the best of what programming offers.

If one wants to try to combine those in a fine-grained way, that’s going to require both advances in programming languages and program synthesis. It’s going to require advances in UI design, and taking into account this conversation between the programmer and what the system can do. If you look at the directions that we’ve been exploring the past few years, it’s a variety of what you could think of as core PL program synthesis kinds of research questions, as well as core user interface questions.

We’re certainly excited to keep pushing on directions in all of these fronts because I think it’s pretty clear that one needs to think about all these in tandem, and not just from the programmer’s point of view or from the designer, the end user’s point of view.

Ivan

Before you started working on Sketch-n-Sketch, what did you see in the world that led you down this path? Did you play with an existing tool and think, “I know how to make this better.” Were you chasing a feeling? How did you arrive at the desire to work on this?

Ravi

It’s a fun question for me to think about because I do think about the original motivations for this project quite often. I think the moments that I look back to are pretty specifically centered around my time in graduate school, where I was doing research on program analysis and types systems. Something really not super related to what I’m working on now.

But, in the process of grad school, I would give conference talks and I would give lectures and things like that. So, I would always use PowerPoint and happily use PowerPoint to create visual, interactive presentations about whatever topics I was talking about. I’m certainly not qualified in graphic design or visual arts, but I always enjoyed the challenge of coming up with really interactive visual ways to explain the ideas about whatever research topic I was talking about.

So, I just really enjoyed how I can do that in PowerPoint because I can really easily try out a bunch of different visual representations. The built in animations often provided a good way to stage transitions and sequence the story of my talk. I always really enjoyed PowerPoint. But, every time I would use it, I would also think, “Okay, there are a whole bunch of operations that are really tedious to perform. And, once I’ve figured out what the basic design of my talk and my visual motif is looking like, it would be much easier if I could make certain changes programmatically instead.”

Instead of making a change early on in my, let’s say, sequence of slides, if I make some fundamental change to the visual motif, instead of having to go back through all the 10, 20, 30, 40 places I copied and pasted and made changes, if I could instead go into a programmatic representation of that sequence and make changes in one or two places, it would be so much easier to build these really complex visual narratives and presentations.

That desire really came up over and over again in grad school. I remember talking with my friends at the time about, “Wouldn’t it be great if PowerPoint was completely programmable?” If under the hood, there was a general purpose programming language that I could use to help interactively build these presentations.

I think, that was the first time that I really thought about this, wanting this combination of, as a programmer, knowing that general purpose programming languages provide all these abstraction capabilities that make certain things easy. But then also, using terrific direct manipulation tools and realizing how they made a different set of interactions very easy.

Some of the examples that I like to use, in the Strange Loop talk, in the recent papers that we had at UIST are, think about even the basic drawing tools in your drawing editor. Drawing a shape, drawing a circle, drawing a polygon. Even those, you might want to define variations of those primitives and have them be the tools that are presented in the toolbox. What makes those definitions of shape and polygon the only way that you can imagine a rectangle tool working?

There’s other variations where certain properties of the rectangle would be different from the defaults that are provided for you. If these primitive tools are instead defined as, let’s say, functions in some programming language, then these tools can actually be libraries that are provided to users, but could be swapped in for other libraries for use cases that might not be the use case for every single user of the tool.

Let’s say I’m in some project where the primitives that I want to use actually have these seven sided stars and snowmen or something. If the GUI system allows these primitive tools to be defined by user or library defined functions, if I’m a programmer or if I’m working with my team of designers, I can define these library functions for seven sided stars and snowman shapes and add those into the toolbox so that we can use those as direct manipulation drawing tools, as if they were the built-ins that had been provided for us.

Clearly, a user interface can’t provide every single tool that every user might possibly want. At some point, a user interface, an application software will have more and more tools, more and more menu items, more and more options. That’s, at the same time, not going to cover every single thing that users might want to do. And, it’s not the most scalable way for a user to use a system, because sometimes, you might really only want five of the 15,000 features that are there. It seems like you want to be able to make user interfaces much more customizable and much extensible so that different users and use cases can customize the UI to do different things.

Ivan

In the Strange Loop talk, one of the demos you give is that you build the lambda logo for Sketch-n-Sketch, used a handful of geometric attributes that were exposed by the graphics editor, like the positions of points, the widths of rectangles, that sort of thing. More recently, in the UIST talk, it looks like you’ve gone much further down the path of exposing geometric attributes. Like the midpoints of lines that you can use to snap another line onto. These additional attributes look like they make it easier to build complex relationships between the shapes. But, at the end of the talk, Brian references this book about programming by demonstration. In that book, there are a number of benchmarks that you can use to evaluate a programming by demonstration system. When he mentioned those benchmarks, he said there’s four of them that Sketch-n-Sketch can currently do perfectly, there’s two that it can kind of do, and then there’s nine more that it can’t really do right now. In order to do some of those nine, a number of features need to be added. One of those features being attaching the end point of one line to an arbitrary position along another line. That sort of feature addition feels like a game of cat and mouse to me. It feels different from what you’ve just talked about, about adding different kinds of preset shapes and that sort of thing. This feels more like needing to change the underlying representation of the graphics or changing the way that the graphics map to the abstractions. Is it something that you feel like you’re working towards making open-ended as part of the system? Or is that something that’s going to be baked in and the end user is not going to be able to add those sorts of additional capabilities to your vector representation?

Ravi

That’s a really good question. The short answer is yes, I think we can and will want to make those choices and those kinds of features also exposed to users or library or tool builders to customize. If you think about even the simplest widgets that you might draw onto a rectangle, let’s say. So, the completely standard feature that you might draw and allow the user to manipulate the corners of the shape, and maybe the midpoints and the center.

It is the case that currently our editor, Sketch-n-Sketch, draws predefined sets of features or predefined widgets for the different kinds of shapes. But you could certainly imagine even those widgets, even those choices to be defined in a library instead. You could imagine there being a library function that describes what to draw on top of the primitive unadorned, undecorated rectangle in the SVG output. And that library function could choose to draw SVG circles or SVG rectangles or whatever it is, that happened to be exactly at the corners and midpoints and centers of those shapes.

You could imagine then, let’s say, choosing a wrapper around rectangles that don’t draw any widgets at all, because let’s say you know that certain rectangles in your output are never going to be interacted with. So, why ever even have the user interface clutter that view with extra widgets? And then, you can imagine in some other part of the design, you have more knowledge about, you’re going to want to be interacting with maybe not even the midpoints, but maybe an arbitrary point on the edge.

You can imagine overlaying the right widgets on the edges of those polygons or on the edges of those shapes, and then hook those up to what the algorithms under the hood that connect to the output value to the program know about. So, I think there’s certainly details about connecting what the user-defined functions have chosen to draw on top of the real, the main values and how to map those interactions to what the underlying program synthesis and program repair algorithms can do.

I think there was a little bit of extra metadata and other kinds of things that you’d have to find there, but I certainly think that this approach would allow those user interface widgets, and actions to certainly be customizable and changed by users and libraries.

Ivan

So, to build something like that, it feels like you’d need to get at the core ideas of whatever your output is, the fundamental first principles. In the case of vector graphics, it might be you have to distill everything down to a point, and then the idea that points are connected in lines, and then bootstrap your way up to all the vector graphics from some fundamental seed. Sort of like that. Does that seem like a fair characterization of what would be needed in order to make that idea work? If so, how would you apply that to other domains?

Ravi

That’s a great point. I do agree that’s a fair way to describe it. I think I mentioned this in the Strange Loop talk as well. I think the way that we see this is, I said that we want this bidirectional connection between programs and their outputs to work in many domains. To do that in a scalable way, it seems that there are going to be certain operations, certain connections, certain changes that ought to be common across whatever application domain you happen to be working on.

But then also for any application domain, there’s going to be custom program analysis, program repair techniques that cater to the kinds of programs that are written in that domain, and also custom user interfaces to expose those capabilities. I think it really is this combination of some set of general purpose tools that are going to be useful, no matter what you’re programming or building. And then, certain tools that are useful, that are specifically designed for a certain domain.

Like you said, for SVG, if one wanted to expose a completely configurable, reconfigurable UI for doing vector graphics, you might want some really general representation, like points. On top of which you could then build basic structured shapes. But then, you would have really the finest grain access to be able to specify constraints over individual points in your output.

I think it is fair to say that for each, let’s say, different type of value in your application domain, or for each application domain, you identify the primitive values in that domain. That defines what the users and libraries can operate over. Currently for SVG, we choose just the normal SVG primitives as our…

Ivan

As our spec.

Ravi

Yeah, exactly. The spec is our domain of values. If one wanted to expose really complex constraints over individual sub features of these shapes, you might have something like you described.

Ivan

That disentangling of the things that are common across domains from the things that are unique to each domain, and coming up with the representation of the things that are unique to a domain in order to expand Sketch-n-Sketch, or a successor or similar tool to work in that domain… That sounds like a very, very hard problem on the level of SDF, or the semantic web, or something like category theory. Or any of these notational systems that are designed to separate out the structure of what constitutes a domain, as opposed to the instances of things that have those structures. Like that sounds like a really big problem. Have you made a a beachhead on that part of the problem yet, when building the core of Sketch-n-Sketch?

Ravi

Yes. Certainly in our current work, there are certain things that seem clearly independent of domain. For example, our programming language really knows nothing about any specific application domain. It’s got built in set of types as usual, user can define new types. And then, you can program with whatever types of values you’re working with. At the end of the day, the connection between the program and the specific domain is the main definition, the main expression that your program computes. That’s the time at which the editor starts to need to translate that main expression, that main value, into the specific domain, in this case, SVG.

Actually, that’s a good question. It makes me realize that… So earlier I described the forward evaluation of programs as a domain independent tracing mechanism where we record evaluation as usual and then domain specific transformations, get to look at that trace information when deciding how to transform programs. But actually there are a couple of situations in which our tracing mechanism is doing SVG domain specific tracing. And so, one of the ideas in the most recent UIST paper is to expose user interface widgets for manipulation on not just the final output and sub values of the final output, but also on some of the intermediate computations that didn’t necessarily draw something in the final output. And so an example of that is in the current version of Sketch-n-Sketch, the evaluation of the program looks for expressions of type point.

Yeah. And it’s funny, this is exactly the artifact that I would really hope to produce in a system that really allowed programming. But yeah, I guess I already said this, so maybe I’ll say it again. Yeah.

Ivan

Yeah. The Sketch-n-Sketch demos that I’ve watched show a specific workflow with direct manipulation at each step, and that workflow is first you draw. And then you add relationships between the things you’ve drawn and then you do graphical grouping, which serves as an abstraction operation where you extract a function that creates the group drawing that you made, and then you tweak your drawing using that abstraction. Each of those steps, creating or refactoring the code. Is that workflow a gilded path, or are there other ways to work within the tool like importing an existing graphic or starting with existing code?

Ravi

Yeah. So a lot of those example demonstrations take that workflow to see what can be done, how expressive a design can be built using just direct manipulation and not also interleaving text-based edits or programmatic edits, but you’re right. Of course, an authoring workflow might really mix and match these two modes of use much more freely. For the question about importing existing graphics, currently we don’t have any tools implemented to make that process easy.

You could imagine taking an existing, let’s say SVG definition and in the limit, just inlining that literal into your program, but even better would be to try to automatically identify numbers and properties that appear over and over again, and suggest those as variables and maybe even function boundaries if they are repetitive patterns in the imported file. You can imagine doing that, but we haven’t spent any effort on that yet.

More directly about this gilded path through the tool. So, in an initial version of Sketch-n-Sketch and the version that we demoed at Strange Loop, there were many requirements about the syntactic structure of the program that if they weren’t satisfied certain interactions in the output would no longer be available. So a simple example is in that initial milestone, the main expression, the main definition of the program essentially had to be a list literal of shapes. And each of those shapes had to be a top level definition in your program and only then could certain interactions be available to users. And so a lot of the work that Brian has done recently has been to relax those restrictions.

And so by doing more general purpose tracing of the program to support more arbitrary programs in the language while still retaining the connection that, “Oh, this value in the output came from certain locations of the program,” that’s supported in a much more general way. But there are certain times when let’s say you want to create some parameterized function that repeats some design. There are certain times you can actually take multiple paths through the tool. So for instance, you can copy and paste a shape or a design multiple times, and then use a tool called merge that looks for syntactic differences in the definitions that generated them. And we’ll take the differences between those programs and turn those into arguments to a function. Or you can say, well, given just a single shape or group, you can use a tool called abstract, which turns it into a function like we just described.

So that’s an example where there’s two different ways you can choose to build one of these parameterized drawings, but there’s other times when you are forced to make a choice about what constraints to add into your program, and then you can’t undo that choice later on. And so there certainly are many times where you do have to pick the right path through the current set of tools, that certainly needs to be address in the future where you want to allow maybe multiple choices to be propagated downstream. So that later on, when you make some subsequent action, maybe then is the right time to decide whether the structure of your program should be one way or another. Currently, there are times at which you have to make a choice that you can’t undo or revert later on.

Ivan

It sounds like you’re not storing an edit history on the graphical side, right? Like every change on the graphical side is immediately propagated back to the code?

Ravi

That’s right. That’s right. So after every interaction, the transformation changes the code and doesn’t store the edits that led up to it. So reasoning about the edits that are being made in the output editor could certainly be a rich source of information for helping to decide or understand what changes to make to the program. And then also keeping track of the history of program edits as well can certainly provide new ways of trying to infer what the user is intending.

Ivan

Yeah because if you provide a lot of different ways to achieve the same result, it could mean that the structure of the code that you end up with is different, depending on which way you went about achieving that result. And it makes me have flashbacks to… For instance, like in Gmail, when you’re trying to do WYSIWYG editing or, not even Gmail, Slack’s recent text editor change to… WYSYWYG editor had this problem, a lot of WYSIWYG text editors have the problem where when you try to add formatting, there is a representation that is invisible behind the scenes, and you can end up with things like here’s a spot that has… It’s surrounded by white space, but if I positioned my carrot in that spot and type it’s bold, even though the text on either side is not bold because there’s an empty bolt node at that spot, those sorts of things.

And so it feels like this is a place where I could imagine it being tricky to get the right balance so that… And I suppose showing the code, and if people who are using the tool are expected to be familiar with the code representation, that gives you a lot of benefit because then people can see, “Oh, when I repeat a shape using this approach, it creates that change in the code. Whereas when I repeat a shape using that approach, it makes a different change.” And so that alleviates you from the burden of having to use something like fancy edit history tracking like a CRDT or something like that, where you try to merge it down to a canonical representation, no matter how you got there.

It alleviates you from the burden of having to make sure that different changes in the output result in the same change in the code that since people can see the code… You’ve moved the burden of correctness and consistency over to the user. And I think that that’s a good thing. Like I think that that’s… You’re giving them leverage rather than foisting complexity on them. Does that feel right to you?

Ravi

I guess I would say yes and no. I guess so certainly I think having a program in a general purpose programming language be the ground truth, be the artifact that matters, I think that’s a good choice. I think that is maybe not in the long-term the best, but I think it’s a very good medium in which to at least allow users, especially expert users, to make this specific choice about what the representation should be. But programs they’re oftentimes where you write something one way, but you might want to express it in a different way instead.

So a very simple example for this specific domain is let’s say I’m writing a program that generates a rectangle. Oftentimes you’ll decide whether the parameterization for this rectangle should include the location, the point of the top left corner and the width and height of the rectangle. Or sometimes you might decide the parametrization should be the top left corner and the bottom right corner, in which case the width and height would be derived in terms of those two points. And so there are times in which you might prefer the former parameterization, there are times in which you might prefer the latter, but with a program you have to pick one, right. And then when you’ve made that choice, all the subsequent code, that depends on it is not very easy to change if you want to go back and change the structure of the initial parameterization. And so I think exposing a general purpose program is a good way of making explicit exactly what the artifact is, what the representation is. But again, there’s times when programs force you to make these kinds of choices that you would ideally like to have even more deferred control over. So, there’s other intermediate representations you can imagine with like program dependence graphs, and other computation, and only have to turn it into a program, like turn it into an abstract syntax tree when it makes sense.

Ivan

Like almost as a rendering step or something like that like export code or something?

Ravi

Right. And then let’s say that you want to move back to the more general purpose, like bag of constraints, bag of computations, do more interactions. And then at some point, you want to say, okay, at this point I know that I want to do some repetitive operation over my data structure, codified as some AST that looks like normal data structure that I can map and fold over so that I can perform some actions, but then go back to this more general representation of the computation for subsequent edits.

Ivan

Ravi

Are they going to use a GUI application that is developed for those domains and give up the abstraction capabilities that they know that programming provides or are they going to pick their favorite programming language or their favorite library that caters to that domain. Oftentimes an expert user might choose to, let’s say, use Beamer to generate slides or use this Racket library called Slideshow to develop their slides. And there’s this clear downside that although you can generate these really complex abstractions and reasonable artifacts, making simple changes like, drag this thing a little bit to the right or copy and paste this thing and then change it, are extremely tedious to make when you have to think about where in the program does that operation stem from. And so, I think there is this pain that expert programmers realize they go through when they’re already programming. Right? There’s this tedious Edit-Compile-Run cycle that, especially when you’re trying to, at the end of the day, generate something that is very visual and interactive and the design process takes lots of iterations, which is common when you’re building something like this. I think expert users run into that pain point and would clearly see how they could benefit from a tool that allows them to do programming, but then also get some of these interactive capabilities for changing the outputs of their programs.

Ivan

Given that that’s the appeal that an expert would see in a tool like this. Let’s look at the other side, what is Sketch-n-Sketch offering beginners, newcomers to programming, or even programmers who are newcomers to graphics that they might not otherwise be able to access?

Ravi

Right. So, I’m certainly no expert in computer science education or programming education, but it seems intuitive that tools that make programming more interactive ought to help with the teaching, the understanding of programming concepts. And so, one thing that I’m interested in doing is using Sketch-n-Sketch and future versions of Sketch-n-Sketch to teach introductory programming to students that maybe want to learn simple graphic design or generative art.

Because my sense is that students that are interested in, let’s say, design or art would of course learn to use tools like Illustrator and Photoshop and all of those tools. And then some of those students might then later on learn a programming language like Processing or p5.js which cater really well to these domains of programming. Instead, what if you could teach the kinds of features that Illustrator and Photoshop provide in the same environment in which you can learn about variables and functions, and have those different concepts in different interaction paradigms just be the same system and not two disparate systems.

And so, I’ve been looking recently at the Processing community. It seems like they’ve done a lot of really cool curriculum development around Processing and p5.js. There’s folks at NYU in particular that I’ve been looking at their work, Daniel Shiffman, Allison Parrish, at UCLA Lauren McCarthy. They’re developing a lot of really interesting content for basically teaching programming to students that are interested in design and art.

And so, one idea that I’m planning to pursue is think about how to teach programming with tools like Sketch-n-Sketch where, not only do you learn different programming constructs, but then you can interact with the output of these programs. And interacting with the output of the programs can suggest changes to the original program and hopefully motivate and teach why you might want to learn about variables, why you want to learn about functions and things like that.

Ravi

I should also make clear that this has really been a big team effort. I’ve hardly done any programming on Sketch-n-Sketch in the past year, two years even. I’m looking forward to getting back into it, but really the heavy lifting has been done by this really terrific group of students. Brian Hempel has been doing a great job over many years.

He was crazy enough to join me when I was starting this project. Justin Lubin, a really terrific undergraduate who’s made tons and tons of contributions to the project. Mikaël Mayer has been doing lots of really awesome work. Cyrus joined my group and brought the idea of holes and programming with holes to the group as well. Nick Collins. And so, it’s been a really terrific team effort. I’ve been really lucky to have such great collaborators.

Ivan

Speak to that a little bit more. Like what have each of those people done? It’d be sort of nice to know for each of those people what they’ve contributed.

Ravi

Sure. So, Brian’s focus over the past couple of years has been extending the expressiveness of what you can do purely with direct manipulation interactions. And so, he was the driving force behind this most recent UIST paper where using just direct manipulation interactions, he’s able to build complex readable programs for a variety of parameterized drawings including recursive drawings, including drawings that have shapes and groups of shapes repeated over various geometric dimensions.

And to do this has required exposing much more about the execution of the program than just the final value that it computes. And so, recording a lot of information about how intermediate program execution ends up affecting the final output value. All this more general purpose tracing of programs and exposing richer widgets for manipulation, exposing a whole bunch of new tools for transforming programs has really been the focus of his work over the past couple of years.

And he’s now thinking about how to expose similar kinds of interactions, building programs based on output interactions, for other domains where you don’t necessarily have very visual representations of values like you do in vector graphics. So, how could you implement a more typical general purpose data structures and data structure, manipulation functions by example, by demonstration. How can you do that in this kind of style? So, that’s been, that’s this kind of current focus right now.

Justin Lubin has made contributions to the project throughout in various aspects. One of the main projects and features that he had led was the design of this text editor interface that we call Deuce. Which is something that actually doesn’t have to do with the bidirectional programming at all. It’s a feature for more traditional text editors or code editors. So, obviously text editors are the main interface through which programmers read and write code.

But of course the program, once it’s parsed into an abstract syntax tree has a lot more structure than just the underlying linear text buffer that created it. And so, refactoring tools offer a variety of structured transformations, like renaming, extract method, things like that. And then structure editors have lower level AST transformations built into the system so, that you don’t have to resort to text editing all the time. And so, a feature that we built for the text editor in Sketch-n-Sketch is what we call Deuce which tries to overlay the structure information of the program, the AST, on top of the normal flat text representation.

And so, Justin led the effort on designing that interface, where you can hover over the code box. And as you hover over different parts of the program text, it shows you different nodes in the AST that you can select. So, for example, you might select some variable definition and then you might select some white space in between two definitions in your program. And one of the tools that Sketch-n-Sketch will then propose is, do you want to move this definition from this part of the program, to the other part that you’ve selected? And so, these kinds of refactorings or structured transformations can be made while staying within the text-based editor. And so, Justin took the lead on lots of that project.

Mikaël Mayer, has worked on one of these algorithms under the hood that we referred to earlier. Which is really the core bidirectional evaluator, which allows changes to be made to the output value in a way that are mapped back to changes to the program. And this bidirectional evaluator has really been developed in a generic domain independent way.

But it’s one of the many features under the hood of Sketch-n-Sketch that allows you to, for example, make changes to things like colors and positions of things, and have those edits be mapped back to corresponding program repairs. He’s also been exploring that idea of bidirectional evaluation applied to the domain of HTML applications as well. And he’s actually pursuing a startup to try to build some of those technologies into more usable tools.

Yeah. And then so, Cyrus obviously had been working on the Hazel project before he joined my group, and hasn’t been working on Sketch-n-Sketch proper. But the idea of programming with holes and running incomplete programs is something certainly that we’re going to incorporate into future versions of Sketch-n-Sketch. We’ve recently been working on a project for taking these programs with holes, partially evaluating them.

Ravi

Oh, so certainly the UIST paper does refer to the sources from which we drew these benchmarks. And so the ones that you mentioned I think are from this Watch What I Do benchmark suite, and it is a very well-known and available resource. I guess one challenge about doing head-to-head comparisons is that there are so many differences among the specific language that is being used. And some of the tools maybe don’t run because they’re 20 years old and some of the goals of the systems are different. And so it’s really hard to, at this point, identify like these are the SPEC benchmarks of live programming or the SPEC benchmarks of output directed programming.

But it certainly is the kind of thing where, yeah, we certainly want to be able to compare different systems on shared examples, if not exactly the same example, at least different incarnations of the same goal or the same concept across multiple systems. And so I don’t know that there’s a single kind of benchmark suite that already exists, but certainly comparing to all these other examples that people are using seems to be a good first step.

Ivan

For the people who are working on these sorts of tools, which I think is most of the listeners of this podcast, those sorts of benchmarks would have utility in that they might force you to approach your tool in a way that you weren’t naturally gravitating towards. And so you can sort of use it to test your assumptions and to test your model and to see where your ideas break down. And that’s the appeal of these sorts of benchmarks to me. The competition thing’s more of a… It’s kind of a joke. It’s more that this is a way of… A lot of our community are working independently. And so they might not have the resources to do extensive user testing or they might be at the wrong stage of their process to do user testing. But those benchmarks might serve as ways to help people think about their model and to just read the benchmark and think, “Hmm, is that something that I could even do within my tool?”

And I was going to… I wanted to reference in this context Brian Eno, who is a very significant figure in the history of generative music and generative art. He made a deck of cards called Oblique Strategies that you’re supposed to use when you’re facing a creative crisis, when you’re staring at a blank page and you don’t know what to write, or when you’ve made something and it’s not finished and you’re unhappy with it but you don’t know what to do to continue working on it. And each of these cards in this deck of cards is sort of an open-ended prompt. And many of them are very unusual in what they ask you to do. And they’re meant for interpretation, and they’re sort of meant to shake you out of a creative rut.

And so I’d love to see, and it sounds like these, the examples from the Watch What I Do book, are a little bit like this, is I’d love to see somebody put together a collection of just, here are intellectual stress tests that you can put your ideas through in order to assess their generality or assess their applicability to the domain or assess things that will force you to think through the problem space so that you don’t lead yourself into a blind alley. So if you know of anything like that, other than the Watch What I Do benchmarks, that would be a really interesting thing to share with our audience.

Ravi

Yeah. I agree that it would be valuable to have that sort of thing. I remember at the end of the LIVE 2018 workshop, there was a little bit of discussion about whether there were common benchmarks or reference points that, like you’re describing, one could look to as stress tests and things like that. I don’t remember if anything came of that discussion.

These aren’t examples of the kind that you described. But one thing that I would mention is this work on what’s called the cognitive dimensions of notation, which are a set of heuristics for evaluating user interfaces, programming languages, other kinds of abstractions like that. And they include things like closeness of mapping, how closely does the notation or the user interface represent the notion of what the user is trying to create, what notion in the world the user is trying to create? Another heuristic is, are there hidden dependencies? Are there other things that the system is doing and knows about that are not exposed to the user for understanding or for manipulation?

And so these are properties that generally are good if they hold. They ought to hold. Some of them are mutually unsatisfiable, but you want to be able to satisfy as many of these as possible. They’re not benchmarks or examples in the sense that we were talking about but are useful heuristics to try to understand whether the system that one is building or trying to build satisfies multiple of these goals or not.

Ivan

Hey, so this is Ivan from the future just cutting in here. I normally adhere to the doctrine that you should not reveal the technical details of how a podcast is made on the podcast unless you are doing so intentionally, so for instance, avoiding letting people hear the glitches that sometimes happen when Skype drops out or one person having a really good sound quality and another person having a really bad sound quality. That kind of behind the curtain stuff is, to my taste, unprofessional.

But I’m going to break my own rule right here and say that at this moment in the conversation, my connection to the internet died. I can’t remember why. But Ravi and I reconnected and picked up the conversation where we left off. I couldn’t salvage this in editing. I’ve had a number of other hiccups like this in my brief time as a podcaster. And I’ve always managed to pull them together through a very deft use of Ableton Live. But this one was just too jarring, and I couldn’t find a way to stitch it together coherently. So, you get to hear me blathering on and apologizing and saying, “I’m sorry. This isn’t how I like to do it. Don’t do me like this.” So anyways, that caveat out of the way, back to the interview.

Ravi

End user programming and programming by demonstration, other kinds of approaches like that have never really succeeded in the mainstream or into really usable, useful solutions. And so I guess one question is, will this ability to actually mix programming and direct manipulation, will it actually help with novices and less expert users, or will this only be limited to experts because this is just another set of tools that need to be used and mastered?

But I guess I’m hopeful because the success story that many people like to talk about are how widely used spreadsheets are and how many people use formulas and macros and a little bit of programming without really knowing it. And I guess I like to think that, what if the standard toolbox that you see in many different GUI applications has a tool that looks like equals X, where you can introduce just a name for something, and then you can drag that name onto multiple properties and multiple objects on your canvas? I feel like very simple notions of giving names to things and very simple relationships between these things seem like they ought to be simple and general enough to be realistically part of end user user interfaces for a variety of domains. But again, I guess it remains to be seen if that’s really going to play out or not.

Ivan

That’s why I brought up Maya and 3D Studio Max and the other Autodesk tools is, and I neglected to mention this, but another ability that they have that’s sort of universally present in 3D animation tools is that they allow you to make any property on any object in your scene the result of a function of another property. And the way that you can wire them together is extremely open-ended.

And as an early teenager, knowing nothing about programming, knowing frankly, nothing about computer graphics and not being a very good artist, I was still able to pick up one of these tools and just through playing around with the user interface, figure out how to make, for example, glitch art by using the incident angle of the camera on the surface, which is an xyz-position, be used as the RGB color attribute. And so the color of the surface is based on the it’s based on the angle that you’re looking at that surface from. And those sorts of mappings from one data type to another, or one representation to another, or one property to another, and applying a function on that mapping in those programs is a really core part of the UI. And a lot of the UIs are built in terms of those mappings between properties.

And it feels like when I use other programs that don’t have that ability, it feels like something’s missing, like I’ve been sort of cheated when I use Photoshop or when I use Illustrator and I can’t do that kind of… It gave me an early taste of what it feels like to be a programmer before I actually learned to be a programmer because I was programming my scene. I was programming my art. And I definitely feel like what you’ve just talked about, about wondering about whether something like Sketch-n-Sketch or something like programming by demonstration would find a home in all of these domains. I really think that it would. I really think that there are examples of where that has happened in existing tools.

And unfortunately, if you pick up a program like Maya or Modo and you open it up, it doesn’t look like Excel. It looks like a nightmare version of Excel where instead of just the one ribbon across the top with way too many tools, it has user interface controls everywhere. It has thousands of features. If you right-click in 3D Studio Max, it brings up four context menus around your mouse. These things are… They’re like the Starship Enterprise. It’s enormous and complicated.

And yet as a 12 year old with no internet at the time and no tutorial books and nothing other than the software on my computer and no expert guidance, I was able to learn how to do it because those interfaces are self-revealing. And so that’s why I wanted to ask about the expert experience and about whether you think about the ramp towards expertise is because I think the focus on making things approachable by beginners, a lot of times people will react to that desire by making a tool that is meant to be handled with boxing gloves or with… It’s like safety scissors. They sort of try to distill it down to some sort of simplified core essence so that people can’t hurt themselves or be scared away by the complicated user interface.

And yet I think in making these tools that have that programmability behind the scenes, there’s going to be a need to expose more complexity and expose that complexity in a way that’s tractable, in a way that’s approachable, in a way that is self-revealing and discoverable, to borrow a Ted Nelson term, to make the interface explain itself, not to make it intuitive, but to make it so that somebody who’s playing around with it with no other reference can figure out what it does.

And I think Sketch-n-Sketch looks to be a great approach to doing that. The ability to see the change that you’re making graphically or in whatever output domain turn into the change in code, like that connection between the complexity of the code and the simplicity of the output, I think gives people that… It gives them that ability to discover the way that the tool works. And so I am optimistic about that being something that we can look forward to in the future.

What I am not optimistic about and what I would be interested in hearing your thoughts on if you have any, is what it takes to make a tool like this have a place in the market. Because we’ve had past examples like HyperCard dying an early death, even though it enjoyed phenomenal success. And Flash more recently, Flash was a beloved tool for doing interactive art that died, I would argue, because of mismanagement by Adobe, but that’s a whole other matter. It seems like these tools have a really hard time surviving in the market. And so do you think about that at all? And what do you think you might be able to do to help fight that trend?

Ravi

Yeah, so market, I guess there’s a couple of ways to think about getting, hopefully getting, these kinds of ideas, technologies, tools into the world. And so one is, can this be built into some system, tool, technology that can be marketed, that can be sold? Another is, can it be implemented and developed and released as open source tools that are used by a variety of people, in a variety of settings?

And so, so far as researchers, a small research group, we’ve chosen, we’ve specifically chosen to do things in a clean slate, prototype, toy little setting because it’s been easiest for us to try out ideas in isolation. But then when we have algorithms and ideas that we think are reusable and generalizable enough, of course we write papers and try to explain the main essence of the idea so that hopefully someday in the future, folks that are building industrial scale languages and editors and environments, hopefully, these kinds of ideas make it into those efforts in the long term.

But in the shorter term, I guess I would say that I think there are certain parts of the techniques, at least like some of the bidirectional algorithms that are making pretty modest changes to the existing program, assuming that a programmer has already written a lot of the high level logic of the program. A lot of those algorithms, I think could be scaled up to practical editors for many languages in the not too distant future.

One of the things that Mikaël Mayer has been working on is scaling up one of these bidirectional evaluation techniques to JavaScript. One way we might do it is actually have a library that takes your JavaScript program and can run that JavaScript program in reverse so that when the output of your JavaScript function is changed, it will suggest to you a bunch of changes to your program that you might want to choose from. And so a library like that, we would hope could be hooked into a bunch of different application domains in which people are writing JavaScript programs to script various things. Could you provide some of this kind of program repair, bidirectional evaluation functionality somewhat for free so that a tool builder or an application builder doesn’t have to do anything more than surface the suggestions, the functionality to the user? I think he and we and I think hopefully others will put in the effort to try to see if we can actually make it a drop-in replacement for… You have eval in JavaScript. Can you have uneval also without a whole lot of extra work on the user’s part?

Ivan

What about artist tools or mixed art and programming tools like HyperCard or Flash?

Ravi

Yeah, so I think in some sense, I mean, the bar is higher for building really usable tools for artists or programmers or novices that don’t have as much programming experience. I think the easiest realization of these ideas in the world, in the short term, I think are as developer tools because you can imagine building plugins for existing languages. You can imagine building plugins for existing editors for… You can imagine building plugins for Chrome and Firefox that expose these capabilities. So I think in terms of the short term, some of these ideas I think could appear in developer tools much more easily than they could for more ambitious domains.

Because I think for a system like this to be useful, it really has to provide a lot of the capabilities that so many existing application GUI tools already provide. And so in terms of marketability for these kinds of application domains, I certainly don’t have any good answer. It’s hard for me to imagine the situations in which this kind of thing could be a useful marketable tool.

I guess one idea is, thinking about the kinds of people that maybe are learning programming not because they want to be programmers but because they want to create some artwork, graphic design and things like that, often times will have to learn programming as part of their workflow and their tool chain. But clearly the state of the world where you have tools that are good for direct manipulation, you have tools that are good at programming, but nothing that combines them. I guess, if there’s the right niche application domain, maybe one could focus building this up for a domain in which there doesn’t already exist a very good solution for the kind of programming side of things. But I guess this is not a really… Yeah, this is not a very helpful answer.

Ivan

It’s hard. One thing I thought about in trying to find out the difference between why it seems tractable to offer this sort of tooling to programmers and intractable to offer it to artists is that programmers are in the business of assembling things. And that extends to the tools that they use. And so rare is the programmer that is using a stock configuration of Emacs or is using a framework with no middleware. We tend to kind of endlessly customize everything we do by assembling other pieces together with a core structure.

And artists don’t do that. Artists buy a complete workstation off the shelf. If you’re a musician, you buy Ableton Live, and Ableton Live lets you use audio unit plugins and VST plugins. But it doesn’t let you add in a new way of… You can’t add in a score notation for writing sheet music within the Ableton Live environment. It’s a much more closed tool, and it’s expected that you will use the extension points to offers and that’s it, whereas programming tools, the same philosophy applies. You can only use the extension points that exist. But programmers are coming with a mindset that they want to customize so let’s make lots and lots and lots of extension points. Let’s make it so that things are modularized and composable.

And so I almost feel like what it might take to be able to, as a small, independent research team, to be able to make a contribution to artists tools would be to find a domain where the artist tools are more modular and where there is sort of the invitation in that domain for people to come in and add the extra slice of functionality that they can add. And it makes me wonder, maybe like scientific visualization or medical imaging or something like that, maybe there’s a domain out there where that is more common practice. And that might be a sort of a way to get the foot in the door to the arts rather than just continuing to offer things to other programmers.

Ravi

Mm-hmm (affirmative). Yeah. Yeah, the thing that scares me about scientific visualization, medical imaging, also 3D animation games, all of which I think are certainly domains in which I would hope these techniques will work for and scale to eventually, what makes me scared in the short term is that the math that goes into the computation of the final artifact is much more complex. And so one of the challenges for connecting programs to their output in both directions is, how well can you invert the operations, right? When you make a change to the output value, how do you map, invert it, and map it back to the program that generated it? And the more and more complex math that you have, the more ambiguity, the harder it is to really get meaningful changes back in the inputs to the computation. It’s certainly not impossible I don’t think, but it’s, I think, more challenging than what we’ve been focusing on so far.