54 • Ella Hoeppner • Vlojure

2022-01-05

Listen in your podcast player by searching for Future of Coding, or via Apple Podcasts | Overcast | RSS

Today’s guest is Ella Hoeppner, who first came onto the radar of our community back in the fall when she released a web-based visual Clojure editor called Vlojure. Here, right off the jump, I’ll include the video that Ella made to introduce the project:

Vlojure - A New Way to Write ClojureScript

I was immediately interested in the project because of the visual style on display — source code represented as nested circles; an earthy brown instead of the usual dark/light theme. But as the video progressed, Ella showed off a scattering of little ideas that each seemed strikingly clever and obvious in hindsight. You’d drag one of the circle “forms” to the bottom right to evaluate it, or to the bottom left to delete it. The sides of the screen are flanked by “formbars” that hold, well, whatever you want. You can reconfigure these formbars to your exact liking. Everything is manipulated with drag. The interface exudes a sense that it was designed with wholly different goals and tastes than what we usually see in visual programming projects — perfect subject matter for our show.

Links

Our guest is Ella Hoeppner, who you can follow on Twitter.

Vlojure runs right in the browser, so click that link to try it for yourself.

Before Vlojure, Ella spent the summer of 2021 releasing a string of generative art videos that I must highlight prominently. Here they are, for your enjoyment:

Smoking: Generalizing the Rainbow Smoke algorithm

Voronoi Diagrams

Flow Fields

The music in these videos was created by Ella’s friend, who (currently) posts as @faycarsons on Instagram.

You can see more of Ella’s generative artwork on Instagram, too: @generative_garden.

One of the libraries Ella uses for generative art is Pixi.js.

Previous podcast guest Toby Schachman’s Recursive Drawing was referenced.

p5.js is another popular generative art library.

Ella’s artificial life project is Broth.

Wolfram’s physics article.

Picbreeder is an aesthetic selection tool.

sfxr is an old video game sound effect tool that has a not-quite aesthetic selection GUI.

The go-to text on aesthetic selection is Genetic Programming: On the Programming of Computers by Means of Natural Selection by John Koza.

Sponsors

Glide are building a batteries-included app development platform where the database is Google Sheets. If you’re excited about making end-user software development a reality, go to glideapps.com/jobs and apply to join their team.

Replit are building an online REPL for the next generation of programmers. It’s a collaborative space for deploying web servers, developing games, training ML models — all driven by the REPL. They’re hiring, looking for people to come enrich the core and ecosystem of Replit.

Transcript

Please know that the episode transcript is challenging to produce. It is unrefined at best, inaccurate or even unreadable at worst. But despite the rough shape, it’s valuable to many people in our community. If you’ve got the time and energy, we would love help cleaning it up.

Ella

When I first learned coding, primarily, I was learning it because I was interested in doing generative art actually, that was kind of the way that I got into programming initially. In the high school that I went to, I was lucky enough that they were able to give all of the students laptops, and that was wonderful for me because when I was sitting in class, instead of listening, I could be doing something actually productive and learning to code. And I would sit in class and make little generative art backgrounds for my computer and other people’s computers. And that’s kind of what got me into coding in the first place. In terms of my professional background, I went to Virginia Tech, graduated from there about two years ago with a degree in computer science. Up until a couple months ago, I was working at the University of Virginia, and the School of Medicine, doing web development stuff for research into online health interventions.

Ella

And then for the last month or two, I have been taking some time off because I’ve got a couple of projects that I’m working on, Vlojure being one of them, that I wanted to have some time to work on.

Ivan

It sounds like you learned how to programme kind of in the later part of high school. And I think by that point, you were probably well aware that programming was a thing. Do you remember at all what you thought that programming was back before you actually learned what it really was? And how those two things compare?

Ella

I can’t remember anything like any particular impressions of what programming was like other than just it being very opaque and kind of almost magical, there were people called programmers who did some magical thing that made computers do interesting stuff. But beyond that, I had no idea the details.

Ivan

So it turns out, it’s in reality very much like you expected.

Ella

Exactly. Yeah.

Ivan

Yeah. There’s a lot of dark magic at least Yeah. I have this tendency when I read the questions for one of these interviews, to leave the very heavy involved deep stuff to the end and then kind of front-load the interview with a bunch of light-hearted, fluffy fun questions. And what inevitably ends up happening is we spend two hours talking about some wild tangent from one of the light fluffy questions and they never get to the actual real meaty stuff.

Ivan

So I am going to want to get into talking about Vlojure pretty quick. But just before we do that, the one kind of lighter thing I wanted to talk a little bit about were the generative art YouTube videos that you’ve made. There are three of them so far, for one as an active listener advocacy, I want to suggest that everybody listening to this actually go to the show notes for this episode. I will have links to these videos I might even embed them on the show page because they are absolutely beautiful and the music in them is just delightful. You could just like zone out and look at them and the generative art that is going on in them is just beautiful to look at and the music accompanying them works perfectly.

Ivan

But then you’re also talking through how the algorithms work, and it has this very hypnotic quality to it. And I’ve just absolutely loved watching these videos. And it made me wish that there was like an infinite stream of them where I could just put them on and have it run for hours and hours. And so, in pursuit of that kind of a goal, maybe, are you planning on doing more of these kinds of videos? What did you do to make these first ones? It seems like they took a lot of work. Where did these videos come from? And do you think you’ll be carrying them on into the future?

Ella

Yeah, I’m certainly planning to. And thank you for your kind words. It’s very nice to hear you say, all the feedback that I’ve received on them has been really wonderful. So I’m absolutely planning on continuing. I’ve been doing generative art for a long time. And I don’t use social media stuff, all that often, I just find it kind of boring. So I would rarely post a lot of the generative art stuff that I did. Oftentimes, I’d show it to a friend or something, and then it would just sit on my laptop for a year or something, doing nothing with it.

Ella

And eventually, I thought I should actually publish some of this stuff and make something cool out of all this time that I’m spending. And so that’s what the idea behind the YouTube channel has been. And they have been a lot of work to produce. And it’s been about maybe two or three months since the last one. And since that one, I’ve been trying to rethink the way that I’m producing the videos, in order to kind of make them less taxing to produce, hopefully, that I can be able to pump them out faster, create that infinite stream of generative art content that you’re talking about. But yes, I’m absolutely looking forward to making more of those in the future.

Ivan

And just for my own curiosity, what’s the process to make one of those videos because there’s like a mixture of animated examples and then also some cases where you’re actually drawing out what the vectors are doing? It just seems there’s a lot of different elements that each need to be separately created and kind of brought together.

Ella

Yeah. So generally, the process just starts with me having some topic and generative art that I find interesting and I’d like to make a video explaining. And I will just kind of start making some code, writing up whatever is necessary for the piece just kind of make a couple of variations like the first video that I did was a video about flow fields, which are kind of a classical technique and generative art, tonnes and tonnes of artists have used them, but they’re still really beautiful and interesting. So I thought that would be a good place to start. And for that video, once I knew that that was going to be my topic, I just sat down and coded up a Flow Field algorithm. And once I was happy with the way that it looked and everything, I started writing a script, and I would go through kind of different various examples of how the algorithm could be used.

Ella

So I start with the coding and making the art and then retrospectively I kind of go through and try to find out what would be the best way to explain this. And then sometimes that involves writing more code or making diagrams, like you said, drawing stuff out, that kind of help people understand what’s going on. And then, yeah, film it, record it, get a friend of mine to make some music for me. And that’s basically the process. By the way, you were mentioning you liked the music to them, and if anybody is interested in that music, then you should check out Inconvenient Body that’s a friend of mine who does all the music that’s been in my videos so far. So I’m really grateful to her.

Ivan

I’m going to do the worst thing that every music reviewer does. And as a musician, I hate this, but as a person listening to music, I love it. It reminds me very much of the early Aphex Twins sort of selected ambient works having that kind of music accompanied with this really beautiful, generative artwork is just such a great mood to find oneself in. So I’ve really enjoyed discovering the work of this musician through your videos as well. So I’ll have a link to their stuff in the show notes also.

Ella

I’m a huge fan of Aphex Twins and so is my friend so she’ll be very happy to hear that.

Ivan

What tools are you using to do this generative art? Is it like a like Clojure Library? Or using processing? Or what kind of generative art tools are you fond of in this work?

Ella

Yeah, I do most or really all of my generative art, nowadays in Clojure, or usually ClojureScript for the videos that I’ve made so far. And I’m using a JavaScript graphics library called Pixi.js as one of my main graphics libraries. And I also do some stuff using various Clojure graphics engines. I’ve been using one recently called just Clojure2D, which is just a great little pure Clojure graphics library. I’ve used a tonne of different graphics libraries over the years and I’m not particularly picky, I can just find something that works, build my own wrapper around it, and get going.

Ivan

So speaking of Clojure, let’s get into the reason that you came onto my radar and why I wanted to bring you on the show, which is your recent project Vlojure, which I would normally do a little spiel where I’d explain what it is, but I actually want to let you explain it like what it looks like, and what it would feel like to work with it for somebody who is listening to this and might not have seen your video introducing the project.

Ella

Sure. Yeah. So the basic idea behind Vlojure is that it is a programming interface for programming in normal ClojureScript, right? So there’s nothing special to the language itself. But what’s special is the interface. So normally writing Clojure involves a text editor, you have nested parentheses and brackets and all that kind of stuff. But in Vlojure, the hierarchy is instead represented with a structure of circles. So wherever you’d have a pair of parentheses in normal Clojure, you will instead have a circle and kind of subforms within larger forms or represented in smaller circles within a bigger circle. And of course, since Clojure includes not just lists, but also things like vectors and maps, that have their own special kind of encapsulating characters, vectors are represented as octagons in Clojure, maps are kind of represented as these circles with little spikes on them.

Ella

And so basically, Vlojure is a sort of drag and drop visual interface for programming ClojureScript. It’s hard to describe the idea that in words, but thankfully, it’s up on Vlojure.io. So if anybody is listening to this, and can’t quite picture what I’m describing in their head, just go look at it for yourself, I worked a lot on trying to make it very intuitive and easy to pick up for people who already know, ClojureScript. So it should be very easy to kind of see what’s going on if you’re familiar with Clojure.

Ivan

And I’ll also include the demo video that you created in the show notes for this so that anybody just happening upon the website for this podcast can watch that presentation you gave, which serves as a really wonderful introduction to this environment. It’s a really concise video, I think it’s only like eight minutes or so. But even in the first three or four minutes, you cover so many really interesting things that feel like they could be explored in great depth in a longer form. So that’s why I wanted to get you on the podcast. Talk a little bit about how you arrived at this idea of evaluating by grabbing a form and dragging it to one of the corners. Was that something that just occurred to you? Was that something that you tried some things and then arrived at this one? How did that come about?

Ella

From the start, I knew that this was going to be a visual interface. And the most natural thing that popped out once I kind of had this idea of representing as expressions as circles with smaller circles inside was that clearly, this interface would be great with a drag and drop system, rather than any kind of other editing system. And so when you’re building a programme in Clojure, you’re sort of dragging circles around, moving them around, copying them, deleting them. So just naturally, the whole thing lends itself to a drag and drop-based workflow. So with that in mind, the idea of just kind of having this rebel zone or eval zone, where you can just drag any piece of code and see what it evaluates to just kind of seemed like the most natural thing to do once the basic idea was there.

Ella

And leading things goes the same way in the bottom right corner of the screen, there’s the eval zone, and in the bottom left corner, there’s a discard zone. And if you ever want to delete something, you just click on it and drag it to the bottom left corner. So I tried to keep the whole thing very drag and drop focused, in part because I want this to work not just on computers but also on mobile devices. I’m aiming this to be very cross-platform tool, that kind of drag and drop system seems like the best way to go.

Ivan

It’s nice because it’s not just cross-platform. But it’s also like cross interface paradigm or whatever term you want to use, like using drag and drop as a way of invoking action rather than something like command return or whatever, like you get out of more traditional REPL is nice because it’s just another way of bringing some spatial character into the way that you interact with this programming tool. And it feels like that sort of a missing ingredient that a lot of programming tools have now that we’re kind of into the mobile world, and maybe someday into the VR world, where if you want to be doing programming that is fully sort of fluid and conversant with the medium that it’s in if that medium is very spatial in nature, the programming also needs to be very spatial in nature. And so having that way of taking some of the very familiar conventions from a traditional REPL based environment, but just adding that little bit of tangibility by making everything about dragging and dropping, just feels really nice.

Ivan

It’s one of those things that seems very obvious in hindsight, but it’s the kind of thing where if it was so obvious in hindsight like we’d have so many more tools that are doing it. And so that’s why I think Vlojure is really interesting, especially because you didn’t just stop at grabbing a form and dragging it to the eval corner and then there’s your result to the discard corner, and then it’s gone. Those two corners also hold on to the last value that was put there. So in the eval corner, it’s whatever the result of the last evaluation wasn’t in the discard corner it’s whatever you discard it. And so in the discard corner, it acts almost like an undo stack. If you change your mind, you can grab something back out of there and bring it back. Was that something that just kind of occurred to you as you were building it? Or how did you arrive at that idea? What sort of led you to have that as a capability?

Ella

Like you said, the idea behind Vlojure is a relatively simple one, it’s really the core is just a way of representing as expressions as kind of this little visual circular structure. And I feel like once you have that idea that the rest of Vlojure just kind of writes itself. So once you’ve got the drag and drop little ripple zone, of course, that’s going to display whatever the result of the eval is. And so it just seemed obvious that if clicking on that and dragging to some other area, that’s not going to do anything anyways. So we might as well make it so that doing that drags out the result of the evaluation.

Ella

Yeah, that just kind of jumped out to me seemed like the obvious thing to do. And one thing I should say, in the future, I’m planning to make it, so that you can pull out not just like the last eval results, or the last thing that you’ve discarded, but you’ll be able to click on the eval zone or the discard zone and see a history of all the evaluations that you’ve done, or all the things that you’ve discarded, and pull any of them back out into the main programme.

Ivan

Yeah, that would be super nice. That seems like another one of those things, where once you’ve added the feature of being able to pull things back out of there, it’s just the next feature writes itself. And I love seeing environments that are designed to that way where it’s like, “Hey, if we do this one little twist, then it shakes all of this fruit out of the tree.” And then this is a terrible analogy because it’s not like after that fruit falls, the rest of the fruit is easier to get. But it’s the kind of thing where you can just keep pulling on the thread and like more and more and more cool stuff comes out. So I like that a lot.

Ivan

And if it sounds like I’m happy about this environment, it’s because it’s like giving me this feeling of excitement about there’s so much possibility here that I really love seeing whenever I find one of these new environments that has some fresh ideas in it that are waiting to be explored. So it’s always fun to get to look at one of these things as it’s early in its evolutionary process.

Ella

Yeah. No, absolutely, I kind of share the excitement I feel like I’m kind of in the same position as the rest of the people who are seeing this where I didn’t do anything particularly special to come up with the idea, it just struck me. And I just kind of been doing the obvious thing ever since I came up with it. So, yeah, I’m very excited to see where this goes to. And more than anything, I’m excited to see kind of the ideas that other people have with on like how to expand Vlojure in the future. So far, this has just been like a solo project. But some people have already forked the repository on GitHub and stuff. So I’m very interested in seeing what directions other people take this.

Ivan

Have anybody shared any good ideas that you’re particularly excited about playing with?

Ella

I’ve gotten tonnes of great suggestions about different things to add just little UI features, you should be able to drag formbars around or stuff like that.

Ivan

Speaking of the formbars, let’s talk about those because that’s another part of the interface that feels like it’s a very simple seed of an idea. But the way that it sort of manifests is, I think, quite, quite rich and quite nice. There’s a little sort of edit the interface mode that you can go into, along each of the four edges of the screen. At the centre of those four edges of the screen, there’s a little plus button. And if you click one of those plus buttons, it’ll make a little bar appear along that edge with a couple of like empty circles in it. And then when you’re back and working on your Vlojure programme, you can grab any form from your programme that’s sort of nested construction of circles at the centre of the screen, and drag it down onto the formbar along the edge where you click that plus button and made the little formbar appear.

Ivan

And you can have these formbars along any side of the screen. And you can also when you’re in that Edit Mode, add a new formbar, kind of on either of the three sides, sort of like the hemisphere away from that edge of the screen, and keep stacking these formbars up and make a larger sort of interface element there if you want to be saving a lot of stuff. And I think in the video that you showed you have formbars along the left and right sides of the screen that have quite a few different things in them, like some common Clojure functions that you might want to use. How are those… Here I go, I’m trying to find a way to repeat the same question that I’ve asked three times already. But here’s what I’ll do. I know how I’ll approach this. I have 2.5-year-old and her favourite thing right now is to reference something and say talk about it. So I’ll point at the conceptual notion of the formbar and say talk about it.

Ella

Sure. Yeah. So I’m happy to talk about it. Yeah, the formbars are, I think, a really interesting part of Vlojure’s design. Because they act kind of both as like a toolbar of common functions that you’ll want to reference frequently. And you can drag them out, and keep them or add them wherever you want. So it can act as that. But it can also act as something much more transient, like sort of a clipboard because at any time, when you’re writing a Vlojure programme, you can click on a circle and drag it into one of your formbars, and it will stay there for as long as you want it to. And then you can drag it back out later, to kind of copy that form you dragged into another place in your programme, or you can throw it into the REPL or discard it from the formbar once you’re done with that.

Ella

So the formbars act as kind of a combination of a normal toolbar, they kind of has a bunch of fixed things on it but also a kind of very versatile clipboard, where if you have something that you’ve written a little piece of code, and you say, “Okay, well, I’ll probably want to write stuff like this all over the place,” then you can throw that into a formbar, and use it whenever you like. So, yeah, the formbars are one feature of Vlojure’s design that I’m definitely happiest with. And, as you said, they can appear on any side of the screen. And I’ve tried to make it so that there is like flexible as possible, in terms of kind of how the user interacts with them, you can make them bigger or smaller, move them around to different sides of the screen that you can have, however, many of them that you want. And a big reason for that kind of amount of customizability is that, like I said, I’m aiming for this to be useful on a bunch of different platforms. So far, I’ve mostly just been using this on my desktop. But I want it to be useful on tablets and phones as well.

Ella

And there isn’t a computer version of Vlojure and a iPhone version of Vlojure or a mobile version, it’s all just the same code base, that just kind of dynamically adjusts to whatever aspect ratio you happen to be using. And so in a landscape aspect ratio, it might be more useful to have your formbars on the side. But then if you switch to landscape, like if you’re on your phone, then, of course, you’ll probably want to have them on the top and the bottom of the screen because having them on the side will end up taking a lot of space in the UI. So, yeah, that was kind of that the idea behind the formbars, just having something that’s very, very flexible on that makes it easy to kind of copy and paste stuff to different areas of your programme. And to have something that can store kind of useful functions or whatever in a long-term way.

Ivan

And they’re very similar in purpose to some things that we’ve seen in other tools that we’ve looked at on this show, and that I’m sure you’ve seen in your own explorations of the wonderful world of software, where normally this kind of a panel would be something that stuck on one specific edge of the screen. So there’d be a left sidebar or something like that where your symbol library goes. And I’m thinking here sort of some of Toby Schachman’s projects like recursive drawing, for any of the listeners who are familiar with that where the left sidebar is where each new shape that you create gets a little slot on that left side.

Ivan

And I’m curious, Ella, if you consider doing an interface for Vlojure, that was more rigidly defined, like on the left there is this, on the right there is that or if right from the start, you were thinking about, “No, I can’t do anything where a specific side of the screen is reserved for a specific function because it’s going to be cross-platform in the way that it is now.”

Ella

The latter. Yeah, I always kind of knew that I wanted this to be a cross-platform project. And for that reason, I wanted the UI to be as customizable as possible. So that was always kind of the intention.

Ivan

It’s neat the design that you’ve arrived at with the formbars because they are a sort of an interesting alternative to the sidebar panels. But of course, sidebar panels aren’t flexible in the way that these formbars are. And they’re also similar to floating windows like you’d get on a windowing, desktop operating system, in that they don’t carve out their own space from the main canvas, they kind of float on top of what’s there. And you have that little bit of flexibility to put them on any edge or to have like four of them one along each edge, like four nested structures of formbars because it’s not just like a single bar, you can have them kind of stacked. And is it like an arbitrary depth? Does it only go like one or two or three deep or is it just as many as you want?

Ella

Oh, yeah, it’s arbitrary depth. Right now if you stack too many of them, then it’ll start just kind of like overlapping all the other UI elements. So it’s easy to kind of if you had five formbars to the left side of the screen, then you’re probably not going to really have any room for anything else. But there’s no hard limit on it right now. So add formbars at your own risk, I guess.

Ivan

Yep. And I love that. It’s this nice balance between the structure that you get out of a rigid UI, but the flexibility that you’d get out of floating panels in a way that sort of seems to take a little bit of the best of both. And so I think that these formbars are just for such a simple conceptual ingredient, there’s a lot of interesting detail to the result that you’ve arrived at that I think is worth other folks, perhaps listening to this. And hopefully, the people out there in the world who are seeing Vlojure, picking up on that, and thinking about that because there’s some cool qualities to this UI that I think are a little bit distinct from what I’m used to seeing in other similar projects. Another aspect of the design that I wanted to ask about is, the arrangement of top-level forms is sort of a long, single axis.

Ivan

So you can have a bunch of different top-level forms that are each able to be evaluated separately. And in your video, by default, you have this sort of main axis that these forms are arranged along going from left to right. And so each time you make a new top-level form, it kind of goes to the right of the previous one, or what have you. And you have the option to rotate that main axis. So you can have your top-level forms going up and down the screen or at a kind of an arbitrary angle of your choosing. And that is interesting to me. Because if I were building Vlojure, I would have immediately gone, “No, it should be a free spatial, 2D canvas, and you can put your forums wherever you want them to go and just have free space and zoom in and out and move things around and organise them.”

Ivan

And what you have is something that is more constrained, it’s more structured. And I’m wondering if that’s something that you did, deliberately, where you thought maybe about doing a free 2D canvas and decided not to, or if it was just something you did because that mirrors nicely the way that Clojure works as a sort of a linear language being text-based? Or if it’s just something where that’s what you did for now and you might change that in the future, as the environment grows and evolves?

Ella

The idea behind that was mainly just to kind of keep coherence with the way that people normally write Clojure code in a Clojure file, the forms that you write that the top-level forms aren’t arbitrary, they have an order to them, I really want to maintain kind of compatibility with kind of the normal structure of code in Vlojure. I don’t want to take this too much into kind of being its own environments, I’d really love for this to just be an interface for the existing environment, that is Clojure and ClojureScript. So I think having just kind of this open 2D canvas where you can place them wherever you want would be interesting. But for now, I think that kind of maintaining the kind of linear ordering of forms makes the most sense.

Ivan

That makes total sense to me. That’s a very kind of pragmatic decision in that sense, where my inclination to always make things free, spatial 2D canvases is purely about aesthetics and more of that impulse, I alluded to earlier about wanting to burn everything down and start over again.

Ella

Totally. Yeah, no, I totally see where you’re coming from, without impulse, I’ve had to fight that myself in developing Vlojure. I actually made a prototype not called Vlojure, I don’t think I really had a name for it, but of this basic idea of a visual list with kind of a circular interface, nine months or a year ago. And in that one, I wasn’t using Clojure as kind of the underlying language, I was just sort of defining a new list ad hoc, and so that was kind of the first direction that I tried to take this project. But then I realised, like, “Oh, it’s really hard to design a whole good programming language.” So I kind of stopped working on that, and kind of took a step back and thought, “Well, Clojure already has so many wonderful tools. There’s no point in reinventing the wheel here, I can just make this work as an interface for programming Clojure, rather than trying to build my whole own programming language, in addition to trying to build my own visual interface.

Ivan

On the aspect of it being sort of a mirror of existing text-based Clojure in that there’s also a little button in the top right corner, you can click on, and it shows you the textual representation of the code that you’re working on so you can flip back and forth between the sort of the graphical circles’ version and the text version with parentheses and indentation and such. Are there any features from a more conventional programming tool like a text editor or terminal or something like that isn’t present in Vlojure that you’re sort of or that you have thought about wanting to find a way to bring across into this more visual realm?

Ella

Yeah, absolutely. So I think that one big thing in Clojure and Lisps, in general, is structural editing. So you’re not really thinking about this programme that you’re writing as a text file, but instead, is this hierarchy defined in the form of a text file. But ultimately, it makes more sense to think of your programme as this kind of tree-like structure, rather than a long string of text. And so structural editing, of course, means kind of editing things from that perspective, rather than from the perspective of text. And Vlojure, in some ways, kind of lends itself very nicely to structural editing you’re not dealing with text at all, you’re just dragging around these different forms to various areas. But of course, there are many kind of elements from normal structural editing like plugins and stuff that are missing in Vlojure. Some obvious kind of structural editing tools are there like dragging things around. But there are all kinds of interesting kind of structural editing operations, that aren’t super easy to kind of translate to the kind of drag and drop interface that of Vlojure has.

Ella

And I very much like to kind of include more features to make the editing more powerful. And what I am planning on doing right now is I’m going to be adding certain… So the formbars right now just contain forms. But a toolbar or whatever normally doesn’t just contain things that you can drag into your programme, but it’ll also contain options and different commands that you can click on. And so I’m planning on adding kind of special forms that you can add to your formbars, where maybe one of them will be like if you click on this little button in your formbar, and then you go and click on some form, then it’ll maybe kind of wrap that form in an enclosing circle or something or whatever kind of structural editing operations you might find useful, could I think be built into just sort of a button that could live on one of your formbars, wherever you’d like it to live, and you can click on that, and then it will pull up a little interface or whatever for doing kind of structural editing stuff.

Ella

So that is something that I’m working on right now. And I think that’ll be a big improvement to kind of the UI of Vlojure. But for now, it doesn’t have anything too special like that.

Ivan

I think that would actually be a really interesting way to add the idea of tools. When I think about tools, I mean, that’s such a broad term, but tools in the sense of in Photoshop or something like that, like the Brush tool, and the Selection tool, and the other brush tool, and the other brush tool, all the brushes. Those tools in a Photoshop-like programme, they’re modal, so you activate one of them, and then clicks into the canvas, invoke that tool, and the tool does whatever it does to your pixels in the canvas. And I think it would be really interesting in a Vlojure like environment to have tools palette-like interface. But where the tools are, rather than being modal, you invoke them the same way you do the other thing is Vlojure, where you just grab one and drag it and drop it on to the thing you want it to have its effect on, especially if there are things that are sort of discrete in the kind of effect that they perform rather than like a continuous operation. Something that is invoking this tool does a thing.

Ivan

It seems to me like there could be some very interesting ways to sort of Vlojurefy that way of working. And so I’m personally excited to see how that goes as you explore that bringing editing tools into this because that does seem like that would be a nice way perhaps to do some more complicated edit operations that aren’t perhaps as enjoyable to do just with directly manipulating the nesting of the circles.

Ella

Right. Yeah. Because one relatively basic thing that’s a bit of a hassle to do in Vlojure right now is if you have some forum, and you want to wrap that forum in another set of parentheses, or in the case of Vlojure in a circle. The only way to really do that right now in Vlojure is to kind of drag an empty circle into wherever you want your thing to be, and then drag the old thing into that empty circle and then go back and delete the original thing, which is a three-step process for something that should be one click. So I think I’m hoping that having this kind of formbars that can contain these little commands will really help things be more convenient.

Ivan

Another angle that I like to look at these tools from when thinking about how they could grow and evolve in the future is there’s the angle we’ve been looking at so far, especially when it comes to ideas here are things we can borrow from more conventional text-based or structural editing tools. And those things from borrowing from those tools are about editing the structure of the code. The other sort of half of this is about bringing in some aspects from the execution side, from the runtime side, from the evaluation side, and bringing those more into the environment.

Ivan

And right now what Vlojure has is very similar to other conventional REPLs where you can evaluate a form and get the result back. But there’s nothing in Vlojure yet that is about visualising the evaluation or adding more leverage over how the evaluation happens. There’s nothing as far as I’m aware, like a step debugger or anything like that. There’s no way to kind of blur the lines between the structure of the code as written, and then the actual operational semantics of the code as it’s executed. And I’m wondering if you’ve had any thoughts about that? Or if that’s an area that you’re interested in exploring as you continue to work on this project?

Ella

Yeah. No, absolutely. I think stuff like that is definitely kind of on my radar and on a more basic level. Right now, Vlojure doesn’t really provide you with anything in terms of like errors. If you drag something into the eval circle, and you know that it encounters an error during execution, it just kind of gives you make X in the eval zone to tell you like, “Hey, something went wrong.” But it doesn’t give you the name of the error or whatever. So the first thing I need to do in terms of making debugging an easier process is just having kind of some way for the user to see what kind of errors occurred. And when it comes to things like kind of stepping through code or more kind of specialised debugging tools, I’d love to work on something like that eventually for Vlojure, but that probably won’t be something that I get to all that soon. I’ve got a bunch of other features that are kind of on my to-do list.

Ella

But in the long run, I’d love to see something like that. And one thing that I haven’t talked about much, but that I think might kind of help understand the perspective that I’m coming from is, at some point, once the Vlojure is a little bit more mature, I love to try to make a version of it that works as an in REPL plugin. So it’s not just something that you can use for ClojureScript in your browser, but it’s also something that you can use for JVM Clojure, where it’ll basically act like a plug-in tools, you’ll just run some command on the command line. And it’ll start up a little webserver that connects to run in Clojure process, and it will let you kind of basically have a REPL for JVM Clojure, but using the Vlojure interface.

Ella

And so I think when something like that is done, it’ll be a lot easier to kind of integrate Vlojure with existing kind of debugging tools. So maybe you can… Even if I don’t do anything specifically for a walkthrough or a step-through interface in Vlojure, you can be doing your stuff in Vlojure. And then if you run into some error, and you want to go dig through the stack trace or whatever, then you can do that and some existing tool that’s kind of more specially built for that kind of thing.

Ivan

That’s nice. And that’s an approach that I’m becoming more and more fond of is this idea of building these new programming environments as part of an ecosystem explicitly rather than as their own siloed thing, allowing them to be opened up to interoperate with other tools so that it relieves you of the burden of having to address every use case, and every need and every special way that people like to work by just saying this tool has its strengths and the things that you would want to use it for. But it’s not going to do everything for everyone.

Ivan

And so just plug it into your existing workflow, and use it for the things that it’s really nice for, and then go and use your other tools for the things that they’re better at, which I think… And this might sound a little bit confrontational, but I think it’s an interesting lens to look at things, what would you say are the strengths of Vlojure compared to something like a traditional text editing tool, like if I’m already using some Parinfer something like that, and I have some nice way of writing Lisp code that I’m happy with? Why might I want to use Vlojure as a programming interface to work with? What things would you say that it offers? I’d like to hear you sort of articulate here are the strengths that Vlojure has that make it compelling to use compared to a conventional tool.

Ella

Yeah, that’s a great question. So I think it’d be way over-ambitious for me to expect that any professional Clojure developers are going to abandon their current workflows and start using Vlojure anytime soon. But where I think that Vlojure probably does have more potential is for people who don’t yet know Clojure, or maybe don’t even know programming at all. The biggest benefit that I think that Vlojure has is that for a lot of people who are kind of looking to learn programming, just the idea of like opening some text file and editing text, and running console commands, that stuff is all just really intimidating. But just saying, “Oh, well, here, you just drag these circles around. And you can do normal programming but through this kind of much more friendly looking interface.” I think that’s where Vlojure’s biggest strength lies. So I’m very interested in kind of making this an educational tool for people who are just learning ClojureScripts or perhaps just learning programming for the first time.

Ivan

I could see that being very appealing in that way. Because it’s something that I think is a wonderful strength of visual programming, in general, is that if nothing else disarmingly different from what people might expect when they imagine what programming is like, which is kind of why I’m fond of that opening question. Because when you’re confronted with one of these visual tools, it feels more like maybe the sort of thing that people would think programming was like if they were just looking at Hollywood where people are waving their arms around and making robots do all sorts of things.

Ella

Yeah.

Ivan

People might look at that, if they are very self-serious, and they’re very professional. And they might sort of dismissively say, “Oh, that’s not programming because it doesn’t look like a black and white text editor with a blinking cursor.” And they might be kind of stuffy about it. But on the other hand, I think there’s a bunch of people for whom its visual character is just sort of, if not charming, then at least kind of, it would dispel some of the fear or the intimidation that they might feel by being confronted with the colder forms of programming that are text in a buffer and need to explicitly run some terminal commands in order to compile your code and go through all those hoops.

Ivan

And so I think that’s a really wonderful thing that these kinds of visual tools can offer. And I’m curious, I’ll phrase this question in a slightly different way because I’m genuinely curious, I don’t have any thoughts about this. Any leading thoughts, I should say. Do you foresee Vlojure, having a potential future, where it grows in a way where you would find yourself using it for your own work? Is there some part of it that really speaks to you, and the way that you like to interact with computation or interact with software, something about it that sort of is driving you towards creating it that you think you might want to use, or is it something that you’re creating for a different reason?

Ella

Yeah, that’s a great question. So I guess one strength of Vlojure that I didn’t mention in the last question is just that sometimes I’ll be out. And I’ll have an idea for a little generative art algorithm or something. And I’ll have some code that I want to write, but I’m nowhere near my computer. So I just have to try to remember it. But I always think it’d be great if there were some convenient way to just pull out my phone and do a little bit of programming, right, and then save that code for later. And so that’s kind of one use case where I can maybe see even kind of professional, experienced developers using something like Vlojure as kind of a more convenient way to kind of interact with Vlojure on a touchscreen.

Sponsor

Glide’s mission is to create a billion software developers by 2030 by making software dramatically easier to build. We all marvel at how successful spreadsheets have been at letting non-programmers build complex software. But spreadsheets are a terrible way to distribute software. They are an IDE and the software built in it, rolled into one, and you can’t separate the two. One way to think of Glide is as a spreadsheety programming model, but with a separable frontend and distribution mechanism.

The way it works right now is that you pick a Google Sheet and Glide builds a basic mobile app from the data in the spreadsheet. You can then go and reconfigure it in many different ways, including adding computations and building some pretty complex interactions. Then you click a button and you get a link or a QR code to distribute the app. The data in the app and in the spreadsheet will automatically keep in sync.

For the Glide team that’s just the beginning. Glide needs to become much more powerful. Its declarative computation system has to support many more use cases without becoming yet another formula language. Its imperative actions don’t even have a concept of loops yet, or of transactions. Glide needs to integrate with tons of data sources and scale up to handle much more data. To do all that Glide needs your help. If you’re excited about making end-user software development a reality, go to glideapps.com/jobs and apply to join the team! My thanks to Glide for helping bring us the future of coding.

Sponsor

I would like to thank Replit for sponsoring the transcript of this podcast. Replit is an online REPL that gives you an immediately productive environment to get up and running with any number of different programming languages, frameworks, and tools that you can use, including Git integration. They have a Multiplayer feature so that multiple people can hop into the same REPL and work on the same project together. It’s all very easy to get started with, and easy to scale-up to much bigger projects. They’ve created this fantastic sandbox for trying out new programming ideas, trying out new languages. They have a constant stream of new things that they’re adding, new ideas, new tools. They’re absolutely firing on all cylinders when it comes to making this environment robust and productive. Even if it’s not the kind of environment that you find yourself working in frequently, it’s the kind of thing that’s useful to have in your back pocket, if you ever need to pop up something on the go, or test something out. Or if you want to, as a weekend project, stretch your legs on a new language, but not spend half the weekend installing the compiler and all the dependencies and getting your environment set up and going through all that work. Go to replit.com to check out their programming environment and all of the tools that come with it. My thanks to Replit for sponsoring the transcript and helping bring us the future of coding.

Ivan

I have one more kind of light and fluffy question about Vlojure that I’m curious about. So it’s a visual environment and there are these circles that are nested within other circles. And it’s like the presentation of the code is two-dimensional in its nature. But because of the nesting circles, and because of the fact that it’s a zoomable interface, it made me feel like as you are zooming in on the circles nested within circles, that I’m actually going into the screen, and that there’s some depth to it. And I’m curious if you also think of it that way, where you sort of imagine it as like going into the distance and sort of into a cave as you are going into your code, or do you see it maybe as like a tower coming out, though, that kind of I guess, defies that sort of depth perspective? Or do you visualise this as just like a flat thing where the things are smaller and smaller?

Ella

Yeah, I think I do mainly just visualise it as something flat where just to see more detail, you just have to zoom in, I don’t quite think of it in like a three-dimensional like depth way. But that’s a cool perspective, I like that. Yeah, mainly I just think of it as like a way of representing a tree that’s fundamentally what the core idea of Vlojure is, it’s not even particular or it’s not even specific to lists anytime that you have a tree, you could represent it with something like Vlojure.

Ivan

And I think it’s my feeling that every time I’m descending into nested forms that I’m going into a cave is probably aided by the fact that the user interface is sort of a dark theme, like a dark background with slightly lighter shades for the foreground elements. And so it does have this kind of feel that I’m like, I’ve got a hardhat with a little headlamp on it. And I’m going into the coding cave, as I’m exploring these deeply nested shapes. And so that’s more of a comment than a question, I guess. But it’s something that I’m curious if that was something that you also felt or not, but I guess not. And that brings me to the end of the questions about Vlojure that I had written down that I’ve remembered, and the ones that I came up with along the way.

Ivan

And I’m curious if you had any other aspects of the project that you wanted to talk about before we move on to the last couple questions that I’ve got for this interview?

Ella

Sure. Yeah, actually, there is one thing I’d love to mention. As I mentioned before, I’m very much a generative artist, that’s how I learned to programme in the first place, and I still do that kind of stuff all the time. And as I said, I want Vlojure or I hope that Vlojure can be used as kind of an educational tool. And I think the one thing that would really contribute to that is having a sort of built-in connection between Vlojure and some kind of JavaScript graphics library so that you can be doing kind of generative art through Vlojure so maybe the left half of your screen is what Vlojure currently looks like. And on the right half of your screen is like an HTML canvas that you can use p5.js or Pixi.js or whatever to kind of interact with on your phone or on your computer or anything. So I’m very excited about that. I’d love for this to be kind of just a little kind of cross-platform generative art studio, that you can use on any device. I think that’ll be really exciting once I’ve got that working.

Ivan

Mm-hmm (affirmative). That was actually something I had wondered about as well because it feels like… And it’s one of those things where when I look at the different tools that are made by the people who like doing generative art, and they tend to make these unconventional interfaces in ways that play nicely with generative art, one of the sort of decisions that they always end up having to go one way or the other on is, does the generative art live in the same space as the visual code? Or is it in a separate space?

Ivan

And so that was something I was curious about where if you were going to be using this for generative art? Would you want the generative art to be on its own separate, its own partition of the screen, or something like that? Or have you thought about having the generative art exist in the same canvas as the code that you’re doing? Because there’s, I think so many different interesting design decisions that can come out of going either direction with that decision. And I don’t think that it’s one of those things where there’s an inherently right answer. And so I’m always curious about seeing which way people go one way or the other with that.

Ella

Yeah, that’s a really interesting point. So far, I just kind of been assuming that it’ll be something like one half of the screen is Vlojure and the other half is your canvas. But I suppose you could have something kind of different than that. And I’m just kind of thinking through this in real-time. But the one thing that comes to mind is rather than having a fixed background colour, the kind of background that you’re seeing could just be your HTML canvas. And the kind of the structure of your code is just overlaid on top of that canvass and maybe if you just sit there and don’t touch anything, then it kind of fades out. So you can see what’s your piece looks like without a bunch of code overlaid on top of it. But, yeah, I don’t know, I’ll have to think about that.

Ivan

Another thought that occurred to me, since we’re doing the thinking through it in real-time thing, which I love to do on this podcast, is you could have it so that any of your forms like your circular shapes in the visual canvas could have a separate view where they show their evaluated result, like overtop of that circle. So you’re flipping between, show it as the code view and show it as the result view. And if the form is evaluating some kind of a renderable thing, like a bitmap or something like that it could just draw right within the circle itself.

Ivan

And so you could have… Whether it’s this one circular construction that I’ve made, that’s running this algorithm is I could flip it to a different view and see it as the rendered result or whether maybe there’s a second top-level form that’s like this form is where my render target is. And so I’m going to be in the one circle over here, building up the algorithm to constantly recompute with the pixels are in that target. And then I on the next circle over on my main axis is where the actual imagery is being displayed. It feels like there’s just a couple of different ways that you could sort of squeeze that idea of showing some of the result of the rendered image that you’re creating kind of right in line with the code itself, though that does a little bit deviate from what’s nice about having the eval corner too.

Ella

Yeah. No, I haven’t really thought about having just sort of live evaluation, like whenever you change something, it just kind of reevaluates the entire forum. Because I’m very much been thinking of this as like a REPL driven workflow like you write your code, and then whenever you want, you send that to the REPL to evaluate it. But I totally see where you’re coming from, that could be a big benefit for something visual, where just as soon as you change any bit of code, you can see what effect that has on your image, that would be pretty cool.

Ivan

It’s one of the things that I harp on about a lot on this show is like, “Show data in the code, show the data and the code in the same place.” So I’m sure I sound like a broken record to a lot of folks listening to this. Sorry, not sorry. So the one other thing I wanted to get into, now that we’re into the back half of the episode, and we can get into this kind of deeper, more nebulous kind of stuff is your project Broth.

Ella

Oh, right. Yeah.

Ivan

Because it’s… I’m coming into this a little bit unprepared. In that, I’ve seen some of the imagery of this project in action as it was when you were working on it in your blog posts sort of talking about it. But I definitely didn’t read the post closely enough to come to a good understanding of how it worked and what it is. But what I found interesting about it was it looks to me like just a very high-level summary. It’s a sort of a virtual, artificial life environment that you created where you set up. And correct me if I’m wrong, but you set up this sort of graphical environment where you have a number of entities in this environment, and they all have their rules guiding their behaviour. And it’s the system where you let an evolutionary process play out. Is that kind of a fair, very high-level summary of it?

Ella

Yeah, yeah. No, that’s, that’s absolutely right. Yes. So Broth is was a project in kind of artificial life. And broadly, the idea behind artificial life, or at least kind of the digital part of it is to create simulations that kind of capture aspects of biological evolution. So random mutation, followed by some kind of natural selection process. And the hope is that we can create simulations, where things just as kind of rich as the biological organisms that we see in real life can evolve within a kind of simulated worlds. And so that’s what Broth was an attempt to doing.

Ivan

I found that project and similar projects, I’m sure a very familiar if not maybe directly related example that we could point to is like Conway’s Game of Life.

Ella

Yeah, absolutely.

Ivan

There’s organisms in that. And it’s evolutionary in its nature. And there have been all sorts of interesting projects, taking that and the idea of cellular automaton and sort of spinning it into all these different permutations and these different things that have grown out of it. And even so far as I think Wolfram is doing some kind of model of physics and trying to unify-

Ella

Right. Yeah, theory of everything.

Ivan

… using something derived from cellular automata is like a foundational principle or something like that. There’s some wild stuff going on there.

Ella

Yeah. Stephen Wolfram is a really interesting guy. I am kind of sceptical that there anything is going to come with that theory. But it’s definitely interesting to read about. Yeah, he’s on really wonderful work on kind of the theory of cellular automata in general. And kind of the graph-based kind of model that he’s using for physics is really interesting. Normally, cellular automata, run-on kind of a fixed square grid or something or sometimes a hexagonal grid or a triangular grid or whatever. But it’s a very kind of fixed regular kind of backgrounds. But what Wolfram is working on right now, instead of having kind of a fixed background, you have this graph that kind of represents both space and the things within space. And so kind of the structure of space itself kind of evolves over time, according to the rules of the simulation. And so that I think is a really interesting kind of elaboration on the idea of cellular automata. And I find that work really inspiring.

Ivan

There was a recent article that he put up about this work, it was interesting to me because it was presented as something that was talking about sort of fundamental aspects of physics, and looking for maybe something even more foundational that he could use to express what we now have a physics in terms of a more foundational thing. And what he arrived at was something that I couldn’t make heads or tails of as a non-physicist, but it was very meaningful to me, in terms of how it made me reflect on the behaviour of programming interfaces, and computation, especially the sensations of time within programming, which is a thing that I’ve been thinking a lot about lately. And so what I liked about looking at your project Broth, even though the aspects of it, which are about artificial life went well over my head, it looked to me like a very interesting environment for coming up with a programming model, it’s got that nice relation that people have made programming models based on cellular automata that’s a classic ISO lang kind of project that folks do.

Ivan

And so I’m curious if in your work on the Broth project if you see anything in there that feels to you like it’s relevant to folks who are working on programming languages, and if there’s any parts of that you might point to as things that you played with the things that you found where it’s like, this was interesting for simulating artificial life and as a simulation of something, but it’s also something that would be need to work with as a way of expressing computation more generally.

Ella

That’s a really interesting question. Unfortunately, I’m not sure if I have an interesting answer. But the idea behind Broth was it was more of a research project than anything into just kind of some kind of ideas about how to achieve some goals in artificial life. So I hadn’t really thought about it in the context of what it might contribute to programming interfaces. Yeah, so I’m not sure if I have much else to say there, unfortunately.

Ivan

It’s totally fine. And that’s one of the things I do on this show is I occasionally will throw kind of a… I won’t say a curveball or I’ll just throw a what-if kind of question at folks and see if there’s anything there.

Ella

Sure, sure.

Ivan

Usually there’s not. That’s what we’re here for. We’re trying-

Ella

We’re trying. Yeah, absolutely.

Ivan

Are there other projects like this that you’ve created where you’ve sort of been playing with dynamic visualisations and interactivity and interfaces and that sort of thing that might be of interest to the folks listening to this show to go and check out?

Ella

Oh, yeah. Absolutely, I’ve got a bunch of cool stuff that I haven’t gotten around to kind of writing up or making videos about yet. And I suppose one thing that one technique that I’ve been thinking a lot about recently, which is useful in generative art, but also, I think might potentially be applicable in a wide variety of areas is this concept of aesthetic selection. And aesthetic selection is a technique that’s very much inspired by kind of evolutionary ideas. Whereas, in biological evolution you have, basically, the two steps are mutation and the natural selection. So mutation is just kind of blind variation of some existing genome or whatever. And then after you’ve created a bunch of variations, you kind of narrow down this pool of variations by natural selection.

Ella

But of course, there are… You don’t have to just use natural selection, you can use any kind of selection mechanism. And there are all kinds of useful techniques and evolutionary computation and machine learning, that kind of apply different kinds of selection techniques to get sort of different evolutionary systems. And aesthetic selection is a technique where you have some system that is producing kind of variations. And then, in addition to that, you yourself kind of act as the selection mechanism. So a good example, here is a project called Picbreeder by Kenneth Stanley, that anybody can Google and it’s very fun to play around with.

Ella

And the idea is that you start off with just kind of a random image generated by an algorithm. And then the system creates a bunch of variations on this image for you. And you go through and select the ones that you like the best. And then you say, okay, well, it creates new variations of these, and this kind of iterative process where this algorithm is generating kind of variations for you, and you’re picking out the ones that you like the best.

Ella

And so it’s kind of this evolutionary process that involves both a computer and a user. And you kind of get to just use your aesthetic sense as kind of the selection mechanism in this evolutionary process. And I’ve been exploring kind of systems that use aesthetic selection a lot in my generative art recently. And I think it’s a really fascinating technique. Because art in general can kind of be thought of as you’re kind of conjecturing new ideas about what might make an interesting piece of art. You’re kind of saying, “Well, maybe this would work, maybe that would work.” And then you say, “Well, no, that probably wouldn’t work or that wouldn’t work.”

Ella

And the way that I think of aesthetic selection, is that it’s essentially getting a computer to do the conjecture part for you or at least help you in doing the conjecture part. And then your job is just limited to kind of criticising to saying, “Okay, well, I don’t like that one. I do like this one. So give me more that look like that.” And so the applications of this technique are most obvious in generative art, but I think that it might have applications much more broadly, I’d love to see systems where maybe a user can customise their avatar on something by doing a sort of aesthetic selection, where it takes the current avatar and kind of modifies it and does stuff like that. So that’s the technique that I’ve been thinking a lot about recently, and I think has a tonne of potential.

Ivan

It makes me wonder kind of two things. And maybe you’re familiar with some of the work in this area, and could say whether these things exist, or if they’ve been shown to not work very well. The one thing I’m wondering is, if there’s any interface that would allow you rather than just picking the example that you prefer the most if there’s some way to sort of enrich the dialogue between the computer side of the system and the human side of it, to say what it is that you like or dislike about a particular thing that you’re selecting?

Ella

Yeah, I’m not familiar with any systems that do something like that. And that would require kind of a pretty big kind of modification to the sort of normal way that aesthetic selection works. Because aesthetic selection, fundamentally, is a very simple process. It’s just that you have this algorithm for kind of blindly generating variations. And then the user gets to pick among those variations. And so if you wanted some kind of system to sort of not just have it be blindly varying but kind of trying to vary in the direction that a user has kind of specified, then that would make things a lot more complicated.

Ella

And one thing that I think is a really important insight that Kenneth Stanley, who I mentioned before, has made about kind of evolutionary systems is that it’s often counterproductive to kind of be too focused in advance on what you want when you’re interacting with kind of an aesthetic selection system if you have a goal in mind like I want to evolve in an image that looks like a cat or something, you’re never going to find something like that, right? It’s just way too hard to kind of search through enough of the space to define something that fits any particular preconceived goal that you have. And instead, aesthetic selection works best as this very exploratory process, where you don’t have anything in particular in mind that you’re looking for, you’re just kind of seeing these different ideas coming at you and just looking at whatever piques your interest. So it’s not I have this particular goal, and I can articulate the things that I like about these ones rather than these ones, instead of just, “Oh, that seems interesting. Let me explore that space more or this seems interesting, let’s go explore that space more.”

Ivan

And is this technique, is this idea of aesthetic selection, is it something that is so far being applied only to images and sort of generating art or imagery? Or are there people applying this idea to other media as well.

Ella

As far as I know, all of the stuff that I’ve seen has been kind of in the medium of visual art either images or like 3D models, or short little animations, but it can absolutely be used in kind of a broader way, something that I’ve been exploring recently has been kind of using it to generate sounds. So I’ve got this kind of little prototype of a sound design tool that I’m working on, where you click a button and it kind of plays you a random sound. And then you can play a bunch of these random sounds, and you pick the ones that you like best, and it’ll generate variations on those. And you can end up finding some really interesting and unique sounds using a tool like this. And so I think anything that can be sort of generated by an algorithm can, in principle be used with aesthetic selection. So historically, I think it’s mostly been used with images and other visual stuff, but I think it can be used for absolutely anything.

Ivan

It reminds me a little bit of this sound effect tool that was very popular in the indie game scene. I mean, it’s still popular, but I think it had its heyday maybe 10 or 15 years ago, called sfxr.

Ella

Oh, that’s right. I used that, like 10 years ago, and you’re just jogging my memory now. And that did have a way that you could kind of vary a sound randomly, right?

Ivan

Yeah. Because it had like… It’s just a very, very simple sort of original Nintendo style synthesis tool where it had like a noise oscillator, and a sine wave and a triangle. And it had a couple of parameters for each of them, like amplitude and frequency and some ADSR kind of stuff. And so it had 20 or 30 different sliders. And you could hit a button to randomise all the sliders. And you’d get just all sorts of weird random noises. Or you could pick one of the presets like this is a sound for an explosion or a laser or a coin pickup or that kind of thing. And then there was a button that vary all of the sliders just a little bit. And so you could jam on that button over and over again and get like slight permutations. But it wasn’t directed. It wasn’t like, generate it in a particular like, “Oh, I like this variation better.” So I can see this aesthetic selection being like a much nicer interface for using a tool like that.”

Ella

Yeah. No, absolutely. I’m glad that you brought that up because I’d forgotten about sfxr. And that’s a great example of kind of a simple little aesthetic selection tool. But, yeah, I think the main thing, probably that was limiting in sfxr was as you said it, the way that it did, the variation was that it had 20 sliders or something. And it just jiggle them around randomly for each one. And I think a much more powerful model of evolutionary computation is rather than having the object that you’re varying being like a vector of numbers that you jiggle around a little bit, you can actually evolve programmes themselves. And Lisp is actually a great language for doing this because a Lisp programme can be thought of as a tree. And it’s very easy to create variations on programmes that are expressed as trees, you can just kind of delete a little part of the tree and fill it in with some random operations. Or you can kind of take two trees and kind of snap a branch off one a tree and kind of shove it into the other tree.

Ella

So what I’m most interested in with aesthetic selection is kind of evolving programmes like this. And the programmes can be used to generate images, to generate sounds, whatever you want, but the fact that you’re kind of searching the space of all possible programmes effectively means that you’re kind of exploring the richest possible space that there is to explore. Whereas, if you’re just kind of varying the positions of 20 different sliders or something, then you’re pretty limited the kinds of sounds that you’re going to produce.

Ivan

No, you do say the richest possible space. But I think I given the choice between exploring programming and exploring procedurally generated chocolates, I think I would much rather have aesthetic selection guiding me towards the ultimate tasty treat. But on a more serious note, the idea of using aesthetic selection with programmes is very interesting because it makes me wonder, all sorts of things like what would you have to do to ensure that the programmes were sound? Is it doing some quick halting problem, like check on each of the variations that it’s generating? Or is it using type systems or something like that to ensure that the programmes that are being generated are actually valid in some way? What do you know about how to make it so that these generated programmes are actually going to actually produce output? Or is that just part of the selection criteria that you, as the user of the tool are going to evaluate these programmes on the basis of?

Ella

Yeah. You raised a bunch of great points there. And if anybody is interested in this kind of idea of kind of evolving programmes, I’d highly recommend John Koza’s book Genetic Programming, which most of what’s inspired my kind of views on kind of evolutionary computation. And he goes over different algorithms for evolving less programmes by kind of swapping different bits of the programme trees. And so when you’re looking to… You mentioned first off of the issue of types of how do you ensure that the right kind of data is being fed to each function or whatever. And one very easy way to do that is just to kind of have it so that your language that you’re working in only has one type, right?

Ella

So maybe all of your functions take in a list of or like a vector of numbers or maybe multiple vectors of numbers and return a vector of numbers. And so if you’re limiting yourself to kind of a situation like that, then that no matter how you connect these functions in a programme tree, you’re going to end up with a valid programme. And you also brought up the problem of halting how do you know that the programmes that you evolve are actually going to halt. And it depends on how you define your language. Because often in genetic programming, you’re not technically exploring the space of all possible programmes because, of course, if you’re exploring that space, then you’re inevitably going to have some programmes that don’t halt.

Ella

So instead rather than exploring the space of all programmes, you can explore some smaller set, like the set of all primitive recursive functions or something, so that you know everything is guaranteed to halt in a finite amount of time. Or an alternative approach would be to say, “Okay, we will include things like recursion and all this stuff that leads to potentially non-holding programmes, but we’re only going to run it for 1,000 steps worth of computation. And at that point, we’re just going to stop it wherever it is, and take the current results.

Ella

So this is very kind of nuanced topic. And there’s a lot of different approaches to solving all the different problems that you mentioned, but thankfully, it’s a well-researched area. And there’s a lot of good literature on the subject.

Ivan

Are there any specific projects that you’re working on, or any ways that you’re working to apply these ideas to your generative artwork or other work that you’re doing that you feel like talking about at this point?

Ella

Yeah, absolutely. I’m hoping to make some videos about kind of the way that I’m using aesthetic selection soon. So hopefully, that’ll be a good opportunity to kind of go into a lot more detail about this stuff. But one thing that I’ve been exploring recently, is just using it to generate kind of little looping animations. And in particular, the way that this works is that you’re evolving a programme, right, which represents a function. And you generate an image from one of these functions by looking at each pixel on an image and kind of taking the X and Y coordinates of that pixel, feeding that into the function. And then the function returns a colour, and you assign that colour to that pixel. And then you go to the next pixel and do this for all the pixels in the image.

Ella

Depending on kind of the character of your function, you’ll get vastly different results. pixels that are nearby one another tend to have similar outcomes from the programme. So there’s still like visual coherence the whole thing. And well, that process, technically would just produce a static image because that doesn’t take time into account. But if you want to create something like an animation rather than a static image, then you do something similar, but rather than just taking in the X and Y coordinates, you also take in a time coordinate so that you’re producing a video rather than just a still image. And the process that I’m using there is very similar to plenty of artworks that have been done in the past. I think I mentioned Picbreeder before by Kenneth Stanley, that’s essentially the same process there that I’m using in my work right now. But I’m just applying it to kind of animations rather than still images.

Ivan

I look forward to seeing a video on your channel about how to use aesthetic selection to generate a YouTube channel where we can have an infinite supply of these videos from you about things like aesthetic selection.

Ella

Yeah, I’ll make that collaborative aesthetic selection programmes so that everybody can design their own generative art YouTube videos.

Ivan

I think that would be a beautiful thing to bring into the room. Well, Ella, thank you so much for coming on our podcast and doing this interview with me.

Ella

Of course, thank you for having me.