On New IDEs

There has been some brouhaha about the state of Common Lisp IDEs, and a few notable reactions to that, so I’m adding my two Euro cents to the conversation.

What is a community ?

It’s a common mistake to refer to some people doing a certain thing as a “community”, and it’s easy to imagine ridiculous examples: the community of suburban lawn-mowing dwellers, the community of wearers of green jackets, the community of programmers-at-large etc…

That’s all fine and you might even have smiled at the examples, but that still doesn’t tell us what a community is. A good definition would be that a group of people need to meet three conditions to qualify as a community:

• there must be a shared resource (the commons) that all of them need to use
• the beneficiaries of the resource want its continued development or the resource is somewhat finite
• the members decide to give themselves a set of rules that govern the common enjoyment of that resource

It then becomes clear that Common Lisp is not a community because it has a standard. The whole point of a standard is to avoid having a community because having common rules also means common objectives and that means that the members have to compromise, and people don’t like to do that; instead they make standards which allows them to go away and do their own thing as long as they comply with the standard (and also because a standard committee sounds awfully similar to a cartel, so companies need a legally codified way to be able to collaborate without running afoul of antitrust laws).

Having a standard is a double-edged sword: you can make your own implementation knowing that all you need is the standard, instead of having to comply with a certain implementation’s features and bugs, which might even change incompatibly every so often; but then the community of the standard is formed by the implementers themselves, not the end users. That works fine as long as there are implementers willing to put the up-to-date (read “commercial interests”) - such as for HTML, C, C++, Ada - but it leaves users shipwrecked when that interest dwindles.

Is there a Lisp one ?

No, Common Lisp has communities around the single implementations: the open-source ones - SBCL, Clisp, ECL - and the proprietary ones, mostly Allegro and Lispworks (this latter is called HUG from the times when Lispworks was called Harlequin Common Lisp).

There are also communities around communication spaces like IRC (now on LiberaChat after the Freenode debâcle), Discord and some fora that I forgot about.

There’s one annual conference that has been carried forth by a very dedicated and competent team since 2011, and it seems that Lisp is Big in Japan (but we seldom hear about that due to geographical and linguistic remoteness).

So where does that leave us ? The nice way to describe the Lisp environment is that you don’t have the strife that you see around communities, and Lispers are classical anarchists who start projects and say “Follow me, O brave warriors !” (but one is lucky to get even one follower most of the times). Like it or not, hierarchical organizations get things done.

The less nice way is to say that we’re like the Western Roman Empire after the fall of Rome: the mighty city has been abandoned and we all live a life of subsistence in tiny villages around the old Empire pouting about the old glory. There are pockets of almost-like-before civilization, but it’s mostly poverty that does not allow undertaking, only dreaming of, large projects.

What does it all have to do with IDEs ?

Who will build it ?

Common Lisp is complicated enough that you need people with sufficient expertise to build a new IDE; a 14-year old won’t do it. I won’t go into details because this point seems self-evident to me (but someone please prove me wrong).

Why build one ?

A new IDE is a very large undertaking, even for experienced developers, that it only makes sense to build one if it represents a significant improvement over the status quo. I don’t mean aiming for a “Computer ! Create new program !” experience à la Star Trek, but it needs to be an improvement for those who make it, and that means an improvement over Emacs+Slime.

This last point is crucial, because while in other places you might get the elders of the community to create tools that they themselves would not use, for the benefit of the neophytes, I’m sure as hell that’s not going to happen with Lispers (if herding cats was ever an apt metaphor for anyone, that’s us Lispers).

What do new users want ?

From conversations with a certain number of people (and beware! This is not a scientifically valid anthropological survey), I’ve gathered that the developers who are put off by Emacs do so because of the lack of a few features:

• The presence of menus for discoverability, which makes sense when there are many features you might use only once a month, but there are many of them and you certainly don’t want to waste hours every time figuring out how to make it work.
• Integration with external tools (linters, static checkers, package managers, Android/iOS emulators). There are a lot of useful tools that are not part of an IDE but very useful.
• Integration with web services such as forges, issue trackers, CI/CD environments (Github, Gitlab, Atlassian, etc…)

Especially the integration with external tools and web services tends to be tedious and difficult to implement, and require constant maintenance because many of the tools were not designed to be automated by an IDE (or even a CI/CD environment) and web services tend to “move fast and break often” (it’s the spirit of the times). I do believe the tedium of it all is a good reason why many who can afford are happy to just pay for an IntelliJ licence rather than contribute to Eclipse.

Notably missing from that list are features that some old-time users of MCL or Symbolics would like to resurrect: in-image modification of the editor itself and the windowing system, deep integration with the OS, debugging via shipping images, building desktop GUI apps, etc… These are all cool features, but not the ones demanded by those who are not dedicated Lispers.

My pet peeve is structured editing, and by that I mean coming up with a new serialization format for Lisp code, which would allow an editor to more easily parse and modify it. That’s another feature that doesn’t garner much interest outside narrow circles.

Why not Lispworks ?

Because many of the new users that we’d like to attract can’t afford it. We’re talking about students and people living in countries outside the wealthy West, for whom the full licence is simply exorbitant; and yes, I know some can afford it. I consider them lucky.

The Lispworks IDE is also not universal, as it’s limited to the Lispworks implementation (well, duh!), but that is also bad because you can’t easily try out or migrate to other implementations.

Why not VSCode ?

From my point of view, it’s a bloated piece of dung with a horrible input latency and other inefficiencies (yes, I have tried to use it), and one that is essentially spyware, given the amount of phoning home it does under the excuse of “A/B testing” or “performance and debugging telemetry”. I’m from the old guard of Free Software supporters and I refuse to have anything to do with it.

What to do then ?

If only experienced Lispers are practically capable of building a new IDE that’s good enough to attract new users, but there’s almost no overlap between the features that the experienced Lispers want and those that the outsiders want, well… not much is going to happen. We need a Deus Ex Machina to break this balance. Who’s volunteering ?