Funny story about drag-and-drop:
At my very first job, we had a weekly meeting to assign tickets and estimates.
Our non-technical PM wrote "drag-and-drop" on a whiteboard, and then my name.
"Gavin, how long do you think this will take?"
"All week. At least. Maybe longer?"
PM laughed, then realized I wasn't making a joke.
"What do you mean 'all week'? That's ridiculous, do I need to give this to someone else?"
"Well, you could, but I wouldn't count on you getting it any sooner."
Appalled, he exclaims: "It's just drag-and-drop!"
After 10-15 minutes of a room full of engineers trying to explain the difficulty, he gave up with an unhappy "Fine."
The phrase "it's just drag-and-drop" became an inside joke for the rest of my time there...
I think non-tech people think that drag and drop is some sort of emergent property because you're pressing the mouse and holding it pressed.
Making something easier and more intuitive for a human is more work as always.
It’s funny that you write this, as when I was at Salesforce, I picked up a nested list drag-and-drop task solo except-testing (as the “go to” UI person on the team), and it took me almost the entire release cycle (1/3rd year release schedule (but picked it up later into it)) to complete it. While it turned out good (and I remember the feature’s implementation getting praise by a senior manager at a sub-org demo meeting), that I was called into a team meeting for a “Root Cause Analysis” (paraphrasing) to discuss why the feature took so long to implement.
I think the Dragology stuff actually gets to the nub of what makes interactive drag-and-drop challenging: it's speculative live editing of a view into your model data.
One of the more complex early modules that I wrote, was an engine for selecting a crop of an image in a scanner control plugin.
It was crazy complicated.
Simple is not simple.
[dead]
While it's a wrong assumption, I don't really think it's dumb for a non-technical person to assume this is fast to implement, and I don't think it's some epic own that he offered pushback on this even though he was wrong.
> do I need to give this to someone else
If said this way he was a jerk. So good on em for making a joke.
"Why would this take so long?" would be better.
My guess like most things "you wanna retrofit audio and data cabling to your house. Find an electrician who can do that in an hour. Yes it is just wires".
>do I need to give this to someone else
Could mean "I didn't know you had so much stuff on your plate that takes precedence". You're giving 0 benefit of doubt to a PM who in my opinion was right to push back because for a non-technical person it's not easy to grasp why drag and drop should be so much harder than other similar impact features.
You could be right. Depends on their tone and previous interactions. This is why it is good to have rituals and processes where e.g. you estimate a bunch of stuff at once and it is clear what kind of estimate it is. Effort, elapsed time or Tony's elapsed time given he is going to a conference Tue to Thu.
> to a PM who in my opinion was right to push back because for a non-technical person...
Ah and there's the rub. Why should a PM of a software project even be a 'non-technical person' (because tbh what's even the point, how would a 'non-technical' person even be able to evaluate technical decisions and their implementation?). And even for multi-disciplinary projects like videogames it's not too much to expect that the project manager has at least a basic understanding of all crafts that work on a game.
E.g.. there's nothing more useless than a PM that's just pushing task bars around in JIRA to make them fit into the next sprint.
...and also: would you fly in a plane, or drive over a bridge that was built under supervision of a 'non-technical' PM? (I know I know... since the Boeing disasters that question is rhetorical, most people don't seem to care)
PM has to be nontechnical so they can manage the symptoms. PM also has to be technical to not trigger the worst of
Or, they can simply be better at management.
The head of a space program doesn't need to be an astronaut. An admiral doesn't have to know how to swim. A PM doesn't have to know all the various engineering disciplines the project may touch.
The comment was derisive, and therefore poor leadership. Period.
Dumb question: why? It’s been a long time since I touched GUI code, but I thought it gave you a message with the URI to the file. How is this significantly different than file-open?
This was circa 2010 or so, and it was a web app. So pre-ES6 JavaScript, before React.
Nowadays you would just "npm install drag-and-drop" but back then you often had to hand-roll solutions.
If anyone can dig up an ancient library on GitHub for drag-and-drop back in those days, I'd imagine it would be several thousand lines long, not to mention the absolute nightmare that testing something like this with a proper framework and accompanying testing library.
Ah, I was thinking desktop applications.
Desktop app drag and drop is also a nightmare. Various frameworks can make it easier or harder, but regardless of what you choose, you have about 5 different states your draggable is in, and all of these states have at least 2 transitions to other states, and all your drop targets have to handle mouse-in/mouse-out/drop-end events, and those events need to expose enough data about the dragged object so that every drop target can not only execute the drop, but also tell you whether dropping is allowed without consuming the data.
It's not insurmountable by any means, but it does have enormous inherent complexity compared to almost every other UI action there is.
It's always been a nightmare.
https://donhopkins.medium.com/the-x-windows-disaster-128d398...
Myth: X Makes Unix “Easy to Use”
Graphical interfaces can only paper over misdesigns and kludges in the underlying operating system; they can’t eliminate them.
The “drag-and-drop” metaphor tries to cover up the Unix file system, but so little of Unix is designed for the desktop metaphor that it’s just one kludge on top of another with little holes and sharp edges popping up everywhere. Maybe the “sag-and-drop” metaphor is more appropriate for such ineffective and unreliable performance.
A shining example is Sun’s Open Windows File Manager, which goes out of its way to display core dump files as cute little red bomb icons. When you double-click on the bomb, it runs a text editor on the core dump. Harmless, but not very useful. But if you intuitively drag and drop the bomb on the DBX Debugger Tool, it does exactly what you’d expect if you were a terrorist: it ties the entire system up, as the core dump (including a huge unmapped gap of zeros) is pumped through the server and into the debugger text window, which inflates to the maximum capacity of swap space, then violently explodes, dumping an even bigger core file in place of your original one, filling up the entire file system, overwhelming the file server, and taking out the File Manager with shrapnel. (This bug has since been fixed.)
But that’s not all: the File Manager puts even more power at your fingertips if you run it as root! When you drag and drop a directory onto itself, it beeps and prints “rename: invalid argument” at the bottom of the window, then instantly deletes the entire directory tree without bothering to update the graphical directory browser.
2010 I'm pretty sure you had jQuery plugins to simplify drag&drop for a lot of use cases. But of course, if your use case was outside of the basic ones easily handled by a jQuery plugin, good luck!
Hit-test where the mouse-down event is fired, hit-test where the mouse-up event is fired. What the hell do you need a library for?
> If anyone can dig up an ancient library on GitHub for drag-and-drop back in those days, I'd imagine it would be several thousand lines long,
I bet, the JS devs who do write code overcompensate for the ones who can only stitch together libraries. I suppose because the expectation is that the library consumer is incapable of writing their own code extending it, so the library itself must support every possible combination of use cases for a million different users, even though the code any individual consumer actually needs is more like 100 loc.
> What the hell do you need a library for?
Think of a grid of cards, and all the possible drag, hover, and drop state transitions it can be in with any other card, and the animations required to make it look nice
Now also consider that you need to persist the final state on the backend, and handle things like rejection, cancellation, failed updates/errors.
It becomes a massive state chart and web of complexity
I mean, maybe we're missing context here. Is that what your boss was actually asking for? 99% of web applications (or, hell, desktop applications) don't have ultra-interactive, fully-responsive-animation drag-and-drop. If he was asking for functional drag-and-drop and the team was implementing top 1% cosmetic drag-and-drop, that's certainly one avenue for a mismatch in expectations.
I think people are talking about two different kinds of drag-and-drop. There is dragging external items (files from the OS, for example) onto the app surface from outside. And there is a list of items, each with its own UI component, being dragged within an app to either change sort order or otherwise change data state. Both complex, and have a lot in common, but many comments here seem to be talking about one or the other without specifying.
What does that look like mid drag, and how?
If you are dragging within the page, then the draggable “ghost” is your problem too (it’s not automatic). As is deciding whether the ghost should affirmatively snap out of and into the source and target containers (that is, whether the destination container should appear to “make room” for the ghost at some transitional point), whether there’s a transition on that, etc.
Drag and drop in desktop non-web GUIs have all sorts of visual affordances that users expect for consistency that are your job in HTML.
I was once quite up-to-date with raw frontend work and I tried to implement this; it’s actually not trivial.
> draggable ghost
Okay, you've added a requirement. Poll the current cursor position after the first hit-test sets the dragging state and call a function to render your object with X% opacity at a higher Z level than anything else. Your 30 minute job has become a 60 minute job.
> As is deciding whether the ghost should affirmatively snap out of and into the source and target containers (that is, whether the destination container should appear to “make room” for the ghost at some point), whether there’s a transition on that,
Most applications don't go this far, including your operating system like Windows that sets baseline UX expectations, so you're well beyond the default expectation for drag-and-drop. If someone asks you to implement drag-and-drop without specifying they want all the bells and whistles, and you respond it's going to take you a week because you're planning to go all-out beyond necessary business requirements, they'd have a valid reason to be like "WTF mate?". Also this is still pretty paltry and not a week's worth of work.
z-index is a CSS property, but doesn't work reliably on the web to always put something on top (IIRC, it positions it within its stacking context - there may be many nested stacking contexts). You need to somewhat carefully make a copy of the dragged DOM node (assuming the thing being moved is all in one DOM node), put it into a separate div which can be in front everything you want it to be in front of; and hide the ghost when you want to do hit testing for the drop. Also, you need to actually think about what you want the drop interaction to do wherever it might land (this last step could scale in complexity indefinitely depending on the webpage).
>Okay, you've added a requirement.
He didn't add a requirement. It was always there, along with many more requirements and complexities and details and nuances. You're just not aware of any of them. Your ignorance and naïveté doesn't justify your aggressive trolling.
>Most applications don't go this far...
Because it's extremely difficult to implement drag-and-drop correctly and completely. Getting those interactions right takes hard work. Their absence doesn't prove they're trivial to implement, just that their developers are lazy.
Your arguments from ignorance are weak, not just because you're hiding behind an anonymous handle, and afraid to stand by your own blustering words and work with your real name, and show us the twelve games you dubiously claim to have implemented drag-and-drop for.
So put up or shut up, and stop condescendingly dismissing other people's arguments and work that you simply don't understand the depth and complexity of, if you refuse to show us your own work that you vaguely allude to and boast of having done.
Did you ever actually do this work?
Do you remember what browser testing was like?
Remember the quirks of browsers’ different understanding of transparency, of z-index, of events, of DOM compliance?
I think the suggestion that getting it right could take a week in the context of a particular application is actually not particularly hyperbolic.
Mind you, I was doing cross-browser testing long enough ago that I am one of those people who can say they independently invented AJAX in-house.
I've done it a dozen times in gamedev, where interfaces and interactivity are routinely built from scratch without any help of a framework. I didn't work on web applications in 2010, but even with the clusterfuck that the web is I find it difficult to believe it was that hard to do.
It could be absolutely exasperating, even in 2010 (especially then since browser support for touch events was still shaking out).
The problem is that even without a framework, you didn’t have a blank slate (which would have been better). You had a moving target of blank slates, you had platform differences, you had drag vs touch-drag, entrenched older browsers and the simple frustration of testing on an array of real devices, all sorts.
Now, if you were talking about 2015, then it starts to get a bit flatter.
From scratch is often way easier than in a browser-compatible way.
It's no accident that after certain complexity people will just go for a blank canvas (afaik Google docs is also canvas-rendered).
The problem is that for drag n drop you mostly want to support normal dom.
It only seems simple to you because you have no idea what you're so overconfidently bloviating about, and you keep dismissing the people who do, even as they explain exactly what you're overlooking. Textbook Dunning-Kruger effect. Your confidence survives every explanation because you refuse to learn anything from it.
You claim you've implemented drag-and-drop "a dozen times", then dismiss other developers' experience as if your anonymous assurances and unsubstantiated claims settle the matter. I'm posting under my own name and linking to my own work. You're asking us to take your anonymous evidence-free expertise on faith while keeping your work conveniently out of reach of the scrutiny you apply so casually to everyone else's. So shut up or put up your dozen examples, or at least merely one. Let us see what you implemented and whether it supports your claims.
I implemented the "drag and drop" direct manipulation object and architecture editing tools in The Sims 1, and it was necessarily excruciatingly complicated to give players effortless control while maintaining the illusion of simplicity. The fact that you may think it's simple is the triumph of years of painstaking design, review, programming, iteration, testing, bug fixing, redesigning, refactoring, tuning, and tweaking.
The Sims, Pie Menus, Edith Editing, and SimAntics Visual Programming Demo:
https://www.youtube.com/watch?v=-exdu4ETscs
In The Sims, you could pick up a chair, carry it without holding the mouse button, see rapid, reversible, and incremental visual feedback as it snapped into valid places or slid through walls and tables where placement wasn't allowed, and rotate it while picking it up or placing it. Making that feel natural takes a lot more thought than saying "just implement drag and drop".
Don’s review of The Sims Design Document, Draft 3 – 8/7/98:
https://donhopkins.com/home/TheSims/TheSimsDesignDocumentDra...
Direct Manipulation:
https://en.wikipedia.org/wiki/Direct_manipulation_interface
>In computer science, human–computer interaction, and interaction design, direct manipulation is an approach to interfaces which involves continuous representation of objects of interest together with rapid, reversible, and incremental actions and feedback. As opposed to other interaction styles, for example, the command language, the intention of direct manipulation is to allow a user to manipulate objects presented to them, using actions that correspond at least loosely to manipulation of physical objects. An example of direct manipulation is resizing a graphical shape, such as a rectangle, by dragging its corners or edges with a mouse.
Randall B. Smith explored this in his paper "Experiences with the Alternate Reality Kit: An Example of the Tension Between Literalism and Magic" (CHI+GI 1987). His Alternate Reality Kit at Xerox PARC let you grab and throw objects with mass and momentum, but also manipulate the physical laws themselves. Deciding where to follow people's physical expectations and where to give them magical powers is part of the design work.
Alternate Reality Kit by Randall Smith at Xerox PARC 1986 (VPRI-0131)
https://www.youtube.com/watch?v=I9LZ6TnSP40
And gamedev actually makes drag-and-drop vastly simpler compared to general purpose desktop drag-and-drop, because you don't have to deal with dragging and dropping to and from other apps that you have no control over, with fundamentally different data models, their own bugs and limitations, partial or downright incorrect implementations of the protocols, and private extensions. You have to query metadata, discover supported formats and representations, and asynchronously negotiate how to transfer, interpret and convert the data, while working around whatever the other apps get wrong.
Have you actually ever worked on implementing a desktop user interface toolkit, and the intricate machinery for general purpose drag-and-drop, or just used them without having any idea how they work under the hood?
From your blithe shallow dismissals of other people's hard work and deep knowledge, it's painfully obvious you've never even heard of the ICCCM, let alone pored over its many chapters of nuanced technical details, diagrams, and state machines, then actually tried to implement non-trivial drag-and-drop that works between any X11 application.
https://donhopkins.medium.com/the-x-windows-disaster-128d398...
>As a result, one of the most amazing pieces of literature to come out of the X Consortium is the “Inter Client Communication Conventions Manual,” more fondly known as the “ICCCM”, “Ice Cubed,” or “I39L” (short for “I, 39 letters, L”). It describes protocols that X clients must use to communicate with each other via the X server, including diverse topics like window management, selections, keyboard and colormap focus, and session management. In short, it tries to cover everything the X designers forgot and tries to fix everything they got wrong. But it was too late — by the time ICCCM was published, people were already writing window managers and toolkits, so each new version of the ICCCM was forced to bend over backwards to be backward compatible with the mistakes of the past.
>The ICCCM is unbelievably dense, it must be followed to the last letter, and it still doesn’t work. ICCCM compliance is one of the most complex ordeals of implementing X toolkits, window managers, and even simple applications. It’s so difficult, that many of the benefits just aren’t worth the hassle of compliance. And when one program doesn’t comply, it screws up other programs. This is the reason cut-and-paste never works properly with X (unless you are cutting and pasting straight ASCII text), drag-and-drop locks up the system, colormaps flash wildly and are never installed at the right time, keyboard focus lags behind the cursor, keys go to the wrong window, and deleting a popup window can quit the whole application. If you want to write an interoperable ICCCM compliant application, you have to crossbar test it with every other application, and with all possible window managers, and then plead with the vendors to fix their problems in the next release.
>In summary, ICCCM is a technological disaster: a toxic waste dump of broken protocols, backward compatibility nightmares, complex nonsolutions to obsolete nonproblems, a twisted mass of scabs and scar tissue intended to cover up the moral and intellectual depravity of the industry’s standard naked emperor.
>"Using these toolkits is like trying to make a bookshelf out of mashed potatoes." - Jamie Zawinski
For more on the complexities of drag-and-drop implementation and interoperability, see my other comment about the myth that X makes Unix "Easy to use":
https://news.ycombinator.com/item?id=50440796
David S H Rosenthal, author of the ICCCM, wrote authoritatively about "The Gorilla Effect", "Recreational Bugs", "The Zero-Tolerance Bugs Policy", "Penalties for Bug Possession", "The Neighborhood Bug-Watch Program", "Hallucinogenic Bugs and Their Symptoms", "Are You Safe if You Use a Toolkit", and other important issues:
even with the clusterfuck that the web is I find it difficult to believe it was that hard to do
This is how I feel about global illumination. Game devs claim it's hard but they're just noobs who don't know how to write a lighting shader.
I'm not a game dev, but it can't be that hard.
so in short, you haven't done "this" work, you have done work in another field with different technical limitations that also has the concept of "drag and drop".
serious question, why, when people from a field you are unfamiliar with tell you that something has particular constraints in that field, do you confidently assert that you know better?
I built a drag and drop content CMS/website builder in native Javascript years ago. The browser APIs are really cumbersome. The other issue is more UX than technical, there are lots of questions to answer. What are you dragging, what is it hovering over actually, what is the intention once it's dropped etc.
Dragging UI items around on a page is very different problem to dragging an external file onto a page
Drag-and-dropping a file directly on the HTML `<input type="file">` file picker works with no code/JS.
If you want to splash up the "drop here" thing it's a frickin nightmare. The events don't fire when they should (e.g. mouse leave doesn't fire so it gets stuck, or you get duplicate events or missing events) so you get a bunch of flicker unless you do a bunch of debouncing and fail safes to cover all the browser quirkiness. And this is the easy drop-a-file scenario.
Doing all the stuff shown in https://www.youtube.com/watch?v=l7v-UwpsEbk I'm not sure I'd even know where to begin.
There are more possible actions than opening files.
I love when non-technical people think they know better than the technical people about technical things...
good thing you had backup. rough hahaha.
My version of that is: “how hard could search be?! Just put a search box there!”
I wish I was making it up.
Obligatory clip https://www.youtube.com/watch?v=XpNdGvbwtf0
everyone at my job knows to never utter the phrase "simple software change" when im within earshot.
Oh, this is very, very cool, and very, very timely!
Paper here: https://declarative-dragging.github.io/D2.pdf
IMO one of the remaining gaps of agentic coding is that it's hard for even multimodal LLMs to check interactive motion effects for correctness and "feel." Like, sure, they can take screenshots of a few snapshotted states, but in terms of how things react as underlying nodes move and (potentially) reflow. How do you QA the feel of moving your wrist rapidly in the context of nested zones?
The answer is that you avoid the need to: you have a (nearly-provably) correct foundation that ensures that behavior is predictable regardless of state, and just render accordingly.
The paper's approach is incredible here: simultaneous optimistic rendering of multiple possible states, and a composable algebra (a small combinator language) over how drop zones and distance are derived - allowing for an extremely high-level interface in practice. Everything feels snappy and right, and the code you write is about as minimal as you can get - though the paper notes that state management is somewhat complex. But this seems solvable!
Once this is ported to HTML, it'll become an incredible part of the modern dev toolkit. I'm very excited for what this means.
i have a basic html version working with solid, using basically the FLIP technique to get the measurements. dunno in what form we might try to release it though, its a bit awkward to generalize over existing frameworks
[flagged]
I think this is a really cool project, but I also really appreciate the introduction to Hazel[0].
[0]: https://hazel.org/
Hazel is such a god tier system, all the respect. Taking programming and making representing things in this live system good is such a huge bridge to a better programming, programming that welcomes the world in.
In huge "Stop Drawing Dead Fish" / "Stop Writing Dead Programs" ways. Which I'm such a fan of. Asked Astra to chase down references on this a couple weeks ago, quite enjoyed the result. https://news.ycombinator.com/item?id=5634662 https://news.ycombinator.com/item?id=33251799 https://news.ycombinator.com/item?id=33270235 https://chatgpt.com/share/6aea25b4-e294-83e9-b925-431bb13b00...
Thanks for the mention. I ran across Declarative Drawing & I really want excuses to use this. But I don't remember Hazel from the video I watched in it!
Appreciable, but what is the novelty exactly? The API design? I see the paper doesn't call out Sketchpad or any of Ivan Sutherland's work anywhere in the paper, neither ThingLab or Juno(-2). They do call out Apparatus, which already introduced speculative rendering. The work also doesn't seem to generate valid transforms, which you have to supply yourself. Discovering valid transforms or solving a more general inverse rendering problem would have been much cooler research.
The Twisted Trees example is really cool. I would not have been able to implement it that well. After playing with it, I think at the most basic the underlying principle is:
> Write your interface as a function of continuous variables, ui = ui(x1,x2,...) = ui(X). When dragging an element, compute the barycentric coordinates λ₁,λ₂,λ₃ of the cursor between the three nearest dropzones. Then render by interpolating the ui parameters between their values at those three dropzones, ui(λ₁X₁ + λ₂X₂ + λ₃X₃).
Interesting idea, but look too complicated for me. What if there are 1000 places where an element can be dropped? What if we select and drag several elements at a time?
Also this approach seems to introduce new complicated concepts like "composables" and even a DSL. Probably writing custom code for Drag-and-drop would be easier that figuring out how this works. I read the page and did not understand much. For example, this phrase is difficult to understand: "Rather than programming all that visual behavior directly, you simply say, for each element you want to make draggable, which future states of your application dragging it should lead to.". Not "simply" at all. Writing new code would be easier especially with LLM.
The idea is that you can describe the predicates, producers, and goals. If I’m understanding correctly. This is how a ton of stuff works: excel, your IaC, etc.
Here the goal can be a render state and it is the solvers problem to figure out what should be there.
So you define these things very atomically, but the behavior automagically finds a valid render state (usually through converting things into a standard planner or solver language, haven’t dived into this repo yet)
Does this also work when the search space becomes large?
I fear this might lead to an exponential blowup.
(The examples shown are all tiny)
I suspect this might almost never be a problem. Because say there are n things on the canvas and they have occupied n slots, you lift one of them and want to drop it somewhere, there are only n spots to drop it to. This gives only n next states, even though there are n! permuatations. You have to drag and drop again to reach some other state.
One artificial example that can generate exponential number of states is this.
You have a list of sets and their subsets below them shown on screen. Now you take one element from one of the sets and add it to another set want things updated, this will trigger exponential computation. Since we (human) can't really visually look at all the exponential subsets anyway it will be not a very useful visulalization.
So exponential state space will be a problem only in the cases we don't care about it. Even then we could choose only few of the states by some criteria.
fwiw even the tiny "drag a ball through a bunch of rotating bent blocks" lags quite noticeably when I start dragging. so I suspect it struggles even when the space is fairly small.
neat technique though!
I didn't read the paper yet, but do you they do space partitioning based on events direction ?
an author here (andrew). down to answer questions though it might be a bit until i get to them
All I have to say is thank you for doing this! This is truly super cool. It’s a rare kind of gem to see thoughtful work like this in our new AI era.
Someone else asked something similiar but basically, how does / should this work when candidate drop targets are many e.g. when dragging a first or second rectangle onto a page in Figma when no relationships have obviously been set.
not 100% sure this is what you're asking but for continuous degrees of freedom i.e. numeric model parameters we have the 'vary' operator, which is used in all our canvas-type examples. more in the paper (section 5 under the numerical optimization header). in the trivial case this allows for unconstrained positioning
so a comment right above yours asks a question I'm curious about too, and I wish that person asked you, so I'm copying it here:
--- Does this also work when the search space becomes large? I fear this might lead to an exponential blowup.
(The examples shown are all tiny) ---
there's a small section on perf in the paper (under implementation). the current implementation has poor asymptotic behavior when there are lots of reachable states. this is somewhat intrinsic to the approach. we've talked about various ideas for better behavior in a lot of common cases, but i think that ultimately for cases with more than a few hundred reachable states, a bespoke implementation may be the way to go
Out of curiosity, does your library support physical springs for animation between states? That’d be a great touch.
so far the library is history-independent wrt transitions, which is a nice simplification but does preclude more physics-y interactions.
Very cool demo! One idea I had in the back of my mind is extending grammar of graphics for interactive visualisations. What you did might be a good way how to approach this! Did you think about doing something like this btw.?
Andrew, thanks for answering questions here. edflsafoiewq's reading of the twisted trees above (take the barycentric coordinates of the cursor between three drop states, then interpolate the UI) is the core of Tom Ngo's Embedded Constraint Graphics (ECG), which I worked on with Tom and Golan Levin at Interval Research in the 1990s (US patent 5933150, expired 2016).
I described ECG on HN just after the patent expired in 2016, when G9.js came up, which jy14898 linked above. Golan drew the faces for his Mouther in the ECG editor, by dragging the eyes and mouth around:
https://news.ycombinator.com/item?id=12572696
In ECG the states are drawn examples at the vertices of simplices, glued along shared faces into a simplicial complex. The state is a point in one simplex, and its barycentric coordinates are the blend weights. You drag any point on the blended picture, the pseudoinverse of the Jacobian turns the drag into a change of weights, and at a boundary the state crosses into the neighboring simplex.
A few parts of that might plug into Dragology: continuous blends between states instead of snapping to the nearest one, several handles held at once (the multitouch case in your limitations section), pinned points, and a complex that says which states may blend, so the author supplies a few examples instead of enumerating every reachable state. Your speculative renders could be the examples: render each candidate state, read off the positions, and that list of numbers is an ECG example.
I just put up an MIT licensed reimplementation with live demos. It starts with a talking face after Golan's Mouther (1995), builds the engine up from two pictures, then points it at a 3D head you turn by the nose, ARKit blendshape coefficients, and window layouts. Beside each drawing it shows the weights, a map of the state space, and the shape of the matrix the solver inverts.
ECG: The Whole Story (live demos):
https://hyperties.org/read/ecg/the-whole-story/
Source, no runtime dependencies:
https://github.com/SimHacker/WillWrightShowForFood/tree/main...
My notes on the limitations in section 9 of your paper and earlier work on each:
https://github.com/SimHacker/WillWrightShowForFood/blob/main...
Nice work! Good to see that someone is still working on making the Human Computer interface that has been around for while and used extensively in many applications more human. Thank you.
Precisely what came to my mind.
Imagine an agentic coding system with a good memory implementation. It's prompted to build an app with sophisticated drag-and-drop features. And then another one... After twenty such apps it realises there must be better way and comes up with declarative dragging.
It's pretty clear to me how far current models are from this being real. I'd say it's a good example of how AGI it not upon us, despite what the big labs are claiming.
Even the remarkable mathematical achievements are in some ways brute-forcing in a limited space. The space of "how could we build this better" is in an entirely different category of scale.
Exactly. There's a big difference between filling in a function or a proof by searching and brute-forcing across existing lemmas or functions and creating truly novel ideas and solutions.
P(Hacker News thread being diverted to talk about AI) is basically 1 isn't it
This API feels as significant to me as Reactive Extensions did when Netflix first debuted it. Thank you for this research! I have a lot to think about!
Very neat, looking forward to playing around with this. We are sorely in need of of more exploration into UX/UI, HCI of the current era of digital tools.
Anyone have an idea of what stack/technique would be used to accomplish a task that this is widely superior for. The work seems cool but I'm having trouble imagining what is it replacing.
Edit: nvm, answered by the Background section of the paper
Speculative rendering sounds awfully expensive.
It’s just rendering? I don’t see why it would be more expensive
Rendering in this context means producing an element tree, or a textual description of one, which is capable of firing off many little (slow) allocations. The implication being that this will not scale to a large number of nodes without slogging the machine unduly.
If it's in memory and off-screen, i think it won't be that slow even for many elements. The actual issue might be the garbage collection pauses, which might become noticeable.
We will continue to reinvent dataflow and planning problems until the end of time; you will be surprised how many fundamental pieces of software use this paradigm
technically all of them since hardware itself is basically a dataflow system
Fantastic work! The interactive demos are incredibly satisfying to play with, visually delightful, and genuinely educational.
Agreed. It’s really good!!
This is revolutionary! I wonder if there's any similar prior art.
The Hazel programming env is dope. I can already imagine using it to explain refactorings to new programmers.
The twisted tree diagrams is also really neat. Is that a new idea as well? I got the beginnings of an idea where it might be a good diagram to display tangled code paths.
all links and buttons in document header are also draggable demonstrations of the concept. very cool
Wow that is an elegant way to do it. This could create a renaissance of drag n drop interactions for the web
I am happy to see this is still on the front page a day later
This is one of the more exciting releases lately in the realm of web dev
It is nice to read a coding article.
Very nice work
Is there any hope this could be turned into a vanilla js library and not attached to a particular SPA framework?
Is there a way to make swiping better?
This is so cool!
Not to be confused with the other D2: the name of a project for Declarative Diagramming: https://d2lang.com/
Interestingly, these 2 technologies would fit well together - putting an agent on this and will hopefully come back with results
I am very pleased with the result
Good work
Drag-and-drop is a user interface feature. More often we need drag tools for DAG (Directed Acyclic Graph), which is an execution architecture. Most work involves dependencies: something must happen before something else can happen. I didn't see DAG here.