✔ Development Work: late 2022 - August 2023: Callable mode, new viewer nodes and more #34
Replies: 27 comments 29 replies
-
|
Hello everyone! 👋 Hope y'all had a nice weekend 🌻! I already finished my research. I'll now be writing about all of my conclusions and how I'll approach the implementation of branching. Thankfully, everything went very very very smoothly, even more than I thought it would. In other words, I expect the branching feature to be implemented in the next few weeks Also, from what I saw from the implementation of looping in other apps, Nodezator works with a different paradigm and can already do everything other apps do in that regard, so it will probably just be a matter of creating tutorials/guides to help people realize the possibilities, but it won't be part of this round of changes. |
Beta Was this translation helpful? Give feedback.
-
|
Hello, everyone, Again, I apologize for the delay. As I said on discord last week, this isn't the only task I've been working on this past weeks, which is why this is taking more time than I expected. Additionally, I think you'll all agree with me that writing about the implementation of branching in a node editor is not something that is so easily done. There's a lot of details, stuff to review, etc. The other projects I've been working on in parallel are also vital for the Indie Python project (which is the parent project of Nodezator) since they target broader demographics and represent the possibility of convincing more people to support the project. Even though my delay is justified, considering my duties as an open source maintainer and also in honor of our patrons who have been selflessly donating to the project I believe it is only fair to share with you my current progress in the branching-related task and a tiny glimpse of what I've been working on in parallel that I've never shared with anyone, not even with people close to me. This will all be in the attachments of this message. For the current progress in the branching related task, I'm sharing with you a PDF of the current state of the draft on the implementation of the branching feature for nodezator. branching_looping_in_nodezator_wip.pdf Have in mind that besides my work on other projects, this draft also requires time not only to write, but to edit the charts/images, take notes, review other texts/code, etc. I've spent a bit of time testing some unrelated utilities to help me with such tasks, like different image editing apps. I ended up using Inkscape, since I think it is more flexible for my needs and allows me to work faster due to my familiarity with the software. For the small glimpse of what I've been working on in parallel, I'm sharing 02 very very very brief videos containing a screen recording of what I've been working on. app_demo.mp4game_demo.mp4I'll offer no comments on those projects at the moment, since I plan to publish an entire video for youtube about such parallel projects, which will make part of the "next phase" of the Indie Python project. As always, all code will be dedicated to the public domain and no content will be hidden behind paywalls. I'd just like to say that work on such other projects will not take priority over my work on Nodezator. Of course I'll need to achieve some balance between my work on Nodezator and the other projects, but I'll always schedule the same amount of time to work on Nodezator that I allocate to work on other apps. At least until the higher priority features are implemented on Nodezator. Just know that I still have a lot of stuff planned for the Indie Python project. It has still a lot to grow over the time. Slowly and steadily! Finally, this task of publishing the full draft should still take more time to finish. I cannot give you a precise estimate, but I expect that between my work on it and on the parallel project I should be able to finish the draft before the end of the month. And hopefully on November we'll have the branching feature implemented on Nodezator. Thank you for your trust and patience. I hope you all have a great rest of week! |
Beta Was this translation helpful? Give feedback.
-
|
Hello, everyone! As I said on the discord server last week, I would share my progress with you as I work on the draft a bit more often, so that you are not left wondering what I'm doing, but instead have an accurate record of my work as I progress in it. In my last message in this discussion I presented the draft as a PDF file, but since the file has SVG images that are difficult to see in the PDF, I thought it would be best for you all if you had access to the draft here on github as a git repository which is actually the format in which I'm working on the draft (using .md files and .svg images). This also has the advantage of allowing you to see the history of my editions in it. I actually only now perfected my workflow on using git for writing pieces like this one rather than for software projects as I'm used to. The tools are the same, of course, but how to structure the project, which formats to use, etc., all of this is different. Have in mind that the first commits were actually created much earlier than shown in the repo history. This is so because I was initially using another version control system called RCS to keep track of my work and only later decided to change back to git. By the way, RCS is still great for single file projects like small texts, scripts and config files. Also remember that I'm working in other stuff for the Indie Python project, and not only in the draft. Here is the link to the github repo containing the draft: it is actually an article on the 📄 Implementation of Branching and Looping in Nodezator. If you have any trouble trying to visualize the SVG images, just right-click the image and select the option to "Open image in new tab" or something similar. Then, in the new tab that opens you can zoom in/out and move freely as you visualize the image. The SVG format is really helpful. I still won't try to guess how much time it will take to finish the draft, but it is almost finished and you can already have a good idea about how I plan to implement the features in Nodezator by reading it on its current state. That's all for now! Have a nice week! |
Beta Was this translation helpful? Give feedback.
-
|
Hello, everyone! I believe it is finally time to go forward with our plans for the implementation of branching/looping in Nodezator. I intend to begin implementing the features in roughly 20 days, on December 10th, to give people additional time to review the draft and share their thoughts. Please, let me know what you think of this. Even though there's still content missing in the draft, such content is not urgent and won't come into play the implementation itself. To be precise, the only pieces missing is a explanation as to why I won't be implementing while-loops (for the looping feature), nor match/case statements (for the branching feature). I'm not abandoning such concepts, but postponing their design and implementation. The reason is that match/case statements will probably require additional tools to be implemented in order for us to be able to take full advantage of them and while-loops, though useful in some cases, are actually seldom used in the apps they are implemented and I believe would also benefit from more time for us to come with proper design (if any, since they are truly not used much on other apps). I repeat, I'm not abandoning such concepts/tools, just postponing them in favor of the more urgent/important if/elif/else and for-loop implementations. I think the nodes representing if/elif/else statements and for-loops will greatly improve the workflow of the Nodezator app and I'd like them to make their way into Nodezator asap. Their design and implementation is already fully explained in the draft. If I do start implementing them by December 10th, we may already have them available by the beginning or middle of January. What do you people, think of this? (as always, this content will be cross-posted on the discord server, so people can answer there as well if they wish) |
Beta Was this translation helpful? Give feedback.
-
|
An unrelated piece of information, since you mentioned dataflow programming at one point: I'm aware of the term, and I remember having looked it up in the past, along with other terms like component-based software engineering/development, but I actually read more about another paradigm, one called Flow-based Programming (FBP). I did so when I was about to develop Nodezator a few years ago. Even so, I only read J. Paul Morrison's book on the topic once, and it was a long time ago. |
Beta Was this translation helpful? Give feedback.
-
This is actually a bug, a nasty one at that. I'm sorry for this. Thank you for pointing it out, I'll solve it ASAP. To be more precise, there's nothing wrong with the underlying code. Both the eval node and the widget are working fine. The problem is my own fault for specifying the wrong settings to the widget. In summary, the widget is set to only allow text representing Python literals (primitives) to be input on it, which is why whenever you input anything that is doesn't conform, the value is set to the previous one. Only the widget had its behaviour limited, the node works fine. For instance, if you pass the string you want from another node, the eval node works fine:
This exact same behaviour can be seen in the widgets in the "Standard lib picks" > ast > This ability to restrict the values that can be typed in a widget is available to the users as well and is described in the manual. It is useful for stuff like preventing that a negative number is typed or to ensure that text typed is a valid Python identifier, for instance. |
Beta Was this translation helpful? Give feedback.
-
|
About the lambda nodes, they are a wonderful idea. 🤩 I actually have been thinking about this for a long time, but postponed their implementation in favor of more urgent stuff. I also wanted to make sure I came up with a design that was simple and easy to use. But honestly, I might actually have been overthinking things in this respect. As you said, a simple node that lets the user write the definition of a lambda (without the I approve and will list it for implementation. Though I suspect the implementation of a node like that would be relatively fast. I may just implement it as soon as I have a little free time. |
Beta Was this translation helpful? Give feedback.
-
|
Hi @KennedyRichard, I've developed a solution that I think is the best of both worlds! The The from typing import TypeVar, Callable, Optional, Any, cast, reveal_type, overload
T = TypeVar('T')
U = TypeVar('U')
@overload
def if_then_else_fp(p: Callable[[T], bool], f: Callable[[T], U], g: Callable[[T], U], x: T) -> U:
...
@overload
def if_then_else_fp(p: Callable[[T], bool], f: Callable[[T], U], g: Callable[[T], U]) -> Callable[[T], U]:
...
def if_then_else_fp(p: Callable[[T], bool],
f: Callable[[T], U],
g: Callable[[T], U],
x: Optional[T] = None) -> Callable[[T], U] | U:
if x is not None:
return f(x) if p(x) else g(x)
else:
return lambda x: f(x) if p(x) else g(x)
main_callable = if_then_else_fpA quick and dirty def lambda_(string_source: str = 'x: x'):
return eval(f'lambda {string_source}')
main_callable = lambda_The FizzBuzz example:
The exported Python code: ### local imports (node callables)
from points2d.point_creation.if_then_else_fp.__main__ import if_then_else_fp
from points2d.point_creation.lambda_.__main__ import lambda_
def if_then_else_fp_fizzbuzz():
"""Execute script version of Python visual graph."""
_4_output = lambda_("x: 'Fizz'", )
_2_output = lambda_("x: 'Buzz'", )
_3_output = lambda_('x: x%5==0', )
_6_output = lambda_("x: 'FizzBuzz'", )
_8_output = lambda_('x: x%15==0', )
_0_output = lambda_('x: str(x)', )
_1_output = 15
_5_output = lambda_('x: x%3==0', )
_15_output = if_then_else_fp(_3_output, _2_output, _0_output, )
_16_output = if_then_else_fp(_5_output, _4_output, _15_output, )
_17_output = if_then_else_fp(_8_output, _6_output, _16_output, _1_output, )
_7_output = print(*(_17_output, ), sep=' ', end='\n', flush=False, )
if __name__ == '__main__':
if_then_else_fp_fizzbuzz()I'm satisfied with this solution because:
What do you think? PS. I think to make things even more convenient for the user, we also need a |
Beta Was this translation helpful? Give feedback.
-
|
Thank you again for your repeated efforts to provide the best solution possible for branching using fp. As always, I'm just stopping by to acknowledge receipt of your message. Will reply properly as soon as I can. At first glance this last update indeed seems to be the best improvement yet over your original solution. Can't wait to pore over each of your recent replies and provide detailed comment on them. Peace. |
Beta Was this translation helpful? Give feedback.
-
|
Update: relevant points were also presented by Mr. @Tom-99 on discord, here's a link to the specific thread. |
Beta Was this translation helpful? Give feedback.
-
|
Since a good amount of time was allocated for the discussions/feedback and new discussions/feedback have not popped up anymore, I believe it is time to begin the implementation. I intend to start the implementation of branching and looping in Nodezator next week. I expect it to take a month or two, but I won't make any promises since these kinds of tasks are hard to predict and I perform other work in parallel in the Indie Python project as well (including planning and performing actions to promote the Indie Python project and the Nodezator app). Have a nice weekend, everybody! |
Beta Was this translation helpful? Give feedback.
-
|
Hello, everyone! Just stopping by to update you on my progress. Please, check the 35 seconds video below: input_play_demo.mp4It looks like a pretty unremarkable video where I demonstrate how to create and print a string variable that says "Hello, world!". However, what is indeed remarkable about this video is that I'm not controlling the mouse or keyboard at all. Nodezator is just playing pre-recorded input in real time. It took me less than 02 weeks to implement this and it will take a few more days to refactor/polish, but everything already works. Many months ago, even before planning on implementing branching and looping, the implementation of automated GUI tests was already a high priority task, as mentioned in the requested/planned features discussion. While planning the implementation of branching and looping, I've been doing some research on how to implement such automated GUI tests and wanted to try some ideas. Thankfully, my attempts proved fruitful and I already have 02 new features working (which I'm still to release): input recording and input playing. This is a great relief for me as the maintainer of Nodezator since these new features solve a lot of key problems and open up new possibilities for the app. The most important contribution of such features is to serve as supporting features for the automated GUI tests to be implemented! We can finally automate every user interaction in Nodezator, thus allowing us to test the app thouroughly every time we want to change it. This will greatly speed up making changes to Nodezator. It will help not only me, but also other people wanting to contribute to Nodezator, helping them test their ideas/proposals faster. In consequence, Nodezator will also be more protected against the introduction of bugs. With Nodezator's source and list of available features growing each day, it was taking an ever-increasing amount of time to test everything manually each time the app was updated. Now I'll finally be able to test faster and more thouroughly. As I just said, new possibilities are opened up as well, since input recording and playing are already pretty useful features on their own, even when not used as support features for testing. For instance, when I'm helping users online to achieve some task in Nodezator, instead of having to describe what they should do, I can just record myself doing what is needed on the app and have them play the recorded input, so they can see the actions play in real time in the app. Another possibility is the implementation of a "demonstration mode" in which the app can demonstrate how to perform specific actions for the users as requested. For the reasons presented above, I intend to invest more time in refactoring/polishing and publishing the input recording and playing features and implementing the automated GUI tests before finally starting my work in branching/looping. I believe my reasoning for doing so (postponing branching/looping) is sound enough. If after what I explained someone still thinks branching/looping shouldn't be postponed further, please, say so, so we can discuss further. It is not my intention to impose my will on the users, regardless of my position as the maintainer. I truly think the inclusion of input recording and playing and automated GUI tests in Nodezator will greatly benefit all the development work to be done in Nodezator in the future. I wish you all a happy new year! Peace |
Beta Was this translation helpful? Give feedback.
-
|
Hello, everyone! How are you doing? Been working on the game project of the Indie Python project this first couple of weeks of the year. I intend to keep working another more week on it, maybe two, then get back to my work on Nodezator. Remember, the Indie Python project has other projects that are also important and that might help bring more donations and supporters to the whole project. Here's a sneak peek: sneak_peek.mp4 |
Beta Was this translation helpful? Give feedback.
-
|
Hello everyone, Sorry for taking so long to update this discussion. I've actually been communicating more frequently on discord, so when in doubt you should always check there. Even so, my objective is to update github discussions with the same frequency. The news I have to share is that you don't need to wait for my local commits to be published to take a look in my freshest work. I shared a folder on Google Drive where you can access copies of my local repositories as I work on my local commits before pushing them. You can access the folder via this easy-to-remember link: https://indiepython.com/local-commits |
Beta Was this translation helpful? Give feedback.
-
|
Just so anyone doesn't accidentally miss it: there's been some new replies in this discussion recently. |
Beta Was this translation helpful? Give feedback.
-
|
Hello, everyone, Just wanted to share a bit of my progress in the implementation of automated GUI testing (and other features on which it depends, like input recording and input playing). I recorded a video of my screen while using the app to demonstrate some of the new stuff. All the actions I performed on the video you can perform as well. Just run the app at commit 2023-Jun-06 edit: although the exact commit hash mentioned above changed due to editions, rebasing, etc., you can still look for a commit near the original date of this post; for instance, the one from 2023-Mar-13 (aa45679) seems like a good candidate. The next time I mention a commit from my local repo I'll include the message as well, to help find it later even if its hash has changed (though the message itself might be edited as well sometimes) input_play_demo_Y2023M03D15.mp4Here are some highlights of the video:
There's still a lot of stuff to implement and it will take a long time, but I'll implement the crucial stuff first so we can take advantage of the tests sooner, and then I’ll implement the rest little by little while implementing other long-awaited features like looping/branching and more at the same time. Even so, I can't give an accurate estimate of the time it will take to finish the crucial stuff, maybe a month or so, but I can't guarantee. Regardless, you'll be updated regularly on the progress of this implementation. That's all for now, Peace |
Beta Was this translation helpful? Give feedback.
-
|
Hello, everyone, Again, a bit of my progress in the implementation of automated GUI testing. I've been working on the front end of different forms while I finish my design of the underlying back end needed to support their operations. There's an inordinate amount of data to move around, so I have to write and ponder a lot not to lose my bearings on the features requirements and resulting design. Specifically, I created some new widgets for the playback forms for both the demonstrations and the automated GUI testing. The widgets are a list box that I'll use to display the available demonstrations and a simple search box where the user will be able to search/filter the available demonstrations. I finished the widgets for now (though there may be stuff to add to them further ahead when new functionality is demanded) and the front end for the demonstrations form is ready as well. Here's a video showing the demonstrations form with the aforementioned widgets: demonstration_form_demo_Y2023M04D03.mp4As always, thank you for your patience, Peace. edit: almost forgot to say: as always, all work is available in the copy of my local repo available here: https://indiepython.com/local-commits If you access the link at a later date and doesn't find the copy of the repo there, it just means the commits were already pushed to the github repo. |
Beta Was this translation helpful? Give feedback.
-
|
Hello, everyone, It's been nearly a month since my last update on the automated GUI testing feature, so I'm here to give you another one. First of all, I'm sorry for taking so much time to update you about the feature. My goal is to decrease the time between my posts to 02 weeks at most. Fortunately, the lack of updates doesn't mean lack of work: it is just that this month I've been working in other stuff in the Indie Python project, rather than the automated GUI testing for Nodezator. Namely, I fixed stuff and improved the manual in the main branch, which you can directly inspect by looking at the commits on the main branch. I also have been reading recommended materials and writing some stuff (still to be posted), all related to the ongoing discussion about my proposal to branching and looping in Nodezator, in response to Mr. Alexander (@OlegAlexander)'s disapproval and concerns (he presented sound arguments related to the benefits of dataflow programming). While dealing with all those tasks, I was able to finally close all the missing gaps in the design of automated GUI testing. As I said in the last post:
Thankfully, the "requirements and resulting design" I mentioned are mostly, if not completely, defined now. In other words, the main stuff is all figured out already which was my biggest challenge, since the feature is so complex and requires so much data to be moved around. Now it is just a matter of time and effort to have the design implemented. However, such time will be quite long still, due to the amount of code to be written and changed. Of course there's still stuff to figure out, but the main stuff is ready in my mind. I plan to write about it soonish so you know how the feature will work and are able to share your opinions as well. As I said in a previous post on March 15th, this is my strategy regarding the implementation of automated GUI testing:
However, I'll probably implement some of the requested features even before implementing the first version of automated GUI tests, though. I'm thinking of implementing the persistent viewer nodes, lambda nodes and enabling drag/drop of files and text into the graph. Nothing is decided yet, I'm just sharing my thoughts. I think that's all for now, Peace |
Beta Was this translation helpful? Give feedback.
-
|
As I mentioned in my last update, I’d be implementing some requested features before resuming my work on the implementation of automated GUI testing. Among the current ones on which I'm working is the feature to cause widgets shadowed by connected sockets to be hidden (since they are not used anyway in such circumstances). This leaves the interface cleaner and free of visual noise. Another one is the feature to enable nodes be collapsed/expanded, that is, to hide/show elements which are not connected (parameters, subparameters and outputs). This also contributes to making the interface cleaner and more compact. Here's a short video, demonstrating such features: collapsible_elements_demo.mp4The changes, though not published yet, are already available in the copy of my local repo, in a new branch called implefeats. Just remember that unpublished commits mean the features are either incomplete (that is, there is still stuff to implement) or the changes weren't tested enough (so there’s probably a lot of errors). These few new commits in the implefeats branch are particularly buggy and the last one is incomplete and will be squashed with the next few complementary commits I’ll make. ETA for pushing these new features is a 02 to 03 weeks, but I still plan to implement more features after publishing these ones. That's all for now! Have a nice weekend! Peace |
Beta Was this translation helpful? Give feedback.
-
|
Hello, everyone, Here's another update on my recent work in Nodezator. As always, a copy of my local repo is available to anyone willing to take a look at the fresh commits. (Edit: I ended up forgetting to upload the copy of my local repo with the latest commits, the ones enabling the callable mode mentioned in this post. I apologize for that. I just uploaded it now. It is prefixed with "_Y2023M05D12", since it is from yesterday.) The short video below demonstrates the ability to make a node pass a reference to the callable it represents. By default, a node represents a call to its callable and thus shows objects representing the callable's signature, that is, the parameters and return value. When a node is set to its "callable mode" it becomes a representation of the callable itself, which can be referenced using the single output socket hanging beside its header. callable_mode_demo.mp4I actually started working on the feature last week, but only this week I could finally put it in action. Still need to update the exporting features (.png, .svg, .html and .py) to take the changes into account. Other changes planned are improvements for the operator nodes, adding the ternary operator ( As always, thank you for your patience. Have a nice weekend! Peace. |
Beta Was this translation helpful? Give feedback.
-
|
Mr. Alexander (@OlegAlexander), Just wanted to let you know that I've been alternating between my work on Nodezator and writing my reply to you. That is, my reply regarding the reading you recommended, your (and Mr. I believe I'm taking a justifiable amount of time, considering all other things I'm doing. However, I failed to give you an accurate estimation of the time I'd take to write my reply. So, even though you told me to take my time, I apologize for taking much longer. Just need to write 02 or 03 more pages to finish it and review some parts (it's currently at 10 pages). I'll let you know in advance that I ultimately decided to give up on my proposed design for branching (and looping) after:
I discuss all of that and more in the reply I'm writing. Have a nice weekend! Peace. |
Beta Was this translation helpful? Give feedback.
-
|
Mr. Alexander (@OlegAlexander), (and anyone else interested in the topic) Thank you for your patience. The text ended up a bit longer than I anticipated. Of course, there's much room to improve, but I believe what's been achieved is enough to address our ongoing discussion regarding branching in Nodezator and its compliance with the dataflow paradigm. Since the text has 19 pages, I'm also sharing this PDF version of it, in case you prefer the format (although the web version has the advantage of allowing the images to be closely inspected by opening them in a tab, etc.). Without further ado, since we have a lot to discuss, let's begin right away! Brief recap and introductionSo, 02 months ago you recommended me a chapter from this book, the chapter about Maya's dependency graph. On the occasion, you claimed to have done so for 02 reasons:
Now that I read the chapter twice, it is finally time to comment on its contents. In addition to all of that, you have been providing ideas of how branching in Nodezator can be achieved using what's already available in Nodezator, as can be seen in the previous replies in this GitHub discussion. Everything based on principles and tools from both functional programming and the dataflow paradigm. So my full reply will address everything, both the chapter and your replies. Here's the structure of this article:
In the first section I'll address the chapter per se, without taking into account the discussion on my branching feature. However, I'll also provide extra info on how Nodezator and other features (not related to branching) relates to what I observed/learned from the chapter. The rest of the articles is much longer, since it is an attempt to stitch many topics together into a concise piece by commenting on their importance/usefulness and the resulting influence on my stance regarding branching in Nodezator. That is, I'll comment on my journey regarding branching in Nodezator and also address your previous replies at the end. I'll also include more information regarding the actual approach I use to make decisions regarding features in Nodezator, my initial stance on branching for Nodezator (even before my proposed design), and my current stance. As I said before I did give up on the implementation of my proposed design for branching (and for looping as well). However, even though I gave up on my proposal, I'll still write about it a bit more, in order to walk you, and whoever reads this, through my decision-making approach and why despite giving up on my proposal I still think it is was valid. The reason for dropping it is simply due to the fact that the current design of Nodezator plus some of the features to be implemented (like the ability to reference the callable on the nodes and the subgraph feature) already provide all the versatility and flexibility needed to handle both branching and looping. The purpose of explaining the validity of my original proposal though is not to pat myself on the back, but rather provide a deeper analysis of the decision-making approach behind it, which I use not only in Nodezator, but the entire Indie Python project. This knowledge will surely be useful to guide future discussions. Because of this, I once more ask you and anyone reading this to read it with an open mind, specially those who didn't agree with my proposal from the beginning. As always, take all the time you need to read and reply. On the contents of the recommended chapterThe contents of the chapter are very accessible and straightforward. It is probably due to how simple, yet powerful, the design of Maya's dependency graph is. That caught me by surprise cause I never thought of a graph as a way to aid in the structure and operation of an app, much less a complex one like Maya. In fact, the concepts are so simple that the entire chapter didn't even need to introduce any code, except for The author even went out of his way to explain how other apps, in contrast, organize/structure their functionality in different modules rather than benefiting from the atomicity, flexibility, extensibility and versatility of a node-based structure. Most of the other concepts following these initial discoveries about Maya's structure based on the dependency graph are actually pretty common not only in the dataflow paradigm, but in other related/similar concepts/paradigms as well, like component-based software engineering/programming (CBSE/CBP) or flow-based prograaming (FBP). Notions like the nodes being individual and independent black-boxes, the extensibility of their operations and how combining them provides versatility, extensibility and power for the environments/apps/features that use them. This is not to say that there was no benefit from revisiting those concepts. On the contrary, seeing how they apply in practice to Maya's design was relevant and enriching. The fact that Maya's dependency graph serves as the foundation over which the graphical user interface operates (mediated by the MEL command engine) was also interesting, especially at the end of the chapter where it is explored how it allows independent features to coexist, like animations driven by the timeline that at the same time allows interactions from the user with the animated objects. It is also interesting how the dataflow behaviour is partially ignored in favor of the push-pull approach. Of course, this is not to say that the dataflow approach was thrown away in the app. As you said before, the push-pull model is just a trick/tool to help optimizing the graph's execution. This particular decision plus other information laid along the chapter provides further insights into the inner workings of Maya and the very nature of the decision-making behind its design. In summary, it was made clear that they understood the value of the data structure that a graph represents per se, independently of any paradigms laid upon it. That is, even before being considered a part of a paradigm, a graph is already useful on its own as a data structure. This perception is what allowed them to bend the dataflow paradigm a bit, for the sake of optimization. They used the dependency graph for what it is above all else: a way to store data representing all attributes, values, behaviours and how they depend on each other. With that in mind, they had freedom to adopt the push-pull approach to efficiently update the app whenever needed, without needing to discard the dataflow paradigm. Nodezator also makes a clear separation between the graph and its execution. So much so, that additional execution behaviours can be easily added. The app has full access to the data in the graph and can use it as it sees fit. For instance, the However, I intend to take advantage of the highly customizable nature of the execution to implement additional execution modes. The current execution algorithm just visits each node, one by one, without a particular order, and checks whether that node has all the data needed to execute. If it has, it is executed and the outputs are sent to the next nodes, if not, it is skipped and we visit the next node. Then we repeat the process revisiting the nodes we skipped earlier until all nodes are visited and executed. During a visit, we also check whether the node is missing data in any of its parameters. If a node doesn't have a needed input from a connection or from a widget, it is deemed inexecutable and the execution is aborted with an error dialogue. There's nothing special about this algorithm, it was just the first thing that came to mind when I created Nodezator. I actually have many written notes about possible improvements to the efficiency of this algorithm and also other execution modes that I want to implement. For instance, the current algorithm executes the node right away as soon as it is visited if it has all data required to execute. This is usually what we want, but if we had a large graph with many nodes and they took considerable time to execute, it would be desirable to first visit all nodes before executing any of them, to see if all their required parameters had connections or widgets providing data. Otherwise, we would only discover later during execution (possibly after a long time) that a node was missing data, thus wasting our time. For now, Nodezator only executes the graph synchronously, one node at a time, but I also have notes describing an additional execution mode that has multiple workers executing nodes as soon as they have all arguments, that is, using concurrency/parallelism. Of course, this is something for the future, after I implement other critical features. I've been studying concurrency/parallelism in Python for quite some time but I feel like I only scratched the surface. The chapter also gave me the idea for another mode where the app also keeps the output of each node cached, and whenever the graph is executed, only the node who have their inputs changed (and the ones downstream from it) are executed. That is, this is similar to what Maya does with the push-pull approach and the dirty flags to avoid executing nodes needlessly. Speaking of the push-pull approach, I'm also glad for having learned about it, cause I was planning to do something similar for the "persistent viewer nodes" feature (it's been in the list of planned/requested features since last year). A persistent viewer node is just a regular viewer node, but instead of just generating the visualization when the graph is executed, the visualization stays visible in the graph after execution. It is something that other node editors have that Nodezator is missing. In addition to having all persistent viewer nodes update when the graph is executed, I also want each individual viewing node to have a button to trigger the update of that node alone, and that's when I plan to do something similar to the push-pull approach. That is, when requesting a single viewing node to update (by clicking in that special button in the node), only the nodes upstream from it will execute to produce the data needed for it to update the visualization. So learning that a similar approach is used in Maya gives me a bit more confidence. There was certainly more content that could be discussed like the implementation of the timeline, how the transform nodes are applied are organized and applied within an hierarchy, etc., but I assume this is beyond what you wanted me to analyze in the chapter. My final conclusion is that the usage of a graph to model and represent all data and dependencies in the Maya gives the app all power and flexibility/versatility/extensibility that it needs. Also, by thinking outside the box, the developers of the app managed to guarantee a better performance for the app by focusing the execution of the graph only in the points where the corresponding data is requested, reducing execution only to the nodes that need to be updated. Brief comment on loopingDespite the fact that my original proposal involved both branching and looping, in order to simplify this already large article, we'll be focusing entirely in comments/analysis of branching designs for Nodezator. Additionally, even though all opposition to my original proposal only ever mention the design of branching, I also rejected its looping-related tools/implications, since they are also indirectly affected by the disapproval of the branching portion, due to also consisting of a deviation of the pure/conventional dataflow-conformant design of Nodezator. My journey regarding branching in NodezatorSummarizing decision making in NodezatorWhen we last discussed about the branching approach in Nodezator, I mentioned an article I'm writing about the decision-making approach behind the birth of Nodezator, the current state of its source, challenges, possibilities, etc. and also provided a copy of its first draft. I'm still working on it, but I'll summarize the part of it which I think is relevant for our discussion: the decision-making approach. First of all, software development is complex. I believe people usually underestimate how difficult it is to implement any kind of meaningful decision-making framework. The Nodezator project is even more difficult in this regard due to its inherent complexity. Another thing to keep in mind is that being an open-source project makes it subject to decision making and discussion by people with very different backgrounds. This factor is actually good because it brings different useful perspectives to the decision making and discussions. It also brings new challenges though. That is, it may also be source of confusion, misunderstandings or just require more time and effort so people involved in a decision/discussion can at least understand the different opinions, regardless of their own positions on the matter in question. This is actually fine though. Good things require time and effort. The trade-offs are worth. I believe in the potential of the whole IndiePython project and its flagship app, Nodezator, to bring more funds to the project over time. Slowly but surely, almost daily, more and more people are becoming aware of their existence and the value they bring to Python development and education. However, until I can fully fund my development time, I can't invest 100% of my time in it (though, fortunately I've been able to invest almost 100% of it), nor hire/outsource people to work full/part time on the projects' tasks. It is okay, though: I'm happy with the slow growth because I'm already quite busy with its current size, so it is better that it keeps growing in small increments as I invest more and more time and resources in the project. My purpose in mentioning all of that is to point out that this situation also influences the decision-making process in the IndiePython project. Until we are big enough, we need to be careful not to adopt a decision-making approach too rigid or complex to the point where it slows down the actual decision making. After all, we must take into consideration that we are a small community with busy people trying to contribute however little time and effort each of us have. With all of that in mind, rather than rules, my approach actually consists of a few principles, some of which I list here:
I think this is all simple and loose enough to allow a healthy and efficient decision making considering all the constraints of the IndiePython project. Now that this decision-making approach was presented, what is specially relevant to our discussion regarding branching in Nodezator is the point about the adoption of paradigms. A lot of node editing apps boldly claim to adopt paradigm X or Y. When subject to closer inspection, though, such claims are often proven inaccurate. When I first studied FBP (flow-based programming) to gather insights for the implementation of Nodezator, the number of apps claiming to be flow-based that were actually proven not to be by the author and other researchers was astounding. This is why I decided not to follow any specific paradigm for Nodezator. Instead, I implemented it based on simple and proven concepts and tools from some useful paradigms. And so, I decided to use a function/callable to represent a blueprint for a node, and a graph structure to store node instances and the relationship between the data that flows through them. I also tried to bring the design as closely related to Python as possible. This approach resulted in many of the useful things that Nodezator has today and that some people take for granted. For instance, I'm not a genius that can develop a generalist node editor that can export a graph to Python code. This Nodezator feature is actually an accident (sweating-smiling emoji). Though I wanted to eventually implement it, I thought it would take some years for Nodezator to be able to do that, but it turns out the concept came to me when I was implementing the other exporting features (.png and .svg). The more one inspects the design, the less impressive it becomes. Nodes are already based in calls to Python callables, so a Python exporting feature is just a matter of "untangling" the nodes in the graph into individual calls and values in a script. This kind of stuff is only possible because I kept the design as simple as possible by implementing stuff iteratively, rather than trying to faithfully reproduce an entire paradigm. My point is not against paradigms, just against stubbornly clinging to them. It is (hard) evidence and/or concise arguments that must guide decision making and adoption/rejection of paradigms or parts of them. Nodezator is not a dataflow or flow-based or functional programming/paradigm application. It is just an app to convert Python callables into nodes that can be arranged into a graph and executed/exported. However, it is true that it adopts dataflow and functional programming principles, concepts and tools. And I want it to keep doing this, but the importance of such principles/concepts/tools comes from their proven usefulness, not from blinding compliance to paradigms. I wanted to make all of this clear just so you and anyone willing to contribute code and ideas to Nodezator have a firm grasp on the decision making behind it. I hope it didn't come off as arrogance on my part. Quite the opposite, it is precisely because I understand my limitations that I must strive to ensure a relevant decision-making approach is put to practice, one which can guide and empower people to discuss and decide on matters related to the project. In addition to that, your feedback and stance against my proposed design to branching in Nodezator, despite always announced as attempts to fiercely defend the dataflow paradigm, have been backed up by concise arguments, specially your detailed demonstrations of leveraging the current design of Nodezator to achieve branching using pure dataflow/functional programming-conformant concepts. In other words, your approach has not been lacking in anything, I just wanted to highlight the importance and relevance of evidence and arguments above paradigm compliance. I acknowledge the strenghts and usefulness of the dataflow paradigm and this is why want to keep Nodezator compatible with it, not of its prestige or "conceptual solidity". With that explained, we can now explore the original problem using the decision-making approach I described. After that, we'll revisit my proposed design. Then, we'll finish by explaining my current stance (the rejection of my proposed design) and how my knowledge on functional programming as well as your comments here on GitHub contributed to it. The problem and my original stanceProposed/requested features must be driven by real problems/use-cases. It was no different in the case of branching and looping in Nodezator. The first person in the project to request such features was Mr. Adams (@WillAdams), which engaged in a lot of discussions since the very release of Nodezator last year. He's currently also a patron and a talented individuals who works with CNC (Computerized Numerical Control), which includes 3D modelling. Our first interactions can be seen in a post on reddit. It spans a lot of comments and even links to another discussion held in a youtube video comment section. Since all of this includes a lot of things unrelated to our discussion of branching and looping, I'll summarize the points discussed and conclusions reached in the following paragraphs. In case you are curious, the original post and comments can be seen here. In summary, Mr. Adams was sharing his concerns regarding his first impressions of the software and was having some problems to perform some actions like looping. As we know, Nodezator currently has no specialized nodes to deal with neither branching nor looping. However, in addition to that, at the time I was not actually even planning on implementing any kind of looping or branching in the app. This is because I wanted to keep the app as simple as possible so it could be used similarly to the Blender app's compositor which also doesn't have branching nor looping. Rather, I wanted branching and looping to be dealt outside Nodezator, by exporting the graphs to a script and doing the necessary modifications there, or just pasting it into another more complete script with all the needed branching/looping logic. Because the Blender app's compositor deals with a finite set of operations and is focused in achieving specific results, like editing an image of image sequence, it doesn't need branching/looping. I was also convinced that, whenever needed, some of the branching and looping could be done inside the nodes instead of in the graph itself. All of this was to avoid making the graph needlessly complex. However, Mr. Adams did present a simple problem which convinced me of the usefulness of branching in Nodezator (I'll present it further below). After pondering about the problem and the differences between Blender and Nodezator I also reached the conclusion that it is only natural for a generalist node editor for the Python language to have branching and looping implemented as well. From them on I've been trying to come up with a satisfactory solution for branching and looping to Nodezator. Then a few months ago I presented my proposal. And that summarizes my conversations with Mr. Adams. Let's now just focus on the problem he presented, which is a simple yet representative example of how one would make use of branching within Nodezator. In his own words, as can be seen in the reddit post previously linked:
My original stance of not implementing branching or nodes specialized for branching in Nodezator would indeed make the problem unnecessarily complex and even a bit less clear, since it would require him to replicate part of his graph across 3 different files. For a software like Blender this is fine, cause one usually have a single result in mind and works in the graph towards such result in mind, tweaking the values as needed. A generalist node editor like Nodezator however, has an infinite set of nodes that can be created and an infinite amount of purposes, depending only on each user's goals. In other words, Nodezator needs a branching feature that allows different paths of execution to be chosen and such paths must be clearly represented in the graph. So, to summarize this entire subsection, I was actually wanting to avoid including branching/looping nodes in Nodezator in order to avoid changing the execution flow. I wanted to keep the current dataflow-compliant execution flow because despite acknowledging the desirability of looping/branching, I didn't think they were strictly necessary inside Nodezator, i. e., they could be explored outside within an external Python script. Additionally, the current dataflow/fp-compliant execution flow already allows looping and branching with Summarizing my solution to branchingAs I mentioned in past conversations, the Elixir language doesn't have for-loops. Instead it uses tools like higher-order functions, recursion, list comprehensions and more to iterate over items. For someone like me, who has only ever used PHP, Javascript and Python, this is understandably mind-boggling at first. Nonetheless, after getting familiar with some functional programming concepts and practices, the usefulness of higher-order functions like Moreover, anyone who spends a bit of time learning about generator-functions and the usage of the Now, what all of those tools from functional programming and Python-specific iteration tools have in common is that they require a bit of understanding of the underlying concepts and how to approach the problems and model the solutions according to those concepts. Not everyone is willing to put the extra effort to understand and employ such concepts in their code/graphs. And quite honestly, some people probably shouldn't, because perhaps what they want to achieve simply doesn't need the extra power/versatility that concepts/tools from FP/dataflow/Python iteration machinery provides. The trade-offs might not be worth for them. This is a serious problem that have potentially killed or hindered the progress of some technologies like the Haskell language and still threatens many others. It is important to be zealous supporters of great tools, but one must not ignore how they are approached by beginners and new adopters, lest they become burdens. In my conversation with Mr. Adams, he jokingly shared this link that points to the anecdote of a programmer that used to write too many lines in his code due to not being aware of the existence of for-loops. Joking aside, the account clearly aims to educate people on the dangers of becoming too invested in specific designs to the detriment of simpler, practical constructs/solutions like the for-loop. I'd like to add that all of those reservations are not about FP/dataflow/Python iteration tools themselves, nor do they imply my disapproval of them or that I found them inherently complex. Quite the contrary, they are awesome tools to bring power, versatility and flexibility to one's code, and many of its constructs/concepts are easy enough to grasp that understanding and using the simplest among them may be enough to significantly improve that quality of the code/systems using them. It is just that, if there's an even simpler solution, we must consider/seek it first. Because of all that, I set to come up with a design as close as possible to branching with if-blocks and looping with for-loops. And thus the solution I proposed (and on which I recently gave up, as I explain further ahead) was born. I believe it is not necessary to explain my proposal here, but I'd like to highlight the aspects that still make the it a valid solution for branching and looping in Nodezator. In addition to such aspects, for the sake of fairness, completeness, and to comply with one of the principles I listed before (the one about being aware of the shortcomings of a solution) I'll also point out my dissatisfactions/the shortcomings of the design. Such highlighted aspects and shortcomings will be briefly presented in the next 02 subsections. After such subsections, we'll finally dive into why I came to gave up on my proposed solution and further comment on your recent replies about exploring the dataflow approach in Nodezator using its existing features. Highlighted aspects that validate the designThere are 03 aspects about my proposal that I'd like to highlight and that I believe makes it a valid option. First, the design for branching matches the analogy of the graph as a set of machines and conveyor belts between them quite accurately, making usage of the analogy to justify the choice of an specific branch in detriment of the others, by ceasing the flow of data into the ignored branches. Of course, the exact mechanism could still be refined, but the core is solid. Second, the design for branching is compatible with the Python exporting feature, that is, it seamlessly exports to a Python script. The only thing that might be of concern is that the graph now contains sets of nodes that won't be executed, which is unusual for node editor in my experience. Nonetheless, considering Nodezator aims to be a 100% match of a Python script, that concern quickly disappears since this is precisely what having if/elif/else-blocks in Python scripts (or really those of most other languages) mean. That is, having if/elif/else-blocks in a script means that parts of the script will execute while other parts won't. The branch whose condition evaluates to True (or the else-block) is executed while the others are ignored. As such, it is not that unnatural for a node editor emulating a Python script to portray such mechanism, even if comes at the expense of a full compliance to the dataflow paradigm. If it is within reason, paradigms can and should be fully or partially put aside. Finally, the design can coexist with a pure dataflow approach. Nodezator is flexible/versatile enough that one can still use a purely dataflow-compliant branching mechanism like the ones you've been demonstrating. Shortcomings/dissatisfactions with the designThere are 03 main shortcomings when it comes to my proposal. They are largely responsible for my ultimate decision of rejecting it. First, though fully or partially parting with a paradigm can be acceptable, it doesn't mean it is encouraged/welcome. Paradigms are not built over irresponsible assumptions. Rather they are usually subject to many tests and scrutiny over time. That means that while blind compliance is a bad thing, justified compliance shields the design of systems against many problems tackled by the paradigm and provide power and versatility inherent to it. Concepts/tools from the dataflow paradigm and functional programming are specially powerful for understanding and greatly simplifying many complex problems and ensure programs/systems remain simple, powerful and flexible at the same time. Second, despite the close resemblance to how if/elif/else-blocks work in Python scripts, there is also some cognitive load to the usage of the specialized nodes that support the design. In short, rather than completely eliminating the cognitive load required to use dataflow/FP concepts and tools, it just replaces it with its own, specially since such design, to my knowledge, is quite unorthodox when it comes to node editors. Just to clarify, I use the term cognitive load rather loosely in this text, meaning an amount of energy and concentration to learn something that is higher than normal. Last but not least, making use of Kanat-Alexander's equation, my proposal has quite the maintenance cost over time. While keeping Nodezator compliant and reliant on dataflow and FP concepts/tools only requires the addition of some feature to fully take advantage of branching and looping in Nodezator, my proposed design would require the creation and maintenance of a different algorithms to handle edition and execution of the graph. Even though the value over time would consist in the full ability to tackle branching and looping in Nodezator for all use-cases/purposes and in a "relatively simple" way, the maintenance cost over time will probably not be worth it. Such cost would be multiplied manifold, whenever implementing new features that influence how the graph is managed/executed. My current stance and reviewing branchingRequirements of a solution to branching/loopingMore or less 05 months since the last update on my proposal and now my stance is not favorable to it anymore. Since the last update in November 2022 I have not been giving my proposal much thought, because I've been working in other unrelated stuff since them. In December I worked on the first version of automated GUI testing for Nodezator, then in January I worked exclusively on the Bionic Blue game and since February I've been further developing automated GUI testing while I refine its design, which I only recently was able to finish. Of course, opportunities to rethink the design were brought up by people disagreeing with it, like Mr. Tom (@Tom-99) in December and you, Mr. Alexander, at the end of March. However, even though both of you expressed your disapproval with sound arguments, I still couldn't abandon my proposal, not because of some personal/emotional attachment to it, but because I still wasn't able to find what I believed to be a better replacement. That is, I wanted an alternative design that could fulfill these requirements as closely as possible:
Of course, although not listed, it must also be 100% translatable into pure Python code, for the Python exporting feature. Problems to achieve the requirementsAchieving the requirements listed above present 02 final problems. I'll present them in the following paragraphs. Since even before publishing my proposal, I was already 100% sure that it was possible to solve any use-case involving looping and branching using only Python's This means I never doubted that keeping Nodezator FP-compliant would allow us to achieve that (and, as a result, we could keep the current dataflow-compliant design as well, which greatly simplifies and empowers the handling the flow and processing of the data). However, here's the first problem: I was still struggling to visualize what Nodezator was lacking in order to take full advantage of such FP tools. That is, I know the FP tools to use, but what does Nodezator lacks so they can be used without any limitations? The second and final problem has to do with the cognitive load of some of the mentioned FP tools. In other words, How to teach and encourage people to model and solve their problems purely using FP tools?. That is, for instance, how do I ask people used to simple for-loops to model all of their problems using This is not a problem inherent to FP, but perhaps it stems from observing other people question the simplicity of FP tools/concepts and struggle to adopt them in their workflow. The links presented earlier about the failure/underachievement of Haskell teach us that we must not underestimate how challenging new tools/concepts are to beginners. I myself, despite my limited knowledge, experienced no particular problems when learning and using concept/tools from FP and from Python iteration machinery in my daily programming. Nodezator's source is filled with examples of usage of higher-order functions like map or reduce, identity functions and recursion (with the However, we must not be tempted to believe other people presented to such concepts will have the same ease of understanding and applying such concepts to their code, much less within a node-based interface, which is yet another unusual factor to take into account. Put simply, though, this second problem is just the obstacle preventing us to meet the second requirement mentioned before. Meeting the requirements with FP toolsOnly recently I was finally able to close the gap preventing me to achieve a solution to the problems and, as a consequence, meet the requirements presented before. What contributed to such achievement was my knowledge of functional programming (limited as it may be), extra research/pondering over the time, and our discussions here on GitHub, Mr. Alexander (not only in this post, but in others as well, like when we discussed subgraphs). All those sources of information, inspiration and discussions were important because the final solution achieved consists of a combination of various ideas, concepts and features. It isn't a feature proposal, cause it doesn't represent a new feature. It is just the usage of features we already know, either already implemented, being implemented right now (the callable mode, for instance) or things already listed for implementation (or at least mentioned in discussions, like group nodes/subgraphs). In other words, the solution is achieved within Nodezator's current design. The features used also represent various FP tools/concepts. Analyzing a problem and presenting our solutionLet's jump straight into action by analyzing how a real problem could be solved with Nodezator and how our method solves the problems and meets the requirements presented earlier. The problem in question is the one originally presented by Mr. Adams: He has data representing a 3D box model and wants the model to be changed depending on the requested kind of lid (hinged, sawn, sliding,). To keep the problem as fundamental/representative as possible, let's reduce the possibilities to 02 options: either "hinged" or "no lid" (the box without a lid could be used as a prop, for instance). Once we solve the problem, we can then extend our solution to include more options as needed. The image below depicts a dummy node that returns data representing a box. At this point we still didn't decide whether the box will have lids or not, the node just returns general data describing a box.
The image below depicts 02 different dummy nodes that represent operations that result in the simplified possibilities: a box with a hinged lid or the same initial box with no lid added to it. As you can see, they take exactly one input and return a single value.
The Finally, the image below shows a viable solution to our branching problem.
At the center of our solution is the The usage of the ternary operator was first mentioned by Mr. Alexander in his many demonstrations/explorations of how branching could be achieved with FP (check this comment and the ones that follow it). However, the usage of if/else blocks had been explored before by Mr. Tom, though in a different way. I'll provide more comments on Mr. Alexander replies in a dedicated subsection after we are done walking through this solution and the next few subsections that extend and solves the problem in other important ways. For now, let's keep focusing on the solution presented above. Everything to the left of the If the requested lid equals 'hinged', then a is used, that is, the The output of This solved problem represents the most atomic/fundamental/basic problem in branching, choosing between an option or doing nothing. In text-based programming and when not using pure FP principles, this would appear in the script as a single if-block without an else-block accompanying it. Something like this: box_data = get_box(...)
if requested_lid == 'hinged':
box_data = add_hinged_lid(box_data)The exported Python code from our solution will instead represent a text-based script using FP. The code below is a simplified representation of how the graph from our solution would be exported as Python code: chosen_func = add_hinged_lid if requested_lid == 'hinged' else return_untouched
box_data = get_box()
box_data = perform_call(chosen_func, box_data)We are showing a representation, rather than actual code generated within Nodezator, because I still need to update the Python exporting feature to take the callable mode into account. Doing that should be pretty quick and straightforward though, because nodes in callable mode are much simpler to handle than regular nodes. This is so because regular nodes have many more elements to export, that is, all the arguments. Nodes in callable node, on the other hand, just represent a reference to a callable. Quick note on the efficiency of our solutionDespite the fact that within a node-based interface like Nodezator the
In other words, our solution is efficient. Extending the problem and our solutionNow let's extend this problem to work with an additional option: a box with a sawn lid. The image below depicts the solution:
The solution above shows how versatile the ternary operator (the In this solution, the first ternary operator (the one closer to the center, in the bottom half of the image) executes and, if the requested lid was a sawed-off one, it passes a reference to the The next ternary operator checks whether the requested lid was a hinged one and, if it is, it passes a reference to Just as shown in the image, this solution can be extended indefinitely, regardless of how much alternatives there are. We just need to chain as much ternary operators as needed. A dict-based simplificationOur solution can still be further simplified in some cases. Whenever the conditions to be evaluated are simple values that correspond to specific options, such different values and respective options can be stored in a dictionary, like demonstrated in the image below:
As seen in the image, the dict is populated with our options, using the respective lid type name. We then retrieve it's This solution is actually not innovative. It works similarly to Completing our solution: different signatures/argumentsThere is one case that our solution still needs to address. What if...
In the problems we just explored, the alternatives had the same signature and we only wanted to pass our However, what if each of the callables required different arguments or we just wanted to pass different arguments to them on our own volition? The answer is actually simple: we would only need to pass each reference of the callables through a This way we could feed the required/desired arguments to the callables even before the chosen one reaches the As an example, let's pretend our
The resulting Alternatively, if the order of the arguments was important, we could feed both the This ability to work with alternatives that have different signatures is actually crucial, because it allow us not only to use alternatives that require different sets of data, but also alternatives that require no data at all. Our solution so far has been addressing a problem with similar alternatives, that is, different kinds of lids (or no lid at all). All of them require some data to be passed to the chosen alternative. Even in the alternative where no lid is added, the identity function used receives the However, there are also cases when no data is passed on at all. For instance, if no lid is required, you might instead want to do something completely unrelated, like sending an email or creating an entirely new 3D model. In such cases, we'll be ignoring the existence of the box data entirely. Here's the resulting graph:
Edit (Y2023M07D18): there's a slight error in the image above. The In other words, the
Fortunately, as demonstrated until this point, keeping Nodezator FP/dataflow-compliant allows us to perform branching with none of the constraints mentioned by Mr. Tom. No signatures enforced, no special nodes required. Completing our solution: subgraphs as a needed tool to handle complexityThere's still a final crucial piece for the completion of our solution: subgraphs (group nodes). Subgraphs are actually an integral part of the solution. Notice that the nodes representing the different alternatives ( In fact, it often won't, which is precisely why we use a node-based interface to program, so we can combine different nodes to achieve a certain result. For instance, the alternative to create a 3D bag instead would probably be followed by other operations related to that alternative. As such, we'd probably use multiple nodes, not only the one to just create the bag. The options where a lid is added and we keep using the box data, would, in practice, maybe require many nodes to be combined and executed in order to add such lid to our box. That's where subgraphs (like group nodes in Blender3D) are useful. This is something that will still take a while to land on Nodezator, but is something indispensable in order to be able to fully tap into branching/looping in Nodezator. Once subgraphs/group nodes are implemented, whenever we need to represent a branch/alternative that requires the usage of multiple nodes, all we'll have to do is to create the nodes, group them together, select which inputs and outputs we want to expose, and use the resulting group node as demonstrated in our solutions: in callable mode, and with help of ternary operators or dictionaries (and I won't offer more comments on subgraphs here, but you can check this comment where I provided my recent thoughts on their design. Wrapping it all up: requirements met, problems solvedIn this quick subsection we'll revisit the requirements and problems we presented earlier, as I comment on how the previous demonstrations met/solved them. The first half of the requirements/problems are related to the efficacy of the branching/looping tools:
To my knowledge, considering all that was demonstrated, there's no doubt anymore that all current and planned features met the requirement and solved the problem related to the efficacy of branching and looping in Nodezator. All the presented features, currently implemented or not, contribute to solve all possible use-cases, that is, all that I could think of. The tools presented give us all the flexibility needed to handle multiple similar or completely different alternatives and with varying degrees of signatures and complexity. The second half of requirements/problems are related to the simplicity and ease of learning of performing branching/looping in Nodezator:
To be completely honest, the demonstrated solutions are only relatively easy to understand, not absolutely easy. I think the demonstrations represent the best available solution to the specific problem and an initial effort to meet the specific requirement. Of course, the demonstrations must be translated into more fundamental step-by-step guides to help beginners grasp the individual concepts and model their problems accordingly. Even so, and in order to be able to begin tackling the challenge, we must admit that for a complete Python beginner, or someone who never relied/got used to FP concepts/tools, it still takes time and effort to grasp the concepts. It just can't compare in terms of simplicity to how straightforward Nonetheless, each day I'm more and more convinced that this cognitive load is inherent to FP thinking and its tools. It is not something that can simply labeled as bad in itself. Rather than thinking only in terms of cost, we must also acknowledge that despite the initial learning curve, the outcome of relying in FP tools is still 100% worth. It gives our solutions modeled and solved in a node-based interface like Nodezator all the power and flexibility needed with a relatively simple design and structure. Because of that, rather than analyzing the usage of FP thinking/tools/modelling to solve our problems from the perspective of its costs, it is more relevant to analyse such usage from the perspective of the trade-offs, that is, the costs and the benefits. Beginners trade a bit of time and effort for unlimited power, flexibility, versatility and a fundamentally simple design and structure that fits a node-based interface perfectly, ensuring high extensibility capabilities and low maintenance cost for Nodezator. My rejected proposal, in comparison to FP tools/concepts, also had a learning curve/cognitive load anyway (and other problems), and with far fewer benefits. Maybe the acknowledgement and disposition to address the beginner's struggles (whether regarding FP thinking/tools/modelling or other subject) is what has been missing in the many failed attempts to popularize technologies like node-based programming and functional programming languages. If we want to do something different, it is only natural that some effort is required. It is just how the natural world works. As such, effort must not be demonized. Another piece of software that has a difficult learning curve is Git. And yet, people go through the difficult process of learning its many features in order to be awarded with the ability to manager their development work effectively and efficiently. Thus, we must accept the effort. That is, as long as the outcome is worth of the effort. And I believe this is the case with FP thinking/modelling for branching in Nodezator. Comment on Mr. Alexander latest replies/demonstrationsMr. Alexander, as I pointed before in the text, the core of my demonstrations was the ternary operator that you had already explored in your replies here on GitHub. When comparing what I demonstrated here with what you have been showing in your own replies my conclusion is that this also applies to many other concepts presented here. Let's go over these more explicitly. Your most recent reply, as the ones before it, already used the ternary operator. You also included a mechanism in your This other reply in particular, shows the usage of partial objects to customize the behaviour of other functions like All of them also explore how ternary operators can be chained. I believe that all of such similarities, despite the different ways in which our demonstrations were represented and different kind of problems tackled, just show how versatile the FP tools/concepts are. When combined with Nodezator's dataflow-compliant design, such tools/concepts allow the same problem to be modeled and solved in many different ways. The fact that my demonstrations explored problems in a broader scope is only due to a few situational factors like:
Given time, and once the few planned features in synergy with FP are implemented (like the callable mode, subgraphs, etc.), there's no doubt that your deeper knowledge and familiarity with FP tools will result in solutions/demonstrations even more sophisticated than the ones I shown here. I say this not as flattery, but as an objective assessment of your capabilities considering our discussions since you've joined the Indie Python community and the proficiency with which you've been exploring its features with your FP knowledge alone. It is precisely because I know a bit of FP and am aware of your knowledge that I must take extra caution when assessing how fit our demonstrations are for beginners. That is, my challenge is to provide this same proficiency to people who are not familiar with FP, so they can easily grasp and start using the most fundamental FP concepts/tools to model and solve their programming problems in Nodezator. Fortunately, I believe the fact that your FP/dataflow knowledge resulted in a proficient usage of Nodezator after such short time means Nodezator is being steered in the right direction, one that provides power and flexibility in problem solving within a node-based interface, using well-established paradigms like FP and dataflow (as long a such paradigms are adopted not for their name alone, but for the practical benefits they bring to our software development). With this I conclude my analysis regarding branching in Nodezator, specially in response to Mr. Alexander's replies/contributions to the topic. As always, Mr. Alexander and anyone reading this, if you think any point here needs extra clarification, feel free to ask me anything. If you have anything to add or feedback on what's been discussed, it is also welcome. Peace. |
Beta Was this translation helpful? Give feedback.
-
|
Dear @KennedyRichard, Sorry it has taken me some time to reply. You've given me a lot to think about. I'm very happy that you enjoyed the chapter about the Maya DAG! One note about the node execution algorithm in Nodezator (not that there's anything wrong with your current algorithm): it's possible to precompute the order in which the nodes should be executed using a Topological Sort. There are several Python graph libraries that have a topological sort function on a directed graph, including NetworkX and igraph. The benefits are that you can precompute the order the nodes should be executed before actually executing them and that it's possible to detect a dependency cycle between nodes and abort execution entirely, letting the user know where the cycle is, perhaps by highlighting the offending edges/nodes. Finally, a topological sort can tell you which nodes are on the "same level" and therefore can be executed in parallel. As is evident in your reply, you've already been thinking about all this! You can also use a graph library to quickly determine which downstream nodes to update and in what order using Dijkstra's shortest path algorithm. This can be useful when updating widgets "live" as soon as a user makes a change to an upstream node without having to press Update in the widget. I'm not certain the push/pull approach that Maya takes is necessary in Nodezator at this time. Right now, a simple push approach seems sufficient. But maybe that will change if/when you add an animation timeline. I know you only briefly discussed looping in your reply, but I have one idea that I think is worth mentioning: We should probably have at least two separate nodes to represent a for loop: The other note (more like a rant) I have about looping has to do with Python's insistence to return lazy iterators from functions like As a Python programmer, I have no choice but to accept the lazy nature of iterators, even though I would prefer eager evaluation by default. However, should we assume that most Nodezator users are also Python programmers and are aware that iterators are lazy? Or should we assume that most Nodezator users are not aware that iterators are lazy and therefore make all looping nodes in Nodezator eager by default--return One final point, since you mentioned iterators briefly in your reply as they relate to FP, is that there's nothing inherently FP about lazy evaluation. I think the association between laziness and FP comes from Haskell. However, the majority of programming languages, including FP languages like F#, OCaml, and the oldest FP language, Lisp (an acronym for list processing), use eager evaluation by default. Since you mention why Haskell is not more popular, I would put lazy evaluation and the problems it causes with (arguably) reasoning about the performance of programs and (actually) space leaks at the top of the list! I think Haskell is a great language in all other regards, but the choice of lazy evaluation by default was, in my opinion, a mistake. (Arguably, a "mistake" that led to novel ideas which would otherwise not be discovered, like the IO monad.) End of rant. Regarding your development process, I really appreciate your Occam's razor approach, and I think we are kindred spirits in that sense. Your minimalist design is one of the main things that attracted me to Nodezator, and it's clear from your reply that you'll continue to look for the simplest practical solution to any problem. I'm not surprised that it took you some time to reconcile the idea of a "node editor" and how FP fits into it. This is because Nodezator is only the third node editor I'm aware of that supports higher-order functions! (The other two are 3ds Max MCG and Viskell.) So we are almost in uncharted territory because most people who have used node editors haven't used higher-order nodes before. Nevertheless, it's obvious that a node editor is a natural way to visualize FP, even though it's not obvious at first what that should look like. As you can imagine, I'm happy that you're willing to explore this new and exciting territory in visual programming. Regarding learning FP concepts for beginners, I completely agree that there is a significant learning curve. We must gently introduce FP concepts to new Nodezator users and assume that even though they may have used a node editor before, they've never used higher-order nodes before. We should also stay far away from FP jargon as it is a huge turn-off for most people. To that end, I propose that we start with carefully choosing good names for higher-order nodes, especially those that don't already have a function in the standard Python library. For example, we should keep names like One more node, besides I absolutely love your running example with the hinged lid and how you were able to disentangle the ternary operator and the callable nodes. I didn't think of that in my if_then_else design above, but it makes perfect sense because the ternary operator node will always execute, so why shouldn't it be a separate node from the callables? I like your design better. I also love how you used the dictionary as a kind of match/case. I think this is quite innovative! We do, of course, need to add a real match/case or if/elif/else node to Nodezator that can check multiple conditions without having to chain ternary operator nodes. But it's nice to know that we can already use a dictionary in the simple case! Apparently, this type of "dictionary of operations" is a "common" way in FP to define what would in OOP languages be known as an Interface allowing for polymorphism. Imagine the classic OOP IAnimal interface defined like this: dog = {'speak': lambda: print('Woof!')}
cat = {'speak': lambda: print('Meow!')}
def speak(animal):
animal['speak']()
>>> speak(dog)
Woof!
>>> speak(cat)
Meow!The use of the I completely agree that in order to complete this design, we need subgraphs! But I understand that subgraphs are a topic all onto themselves and will take a long time to land in Nodezator. I think your intuition that the benefits of FP outweigh the cost of learning it is totally correct. That has certainly been my experience, even though I'm a very skeptical person. When I first started learning FP, I had a lot of questions, like "What about exceptions?", "What about performance?", and "What the heck are lenses?" Many of these questions were answered only after I started watching videos by Scott Wlaschin and subsequently learned the basics of F#. But FP is different from the imperative programming I was used to, and it took a long time to rewire my brain. That's not to say I'm an FP expert--I think my journey is just beginning. Thank you very much for sharing your thought process on this topic with me! You've shown me ways to use Nodezator that I haven't considered. As I've said, we are in mostly uncharted territory, trying to marry node-based editing with FP. It's clear that the two concepts are strongly related, but you have to wonder why, after 30+ years of visual programming, the two concepts are rarely seen together. We must therefore proceed carefully and thoughtfully so as not to alienate too many people. I'm excited to continue this journey with you! |
Beta Was this translation helpful? Give feedback.
-
|
Thank you for your well-thought-out reply, Mr. Alexander. It is always my pleasure to discuss things in depth with you and other members of the community because it ensures people are on the same page (regardless of whether people end up in agreement or not) and more ready to handle the many challenges of an open-source project. Learning new things or further exploring things one already knows is also very exciting to me. That is why I can never stress enough that the IndiePython community is as much about the journey as it is about the destination. I'll be glad to provide a complete reply as soon as I can find the time. I'll probably be able to find some throughout this week. Have a nice weekend. Peace |
Beta Was this translation helpful? Give feedback.
-
|
Mr. Alexander, Here's a more complete reply to your most recent comment. Thank you for your continuous feedback, discussions and support. Please, keep the ideas/questioning/suggestions/requests coming. It's is always a pleasure and very instructive and mind-broadening to discuss things with you. Challenging as well, in the best sense of the word.
You are absolutely right that there's much to consider when it comes to Nodezator's execution algorithm. As you could see in my last reply, I've been thinking about this for a long time. However, I didn't know where to start, so your suggestion is much appreciated. If I recall correctly it is not the first time in our discussions that you bring up the topological sort algorithm, so it is something to which I'll give priority when it is time to work on this. I won't look into the linked wikipedia article just yet, since I am currently working on another thing, but I think it is finally time that we take our first steps towards discussing these topics related to graph execution algorithms thoroughly. I'll start by listing "Expand and improve execution algorithms" in the list of planned/requested features and create a dedicated discussion so people can share their ideas/feedback over the time as well. Please, give me a week or two to gather my thoughts on the matter and I'll do just that. This way we can finally advance towards this goal. My overall design/idea for Nodezator regarding graph execution algorithms is that instead of listing a couple of them in the Graph menu as we do now ("Execute graph", "Execute with custom stdout"), we would actually an Execute Graph submenu that would list many available algorithms/strategies. The same way different problems require different algorithms, different kinds of graphs can benefit from different kinds of graph execution algorithms as well. I want users to have as much control and choice as possible when it comes to algorithms used in Nodezator. And this applies not only to graph execution but other areas as well. In summary, if there's different ways to do something and they have distinct advantages and disadvantages, I want the users to be able to decide. In practice, users will probably be switching between different algorithms, each one for a specific kind of problem they have to tackle. My current solution for the graph execution, for instance, is great for many use-cases but inappropriate for many others as well. For instance, I'd say for beginners or advanced users working with small to medium-sized graphs it is more than enough and maybe even more performant than advanced algorithms, because it doesn't spend any time doing pre-calculations and just jumps straight into visiting and executing the nodes. However, for more complex graphs and specially those with many time-consuming nodes, my solution is too naive and could lead to a lot of wasted time. For example, since it executes a node as soon as it is visited, it may execute many nodes during graph execution before finding out that one of the node is missing a required input, making it so the graph execution can't be finished, and thus wasting all the previous effort. Nonetheless, even for more complex graphs it might still be desirable if the user turns on a configuration that causes node outputs to be cached for later use (using dirty flags in the nodes, just like Maya does, a feature that I want to implement in Nodezator as well, but as something that the user can turn on or off). In such case, the nodes executed before finding the problematic node won't be wasted. There's a lot more I'd like to discuss about this, but I'll hold off on that even though this is such interesting topic for me. Let's discuss this more thoroughly when I create the dedicated discussion, so that we keep our thoughts organized in a single place. Of course, if you want to start discussing this right away, you can create the discussion yourself and start sharing your mind on the topic. If you do so I'll join later by adding all my ideas and feedback as well. You already know that, but no one needs my permission to start anything in the IndiePython community. There's only another thing that I'd like to say in advance and that applies not only to this topic, but all of development in Nodezator or any other IndiePython software: as much as possible, but only when viable, I'd like to avoid using external libraries in Nodezator. No, I'm not against it, and I'll certainly take a look in all libraries mentioned. It is just that, if we can implement the algorithms ourselves, than the initial cost of implementation is quickly surpassed by the value brought by avoiding external dependencies. It is not a strict rule, just a general principle. By doing so, we not only avoid future dependency problems, but we can customize our tools as much as we need for the users' needs. I'll give you a concrete example: Nodezator already has an error prevention mechanism that makes it impossible for the user to create dependency cycles in the graph (in management, such mechanisms are called Poka yoke mechanisms, and I rate them very highly as useful tools). If connecting two sockets would create a dependency cycle, the connection is aborted. Therefore, any solution that tries to calculate this is wasting time and processing effort to check for a problem that cannot exist in the first place. Of course, I'm saying this from the perspective of someone who never studied graph algorithms or the mathematical concepts behind graphs. It is just only speculation for now. So, in practice, implementing the algorithms ourselves might be a task too arduous, too time-consuming or just beyond our current skill level and thus we might be better off just adopting a third-party library to do the job in the end. If it is the case, then there's no problem in doing so. Otherwise, though, even if it takes a bit of time, it is better to come up with our own solutions. It is also more fun that way, no? The IndiePython project is an open-source project, not pressured by time or budget, so we might as well take advantage of that fact and enjoy the process and the learning. Oh, one more thing: even if we end up implementing our own solution, I'll still take a look into the implementation from the third-party libraries you mentioned. In open-source development, and probably in any kind of content creation in general, some people often avoid looking into solutions from other people in order to claim that their own solution is totally original or at least that they made it by themselves. In my honest opinion, this is a harmful, arrogant and antiscientific way of thinking. It is only by fully knowing others' implementations that we can claim that our solution is original and only by doing so we can ensure our solution is as best as it could possibly be (since we'll be able to learn from others' solutions, their mistakes and successes). This is just how science works and it saddens me seeing people ignore the effort from others.
Great, this applies to what I said above: let's discuss this in in-depth in a dedicated discussion. For now, I'll just add that we already have mechanisms to:
The only reason I didn't implement such automatic behaviour yet is that I want it to be activated by a flag the user turns on or off at will. This is so because this behaviour is desirable in some situations, but not in others (for instance, when the triggered executions are too time consuming, so the user might prefer to trigger such executions manually, not upon each change). Even this is actually simple to implement, but I'll do so in another release, since the next one already has too many new features. Even so, I repeat what I said before: I want users to have as many control and choice as possible when it comes to algorithms used, so I'll take your suggestion into account as well and make them available, as long as they bring something useful and, preferably, by implementing stuff ourselves rather than depending on external libraries (when possible).
Yes, you are right, we don't need both push and pull. But, the one we use is actually a pull approach. Like the chapter on Maya's dependency graphs says, When you pull information from a node, it propagates that request through all the nodes that provide input connections to the node.. That is, in the new viewer nodes to be released soon, when a user clicks the Reload button to update a visualization, it just executes its upstream nodes and itself in order to update its visualization, so it is a pull action.
Great insights, truly. In the end, all hinges in the answer to this question that you asked: "...should we assume that most Nodezator users are also Python programmers and are aware that iterators are lazy? Or should we assume that most Nodezator users are not aware that iterators are lazy and therefore make all looping nodes in Nodezator eager by default--return list(iterator)?". My answer is that people working with Nodezator should know Python, or at least actively seek to improve their knowledge of Python. In the IndiePython project in general I like to provide as much handholding as healthily possible. In this particular case, though, I believe it would not be healthy. That is, even if we succeeded in making beginners avoid mistakes, we'd risk teaching them inappropriate workarounds around Python's intended design decisions. We must not avoid Python design decisions, but embrace them, since Nodezator is a generalist/multi-purpose Python node editor. Because of that, I believe it is only natural that people should understand Python in order to be able to use it properly. However, it is true that flexibility is needed in order to solve an actual problem of Python's lazy evaluation of So, as you pointed out, we can already force the evaluation of a map object by passing it to a list or another collection. However, what Python lacks, is a way to force this evaluation without having to catch the result. Python doesn't have an operation to do that. Instead, only a for-loop can do such. So, as you can see in the copy of my local repo, there's a commit from June 1st in which I introduced a new snippet node to deal with this problem. The commit is entitled "add: 02 new snippet nodes" (the first one is just the
This new node effectively solves the problem of executing functions for their side-effects that you astutely pointed out. Admittedly, beginners will still struggle with the concept of lazy evaluation of iterators, but this is a problem that originates from Python's design, not Nodezator which just aims to be as faithful as possible. Even so, I personally think that just like paradigms/tools like FP and git, the learning curve is well worth it, cause lazy evaluation allows many operations to be chained together efficiently (as explained by Mr. Beazley in the link I provided in my last long reply). It also allows developers to write generator-functions that worry only about processing individual items, without having to worry whether they will end up in a specific collection or passed further to yet another function. For the sake of transparency though, I must say I not only appreciate but use such lazy evaluation profusely in my own graphs. I even have a node pack that I mean to release in the future which uses a lot of chained generator-functions to produce position data for game animations. Despite that, though, I still think lazy evaluation has objective merits that justify its existence in Python, otherwise I would resist the temptation to defend it. And again, this is all due to how Python does things, so we must try not to fight against it, despite any reservations we might have. However, even Python 2 had a I could say more about this, but really, this in itself could be its own discussion as well. Of course, if you wish to discuss this further, feel free to do so, and I'll provide further reasoning. Regardless of all that, with
I think you misunderstood the point about me associating FP with lazy evaluation, but I don't blame your for that, since I do think my long reply about branching could use a bit of revising and improving. I even tried to make this distinction clear in the text by listing them as two different things, like in this excerpt:
Even so, I repeat, the text could be revised further to make this distinction clearer, so I don't blame you. I actually didn't even have to study FP deeply to realize that lazy evaluation is not related to FP. Just by knowing that, from the perspective of FP, functions must not have side-effects, the concept of iteration in Python is completely ruled out from any relationship with FP. That is, when we chain generator-functions to one another in Python, the items actually travel the chain one by one (and even that is not guaranteed in some cases). In other words, a call at the end of chain might raise an exception caused by iteration that was happening in the beginning of the chain, which is crazy. This actually offends even the dataflow and component-based paradigms as well, I believe. Even so, I don't disapprove Python's lazy evaluation because it doesn't prevent the user from evaluating the map or whichever iterator they are using right away if they decided to do so. It just requires the additional step of triggering such evaluation, either by passing the iterator to a collection or to a for-loop. Even in my node pack that I mentioned, I could easily change the generator-functions back to regular functions without affecting existing graphs, since it is all done in the body of the function. But again, this extra step may be very annoying to people used to eager evaluation. I didn't use Python 2 for long, so this expectation of
Thank you for the appreciation. The gif pretty much says it all! I wish I could say I'm a big Star Wars fan, but despite liking the franchise, the complex political plots are too distracting and energy-draining for me to follow, so I stopped following the franchise after watching the movies. I'm probably missing a lot of good stuff though. I do like the political plots though, as contradictory as it may sounds. I guess when I'm watching something my mind enters an entertainment-only mode that refuses to spend energy thinking too hard about things. I'm glad I have people like you aboard, because simplicity is not easy to achieve and requires a lot of discussion, effort and time. If it were not for your intervention, for instance, Nodezator's design would become dangerously much complex with my original now-rejected proposal to branching. It is no exaggeration to say that the resulting increase in complexity could have easily threatened the very survival of the project. And yet, as I explained in my long reply, I had my reasons to pursue my first proposal at the beginning. Precisely because of how easy it is to overlook such hidden costs is that cooperation, properly held discussions and open-mindedness are so valuable. But that's the point in open-source development. At least in healthy open-source communities, where the maintainer(s) are open-minded. So again, thank you for your inestimable cooperation.
I'm happy and excited for this new direction too! I can't even take credit for it, because despite being a notion that took time to form in my mind, it is a super common/natural concept in FP. Nodezator was created because I always thought of nodes as function calls and yet existing frameworks required me to create classes to define even the simplest of nodes. Add to that the concept that functions are first-class citizens in Python and the notion of a node representing the function itself rather than a call to it should come almost naturally. I just made the link between the concepts. However, had I researched a bit more and came to know about 3ds Max MCG and Viskell earlier I might have brought the feature to Nodezator much sooner. This is why it is important to know other solutions, like I said before. It is the scientific way to do things. Unfortunately, in my many attempts at learning about other apps, I often find the knowledge to not be published at all (for instance, when the source is closed or when the creators didn't bother with discussing the design/ideas behind their apps) or hidden behind paywalls one way or another. Of course Nodezator has much to improve in making the knowledge about its design available as well. I'm already writing about its design in detail to publish soon. But, at least, since the project was first announced, I'm willing to explain and discuss everything in detail with anyone that asks. It is just that I've been prioritizing development, but I'll slowly publish more and more about its design and the design of its features.
Oh, that's great! I prefer names that describe the meaning/purpose of things, hence names like
I'm sold on the concept already and have no doubts about its usefulness. I even use a higher order function in the nodezator codebase that does something a bit similar: when I pass a function to it, it returns a function that will ignore any arguments given to it, a single one, multiple ones, keyword-arguments, etc. I call it get_oblivious_callable(), since it is what it does. That is, it receives a callable and returns a new one which does and returns the same thing but is oblivious to any received data. When researching a bit more about your suggestion I also found this funny answer in SO that helped me understand its usage as well. About the naming, I think names like Usefulness and naming aside, I still have just one concern: as we progress with our usage of Nodezator, we'll naturally stumble upon many other useful functions that would be nice to have in Nodezator by default. And I'm not only referring to functions/operations from FP. As such, we must take care not to overpopulate Nodezator with too many default nodes, especially when users can define them as custom nodes or as the lambda nodes that will be added to Nodezator as well. Your suggestion in particular looks like a great candidate for a lambda node that the user could add in a few seconds by instantiating a lambda node and using Even so, I have no objective reason to deny your suggestion (nor the desire), because the same could be said of the Making a function available by default in Nodezator is simply a convenience to users, so they don't need to define it by themselves. However, there's a bit more to it as well: we might also say that by making a function available by default, we are also vouching for the popularity and usefulness of that function. Therefore, let me ask you: are you positive that making this function available by default rather than letting the users define it by themselves as needed will really help users in general? This is an honest question, not a suggestive one meant to discourage you. As you know I'm not that versed in FP, so I cannot give an accurate answer to this question myself. Also consider that Nodezator users won't always seek FP tools to solve their problems. I actually think that a lot of them, probably most of them, will just use Nodezator as a node-based interface for working with Python. Of course, this usage is perfectly fine. However, I do believe the ones who choose to rely in FP tools will likely make the most out of Nodezator, not due to Nodezator or Python, but because of the inherent usefulness and versatility of FP tools/concepts. If after considering all of this you can still honestly say yes to my question, then I'll add it as a default node without problems. There's also an alternative that I find even more desirable, something about which I'm still pondering: perhaps, the submenu that contains the snippet nodes ("Useful encapsulations") could have yet another submenu entitled "extra FP tools", where we could add your suggestion. This way, additional FP nodes would have a dedicated space for them and overpopulating the "Useful encapsulations" submenu wouldn't be a probably anymore. We would also be able to add more additional FP nodes there, not only your suggestion. It would also be a convenient way for FP-reliant users to have additional extra tools easily available to them. Regardless of all of this, I still think we must be selective about which nodes we add, even if we were to add this special submenu I just mentioned, lest we end up wasting space with a lot of underused nodes. Please, let me know what you think about all of this. This is all very interesting to me as well.
I'm glad we are on the same page on that. It is nothing you could not come up with by yourself though, it is just that I had the extra time to work on it since the project is my main activity.
Great, yet another thing on which we are on the same page! Not that disagreeing is a bad thing, it is an arduous but important path towards understanding things better. But the opposite, when our thoughts are aligned, feels great as well!
Oh, I didn't know that. Whenever I imagine myself working without OOP I usually don't fear the possibility because I can clearly see how dictionaries and other collections could be used to store varying behaviours and data like that. Once you start seeing everything as data, even functions as data that describe behaviours, you start to see that there's nothing inherently special about OOP/classes. They are great, don't get me wrong. What I mean is that their value consists of abstracting away operations that could still be done without them, but that are made easier/more convenient by using them. Not understanding this may lead (and often does, in my opinion) to overuse/abuse of classes in cases where simple functions would be more appropriate (one of the reasons why Nodezator defines nodes from callables rather than classes).
Yes. It is not easy, but a task from which we should not shy away lest we end up like other fine technologies that died or never became popular because they didn't spend enough time and effort to accommodate new users.
Ditto. Though, conversely, I don't think subgraphs are inherently complex. It is just that, given the inordinate amount of things in Nodezator to accommodate into the feature, it will take time to plan and implement everything. However, it is more a question of time than a question of effort. A bit of refactoring will also be needed in order to avoid increasing the complexity of the app needlessly, but this is a thing that can be said about most software.
My favorite application of this notion is git. It has a very unorthodox data model that makes it a bit difficult to grasp at first and yet, once you spend time and effort to learn it you find out that the benefits massively outweigh the initial energy and time spent. FP is naturally another great example.
That looks like a great journey. Thank you for this link as well, I didn't know this resource. I'll sure consider taking a look when I have the time to deepen my knowledge of FP, something that I want to do a lot, since I know it will benefit my development knowledge and practice considerably. Learning a new paradigm is not easy and can even be a bit intimidating when it is so different from what we are used too, even more when it lacks a lot of elements we are used to. I believe that when it comes to scientific stuff, skepticism is a great thing. Without it we risk being led into all kinds of traps. Instead, we must question the principles behind the tools we use and must not blindly accept the reasoning behind the designs/concepts, but only after examining the evidence. I think your skepticism made and makes your FP journey all the more valuable. It can be a double-edged sword though, if it prevents you from looking at things with an open mind.
I too am beyond excited to continue this journey with you and thank you for being part of it. As I pointed out before, Nodezator would not be the same app without your talent and dedication. I have a lot of fun discussing all node-based, dataflow and FP topics with you. Even when we disagree it is a source of questioning and improvement. As I keep saying, the IndiePython project is as much about the journey as it is about the destination, so this is the kind of thing that we must encourage. I think the potential of Nodezator is massive. No, I don't have delusions of grandeur or expectations of great fortune. I just think that, by doing our work diligently and step by step, our little work can result in a simple yet powerful tool used to help a lot of people and become the base of even more impressive creations. That is, through this marriage between node-based editing and FP. |
Beta Was this translation helpful? Give feedback.
-
|
Hi @KennedyRichard, Thank you for your thoughtful reply. I'll be happy to discuss graph algorithms in another thread if you start one. If you don't want to add another dependency to Nodezator, then I recommend you take a look at NetworkX which is written in pure Python. If you find a useful algorithm there, you can selectively copy it to Nodezator, giving credit to the source, of course. In the end, I have to agree with you that "We must not avoid Python design decisions, but embrace them, since Nodezator is a generalist/multi-purpose Python node editor." I also agree with your insight that a chain of iterators "actually offends even the dataflow and component-based paradigms" and that there's not much we can do about it if we are to embrace iterators. Another name for "exhaust" that is also used in the Python documentation is "consume". But either name is fine. This function is available in the I do think we need the >>> from operator import mul
>>> W = lambda f: lambda x: f(x, x)
>>> square = W(mul)
>>> square(5)
25In Nodezator, the user can square a number by feeding it into both the a and b inputs of the multiply node. So, I agree that we shouldn't clutter Nodezator with too many frivolous built-in nodes. But I think the That's all for now. I look forward to trying the next version of Nodezator! (I know I can already try it from your local repo, but I prefer to update it through pip.) |
Beta Was this translation helpful? Give feedback.
-
|
Thank you for your reply, Mr. Alexander.
Great! I'll start the new discussion as soon as I release the new Nodezator version. There's a lot I want to discuss, share and learn about this. I'll schedule a thorough inspection of the NetworkX library as well, so I can discuss it further in the new discussion to be created.
I totally agree! Though, in hindsight, perhaps it is more accurate to say that we should not stand in the way of Python design decisions rather than openly embrace them. For instance, I certainly like how iteration is handled in Python and think chaining iterators is very useful in a lot of situations, but I also think that the FP paradigm (also dataflow and component-based) has its reasons for keeping functions free of side-effects and as "pure" as possible, so I certainly don't want to solely rely on Python's approach for all problems regarding iteration. As I said before, when it comes to a node-based interface, relying on FP principles will likely make the produced solutions more powerful and versatile. It is just that for iteration, Python's approach is rather suitable as well.
I personally prefer "consume" over "exhaust". However, though a bit more advanced, generators in Python also have a capability that can be described as an ability to consume values, which is the # yield can receive a value instead
def receiver():
while True:
item = yield
print('Got', item)
# It defines a generator that you send things to
recv = receiver()
next(recv) # Advance to first yield
recv.send('Hello')
recv.send('World')Check this excerpt from yet another of his slides: # Consume a generator and send to consumers
def broadcast(source, consumers):
for item in source:
for c in consumers:
c.send(item)These are also points from the same set of slides as the previous code block:
See how the term ends up a bit overused? "consume a generator and send to consumers", etc. This is why I want to avoid mixing the two terms and use "exhaust" instead, which I believe to be less prone to ambiguity. I'm not criticizing the author's usage of the term either. Even he admits this kind of Python tools are very mind-bending. He also seems personally more inclined to use generators for iteration and not so much as co-routines. This ability to send data to a co-routine after its call is even more egregious for FP/dataflow/component-based paradigms than chaining iterators. A sacrilege even. Not to mention other methods like
I've heard of the library before. Now that you are recommending it I have even more reason for checking it out. Thank you for yet another great recommendation.
Thank you for asking my question. This answer, plus some quick personal research I did, provided all the confirmation I needed. I also personally think it will be useful in many cases. I approve of its addition as an app-defined node, that is, provided by default in Nodezator. I intend to add it in this soon-to-be-released Nodezator version or the one after it at most. Also thank you for the demonstration of the W combinator. Even though you deemed it unnecessary as a default node I still appreciate the demonstration and explanation, since I find all of this very interesting. For the same reason, thank you for the link to the gist presenting other combinators. As always, thank you for the feedback, and keep the questions/suggestions/requests coming. I look forward to our next discussions regarding graph execution algorithms in the discussion to be created. Peace |
Beta Was this translation helpful? Give feedback.
-
|
Hello, everyone, Here's another update on my recent work in Nodezator. The new commits can be accessed, as always, in this copy of my local repo. The new commits consist of fixes and improvements on the new collapsed mode and callable mode features. The highlight is the fact that operator nodes now have a callable mode as well, as demonstrated in the video below: operator_node_in_callable_mode.mp4I also changed the size of the operator nodes, specially the big characters in their center, as can be seen in the image below, with a
The characters are still big, so they can be easily spotted in the graph, but they are not needlessly big anymore. I'm thinking of changing the red in the big characters a bit, just to add more contrast, specially to make the difference between the The only thing missing for this batch of new changes to be released is to update exporting features (.png, .svg, .html and .py) and add the ternary operator ( Have a nice weekend! Peace. |
Beta Was this translation helpful? Give feedback.
-
|
Hello, everyone, Yet another update on my recent work in Nodezator: the ability to view the Python code representing the graph without needing to export it to a file, as can be seen in the video below: view_as_python_demo.mp4Currently, the user has to save the exported code to a file before being able to see it. And even so, the user has to open it in another app. In the next release, the user will be able to view the code instantly, by clicking in the "View as Python" command in the menubar or simply pressing Shift+Ctrl+P I also rebased the new commits from the implefeats branch on top of the commits from the apptesting branch (bringing the commits to the implefeats branch). Since the commits from apptesting are now in the implefeats branch, the apptesting branch was deleted. All code needed for the new modes (expanded mode, collapsed mode and callable mode) to work is already implemented! Now I just need to refactor everything, implement the ternary operator (which will be super quick, since it requires just a few lines to be added) and also make a few small additions. Then I’ll finally be able to release a new version with the few recent features. As always, the commits are available in the online copy of my local repo. Peace. |
Beta Was this translation helpful? Give feedback.
-
|
Neat - is this the implementation of idea @ #33 (comment) ? |
Beta Was this translation helpful? Give feedback.
-
|
Well spotted! That's precisely it! Though I forgot I posted that comment. It is just that this stuff is always in the back of my mind, so sometimes I manage to recall it by myself. I'm glad I always post/write that stuff down, though, since it helps to keep the memory fresh and sometimes is the only way for me to remember something. |
Beta Was this translation helpful? Give feedback.
-
|
Hello, everyone! Here's a last minute addition to the list of features that will be published soon. It is the persistent viewer nodes feature, that is, nodes which can display visualizations inside the graph/canvas (for instance, from their outputs). Though, since they supersede the current viewer nodes, they will simply be called viewer nodes as well. viewer_nodes_demo.mp4I'll explain this new feature in new content that I'll write for the manual. What's great about this feature is that it requires very little additions to set up a visualization for your own nodes. And it is backward compatible. Actually, all recent features to be released soon are backward compatible, so you'll be able to open your current files without any problems and the old viewer nodes will still work as usual or can be upgraded at your own discretion. This feature only exists because of request and feedback from Mr. @Tokarzewski and Mr. @Tom-99, as can be seen in the discussion related to this comment on discord: https://discord.com/channels/980193047000272967/980193047461630113/1013720084516179989 This new addition will cause the release of the new features to take a bit longer. I can't give a precise date, but it should still be very soon (maybe 02 weeks from now? I honestly don't know exactly), since there's little left to implement and a bit of fine-tuning and refactoring to do. As always, you can check the fresh commits in the online copy of my local repo: https://indiepython.com/local-commits Thank you again for your patience and encouragement. Have a nice week! Peace |
Beta Was this translation helpful? Give feedback.
-
|
Hello, everyone, Hopefully this will be my last update before the next Nodezator version is released! All planned features for this version are implemented and many other last minute additions as well:
The only thing left to do now is updating the manual to include information on how to use the new viewer nodes. I'll also make a short video explaining the main changes in more detail when I release the new version. One of the most recent changes was fixing the doctests and converting them into markdown files. Now it is more pleasant to browse through them, since markdown can display both stylized hypertext (headings, paragraphs, blockquotes, links, images, lists, tables and even HTML tags) and code that can be executed as doctests. Even the font used for rendering in markdown seems to be more easy on the eye. The code blocks are also syntax highlighted, which helps reading the code as well. See the comparison below between a doctest written in Python and the same one edited as a markdown file:
As always, you can see my latest commits in the copy of my local repo: https://indiepython.com/local-commits Have a nice weekend Peace |
Beta Was this translation helpful? Give feedback.
-
|
Hello, every just a very tiny update on the new release: I've been working on it continuously over the past weeks. Even though all the changes are ready, I've been working on updating the manual to take the changes into account. Despite looking like a simple task due to not involving coding directly, it actually takes a lot of time because I must make sure the material is as simple as possible and easy to understand and it also requires me to write/rewrite/refine example code/demonstrations which takes a lot of time too. Additionally, writing the new manual content does occasionally help me identify a bug or two in the code, which is a good thing but also makes the final changes take a bit longer. I also want to update the manual all at once in a single commit, since this is what makes more sense to me, which is why I haven't updated the copy of my local repo in a while, but it should be done soon, once I'm finished. All in all, things are progressing fine, it is just that they naturally take time, so I ask you that you keep waiting a bit more. I can't give you a precise ETA for this new release, because as I explained, it is hard to predict all the subtasks that pop up along the way, but personally, I can't imagine the changes taking more than a week or so. Thank you all for your patience so far. Have a nice week. Peace |
Beta Was this translation helpful? Give feedback.
-
|
Hello, everyone, I'm here with yet another update. As I said before, I cannot give you a precise ETA for the new release, because the subtasks are just too numerous. But again, I'm very very close to finishing everything needed, so the new version might drop in just a few days. Also, as always, for the sake of transparency, you can always check the online copy of my local repo that I update from time to time (I updated it today too), in case you wanna personally inspect my work: https://indiepython.com/local-commits Gladly I'm not experiencing any kind of burnout or fatigue. I have been working daily in an efficient and healthy pace. It is just that the changes do take this long. Thank you for your patience. Peace |
Beta Was this translation helpful? Give feedback.
-
|
Hello, everyone, As always, it is my duty to update you on my progress. As I cannot stress enough, I cannot give a precise ETA for the release, because of the many subtasks involved with the final preparations. The recent work however, isn't related to such subtasks at all. It is actually a bit of new stuff being implemented. In summary, since the new Nodezator version to be released allows visualizations to be displayed in the graph itself beside viewer nodes, users will need extra nodes to work with pygame surfaces, which are used for the visualizations. Of course such nodes could be created by the users themselves, but to avoid a lot of code repetition across different node packs created by users, I'm including the most basic operations in the form of new default nodes that can be used to create/manipulate surfaces. Here's a quick demonstration: surface_viewing_editing_demo.mp4Again, you can check the online copy of my local repository (which I updated today) if you want to inspect my work in this link: https://indiepython.com/local-commits I'm as excited as you for the new version, but unfortunately I can't rush things. All changes must be properly tested and retested. Even so, thank you all for your patience and support, specially Mr. @OlegAlexander and Mr. @BMaxV who are awaiting the creation of a new discussion on Nodezator execution algorithms and the review of a pull request, respectively. Peace |
Beta Was this translation helpful? Give feedback.
-
|
Hello, everyone, Nodezator 1.4 is finally out! There's already a couple of patches with last minute fixes (latest version is 1.4.2)! You can download it right now with pip:
And learn more about it here: https://github.com/IndiePython/nodezator/releases/tag/v1.4.0 The online manual is updated as well: https://manual.nodezator.com (it is also available within the app) Thank you for your patience and support! Please, (only if you can and are willing to) consider becoming a patron or making a one-time donation (any amount helps a lot): https://patreon.com/KennedyRichard I'm also very grateful towards those who cannot help fund the project, but have been helping the project a lot in other ways like engaging in discussions, submitting PRs, suggesting ideas/improvements/features. Thank you everyone! Peace |
Beta Was this translation helpful? Give feedback.
-
|
Hello, everyone! Having finally released the new Nodezator version, I can finally resume work on other pending tasks! Thank you for your patience! My priority now is creating and publishing a new discussion at the request of Mr. @OlegAlexander where we and everyone interested can discuss improvements and availability of graph execution algorithms in Nodezator. Once I start the discussion, I will switch my attention to Mr. BMaxV PR #53. At the same time, from now on, I'll dedicate small amounts of time throughout the week here and there to tend to other administrative tasks of the Indie Python project, like promoting the project online and supporting users by answering pending discussions in the Nodezator repo. Since this specific discussion where I'm posting this message already has more than 50 comments, a lot of which are related to stuff already implement or discussions already finished, I'll be closing this discussion in a few days and creating a new one listing the next work scheduled for the Nodezator app. I'll also rename its title to reflect the period of time over which the related work was done (for instance, rather than Current work... it will start with Work from late 2022 to first half of 2023 etc. I'll also update the top post summarizing the contents of the discussion. To help users and the community in general to visualize the planned work for Nodezator I'll also publish an image of a roadmap for Nodezator in the discussion containing the list of planned/requested features (the image will probably be hosted somewhere else due to its size and referenced there, I'm still thinking about this). This will be just a visual summary. Nothing will be set in stone and any point can be contested by anyone at any point in time. I think that's all for now. Peace. |
Beta Was this translation helpful? Give feedback.
-
|
FYI, if you want to do 3D modeling, there's a new version of OpenSCAD in development/testing which uses Python: http://www.guenther-sohler.net/openscad/ which if there was just an easy way to launch OpenSCAD and point it at the Python file from Nodezator, would be pretty easy to support. |
Beta Was this translation helpful? Give feedback.
-
|
Thank you for the suggestion, Mr. Adams. Always bringing something new to the table. This looks very promising. I just read the entire page you linked and also the entire conversation in the related pull request, including your interactions. Now that Nodezator can display visuals on the graph I plan to keep making a lot of small demos to explore the possibilities, not only on 3D processing and rendering but a lot of different topics, since Nodezator applications are limited only by the Python ecosystem, which is enormous and ever-expanding. Just by connecting existing packages one can produce almost unlimited results. As such I do plan to start exploring with basic 3D stuff. Nodezator won't be ready to fully support 3D visualizations until I implement the OpenGL backend but there's a lot that can be achieved already like basic modelling and rendering of stills/previews with the right tools. My job for now will be to determine what are those "right tools". After all, Nodezator power comes from its simplicity. It is more of a meta-app than an app per se, that is, its job is to move data around operations. As long as a problem can be described as a data pipeline Nodezator can be used to solve it or at least assist in the solution. This is very exciting and fun work. I have some small demonstration work to do with scalar vector graphics (SVGs) and also math formulas rendering with the ASCIIMath format before delving into this, however. I'm estimating I'll be able to start having some 3D fun in a couple of weeks though. |
Beta Was this translation helpful? Give feedback.
-
Hello, everyone,
Every time I'm about to dive into working on a new feature/couple of features I create a discussion like this in order to...
Changes planned:
June 2023 update
A lot happened since the start of this specific batch of development work and at this point in time it has already more than 40 comments, so here's a summary:
Final update (related to release of version 1.4 that happened at the very end of August 2023)
As discussed in the last update above, the original changes planned in this discussion were either rejected or postponed. Instead, many other features were implemented, all listed in this released post: https://github.com/IndiePython/nodezator/releases/tag/v1.4.0
Since such work is already implemented and other discussions in this post were either finished as well or will be continued in other discussions to be created, I'm finally closing this discussion (after renaming it to reflect to period of time when the work was done) and will create a new one in a few moments with plans for the next Nodezator version.
Original changes planned (text below represents original content of this post)
Important
This changes were either rejected or postponed; other changes were implemented instead and you can read about them in the rest of this discussion or summarized here: https://github.com/IndiePython/nodezator/releases/tag/v1.4.0; the following paragraphs and subsections are only kept here for completeness;
From this late September and into October the next features I decided to add to Nodezator are branching (if/elif/else) and looping (for-loops).
I originally intended to implement looping at a later date, but I decided to implement them together because both features affect the execution flow of the graph and they require the same kind of research, that is:
Due to the amount of research and discussion needed to ensure the features were properly planned and designed and due to the amount of work done in parallel in Nodezator's parent project (the Indie Python project), the planning phase took more time than anticipated but it finally ended. The implementation should finally begin this late December 2022 but will be postponed again in favor of the release of the input recording/playing features and automated GUI tests, as explained in this comment from this discussion.
Brief thoughts on branching/looping
I already have a complete design in my head for branching, since I've been giving a lot of thought to it. For looping, though, it is not that I couldn't think of how implementing it on Nodezator, it is just that I never used this feature before in any other software, so I don't even have a mental image of the feature. Because of that, I'll need to research the usage of the feature in other apps.
Also, even though I have a complete design of branching in my mind, It doesn't exempt me from researching into the usage of this feature in other apps as well, because it would be irresponsible not to do, preventing me from learning from them, knowing major pitfalls, different possibilities, etc. By research I don't mean anything deep nor time-consuming. Reading a bit and watching demonstrations of the features in action and things like that should be enough.
Next steps
Here are the next steps planned:
Regarding branching and looping, I expect most tasks to proceed smoothly. What is usually more time-consuming is not the features per se, but all the additional changes needed to support the features, like the creation of new UI elements and the code needed to manage them, then fine-tuning and testing everything manually.
During all this I welcome any feedback or suggestions from you all about the features, including pointing out resources I should look into. In other words, is there any particular implementation of such features in another app that you are fond of? Please, let me know, so we can bring into Nodezator the best designs in existence.
Beta Was this translation helpful? Give feedback.
All reactions