Version 2.0 of the Lua Develoer FAQ #1899
Conversation
docs/en/lua-developer-faq.md
Outdated
| This FAQ does not aim to help you to learn to program or even how to program in Lua. There are plenty of resources on the Internet for this, some of which are listed in [Where to start](#where-to-start). What this FAQ does is to answer some of the common questions that a competent Lua developer would ask in learning how to develop Lua applications for the ESP8266 based boards running the [NodeMcu](http://NodeMCU.com/index_en.html) firmware. | ||
| This FAQ does not aim to help you to learn to program or even how to program in Lua. There are plenty of resources on the Internet for this, some of which are listed in [Where to start](#where-to-start). What this FAQ does is to answer some of the common questions that a competent Lua developer would ask in learning how to develop Lua applications for the ESP8266 based boards running the [NodeMcu firmware](https://github.com/nodemcu/nodemcu-firmware). This includes the NodeMCU Devkits. However, the scope of the firmware is far wider than this as it can be used on any ESP8266 module. | ||
|
|
||
| ## What has changed since the first version? |
There was a problem hiding this comment.
As I said in the earlier discussion I still think that this whole chapter doesn't belong here (I suggested README, https://www.facebook.com/NodeMCU or even https://en.wikipedia.org/wiki/NodeMCU instead).
There was a problem hiding this comment.
Marcel, I agree that I could replace this with a reference, but IMO the source needs to be wthin our documentation set. Perhaps you could consider reviewing our history and adding this?
- I don't know who owns the NodeMCU project Facebook page, but Facebook is closed resource and many of us may not even have a Facebook about (me for one) and really object to getting bonbarded to "sign up to Facebook messages".
- I am a regular Wkipedia contributor, and this is not how WP works. Whilst articles can include a history section, as a contributor you are still not allowed to add primary research to a WP page. So for example, if a history page is included on this site and the content is both non-falsifiable and therefore non-controversial, then a WP contributor might be able to update the History by quoting this reference as a source.
So my proposal is that you draft a more general "changes in NodeMCU section for the documentation and I reference this here. We've all done so much over the last two years, that I feel that we should be open and explicit about what we've achieve.
There was a problem hiding this comment.
You have a point here - more than one actually.
I don't know who owns the NodeMCU project Facebook page
"We" do. I don't know who else has write access to it but at least the originators and I do.
but Facebook is closed resource and many of us may not even have a Facebook account
True, I keep forgetting that (what a shame it is closed btw). I don't have a personal account either and I personally couldn't care less about FB but I just accept that it's a useful medium for a project like ours.
You're right about WP as well.
I'll be happy to create a "Changes in NodeMCU" section and transfer your chapter to it. It'll be on the README, though. The space between the "Releases" and "Support" sections looks appropriate to me.
docs/en/lua-developer-faq.md
Outdated
| * The NodeMCU development team recommends that you consider using a tailored firmware build, which only includes the modules that you plan to use before developing any Lua application. Once you have the ability to make and flash custom builds, the you also have the option of moving time sensitive or logic intensive code into your own custom module. Doing this can save a large amount of RAM as C code can be run directly from Flash memory. If you want an easy-to-use intermediate option then why note try the [cloud based NodeMCU custom build service](https://nodemcu-build.com)?. | ||
| ### How to reduce the size of the firmware? | ||
|
|
||
| * We recommend that you consider using a tailored firmware build; one which only includes the modules that you plan to use in developing any Lua application. Once you have the ability to make and flash custom builds, the you also have the option of moving time sensitive or logic intensive code into your own custom module. Doing this can save a large amount of RAM as C code can be run directly from Flash memory. If you want an easy-to-use intermediate option then why note try the [cloud based NodeMCU custom build service](https://nodemcu-build.com). |
There was a problem hiding this comment.
We recommend that you consider...
Users don't really have a choice now 😉 On the first mention of the term build you could link to ./build.md.
There was a problem hiding this comment.
Yup, since the FAQ is written in a generally personal voice, the shift to first person works; however, the grammar still isn't quite right here, so I'll update accordingly and reference the build section.
One of the thing that has changes since V1 is that it's primary host was esp8266.com and decent documentation didn't exist. I have tried to integrate intra-documentation references and avoid too much duplication when practical.
docs/en/lua-developer-faq.md
Outdated
| * Unlike Arduino or ESP8266 development, where each application change requires the flashing of a new copy of the firmware, in the case of Lua the firmware is normally flashed once, and all application development is done by updating files on the SPIFFS file system. In this respect, Lua development on the ESP8266 is far more like developing applications on a more traditional PC. The firmware will only be reflashed if the developer wants to add or update one or more of the hardware-related libraries. | ||
| * Those developers who are used to dealing in MB or GB of RAM and file systems can easily run out of memory resources, but with care and using some of the techniques discussed below can go a long way to mitigate this. | ||
| * The ESP8266 runs the SDK over the native hardware, so there is no underlying operating system to capture errors and to provide graceful failure modes. Hence system or application errors can easily "PANIC" the system causing it to reboot. Error handling has been kept simple to save on the limited code space, and this exacerbates this tendency. Running out of a system resource such as RAM will invariably cause a messy failure and system reboot. | ||
| * Note that in the 3 years since the firmware was first developed, Espressif has developed and released a new RTOS alternative to the non-OS SDK, and and the latest version of the SDK API reference recommends using RTOS. Unfortunately, the richer RTOS has a significantly larger RAM footprint. Whilst our port to the ESP-32 (with its significantly larger RAM) uses the ESP-IDF which is based on RTOS, the ESP8266 RTOS versions don't have enough free RAM for a RTOS-based NodeMCU firmware build to have sufficient free RAM to write usable applications. |
There was a problem hiding this comment.
There was a problem hiding this comment.
There was a problem hiding this comment.
Thanks, I will add the links.
docs/en/lua-developer-faq.md
Outdated
|
|
||
| The NodeMCU Lua firmware is an ESP8266 application and must therefore be layered over the ESP8266 SDK. However, the hooks and features of Lua enable it to be seamlessly integrated without loosing any of the standard Lua language features. The firmware has replaced some standard Lua modules that don't align well with the SDK structure with ESP8266-specific versions. For example, the standard `io` and `os` libraries don't work, but have been largely replaced by the NodeMCU `node` and `file` libraries. The `debug` and `math` libraries have also been omitted to reduce the runtime footprint (`modulo` can be done via `%`, `power` via `^`). | ||
| The NodeMCU Lua firmware is an ESP8266 application and is therefore be layered over the ESP8266 SDK. However, the hooks and features of Lua enable it to be seamlessly integrated without losing any of the standard Lua language features. The firmware has replaced some standard Lua modules that don't align well with the SDK structure with ESP8266-specific versions. For example, the standard `io` and `os` libraries don't work, but have been largely replaced by the NodeMCU `node` and `file` libraries. The `debug` and `math` libraries have also been omitted to reduce the runtime footprint (`modulo` can be done via `%`, `power` via `^`). |
There was a problem hiding this comment.
is therefore be layered over the ESP8266 SDK
Did you mean to write " is therefore to be layered "?
There was a problem hiding this comment.
Sorry, I am a sloppy writer (my fingers don't fully obey my thinking 😃)and my stuff always needs proofing. What I am trying to say is the that in some sense the NodeMCU firmware is just another NodeMCU application so the phrase should read "and must therefore be layered over the ESP8266 SDK."
docs/en/lua-developer-faq.md
Outdated
| * Current versions of the ESP8266 run the SDK over the native hardware so there is no underlying operating system to capture errors and to provide graceful failure modes, so system or application errors can easily "PANIC" the system causing it to reboot. Error handling has been kept simple to save on the limited code space, and this exacerbates this tendency. Running out of a system resource such as RAM will invariably cause a messy failure and system reboot. | ||
| * There is currently no `debug` library support. So you have to use 1980s-style "binary-chop" to locate errors and use print statement diagnostics though the systems UART interface. (This omission was largely because of the Flash memory footprint of this library, but there is no reason in principle why we couldn't make this library available in the near future as an custom build option). | ||
| * The LTR implementation means that you can't easily extend standard libraries as you can in normal Lua, so for example an attempt to define `function table.pack()` will cause a runtime error because you can't write to the global `table`. (Yes, there are standard sand-boxing techniques to achieve the same effect by using metatable based inheritance, but if you try to use this type of approach within a real application, then you will find that you run out of RAM before you implement anything useful.) | ||
| * The ESP8266 uses a combination of on-chip RAM and off-chip Flash memory connected using a dedicated SPI interface. Code can be executed directly from Flash (the ESP hardware first faults the Flash code into an L1 cache in RAM, but this largely transparent in terms of programming ESP8266 applications). The Lua firmware largely runs out of Flash. Both the RAM and the Flash memory are *very* limited when compared to systems than most application programmers use. |
There was a problem hiding this comment.
to systems than most application programmers use
"that" instead of "than"?
docs/en/lua-developer-faq.md
Outdated
| * The LTR implementation means that you can't easily extend standard libraries as you can in normal Lua, so for example an attempt to define `function table.pack()` will cause a runtime error because you can't write to the global `table`. (Yes, there are standard sand-boxing techniques to achieve the same effect by using metatable based inheritance, but if you try to use this type of approach within a real application, then you will find that you run out of RAM before you implement anything useful.) | ||
| * The ESP8266 uses a combination of on-chip RAM and off-chip Flash memory connected using a dedicated SPI interface. Code can be executed directly from Flash (the ESP hardware first faults the Flash code into an L1 cache in RAM, but this largely transparent in terms of programming ESP8266 applications). The Lua firmware largely runs out of Flash. Both the RAM and the Flash memory are *very* limited when compared to systems than most application programmers use. | ||
| * Over the last two years, both the Espressif non-OS SDK developers and the NodeMCU team have made a range of improvements and optimisations to increase the amount of RAM available to to developers, from a typical 15Kb or so with Version 0.9 builds to some 45Kb with the current firmware Version 2.x builds. | ||
| * The early ESP8266 modules were typically configured with 512Kb Flash. Fitting a fully featured Lua build with a number of optional libraries and still enough usable Flash to hold a Lua application was a struggle. However the code-size of the SDK has grown significantly between the early versions and the current 2.0 version. Applications based on the current SDK can no longer fit in 512Kb Flash memory, and so all currently produced ESP modules now contain a minimum of 1Mb with 4 and 16Mb becoming more common. The current NodeMCU firmware will fit comfortably in a 1Mb Flash and still have ample remaining Flash memory to support Lua IoT applications. |
There was a problem hiding this comment.
You could mention here that we froze the 1.5.4.1-final branch to still support those 512Kb modules. Even though it was introduced less than a month ago the stats of my build service indicate that there actually is demand for it.
| * The runtime system runs in interactive-mode. In this mode it first executes any `init.lua` script. It then "listens" to the serial port for input Lua chunks, and executes them once syntactically complete. There is no `luac` or batch support, although automated embedded processing is normally achieved by setting up the necessary event triggers in the `init.lua` script. | ||
| * The various libraries (`net`, `tmr`, `wifi`, etc.) use the SDK callback mechanism to bind Lua processing to individual events (for example a timer alarm firing). Developers should make full use of these events to keep Lua execution sequences short. *If any individual task takes too long to execute then other queued tasks can time-out and bad things start to happen.* | ||
| * Non-Lua processing (e.g. network functions) will usually only take place once the current Lua chunk has completed execution. So any network calls should be viewed at an asynchronous request. A common coding mistake is to assume that they are synchronous, that is if two `socket:send()` are on consecutive lines in a Lua programme, then the first has completed by the time the second is executed. This is wrong. Each `socket:send()` request simply queues the send operation for dispatch. Neither will start to process until the Lua code has return to is calling C function. Stacking up such requests in a single Lua task function burns scarce RAM and can trigger a PANIC. This true for timer, network, and other callbacks. It is even the case for actions such as requesting a system restart, as can be seen by the following example: | ||
| * The runtime system runs in interactive-mode. In this mode it first executes any `init.lua` script. It then "listens" to the serial port for input Lua chunks, and executes them once syntactically complete. |
There was a problem hiding this comment.
Hyperlink-maniac speaking again 😉 How about linking to upload.md#initlua?
There was a problem hiding this comment.
Yes, but the target link needs improving IMO -- however, that is a separate issue.
docs/en/lua-developer-faq.md
Outdated
| * The runtime system runs in interactive-mode. In this mode it first executes any `init.lua` script. It then "listens" to the serial port for input Lua chunks, and executes them once syntactically complete. | ||
| * There is no batch support, although automated embedded processing is normally achieved by setting up the necessary event triggers in the `init.lua` script. | ||
| * The various libraries (`net`, `tmr`, `wifi`, etc.) use the SDK callback mechanism to bind Lua processing to individual events (for example a timer alarm firing). Developers should make full use of these events to keep Lua execution sequences short. | ||
| * Non-Lua processing (e.g. network functions) will usually only take place once the current Lua chunk has completed execution. So any network calls should be viewed at an asynchronous request. A common coding mistake is to assume that they are synchronous, that is if two `socket:send()` are on consecutive lines in a Lua programme, then the first has completed by the time the second is executed. This is wrong. A `socket:send()` request simply queues the send task for dispatch by the SDK. This task can't start to process until the Lua code has returned to is calling C function to allow this running task to exit. Stacking up such requests in a single Lua task function burns scarce RAM and can trigger a PANIC. This true for timer, network, and other callbacks. It is even the case for actions such as requesting a system restart, as can be seen by the following example: | ||
|
|
||
| ```lua | ||
| node.restart(); for i = 1, 20 do print("not quite yet -- ",i); end |
There was a problem hiding this comment.
To drive the point home it'd IMO be helpful to add a (stripped) sample output.
|
Marcel, thanks for your feedback. |
|
There's more to come, I'm only half-way through 😉 |
| * There are standard libraries to provide access to the various hardware options supported by the hardware: WiFi, GPIO, One-wire, I²C, SPI, ADC, PWM, UART, etc. | ||
| * The runtime system runs in interactive-mode. In this mode it first executes any `init.lua` script. It then "listens" to the serial port for input Lua chunks, and executes them once syntactically complete. There is no `luac` or batch support, although automated embedded processing is normally achieved by setting up the necessary event triggers in the `init.lua` script. | ||
| * The various libraries (`net`, `tmr`, `wifi`, etc.) use the SDK callback mechanism to bind Lua processing to individual events (for example a timer alarm firing). Developers should make full use of these events to keep Lua execution sequences short. *If any individual task takes too long to execute then other queued tasks can time-out and bad things start to happen.* | ||
| * Non-Lua processing (e.g. network functions) will usually only take place once the current Lua chunk has completed execution. So any network calls should be viewed at an asynchronous request. A common coding mistake is to assume that they are synchronous, that is if two `socket:send()` are on consecutive lines in a Lua programme, then the first has completed by the time the second is executed. This is wrong. Each `socket:send()` request simply queues the send operation for dispatch. Neither will start to process until the Lua code has return to is calling C function. Stacking up such requests in a single Lua task function burns scarce RAM and can trigger a PANIC. This true for timer, network, and other callbacks. It is even the case for actions such as requesting a system restart, as can be seen by the following example: |
There was a problem hiding this comment.
This true for timer -> This is true for timer
docs/en/lua-developer-faq.md
Outdated
| So all Lua callbacks are called by C wrapper functions that are themselves callback activated by the SDK as a result of a given event. Such C wrapper functions themselves frequently need to store state for passing between calls or to other wrapper C functions. The Lua registry is simply another Lua table which is used for this purpose, except that it is hidden from direct Lua access. Any content that needs to be saved is created with a unique key. Using a standard Lua table enables standard garbage collection algorithms to operate on its content. | ||
| All Lua callbacks are called by C wrapper functions within the NodeMCU libraries that are themselves callbacks that have been activated by the SDK as a result of a given event. Such C wrapper functions themselves frequently need to store state for passing between calls or to other wrapper C functions. The Lua registry is a special Lua table which is used for this purpose, except that it is hidden from direct Lua access, but using a standard Lua table for this store enables standard garbage collection algorithms to operate on its content. Any content that needs to be saved is created with a unique key. The upvalues for functions that are global or referenced in the Lua Registry will persist between event routines, and hence any upvalues used by them will also persist and can be used for passing context. | ||
|
|
||
| * If you are running out of memory, then you might not be correctly clearing down Registry entries. One example is as above where you are setting up timers but not unregistering them. Another occurs in the following code fragment. The `on()` function passes the socket to the connection callback as it's first argument `sck`. This is local variable in the callback function, and it also references the same socket as the upvalue `srv`. So functionally `srv` and `sck` are interchangeable. So why pass it as an argument? Normally garbage collecting a socked will automatically unregister any of its callbacks, but if you use a socket as an upvalue in the callback, the socket is now referenced through the Register, and now it won't be GCed because it is referenced. Catch-22 and a programming error, not a bug. |
docs/en/lua-developer-faq.md
Outdated
| * The **File system** is a special case of persistent global, so there is no reason in principle why it can't be used to pass context. However the ESP8266 file system uses flash memory and this has a limited write cycle lifetime, so it is best to avoid using the file system to store frequently changing content except as a mechanism of last resort. | ||
| * **Upvalues**. When a function is declared within an outer function, all of the local variables in the outer scope are available to the inner function. Since all functions are stored by reference the scope of the inner function might outlast the scope of the outer function, and the Lua runtime system ensures that any such references persist for the life of any functions that reference it. This standard feature of Lua is known as *closure* and is described in [Pil 6]. Such values are often called *upvalues*. Functions which are global or [[#So how is the Lua Registry used and why is this important?|registered]] callbacks will persist between event routines, and hence any upvalues referenced by them can be used for passing context. | ||
| * It is important to understand that a single Lua function is associated with / bound to any event callback task. This function is executed from within the relevant NodeMCU library C code using a `lua_call()`. Even system initialisation which executes the `dofile("init.lua")` is really a special case of this. Each function can invoke other functions and so on, but it must ultimately return control to the C library code which then returns control the SDK, terminating the task. | ||
| * By their very nature Lua `local` variables only exist within the context of an executing Lua function, and so locals are unreferenced on exist and any local data (unless also a reference type such as a function, table, or user data which is also referenced elsewhere) can therefore be garbage collected between these `lua_call()` actions. |
There was a problem hiding this comment.
unreferenced on exist -> unreferenced on exit
docs/en/lua-developer-faq.md
Outdated
|
|
||
| ### Why is it importance to understand how upvalues are implemented when programming for the ESP8266? | ||
|
|
||
| Routines directly or indirectly referenced in the globals table, **_G**, or in the Lua Registry may use upvalues. The number of upvalues associated with a given routine is determined by the compiler and a vector is allocated when the closure is bound to hold these references. Each upvalues is classed as open or closed. All upvalues are initially open which means that the upvalue references back to the outer functions's register set. However, upvalues must be able to outlive the scope of the outer routine where they are declared as a local variable. The runtime VM does this by adding extra checks when executing a function return to scan any defined closures within its scope for back references and allocate memory to hold the upvalue and points the upvalue's reference to this. This is known as a closed upvalue. | ||
| The use of upvalues is a core Lua feature, that are explain in detail in PiL. Any Lua routines defined within an outer scope my use them. This can include routines directly or indirectly referenced in the globals table, **_G**, or in the Lua Registry. |
There was a problem hiding this comment.
that are explain -> that are explained
docs/en/lua-developer-faq.md
Outdated
| - Johny Mattsson is currently leading an ESP32 port. | ||
| - We have a lot more hardware modules supported. | ||
|
|
||
| Because the development is is active this list will no doubt continue to be revised and updated. |
docs/en/lua-developer-faq.md
Outdated
|
|
||
| The [NodeMCU company](http://NodeMCU.com/index_en.html) was set up by [Zeroday]([email protected]) to develop and to market a set of Lua firmware-based development boards which employ the Espressif ESP8266 SoC. The initial development of the firmware was done by Zeroday and a colleague, Vowstar, in-house with the firmware being first open-sourced on Github in late 2014. In mid-2015, Zeroday decided to open the firmware development to a wider group of community developers, so the core group of developers now comprises 6 community developers (including this author), and we are also supported by another dozen or so active contributors, and two NodeMCU originators. | ||
|
|
||
| This larger active team has allowed us to address most of the outstanding issues present at the first version. These include: |
There was a problem hiding this comment.
It's not clear to what you refer to, FAQ or firmware. I assume you mean the "first version of the FAQ"?
docs/en/lua-developer-faq.md
Outdated
|
|
||
| The ESP8266 was designed and is fabricated in China by [Espressif Systems](http://espressif.com/new-sdk-release/). Espressif have also developed and released a companion software development kit (SDK) to enable developers to build practical IoT applications for the ESP8266. The SDK is made freely available to developers in the form of binary libraries and SDK documentation. However this is in a *closed format*, with no developer access to the source files, so ESP8266 applications *must* rely solely on the SDK API (and the somewhat Spartan SDK API documentation). | ||
| The ESP8266 was designed and is fabricated in China by [Espressif Systems](http://espressif.com/new-sdk-release/). Espressif have also developed and released a companion software development kit (SDK) to enable developers to build practical IoT applications for the ESP8266. The SDK is made freely available to developers in the form of binary libraries and SDK documentation. However this is in a *closed format*, with no developer access to the source files, so anyone developing ESP8266 applications must rely solely on the SDK API (and the somewhat Spartan SDK API documentation). |
There was a problem hiding this comment.
Maybe worth mentioning that Espressif abandoned the closed-source model with the IDF for ESP32.
docs/en/lua-developer-faq.md
Outdated
|
|
||
| NodeMCU Lua is based on [eLua](http://www.eluaproject.net/overview), a fully featured implementation of Lua 5.1 that has been optimized for embedded system development and execution to provide a scripting framework that can be used to deliver useful applications within the limited RAM and Flash memory resources of embedded processors such as the ESP8266. One of the main changes introduced in the eLua fork is to use read-only tables and constants wherever practical for library modules. On a typical build this approach reduces the RAM footprint by some 20-25KB and this makes a Lua implementation for the ESP8266 feasible. This technique is called LTR and this is documented in detail in an eLua technical paper: [Lua Tiny RAM](http://www.eluaproject.net/doc/master/en_arch_ltr.html). | ||
|
|
||
| The mains impacts of the ESP8266 SDK and together with its hardware resource limitations are not in the Lua language implementation itself, but in how *application programmers must approach developing and structuring their applications*. As discussed in detail below, the SDK is non-preemptive and event driven. Tasks can be associated with given events by using the SDK API to registering callback functions to the corresponding events. Events are queued internally within the SDK, and it then calls the associated tasks one at a time, with each task returning control to the SDK on completion. *The SDK states that if any tasks run for more than 10 mSec, then services such as Wifi can fail.* | ||
| The mains impacts of the ESP8266 SDK and together with its hardware resource limitations are not in the Lua language implementation itself, but in how *application programmers must approach developing and structuring their applications*. As discussed in detail below, the SDK is non-preemptive and event driven. Tasks can be associated with given events by using the SDK API to registering callback functions to the corresponding events. Events are queued internally within the SDK, and it then calls the associated tasks one at a time, with each task returning control to the SDK on completion. *The SDK states that if any tasks run for more than 15 mSec, then services such as WiFi can fail.* |
There was a problem hiding this comment.
Do you mean "main" instead of "mains"?
docs/en/lua-developer-faq.md
Outdated
| * Current versions of the ESP8266 run the SDK over the native hardware so there is no underlying operating system to capture errors and to provide graceful failure modes, so system or application errors can easily "PANIC" the system causing it to reboot. Error handling has been kept simple to save on the limited code space, and this exacerbates this tendency. Running out of a system resource such as RAM will invariably cause a messy failure and system reboot. | ||
| * There is currently no `debug` library support. So you have to use 1980s-style "binary-chop" to locate errors and use print statement diagnostics though the systems UART interface. (This omission was largely because of the Flash memory footprint of this library, but there is no reason in principle why we couldn't make this library available in the near future as an custom build option). | ||
| * The LTR implementation means that you can't easily extend standard libraries as you can in normal Lua, so for example an attempt to define `function table.pack()` will cause a runtime error because you can't write to the global `table`. (Yes, there are standard sand-boxing techniques to achieve the same effect by using metatable based inheritance, but if you try to use this type of approach within a real application, then you will find that you run out of RAM before you implement anything useful.) | ||
| * The ESP8266 uses a combination of on-chip RAM and off-chip Flash memory connected using a dedicated SPI interface. Code can be executed directly from Flash (the ESP hardware first faults the Flash code into an L1 cache in RAM, but this largely transparent in terms of programming ESP8266 applications). The Lua firmware largely runs out of Flash. Both the RAM and the Flash memory are *very* limited when compared to systems than most application programmers use. |
There was a problem hiding this comment.
I don't understand "to fault" in this context.
docs/en/lua-developer-faq.md
Outdated
| * My approach is avoid using then unless I have a *very* good reason to justify this. I track them statically by running a `luac -p -l XXX.lua | grep GLOBAL` filter on any new modules and replace any accidental globals by local or upvalued local declarations. | ||
|
|
||
| * On NodeMCU, _G's metatable is _G, so you can create any globals that you need and then 'close the barn door' by assigning | ||
| `_G.__newindex=function(g,k,v) error ("attempting to set global "..k.." to "..v) end` and any attempt to create new globals with now throw an error and give you a traceback of where this has happened. |
docs/en/lua-developer-faq.md
Outdated
|
|
||
| One further complication is that some library functions don't correctly dereference expired callback references and as a result their upvalues may not be correctly garbage collected (though we are tracking this down and hopefully removing this issue). This will all be manifested as a memory leak. So using upvalues can cause more frequent and difficult to diagnose PANICs during testing. So my general recommendation is still to stick to globals for this specific usecase of passing context between event callbacks, and `nil` them when you have done with them. | ||
| One further complication is that some library functions don't implicitly dereference expired callback references and as a result their upvalues may not be garbage collected and this application error can be be manifested as a memory leak. So using upvalues can cause more frequent and difficult to diagnose PANICs during testing. So my general recommendation is still to stick to globals during initial development, and explicitly dereference resources by setting them to `nil` them when you have done with them. |
docs/en/lua-developer-faq.md
Outdated
|
|
||
| ### How do I avoid a PANIC loop in init.lua? | ||
|
|
||
| Most of us have fallen into the trap of creating an `init.lua` that has a bug in it, which then causes the system to reboot and hence gets stuck in a reboot loop. If you haven't then you probably will do so at least once. | ||
|
|
||
| - When this happens, the only robust solution is to reflash the firmware. | ||
| - The simplest way to avoid having to do this is to keep the `init.lua` as simple as possible -- say configure the wifi and then start your app using a one-time `tmr.alarm()` after a 2-3 sec delay. This delay is long enough to issue a `file.remove("init.lua")` through the serial port and recover control that way. | ||
| - Also it is always best to test any new `init.lua` by creating it as `init_test.lua`, say, and manually issuing a `dofile("init_test.lua")` through the serial port, and then only rename it when you are certain it is working as you require. | ||
| - Another trick is to poll a spare GPIO input pin in your startup. I do this omy boards by taking this GPIO plus Vcc to a jumper on the board, so that I can set the jumper to jump into debug mode or reprovision the software. |
docs/en/lua-developer-faq.md
Outdated
| * **Generate a bytecode listing on your development PC**. The Lua 5.1 code generator is basically the same on the PC and on the ESP8266, so whilst it isn't identical, using the standard Lua batch compiler `luac` against your source on your PC with the `-l -s` option will give you a good idea of what your code will generate. The main difference between these two variants is the size_t for ESP8266 is 4 bytes rather than the 8 bytes size_t found on modern 64bit development PCs; and the eLua variants generate different access references for ROM data types. If you want to see what the `string.dump()` version generates then drop the `-s` option to retain the debug information. | ||
| * **Upload your `.lc` files to the PC and disassemble then there**. There are a number of Lua code disassemblers which can list off the compiled code that you application modules will generate, `if` you have a script to upload files from your ESP8266 to your development PC. I use [ChunkSpy](http://luaforge.net/projects/chunkspy/) which can be downloaded [here](http://files.luaforge.net/releases/chunkspy/chunkspy/ChunkSpy-0.9.8/ChunkSpy-0.9.8.zip) , but you will need to apply the following patch so that ChunkSpy understands eLua data types: | ||
| * **Generate a bytecode listing on your development PC**. The Lua 5.1 code generator is basically the same on the PC and on the ESP8266, so whilst it isn't identical, using the standard Lua batch compiler `luac` against your source on your PC with the `-l -s` option will give you a good idea of what your code will generate. The main difference between these two variants is the size_t for ESP8266 is 4 bytes rather than the 8 bytes size_t found on modern 64bit development PCs; and the eLua variants generate different access references for ROM data types. If you want to see what the `string.dump()` version generates then drop the `-s` option to retain the debug information. You can also build `luac.cross` with this firmware and this generate lc code for the target ESP architecture. | ||
| * **Upload your `.lc` files to the PC and disassemble then there**. There are a number of Lua code disassemblers which can list off the compiled code that you application modules will generate, `if` you have a script to upload files from your ESP8266 to your development PC. I use [ChunkSpy](http://luaforge.net/projects/chunkspy/) which can be downloaded [here](http://files.luaforge.net/releases/chunkspy/chunkspy/ChunkSpy-0.9.8/ChunkSpy-0.9.8.zip) , but you will need to apply the following patch so that ChunkSpy understands eLua data types: |
There was a problem hiding this comment.
and disassemble them there
the compiled code that your application modules will generate
docs/en/lua-developer-faq.md
Outdated
| Here bigger means less RAM used. | ||
|
|
||
| Of course you should still use functions to structure your code and encapsulate common repeated processing, but just bear in mind that each function definition has a relatively high overhead for its header record and stack frame (compared to the 20 odd KB RAM available). *So try to avoid overusing functions. If there are less than a dozen or so lines in the function then you should consider putting this code inline if it makes sense to do so.* | ||
| Functions have fixed overheads, so in general the more that your group your application code into larger function, then less RAM used will be used overall. The main caveat here is that if you are starting to do "copy and paste" coding across functions then you are wasting resources. So of course you should still use functions to structure your code and encapsulate common repeated processing, but just bear in mind that each function definition has a relatively high overhead for its header record and stack frame. *So try to avoid overusing functions. If there are less than a dozen or so lines in the function then you should consider putting this code inline if it makes sense to do so.* |
There was a problem hiding this comment.
the more that you group your application
then less RAM will be used overall
docs/en/lua-developer-faq.md
Outdated
| * Lua variable can be assigned two broad types of data: **values** such as numbers, booleans, and strings and **references** such as functions, tables and userdata. You can see the difference here when you assign the contents of a variable `a` to `b`. In the case of a value then it is simply copied into `b`. In the case of a reference, both `a` and `b` now refer to the *same object*, and no copying of content takes place. This process of referencing can have some counter-intuitive consequences. For example, in the following code by the time it exists, the variable `timer2func` is out of scope. However a reference to the function has now been stored in the Lua registry by the alarm API call, so it and any upvalues that it uses will persist until it is eventually entirely dereferenced (e.g. by `tmr2:unregister()`. | ||
| ```Lua | ||
| do | ||
| local tmr2unc = function() ds.convert_T(true) tmr1:start() end |
There was a problem hiding this comment.
local tmr2unc = function() ds.convert_T(true) tmr1:start() end
->
local tmr2func = function() ds.convert_T(true); tmr1:start() end
& correction of tmr2unc on the following line
There was a problem hiding this comment.
Actually the ; here is syntactic sugar and the code compiles and runs fine without it, but I agree that the code is clearer with the separator.
docs/en/lua-developer-faq.md
Outdated
| end | ||
| end | ||
| ``` | ||
| * This loop is compiled once when the module is required. Th opcode vectors for the read and write functions are created during the compile, along with a header which defines how many upvalues and locals are used by each function. However, these two functions are then bound _four_ times as different functions (e.g. `mcp23008.writeIODOR()`) and each closure inherits it's own copies of the upvalues it uses so the `regAddr` for this function is `0x00`). The upvalue list is created when the closure is created and though some Lua magic, even if the outer routine which initially declared then is no longer in scope and is GCed (Garbage Collected), the Lua RTS ensures that upvalue still persist whilst the closure persists. |
There was a problem hiding this comment.
This example, although very elegant, is a bit complex for what is being explained here.
I would prefer to see something very elementary. The following trivial example helps me understand:
The following two lines of code have the same functionality but the second one consumes less memory (50 bytes less in this case) and is more efficient.
tmr.create():alarm(1000, tmr.ALARM_AUTO, function() print(node.heap()) end)local node_heap=node.heap; tmr.create():alarm(1000, tmr.ALARM_AUTO, function() print(node_heap()) end)The first callback function every time it is called needs to search for a member of the node table called heap (opcode GETGLOBAL + GETTABLE) but it also needs to store the string "heap" so it knows what to look for.
The second example uses just the upvalue node_heap. At the time the timer is printing the amount of free memore it no longer needs to know the string "heap" and does not have to look it up in the table node. It uses the reference to the function node.heap()stored in the upvalue (opcodeGETUPVAL`). As noted it is more efficient and less memory demanding.
There was a problem hiding this comment.
I agree that using locals will in general generate less code and execute faster, but that is a different point than the issue about locals used as upvalues inside functions will still persist even though the binding function is not longer scoped.
The second example is complicated but it does underline the basic point that a function is compiled once but can be bound many times at run time.
|
Thanks guys for all this feed back. I will go through the doc in the next couple of days and incorporate this. Any Qs or topics that you feel should be in this, that I've missed? |
|
Poke? |
|
On my TODO list. I'll push out an update this w/e. I was hoping for a few extra points, but obviously none are forthcoming. |
|
Please cover also Another very typical example: a server. How to implement a memory efficient handling of receive events? Can we call dofile() from within the receive handlers? Is it smart? I guess it's a tradeoff between speed and memory but we generally miss memory, not speed. Perhaps the handlers can contain just dofile() and nothing else? Will it free the memory as soon as possible? What is the impact on speed, anyway? Last but not least: is it possible to free the memory used by init.lua if one of the last commands is |
|
In that |
|
Actually Marcel, whoever this guy or gal is, he / she is terribly ill informed and giving bad advice, so I don't recommend that we link to it. The advice seems to be don't use locals or globals (and by implication any variables) or tables or functions, which is all just plain silly. We could with some targeted advice in the FAQ, but the main thing is that Lua coders should have and understanding of the rules for the life of any variable and how the GC works. You can write some pretty serious embedded applications and still stay within the RAM constraints of the ESP8266 if you understand these. |
I agree, you may reduce the memory footprint of your application but any such code will be terribly hard to read and thus hard to understand and maintain. It's just that the search engines rank that page pretty high. |
|
@joysfera Petr, the GC in Lua is really quite simple: if a variable or resource is referenced directly or indirectly then it won't be GCed. Conversely if it isn't then it can be. Doing a Tables, userdata, function closures, strings are all treated as collectable resources. So if they are referenced then they won't be collected and if they aren't then they can be. There is no time expiry on this. The only reason that a required module sticks around is that the |
|
I've just pushed a version with these changes in it. Unless anyone can see any show-stoppers, then I suggest that we merge this. It's not perfect, but it never will be. I would really prefer something more fluid that we could collaborate on, such as a wiki pageset and which we occasionally snap to the readthedocs reference, but I've already lost that one 😄 |
|
I have just a few global variables and functions, all the rest is local, most of the stuff is loaded on demand only yet I have less than 23 kB free. This is not enough for TLS, unfortunately. That's why I am looking for any advice how to write code that keeps as much memory free as possible. |
Not that anything is wrong but you promised to include #1861 😜 Sorry for being such a pest. |
|
Marcel, I want to print this off and consider again. I asked you guys if you had any other FAQs, and I've considered a list. Some are very ESP-specific and some are really general Lua, but not really something that you worry too much about except when you are running in a tight RAM configuration as in the ESP8266.
The problem is that there's a lot of ground to cover here and a lot is covered in other references that I call out, but people don't seem to read them. maybe we just need to break this up more. Need to think, but let's get this out for new and move this sort of stuff into a separate issue/PR. |
Fixes #937.
devbranch rather than formaster.This is a major rewrite of the Lua Developer FAQ to bring it up to date with the current state of development.
This type of FAQ is always going to tend to be the views of one author and in the case me. The commnets I would be really interested in are:
Note that the MD format is pretty diff hostile for this level of change, so you might just want to take a view on the new document as a whole.