Migrate nestboxes to lua and add to autobutcher

[Birkow]

Nestboxes rewritten in lua with enhanced functionality, added to autobutcher.
Started with DFHack/scripts#1296, moved to dfhack to add as part of autobutcher
Attached performance check after ~1 year with 14 nestboxes and birds assigned to all of them. Script checks created egg item when called from eventful on ITEM_CREATED event.

dfhack-perf-counters.dat

Summary

Measuring unpaused time only

elapsed 1299094 ms (21m 39s)
dfhack 118211 ms ( 9.10% of elapsed)

DFHack details

keybinding 16 ms ( 0.01% of dfhack, 0.00% of elapsed)
update 25355 ms ( 21.45% of dfhack, 1.95% of elapsed)
overlay 92809 ms ( 78.51% of dfhack, 7.14% of elapsed)
zscreen 31 ms ( 0.03% of dfhack, 0.00% of elapsed)

Update details

event manager 4087 ms ( 16.12% of update, 0.31% of elapsed)
plugin onUpdate 17178 ms ( 67.75% of update, 1.32% of elapsed)
lua timers 3194 ms ( 12.60% of update, 0.25% of elapsed)

Event manager per event type

      UNIT_NEW_ACTIVE     1380 ms ( 33.77% of event manager,   0.11% of elapsed)
          JOB_STARTED     1296 ms ( 31.71% of event manager,   0.10% of elapsed)
        JOB_COMPLETED      847 ms ( 20.72% of event manager,   0.07% of elapsed)
        JOB_INITIATED      127 ms (  3.11% of event manager,   0.01% of elapsed)
             BUILDING       61 ms (  1.49% of event manager,   0.00% of elapsed)
         ITEM_CREATED       31 ms (  0.76% of event manager,   0.00% of elapsed)

            framework      345 ms (  8.44% of event manager,   0.03% of elapsed)
        all subtimers     3742 ms ( 91.56% of event manager,   0.29% of elapsed)

Event manager JOB_INITIATED event per plugin

             eventful       63 ms ( 49.61% of event type,   0.00% of elapsed)

            framework       64 ms ( 50.39% of event type,   0.00% of elapsed)
        all subtimers       63 ms ( 49.61% of event type,   0.00% of elapsed)

Event manager JOB_COMPLETED event per plugin

            framework      847 ms (100.00% of event type,   0.07% of elapsed)
        all subtimers        0 ms (  0.00% of event type,   0.00% of elapsed)

Event manager BUILDING event per plugin

            framework       61 ms (100.00% of event type,   0.00% of elapsed)
        all subtimers        0 ms (  0.00% of event type,   0.00% of elapsed)

Event manager UNIT_NEW_ACTIVE event per plugin

            framework     1380 ms (100.00% of event type,   0.11% of elapsed)
        all subtimers        0 ms (  0.00% of event type,   0.00% of elapsed)

Event manager ITEM_CREATED event per plugin

             eventful       16 ms ( 51.61% of event type,   0.00% of elapsed)

            framework       15 ms ( 48.39% of event type,   0.00% of elapsed)
        all subtimers       16 ms ( 51.61% of event type,   0.00% of elapsed)

Event manager JOB_STARTED event per plugin

               burrow       15 ms (  1.16% of event type,   0.00% of elapsed)

            framework     1281 ms ( 98.84% of event type,   0.10% of elapsed)
        all subtimers       15 ms (  1.16% of event type,   0.00% of elapsed)

Update per plugin

              overlay    12762 ms ( 74.29% of update,   0.98% of elapsed)
             autochop      316 ms (  1.84% of update,   0.02% of elapsed)
       preserve-rooms      265 ms (  1.54% of update,   0.02% of elapsed)
            logistics      237 ms (  1.38% of update,   0.02% of elapsed)
             autofarm      203 ms (  1.18% of update,   0.02% of elapsed)
             dwarfvet      186 ms (  1.08% of update,   0.01% of elapsed)
 RemoteFortressReader      169 ms (  0.98% of update,   0.01% of elapsed)
       suspendmanager      140 ms (  0.81% of update,   0.01% of elapsed)
           stonesense      125 ms (  0.73% of update,   0.01% of elapsed)
         buildingplan      110 ms (  0.64% of update,   0.01% of elapsed)
          add-spatter      109 ms (  0.63% of update,   0.01% of elapsed)
            seedwatch      108 ms (  0.63% of update,   0.01% of elapsed)
          autobutcher      107 ms (  0.62% of update,   0.01% of elapsed)
               burrow       94 ms (  0.55% of update,   0.01% of elapsed)
              3dveins       93 ms (  0.54% of update,   0.01% of elapsed)
          autonestbox       79 ms (  0.46% of update,   0.01% of elapsed)
         autoclothing       78 ms (  0.45% of update,   0.01% of elapsed)
                tweak       77 ms (  0.45% of update,   0.01% of elapsed)
               tailor       63 ms (  0.37% of update,   0.00% of elapsed)
            cxxrandom       62 ms (  0.36% of update,   0.00% of elapsed)
             autoslab       62 ms (  0.36% of update,   0.00% of elapsed)
              hotkeys       48 ms (  0.28% of update,   0.00% of elapsed)
         pet-uncapper       47 ms (  0.27% of update,   0.00% of elapsed)
                 zone       47 ms (  0.27% of update,   0.00% of elapsed)
                plant       47 ms (  0.27% of update,   0.00% of elapsed)
             spectate       32 ms (  0.19% of update,   0.00% of elapsed)
               deramp       32 ms (  0.19% of update,   0.00% of elapsed)
               misery       32 ms (  0.19% of update,   0.00% of elapsed)
             showmood       32 ms (  0.19% of update,   0.00% of elapsed)
       preserve-tombs       32 ms (  0.19% of update,   0.00% of elapsed)
              aquifer       31 ms (  0.18% of update,   0.00% of elapsed)
           changevein       31 ms (  0.18% of update,   0.00% of elapsed)
            nestboxes       31 ms (  0.18% of update,   0.00% of elapsed)
            autolabor       31 ms (  0.18% of update,   0.00% of elapsed)
        fix-occupancy       31 ms (  0.18% of update,   0.00% of elapsed)
           cursecheck       31 ms (  0.18% of update,   0.00% of elapsed)
            tiletypes       31 ms (  0.18% of update,   0.00% of elapsed)
             pathable       30 ms (  0.17% of update,   0.00% of elapsed)
            fastdwarf       16 ms (  0.09% of update,   0.00% of elapsed)
            getplants       16 ms (  0.09% of update,   0.00% of elapsed)
                  dig       16 ms (  0.09% of update,   0.00% of elapsed)
                flows       16 ms (  0.09% of update,   0.00% of elapsed)
              liquids       16 ms (  0.09% of update,   0.00% of elapsed)
                 sort       16 ms (  0.09% of update,   0.00% of elapsed)
           createitem       16 ms (  0.09% of update,   0.00% of elapsed)
              dig-now       16 ms (  0.09% of update,   0.00% of elapsed)
            faststart       16 ms (  0.09% of update,   0.00% of elapsed)
          filltraffic       16 ms (  0.09% of update,   0.00% of elapsed)
              regrass       16 ms (  0.09% of update,   0.00% of elapsed)
          changelayer       16 ms (  0.09% of update,   0.00% of elapsed)
               reveal       16 ms (  0.09% of update,   0.00% of elapsed)
           cleanconst       16 ms (  0.09% of update,   0.00% of elapsed)
               orders       16 ms (  0.09% of update,   0.00% of elapsed)
               stocks       15 ms (  0.09% of update,   0.00% of elapsed)
           changeitem       15 ms (  0.09% of update,   0.00% of elapsed)
            blueprint       15 ms (  0.09% of update,   0.00% of elapsed)
               design       15 ms (  0.09% of update,   0.00% of elapsed)
             tubefill       15 ms (  0.09% of update,   0.00% of elapsed)
          strangemood       15 ms (  0.09% of update,   0.00% of elapsed)
             eventful       15 ms (  0.09% of update,   0.00% of elapsed)

            framework      792 ms (  4.61% of update,   0.06% of elapsed)
        all subtimers    16386 ms ( 95.39% of update,   1.26% of elapsed)

State change per plugin

            framework    17178 ms (100.00% of update,   1.32% of elapsed)
        all subtimers        0 ms (  0.00% of update,   0.00% of elapsed)

Lua timer details

     control-panel/fix/noexert-exhaustion      578 ms ( 18.10% of lua timers,   0.04% of elapsed)
                    control-panel/combine      566 ms ( 17.72% of lua timers,   0.04% of elapsed)
              control-panel/fix/ownership      328 ms ( 10.27% of lua timers,   0.03% of elapsed)
     control-panel/fix/empty-wheelbarrows      297 ms (  9.30% of lua timers,   0.02% of elapsed)
      control-panel/fix/stuck-instruments      237 ms (  7.42% of lua timers,   0.02% of elapsed)
          control-panel/fix/stuck-worship      220 ms (  6.89% of lua timers,   0.02% of elapsed)
         control-panel/fix/general-strike      220 ms (  6.89% of lua timers,   0.02% of elapsed)
                                 autofish      140 ms (  4.38% of lua timers,   0.01% of elapsed)
             control-panel/fix/engravings      124 ms (  3.88% of lua timers,   0.01% of elapsed)
            control-panel/fix/dry-buckets       80 ms (  2.50% of lua timers,   0.01% of elapsed)
          control-panel/orders-reevaluate       16 ms (  0.50% of lua timers,   0.00% of elapsed)
             control-panel/fix/dead-units       15 ms (  0.47% of lua timers,   0.00% of elapsed)
                   control-panel/automilk       15 ms (  0.47% of lua timers,   0.00% of elapsed)
                  control-panel/autoshear       15 ms (  0.47% of lua timers,   0.00% of elapsed)

                                framework      343 ms ( 10.74% of lua timers,   0.03% of elapsed)
                            all subtimers     2851 ms ( 89.26% of lua timers,   0.22% of elapsed)

Overlay details

              agitation-rebalance.monitor    13287 ms ( 14.32% of overlay,   1.02% of elapsed)
                         gui/notify.panel     7348 ms (  7.92% of overlay,   0.57% of elapsed)
                             hotkeys.menu     3108 ms (  3.35% of overlay,   0.24% of elapsed)
                   sort.location_selector     2701 ms (  2.91% of overlay,   0.21% of elapsed)
                   suspendmanager.overlay     2041 ms (  2.20% of overlay,   0.16% of elapsed)
                                sort.info     1292 ms (  1.39% of overlay,   0.10% of elapsed)
                      sort.elevate_barony     1236 ms (  1.33% of overlay,   0.10% of elapsed)
                          notes.map_notes     1110 ms (  1.20% of overlay,   0.09% of elapsed)
             gui/civ-alert.big_red_button      910 ms (  0.98% of overlay,   0.07% of elapsed)
                                sort.slab      893 ms (  0.96% of overlay,   0.07% of elapsed)
                               sort.world      871 ms (  0.94% of overlay,   0.07% of elapsed)
                          sort.candidates      866 ms (  0.93% of overlay,   0.07% of elapsed)
                              sort.places      784 ms (  0.84% of overlay,   0.06% of elapsed)
                          confirm.overlay       63 ms (  0.07% of overlay,   0.00% of elapsed)
                    gui/design.dimensions       16 ms (  0.02% of overlay,   0.00% of elapsed)
                      zone.retirelocation       15 ms (  0.02% of overlay,   0.00% of elapsed)

                                framework    56268 ms ( 60.63% of overlay,   4.33% of elapsed)
                            all subtimers    36541 ms ( 39.37% of overlay,   2.81% of elapsed)

ZScreen details

                            warn-stranded       31 ms (100.00% of zscreen,   0.00% of elapsed)

                                framework        0 ms (  0.00% of zscreen,   0.00% of elapsed)
                            all subtimers       31 ms (100.00% of zscreen,   0.00% of elapsed)

[Birkow]

pre-commit.ci autofix

[lethosor]

I may be missing some earlier discussion on this, but why the merge into autobutcher? The two tools don't really seem all that similar to me - sure, they both deal with animals, but autobutcher is intended to control which units get butchered automatically, and nestboxes is intended to protect the eggs of egglayers. I think autobutcher nestboxes makes less sense than nestboxes from an end-user perspective.

[myk002]

why the merge into autobutcher?

It's because of the configuration overlap. we're adding a nestboxes feature that essentially tracks targets per race, which is very aligned with autobutcher's concept and interface. if you configure one, you are very likely to want to configure the other the same way, so it seemed like it made sense to combine them

[lethosor]

Would some other name make more sense?

As a historical example, I feel like this combination makes about as much sense as combining autobutcher and zone, which were previously combined until we split them up a couple years back.

[myk002]

For the record, I didn't understand the autobutcher/zone combination at all. They didn't share code or configuration. Moreover, one was an automation tool and the other was a productivity tool. In this case, both are automation tools with aligned purposes.

I think this was the discussion on Discord: https://discord.com/channels/793331351645323264/807444515194798090/1284174784472547338

[Birkow]

Nestboxes is already recognized as egg handling thing, leaving it might make changes easier on end users already familiar with name. Although I'm not really attached to it. Started working on it with eggwatch name, any name would be fine with me. There is still possibility of renaming autobutcher like myk mentioned to something like animal-husbandry to cover expanded functionality.

Adding nestboxes as part of autobutcher makes sense to me as it can be used to make sure butcher will not be overworked by handling excess hatchlings. With default settings only protect enough fertile eggs to reach autobutcher targets for the population (plus few extra to deal with random gender).

[myk002]

I'm finding the proposed API a bit confusing. Let's hash out the behavior and interface in the docs before I look at the actual code

[myk002]

Maybe it would be better if you could explain/mock up how the new configuration will fit into gui/autobutcher:

I was hoping there would be one extra column and nothing else, but the complexity of the API makes me doubt that that is possible. However, I'm also not convinced that all the new options are necessary, and I think some of them could be trimmed.

[Birkow]

Edit: changed screenshot to include shortcuts for setting new per race fields:

I think from "global" variables for Nestboxes module it's worth to only expose enabled status, others can be hidden/removed. Added only that one to mock up.
For per race config added everything to mock up. I have no idea if any of those should be hidden/removed.

Obviously new fields would be filled only for egglayers and first 2 rows with default/all, just added it here as static text to everything.

Currently there are following variables saved in persisted site config:

• enabled
  enabled status for nestboxes module, added to mock up
• ignore_autobutcher
  would allow nestboxes work on it's own in without autobutcher enabled, good candidate to remove/not show in gui
• split_stacks
  if egg stacks should be split when stack goes partially over target amount of protected eggs, good candidate to
  remove and always split stacks
• default
  default settings for new races
• config_per_race
  table with per race settings
  Not exposed to user:
• verbose
  probably good idea to remove from persisted config and move to separate global variable, when enabled prints a details with debug info
• migration_from_cpp_to_lua_done
  flag to mark migration of enabled status from cpp nenstboxes

Settings per race:

• watched
  If eggs for race should be protected
• target
  base amount of protected eggs
• ama
  Add Missing Animals to base target for protected eggs, difference between current amount of animals and target set for autobutcher (both for kids and adults)
• stop
  stop protecting eggs once autobutcher target for live animals is reached