RFC: Functions Registry (a.k.a Loader) [RELEASED]

[42atomys]

Introduction

This RFC introduces enhancements to the loader feature of the go-sprout/sprout project, focusing on optimizing the management and execution of functions within Go templates. This aims to prevent the unnecessary loading of functions and enhance developer control over function management.

---

Important

Actual implementation selected are available here https://github.com/orgs/go-sprout/discussions/31#discussioncomment-10013601

Tip

RELEASED: The registry system has released on v0.5.0 🎉 Thanks for all to have contributed to the RFC, reviews, and feedbacks 🌱 💜
If you have concerns don't hesitate to talk about on release discussion or open an issue !

---

Motivation

In large-scale applications, the efficiency of template processing is crucial. Loading all available functions into every template instance not only affects performance but also increases memory usage unnecessarily. This proposal seeks to provide a solution where developers can selectively load functions, optimizing resource use and processing speed.

Proposal

We propose the development of a selective function loading mechanism within the loader structure, allowing developers to specify which functions are loaded into a template environment.

And also have a custom function loading to enhance the flexibility and utility of sprout, we propose allowing developers to load custom functions specific to their projects. This would enable them to not only utilize but also extend sprout's core functionalities according to their unique requirements, thereby maximizing the adaptability and application of the loader system across diverse scenarios.

Technical Description

• Selective Loading: Functions are no longer automatically loaded into every template. Instead, developers can specify which functions are needed for a particular template.
• Loader Enhancements: The loader will include methods to manage function availability on a per-handler basis, allowing global or per-template mechanism.
• Function Groups: Introduce the concept of function groups, where related functions can be grouped and loaded together. This allows for more organized management and can be particularly useful in templates that repeatedly use certain sets of functions.
• Dynamic Loading at Runtime: Templates can request additional functions as needed during their execution, which the loader can then resolve and provide dynamically.
• Safe/Must Function: Each function will be implemented with the signature func(...any) (any, error) (must in sprig). The Loader will automatically register a safe version of the function func(...any) any.

Code example

This code is an example and may be subject to changes.

// Define a custom handler struct with methods that can be loaded into the sprout loader.
type myHandler struct{}

// Hello method for the myHandler struct, returns a simple greeting string.
func (_ *myHandler) Hello() string {
	return "Hello World!"
}


func main() {
	// Initialize a new loader from the sprout package.
	loader := sprout.NewLoader()

	// Assume that mathSum and mathPow functions are predefined elsewhere in the application.
	// Load a predefined group of math functions into the loader.
	loader.LoadFunctionGroup("math", mathSum, mathPow)
	
	// Load a single function into the loader.
	loader.LoadFunction("sum", mathSum)
	
	// Load all methods from the myHandler struct as functions into the loader.
	loader.LoadFunctionFromStruct(&myHandler{})

	exampleTemplateFuncs(loader)
}

// Example showing how to initialize function maps for use in Go templates.
func exampleTemplateFuncs(loader *sprout.Loader) {
	// Creating a function map from a loader with all loaded functions.
	funcsMap := sprout.FuncsMap(sprout.WithLoader(loader))

	// Creating a function map from a loader but only including functions from the "math" group.
	funcsMap := sprout.FuncsMap(sprout.WithLoaderGroup(loader, "math"))

	// Creating a function map from a loader but only including the "sum" function.
	funcsMap := sprout.FuncsMap(sprout.WithLoaderFunction(loader, "sum"))

	// Example of how to use these function maps with Go's text/template package to parse and execute a template.
	// This template utilizes the "sum" function loaded into funcsMapSum.
	tmpl, err := template.New("exampleTemplate").Funcs(funcsMapSum).Parse(`{{sum 1 2 3}}`)
	if err != nil {
		log.Fatalf("Failed to parse template: %v", err)
	}

	// Execute the parsed template with no data (nil).
	var buf bytes.Buffer
	if err := tmpl.Execute(&buf, nil); err != nil {
		log.Fatalf("Failed to execute template: %v", err)
	}

	// Output the result of the template execution.
	fmt.Println(buf.String())
}

Benefits

• Performance: Reduces the overhead associated with loading unnecessary functions.
• Flexibility: Developers can tailor function availability to the needs of specific templates.
• Maintainability: Easier to manage and debug templates when only relevant functions are loaded.

Drawbacks

• Complexity in Configuration: Requires additional setup from developers to manage function groups and template-specific function availability. (Can be addressed by having built-ins groups and one function to load all functions to keep the idea of sprig)
• Initial Learning Curve: Developers need to familiarize themselves with the new loading mechanism.

Alternatives Considered

• Lazy Loading: Functions could be loaded on-demand when called in a template. However, this might introduce latency during template execution, so this is excluded.

Open Questions

1. Configuration Management: How can function groups be defined and managed to ensure user-friendliness and security?
   a. For the LoadFunctionFromStruct method, should a specific prefix, such as TemplateFunc, be mandated to explicitly secure and clarify function purposes?
   b. For the WithLoaderGroup method, which is better: using a string-based identifier (e.g., "math") or a type-based identifier (e.g., sprout.MathFunctionGroup) for defining function groups?
   c. For the LoadFunction method, should function identification rely on string-based names (e.g., "print") or type-based identifiers (e.g., sprout.PrintFunction)? What are the implications of each approach for system robustness and usability?

Future Extensions

Intelligent Preloading: Based on a pre-read of the template, automatically determine which functions to load for a template.
Resource Monitoring: Tools to monitor the performance impact of loaded functions on template processing.

Conclusion

Your insights and comments are crucial for refining and improving the proposed loader feature, ensuring it meets the needs and expectations of a broad range of developers. Please share your thoughts on the proposed functionalities, any concerns you might have, and any additional features you believe should be considered. This collaborative approach is essential for us to achieve a robust and versatile implementation. We look forward to your contributions and are excited to see how together we can enhance the capabilities of sprout.

[ripienaar]

I've never really had needs for such complexity - with funcmaps just being maps it's easy to add/remove functions. But I do not remember a case where I wanted to remove functions

I did need to replace functions with others to facilitate backward compatible introduction of sprig into existing categories. It would be beneficial to support easily preventing some function/group but loading the rest.

What is a problem is the compile time dependencies for things like the crypt functions and this mechanism would not address that - unused functions would still be compiled in as the resolution of which to load would be runtime and potentially dynamic.

If dynamic of template initiated loading is supported that should be off by default for security needs when templates are user supplied which would be the most common case.

[42atomys]

Thank you for your feedback. You raise an important point regarding the build-time complexity.

One potential solution, I have in mind when I read your comment to address the issue of unwanted compile-time dependencies, especially for pre-built functions in sprout, could be the use of sub-module imports. This approach allows for selective inclusion of dependencies at build time, ensuring that only the desired functionalities are compiled. For example:

import (
  "github.com/go-sprout/sprout"
  _ "github.com/go-sprout/sprout/modules/math"
  _ "github.com/go-sprout/sprout/modules/crypto"
)

By using this method, you can include only the specific modules needed for your project, reducing unnecessary compilation of unused functions.

Your concerns about security with dynamically or template-initiated loading are valid and I share it. Indeed, This kind of features must be disabled by default to safeguard against potential security vulnerabilities. 👮

---

Additionally, the dynamic loader feature of sprout supports loading functions from external libraries while leveraging sprout's standardized features for template management. This can greatly enhance flexibility and maintain standardization across different parts of your application.

[ripienaar]

Yeah an import based approach would be good.

I am curious if anyone else monitoring this thread ever had a need for the complexity outlined here, what real world scenarios this is for and maybe some repos that exist on github exhibits the need. I am struggling to see the use case.

[andreynering]

@ripienaar is right. Choosing which functions we're going to include is nice and we could have that, too. But by far the most important here is to have crypto/etc on a separate package so that won't even be imported/compiled for those that don't want to, to save compilation time and binary size.

Talking about crypto... I think there's room to have two packages here. There are some sprig function like sha256 that could perhaps be useful to more folks, but bcrypt and certificate functions (and etc) should be pretty rare. So the options are:

1. Keep only checksum functions, but on a separate package. Ditch other useless crypto functions out.
2. If we want to keep compatibility with sprig, we could that those crypto functions but on a separated package, only for those who want it. Separate checksumming and crypto.

[42atomys]

Thanks for your feedback, @andreynering. We are aligned on the necessity of separation. For backward compatibility, my vision is to provide a v0.x version compatible with seamless migration, accompanied by a simple migration guide to transition to v1.0 once it is released.

Regarding the crypto/checksum functions, we can indeed separate them easily. We can create more packages to allow for a more granular selection of functions.

Some libraries or applications may require specific functions. Therefore, having a package import strategy as the default is essential. Additionally, we can leverage the power of go generate to let libraries or applications compile their package with only the necessary functions, achieving an "import perfect" setup. (This can fit perfect with the other RFC safe/must)

[42atomys]

I will start working on this in a few days/weeks to give people more time to discuss the RFC if needed. :)

[42atomys]

UPDATE: Some IRL (IRL is a curious thing that forces me to go outside of my cave) - I'll take my time this week. I probably will start a POC (proof of concept) this week and propose the pull request here to get feedback!

See you 🌱 💜

[42atomys]

UPDATE: After somes complication, I'm back ! So after takes pros and cons for import based stractegy (and to excluse crypto as well 🛴)

So that is the

Setup and Usage in Main Application (main.go)

A main.go example of how to use and import various handlers. Here's how you can integrate the CryptoHandler with the Sprout library: (I select this one for the example because that the sun in the kitchen)

package main

import (
	"html/template"

	"github.com/go-sprout/sprout"
	"github.com/go-sprout/sprout/pkg/crypto"
)

func main() {
	fh := sprout.NewFunctionHandler(
		// Customization of the global handler can be done here (e.g., add aliases or custom logger)
	)

	// Register handlers in the registry for use in templates
	fh.RegisterHandlers(crypto.NewCryptoHandler(fh))

	// Execute the template
	tpl := template.Must(
		template.New("base").Funcs(fh.Registry()).ParseGlob("*.tmpl"),
	)
}

Crypto Package Setup (crypto package)

The crypto package defines its handler which embeds the Sprout FunctionHandler. This design facilitates the easy addition of cryptographic functions

package crypto

import (
	"github.com/go-sprout/sprout"
)

type CryptoHandler struct {
	*sprout.FunctionHandler // Embedding FunctionHandler for shared functionality
}

// NewCryptoHandler creates a new instance of CryptoHandler with an embedded FunctionHandler.
func NewCryptoHandler(embedFunctionhandler *sprout.FunctionHandler) *CryptoHandler {
	return &CryptoHandler{FunctionHandler: embedFunctionhandler}
}

// RegisterFunctions adds all crypto-related functions to the provided registry.
func (ch *CryptoHandler) RegisterFunctions(registry sprout.FunctionsRegistry) {
	sprout.AddFunctionToRegistry(registry, "bcrypt", ch.Bcrypt)
	sprout.AddFunctionToRegistry(registry, "htpasswd", ch.Htpasswd)
	sprout.AddFunctionToRegistry(registry, "derivePassword", ch.DerivePassword)
	sprout.AddFunctionToRegistry(registry, "generatePrivateKey", ch.GeneratePrivateKey)
	sprout.AddFunctionToRegistry(registry, "parsePrivateKeyPEM", ch.ParsePrivateKeyPEM)
	sprout.AddFunctionToRegistry(registry, "getPublicKey", ch.GetPublicKey)
    // Additional cryptographic functions can be added here...
}

Function Registry Implementation (registry.go in Sprout)

registry.go in the Sprout package manages the function registration, allowing for a structured way to include multiple handlers:

package sprout

import (
	template "text/template"
)

// FunctionsRegistry is an alias for template.FuncMap, which maps function names
// to functions. This registry is used to register all template functions.
type FunctionsRegistry = template.FuncMap

// FunctionRegistry is an interface that defines the method to register functions
// within a given FunctionsRegistry. This interface should be implemented by
// any component or handler that provides template functions.
type FunctionRegistry interface {
	// RegisterFunctions adds the provided functions into the given registry.
	RegisterFunctions(registry FunctionsRegistry)
}

// RegisterHandler registers a single FunctionRegistry implementation (e.g., a handler)
// into the FunctionHandler's internal function registry. This method allows for integrating
// additional functions into the template processing environment.
func (fh *FunctionHandler) RegisterHandler(handler FunctionRegistry) {
	handler.RegisterFunctions(fh.funcsRegistry)
}

// RegisterHandlers registers multiple FunctionRegistry implementations into the
// FunctionHandler's internal function registry. This method simplifies the process
// of adding multiple sets of functionalities into the template engine at once.
func (fh *FunctionHandler) RegisterHandlers(handlers ...FunctionRegistry) {
	for _, handler := range handlers {
		handler.RegisterFunctions(fh.funcsRegistry)
	}
}

// Registry retrieves the complete function registry that has been configured
// within this FunctionHandler. This registry is ready to be used with template engines
// that accept FuncMap, such as html/template or text/template.
//
// NOTE: This will replace the `FuncsMap()`, `TxtFuncMap()` and `HtmlFuncMap()` from sprig
func (fh *FunctionHandler) Registry() FunctionsRegistry {
	fh.registerAliases() // Ensure all aliases are processed before returning the registry
	return fh.funcsRegistry
}

// AddFunctionToRegistry adds a new function under the specified name to the given registry.
// If the function name already exists in the registry, this method does nothing to
// prevent accidental overwriting of existing registered functions.
func AddFunctionToRegistry(registry FunctionsRegistry, name string, function any) {
	if _, ok := registry[name]; ok {
		return // Prevent overwriting existing functions
	}
	registry[name] = function
}

The focus will be on the usability, modularity, and potential improvements or concerns related to the current implementation.

✅ Ease of Integration: New function handlers are integrated as standalone modules for each package.
✅ Performance Considerations: The import strategy enhances build-time performance and optimizes the final application's space usage. At runtime, unused functions are not loaded into RAM, conserving resources.
✅ Extensibility: Allows external contributors to create new packages without the risk of breaking the rest of the project, ensuring scalability.
✅ Design Pattern: Adopts a declarative and explicit pattern over ordered import logic, promoting clearer and more maintainable code.
✨ Clean: No more default crypto as well (Gift for @andreynering)

Community feedback are invaluable for refining this approach and ensuring it meets the diverse needs of developers using the Sprout library. 🌱💜

I am extending the feedback period by one more week. Following this, we will begin to split, implement, and document the project, starting on July 17, 2024.

See you grow !

[ripienaar]

Why does the crypto import not auto register itself in the registry on import then all that is needed is to import it. None of the other code.

[42atomys]

The most notable case involves the import of third-party packages in a package that renders multiple templates. An example will make it clearer:

// mypackage/render.go
package mypackage

import (
  "github.com/go-sprout/sprout"
  "github.com/go-sprout/sprout/hello"
  "github.com/go-sprout/sprout/crypto"
  "github.com/go-sprout/sprout/pkga"
  "github.com/go-sprout/sprout/pkgb"
)

func renderTemplateA(handler *sprout.FunctionHandler) {
  // Register handlers in the registry for use in templates
  handler.RegisterHandlers(crypto.NewCryptoHandler(handler))

  // Execute the template
  tpl := template.Must(
    template.New("base").Funcs(handler.Registry()).ParseGlob("*.tmpl"),
  )
}

func renderTemplateB(handler *sprout.FunctionHandler) {
  // Register handlers in the registry for use in templates
  handler.RegisterHandlers(pkga.NewAHandler(handler), pkgb.NewBHandler(handler))

  // Execute the template
  tpl := template.Must(
    template.New("base").Funcs(handler.Registry()).ParseGlob("*.tmpl"),
  )
}

func renderAll() {
  globalHandler := sprout.NewFunctionHandler(
    sprout.WithLogger(slogLogger),
    sprout.WithAlias("hello", "hi"),
  )

  globalHandler.RegisterHandlers(hello.NewHelloHandler(globalHandler))

  renderTemplateA(globalHandler)
  renderTemplateB(globalHandler)
}

Automatically registering a package upon import can be convenient in certain scenarios but problematic in others. Here are the key points:

1. Declarative Approach:

• A declarative approach provides clarity and control over the code's behavior.
• Explicitly registering handlers makes it clear which components are being used and where.

2. Configuration Sharing:

• Sharing a general configuration (e.g., FunctionHandler with a logger, error handler, etc.) with third-party packages becomes more complex if automatic registration is enforced.
• By manually registering packages, the user has full control over how and when configurations are shared.

3. User Choice:

• Allowing users to register packages declaratively empowers them to make informed decisions about their codebase.
• This flexibility ensures that the code remains modular and maintainable, especially in larger projects with multiple dependencies.

In summary, while automatic registration of packages upon import might seem convenient, it undermines the benefits of a declarative approach and complicates configuration management.
It is better to give users the choice to register packages explicitly, maintaining clarity and control over their code.

[ripienaar]

Again, I would love to see actual real world cases where these use cases and requirements came up.

To me, this is a show stopper, this project seem to HEAVILY optimise for the most complex way to solve any problem without any evidence that this is needed.

As a fork of sprig, this makes it a no go to me. I am probably done with this tbh.

[ripienaar]

It's important to keep in mind the ease of use and simplicity of sprig and the fact that the functions are just a map and how easy it is to manipulate that in just pure go, without all these Javaescue design patterns.

It has to be easy to use and get out of your way, the direction here is very much to get in your way as much as possible.

Now I appreciate that complexity is needed and often adds value - but thus far there is no evidence that anyone in sprig or here ever needed this and the ROI of this complexity is going to be net negative imo.

[42atomys]

Thank you for the thoughtful discussion on the registration mechanism for handlers. After carefully considering the pros and cons of both automatic and explicit registration, I believe that explicit registration aligns better with the principles and long-term goals of this project.

As a fork

As a fork we needs to be easy to use, and seamless from sprig. That the challenge : have a seamless transitions with a good future following vision, and go specification, for new projects and large projects.

The discussion between vision are done in the issue #1, I can understand for small project that an inconvenient, for larger projects, this is another thing.

Reasons for Choosing Explicit Registration

1. Clarity and Transparency: Explicit registration makes it clear which handlers are being registered and when. This improves code readability and makes it easier for new contributors to understand the system. As highlighted in the Effective Go guide, clarity and simplicity are fundamental principles of Go. Explicitly stating what is being registered enhances transparency and adheres to Go’s idiomatic practices .
2. Control and Maintainability: Explicit registration allows for fine-grained control over what gets registered, reducing the risk of unintentional registrations or conflicts. This is particularly important as the project grows and becomes more complex. The Go Language Specification emphasizes precise and controlled coding practices, which are best achieved through explicit declarations .
3. Debugging and Modularity: When issues arise, explicit registration makes it easier to trace the source of the problem. Additionally, it supports a more modular design, which is beneficial for maintaining and scaling the project. Go’s Effective Go also discusses the importance of modularity and clear code organization for maintainability and ease of debugging .

Addressing Concerns

I understand that some contributors find the automatic registration more convenient. Here are a few steps we can take to mitigate the perceived drawbacks of explicit registration:

1. Improved Documentation: We will enhance our documentation to include detailed guidelines and examples on how to register handlers explicitly. This should help reduce the learning curve and make the process more straightforward.
2. Helper Functions: We can provide utility functions to streamline the registration process. These functions will encapsulate common registration patterns, reducing boilerplate while maintaining the benefits of explicit control.
3. Feedback Loop: We will establish a channel for continuous feedback where contributors can share their experiences and suggest improvements. This will help us refine our approach and address any pain points promptly.

Conclusion

While automatic registration offers simplicity, the long-term benefits of explicit registration in terms of clarity, control, and maintainability make it the preferred approach for our project. I appreciate your understanding and cooperation as we implement this change. Let’s continue to work together to build a robust and maintainable codebase.

For further reading on Go’s best practices, I recommend the Effective Go document and the Go Language Specification

Thank you again for your contributions and ongoing support.

[42atomys]

PS: I’m not good with the idea of making auto-register and declarative ways works together to be everyone happy, but I still open to discuss with you and others people to have a bigger feedbacks loop

PS2: I think the best choose are to make two projects: sprout with new changes and go-sprout/sprig the fork without any changes

[42atomys]

UPDATE: A Huge pull request to migrate all stuffs are created #46

I will test the branch with my personal projects to test performance and devex, and also start a branch for documentation update 📜

[42atomys]

UPDATE: The registry system has released on v0.5.0 🎉 Thanks for all to have contributed to the RFC, reviews, and feedbacks 🌱 💜

If you have concerns don't hesitate to talk about on release discussion or open an issue !