New PPC: Attributes v2

[leonerd]

Abstract

Define a new way that attribute definitions can be introduced such that the parser can invoke the third-party custom logic they provide; and additionally permit them to be attached to more types of target than Perl currently allows.

---

Still currently in draft as I work out a few more minor questions, but now at the stage where it should be suitable for comments.

[happy-barney]

instead of ver it will be better to put it into registration function and struct, eg:

register_attribute_540 (struct Perl_Attribute_Definition_540 *);

that will allow to change structure definition along with version which will be easier to validate using C-compiler capabilities.

[happy-barney]

how to implement scenario where it make sense to define default value? for example attribute required, default value true

cannot parse be reused for that? (getting NULL when there is no value?)

[leonerd]

Easiest just done in the apply function by noting the lack of a (defined) value being passed in.

[happy-barney]

What about to define universal SV to hold any kind of information identified by U32 ?
Such value can be used as prefix in pad as well and defines implementation pattern for any kind of symbol extension (eg Named regexp patterns, in fact same thing I need for "independent symbol spaces")

[happy-barney]

Another possible context:

• regexp attributes

s
    : multiline
    : extended
    : non_capture
    : return
    ( )
    ( )
    ;

• use statement, for example:

use My::Almighty::Set::Of::Utils
   :attributes(import grammar of attributes to be imported)
   :pattern(C_Comment CPP_Comment Perl_Comment)
   :other_symbol_space(import grammar of attributes to be imported)
   ;

• require statement

require :module q (Foo);
require :relative q (file-to-require.pl);

• smartmatch expression

- given ($foo :using(Str_Or_Regex)) { ... }
if ($foo ~~ :using(Str_Or_Regex)) { ... }

• function invocation scope, for example:

select
    :table(foo)
    :join(bar ON (foo.id = bar.foo_id and bar.valid_from > $valid_from))
    :columns(foo.*)
    :where(foo.column BETWEEN $min AND $max)
;

# with alternative attribute syntax
select (
    := table   => foo
    := join    => bar ON (foo.id = bar.foo_id and bar.valid_from > $valid_from)
    := columns => foo.*
    := where   => foo.column BETWEEN $min AND $max
)

[leonerd]

These are all things to consider in the future, yes. The point of this particular proposal isn't to allow new syntax, but to create a better implementation base such that in future new syntax could be considered.

I've added words to that effect.

[happy-barney]

when parse is invoked, it knows context, so why not to defined enumeration of context identifiers and send it along with text, so it can return value aware of such context?

should this "enumeration be extendable, such mechanism can be reused by 3rd party extensions to declare new contexts for their use (along with core contexts) and use attribute handling provided by core

[happy-barney]

I'd suggest to define reusable (from core as well from 3rd party modules) syntax to declare additional symbols (taken from my "independent symbol spaces"). For attributes:

declare attribute Foo as
    on CODE   => { ... do something ... }
    on SCALAR => { ... do something ... }
    on ARRAY | HASH => { ... do something ... }
  ;

[mauke]

In general, a "function value" should be represented in C as a pair of (function pointer, context pointer) to enable closure-like behavior. So I would expect to see something more like

    void *parse_ctx;
    SV *(*parse)(pTHX_ void *ctx, SV *text);
    void *apply_ctx;
    void (*apply)(pTHX_ void *ctx, SV *target, SV *attrvalue);

[leonerd]

Ahyes, some sort of void *ctx pointer in the functions would be useful, though I don't think its value should come from the struct PerlAttributeDefinition itself. If following similar patterns from many of my XS modules, that struct would likely be static const in the .xs file itself. A void *ctx would probably be supplied some kind of registration function, or from the SV in the scope pad, or somesuch.

[leonerd]

I believe I've added this now in the latest wording. The pointer value is supplied as an argument to newSVinternal_attribute_definition(), and the callback functions in the struct all extended to use it.

[mauke]

What does parse return? If parse isn't supposed to modify the text parameter in place, shouldn't it be marked const?

[happy-barney]

@mauke good point ... in fact, shouldn't parse be rather Perl_keyword_plugin_t ?

[leonerd]

Ahyes, good idea. It could just return the SV itself or it could return some new one. Would have to think a bit about refcounts, but yes it shouldn't modify the one passed in.

[mauke]

I think for Perl-level operations it makes much more sense to have the opposite of this proposal: Attributes that apply to Perl entities (variables, subroutines), not parser/interpreter artifacts (GVs, pad slots, CVs, protosubs). Since most of these entities (lexical variables, anonymous closures) only exist at runtime, it would mean providing a runtime hook that is invoked when the entity is instantiated.

But that's a very different proposal.

[leonerd]

Since most of these entities (lexical variables, anonymous closures) only exist at runtime, it would mean providing a runtime hook that is invoked when the entity is instantiated.

But that's a very different proposal.

Well, quite ;) That's the "Hooks" part of the idea, which is separate enough (and rather more vague) to warrant its own separate PPC doc. The attributes here aren't directly related, but they're likely useful when combined.

[leonerd]

Many small updates clarifying wording, narrowing down a few open issues, adding more specification in terms of new API support functions and behaviours.

[happy-barney]

what about change parse to custom_grammar returning something that may or may not be OP * ? with following behaviour:

• when perl starts:
  • push grammar stack unbalanced grammar
  • push grammar stack perl grammar
• when encountering places where switch to known custom grammar is needed
  • push grammar stack known custom grammar (custom grammar can be perl as well)

there already is similar mechanism in toke.c - Perl_lex_start. This mechanism will just provide generic API to that plus it will provide Perl grammar itself as custom grammar.

That itself allows to implement use VERSION as custom grammar which may allow to introduce changes will require incompatible changes to perly.y.

Example: hypothetical import syntax may reuse this attribute mechanism for itself (by introducing custom attribute context)

use builtin
  :functions (
    foo :as (bar)
    baz
  )
  :attributes (magic)
  ;

[happy-barney]

what about replacing this with some set (linked list? tree? ...) of structures:

struct attribute_apply {
    U32 attribute_context_id;
    Attribute_Apply_Function apply;
};

[leonerd]

There's not been any new comments on this for a while, so I'm going to declare it ready-for-review. We'll see how it looks.