Split PV typemap into mutable and immutable

[Leont]

This splits the current string typemap into a mutable and an immutable variant. This will make a bunch of XS code do the right thing with COW strings.

However, this can also break old-fashioned XS code whose arguments aren't marked const properly. As is evidenced by the fast that it broke 11 modules in core. This may break "a few things" on CPAN as well.

• This set of changes requires a perldelta entry, and it is not yet included.

[bulk88]

maybe rmv white space only changes on unrelated lines? makes future git blames confusing

[Leont]

The variables are aligned, I'm keeping that alignment here.

[bulk88]

Did you try a core smoke with a perl interp built with that linux-only nobody-uses-it hardware-RO-const-from-malloc SVPV COW buffers feature? I've never heard of anyone using it but it exists in core interp src code. The interp build flag feature sounds very good but that feature is always off (assert-like) AFAIK b/c of slow-ness of continuously asking the ring 0 kernel to flip your RWX bits MMU bits.

I made a branch but never finished it where I made sv = newSVpvs(""); made a POK COW, where ptr in SvPVX() WILL SEGV if you write to it. This saves many KBs but not MBs of memory, b/c you not keeping 100s/1000s/10000s of separate 16+8, 16+16 or on WinPerl64 16+16+24 long malloc buffers, with only 1 byte init-ted and 15 un-init/unused.

Remember, in many many places, sv = newSV();, newSV_type(SVt_PV); and sv = &PL_sv_no; are spiritual or actual fatal errors on a PP level and only sv = newSVpvs(""); can be written.

So currently there is P5P written/maint code (did a git blame) somewhere inside 5.41 code that does

sv = newSVpvs("");
pv = SvPV_nolen(sv);
pv[0] = 'f';
pv[1] = 'a';
pv[2] = 'i';
pv[3] = 'l';
pv[4] = '\0';

or maybe it pointlessly (code rot) does

sv = newSVpvs("");
pv = SvPV_nolen(sv);
pv[0] = '\0';

or

sv = newSVpvs("");
pv = SvPV_nolen(sv);
pv[SvCUR(sv)] = '\0';

So if some core P5P person commit that to a core .c interp blead somewhere in the last 10-20 years, very rarely its gonna appear, in CPAN XS or private XS b/c statistics and nobody is perfect. But its guaranteed it will appear again atleast 1x over the next 40 years. A dev could bikeshed a 1 byte malloc() block that SEGVs at str[1] = '\0'; is illegal and impossible because alignment to sizeof(void*), sizeof(double), or sizeof(SSE_REG). I believe the lawsuit at the TC about clause "suitable aligned for all types" is still being litigated.

In any case,

sv = newSVpvs("");
pv = SvPV_nolen(sv);
pv[0] = 'f';
pv[1] = 'a';
pv[2] = 'i';
pv[3] = 'l';
pv[4] = '\0';

sv = newSVpvs("");
pv = SvPV_nolen(sv);
pv[0] = 6;

sv = newSVpvs("");
pv = SvPV_nolen(sv);
pv[0] = '\0';

IMO all of these are invalid regarding Perl C. Ask for 0 bytes? You get 0 bytes! The person didn't execute SvGROW() or sv_grow().

sv = newSVpvs("");
pv = SvPV_nolen(sv);
pv[0] = 6;

is a CVE attempt or something b/c a not null-terminated SVPV was just introduced into the VM and will start to circulate, and I doubt the end user is creating a 1 byte long C struct to pass to a 3rd part clib that doesn't need a wasteful extra secret '\0' byte and call it an optimization attempt.

Last sentence is a BIG problem with Perl XS/CORE's API, when someone wants to store a 16 byte C struct in SvPVX(), and core's public API turns that into 32 byte long buffers, and wastes 16 bytes at time, over and over.

But back to this PR, changing CPAN XS public API, to start blindly executing SvPV_force_nolen() 50% or 90% of the time, may or may not, have disastrous pref consequences.

More research is needed with "average" or "typical" PP production quality code, and "average" or "typical" CPAN XS modules to see how pure/clean they (their authors) are about _const( and const char marking their code. 95% of CPAN XS authors in 2025 still have no idea what PERL_NO_GET_CONTEXT pTHX aTHX are and will start a fight with you that you are proposing to add a syntax error to their XS module.

SvPV_force_nolen() is really expensive if done by accident. And a 50%-75% chance of heap corruption if writing to buffer is intended correct behavior. You didn't do a bounds check! You didn't type SvGROW(). Why are writing or 2-way read/writing to SvPVX() with SvPV_force_nolen()?

There are few people crazy (me)/rare enough to type

pv = SvPV_force_nolen();
if(SvCUR() < 1234 || SvLEN() < 1234) 
    pv = sv_grow(1234);

In 5.41 git repo, there is ZERO, NO, NADA usage of macro SvLEN() in any .xs file. Besides there is a perf problem with

pv = SvPV_force_nolen();
if(SvCUR() < 1234 || SvLEN() < 1234) 
    pv = sv_grow(1234);

It really should be (not perfect ahead, please debate)

if(SvTHINKFIRST() || !SvPOK() || SvCUR() < 1234 || SvLEN() < 1234) 
    pv = sv_grow(1234);
else
   pv = SvPVX();

so libc realloc doesnt execute twice, just once.

So summary, some more thinking or answers, or opinion, are needed on topics

• are there perf consequences if any, if running actively maintained medium-quality CPAN XS code not written by first timers

• do we need to add fatal errors somewhere? And its a hard fatal error suddenly introduced in stable perl. And tell end users, "tough luck" when they argue

My code is correct. It does not have heap corruption. Bash didn't print process terminated with SIGSEGV. My module passes all unit tests. Perl 5 Porters broke my module. Send me a check if you want me to google the word valgrind and learn how to write code for that language I never heard of.

• refactor/cleanup/make more-portable the exotic unused SVPV COW bufs stored in HW RO pages (mprotect/VirtualProtect) feature, and make it a stable perl default-on?

[Leont]

Did you try a core smoke with a perl interp built with that linux-only nobody-uses-it hardware-RO-const-from-malloc SVPV COW buffers feature? I've never heard of anyone using it but it exists in core interp src code.

I know Father C wrote and used it when he worked on COW.

The interp build flag feature sounds very good but that feature is always off (assert-like) AFAIK b/c of slow-ness of continuously asking the ring 0 kernel to flip your RWX bits MMU bits.

I think it uses a spectacular amount of RAM because it needs a full page for every allocation, and really slow because that really trashes the MMU. Great for debugging, absolutely unusable in production.

There are few people crazy (me)/rare enough to type
…
In 5.41 git repo, there is ZERO, NO, NADA usage of macro SvLEN() in any .xs file.

There is the SvGROW() macro for that purpose.

[bulk88]

There is the SvGROW() macro for that purpose.

SvGROW() has alot of problems.

//////////////////
/* scalar SVs with SVp_POK */
#define SVppv_STATIC    0x40000000 /* PV is pointer to static const; must be set with SVf_IsCOW */
//////////////////
#define SVf_AMAGIC	0x10000000  /* has magical overloaded methods */
#define SVf_IsCOW	0x10000000  /* copy on write (shared hash key if SvLEN == 0) */
//////////////////
#define SVf_THINKFIRST	(SVf_READONLY|SVf_PROTECT|SVf_ROK|SVf_FAKE \
                        |SVs_RMG|SVf_IsCOW)
//////////////////
#define SvTHINKFIRST(sv)	(SvFLAGS(sv) & SVf_THINKFIRST)
/////////////////
#define SvIsCOW(sv)              (SvFLAGS(sv) & SVf_IsCOW)
#define SvIsCOW_on(sv)           (SvFLAGS(sv) |= SVf_IsCOW)
#define SvIsCOW_off(sv)          (SvFLAGS(sv) &= ~(SVf_IsCOW|SVppv_STATIC))
#define SvIsCOW_shared_hash(sv)  ((SvFLAGS(sv) & (SVf_IsCOW|SVppv_STATIC)) == (SVf_IsCOW) && SvLEN(sv) == 0)
#define SvIsCOW_static(sv)       ((SvFLAGS(sv) & (SVf_IsCOW|SVppv_STATIC)) == (SVf_IsCOW|SVppv_STATIC))
//////////////////////
#ifdef PERL_COPY_ON_WRITE
#   define SvCANCOW(sv)					    \
        (SvIsCOW(sv)					     \
         ? SvLEN(sv) ? CowREFCNT(sv) != SV_COW_REFCNT_MAX : 1 \
         : (SvFLAGS(sv) & CAN_COW_MASK) == CAN_COW_FLAGS       \
                            && SvCUR(sv)+1 < SvLEN(sv))
   /* Note: To allow 256 COW "copies", a refcnt of 0 means 1. */
#   define CowREFCNT(sv)	(*(U8 *)(SvPVX(sv)+SvLEN(sv)-1))
#   define SV_COW_REFCNT_MAX	nBIT_UMAX(sizeof(U8) * CHARBITS)
#   define CAN_COW_MASK	(SVf_POK|SVf_ROK|SVp_POK|SVf_FAKE| \
                         SVf_OOK|SVf_BREAK|SVf_READONLY|SVf_PROTECT)
#endif

#define CAN_COW_FLAGS	(SVp_POK|SVf_POK)
//////////////////////
#ifdef PERL_ANY_COW
# define SvGROW(sv,len) \
        (SvIsCOW(sv) || SvLEN(sv) < (len) ? sv_grow(sv,len) : SvPVX(sv))
#else
# define SvGROW(sv,len) (SvLEN(sv) < (len) ? sv_grow(sv,len) : SvPVX(sv))
#endif
///////////////////////////

SvGROW() doesn't look at these SVf_THINKFIRST-style flags SVf_READONLY|SVf_PROTECT|SVf_ROK|SVf_FAKE.

SvGROW() doesn't check SvTYPE() >=, instant SEGV if you pass a SV* with type SVt_NULL. I have to hand roll this on CPAN.

Passing SVt_IV/SVt_NV is 80% safe but does 20% random random heap corruption/bc SvCUR()/SvLEN() are uninitted bytes (to viewer) from a directly adjacent SV Body/head struct in the arena. Passing a SVt_IV/SVt_NV to SvGROW on pre 5.9.3 perl is a insta-crash.

Remember NC refactored SV heads from 3 members to the 4 we know now around 5.9.3.

SvGROW() on a SV SvROK_on, can go 2 different way, depending if the SV ROK is a SVt_IV bodyless, or has non-ghost propert zeroed out SvLEN and SvCUR fields b/c its a SVt_PVMG for some reason.

What do the SvLEN/SvCUR fields (SVt_PV to SVt_PVMG) even mean for a SvROK_on 😳?????!!!!!

SVf_READONLY|SVf_PROTECT ignored plan as day. Time to SvGROW(&PL_sv_yes,32);?

SVf_FAKE IDK what this flag does in core but SVf_THINKFIRST has it, SvGROW() does not.

////////////////////////////
#define SvPOK_pure_nogthink(sv) \
    ((SvFLAGS(sv) & (SVf_POK|SVf_IOK|SVf_NOK|SVf_ROK|SVpgv_GP|SVf_THINKFIRST|SVs_GMG)) == SVf_POK)

///////////////////////////
PERL_STATIC_INLINE char *
Perl_SvPV_helper(pTHX_      SV * const sv, STRLEN * const lp,const U32 flags, const PL_SvPVtype type,
                 char * (*non_trivial)(pTHX_ SV *, STRLEN * const, const U32),const bool or_null,const U32 return_flags
                )
{
    /* 'type' should be known at compile time, so this is reduced to a single
     * conditional at runtime */
    if (   (type == SvPVbyte_type_      && SvPOK_byte_nog(sv))
        || (type == SvPVforce_type_     && SvPOK_pure_nogthink(sv))
/////////////////////////////////////////////////
#define SvPV_force_flags_nolen(sv, flags)                                   \
    Perl_SvPV_helper(aTHX_ sv, NULL, flags, SvPVforce_type_,                \
                     Perl_sv_pvn_force_flags, FALSE, 0)
/////////////////////////////////////////////////
#define SvPV_force_nolen(sv)        SvPV_force_flags_nolen(sv, SV_GMAGIC)
/////////////////////////////////////////////////

So also SvPV_force_nolen(sv) will go down the heavy path for SVpgv_GP and SvGROW() doesnt check for SVpgv_GP or SvTYPE() SVt_PVGV

GP* and RV*are permanently conflicted with SVPV's SvPVX() field b/c all 3 use it with no other tricks available, other traicks like IV/NVs. NC in 2005 5.9.X, split macro SvRV() from SvIVX() , previously they were identical. Post 5.9 I can create a NV NOK, IV IOK and RV ROK tri-var if I wanted to. But you can't make a SvROK+SvPOK dual var (SHOOO!!! AMAGIC and overload.pm stringifier operator ) because they collide on ownership of SVPVX in the head.

There are a bunch of issues big small or non-issues with SvGROW(), but its definitely has edge cases and isnt a universal throw it at any SV* <= PVLV in the Perl VM and SvGROW() will be safe and do the right thing.

[tonycoz]

However, this can also break old-fashioned XS code whose arguments aren't marked const properly. As is evidenced by the fast that it broke 11 modules in core. This may break "a few things" on CPAN as well.

I like the change, but I'd expect it to break silly amounts of modules, especially old ones that aren't maintained any more.

[bulk88]

However, this can also break old-fashioned XS code whose arguments aren't marked const properly. As is evidenced by the fast that it broke 11 modules in core. This may break "a few things" on CPAN as well.

I like the change, but I'd expect it to break silly amounts of modules, especially old ones that aren't maintained any more.

what do you think of adding a XS/C side analog of https://perldoc.perl.org/warnings or the formal https://perldoc.perl.org/warnings PP API having warning options that DIRECTLY only affect execution of XSUBs/C funcs/machine code, not PP OP* bytecode?

Basically macros SvPV() and SvPV_nolen() become 1x self latching STDERR console warnings without

#define PERL_NO_GET_CONTENT_REVISION_2_MAX_VER 42
#include "EXTERN.h"
#include "perl.h"

?

I can sort of implement high-ish performance hardware RO/MMU controlled, bad-write protection COW for Win32/Win64 only if someone really wants or I find a reason to need it for personal reasons. The backend ntdll.dll Rtl*() funcs behind MS's HeapAlloc/HeapCreate, do have optional hardware read-only protection that can be flipped on and off 4KB aka HeapSize() aka msize() at a time through certain undocumented highish-level ntdll.dll Rtl*() features. Either by the minimally public API documented middle (2nd) flag arg, or dedicated Rtl*() function calls. Its not the MS public API "page heap" feature which gives a dedicated 4 KB VM block to a 1 byte long malloc()/HeapAlloc() request.

This PR also brings up big BIG criticism from certain former perl devs on how Perl SVPV COW in 2014, and Perl's HEK* in the Nicholas Clark/JHI era were implemented. The flaw is by keeping the constantly runtime ++ing --ing refcount integers, right next to the PV buffer, in address space, its impossible to implement hardware const/RO protection to PP-level/PP-lang/XS-level/C-leve constant SVPV/HE/HEK strings.

Twice in the last 15 years, forks of Perl, changed the current P5P HEK struct's I32 refcount field, to a POINTER to a I32 refcount. I think in my "SHEK" PR/branch which is still open right now, but is sorta out of date, ever since RCPV API was added in 2019/2020. In my "SHEK" branch I turned the current HEK's refcount logic, that if its refcount I32 -1, an unaligned 4 or 8 byte I32 * exists after the current U8 HEK_FLAGS() at the end of the PV buffer of the var length/VLA HEK struct.

Doing the above probably kills 90% the runtime RW->R->RW->R->RW kernel ring-0 mprotect calls, HEKs/SHEKs/GHEKs usually live for the rest of the perl process lifetime since they almost always represent the PP optree and *main::SomePkg things. Nobody unloads or deletes or tree shaked .pm or .xs modules dynamically at runtime once loaded even though nothing in P5 stops a user from doing that at runtime.

I have a hypothesis, that a very basic and very simple capture of the first 2 bytes and last 2 bytes, [some logic omited], of the PV buffer, inside the HEK or max 255 RC SVPV COW, then constant comparison and runtime asserting, of that U32, against PP visible first and last 2 bytes, could benchmark to 0 runtime overhead, but provide 99% of the same protection against accidentally bad XS modules on CPAN. But my hypothesis/proposal has 0% protection against malicious XS modules on CPAN but those are never a design consideration for any end user PP/C/XS code written for the Perl 5 interp b/c day 1 fundamental design reasons of the Perl 5 interp/lang reasons.

https://metacpan.org/pod/Safe is long since been declared obsolete and useless.

PP Taint probably too.

And https://perldoc.perl.org/perlsecpolicy#Feeding-untrusted-code-to-the-interpreter has (day 1?) said kernel ring-0 aka vmware aka ulimit aka docker aka WinKernel's ACLs/Job API are the only ways to deal with CVEs/bug bounty/blackhat PP or XS code.

Google V8's blackhat-proof guarantee and Chrome's W3C blackhat-proof guarantees are N/A for Perl ecosystem.

Remember NodeJS doesn't have that guarantee and the NodeJS interp the exact same "Arbitrary Machine Code Execution" exploit that Perl 5 and ISO C have 😇

Also remember Google V8's blackhat-proof guarantee requires CC build time disabling (compile out) of JIT/JS string eval/machine codegen inside the jail-ed V8 interp binary, and loading pre-compiled V8 bytecode/V8 machine code/V8 ELF .so'es from disk that were made in another full service V8 interpreter.

And Chrome doesn't even trust V8's blackhat proof-ness anyways and always runs unsafe WWW https:// .js src code in a super OS-level restricted 2nd (3rd/4th/etc) OS process with just a single unix domain socket doing RPC between 2 chrome.exe OS processes. Gotta go back to basic Jan 1 1970 C POSIX security design rules and HW MMU protection. Don't trust your human programmers regardless of resume and salary. Escaping the sandbox is a Tovalds or Intel problem, not a Google or P5P's problem. I agree.

A Perl interp core attempt at a high performance hardware RO MMU protected COW implementation should be tried and benchmarked, and see what happens, maybe first 2 bytes and last 2 bytes will work perfect. Maybe NtVirtualMemoryAdjust() and mprotect() take 2-9 micro seconds (20, 200, or 2000 hertz), not 200 or 900 microseconds (0.2/0.9 milliseconds, aka eternity) to flip the CPU's page table.

Maybe take a Firefox/Trident Edge 11 technique, and keep 2 separate mmap views or virtual address space views to Perl's "legacy" HEKs and SVPV max 255 COWs. This has 0 bytes of phy ram or page file overhead on Windows and Linux. Perl C API's RC++ RC-- macros know the secret, but macros SvPV() and SvPVX() and 100% of CPAN XS authors don't know the secret and all 3 instantly SEGV 100% of the time guaranteed.