Add non wagtail URLs to sitemap.xml

[kesara]

#135 added sitemap.xml to the wagtail website.
This only contains the URLs that are generated by the wagtail website.
We should expose other URLs that are on www.ietf.org that is not generated by wagtail.
Examples:

• https://www.ietf.org/id/
• https://www.ietf.org/archive/id/ (be careful - that one will exhaust your browser's resources if you just open it)
• https://www.ietf.org/ietf-ftp/

[ghwood]

Good point to raise, I'd be interested in others thoughts about this, and particularly these examples. A few comments below about these particular examples but my overall thought is that it would be useful to have a more general discussion about our SEO strategy (particularly related to I-Ds and RFCs).

For example, thinking about SEO for things like Datatracker (including publishing a more comprehensive sitemap.xml) could be useful to help people find things like meeting proceedings (though my tests returned good results).

For IETF-FTP, that's on track to be retired (eventually):
https://mailarchive.ietf.org/arch/msg/tools-development/dI1DQXo-oCSMJcKg4Y0bte328I0/

For I-Ds, I know there's been some consideration of the multiple publication points, and I'm thinking we might want to nudge folks to Datatracker?

[rjsparks]

Read that message about IETF-FTP and particularly RFC9141 more carefully. We are removing the ftp service, but 9141 enshrined that URL path for the http service:

3.26. Generic Guidance

If any other RFC not explicitly mentioned in an earlier section
contains a reference of the form "ftp://ftp.ietf.org/", the
reference MUST be replaced with a URI of the form
"https://www.ietf.org/ietf-ftp/".

[ghwood]

Yep, I misstated that point so thanks for the correction. I do think it is still worth considering with some care to where we'd like to nudge folks for the items under https://www.ietf.org/ietf-ftp/

[rjsparks]

I agree, and if possible nudge them out from there. I would, in the very long game, like to get rid of that url segment.