Add an option to the add-on store, allowing users to customize the mirror URL of metadata

[cary-rowen]

Is your feature request related to a problem? Please describe.

In China, due to the government firewall or some other reason, NVDA users cannot use the add-on store, get metadata, and cannot download add-ons.
I don't know of any other potential users in regions that are affected by this.

Describe the solution you'd like

Add an option in NVDA that allows users to set the metadata mirror URL of the add-on store, and the Chinese team can maintain this mirror. Just like it currently does for the add-on updater.

Describe alternatives you've considered

Use Monkey Patch for getAddonStoreURL in source/addonStore/network.py.

Additional context

None

[nvdaes]

@cary-rowen , in case this takes some time, since they are add-ons not added to the addons.nvda-project.org website, as you may know, websites have been created to fetch/present add-ons available in the store.
I created a webpage for this when I was requested to make preliminary tests. My webpage has a form at the bottom (you can move there with d since this is a landmark), and there you can upload a downloaded add-on and check the sha256 to ensure that this is not modified. I understand that this is not ideal, but it maybe useful as an alternative. API version of presented add-ons is latest.

https://nvdaes.github.io/nvdastore

[cary-rowen]

Thanks @nvdaes

I can access all services of NVAccess and Github with no problem using VPN.
Also, I can also test the latest alpha version.
Just from the perspective of the Chinese community, I hope to have such a feature to solve the problems faced by Chinese users.

[seanbudd]

Hi @cary-rowen - are you certain of this?

The following tool seems to suggest that nvaccess.org can be accessed just fine from China:
https://viewdns.info/chinesefirewall/?domain=nvaccess.org

Other similar tools have confirmed this as well.

[wmhn1872265132]

Whether you can successfully access nvaccess.org and github is uncertain, and even if you can access related websites, the access speed is not very satisfactory. If you need to download a large amount of data, it will consume a lot of time

[cary-rowen]

Hi @seanbudd
Not all carrier networks can access NVAccess or download files from Github. The most serious problem may be users using China Mobile.
Worst of all is the inability to download files from github, manifested as the add-on being in a "downloading" state for a long time.
This is a very common phenomenon, and most of the time, NVDA's update check cannot be used in China.
In order to solve this problem, we specially created a mirror (nvaccess.mirror.nvdadr.com) of NVAccess. I once sent an email to [email protected] to explain this.

The following add-on apply this mirror source:
https://github.com/nvdacn/NVDAUpdateMirror

[seanbudd]

If NVDA users in China already have to use an add-on to use a mirror for update check, wouldn't adding a general mirror for nvaccess.org be more appropriate than just for the add-on store?

It's difficult to say whether supporting mirrors is in scope of NVDA, or should be performed by add-ons only.

[cary-rowen]

If NVDA users in China already have to use an add-on to use a mirror for update check, wouldn't adding a general mirror for nvaccess.org be more appropriate than just for the add-on store?

Yes, this is the ideal solution.

Rather than asking users to go to a specific site to download a separate mirror add-on, it would be better if there was an option to fill in the mirror URL in NVDA's advanced settings, or somewhere more appropriate.

[XLTechie]

It's difficult to say whether supporting mirrors is in scope of NVDA, or should be performed by add-ons only.

While I have no comment on that, I will say that allowing the metadata URL to be easily changed by an add-on without monkeypatching, could make life easier for add-on maintainers. Both in these situations, where China and Iran, to give two examples who have been known to block all kinds of things in the past, might need add-ons to patch the URL; and for alternative add-on catalogs. I would prefer there weren't alternative add-on catalogs, but we know that there are, for example several with language specific add-ons only in them. Allowing a catalog specific add-on to easily patch the URL from which data is fetched, could open certain options. Either an extensionPoint Filter, or a globally available setter for the URL, would solve this with minimal maintenance requirements on the NV Access side..

[cary-rowen]

Either an extensionPoint Filter, or a globally available setter for the URL,
would solve this with minimal maintenance requirements on the NV Access side..

This is also nice and will be friendlier for add-on maintainers too.

[nvdaes]

Imo, seems reasonable to allow URLS mirroring the whole NV Access data available for users, not just the store, to make easier NVDa updates. This is absolutely needed for user of certain countries. In the other hand, I don"t thing that the fact that different catalogs for add-ons which may not be needed, given the fact that sending add-ons to the store can be done easily, I don"t thing that this should be prioritized at all. This maybe even harmful for gattering stats from NV Access, and may have the effect that catalogs are created to promote certain servers without considering the efforz made for the store. But please don"t think that I"m suggesting that providing a way for users that cannot access data ue to restrictions in countries shouldn"t be prioritized. My concern is about people who may create their own catalogs for their own prestige or benefit, at least hypotetically. This cannot be avoided, but certain policies maybe provided, or at least recommendations to use the NV Access catalog when possible.

[seanbudd]

We can improve the API for this in the meantime, before considering including it in NVDA.

[cary-rowen]

We can improve the API for this in the meantime, before considering including it in NVDA.

that would be great, thanks

[cary-rowen]

Hello,

Monkey Patching addonStore.dataManager.getAddonStoreURL with addons is complicated by NVDA's pre-caching of addon store metadata.
Because "initializing addonStore data manager" happens before the addon system is initialized.
If this issue is not fixed in 2023.2 then users in certain regions will not be able to use the add-on store.

The add-on I developed for this:
https://github.com/nvdacn/NVDAUpdateMirror

@seanbudd
Do you have any suggestions for this?

Thanks

[seanbudd]

I think we should create a new centralized class like NVDAState.WritePaths, URLs. It seems like a lot of URLs in NVDA should be centralized. Some are quite dated, e.g. using http not https. The NVDA website in the NVDA menu links to http://www.nvda-project.org!

[XLTechie]

@seanbudd I may tackle that one if you like. I did some work in add-on updater that might be appropriate to port across for an URLs class.

[seanbudd]

Sure thing @XLTechie - perhaps it should go in utils?

[XLTechie]

@seanbudd I was going to do it as its own base module, but if you would rather it in utils, it doesn't matter to me.

[seanbudd]

I think its own module in utils would be ideal

[cary-rowen]

Has this been postponed to 2023.3? It means that users in some regions in 2023.2 cannot use the add-on store.

[XLTechie]

@cary-rowen asked:

Has this been postponed to 2023.3? It means that users in some regions in 2023.2 cannot use the add-on store.

I'm sorry. I've been too busy with work recently to finish my part of this. I'll see what I can do in the first half of this week. I had one or two bugs to work out, but it was close to completion.

[seanbudd]

Unfortunately solving the original issue is a major piece of work and couldn't fit into this development cycle.
Our development cycles rarely push out fully complete features, rather an incremental approach aiming for a stable release.
We'd like to address this problem fully in 2023.3.

The current workaround solution for mirrors is handled by add-ons.
The request for a more stable add-on API in 2023.2 was considered a nice-to-have, as the add-on store API is unstable, and is likely to change in 2023.3. Even implementing a good API for this is issue is complex, i.e. #15073.
This PR might not make the cut for the first beta, but may be able to go into a subsequent beta release.

Some possible workarounds
A workaround for add-on authors is to monkeypatch the private function.
You should import and monkeypatch only if the NVDA release is 2023.2, as this issue will be fixed in 2023.3.
This should prevent compatibility issues in future releases, and handle the temporary issue with 2023.2.

There are also existing website mirrors:

• https://nvda.store/
• https://nvda.es/files/get.php

[cary-rowen]

@seanbudd
Thanks for explaining this.
As you said, if the problem cannot be completely solved in 2023.2, I hope to use MonkeyPatching to patch the private function, but the current problems encountered are as follows, can you give some feasible suggestions?

Monkey Patching addonStore.dataManager.getAddonStoreURL with addons is complicated by NVDA's pre-caching of addon store metadata.
Because "initializing addonStore data manager" happens before the addon system is initialized.

Thanks

[seanbudd]

Is this an issue because it causes the first attempt to fetch from the NV access server rather than the mirror?
Or is the complication something else? Does it prevent the monkeypatch from occurring at all?

[cary-rowen]

Is this an issue because it causes the first attempt to fetch from the NV access server rather than the mirror?

Yes, that's the problem.

[seanbudd]

Could you explain what the behaviour is when monkey patching the add-on? My understanding is that even if the first attempt fails, subsequent fetches to the add-on store should succeed given a monkeypatched mirror

[cary-rowen]

Hi, @seanbudd
Sorry for not explaining clearly in the previous comment:

There are two possible cases that might occur before the monkey-patching add-on is loaded:

1. NVDA fetches the add-on metadata successfully from the NV Access server.
2. NVDA can't fetch the metadata from the NV Access server due to issues like the Great Firewall (GFW) or other network problems.

If the first case happens and the monkey-patching add-on loads successfully, NVDA won't try to fetch the metadata again when the user opens the add-on store. Users can see the available add-ons, but they can't download them because the download URL in the cached metadata isn't mirrored.

In the second case, if NVDA can't load the metadata during initialization, there's no cache. When the user opens the add-on store, NVDA will use the mirror server to fetch the metadata successfully. Here, the download URL in the metadata is from the mirrored address, not the Github address. This means users can use the add-on store without any problems.

It seems to me that this is the problem currently encountered.

Thanks

[seanbudd]

To handle the first case, you should be able to fix this by monkey patching the cacheHash url.
Your mirror should create an independent cache hash every time you update your views.
You could do this by re-hashing the value returned by our endpoint.

[cary-rowen]

Your mirror should create an independent cache hash every time you update your views.
You could do this by re-hashing the value returned by our endpoint.

oh, Thanks, What is the generation algorithm of this cache hash? How often should I update the cache hash?

[seanbudd]

The hash returned by our endpoint matches the latest commit of the views branch https://github.com/nvaccess/addon-datastore/commits/views

Ideally, your hash should always return a value different to NV Access's and should be updated whenever we update our hash.

You have a few options for generating the hash:

• mirroring the endpoint directly. This would be a good solution if the mirrored data was identical, but the data is not. Using this solution would mean that the add-on download bug would persist. As the mirrored data is different, we want your hash to never match an NV Access hash.
• hashing the value returned by the endpoint directly. This is the best solution in my opinion. To do this, simple take the hash of what the NV Access endpoint returns. The value is then updated whenever the NV access end point is updated, and the value is unique to NV access's hash.
• updating the hash every X minutes.
  How often do you update the mirrored data for your endpoint? Is it always up to date?
  You could update your hash value every X minutes, either matching when you update your data, or at an interval you find ideal.
  This means an NVDA user will only get the latest data if it has been X minutes since they last opened the store.
• If you don't mind your users being able to easily spam your endpoint for requests, you could return a random value each time your hash endpoint is fetched. Please don't do this if each time your endpoint is fetched, it fetches from NV Access's endpoint. Only do this if add-on data is cached by your own mirror.

[seanbudd]

When adding the mirror URL, an option should be added to set it to the URL to None to disable all telemetry