-
-
Notifications
You must be signed in to change notification settings - Fork 79
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
json_body should not read entire request BODY #1730
Comments
I was patching this like this:
But as @mauritsvanrees pointed out: "We should not read the complete request BODY in memory." and so that's probably not helping. I wasn't able to come up with a test that would actually push a request through Zope's machinery to address this issue. I also think that a "fix" like the one I show here would not really eliminate the D.O.S. threat that potentially was the reason for introducing the limits in the first place. The only "good" fix would be to have every request body be a stream in all cases, but I wonder if that's feasible. Can we use a data structure that is a stream and behaves like "in memory data"? |
Additionally we "fixed" the problem by upping the limits in zope.conf like this:
|
... maybe at one point we will have to live with "huge" data in memory. |
Can this problem cause high memory usage on portals that upload large files? |
It might be possible to create a new function which returns a specific key/value from a JSON document without reading the entire structure into memory, using something like https://pypi.org/project/ijson/ -- but you need to look at the specific places where json_body is used to see if that is helpful. For the use case of file uploads in particular, I think the current approach that volto uses (base64-encoded file data in a JSON body) is not optimal. I think we should explore adding support for a different serialization format in plone.restapi, with a multipart/form-data body that includes both the content as JSON and separate files as attachments (and some convention for referencing the files from within the JSON structure). |
So this line is bad:
With Zope 5.8.4+ this fails when the request BODY is larger than 1MB.
See discussion in plone/Products.CMFPlone#3848 and zopefoundation/Zope#1142.
See my summary of the history and the steps taken so far.
Temporarily fixed in #1729 by disabling a Zope form memory limit check.
The text was updated successfully, but these errors were encountered: