Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

HTTP POST request to compute times out #7

Open
calebwin opened this issue Aug 4, 2021 · 9 comments
Open

HTTP POST request to compute times out #7

calebwin opened this issue Aug 4, 2021 · 9 comments
Labels
banyan-jl Concerning Banyan.jl bug Something isn't working

Comments

@calebwin
Copy link
Contributor

calebwin commented Aug 4, 2021

The returned error message:

Basic usage of BanyanDataFrames: Error During Test at /home/calebwin/Projects/banyan-julia/BanyanDataFrames/test/test_groupby_agg.jl:1
  Got exception outside of a @test
  IOError(Base.IOError("read: connection timed out (ETIMEDOUT)", -110) during request(https://hcohsbhhzf.execute-api.us-west-2.amazonaws.com/dev/evaluate))
  
  Stacktrace:
    [1] wait_readnb(x::Sockets.TCPSocket, nb::Int64)
      @ Base ./stream.jl:394
    [2] eof(s::Sockets.TCPSocket)
      @ Base ./stream.jl:106
    [3] wait_for_encrypted_data(ctx::MbedTLS.SSLContext)
      @ MbedTLS ~/.julia/packages/MbedTLS/4YY6E/src/ssl.jl:374
    [4] wait_for_decrypted_data(ctx::MbedTLS.SSLContext)
      @ MbedTLS ~/.julia/packages/MbedTLS/4YY6E/src/ssl.jl:294
    [5] eof
      @ ~/.julia/packages/MbedTLS/4YY6E/src/ssl.jl:186 [inlined]
    [6] read_to_buffer(t::HTTP.ConnectionPool.Transaction{MbedTLS.SSLContext}, sizehint::Int64)
      @ HTTP.ConnectionPool ~/.julia/packages/HTTP/IAI92/src/ConnectionPool.jl:251
    [7] readuntil(t::HTTP.ConnectionPool.Transaction{MbedTLS.SSLContext}, f::Function, sizehint::Int64)
      @ HTTP.ConnectionPool ~/.julia/packages/HTTP/IAI92/src/ConnectionPool.jl:271
    [8] readuntil
      @ ~/.julia/packages/HTTP/IAI92/src/ConnectionPool.jl:269 [inlined]
    [9] readheaders
      @ ~/.julia/packages/HTTP/IAI92/src/Messages.jl:471 [inlined]
   [10] startread(http::HTTP.Streams.Stream{HTTP.Messages.Response, HTTP.ConnectionPool.Transaction{MbedTLS.SSLContext}})
      @ HTTP.Streams ~/.julia/packages/HTTP/IAI92/src/Streams.jl:155
   [11] macro expansion
      @ ~/.julia/packages/HTTP/IAI92/src/StreamRequest.jl:67 [inlined]
   [12] macro expansion
      @ ./task.jl:387 [inlined]
   [13] request(::Type{HTTP.StreamRequest.StreamLayer{Union{}}}, io::HTTP.ConnectionPool.Transaction{MbedTLS.SSLContext}, req::HTTP.Messages.Request, body::String; reached_redirect_limit::Bool, response_stream::Nothing, iofunction::Nothing, verbose::Int64, kw::Base.Iterators.Pairs{Union{}, Union{}, Tuple{}, NamedTuple{(), Tuple{}}})
      @ HTTP.StreamRequest ~/.julia/packages/HTTP/IAI92/src/StreamRequest.jl:57
   [14] request(::Type{HTTP.ConnectionRequest.ConnectionPoolLayer{HTTP.StreamRequest.StreamLayer{Union{}}}}, url::HTTP.URIs.URI, req::HTTP.Messages.Request, body::String; proxy::Nothing, socket_type::Type, reuse_limit::Int64, kw::Base.Iterators.Pairs{Symbol, Union{Nothing, Bool}, Tuple{Symbol, Symbol}, NamedTuple{(:iofunction, :reached_redirect_limit), Tuple{Nothing, Bool}}})
      @ HTTP.ConnectionRequest ~/.julia/packages/HTTP/IAI92/src/ConnectionRequest.jl:96
   [15] request(::Type{HTTP.ExceptionRequest.ExceptionLayer{HTTP.ConnectionRequest.ConnectionPoolLayer{HTTP.StreamRequest.StreamLayer{Union{}}}}}, ::HTTP.URIs.URI, ::Vararg{Any, N} where N; kw::Base.Iterators.Pairs{Symbol, Union{Nothing, Bool}, Tuple{Symbol, Symbol}, NamedTuple{(:iofunction, :reached_redirect_limit), Tuple{Nothing, Bool}}})
      @ HTTP.ExceptionRequest ~/.julia/packages/HTTP/IAI92/src/ExceptionRequest.jl:19
   [16] (::Base.var"#70#72"{Base.var"#70#71#73"{ExponentialBackOff, HTTP.RetryRequest.var"#2#3"{Bool, HTTP.Messages.Request}, typeof(HTTP.request)}})(::Type, ::Vararg{Any, N} where N; kwargs::Base.Iterators.Pairs{Symbol, Union{Nothing, Bool}, Tuple{Symbol, Symbol}, NamedTuple{(:iofunction, :reached_redirect_limit), Tuple{Nothing, Bool}}})
      @ Base ./error.jl:288
   [17] #request#1
      @ ~/.julia/packages/HTTP/IAI92/src/RetryRequest.jl:44 [inlined]
   [18] request(::Type{HTTP.MessageRequest.MessageLayer{HTTP.RetryRequest.RetryLayer{HTTP.ExceptionRequest.ExceptionLayer{HTTP.ConnectionRequest.ConnectionPoolLayer{HTTP.StreamRequest.StreamLayer{Union{}}}}}}}, method::String, url::HTTP.URIs.URI, headers::Base.Vector{Pair{SubString{String}, SubString{String}}}, body::String; http_version::VersionNumber, target::String, parent::Nothing, iofunction::Nothing, kw::Base.Iterators.Pairs{Symbol, Bool, Tuple{Symbol}, NamedTuple{(:reached_redirect_limit,), Tuple{Bool}}})
      @ HTTP.MessageRequest ~/.julia/packages/HTTP/IAI92/src/MessageRequest.jl:51
   [19] request(::Type{HTTP.BasicAuthRequest.BasicAuthLayer{HTTP.MessageRequest.MessageLayer{HTTP.RetryRequest.RetryLayer{HTTP.ExceptionRequest.ExceptionLayer{HTTP.ConnectionRequest.ConnectionPoolLayer{HTTP.StreamRequest.StreamLayer{Union{}}}}}}}}, method::String, url::HTTP.URIs.URI, headers::Base.Vector{Pair{SubString{String}, SubString{String}}}, body::String; kw::Base.Iterators.Pairs{Symbol, Bool, Tuple{Symbol}, NamedTuple{(:reached_redirect_limit,), Tuple{Bool}}})
      @ HTTP.BasicAuthRequest ~/.julia/packages/HTTP/IAI92/src/BasicAuthRequest.jl:28
   [20] request(::Type{HTTP.RedirectRequest.RedirectLayer{HTTP.BasicAuthRequest.BasicAuthLayer{HTTP.MessageRequest.MessageLayer{HTTP.RetryRequest.RetryLayer{HTTP.ExceptionRequest.ExceptionLayer{HTTP.ConnectionRequest.ConnectionPoolLayer{HTTP.StreamRequest.StreamLayer{Union{}}}}}}}}}, method::String, url::HTTP.URIs.URI, headers::Base.Vector{Pair{SubString{String}, SubString{String}}}, body::String; redirect_limit::Int64, forwardheaders::Bool, kw::Base.Iterators.Pairs{Union{}, Union{}, Tuple{}, NamedTuple{(), Tuple{}}})
      @ HTTP.RedirectRequest ~/.julia/packages/HTTP/IAI92/src/RedirectRequest.jl:24
   [21] request(::Type{HTTP.RedirectRequest.RedirectLayer{HTTP.BasicAuthRequest.BasicAuthLayer{HTTP.MessageRequest.MessageLayer{HTTP.RetryRequest.RetryLayer{HTTP.ExceptionRequest.ExceptionLayer{HTTP.ConnectionRequest.ConnectionPoolLayer{HTTP.StreamRequest.StreamLayer{Union{}}}}}}}}}, method::String, url::HTTP.URIs.URI, headers::Base.Vector{Pair{SubString{String}, SubString{String}}}, body::String)
      @ HTTP.RedirectRequest ~/.julia/packages/HTTP/IAI92/src/RedirectRequest.jl:21
   [22] request(method::String, url::String, h::Tuple{Tuple{String, String}, Tuple{String, String}}, b::String; headers::Tuple{Tuple{String, String}, Tuple{String, String}}, body::String, query::Nothing, kw::Base.Iterators.Pairs{Union{}, Union{}, Tuple{}, NamedTuple{(), Tuple{}}})
      @ HTTP ~/.julia/packages/HTTP/IAI92/src/HTTP.jl:314
   [23] request
      @ ~/.julia/packages/HTTP/IAI92/src/HTTP.jl:314 [inlined]
   [24] #post#14
      @ ~/.julia/packages/HTTP/IAI92/src/HTTP.jl:405 [inlined]
   [25] post
      @ ~/.julia/packages/HTTP/IAI92/src/HTTP.jl:405 [inlined]
   [26] send_request_get_response(method::Symbol, content::Dict{String, Any})
      @ Banyan ~/Projects/banyan-julia/Banyan/src/utils.jl:340
   [27] send_evaluation(value_id::String, job_id::String)
      @ Banyan ~/Projects/banyan-julia/Banyan/src/requests.jl:283
      ```
This fails on the client side although the last call to a backend service finished in <5s. So its possible that the client machine lost Internet connection or something else interrupted the communication.
@calebwin calebwin added bug Something isn't working banyan-jl Concerning Banyan.jl labels Aug 4, 2021
@calebwin calebwin mentioned this issue Aug 4, 2021
Closed
@calebwin
Copy link
Contributor Author

calebwin commented Aug 4, 2021

This happens rarely and was hard to reproduce.

@calebwin
Copy link
Contributor Author

calebwin commented Aug 4, 2021

This could be resolved by retrying calls to idempotent backend functionality.

@calebwin
Copy link
Contributor Author

calebwin commented Aug 9, 2021

On another instance, this occurred because somehow the job was being destroyed at the same time as a call to evaluate:

Basic data analytics on a small dataset: Error During Test at /home/calebwin/Projects/banyan-julia/BanyanDataFrames/test/test_small_dataset.jl:1
  Got exception outside of a @test
  "Invalid job_id."
  Stacktrace:
    [1] send_request_get_response(method::Symbol, content::Dict{String, Any})
      @ Banyan ~/Projects/banyan-julia/Banyan/src/utils.jl:355
    [2] send_evaluation(value_id::String, job_id::String)
      @ Banyan ~/Projects/banyan-julia/Banyan/src/requests.jl:287
    [3] compute(fut::Future)
      @ Banyan ~/Projects/banyan-julia/Banyan/src/requests.jl:192
    [4] collect(fut::DataFrame)
      @ Banyan ~/Projects/banyan-julia/Banyan/src/requests.jl:328
    [5] (::var"#15#23")(job::String)
      @ Main ~/Projects/banyan-julia/BanyanDataFrames/test/test_small_dataset.jl:16
    [6] (::var"#3#6"{var"#15#23"})(j::String)
      @ Main ~/Projects/banyan-julia/BanyanDataFrames/test/runtests.jl:77
    [7] with_job(f::var"#3#6"{var"#15#23"}; kwargs::Base.Iterators.Pairs{Symbol, Any, Tuple{Symbol, Symbol}, NamedTuple{(:job, :destroy_job_on_exit), Tuple{String, Bool}}})
      @ Banyan ~/Projects/banyan-julia/Banyan/src/jobs.jl:258
    [8] run_with_job(test_fn::var"#15#23", name::String)
      @ Main ~/Projects/banyan-julia/BanyanDataFrames/test/runtests.jl:72
    [9] macro expansion
      @ ~/Projects/banyan-julia/BanyanDataFrames/test/test_small_dataset.jl:2 [inlined]
   [10] macro expansion
      @ /buildworker/worker/package_linux64/build/usr/share/julia/stdlib/v1.6/Test/src/Test.jl:1151 [inlined]
   [11] top-level scope
      @ ~/Projects/banyan-julia/BanyanDataFrames/test/test_small_dataset.jl:2
   [12] include
      @ ./client.jl:444 [inlined]
   [13] include_tests_to_run
      @ ~/Projects/banyan-julia/BanyanDataFrames/test/runtests.jl:14 [inlined]
   [14] include_all_tests()
      @ Main ~/Projects/banyan-julia/BanyanDataFrames/test/runtests.jl:99
   [15] (::var"#9#10")(j::String)
      @ Main ~/Projects/banyan-julia/BanyanDataFrames/test/runtests.jl:114
   [16] with_job(f::var"#9#10"; kwargs::Base.Iterators.Pairs{Symbol, String, Tuple{Symbol}, NamedTuple{(:job,), Tuple{String}}})
      @ Banyan ~/Projects/banyan-julia/Banyan/src/jobs.jl:258
   [17] top-level scope
      @ ~/Projects/banyan-julia/BanyanDataFrames/test/runtests.jl:104
   [18] include(fname::String)
      @ Base.MainInclude ./client.jl:444
   [19] top-level scope
      @ none:6
   [20] eval
      @ ./boot.jl:360 [inlined]
   [21] exec_options(opts::Base.JLOptions)
      @ Base ./client.jl:261
   [22] _start()
      @ Base ./client.jl:485
  
  caused by: HTTP.ExceptionRequest.StatusError(500, "POST", "/dev/evaluate", HTTP.Messages.Response:
  """
  HTTP/1.1 500 Internal Server Error
  Date: Mon, 09 Aug 2021 01:17:55 GMT
  Content-Type: application/json
  Content-Length: 17
  Connection: keep-alive
  x-amzn-RequestId: a1b3d12f-a952-4a92-b0e1-db0269799484
  Access-Control-Allow-Origin: *
  Access-Control-Allow-Headers: Content-Type,X-Amz-Date,Authorization,X-Api-Key,X-Amz-Security-Token
  x-amz-apigw-id: DxlKXE3XPHcFjvg=
  Access-Control-Allow-Methods: OPTIONS,POST
  X-Requested-With: *
  X-Amzn-Trace-Id: Root=1-61108242-1e2accc30d3d055145422ea4;Sampled=0
  Access-Control-Allow-Credentials: true
  
  "Invalid job_id."""")
  Stacktrace:
    [1] request(::Type{HTTP.ExceptionRequest.ExceptionLayer{HTTP.ConnectionRequest.ConnectionPoolLayer{HTTP.StreamRequest.StreamLayer{Union{}}}}}, ::HTTP.URIs.URI, ::Vararg{Any, N} where N; kw::Base.Iterators.Pairs{Symbol, Union{Nothing, Bool}, Tuple{Symbol, Symbol}, NamedTuple{(:iofunction, :reached_redirect_limit), Tuple{Nothing, Bool}}})
      @ HTTP.ExceptionRequest ~/.julia/packages/HTTP/IAI92/src/ExceptionRequest.jl:22
    [2] (::Base.var"#70#72"{Base.var"#70#71#73"{ExponentialBackOff, HTTP.RetryRequest.var"#2#3"{Bool, HTTP.Messages.Request}, typeof(HTTP.request)}})(::Type, ::Vararg{Any, N} where N; kwargs::Base.Iterators.Pairs{Symbol, Union{Nothing, Bool}, Tuple{Symbol, Symbol}, NamedTuple{(:iofunction, :reached_redirect_limit), Tuple{Nothing, Bool}}})
      @ Base ./error.jl:288
    [3] #request#1
      @ ~/.julia/packages/HTTP/IAI92/src/RetryRequest.jl:44 [inlined]
    [4] request(::Type{HTTP.MessageRequest.MessageLayer{HTTP.RetryRequest.RetryLayer{HTTP.ExceptionRequest.ExceptionLayer{HTTP.ConnectionRequest.ConnectionPoolLayer{HTTP.StreamRequest.StreamLayer{Union{}}}}}}}, method::String, url::HTTP.URIs.URI, headers::Base.Vector{Pair{SubString{String}, SubString{String}}}, body::String; http_version::VersionNumber, target::String, parent::Nothing, iofunction::Nothing, kw::Base.Iterators.Pairs{Symbol, Bool, Tuple{Symbol}, NamedTuple{(:reached_redirect_limit,), Tuple{Bool}}})
      @ HTTP.MessageRequest ~/.julia/packages/HTTP/IAI92/src/MessageRequest.jl:51
    [5] request(::Type{HTTP.BasicAuthRequest.BasicAuthLayer{HTTP.MessageRequest.MessageLayer{HTTP.RetryRequest.RetryLayer{HTTP.ExceptionRequest.ExceptionLayer{HTTP.ConnectionRequest.ConnectionPoolLayer{HTTP.StreamRequest.StreamLayer{Union{}}}}}}}}, method::String, url::HTTP.URIs.URI, headers::Base.Vector{Pair{SubString{String}, SubString{String}}}, body::String; kw::Base.Iterators.Pairs{Symbol, Bool, Tuple{Symbol}, NamedTuple{(:reached_redirect_limit,), Tuple{Bool}}})
      @ HTTP.BasicAuthRequest ~/.julia/packages/HTTP/IAI92/src/BasicAuthRequest.jl:28
    [6] request(::Type{HTTP.RedirectRequest.RedirectLayer{HTTP.BasicAuthRequest.BasicAuthLayer{HTTP.MessageRequest.MessageLayer{HTTP.RetryRequest.RetryLayer{HTTP.ExceptionRequest.ExceptionLayer{HTTP.ConnectionRequest.ConnectionPoolLayer{HTTP.StreamRequest.StreamLayer{Union{}}}}}}}}}, method::String, url::HTTP.URIs.URI, headers::Base.Vector{Pair{SubString{String}, SubString{String}}}, body::String; redirect_limit::Int64, forwardheaders::Bool, kw::Base.Iterators.Pairs{Union{}, Union{}, Tuple{}, NamedTuple{(), Tuple{}}})
      @ HTTP.RedirectRequest ~/.julia/packages/HTTP/IAI92/src/RedirectRequest.jl:24
    [7] request(::Type{HTTP.RedirectRequest.RedirectLayer{HTTP.BasicAuthRequest.BasicAuthLayer{HTTP.MessageRequest.MessageLayer{HTTP.RetryRequest.RetryLayer{HTTP.ExceptionRequest.ExceptionLayer{HTTP.ConnectionRequest.ConnectionPoolLayer{HTTP.StreamRequest.StreamLayer{Union{}}}}}}}}}, method::String, url::HTTP.URIs.URI, headers::Base.Vector{Pair{SubString{String}, SubString{String}}}, body::String)
      @ HTTP.RedirectRequest ~/.julia/packages/HTTP/IAI92/src/RedirectRequest.jl:21
    [8] request(method::String, url::String, h::Tuple{Tuple{String, String}, Tuple{String, String}}, b::String; headers::Tuple{Tuple{String, String}, Tuple{String, String}}, body::String, query::Nothing, kw::Base.Iterators.Pairs{Union{}, Union{}, Tuple{}, NamedTuple{(), Tuple{}}})
      @ HTTP ~/.julia/packages/HTTP/IAI92/src/HTTP.jl:314
    [9] request
      @ ~/.julia/packages/HTTP/IAI92/src/HTTP.jl:314 [inlined]
   [10] #post#14
      @ ~/.julia/packages/HTTP/IAI92/src/HTTP.jl:405 [inlined]
   [11] post
      @ ~/.julia/packages/HTTP/IAI92/src/HTTP.jl:405 [inlined]
   [12] send_request_get_response(method::Symbol, content::Dict{String, Any})
      @ Banyan ~/Projects/banyan-julia/Banyan/src/utils.jl:340
   [13] send_evaluation(value_id::String, job_id::String)
      @ Banyan ~/Projects/banyan-julia/Banyan/src/requests.jl:287
   [14] compute(fut::Future)
      @ Banyan ~/Projects/banyan-julia/Banyan/src/requests.jl:192
   [15] collect(fut::DataFrame)
      @ Banyan ~/Projects/banyan-julia/Banyan/src/requests.jl:328
   [16] (::var"#15#23")(job::String)
      @ Main ~/Projects/banyan-julia/BanyanDataFrames/test/test_small_dataset.jl:16
   [17] (::var"#3#6"{var"#15#23"})(j::String)
      @ Main ~/Projects/banyan-julia/BanyanDataFrames/test/runtests.jl:77
   [18] with_job(f::var"#3#6"{var"#15#23"}; kwargs::Base.Iterators.Pairs{Symbol, Any, Tuple{Symbol, Symbol}, NamedTuple{(:job, :destroy_job_on_exit), Tuple{String, Bool}}})
      @ Banyan ~/Projects/banyan-julia/Banyan/src/jobs.jl:258
   [19] run_with_job(test_fn::var"#15#23", name::String)
      @ Main ~/Projects/banyan-julia/BanyanDataFrames/test/runtests.jl:72
   [20] macro expansion
      @ ~/Projects/banyan-julia/BanyanDataFrames/test/test_small_dataset.jl:2 [inlined]
   [21] macro expansion
      @ /buildworker/worker/package_linux64/build/usr/share/julia/stdlib/v1.6/Test/src/Test.jl:1151 [inlined]
   [22] top-level scope
      @ ~/Projects/banyan-julia/BanyanDataFrames/test/test_small_dataset.jl:2
   [23] include
      @ ./client.jl:444 [inlined]
   [24] include_tests_to_run
      @ ~/Projects/banyan-julia/BanyanDataFrames/test/runtests.jl:14 [inlined]
   [25] include_all_tests()
      @ Main ~/Projects/banyan-julia/BanyanDataFrames/test/runtests.jl:99
   [26] (::var"#9#10")(j::String)
      @ Main ~/Projects/banyan-julia/BanyanDataFrames/test/runtests.jl:114
   [27] with_job(f::var"#9#10"; kwargs::Base.Iterators.Pairs{Symbol, String, Tuple{Symbol}, NamedTuple{(:job,), Tuple{String}}})
      @ Banyan ~/Projects/banyan-julia/Banyan/src/jobs.jl:258
   [28] top-level scope
      @ ~/Projects/banyan-julia/BanyanDataFrames/test/runtests.jl:104
   [29] include(fname::String)
      @ Base.MainInclude ./client.jl:444
   [30] top-level scope
      @ none:6
   [31] eval
      @ ./boot.jl:360 [inlined]
   [32] exec_options(opts::Base.JLOptions)
      @ Base ./client.jl:261
   [33] _start()
      @ Base ./client.jl:485

@calebwin
Copy link
Contributor Author

This has only come up when testing so maybe this happens when an error occurs but then the test proceeds past the failed assertion and even though the job has been destroyed it calls compute again and then compute hangs because its trying to get a queue or something. I'm not sure that this always happens when their is a failure though.

@calebwin
Copy link
Contributor Author

Both job destruction (in #11) and calls to compute are now idempotent so we should be able to simply implement retry logic to fix this.

@calebwin
Copy link
Contributor Author

calebwin commented Aug 20, 2021 •
edited
Loading

Another time this failed with the following:

Basic data analytics on a small dataset: Error During Test at /home/calebwin/Projects/banyan-julia/BanyanDataFrames/test/test_small_dataset.jl:63
  Test threw exception
  Expression: nrow(iris_filtered) == 306
  IOError(Base.IOError("read: connection timed out (ETIMEDOUT)", -110) during request(https://hcohsbhhzf.execute-api.us-west-2.amazonaws.com/dev/evaluate))
  
  Stacktrace:
    [1] wait_readnb(x::Sockets.TCPSocket, nb::Int64)
      @ Base ./stream.jl:394
    [2] eof(s::Sockets.TCPSocket)
      @ Base ./stream.jl:106
    [3] wait_for_encrypted_data(ctx::MbedTLS.SSLContext)
      @ MbedTLS ~/.julia/packages/MbedTLS/4YY6E/src/ssl.jl:374
    [4] wait_for_decrypted_data(ctx::MbedTLS.SSLContext)
      @ MbedTLS ~/.julia/packages/MbedTLS/4YY6E/src/ssl.jl:294
    [5] eof
      @ ~/.julia/packages/MbedTLS/4YY6E/src/ssl.jl:186 [inlined]
    [6] read_to_buffer(t::HTTP.ConnectionPool.Transaction{MbedTLS.SSLContext}, sizehint::Int64)
      @ HTTP.ConnectionPool ~/.julia/packages/HTTP/IAI92/src/ConnectionPool.jl:251
    [7] readuntil(t::HTTP.ConnectionPool.Transaction{MbedTLS.SSLContext}, f::Function, sizehint::Int64)
      @ HTTP.ConnectionPool ~/.julia/packages/HTTP/IAI92/src/ConnectionPool.jl:271
    [8] readuntil
      @ ~/.julia/packages/HTTP/IAI92/src/ConnectionPool.jl:269 [inlined]
    [9] readheaders
      @ ~/.julia/packages/HTTP/IAI92/src/Messages.jl:471 [inlined]
   [10] startread(http::HTTP.Streams.Stream{HTTP.Messages.Response, HTTP.ConnectionPool.Transaction{MbedTLS.SSLContext}})
      @ HTTP.Streams ~/.julia/packages/HTTP/IAI92/src/Streams.jl:155
   [11] macro expansion
      @ ~/.julia/packages/HTTP/IAI92/src/StreamRequest.jl:67 [inlined]
   [12] macro expansion
      @ ./task.jl:387 [inlined]
   [13] request(::Type{HTTP.StreamRequest.StreamLayer{Union{}}}, io::HTTP.ConnectionPool.Transaction{MbedTLS.SSLContext}, req::HTTP.Messages.Request, body::String; reached_redirect_limit::Bool, response_stream::Nothing, iofunction::Nothing, verbose::Int64, kw::Base.Iterators.Pairs{Union{}, Union{}, Tuple{}, NamedTuple{(), Tuple{}}})
      @ HTTP.StreamRequest ~/.julia/packages/HTTP/IAI92/src/StreamRequest.jl:57
   [14] request(::Type{HTTP.ConnectionRequest.ConnectionPoolLayer{HTTP.StreamRequest.StreamLayer{Union{}}}}, url::HTTP.URIs.URI, req::HTTP.Messages.Request, body::String; proxy::Nothing, socket_type::Type, reuse_limit::Int64, kw::Base.Iterators.Pairs{Symbol, Union{Nothing, Bool}, Tuple{Symbol, Symbol}, NamedTuple{(:iofunction, :reached_redirect_limit), Tuple{Nothing, Bool}}})
      @ HTTP.ConnectionRequest ~/.julia/packages/HTTP/IAI92/src/ConnectionRequest.jl:96
   [15] request(::Type{HTTP.ExceptionRequest.ExceptionLayer{HTTP.ConnectionRequest.ConnectionPoolLayer{HTTP.StreamRequest.StreamLayer{Union{}}}}}, ::HTTP.URIs.URI, ::Vararg{Any, N} where N; kw::Base.Iterators.Pairs{Symbol, Union{Nothing, Bool}, Tuple{Symbol, Symbol}, NamedTuple{(:iofunction, :reached_redirect_limit), Tuple{Nothing, Bool}}})
      @ HTTP.ExceptionRequest ~/.julia/packages/HTTP/IAI92/src/ExceptionRequest.jl:19
   [16] (::Base.var"#70#72"{Base.var"#70#71#73"{ExponentialBackOff, HTTP.RetryRequest.var"#2#3"{Bool, HTTP.Messages.Request}, typeof(HTTP.request)}})(::Type, ::Vararg{Any, N} where N; kwargs::Base.Iterators.Pairs{Symbol, Union{Nothing, Bool}, Tuple{Symbol, Symbol}, NamedTuple{(:iofunction, :reached_redirect_limit), Tuple{Nothing, Bool}}})
      @ Base ./error.jl:288
   [17] #request#1
      @ ~/.julia/packages/HTTP/IAI92/src/RetryRequest.jl:44 [inlined]
   [18] request(::Type{HTTP.MessageRequest.MessageLayer{HTTP.RetryRequest.RetryLayer{HTTP.ExceptionRequest.ExceptionLayer{HTTP.ConnectionRequest.ConnectionPoolLayer{HTTP.StreamRequest.StreamLayer{Union{}}}}}}}, method::String, url::HTTP.URIs.URI, headers::Base.Vector{Pair{SubString{String}, SubString{String}}}, body::String; http_version::VersionNumber, target::String, parent::Nothing, iofunction::Nothing, kw::Base.Iterators.Pairs{Symbol, Bool, Tuple{Symbol}, NamedTuple{(:reached_redirect_limit,), Tuple{Bool}}})
      @ HTTP.MessageRequest ~/.julia/packages/HTTP/IAI92/src/MessageRequest.jl:51
   [19] request(::Type{HTTP.BasicAuthRequest.BasicAuthLayer{HTTP.MessageRequest.MessageLayer{HTTP.RetryRequest.RetryLayer{HTTP.ExceptionRequest.ExceptionLayer{HTTP.ConnectionRequest.ConnectionPoolLayer{HTTP.StreamRequest.StreamLayer{Union{}}}}}}}}, method::String, url::HTTP.URIs.URI, headers::Base.Vector{Pair{SubString{String}, SubString{String}}}, body::String; kw::Base.Iterators.Pairs{Symbol, Bool, Tuple{Symbol}, NamedTuple{(:reached_redirect_limit,), Tuple{Bool}}})
      @ HTTP.BasicAuthRequest ~/.julia/packages/HTTP/IAI92/src/BasicAuthRequest.jl:28
   [20] request(::Type{HTTP.RedirectRequest.RedirectLayer{HTTP.BasicAuthRequest.BasicAuthLayer{HTTP.MessageRequest.MessageLayer{HTTP.RetryRequest.RetryLayer{HTTP.ExceptionRequest.ExceptionLayer{HTTP.ConnectionRequest.ConnectionPoolLayer{HTTP.StreamRequest.StreamLayer{Union{}}}}}}}}}, method::String, url::HTTP.URIs.URI, headers::Base.Vector{Pair{SubString{String}, SubString{String}}}, body::String; redirect_limit::Int64, forwardheaders::Bool, kw::Base.Iterators.Pairs{Union{}, Union{}, Tuple{}, NamedTuple{(), Tuple{}}})
      @ HTTP.RedirectRequest ~/.julia/packages/HTTP/IAI92/src/RedirectRequest.jl:24
   [21] request(::Type{HTTP.RedirectRequest.RedirectLayer{HTTP.BasicAuthRequest.BasicAuthLayer{HTTP.MessageRequest.MessageLayer{HTTP.RetryRequest.RetryLayer{HTTP.ExceptionRequest.ExceptionLayer{HTTP.ConnectionRequest.ConnectionPoolLayer{HTTP.StreamRequest.StreamLayer{Union{}}}}}}}}}, method::String, url::HTTP.URIs.URI, headers::Base.Vector{Pair{SubString{String}, SubString{String}}}, body::String)
      @ HTTP.RedirectRequest ~/.julia/packages/HTTP/IAI92/src/RedirectRequest.jl:21
   [22] request(method::String, url::String, h::Tuple{Tuple{String, String}, Tuple{String, String}}, b::String; headers::Tuple{Tuple{String, String}, Tuple{String, String}}, body::String, query::Nothing, kw::Base.Iterators.Pairs{Union{}, Union{}, Tuple{}, NamedTuple{(), Tuple{}}})
      @ HTTP ~/.julia/packages/HTTP/IAI92/src/HTTP.jl:314
   [23] request
      @ ~/.julia/packages/HTTP/IAI92/src/HTTP.jl:314 [inlined]
   [24] #post#14
      @ ~/.julia/packages/HTTP/IAI92/src/HTTP.jl:405 [inlined]
   [25] post
      @ ~/.julia/packages/HTTP/IAI92/src/HTTP.jl:405 [inlined]
   [26] send_request_get_response(method::Symbol, content::Dict{String, Any})
      @ Banyan ~/Projects/banyan-julia/Banyan/src/utils.jl:340
   [27] send_evaluation(value_id::String, job_id::String)
      @ Banyan ~/Projects/banyan-julia/Banyan/src/requests.jl:287
   [28] compute(fut::Future)
      @ Banyan ~/Projects/banyan-julia/Banyan/src/requests.jl:192
   [29] collect(fut::Future)
      @ Banyan ~/Projects/banyan-julia/Banyan/src/requests.jl:332
   [30] nrow
      @ ~/Projects/banyan-julia/BanyanDataFrames/src/df.jl:137 [inlined]
   [31] (::var"#17#31")(job::String)
      @ Main ~/Projects/banyan-julia/BanyanDataFrames/test/test_small_dataset.jl:63
   [32] (::var"#3#6"{var"#17#31"})(j::String)
      @ Main ~/Projects/banyan-julia/BanyanDataFrames/test/runtests.jl:83
   [33] with_job(f::var"#3#6"{var"#17#31"}; kwargs::Base.Iterators.Pairs{Symbol, Any, Tuple{Symbol, Symbol}, NamedTuple{(:job, :destroy_job_on_exit), Tuple{String, Bool}}})
      @ Banyan ~/Projects/banyan-julia/Banyan/src/jobs.jl:258
   [34] run_with_job(test_fn::var"#17#31", name::String)
      @ Main ~/Projects/banyan-julia/BanyanDataFrames/test/runtests.jl:78
   [35] macro expansion
      @ ~/Projects/banyan-julia/BanyanDataFrames/test/test_small_dataset.jl:54 [inlined]
   [36] macro expansion
      @ /buildworker/worker/package_linux64/build/usr/share/julia/stdlib/v1.6/Test/src/Test.jl:1151 [inlined]
   [37] top-level scope
      @ ~/Projects/banyan-julia/BanyanDataFrames/test/test_small_dataset.jl:23

@calebwin
Copy link
Contributor Author

This may have something to do with this:

Hello,

We had previously contacted you about a change in the behavior of asynchronous AWS Lambda functions while reserved concurrency is set to zero (0), where events would be sent to the dead letter queue (DLQ) or on-failure event destination immediately instead of being retried for up to 6 hours (if neither is configured, events are dropped immediately).

This change was originally planned to begin on August 16, 2021. However, to provide more time to customers to assess impact, we have decided to delay implementing this change in behavior to August 30, 2021. After August 30, events sent to your Lambda functions while reserved concurrency is set to zero will be sent to the DLQ or on-failure destination immediately.

This change does not affect functions that don't have reserved concurrency configured or have it set to a non-zero value. This change applies to all AWS Regions. Please refer to the AWS Lambda User Guide for information on how to configure a DLQ [1] or an on-failure event destination [2].

Please contact us through AWS Support [3], the AWS Lambda Developer Forum [4], or your AWS account team should you have any questions or concerns. You may also refer to the forum post below [5] for more details.

[1] https://docs.aws.amazon.com/lambda/latest/dg/invocation-async.html#dlq
[2] https://docs.aws.amazon.com/lambda/latest/dg/invocation-async.html#invocation-async-destinations
[3] https://aws.amazon.com/support
[4] https://forums.aws.amazon.com/forum.jspa?forumID=186
[5] https://forums.aws.amazon.com/ann.jspa?annID=8834

Sincerely,
Amazon Web Services

@calebwin
Copy link
Contributor Author

calebwin commented Aug 21, 2021 •
edited
Loading

On another occasion, the same issue occurred but this time with describing clusters. This happened when no errors were reported on the client side. No error was expected to occur. It's not immediately clear why describe-clusters failed.

Basic data analytics on a small dataset: Error During Test at /home/calebwin/Projects/banyan-julia/BanyanDataFrames/test/test_small_dataset.jl:22
  Got exception outside of a @test
  IOError(Base.IOError("read: connection timed out (ETIMEDOUT)", -110) during request(https://hcohsbhhzf.execute-api.us-west-2.amazonaws.com/dev/describe-clusters))
  
  Stacktrace:
    [1] wait_readnb(x::Sockets.TCPSocket, nb::Int64)
      @ Base ./stream.jl:394
    [2] eof(s::Sockets.TCPSocket)
      @ Base ./stream.jl:106
    [3] wait_for_encrypted_data(ctx::MbedTLS.SSLContext)
      @ MbedTLS ~/.julia/packages/MbedTLS/4YY6E/src/ssl.jl:374
    [4] wait_for_decrypted_data(ctx::MbedTLS.SSLContext)
      @ MbedTLS ~/.julia/packages/MbedTLS/4YY6E/src/ssl.jl:294
    [5] eof
      @ ~/.julia/packages/MbedTLS/4YY6E/src/ssl.jl:186 [inlined]
    [6] read_to_buffer(t::HTTP.ConnectionPool.Transaction{MbedTLS.SSLContext}, sizehint::Int64)
      @ HTTP.ConnectionPool ~/.julia/packages/HTTP/IAI92/src/ConnectionPool.jl:251
    [7] readuntil(t::HTTP.ConnectionPool.Transaction{MbedTLS.SSLContext}, f::Function, sizehint::Int64)
      @ HTTP.ConnectionPool ~/.julia/packages/HTTP/IAI92/src/ConnectionPool.jl:271
    [8] readuntil
      @ ~/.julia/packages/HTTP/IAI92/src/ConnectionPool.jl:269 [inlined]
    [9] readheaders
      @ ~/.julia/packages/HTTP/IAI92/src/Messages.jl:471 [inlined]
   [10] startread(http::HTTP.Streams.Stream{HTTP.Messages.Response, HTTP.ConnectionPool.Transaction{MbedTLS.SSLContext}})
      @ HTTP.Streams ~/.julia/packages/HTTP/IAI92/src/Streams.jl:155
   [11] macro expansion
      @ ~/.julia/packages/HTTP/IAI92/src/StreamRequest.jl:67 [inlined]
   [12] macro expansion
      @ ./task.jl:387 [inlined]
   [13] request(::Type{HTTP.StreamRequest.StreamLayer{Union{}}}, io::HTTP.ConnectionPool.Transaction{MbedTLS.SSLContext}, req::HTTP.Messages.Request, body::String; reached_redirect_limit::Bool, response_stream::Nothing, iofunction::Nothing, verbose::Int64, kw::Base.Iterators.Pairs{Union{}, Union{}, Tuple{}, NamedTuple{(), Tuple{}}})
      @ HTTP.StreamRequest ~/.julia/packages/HTTP/IAI92/src/StreamRequest.jl:57
   [14] request(::Type{HTTP.ConnectionRequest.ConnectionPoolLayer{HTTP.StreamRequest.StreamLayer{Union{}}}}, url::HTTP.URIs.URI, req::HTTP.Messages.Request, body::String; proxy::Nothing, socket_type::Type, reuse_limit::Int64, kw::Base.Iterators.Pairs{Symbol, Union{Nothing, Bool}, Tuple{Symbol, Symbol}, NamedTuple{(:iofunction, :reached_redirect_limit), Tuple{Nothing, Bool}}})
      @ HTTP.ConnectionRequest ~/.julia/packages/HTTP/IAI92/src/ConnectionRequest.jl:96
   [15] request(::Type{HTTP.ExceptionRequest.ExceptionLayer{HTTP.ConnectionRequest.ConnectionPoolLayer{HTTP.StreamRequest.StreamLayer{Union{}}}}}, ::HTTP.URIs.URI, ::Vararg{Any, N} where N; kw::Base.Iterators.Pairs{Symbol, Union{Nothing, Bool}, Tuple{Symbol, Symbol}, NamedTuple{(:iofunction, :reached_redirect_limit), Tuple{Nothing, Bool}}})
      @ HTTP.ExceptionRequest ~/.julia/packages/HTTP/IAI92/src/ExceptionRequest.jl:19
   [16] (::Base.var"#70#72"{Base.var"#70#71#73"{ExponentialBackOff, HTTP.RetryRequest.var"#2#3"{Bool, HTTP.Messages.Request}, typeof(HTTP.request)}})(::Type, ::Vararg{Any, N} where N; kwargs::Base.Iterators.Pairs{Symbol, Union{Nothing, Bool}, Tuple{Symbol, Symbol}, NamedTuple{(:iofunction, :reached_redirect_limit), Tuple{Nothing, Bool}}})
      @ Base ./error.jl:288
   [17] #request#1
      @ ~/.julia/packages/HTTP/IAI92/src/RetryRequest.jl:44 [inlined]
   [18] request(::Type{HTTP.MessageRequest.MessageLayer{HTTP.RetryRequest.RetryLayer{HTTP.ExceptionRequest.ExceptionLayer{HTTP.ConnectionRequest.ConnectionPoolLayer{HTTP.StreamRequest.StreamLayer{Union{}}}}}}}, method::String, url::HTTP.URIs.URI, headers::Base.Vector{Pair{SubString{String}, SubString{String}}}, body::String; http_version::VersionNumber, target::String, parent::Nothing, iofunction::Nothing, kw::Base.Iterators.Pairs{Symbol, Bool, Tuple{Symbol}, NamedTuple{(:reached_redirect_limit,), Tuple{Bool}}})
      @ HTTP.MessageRequest ~/.julia/packages/HTTP/IAI92/src/MessageRequest.jl:51
   [19] request(::Type{HTTP.BasicAuthRequest.BasicAuthLayer{HTTP.MessageRequest.MessageLayer{HTTP.RetryRequest.RetryLayer{HTTP.ExceptionRequest.ExceptionLayer{HTTP.ConnectionRequest.ConnectionPoolLayer{HTTP.StreamRequest.StreamLayer{Union{}}}}}}}}, method::String, url::HTTP.URIs.URI, headers::Base.Vector{Pair{SubString{String}, SubString{String}}}, body::String; kw::Base.Iterators.Pairs{Symbol, Bool, Tuple{Symbol}, NamedTuple{(:reached_redirect_limit,), Tuple{Bool}}})
      @ HTTP.BasicAuthRequest ~/.julia/packages/HTTP/IAI92/src/BasicAuthRequest.jl:28
   [20] request(::Type{HTTP.RedirectRequest.RedirectLayer{HTTP.BasicAuthRequest.BasicAuthLayer{HTTP.MessageRequest.MessageLayer{HTTP.RetryRequest.RetryLayer{HTTP.ExceptionRequest.ExceptionLayer{HTTP.ConnectionRequest.ConnectionPoolLayer{HTTP.StreamRequest.StreamLayer{Union{}}}}}}}}}, method::String, url::HTTP.URIs.URI, headers::Base.Vector{Pair{SubString{String}, SubString{String}}}, body::String; redirect_limit::Int64, forwardheaders::Bool, kw::Base.Iterators.Pairs{Union{}, Union{}, Tuple{}, NamedTuple{(), Tuple{}}})
      @ HTTP.RedirectRequest ~/.julia/packages/HTTP/IAI92/src/RedirectRequest.jl:24
   [21] request(::Type{HTTP.RedirectRequest.RedirectLayer{HTTP.BasicAuthRequest.BasicAuthLayer{HTTP.MessageRequest.MessageLayer{HTTP.RetryRequest.RetryLayer{HTTP.ExceptionRequest.ExceptionLayer{HTTP.ConnectionRequest.ConnectionPoolLayer{HTTP.StreamRequest.StreamLayer{Union{}}}}}}}}}, method::String, url::HTTP.URIs.URI, headers::Base.Vector{Pair{SubString{String}, SubString{String}}}, body::String)
      @ HTTP.RedirectRequest ~/.julia/packages/HTTP/IAI92/src/RedirectRequest.jl:21
   [22] request(method::String, url::String, h::Tuple{Tuple{String, String}, Tuple{String, String}}, b::String; headers::Tuple{Tuple{String, String}, Tuple{String, String}}, body::String, query::Nothing, kw::Base.Iterators.Pairs{Union{}, Union{}, Tuple{}, NamedTuple{(), Tuple{}}})
      @ HTTP ~/.julia/packages/HTTP/IAI92/src/HTTP.jl:314
   [23] request
      @ ~/.julia/packages/HTTP/IAI92/src/HTTP.jl:314 [inlined]
   [24] #post#14
      @ ~/.julia/packages/HTTP/IAI92/src/HTTP.jl:405 [inlined]
   [25] post
      @ ~/.julia/packages/HTTP/IAI92/src/HTTP.jl:405 [inlined]
   [26] send_request_get_response(method::Symbol, content::Dict{String, Any})
      @ Banyan ~/Projects/banyan-julia/Banyan/src/utils.jl:340
   [27] get_clusters(; kwargs::Base.Iterators.Pairs{Union{}, Union{}, Tuple{}, NamedTuple{(), Tuple{}}})
      @ Banyan ~/Projects/banyan-julia/Banyan/src/clusters.jl:555
   [28] get_clusters
      @ ~/Projects/banyan-julia/Banyan/src/clusters.jl:553 [inlined]
   [29] get_cluster(::String)
      @ Banyan ~/Projects/banyan-julia/Banyan/src/clusters.jl:575
   [30] get_cluster()
      @ Banyan ~/Projects/banyan-julia/Banyan/src/clusters.jl:576
   [31] (::var"#17#30")(job::String)
      @ Main ~/Projects/banyan-julia/BanyanDataFrames/test/test_small_dataset.jl:55
   [32] (::var"#3#6"{var"#17#30"})(j::String)
      @ Main ~/Projects/banyan-julia/BanyanDataFrames/test/runtests.jl:84
   [33] with_job(f::var"#3#6"{var"#17#30"}; kwargs::Base.Iterators.Pairs{Symbol, Any, Tuple{Symbol, Symbol}, NamedTuple{(:job, :destroy_job_on_exit), Tuple{String, Bool}}})
      @ Banyan ~/Projects/banyan-julia/Banyan/src/jobs.jl:258
   [34] run_with_job(test_fn::var"#17#30", name::String)
      @ Main ~/Projects/banyan-julia/BanyanDataFrames/test/runtests.jl:79
   [35] macro expansion
      @ ~/Projects/banyan-julia/BanyanDataFrames/test/test_small_dataset.jl:54 [inlined]
   [36] macro expansion
      @ /buildworker/worker/package_linux64/build/usr/share/julia/stdlib/v1.6/Test/src/Test.jl:1151 [inlined]
   [37] top-level scope
      @ ~/Projects/banyan-julia/BanyanDataFrames/test/test_small_dataset.jl:23
   [38] include
      @ ./client.jl:444 [inlined]
   [39] include_tests_to_run
      @ ~/Projects/banyan-julia/BanyanDataFrames/test/runtests.jl:21 [inlined]
   [40] include_all_tests()
      @ Main ~/Projects/banyan-julia/BanyanDataFrames/test/runtests.jl:105
   [41] (::var"#9#10")(j::String)
      @ Main ~/Projects/banyan-julia/BanyanDataFrames/test/runtests.jl:131
   [42] with_job(f::var"#9#10"; kwargs::Base.Iterators.Pairs{Symbol, String, Tuple{Symbol}, NamedTuple{(:job,), Tuple{String}}})
      @ Banyan ~/Projects/banyan-julia/Banyan/src/jobs.jl:258
   [43] top-level scope
      @ ~/Projects/banyan-julia/BanyanDataFrames/test/runtests.jl:121
   [44] include(fname::String)
      @ Base.MainInclude ./client.jl:444
   [45] top-level scope
      @ none:6
   [46] eval
      @ ./boot.jl:360 [inlined]
   [47] exec_options(opts::Base.JLOptions)
      @ Base ./client.jl:261
   [48] _start()
      @ Base ./client.jl:485

@calebwin
Copy link
Contributor Author

Maybe we need to update the version of HTTP.jl we are using. Some potentially rlevant commits that we need:

We are using release v0.8.19 while the latest version is v0.9.13.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees
No one assigned
Labels
banyan-jl Concerning Banyan.jl bug Something isn't working
Projects
None yet
Milestone
No milestone
Development

No branches or pull requests
1 participant
@calebwin