Add 'use_maps' option to fxml_stream.parse_element/2. #50
Conversation
|
Ok, it passes the fast_xml standalone tests. I'll test if ejabberd passes its tests when using this PR and compiled with rebar3 without elixir support, rebar3 with elixir support, and mix. |
|
After applying this patch to ejabberd git: diff --git a/mix.exs b/mix.exs
index a82addd7b..7d1b2486e 100644
--- a/mix.exs
+++ b/mix.exs
@@ -136,7 +136,7 @@ defmodule Ejabberd.MixProject do
{:eimp, "~> 1.0"},
{:ex_doc, "~> 0.31", only: [:dev, :edoc], runtime: false},
{:fast_tls, ">= 1.1.18"},
- {:fast_xml, ">= 1.1.51"},
+ {:fast_xml, git: "https://github.com/goodgamechat/fast_xml.git", ref: "c2de8471cd2ae8ba8c43445a306eba1c96e381fa", override: true},
{:fast_yaml, "~> 1.0"},
{:idna, "~> 6.0"},
{:mqtree, "~> 1.0"},Compilation suceeds with rebar3, with or without elixir support. Compilation with mix and erlang 26.2 fails: |
|
Strange, possibly mix is using different erlang version where there is issue with that module? |
|
Also this doesn't even touches this parse transform... |
|
I was receiving this error too but thought it was something wrong with my local installation. I'm running elixir 1.16 and the After digging a little more, I was able to find a way to get it work but doesn't explain why mix compile.erlang
src/fxmlrpc_codec.erl:6:2: warning: export_all flag enabled - all functions will be exported
src/fxmlrpc_codec_external.erl:6:2: warning: export_all flag enabled - all functions will be exported
Compiling 1 file (.erl)
mix compile.elixir
Compiling 1 file (.ex)
mix compile
src/fxmlrpc_codec.erl:6:2: warning: export_all flag enabled - all functions will be exported
src/fxmlrpc_codec_external.erl:6:2: warning: export_all flag enabled - all functions will be exported
Generated fast_xml appIt's not clear why |
|
It seems the problem is that the fast_xml git repository contains some old and obsolete files that were used to compile that library using Mix many time ago. When compiling the library obtained from git repository using ejabberd and Mix, those files are used and they screw compilation: mix then uses rebar2 instead of rebar3, and fails even with fast_xml master git branch: $ git diff mix.exs
diff --git a/mix.exs b/mix.exs
index a82addd7b..6dd246b57 100644
--- a/mix.exs
+++ b/mix.exs
@@ -135,9 +135,9 @@ defmodule Ejabberd.MixProject do
{:dialyxir, "~> 1.2", only: [:test], runtime: false},
{:eimp, "~> 1.0"},
{:ex_doc, "~> 0.31", only: [:dev, :edoc], runtime: false},
- {:fast_tls, ">= 1.1.18"},
- {:fast_xml, ">= 1.1.51"},
- {:fast_yaml, "~> 1.0"},
+ {:fast_tls, git: "https://github.com/processone/fast_tls.git", override: true},
+ {:fast_xml, git: "https://github.com/processone/fast_xml.git", override: true},
+ {:fast_yaml, git: "https://github.com/processone/fast_yaml.git", override: true},
{:idna, "~> 6.0"},
{:mqtree, "~> 1.0"},
{:p1_acme, "~> 1.0"},
$ rm -rf deps/fast_*
$ mix deps.unlock fast_xml fast_yaml fast_tls
Unlocked deps:
* fast_xml
* fast_yaml
* fast_tls
$ mix deps.get
* Getting fast_tls (https://github.com/processone/fast_tls.git)
remote: Enumerating objects: 1454, done.
remote: Counting objects: 100% (231/231), done.
remote: Compressing objects: 100% (121/121), done.
remote: Total 1454 (delta 124), reused 202 (delta 103), pack-reused 1223
origin/HEAD set to master
* Getting fast_xml (https://github.com/processone/fast_xml.git)
remote: Enumerating objects: 2082, done.
remote: Counting objects: 100% (393/393), done.
remote: Compressing objects: 100% (165/165), done.
remote: Total 2082 (delta 226), reused 327 (delta 211), pack-reused 1689
origin/HEAD set to master
* Getting fast_yaml (https://github.com/processone/fast_yaml.git)
remote: Enumerating objects: 955, done.
remote: Counting objects: 100% (77/77), done.
remote: Compressing objects: 100% (52/52), done.
remote: Total 955 (delta 37), reused 53 (delta 23), pack-reused 878
origin/HEAD set to master
Resolving Hex dependencies...
Resolution completed in 0.119s
Unchanged:
base64url 1.0.1
cache_tab 1.0.31
dialyxir 1.4.3
earmark_parser 1.4.41
eimp 1.0.23
elixir_make 0.8.4
epam 1.0.14
eredis 1.2.0
erlex 0.2.7
esip 1.0.54
ex_doc 0.34.2
exsync 0.4.1
ezlib 1.0.13
file_system 1.0.0
idna 6.1.1
jiffy 1.1.2
jose 1.11.10
luerl 1.2.0
makeup 1.1.2
makeup_elixir 0.16.2
makeup_erlang 1.0.0
mqtree 1.0.17
nimble_parsec 1.4.0
p1_acme 1.0.23
p1_mysql 1.0.24
p1_oauth2 0.6.14
p1_pgsql 1.1.27
p1_utils 1.0.26
pkix 1.0.10
sqlite3 1.1.15
stringprep 1.0.30
stun 1.2.14
unicode_util_compat 0.7.0
xmpp 1.8.3
yconf 1.0.16
$ make
mix compile
===> Fetching pc v1.15.0
===> Analyzing applications...
===> Compiling pc
===> Compiling c_src/fast_yaml.c
===> Linking /home/bernar/e/git/ejabberd/_build/dev/lib/fast_yaml/priv/lib/fast_yaml.so
===> Analyzing applications...
===> Compiling fast_yaml
===> Analyzing applications...
===> Compiling yconf
===> Analyzing applications...
===> Compiling p1_acme
===> Fetching pc v1.15.0
===> Analyzing applications...
===> Compiling pc
===> Compiling c_src/fast_tls.c
===> Compiling c_src/ioqueue.c
===> Compiling c_src/p1_sha.c
===> Linking /home/bernar/e/git/ejabberd/_build/dev/lib/fast_tls/priv/lib/fast_tls.so
===> Linking /home/bernar/e/git/ejabberd/_build/dev/lib/fast_tls/priv/lib/p1_sha.so
===> Analyzing applications...
===> Compiling fast_tls
===> Analyzing applications...
===> Compiling stun
===> Linking /home/bernar/e/git/ejabberd/_build/dev/lib/esip/priv/lib/esip_drv.so
===> Analyzing applications...
===> Compiling esip
==> fast_xml
make[1]: Entering directory '/home/bernar/e/git/ejabberd/deps/fast_xml'
rebar get-deps
==> fast_xml (get-deps)
Pulling p1_utils from {git,"https://github.com/processone/p1_utils",
{tag,"1.0.26"}}
S'estàclonant a ëp1_utilsû...
==> p1_utils (get-deps)
rebar compile
==> p1_utils (compile)
Compiled src/p1_server.erl
Compiled src/p1_queue.erl
Compiled src/treap.erl
Compiled src/p1_utils_sup.erl
Compiled src/p1_utils.erl
Compiled src/p1_time_compat.erl
Compiled src/p1_nif_utils.erl
Compiled src/p1_shaper.erl
Compiled src/p1_rand.erl
Compiled src/p1_options.erl
Compiled src/p1_http.erl
Compiled src/p1_proxy_protocol.erl
Compiled src/p1_prof.erl
Compiled src/p1_file_queue.erl
Compiled src/p1_fsm.erl
==> fast_xml (compile)
Compiled src/fxml_gen_pt.erl
Compiled src/fxml_sup.erl
Compiled src/fxml_stream.erl
Compiled src/fxml.erl
Compiled src/fxmlrpc.erl
Compiled src/fast_xml.erl
Compiled src/fxmlrpc_codec_external.erl
Compiled src/fxmlrpc_codec.erl
Compiled src/fxml_gen.erl
Compiling c_src/fxml.c
Compiling c_src/fxml_stream.c
make[1]: Leaving directory '/home/bernar/e/git/ejabberd/deps/fast_xml'
Compiling 9 files (.erl)
src/fxmlrpc_codec_external.erl:6:2: Warning: export_all flag enabled - all functions will be exported
% 6| -compile(export_all).
% | ^
src/fxml_gen.erl: error in parse transform 'fxml_gen_pt':
exception error: undefined function erl_syntax_lib:map/2
in function fxml_gen_pt:'-parse_transform/2-fun-1-'/1 (src/fxml_gen_pt.erl, line 23)
in call from lists:map/2 (lists.erl, line 1559)
in call from compile:foldl_transform/3 (compile.erl, line 1195)
in call from compile:fold_comp/4 (compile.erl, line 410)
in call from compile:internal_comp/5 (compile.erl, line 394)
in call from compile:'-internal_fun/2-anonymous-0-'/2 (compile.erl, line 227)
in call from compile:'-do_compile/2-anonymous-0-'/1 (compile.erl, line 217)
src/fxmlrpc_codec.erl:6:2: Warning: export_all flag enabled - all functions will be exported
% 6| -compile(export_all).
% | ^
could not compile dependency :fast_xml, "mix compile" failed. Errors may have been logged above. You can recompile this dependency with "mix deps.compile fast_xml --force", update it with "mix deps.update fast_xml" or clean it with "mix deps.clean fast_xml"
make: *** [Makefile:219: src] Error 1We didn't notice those problematic files, because they are not included in the hex package. Temporary workaround: remove the files before compiling the library, and compilation suceeds even with this PR: $ rm -rf deps/fast_*
$ mix deps.get
* Getting fast_xml (https://github.com/goodgamechat/fast_xml.git - c2de8471cd2ae8ba8c43445a306eba1c96e381fa)
remote: Enumerating objects: 2019, done.
remote: Counting objects: 100% (356/356), done.
remote: Compressing objects: 100% (143/143), done.
remote: Total 2019 (delta 211), reused 288 (delta 196), pack-reused 1663
Resolving Hex dependencies...
Resolution completed in 0.15s
New:
fast_tls 1.1.21
fast_yaml 1.0.37
Unchanged:
base64url 1.0.1
cache_tab 1.0.31
dialyxir 1.4.3
earmark_parser 1.4.41
eimp 1.0.23
elixir_make 0.8.4
epam 1.0.14
eredis 1.2.0
erlex 0.2.7
esip 1.0.54
ex_doc 0.34.2
exsync 0.4.1
ezlib 1.0.13
file_system 1.0.0
idna 6.1.1
jiffy 1.1.2
jose 1.11.10
luerl 1.2.0
makeup 1.1.2
makeup_elixir 0.16.2
makeup_erlang 1.0.1
mqtree 1.0.17
nimble_parsec 1.4.0
p1_acme 1.0.23
p1_mysql 1.0.24
p1_oauth2 0.6.14
p1_pgsql 1.1.27
p1_utils 1.0.26
pkix 1.0.10
sqlite3 1.1.15
stringprep 1.0.30
stun 1.2.14
unicode_util_compat 0.7.0
xmpp 1.8.3
yconf 1.0.16
* Getting fast_tls (Hex package)
* Getting fast_yaml (Hex package)
$ rm -v deps/fast_xml/*mix*
s’ha eliminat 'deps/fast_xml/Makefile.mix'
s’ha eliminat 'deps/fast_xml/mix.exs'
s’ha eliminat 'deps/fast_xml/mix.lock'
$ make
/usr/bin/sed -e "s*{{installuser}}**g" \
-e "s*{{config_dir}}*/usr/local/etc/ejabberd*g" \
-e "s*{{logs_dir}}*/usr/local/var/log/ejabberd*g" \
-e "s*{{spool_dir}}*/usr/local/var/lib/ejabberd*g" \
-e "s*{{bindir}}*/usr/local/bin*g" \
-e "s*{{libdir}}*/usr/local/lib":/home/bernar/.asdf/installs/elixir/1.16.3-otp-26/bin/../lib"*g" \
-e "s*ERTS_VSN*# ERTS_VSN*g" \
-e "s*{{iexpath}}*/home/bernar/.asdf/shims/iex*g" \
-e "s*{{erl}}*/home/bernar/.asdf/shims/erl*g" \
-e "s*{{epmd}}*/home/bernar/.asdf/shims/epmd*g" ejabberdctl.template \
> ejabberdctl.example
mix compile
===> Fetching pc v1.15.0
===> Analyzing applications...
===> Compiling pc
===> Compiling c_src/fast_yaml.c
===> Linking /home/bernar/e/git/ejabberd/_build/dev/lib/fast_yaml/priv/lib/fast_yaml.so
===> Analyzing applications...
===> Compiling fast_yaml
===> Analyzing applications...
===> Compiling yconf
==> makeup_erlang
Compiling 4 files (.ex)
Generated makeup_erlang app
==> ex_doc
Compiling 26 files (.ex)
Generated ex_doc app
===> Analyzing applications...
===> Compiling p1_acme
===> Fetching pc v1.15.0
===> Analyzing applications...
===> Compiling pc
===> Compiling c_src/fast_tls.c
===> Compiling c_src/ioqueue.c
===> Compiling c_src/p1_sha.c
===> Linking /home/bernar/e/git/ejabberd/_build/dev/lib/fast_tls/priv/lib/fast_tls.so
===> Linking /home/bernar/e/git/ejabberd/_build/dev/lib/fast_tls/priv/lib/p1_sha.so
===> Analyzing applications...
===> Compiling fast_tls
===> Analyzing applications...
===> Compiling stun
===> Linking /home/bernar/e/git/ejabberd/_build/dev/lib/esip/priv/lib/esip_drv.so
===> Analyzing applications...
===> Compiling esip
===> Fetching pc v1.15.0
===> Analyzing applications...
===> Compiling pc
===> Compiling c_src/fxml.c
===> Compiling c_src/fxml_stream.c
===> Linking /home/bernar/e/git/ejabberd/_build/dev/lib/fast_xml/priv/lib/fxml.so
===> Linking /home/bernar/e/git/ejabberd/_build/dev/lib/fast_xml/priv/lib/fxml_stream.so
===> Analyzing applications...
===> Compiling fast_xml
===> Linking /home/bernar/e/git/ejabberd/_build/dev/lib/xmpp/priv/lib/jid.so
===> Linking /home/bernar/e/git/ejabberd/_build/dev/lib/xmpp/priv/lib/xmpp_uri.so
===> Linking /home/bernar/e/git/ejabberd/_build/dev/lib/xmpp/priv/lib/xmpp_lang.so
===> /home/bernar/e/git/ejabberd/deps/xmpp/asn1/XmppAddr.asn1
===> Analyzing applications...
===> Compiling xmpp
===> Analyzing applications...
===> Compiling p1_pgsql
==> ejabberd
Compiling 117 files (.erl)
Generated ejabberd appLooking at those three mix files ( |
|
Thanks for looking into it. I agree there might be some sort of mixup with generating the hex package because the module |
I suspect that compiling fast_xml with the mix tool has not been used since 2018, and nobody cared about that fact. I would days that On the other hand,
I removed mix.exs and all elixir files in fast_xml, then compilation suceeds using mix in the parent application, and this is the result of parsing the elements: $ iex -S mix
iex(1)> :fxml_stream.parse_element("<root></root>")
{:xmlel, "root", [], []}
iex(2)> :fxml_stream.parse_element("<root></root>", [:use_maps])
%{name: "root", __struct__: FastXML.El, children: [], attrs: %{}}
iex(10)>
:fxml_stream.new(self(), :infinity, [:no_gen_server, :use_maps])
|> :fxml_stream.parse("<root>")
|> :fxml_stream.parse("<xmlelement>content cdata</xmlelement>")
|> :fxml_stream.parse("<xmlelement><empty/><subelement attribute='true'>content cdata</subelement></xmlelement>")
|> :fxml_stream.parse("</root>")
|> :fxml_stream.close()
iex(10)> receive do result -> result end
%{name: "root", __struct__: FastXML.StreamStart, attrs: %{}}
iex(11)> receive do result -> result end
%{
name: "xmlelement",
__struct__: FastXML.El,
children: ["content cdata"],
attrs: %{}
}
iex(12)> receive do result -> result end
%{
name: "xmlelement",
__struct__: FastXML.El,
children: [
%{name: "empty", __struct__: FastXML.El, children: [], attrs: %{}},
%{
name: "subelement",
__struct__: FastXML.El,
children: ["content cdata"],
attrs: %{"attribute" => "true"}
}
],
attrs: %{}
}
iex(13)> receive do result -> result end
%{name: "root", __struct__: FastXML.StreamEnd} |
|
Gotcha, this looks fine to me on my end. |
|
I've updated fast_xml accordingly, and also applied other changes. Can you rebase your PR against the current master, so it can be merged? Notice that elixir files were removed. |
|
Ok, all seems perfect now, including inclusion in ejabberd. As Elixir files were not used long ago, I guess nobody used the mix compilation, only rebar3 one (even when the library is added as a dependency in an elixir application managed by mix). |
I'm attempting to surface the elixir structs for my project and thought this would be useful.