-
Notifications
You must be signed in to change notification settings - Fork 0
/
Copy pathChangeLog
3858 lines (3858 loc) · 288 KB
/
ChangeLog
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
72
73
74
75
76
77
78
79
80
81
82
83
84
85
86
87
88
89
90
91
92
93
94
95
96
97
98
99
100
101
102
103
104
105
106
107
108
109
110
111
112
113
114
115
116
117
118
119
120
121
122
123
124
125
126
127
128
129
130
131
132
133
134
135
136
137
138
139
140
141
142
143
144
145
146
147
148
149
150
151
152
153
154
155
156
157
158
159
160
161
162
163
164
165
166
167
168
169
170
171
172
173
174
175
176
177
178
179
180
181
182
183
184
185
186
187
188
189
190
191
192
193
194
195
196
197
198
199
200
201
202
203
204
205
206
207
208
209
210
211
212
213
214
215
216
217
218
219
220
221
222
223
224
225
226
227
228
229
230
231
232
233
234
235
236
237
238
239
240
241
242
243
244
245
246
247
248
249
250
251
252
253
254
255
256
257
258
259
260
261
262
263
264
265
266
267
268
269
270
271
272
273
274
275
276
277
278
279
280
281
282
283
284
285
286
287
288
289
290
291
292
293
294
295
296
297
298
299
300
301
302
303
304
305
306
307
308
309
310
311
312
313
314
315
316
317
318
319
320
321
322
323
324
325
326
327
328
329
330
331
332
333
334
335
336
337
338
339
340
341
342
343
344
345
346
347
348
349
350
351
352
353
354
355
356
357
358
359
360
361
362
363
364
365
366
367
368
369
370
371
372
373
374
375
376
377
378
379
380
381
382
383
384
385
386
387
388
389
390
391
392
393
394
395
396
397
398
399
400
401
402
403
404
405
406
407
408
409
410
411
412
413
414
415
416
417
418
419
420
421
422
423
424
425
426
427
428
429
430
431
432
433
434
435
436
437
438
439
440
441
442
443
444
445
446
447
448
449
450
451
452
453
454
455
456
457
458
459
460
461
462
463
464
465
466
467
468
469
470
471
472
473
474
475
476
477
478
479
480
481
482
483
484
485
486
487
488
489
490
491
492
493
494
495
496
497
498
499
500
501
502
503
504
505
506
507
508
509
510
511
512
513
514
515
516
517
518
519
520
521
522
523
524
525
526
527
528
529
530
531
532
533
534
535
536
537
538
539
540
541
542
543
544
545
546
547
548
549
550
551
552
553
554
555
556
557
558
559
560
561
562
563
564
565
566
567
568
569
570
571
572
573
574
575
576
577
578
579
580
581
582
583
584
585
586
587
588
589
590
591
592
593
594
595
596
597
598
599
600
601
602
603
604
605
606
607
608
609
610
611
612
613
614
615
616
617
618
619
620
621
622
623
624
625
626
627
628
629
630
631
632
633
634
635
636
637
638
639
640
641
642
643
644
645
646
647
648
649
650
651
652
653
654
655
656
657
658
659
660
661
662
663
664
665
666
667
668
669
670
671
672
673
674
675
676
677
678
679
680
681
682
683
684
685
686
687
688
689
690
691
692
693
694
695
696
697
698
699
700
701
702
703
704
705
706
707
708
709
710
711
712
713
714
715
716
717
718
719
720
721
722
723
724
725
726
727
728
729
730
731
732
733
734
735
736
737
738
739
740
741
742
743
744
745
746
747
748
749
750
751
752
753
754
755
756
757
758
759
760
761
762
763
764
765
766
767
768
769
770
771
772
773
774
775
776
777
778
779
780
781
782
783
784
785
786
787
788
789
790
791
792
793
794
795
796
797
798
799
800
801
802
803
804
805
806
807
808
809
810
811
812
813
814
815
816
817
818
819
820
821
822
823
824
825
826
827
828
829
830
831
832
833
834
835
836
837
838
839
840
841
842
843
844
845
846
847
848
849
850
851
852
853
854
855
856
857
858
859
860
861
862
863
864
865
866
867
868
869
870
871
872
873
874
875
876
877
878
879
880
881
882
883
884
885
886
887
888
889
890
891
892
893
894
895
896
897
898
899
900
901
902
903
904
905
906
907
908
909
910
911
912
913
914
915
916
917
918
919
920
921
922
923
924
925
926
927
928
929
930
931
932
933
934
935
936
937
938
939
940
941
942
943
944
945
946
947
948
949
950
951
952
953
954
955
956
957
958
959
960
961
962
963
964
965
966
967
968
969
970
971
972
973
974
975
976
977
978
979
980
981
982
983
984
985
986
987
988
989
990
991
992
993
994
995
996
997
998
999
1000
b59fc7d7 (HEAD -> master, origin/master) When checking for empty strings, use 'not' instead of 'empty()'.
fa88ed53 Correction to handling of exclusionary list member indexing
3671de18 Remove dead code
bf3bb0e5 Update README.rst
485429d7 double the buffer size
d6e4b94c workaround underflow? verbosity+
5426d918 some lint stuff
cea87758 Remove non-ICU code paths as a consequence of issue #501
27b1eadb Preserve sigma set when converting to and from OL formats
a0099ea5 (origin/pmatch-arc-compression) Merge branch 'empty_pmatch_alphabet'
dc158fe2 Replace "use-character-classes" option with "unicode-character-classes"
30069af7 Merge branch 'pmatch-unicode-character-classes'
db931052 Blank alphabets in RTN transducers
ff935594 Don't waste time parsing redundant alphabets
0aa7d314 (origin/pmatch-unicode-character-classes) Add caching mechanism to avoid calling expensive unicode functions
ba777b01 Ignore empty lists
82f5e042 Merge branch 'pmatch-unicode-character-classes' of https://github.com/hfst/hfst into pmatch-unicode-character-classes
d31667f9 Fix copypaste errors
65380472 don't escape ] in xerox mode
47a9ee20 Correctly indent warnings about nonsubtractable symbols
e9b82181 Warn about subtractions that probably break due to metasymbols
c55ff9a4 Add new unicode special symbols to is_special_symbol(string)
63b12e04 Added runtime ICU versions of more character classes
97540c3c An implementation & use of character class UnicodeAlpha
50ee623c Improve warning about nested context conditions
08716576 Improvements to reporting of function call evaluation in verbose mode
3de9f4ef Improvements to verbose reporting of compilation steps
dff92692 An implementation & use of character class UnicodeAlpha
82c3b87e Merge branch 'master' of https://github.com/hfst/hfst
cecdaec1 Added a potential new symbol, UnicodeAlpha
9b1dbccd skip sub-readings of partial unks
9432431c use ?? as unk sign too
fe9a89b2 force ICU, fixes #501
8f4a23a5 oops debug pritns
4de5a50d logic fail
7cb2eca7 blah
dcafd64d warn before reweighting unweighted argh
848956c0 hack to restore composition middletape
3b238265 Drop leftover checks for C++11 headers
db1da471 SVN is obsolete, refer to GIT instead
f5df65e4 Fix memory leak in TransducerAlphabet: use virtual destructor.
cfb5dbc8 Use empty method for std::string.
fcb7eb91 Drop support for TR1 standard assuming that C++14 is minimal requirement.
65296280 Use range-based loops in HFST transducer. (#550)
8bea3e68 Drop read_prop_line because it is a duplicate of add_prop_line.
432c2016 Also run action for PRs
bda85b7b Cleanup XFST compiler (#548)
abc9de56 Use unique_ptr for input/ouput stream memory allocation (closes #547)
e2e9c95b Fix memory leak in hfst-lexc-compiler
fd5008c7 Fix compiler warning: return constant string.
9eba5b60 Fix memory leak in FST type formatting.
417b9494 Fix memory leak in hfst-strings2fst tool.
3511d26a Fix memory leak in XFST substitution operation.
84bf68b1 Fix memory leak in hfst-summarize tool.
a0c8654e Fix memory leak in SFST transducer.
e34c7e66 Fix memory leaks when transducer is redefined in XFST compiler.
1916181b Fix memory leak in print DOT and modernize for-loop.
4e9363e9 Fix memory leaks in PC-KIMMO print.
e467c8c3 Fix implicit fall-through in hfst-optimized-lookup
f09ddc48 Fix memory leaks in unit tests. (#533)
fa2ca7f4 Drop legacy wrappers for foma and lexc. (#532)
eaf901eb Use upstream Foma; Use C++17, or C++11 with bundled OpenFST; Use SSE where available
66f047d1 Fix segfault in hfst-summarize: skip first input symbols if FST is empty.
ad6854dd Newer OpenFST don't `using namespace std`
76b00004 Fix segfault when Foma sigmas are merged. (#530)
e4b5bbc5 Update external links for sfst, foma and flex.
734ae775 Use upstream OpenFST.
3977c5c1 OFTC
3486e40c Fix segfault when disjunct is applied to hude transducer.
727c1387 Fix segfault in Foma flag elimination.
1c47bbce Add lex/yacc sources to dist (see issue #524)
76dc60fd Fix compiler warnings for SFST error reporting functions.
d174a7a9 Fix memory leak when ranges are released
7317f7e8 (tag: v3.15.5) Merge branch 'master' of https://github.com/hfst/hfst
70c48438 Release 3.15.5.
fd7f7770 allow relative paths in source command
f68fb11f Drop string copy function because it leads to stack corruption and is not used.
1de31ce6 Fix memory leaks when input/output streams are not released.
25becf42 Fix memory leak when parallel rule mismatch error is detected.
f8d464dd Fix memory leak in command line arguments parsing code of hfst-xfst.
cede7aa7 Fix memory leak in are_equivalent method of FomaTransducer.
afb3cfa6 Fix alloc-dealloc-mismatch (malloc vs operator delete) for function name variable.
49050967 Fix memory leak in foma transducer: avoid double allocation of sigma.
63caf314 Fix memory leak in XFST and XRE compilers undefine method.
badaa605 Fix link typos
4e73272c Fix memory leaks in LexcCompiler detected by address sanitizer.
c4349fa4 Avoid memory leak in XfstCompiler::write_stack: free outstream after usage.
2a454e3c Avoid segfault when regexp parsing fails in lexicon.
fef85e4f Snip flex version warning as all supported platforms have new enough flex (closes #508); Python 3.x
7b8eb265 Merge branch 'master' of https://github.com/hfst/hfst
6d967ad5 Modify tracking of entry points to allow for RTNs in Captures()
a2741510 truncate long names, fixes #488
2a51e9dd Initialize input_mark_symbol (works around https://github.com/divvun/libdivvun/issues/43)
3b91edb0 hfst-tokenise: skip modifier letters in is_cg_tag
1eafb28f (tag: v3.15.4) Intended license was LGPLv3+ so formalize that, and explicitly mention that the SFST-derived files are used under LGPLv3+ agreement (closes #494)
1a29cf31 Change https://kitwiki.csc.fi/twiki/bin/view/KitWiki -> https://github.com/hfst/hfst/wiki (fixes #452)
4a10952e Allow hfst-xfst to source from absolute paths
f7c208b4 Properly open input files for hfst-expand-equivalences (#495)
be76d456 Release 3.15.4; Import Foma patches with license change to Apache-2.0 and minimal other fixes; Allow regeneration of Foma files so the generated files can be omitted; Explicit license file for SFST
6128b1c8 (tag: v3.15.3) accoring -> according
94b48cb5 Release 3.15.3
ed49cfde Merge pull request #456 from hfst/440-lemma-vs-tag-vs-non-composed-single-arc
5b231fce (origin/440-lemma-vs-tag-vs-non-composed-single-arc) ${ICU_CPPFLAGS} to AM_CPPFLAGS
437e9eb3 Use ICU if possible to find characterBoundary in is_cg_tag
c78084b3 escape all \ for cg output fixes #491
fd35fa6e whitespace only
8bc6c247 bump reqd automake in autogensh
4e2e8be7 Merge pull request #489 from mr-martian/expand-help-string
143098ce I promise I can count
321bc2d4 don't follow cycles in example
0d8e021e clearer help text for hfst-expand
5dadce48 try newer automake travis
b01f4220 git tries to modify line-endings in binaries
6a4eeaf7 sedded the object references
4748ae87 add subdir-objects fixes #486
b7fef31f Merge pull request #471 from ftyers/free-insert
a40be44f theres some bug in eliminate flags that overflows
1d19c46d Priority union (#480)
eabf0408 Realign (#468)
4d6b15ad Parse wordbound blanks as normal blanks in hfst-proc (#478)
a140e286 Travis macOS Homebrew just doesn't work
452b7289 Some files should keep crlf
342e98a8 Fix Travis; Add Github Action; Force text and eol=lf
088e48ac Fix Travis; Add Github Action; Force text and eol=lf
c1a4887a Fst2txt union (#475)
b2129ff2 improve handling of input based on hfst-substitute
323ea043 insert freely manpage
01c5f907 add insert-freely
265bc37a some fixes to hfst-proc (#465)
f7e1636d Add clarifying line to help message
78bcf55f Merge pull request #438 from mr-martian/master
a761da1a hfst-proc: escape apertium reserved chars in lemma
505aac3d (tag: v3.15.2) Release 3.15.2.
57dd2e18 Fix off-by-one bug in checking captures
95ec8680 Change Travis: ccache fails for Python bindings
dee52b33 Put Python bindings under autotools via --enable-python-bindings
356ddd30 (tag: v3.15.1) Fix Travis
b456e6e7 Release 3.15.1; Python 2 -> 3; Trusty -> Xenial
f1dc19aa Handle all-space-forms in hfst-tokenize, cf. #443
0a796763 fix typo in HfstBasicTransducer.write_xfst()
6a318906 HFST does not need nightly
12a16951 EL7 has ICU 50
5a25971d icu-config is gone in newer distros
99e606b9 doh, switched upper/lower here
7dbd2df3 test non-latin1 hfst-proc uppercasing
1c8aa88a travis: test icu config too
7d1f88f9 we need $(ICU_LIBS) here on mac
c1a18f60 go back to icu-config since .pc isn't in old debians
3fa86a80 Add $(ICU_LIBS) to LDADD to make it compile
5f0c4895 (origin/icu) no GLIB_CFLAGS/ICU_CFLAGS in hfst.pc
0250e35c use pkg-config instead of icu-config (deprecated)
490efdd5 implement icu handler in hfst-proc and hfst-pmatch2fst
7ad603a5 real configure.ac check for icu
16b1b381 Unicode support for pmatch casing functions, if available
de22d584 Include glib headers if we have them to support unicode
024c111d Fixes https://github.com/hfst/hfst/issues/429
c2f13c6e Adjust spacing of help message to better fit in 80 columns
ec14ee3b Fixes to help strings as per KP-2152
a3911243 Fix handling of DefInsed nets when precomputing allowed initial symbols
368f15b9 Sort PROPtags so they appear in consistent order
61413f63 Add tests
d05a7b73 New version
506337c6 Fix bug where finnish-tokenize was printing analyses too
4f72bb20 Set PYTHONIOENCODING environment variable
4c9ebad1 New package structure
1b601689 Add throw specifiers to destructors, one inheriting the other
d26cd6bb hfst-tokenize: drop trailing zeroes (except first) off weights
f09fc9bd Also mention python 2.7.
ae56dbaa Update pypi README.
f06b5f34 Support python version 2 in tests.
b89d26a7 Fix issue #409.
c75708cc Use hfst_commandline module in all python command line tools.
72c97f5b Rewrite output stream processing.
58f31fa2 Use module hfst_commandline for option parsing.
b2d63868 Define functions for output stream handling as well as single input stream handling.
49d3b339 Fix free parameter handling.
89938963 Use hfst_commandline module in all binary tools.
91d12188 Make hfst-compose use hfst_commandline.
0cb784db Handle input streams in a separate module.
5be41fb1 Improve command line argument handling.
701048a4 Rewrite comamnd line option processing.
0e26ecff pmatch: Make sure @PMATCH_BACKTRACK@ is not treated as special symbol either
e98973ad Define encoding.
9abdc7c6 Fix some things noticed when compiling with clang with c++11 disabled.
dd4420af (tag: v3.15.0) Ready for release 3.15.0.
ad302837 Do not print warnings.
7ab98bee Small fixes noticed when compiling on windows.
a76e3c96 Ignore hfst-eliminate-flags
1c8e5215 pmatch: Make INPUT_MARK not special_symbols, so we can refer to it in pmscripts
432203dd When verbose, print what goes into Like() operations
e80aadeb as_string() for symbols
93f3280d Allow negative weights
f1bc2475 Merge branch 'master' of https://github.com/hfst/hfst
da7bcee8 Remove "." from internal punctuation & adjust url rule
2ae7765f Option to print weights in location mode
35557c0d Allow both escaped and unescaped zeros in multicharacters in all lexicon entries. Fixes issue #274.
640b91d1 Use c strings with printf.
34cbfc3b Allow both escaped and unescaped zeros in multicharacters in one-level lexicon entries. Partially fixes issue #274.
cc258f5d Make link to hfst-apertium-proc only if proc is enabled.
14309ef7 Add tool hfst-eliminate-flags.
d69404d0 Warn user if transducer file contains more than one transducer in hfst-optimized-lookup.
37bdeded Expand @include"filename" directives as a preprocessing stage
732d645e Also reweight state that is both initial and final in hfst-reweight. Fixes issue #368.
6d9c495f Don't delete the implementation's FdTable
3048e03a Print warnings and error messages also in earlier stages of twol compilation.
a7f27143 Do not let distributed files depend on configuration.
cbf1e941 Update man files and include all of them in dist.
444c8860 Remove nonexistent files from dist.
59e314dc Add option --execute-and-quit (-E) to hfst-xfst.
b0a2be55 Merge pull request #392 from frankier/fix-python-lookup-leak
6a7ac658 Fix memory leak in Python .lookup(...) method
99dc9ffd Add returns in non-void functions
22448e08 Option --no-readline should be enough.
a6c923b3 Document datatype Location and function locate and make them visible in the API.
245ae4d6 Add missing file pmatch_tokenize.cc.
d6258bcc Fix incorrect use of delete; Fix missing virtual dtor; Remove some (void)
a1eeeada Fix typo
630fd7b2 Merge branch 'pmatch-implode-explode-fixes'
1afaca88 Make Implode() and Explode() allow symbol expansion
ae00e6b5 Preserve empty lines; Flush
f1ff7731 Merge branch 'master' into pmatch-implode-explode-fixes
295b1264 Don't try to collect initial symbols if initial state is -1
87e74a61 Merge branch 'master' of https://github.com/hfst/hfst
01cfc9f9 Fix internal punctuation case
9073e218 Name transducers for langname instead of langcode in these scripts too
8df796e7 Correct handling of markup markers
a6e96d32 No need for an allocation there; Fix final read
bb8e4d44 Be more verbose in READMEs.
4de9548a Add urns to metashare articles.
856902d0 Make implode and explode accept more than just bare symbolsw
3184e2ad Return a list of strings by default in tokenize.
18f88cc1 Use langname instead of langcode in transducer names
5f694239 Merge branch 'master' of https://github.com/hfst/hfst
61713be8 Fix indentation
b4f726d4 Replace containment implementations with correct ones
f7985c78 Add keyword arguments to PmatchContainer.tokenize.
75a69b35 Quote array expansions, otherwise they break on spaces
1d0d0620 Use * to concatenate array to string
f3b5acb0 Quote grep pattern so shell won't interpret
859de389 Convert dos2unix line endings
af328c48 Force type
8b297fd6 Protect line-based XML/HTML tags (WIP issue #379)
fe4c2f69 Modify tokenization implemented via PmatchContainer.
ceb5d2b0 Merge branch 'master' of https://github.com/hfst/hfst
d44f3947 Don't free data pointer until after we've evaluated everything
80a2bcd1 Add package data, i.e. readme and licensing information for each language package.
1c463a0c Add metadata for each language package for download directory.
683a5f4e (tag: v3.14.0) Ready for release 3.14.0.
e824094a Update morphology tests so that they pass after small changes in pmatching and tokenization.
48e3c363 Move hacky foreign conjugation guessing to unknown token part
85ef0616 Don't tag numbers after all
bcde7b96 One more weight typo
c815db6c Fix same typo again and merge changes from skeleton
ddac854f Foreign word conjugations
eb8efa4c Annotate numerical expressions
e980a909 Adjust indentation
12315805 Fix typo
9b8e3bb6 Merge branch 'master' of https://github.com/hfst/hfst
4b215fca Use language-specific patches when provided
9365b6c9 Bugfixes related to internal punctuation
ce5a05a8 Some language-specific customisations to the tokenizer
c685c412 Implement Ins() arcs inside right contexts
de9cc378 Fix to xerox output mode with plain, unanalyzed tokens
c4056f82 Normalize unicode apostrophies
4f2a4cec Rename morphology directory accordingly.
fcc122b3 Get rid of *new in new morphology tests.
d5e87dbb Rename old morphology tests as *old.
904647ef Tentatively add tests for upcoming new morphology scripts.
e019c23a Merge branch 'master' of https://github.com/hfst/hfst
86f98c8b When uniqifying without regard for weights, make sure to sort by weights
857624ec Return a skip value if tool is not found.
47ba60b6 Update morphology tests: omit deprecated .sh extensions and omorfi-analyze/generate. Do not include python tests in tool test.
3ab93036 Add missing file.
ffd17976 Add missing files to dist.
1032bf75 Explicitly list all functionality tests in dist.
90e161a6 Implement a cautious form of first-input-symbol scanning
9730a4e8 Correct variable name
a450d58b When not printing weights and in --unique xerox mode, suppress weight differences
4b05205c --uniq does not require an arg!
85f305ec Check for tokenizer, not analyzer
27ec41d8 Remove duplicate analyses
4295c847 Improve punct-handling
95f49df0 Add paragraph-tokenizing newline behaviour to --xerox too
82165de2 Migrate to using hfst-tokenize --xerox instead of hfst-lookup
0bef94da Not generating analyzers anymore
00a9cf93 Reflect new --xerox functionality
d0945dfd Don't use the fancy Omorfi tokenizer
393407e6 Nuke old contents of installable dir before proceeding
9863b099 Redirect stdout from command, not stderr..
7b4042d0 Check that root path was set
251108f6 Add & parallelization to Finnish
b470ec4e Fix tests for existing programs
746c069b Wait for everything to finish before making replacements
6f7ca8e3 Make test expect result of previous commits
ff232a41 Utility script for building everything at once
c02ab2df Check for programs and give info if missing
02986612 Unify blankline behaviour of --finnpos and --tokenize
9c45fc95 Add function HfstTransducer::get_initial_input_symbols.
633fee2a Rewrite morphology tests.
3fc0c433 Add stubs for collecting initial symbols
d1191f9b Handle default arcs in pmatch & fix spelling of initial symbols variable
cb6e3c39 Fix definition of unknown_token
51b9c411 Simplify main morpho-tokenization
6d08f5e4 Remove messy first-possible-symbol detection code, read it from hopefully precompiled info instead
05382326 Add clarifying comment
13a14243 Get a bit more speed by pre-reserving space for the vectors
efeb7039 Implement reading vector data from .bin binary format (much faster)
98e6697f Ins() the morphology check
1fa87847 When we have a dictionary word, don't override it
f6c33a59 When returning from a RTN, don't overwrite the original local variables!
6413d710 Add a script for testing the new morphology packages.
38af4eef Handle weight cutoffs inside pmatch
f35c5e28 Fix help message
28d8ce1a Optimized-lookup does work with hfst3 headers, so let's support it
9b53cb33 Add universal tokenizer
253d3e65 Disjunct empty string so transducers "know" about sentence boundaries
9d517a61 Improve usage notes
8b56ed6a Merge branch 'master' of https://github.com/hfst/hfst
132ff958 Link to github wiki instead of kitwiki.
f3858d05 Note about weights in existing morphologies
381bad57 Change default to weighted
13a43f44 Add notes about existing morphologies
241153bf Correct name of temp morphology for building tokenizer
3d2ef763 Add option to build installable dir in-place in specified dir
548a1370 Add options for capital casing and punctuation including
0e162d8c More elaborate tokenizer based on omorfi-tokenize
1949af28 General README
526cf4e5 Punctuation list
0e9d26d0 Merge branch 'master' of https://github.com/hfst/hfst
59df066e Add default tokenizer
0a0ebe2a Tentatively add match_and_print to python api as 'pmatch_tokenize'.
10e65659 Add some unicode quotes and relax the "don't know what to do" case
fc23a214 Try to guess whether we're at a sentence boundary or ordinal period.
334551e5 Handle colon separated conjugations differently to avoid size explosion
338b725a Cut out numerical parts of OmorFi to fix some bugs in it.
cbda6710 Improve dash, hyphen & range handling
07e6f8de Assume that omorfi handles uppercasing correctly; add xml tags
54bd3c88 Merge branch 'master' of https://github.com/hfst/hfst
8232010c Add introspecion mechanism to PmatchObjects; use for optimization
e0193229 travis: include macos
74da1a4f on macos, when bash is called as /bin/sh, echo prints '\x22' as "
c9eb7cb6 Use input symbols in lookup. Add a test for cases such as looking up string 'foo' in transducer [f:0 o:0 o:foo].
c71da223 Use only input symbols for tokenization in apply down. Add tests for cases where this matters, e.g. [f:foo o:0 o:0].
5da96bad Add one more test for compile-replace.
b1010cc2 Align regexp start and end tags with epsilons in compile-replace for better alignment between oridinary symbols.
220f031d Allow arbitrary symbols (except whitespace) in strings given as input to apply up and appy down commands.
8e154aed Implement variable 'retokenize'.
2db4eadf Add precompiled tokenizers
ca6b412e Fix typo
54be1901 Renamed to analyze-words and generate-words, for clarity
1a4d52c5 Add README
65994b0d Add missing files.
75fd90c9 Add function HfstInputStream::read_all and support an iterable object of several transducers in HfstOutputStream::write. Also fix some indentations.
7865f323 handle escaped @ as part of form, keep escape in output
693fe064 Add an extra parameter int max_number to HfstBasicTransducer::lookup, the default being -1 (infinite). Also fix a template issue with function compile_first of python API.
ba5ae812 Merge branch 'master' of https://github.com/hfst/hfst
d476a8ee Update documentation link.
715eee48 More loop protection
c8d197c4 Endless loop protection
384ecbd9 hfst-proc: more fixes and tests for serial unicode ranges
eff43225 tests: use $TOOLDIR, things are not installed yet :)
3720c0c6 travis: make verboser check so we can see what failed
a3445f47 hfst-proc: seems like the test for serial unicode ranges was flipped
0dbbcdb0 Add preserving "plain" omorfi-based Finnish tokenizer.
339ccc77 If compile_sfst_file is given an empty string as filename, read from standard input.
ae77b0a3 Add option -H. Add symbol-harmonization to python tests.
6bdcea82 Specify exceptions thrown. Add functions to the interface. Replace quoted values 'True' and 'False' with True and False.
917801d1 Improvements to python command line tools.
040d2e9c Enable python tools.
9f3d164f Add hfst-strings2fst.py to tests.
43eab05c Add a skeleton for hfst-fst2strings.py.
d54e180e Add hfst-substitute.py.
4286e36d Add implementation for lexc compiler. It still fails with one test case.
bb15ded5 Implement hfst-fst2txt.py.
89a0ee27 Add compose intersect tool.
26d67f0a Add pmatch tools.
549e2254 Implement python regexp parser.
29aabb65 Add tool for pushing weights.
132ba1c4 Add new python tools.
1a4ae938 Add hfst-subtract.py tool.
eb520631 Fix parameter handling and return values.
377d2ebc Add a script for testing python command line tools. Add option --prolog to hfst-txt2fst.py tool.
c5e6cd58 Add hfst-tail python command line tool.
7b584ae6 Add new tools.
2050ec67 Add new python command line tools. Fix a typo in hfst-compose.py.
ec3d25ab Return a skip value for python tests.
82238887 Tools hfst-fst2strings and hfst-head.py implemented.
5ec5cc74 Fix tests.
c5ca695c Fix scripts.
34e42a84 Fix rest of variables in tests.
e8de5ef1 Fix variables in scripts.
445d9e66 Modify scripts. Add a new python command line tool.
aae549f9 Add python versions of the tests.
bcd208d1 Add a python command line tool hfst-format.py.
82513574 Use string instead of char pointers in VarMaps and SVarMaps.
c14bf89f Check if foldername is the empty string and do not add a '/' in that case.
b58f63ed Print an error message if input is in optimized lookup format.
12b55059 update man page about --giella-cg tag assumptions
701b0328 Yet more python tools.
442d6399 Add new (minimal) python versions of command line tools.
95dc1f28 Add option --python to rest of functionality tests.
f77e97f9 Add more --python options to functionality tests.
6c9aa62c Tentatively add option --python to functionality tests. If the option is used, the tests are run with python API, if available.
b17c2864 Add a simple python implementation for hfst-calculate. Enable it with option --python in calculate-functionality.sh.
fd8dfea7 Keep separate RTN stacks for each depth level
29e9155d New morphology scripts
0f768266 Fix some typos, including 'kvargs' -> 'kwargs'.
c6686094 correct home page url of HfstPushWeights
1e2c797b Merge pull request #365 from frankier/fix-print-dot-labels
b0d3218d don't tokenize_multichar as default in --giella-cg yet
c6331179 put tokenize_multichar in TokenizeSettings; default true for --giella-cg
a1894737 Update finnish morphology script.
c187d5ae (tag: v3.13.0) Ready for release 3.13.0.
f6bc1cde -W/--no-weights overrides earlier -w (or -w implied by -g) options
cfa49ab8 Ensure print_dot labels are valid utf-8
cb9916d5 Add missing files.
c92bcaf1 Add missing files to dist.
d98ffa7b Initial commit of finnish-tagtools scripts Includes a few ones from FinnPos modified for our purposes
ee4662b4 Merge branch 'master' of https://github.com/hfst/hfst
d6a247e7 Added a variable "xerox-composition", default to "on"
355aaf9c allow creating HfstInputStream's from std::istream's
8ff49aaa Use libreadline in hfst-pmatch when available
b5b7995b [].with(X = Y) feature in pmatch This provides support in the pmatch2fst compiler to define "global flags" with the syntax [your expression here].with(X = Y). This will set up a flag diacritic readable from anywhere within the brackets, even if Ins()ed.
0858e633 Support getting a list of operations involved with a particular feature. This also involves making define_diacritic() redefine the meaning of symbols if called multiple times on the same symbol number.
4d4471de tokenize tests should pass again
f81dcb53 tokenize --giella-cg: assume FST handles space around tags
b07c47b9 Merge branch 'master' of https://github.com/hfst/hfst
60e5c288 Check for success of the now-working test suite.
acf3c785 Cleanup of remainder of the pmatch test suite Many cases of adding set "need-separators off"; many cases of fixing mistaken test assumptions; one case of fixing code; one case of removing test that was already in pmatch2fst-functionality.sh; one case of removing entire test because it was trying to do something impossible; one case of relaxing test assumptions that should be possibly revisited. Multiple FIXMEs left but all the tests now pass.
2669274a Make both-sided OptCap() be actually both-sided. It still doesn't downcase the continuation of words, which is possibly a bug.
64a18742 tokenize --giella-cg: remove as_cg_tag, just need is_cg_tag now
9decc11c Modify check for list symbols. It was conflicting with left markers for Xerox rules (@LM@), causing a test to very surprisingly break (it didn't even seem to be using optimized-lookup format at all..)
4e4ba7c9 Add tests for Lst() and Exc()
7a0176ac Avoid keeping an extra copy of the tokenizing dictionary in memory (fixes https://github.com/hfst/hfst/issues/362)
8c10e1df Consider list symbols (@L..." and "@X...") to be special
e4dff091 hfst-tokenize --giella-cg, is_tag: check if symbol Multichar_symbol
3ec59daa tokenize --giella-cg: non-verbose as default
862ea0ec Add C and C++ dlls to dist.
d6531234 Fix misspelling of nrc for nlc
c005a4aa Merge branch 'master' of https://github.com/hfst/hfst
86dcec64 Fix runtime handling of contexts and compilation of negative contexts. This was broken when the processing state stack was changed to accommodate "RTN safety". The local state is now pushed into a stack when entering a context and pushed again when exiting it, and popped when backtracking out of those situations.
10967c48 set need-delimiters no for more tests that were broken by it
06cbb209 Made Xerox output of unknowns follow Xerox
97ecf446 Merge pull request #359 from kartikm/patch-1
23dcc545 Fixed typo
134d5a5a Works around issue #358
21b0e820 Actually support multiple different Capture() names
be9e214f Instead of names, use pointers to the calling transducers for rtn calls
34bf445c Don't track returns from Ins() arcs in the call stack Pushing returns from Ins() calls to the call stack were clobbering nested calls. This is unnecessary, because we have the necessary information about the correct return location from the stack depth variable anyway.
1358576a Use hfst-pmatch --newline in the tests so we don't see irrelevant blank lines
e6f7067b Decrement recursion counter when returning from main loop due to no input
528952fa A somewhat hacky fix to a bug introduced by changes to RTN processing pmatch is lacking recursion depth with each RTN entered. This is mostly a problem due to eventually running out of depth and truncating results. For now we reset depth at the start of each input, which makes sense anyway, but the leak should really be fixed.
c45f620a In locate mode, omit blank lines even if in blankline mode if we didn't print anything
74f3a6fd Remember captures for the duration of the entire input. Also avoid copying the captured vectors around, instead passing iterators to input
01c2fd76 Fix short form of end tagging also disable automatic disabling of need-separators when encountering a context
f2b6c8c0 Merge branch 'master' of https://github.com/hfst/hfst
64528f78 In blankline-separated mode, keep blanklines in output too
b0a17a28 Merge branch 'master' of https://github.com/hfst/hfst
31f0767d Include pre-swig-generated wrappers to source distribution. Update README.
de22111c Remove bashisms that was breaking tests that apparently are not run with bash
368acefd Fix tokenize backtrack regression, need to handle @PMATCH_BACKTRACK@ explicitly
da0ab47e Fix test which was probably written with a misunderstanding of replace rules
ed5ac3bb Reinstate and fix some tests by moving them from pmatch-tests.sh (where escaping done by the testing script breaks some tests) to pmatch2fst-functionality.sh
882ee743 Allow term complement of named transducers
26865255 Merge branch 'master' of https://github.com/hfst/hfst
215a4829 Support backreferences (Capture())
eea79b86 Update README.
e1d519c5 Link to exact place in page.
27d3e1c1 Update README.
2e935163 For installation instructions, link to PyPI pages instead of KitWiki.
b68f1461 Update README.
7a3618ea Make Like() and Unlike() much faster by not sorting the whole vocabulary (by doing an insertion sort of the n best words) and only calculating each comparison key once
85f8a8cb Fixed binding of freely insert operator in hfst-twolc.
83eea8b0 Remove unnecessary (debug?) prints.
e10be33e Actually use other than ascii characters in att file.
5102d2d0 Fix some typos in test.sh. Replace epsilon with empty string in fst2strings.py. Read input in utf-8 format in prolog2fst.py, if possible.
394284b4 Fix a typo.
b52f348b Test if hfst-xfst exists after all options are processed.
cdaa748c Implement an equivalent python script for all command line tools needed in tests.
c6dae7a6 When using python for tests, also replace part of command line tools with equivalent python scripts.
88a70d33 Fix a couple of typos.
ef656676 Update python package version number. Use .cpp extension.
7bc8e180 Update README.
778ebcac Get rid of unnecessary CommandLine in python api. Fix some typos.
1c345ef9 Error functions are void, not int.
b8edb3aa Import sys.version_info before using it...
6a2107cb Use extension cpp for all c++ files in pypi package. Also use the c++ implementation of foma backend by default.
43190464 A couple more fixes.
3f63a4e7 Fix some issues noticed by clang.
27c30eed Add a c++ version of foma backend for testing.
31c91c39 Add arguments to function declaration.
0b468456 Fix a couple more typos.
a117e056 Fix some typos.
76048509 Improve test script and add expected results for cases where xfst solution does not exist but the result from hfst script is checked.
c4592072 Add two more python implementations for fsmbook tests.
c75cbb8e Merge branch 'master' of https://github.com/hfst/hfst
61be4541 Add Unlike() operation, which could be better named also add option for whether information about cosine distances should be included in the result when compiling word vector operations
87cb7d6a Two more python tests...
42ba171c Add two more python tests.
221f8dcc Add more tests.
e25e2d46 Add python test for FinnishNumerals
7d917a49 Allow empty string as input for hfst.fst and hfst.fsa and interpret it as epsilon.
3842501e Use xfst instead of python for slow tests.
5962b3a6 Add one more python test.
7b3dccbf Add python test.
dd5ed1f7 Fix broken test assumption
a45a97d8 Prohibit EndTagging printable-empty lengths of input
77d5c7b2 Fix single-character transducers not getting automatic context separators
432d16c9 Verbose message about automatic separators
7bf07dd2 Make using Ins() -arcs safe Ins() arcs are no longer locally greedy, so there is no semantic difference between network inclusion with Ins() or by explicitly embedding. eg. define greedy "a+"; now the following work the same way: Ins(greedy) {abc} greedy {abc}
adc3b0b5 Use temp weights rather than subtractions like in pmatch
1378513d Fix test by having need-separators off
849d2973 Minimize automatic delimiter contexts
97a7b050 Merge pull request #352 from unhammer/tokenise-lib
2fcc6659 Merge branch 'master' into tokenise-lib
ef026f10 Add option --silent to tests.
c0996a8c Add .0000000000 to tests; ToDo: Trim trailing 0 after the dot, then trim the dot; ToDo: Investigate where std::fixed gets disabled
1fe2af94 Wrap compile_twolc_file inside a class as TwolcCompiler::compile.
dbd0d14f Revert "Removed rounding"
813f2d39 Add keyword arguments to compile_twolc_file.
786af199 Removed rounding
702425ee Add twolc files to pypi setup.
30e3c4f9 Fix a typo.
e8acc1a5 Flush and close the stream.
2b26a3b0 Add tests for 'compile_twolc_file'.
b27f9932 Remember to reset also the second parser.
7fb07f52 Add a possibility to reset twolc parsers between reading several twolc files.
251cef5e Add option --local-hfst to setup.py.
87a77e75 Add a tentative implementation of twolc compilation.
45676091 Make code clearer and add documentation.
87382726 Keep track of weights along context checking paths and unify weight handling
539592b7 Merge branch 'master' of https://github.com/hfst/hfst
31950dea Add beam mode to result filtering
532925d4 Make sure TOP has the right name if it's eg. loaded from a @bin""
dc9bf4e9 Make sure the HfstTransducer name and the container names match
5289eb35 Fixes issue 353. Unbelievably, the return value for a weight-returning get_weight() was bool, which after casting mostly worked until it ran into negative weights.
2bfb55c0 Fix problem with rtns that are referred but not really present
2dd77678 Improve help messages of scripts.
15484805 Add hfst-specific options --with-c++11 and --without-c++11 to setup.py.
f2854007 Use HFST_THROW macro instead of plain throw for HfstExceptions.
4be886d5 If USE_FOMA_EPSILON_REMOVAL is defined, use foma for epsilon removal in minimization for unweighted TropicalWeightTransducers to make it faster.
7b03fd00 Add an option --verbose to test.sh.
5463306b Fix the way unordered containers are used on windows with python2.7.
48de6fdd Give pythonpath as first optional argument for each test, except for stream tests that take it as a second optional argument. Also specify coding for each test.
dbb055f7 round weight to zero decimals, non-sci, in cg/giellacg vislcg3 doesn't (yet) accept floating point weights
8d389257 Use by default c++11 unordered_map and unordered_set, unless otherwise specified via definitions (INCLUDE_TR1_UNORDERED_MAP_AND_SET and USE_TR1_UNORDERED_MAP_AND_SET).
31cd16c3 Merge branch 'master' of https://github.com/hfst/hfst
8f17d4ec Add boolean CPP_STD_11=True to control c++11 support.
baec6872 Move tokenize to correct alphabetical place.
194b9cca Couple of fixes to vc 2008 compilation without c++11 support.
38ad5946 Fix issues noticed when compiling without c++11 support with VC 2008 on Windows.
89ddca2c Merge branch 'master' of https://github.com/hfst/hfst
8121be60 Fixed bug 341 (priority union)
318fdb58 Add an option --without-c++11 (defaults to 'no') to compile hfst without c++11 support.
d3a5c5da c++ standard is set in configure, do not set it here
ee84e29d make hfst-tokenise usable as a lib; include simple string→string fn
fbf8d49a Update documentation and setup.
9617edb0 Add a quick fix to missing 'strtof' in msvc 2012.
b22626ed Add a function cross_product that takes an iterable object of transducers. Document other similar functions.
b1fbf8bc Add simple help messages.
195ef733 Add a function 'compose' that takes a list of transducers. Comment out a debug print.
8cb5ee5a Skip calculate_funtionality.py test until it is fixed.
b9aa6186 Add a script that copies files needed for creating a distribution for windows.
72b14638 Add a script for generating cc and hh files from yacc/flex sources for windows. Update copy script for windows. Change flex token ECHO to ECHO_ to prevent collision with flex/yacc macro with the same name.
0b61a94a Make script faster.
06a6c9dd More fsmbook python scripts.
88cc5bf8 Add a simple --help message to test.sh. Add python versions of two tests.
36490188 Add new python scripts.
3ec8d35b If --python is requested in test.sh, use python API also for hfst scripts that have an equivalent python script.
0ebc87e6 Add options --python and --pythonpath to test.sh which enable compiling xfst files also via the python API.
7266f932 Add an example for HfstBasicTransducer.remove_transition.
f2a547fe clang requires libc++ and osx version >= 10.7.
9a60444b Add function HfstBasicTransducer.remove_final_weight and document it. Also improve documentation of HfstBasicTransducer.remove_transition and HfstBasicTransducer.add_transition.
a2dbe8eb Add function HfstBasicTransducer::remove_final_weight.
9950d90f Add option --restricted-mode (-R) to hfst-xfst. If it is requested, Write and read operations are allowed only in current directory (i.e. pathnames cannot contain '/' or '\') and system calls are disabled.
059b5b02 Add tests for hfst-xfst's 'negate' command.
cdab3f74 Add a function HfstTransducer::negate() which is equivalent to [?* - A] (where A is the transducer) with the exception that flag diacritics are treated as ordinary symbols. Use the function to implement 'negate' command of hfst-xfst.
4001b451 Allow complement/negation only for automata.
a741bdc5 (tag: v3.12.2) Ready for release 3.12.2.
56534b34 Add missing file.
cc2dea54 Merge branch 'master' of https://github.com/hfst/hfst
22383de8 Check more carefully which backends are actually enabled.
4cebb756 Actually remove commented code instead of making it visible.
c811b357 Remove commented code.
357ab0fe Remove commented code.
f9796b35 Update README.
60742c42 Add extra compile option -std=c++0x.
25ce84e8 Reapply changes in commit de59747. Require at least automake 1.12, unless building from pre-yacc-generated sources.
750a3cae Revert changes in commit de59747. Allow automake older than 1.12.
0ae7b1e9 Use std::unordered_map instead of hash_map.
0b3a248f Use unordered_map instead of hash_map.
8b4b1d4b Skip lexc wrapper test.
d11b00e3 Add a simple exception handling mechanism.
02e16cbd Fix a typo.
de597474 Stop supporting automake < 1.12. Warn that build will fail unless building with pre-flex-generated sources.
e776ecc1 Remove commented code. Remove unnecessary variables.
8b30936a Skip hfst-train-tagger and hfst-twolc-loc tools in version and help message testing.
ba28c397 Add missing file to dist.
b29e140c Remove lexc wrapper tests.
4033c44e Disable lexc and foma wrappers unless explicitly requested. Fix a typo in --enable-expand-equivalences.
50351323 Actually enable hfst-calculate and hfst-xfst by default as promised in commit 244b9b5...
31fc7c70 Add missing condition.
5191c886 Add option --with-openfst-log=lean to configure which supports only reading, writing and converting log openfst transducers.
30d3792a Actually check if --with-sfst=lean is configured when compiling sfst functions.
94eda73a Add configure option --with-sfst=lean which supports a limited set of sfst functions (reading, writing and converting between formats).
5df548e3 Add missing const to definition, remove second argument of HFST_THROW.
214ab27b Add function 'has_weights'. Make function 'get_profile_seconds' public.
a27f845c Add function 'get_profile_seconds' for profiling foma back-end.
1b64f33b Add function 'has_weights' and make function 'get_profile_seconds' static.
244b9b57 Remove duplicate of AC_ARG_ENABLE([xfst]). Enable hfst-calculate and hfst-xfst by default.
e47f0100 Remove commented code. Make sure that code compiles when openfst backend is disabled. Add comments to some functions.
d0a1e91c Use iosfwd header instead of iostream when possible. Try to include only header files that are actually needed.
389679db Clean temporary files generated by tests. Add some files to be ignored.
610572fd Merge branch 'master' of https://github.com/hfst/hfst
f66f396c Check for nested context conditions, ignoring the inner ones
d8db5525 Update man pages. Remove man pages of tools that are no longer supported. Update copyright year.
62da656e Disable hfst-twolc script and hfst-train-tagger tool unless explicitly enabled with --enable-twolc-script and --enable-train-tagger.
e369608c Add files to be ignored.
142bfca5 By default, implement hfst-twolc as a single program and disable the script (can be controlled via --enable-twolc-script).
98e08f62 Get rid of hfst-twolc-system tool. The script hfst-twolc will soon be replaced by a single program.
e30202c2 Update windows scripts according to recent changes in twolc processing.
d6760437 Update files ignored by git.
ae820a5b Revert to earlier version of htwolc-main.cc which calls twolc parsers directly instead of TwolcCompiler class. The latter approach sometimes causes a segmentation fault.
d96b441c Define warning and error streams in TwolcCompiler constructor.
9c0cca49 Throw an error instead of calling exit(1) when parsing input.
b820f11c Handle twolc error and warning streams.
e116a74c Add TwolcCompiler class and use it in htwolc program. Move CommandLine under libhfst/src/parsers.
9a190880 Move most twolc code from tools/src to libhfst/src/parsers.
0705bec9 if no --superblanks, newlines won't be in [], so need to print on unblanked newlines
13512574 --giella-cg: only treat superblanks if given --superblanks
efa0964f Update list of files ignored by git.
19fd1d1b Add an option to use htwolc in twolc tests. Disable it by default.
59cfdce9 Rename twolc files.
a6029ef4 Rename twolc parser and lexer files. Move functions under namespace hfst::twolcpre[1|2|3].
5844621d Actually exclude htwolc from installation.
7661b389 Add an alternative implementation of hfst-twolc which does all processing in a single program. Exclude it from installation until it is properly tested.
52d52573 only warn for backtrack-on-substrings-without-analyses if --verbose
3936fdbe Control output of htwolcpre1.
0cc7582e Change names of extern twolc variables and add functions to access some of them.
f9a69b21 Add missing new file hfst_sfst_extensions.cc.
d5baf5ad Separate twolc parsing and command line tools to their own files.
035b6aba Use separate prefixes in twolc lexers and parsers.
d3c4de92 Handle '^<' and '^>' operators in lexc regular expressions correctly. Should fix issue #346.
61653471 Space-separated mode
144a3f34 Ignore generated sfst source files.
dc906026 Merge branch 'master' of https://github.com/hfst/hfst
9f7208bf Improve pmatch compilation error handling, still leaks memory at exit
3767b513 Add new function 'compile_sfst_file'.
512245b4 Merge branch 'master' of https://github.com/hfst/hfst
4649b83f Reorder and add norm caching
460cded6 clearer error message for when there is backtracking into substrings that have no analyses
0705c3cf Apply recent changes in filenames and locations.
3560d0b6 Move sfst compilation from tools/src to libhfst/src/parsers.
bc49e55b Fix namespace and class names.
9f356e7a Change file and namespace names.
3f8d4573 Small fixes to sfst compilation.
c28d2c53 Further separate sfst compilation and command line program.
0ccd6fe7 Reorganize sfst compilation into separate compilation units.
b9b85850 Fix xfst and twolc tests. Use rm -f when removing files that are conditionally created.
52d3cbf2 Remove all exe extensions from hfst-twolc and hfst-twolc-loc because they are scripts.
99d37e68 Remove exe extensions from hfst-twolc and hfst-twolc-loc because they are scripts.
b9b678ee Start separating sfst parser and command line program to their own files. Add a prefix 'sfst' to flex/bison functions and variables.
97338125 Rename HfstCompiler, hfst-compiler and hfst-scanner to SfstCompiler, sfst-compiler and sfst-scanner.
ea9fb6ba Add a test for hfst.regex using 'definitions'.
2a7ad84e Add a keyword argument 'definitions' to regex that takes a dictionary mapping transducer names into transducers.
6c761284 Add regexp operator documentation also to web pages.
2322ecc8 Modify regex documentation.
31559550 Tentatively add documentation about regexp operators.
77b64445 use ccache if possible, slightly faster travises
1ab384d4 Update fst_to_fsa and fsa_to_fst. Support both HfstTransducer and HfstBasicTransducer formats.
663224d8 Catch exceptions when compiling pmatch expressions. Add new tests for pmatch functions.
43e06449 Do not exit on pmatch compiling errors, throw 1 and catch it in command line program.
0556161b Merge remote-tracking branch 'refs/remotes/origin/master'
51b811ea Small math fix + revert to float, which wasn't causing problems after all
e1c9af03 Modify fst_to_fsa and fsa_to_fst.
aaf2f8b7 Support special symbols and single-character symbols without separator in fsa_to_fst. Update also documentation.
ebea326d Add simple tests for fst_to_fsa and fsa_to_fst. Also test modifying transitions of an HfstBasicTransducer.
ecb78d05 Rewrite HfstBasicTransducer's transitions() and state_and_transitions(). Tentatively add an implementation for functions fst_to_fsa and fsa_to_fst.
a65ca944 Remove HfstBasicTransitions and HfstStates from HfstBasicTransducer and use hfst::implementations::HfstBasicTransitions and hfst::implementations::HfstBasicStates instead.
4579f7d6 Give the pythonpath as command line argument for tests that use 'from __future__ import' instead of catenating it to the beginning of file. The import must happen in the beginning of file.
3dda726a Rename HfstTransitions to HfstBasicTransitions.
68155b3d Add docstrings and dummy implementations for functions fst_to_fsa and fsa_to_fst.
4a2814dc Document the changes to api.
3d1a1ce3 Use the version of HfstBasicTransducer.transitions() returning a reference. Add HfstBasicTransition.set_weight to python api. Add tests for modifying transition weights.
17e00271 Fix a typo.
04accecf Add help message to copy-for-windows.sh and comments to foma wrappers.
3688695f Typedef and templatize vector math, reformulate as per article description
95c022c6 Add help message.
1cd8cb85 Add help message.
dd2f2302 Fix vector reading for case where lines don't end with a separator
5bcb93db Support hfst.compile_xfst_file only for python3.
49959535 Fix a typo.
20f39cdc Fix copying files on mac as well as pip packaging issues.
7f600058 Update file instead of deleting it...
49b0ef97 Tentatively also support python version 2.
52016b22 Actually copy all files.
01744fa3 Omit sdk directory on windows if compiling for python3.5 or higher. Do not redefine _MSC_VER. Do not call subprocess on mac until it works correctly.
b44f91d5 Update README.
f6779722 Check if we are compiling with visual studio 14.
b17eb3d7 Update PyPI documentation.
a6ca66bf Add missing file.
410d6ef9 Get rid of unnecessary libraries.
f1572e1c Merge branch 'master' of https://github.com/hfst/hfst
75fcdd90 Fix chaining of functions in example.
d6552290 Minor version bump to combat repo inconsistencies
91539f01 oops, reset variable
55664266 add transliterate output mode
416373d4 Update documentation.
b5736dcb Add missing file.
43db07a0 Move ReplaceType under hfst.xerox_rules. Reorganize some internal functions.
bc8983e7 Move python code from swig interface file under module hfst.
3aff1428 Check if readline package is available.
391f73ab Do not keep input to interactive commands in readline history.
68e1b878 Update description.
0ff98211 Support apply up and apply down in hfst.start_xfst().
c687c097 Document return values of compile_xfst_file and compile_lexc_file.
12da4aef Warn user about missing readline support if --with-readline is not requested. Issue an error if --with-unicode-handler is called as such or with a value of 'yes' or 'no'. Make it possible to disable loading entries from shared object (openfst back-end feature).
9731e75a Update version number to 3.12.1. Handle links to future and earlier releases so that they are easier to update. Add a README.
764399e4 Remove doc/libhfst.py as documentation has been moved under directory doc/hfst/.
f3df0b9f Add more documentation for XreCompiler. Fix links to installation instructions.
d2666d6e pre-increment (thanks, cppcheck)
57120541 Add the script for creating python package. Convert README to dos format.
4abf93e7 Update package scripts.
4888fc6a Add a script for creating package which contains 64-bit python bindings for python3 for mac osx.
18df4685 Add a script fpr creating 64-bit python bindings for python 3.3 and 3.4 for windows.
47909865 Add README files for python packages for mac and windows.
3519e4c2 Add a script for creating the lgpl package.
0cb9fb53 Add alternative files for making LGPL release of HFST.
fea67231 (tag: v3.12.1) Add missing files to dist. Do not print termination message when performing tokenize tests.
65962bf8 Ready for release 3.12.1.
77c845dd Fix flag elimination bug (reported in issue #342).
c827bb62 Do not call fsm_destroy until issue with 'double free or corruption' has been fixed.
3b0e58a2 Provide a get_current_dir_name() for platforms without one
b0fb7a67 Merge branch 'master' of https://github.com/hfst/hfst
65a4d799 Search for included files under scriptdir, not working dir. If an included filename is non-absolute, prefix it with the directory the script was in when trying to read it. This only happens in non-Windows environments for now.
f5e94d73 Free allocated memory.
6144e75c tokenise -g: check for eof
cf46e69d Apply more changes from latest release of foma.
a9f8397b Apply some changes from newest release of foma.
cc8c96ba Make a separate directory python/pypi for creating pypi packages.
479789e3 Rename the pypi package to 'hfst' and use a four-number versioning scheme. Also define '__version__'. Add an empty config file as hfst c++ headers include it.
eb304f0f tests: trying sed instead of awk for exit on <STREAMCMD:FLUSH>
798383dc tokenize: handle Apertium-style superblanks in --giella-cg format
7922399a Provide a way to scale the vector similarity calculations via variable
166d418d Merge branch 'master' of https://github.com/hfst/hfst
9bc5315e parse word2vec -style text format and assume first line is info
5d3f6b8f Do not allow unescaped dots in regular expressions.
ce5e2a5f Merge branch 'master' of https://github.com/hfst/hfst
a8f21f0b Observe HAVE_SFST and HAVE_FOMA when including back-end header files.
8a9c0dec nwords catenate-style argument for Like() function
1433e453 Merge branch 'master' of https://github.com/hfst/hfst
1ec8e03e Add experimental two-vector model for word sense
72d4f72c (tag: v3.12.0) Ready for release 3.12.0.
da866477 Read vector models and do simple ranking
42df833e Small fixes to memory leaks and efficiency.
e2abd896 Fix some memory leaks.
da54acca Fix leaks reported by valgrind.
92cf4dc9 Fix more memory leaks.
9bd97498 Fix some lost memory issues reported by valgrind.
6e129a4f Remove unnecessary variable. Delete transducer definitions and latest regexp when destructor is called.
d7003095 flush given outstream, not necessarily cout
8d4f8249 Add a destructor to XfstCompiler.
6b10aabc Update README.
863e94aa Add missing files to dist and remove generated ones.
e4c13049 Rename hfst.rules into hfst.sfst_rules. Fix hfst.types into hfst.ImplementationType in tests.
80f18a82 Use single quotes instead of double quotes in docstrings.
52a9d75c Add test for hfst.xerox_rules.
00b020c0 Rename xerox_replace_left and xerox_restriction back to replace_left and restriction in package hfst.xerox_rules.
d1556dd0 Fix documentation of parameter 'output' of function 'extract_paths'.
3df4a48c Sentence separation for finnpos-mode
d3058c74 Issue a warning if automake is too old, which can make build fail if pre-generated yacc files are modified.
9166cdc5 Move ImplementationTypes and ReplaceTypes to their own classes and get rid of package hfst.types.
6d6c6315 Fix datatypes used in for loops.
c6aa5e16 Fix some typos in documentation.
8073675b Use renamed versions of xerox rule functions.
ae70056c Tentatively add HfstXeroxRules to python API in package hfst.xerox_rules.
dee0176a Fix some more warnings.
37edbbb9 Fix minor warnings generated by clang.
faa76472 Fix typos in ifdef guards.
41c5a32c Add a simple script for fetching statistics about hfst downloads.
d9d80771 Update pypi setup.
822f4bb8 doh, forgot to add test expected output
cccba226 hfst-tokenise giella-cg: fix indentation issue with backtracking
5a262692 Fix still some warnings.
32c82b30 giella-cg backtracking: ensure ignored spaces are still output
03d6a237 Do not warn about pragma push/pop issues or unsafe cstring functions on windows
9c12094a Use hfst::hfst_fopen instead of fopen. Fix some warnings related to datatype conversions.
d64cb49a Add function hfst::hfst_fopen that uses fopen_s when compiling with msvc.
5ed27eae Fix some warnings caused by conversions (possible loss of data).
b93a1442 Do not use extern templates when compiling with msvc.
73f7e18e Remove files generated by the tests.
61ac4f83 Add a batch file version of test.sh for testing the python bindings on windows.
b53a128a Fix warnings.
7cb8740c Update pypi setup script.
cc9cca46 Remove unnecessary back-end include directories as well as commented code from Makefiles.
fd5f0074 Fix some minor warnings.
ec114ea8 --giella-cg support for @PMATCH_BACKTRACK@ symbol (#337)
15fbb1fc Update windows script.
097826c1 Remove extra comma from initialization.
d6b001d7 Add missing header file.
9b3c1227 Fix warnings generated when compiling with -Wall.
a0dee642 Fix most of the warnings generated when compiling with -Wall.
89b20612 rename --gtd to --giella-cg, and more descriptive help text
335df14e Add support for compiling on windows.
85585c4b Fix a typo.
e628f556 Add a manifest file for pypi packaging.
48c34ef0 Tentatively add a setup file for creating pypi packages.
6abde490 update to proc to make it possible to weight surface forms
5e379fc7 Update windows compilation script.
1f55a514 Define int64 for msvc.
21f52d0e Implement snprintf for msvc compilation.
c05469e2 Replace definition _Bool with Boolean, as _Bool conflicts with typedefs in msvc header files.
00bbb1d2 Get rid of unnecessary typenames and typos noticed during compiling HFST on windows.
b160c07d Forward declare back-end implementation datatypes and wrap their destructors.
067c3c92 Add missing <iterator> header.
2f0c8dff Add missing std:: to vector.
34e99ad6 Issue an error message if glib unicode handler is requested but not found.
d28075c3 Declare all functions that are needed in start_xfst().
ec34dea1 Add math.h
4a3be6b7 Make preparations for supporting OpenFst version 1.5.4.
c9940f65 Include HfstBasicTransducer.h instead of HfstTransitionGraph.h.
8560638b Add extern templates.
dc969538 Remove unnecessary OpenFst headers and use forward declarations instead.
699c0985 Wrap StdVectorFst and LogFst destructors as both classes are forward declared.
7f81f41b Forward declare as much as possible when using OpenFst library. Also remove unnecessary include directives and add missing std namespaces.
a0b2fb35 Prepend missing 'std::'.
c8137fac Make HfstBasicTransducer and HfstBasicTransition fully implemented classes instead of generated template specializations.
eb5b877a Improve the way error messages are handled.
b8278ec5 Change 'import libhfst' to 'import hfst'.
2b9e44b6 Move implementations of HfstTropicalTransducerTransitionData away from header file.
ce05c672 Merge branch 'master' of https://github.com/hfst/hfst
3ec0d09d Alter expected result to conform with newly expected behaviour
759605c7 Do not allow newline characters inside quoted literals.
f296b6bd Merge branch 'master' of https://github.com/hfst/hfst
63d7544b Allocate enough memory for XRE error message.
f968ebfa Trying out facility for communicating sentence boundaries to plain tokenization mode
6a78c4c2 Merge branch 'master' of https://github.com/hfst/hfst
bf2ded3e Don't print size info when unverbose
5a33841f Don't separate tokens with empty lines in plain tokenization mode
d0a1b550 Warn about undefined multichar symbols when compiling lexc in verbose mode.
5d2c67d8 Fix a typo in option handling.
e2fee814 Tentatively add an option --do-not-minimize to hfst-regexp2fst which performs determinization instead of minimization. Also make hfst-xfst use determinization if variable 'minimal' is set to 'OFF'. Implement this by adding function hfst::set_minimization(bool) to HFST API and a function HfstTransducer::optimize() which performs minimization, or determinization if hfst::set_minimization(false) has been called. Replace calls to minimize() with optimize() in parts of HFST code relevant to the tools mentioned.
82aa26ec Get rid of CR-LF newlines.
44f2e212 Remove commented code.
94b76073 Print an error message if output type is defined several times.
966a8954 Fix mismatched free/delete reported by valgrind.
6dc1d8f4 Do not use strdup with -std=c99.
407a5178 Use delete_foma instead of delete/free when deallocating memory used by a foma transducer.
07508b50 Limit to C++11 for now
d15b4be3 Use free with malloc, not delete.
b693b3af Add a function 'void LexcCompiler::reset()' and use it in XfstCompiler when reading lexc input again.
bc5ac275 Remove revision info from output of hfst-info as it doesn't work correctly in git.
ace48016 Pass cascade argument by reference in lookup_cascading functions.
e2ea4cb1 Fix a typo in function 'strip_percents' which resulted in incorrectly escaped zeros.
92a7152e delete more lines
9f940a35 remove osx lines from yml
f7bd0858 Unobservable change - nothing happened here
d1bf79b0 close #336: off-by-one segfaults in OlLetterTrie
2574e27a Avoid using projections in casing functions (which cause minimization issues with large weighted transducers).
73a3ff0a Parametrize optionality in casing functions
0adc1f93 Make sure there's no case fallthrough
58eb35db Fix other casing functions too
c366c4cb Another fix to Cap()
704c1c7d Merge branch 'master' of https://github.com/hfst/hfst
5a5df2d7 Change operation of Cap() and make timing per-object
44f0ae7a Fix missing empty lines in output when cascading composition is used.
284f9d0d Add an option --max-number=N for OL transducers which limits the number of results. If not defined, defaults to 5 for infinitely ambiguous input and to infinity for finitely ambiguous input.
5686864b There are some problems in using the bindings with python version 2, so refer only to python3 in README.
5a17fb02 Add a missing 'break' to switch block, making variable infinite_cutoff work correctly.
4c2f26fb If no results are found, print it only once.
d703776c Tentatively add support for --xfst=print-pairs for OL transducers.
b7fcab93 Remember to flush the streams also when no results are found.
7ba32cf8 Merge branch 'master' of https://github.com/hfst/hfst
bc7a80a5 Implement a pair-lookup for optimized-lookup transducers.
ce3133fd Add a fourth argument of type hfst::ImplementationType to the constructor and a function get_type() to ImplementationTypeNotAvailableException.
4ffe105a Print to console on windows if print-pairs is specified.
587f3c04 Do not print extra newlines in cascading composition.
43538943 Fix a bug/typo in string comparison.
72de7821 Improve documentation in HfstTransducer.lookup.
0c888e9c Rename HfstBasicTransducer.lookup_fd to HfstBasicTransducer.lookup and add an argument 'obey_flags'.
d5493aca Rename HfstTransitionGraph's lookup_fd to lookup and change some parameter names to be more descriptive.
56a8d2f6 Add instructions for linking hfst c++ library statically in comments.
67e25ffd Add missing file to windows scripts.
8db14ce0 (tag: v3.11.0) Ready for release 3.11.0.
fe37aa00 Update morphology checks.
8d3f162b Update EXTRA_DIST.
6dac2464 Do not use binary transducers in tests.
ee62c447 Add missing doc file.
a3676454 Mention that python tests are no longer available.
da2678db Skip Python tests as they can now be performed in the folder python/test.
ea47f0b1 One more stray memory leaking cache
95e3480c Memory conservation: only cache named objects and always minimize them.
6b113b24 Merge branch 'master' of https://github.com/hfst/hfst
974d2403 Turn off automatic delimiter-contexts ("set need-separators off") in the presence of user-defined contexts
01fccfd5 Use docstrings.i instead of swig's autodoc. Document the contents of hfst and hfst.exceptions modules.
4322ca72 Add a docstrings file created semiautomatically from doxygen documentation.
816ce6c1 Add at least a simple docstring for all functions.
82c8f6a5 Merge branch 'master' of https://github.com/hfst/hfst
1b11fbff Obey -i option for giving the pmatch archive filename
4620b1b7 std=c++ works on OS X but not Linux, so disable Linux+Clang
d245a745 Test if gnu++ appeases Travis' clang
3d9c66b5 Include needed m4
5e73db33 Copy hfst-ospell's C++11+ detection
9af66d0b short int -> short; long int -> long
daa457e9 Yet more documentation..
e1b01ff0 More docstrings.
711a0898 Add docstrings.
acf56964 tokenise: don't skip output-empties
b5b4434c tokenise: don't let empty analyses affect weight classes
18f5a2b6 travis: fix indentation
62d216a6 getopt_long() returns an int, not char; this is important because char is unsigned on many platforms which means char(-1) is 255
39d28d1f Do not check if lookup is infinitely ambiguous if --time-cutoff is defined.
c42af376 Remove redundant flag diacritic check. It was once removed, but got pushed there again.
b7b58825 Add support for --cascade also for optimized-lookup transducers.
9169e38a Support combination --cascade=composition --xfst=print-pairs.
51a0b61a Merge branch 'master' of https://github.com/hfst/hfst
ef6ed81d Fix the way cycle limit argument is passed.
9ad09503 Merge branch 'master' of https://github.com/hfst/hfst
6d94c906 Add a preliminary omorfi-friendly conll-u mode
45a0bf29 Remove redundant filtering of paths containing flag diacritics. Flags are handled already in function HfstBasicTransducer::lookup_fd.
5c7135ac Implement option --cascade.
18d5e66e Tentatively add option --cascade to hfst-lookup.
1dcad09d Print error message if --xfst=obey-flags is used with optimized-lookup transducers. They always obey flag diacritics.
670ee39d Merge branch 'master' of https://github.com/hfst/hfst
f903d9cf When doing apply up or down in hfst-xfst's, check if 'obey-flags' is ON or OFF.
8e9c6e65 travis: --enable-silent-rules
a253b8d4 Add an argument obey_flags=false to is_lookup_infinitely_ambiguous functions.
14bdd9c7 Make is_lookup_infinitely_ambiguous aware of flags. Use typedef hfst::StringVector in HfstLookupFlagDiacritics to prevent conflicting definitions in hfst-twolc which defines a class StringVector.
349fb735 Add option obey_flags=false to lookup_fd.
fdf912ab Add tentative script for converting from doxygen documentation to docstrings.
7618907d Modify function visibility.
bc01c0bc Control visibility of functions in HfstTransducer.
b535aa6f Follow foma and xfst convention for operator ignore '/', i.e. expand any identities and unknowns before inserting the strings to be ignored.
c1b43412 Add an argument 'force' to HfstTransducer::harmonize which makes the function harmonize also FOMA_TYPE transducers.
724b4e95 Obey variable 'print-sigma' when print_net is called.
6bec5076 Add brief descriptions for hfst packages.
ca892d14 Document the implementation types.
98764da7 Fix bugs in options --xfst=show-flags and --xfst=obey-flags when non-optimized lookup is performed.
93b9fe27 Close the outstream when done. Not doing this was causing a bug where some buffered data never got written to the output file.
7d971066 Obey variable 'print-pairs' when performing apply down in non-optimized format.
ac2fc3ab Add parentheses in conditionals of function 'flag_build' to ensure correct interpretation. Also remove redundant code.
28867c16 Allow punctuation as context in the "need-separators on" mode
0cd05ee8 Add "need-separators no" to test script to reflect changed default in pmatch
25977996 exclude gcc from osx builds
7dfa8ec8 Track used symbol names
81128c23 Allow lowercase define()
52a4ca45 Merge branch 'master' of https://github.com/hfst/hfst
af71662c Add side arguments to casing functions
7c1b3886 Revert back to the original extra_link_args in python bindings.
2d5c670a Rename examples.py to test_examples.py. Perform tests in a more logical order.
4c8a1e30 Update documentation about linking to the HFST C++ library.
9f8ca0f6 If --enable-no-tools is used, skip tests in test/tools/.
41dc9a27 Mention module hfst.types.
a8c3e566 Remember to apply modifications to weight pushing to tests and hfst package.
835f0efc Get rid of TO_FINAL_STATE and TO_INITIAL_STATE and replace function 'push_weights' with 'push_weights_to_start' and 'push_weights_to_end'.
d05b9dbf Wrap HfstTransducer implementation types in package 'hfst.types'.
5e1a2456 Add missing files to dist.
58d46022 Update README. Handle linking to extension library better.
de888b6f Update README. Handle cases where an implementation type is not available. Add missing files to dist.
1a68126f Fix a typo in URL.
7a2296be Update link in README.
861baf6b tokeniser: --weight-classes=N opt, as in hfst-proc
151f1fa4 Add functions lookup_optimize, remove_optimization and copy to HfstTransducer.
4efc9af7 Make HfstTransducer() create the empty transducer.
0f237c17 Comment out hfst_ol::set_extract_tags_mode(bool). Improve error messages in att and prolog readers.
57426cae Remove unnecessary commented file.
b7139cde Fix HfstException documentation.
3b13bf58 Merge branch 'master' of https://github.com/hfst/hfst
7e6d27c9 Fix a typo.
9f4ceb76 Merge branch 'master' of https://github.com/hfst/hfst
829c9e21 Implement remaining variables in the library and support them in hfst-pmatch, with the exception of need-separators which has to be specified during rule compilation, at least for now.
457c5de5 Document AttReader and PrologReader.
8695c47c Add HfstException::what() to python api.
2cffffac Add an alias what() for operator() to HfstException.
5fdf61a1 Add class PrologReader and tests for it.
77eda555 Add a new class AttReader and tests for it.
18edac59 Add package hfst to setup. The package is essentially a wrapper around libhfst module which is still available as before.
e41f131e travis: re-encrypt channel to #hfst
6329bb89 Use all implementation formats in tests.
5a5018c6 Do not use default_fst_type as pmatch transducers are always in ol format.
b38f87bd Add stream tests.
ccaea0e0 Represent empty transducers as empty lines so that read_att_transducer will work correctly.
fe66b184 Remove unnecessary files.
bf5511d5 Forgot to do git add before committing...
3710f91b Update test examples.py and add it to test.sh. Remove extra files.
7d5064de Add directory tree for package 'hfst' and its subpackages 'exceptions' and 'rules'. This package encapsulates module libhfst.
7818c1ee Delete unnecessary file.
91b86a0e Rewrite part of tests. Document the interface file.
657013d5 Add new tests.
3b4c1867 Merge pull request #323 from unhammer/tokenise--gtd-tags-are-multichars
67125ab1 rm unneeded import
d927d795 pmatch: store input/output_symbol_strings, _parts refer to them
c665f8a8 Add missing endverbatim tag.
884d1dcc Add missing file.
bc87fc58 Get rid of HfstFile wrapper.
30cde028 Try to preserve the names of transducers when converting.
90b33287 Expand prolog tests. Fix a small bug in naming transducers.
8e61537f Forgot to update testfile.
63a04567 Add a new function for reading prolog input and improve the way att input is handled.
9ef38f7d Split part of code related to reading att input into a separate function. Change the way transducer name is set when reading prolog input.