Package refactor (#47)

* Rename token types
* Switch to govend for dep management
* Remove Line and StartPosition types
* omitempty for Line and StartPosition in node.go
* Fix travis build / Add travis-ci badge
* Update TODO.txt

ALL THE TESTS PASS AGAIN

:sparkles: :grin: :sparkles:

.gitignore

@@ -1,7 +1,6 @@
 !*
 *.test
 c.out
-pkg/
-bin/go-rst2html
 .cover/
 *-session.vim
+refactor*

.travis.yml

@@ -1 +1,6 @@
 language: go
+before_install:
+  - go get github.com/mattn/goveralls
+script:
+  - go test -v ./pkg/...
+  - goveralls -service=travis-ci -v -package ./pkg/... -repotoken $COVERALLS_TOKEN || true

README.rst

@@ -2,8 +2,8 @@
 go-rst - reStructuredText for Go
 ================================
 
-.. image:: https://drone.io/github.com/demizer/go-rst/status.png
-    :target: https://drone.io/github.com/demizer/go-rst/latest
+.. image:: https://travis-ci.org/demizer/go-rst.svg?branch=master
+    :target: https://travis-ci.org/demizer/go-rst
 .. image:: https://coveralls.io/repos/github/demizer/go-rst/badge.svg?branch=master
     :target: https://coveralls.io/github/demizer/go-rst?branch=master
 .. image:: https://goreportcard.com/badge/github.com/demizer/go-rst

TODO.txt

@@ -1,43 +1,18 @@
-* Mon Jan 23 21:50 2017: Remove custom json similarity checks and used jsondiff (jd)
-
-  The lexer/parser test checker is overly complex at about 2000 lines. We need to make sure the json output of the
-  lexer/parser match the expected results and that's it. This can be done with the json diff library.
+* Sun Feb 26 14:09 2017: Convert test checks to use testify package
 
 * Sun Feb 14 22:38 2016: Fix gocyclo on https://goreportcard.com/report/github.com/demizer/go-rst
 
-* Sun Jan 22 02:45 2017: Add MarshalJSON() to all Nodes in node.go
-
-  otherwise diff output on errors shows an index number and not the type!
-
-* Sun Jan 22 00:26 2017: Use travis.ci for building grst
-
-  drone.io went belly up, beta.drone.io doesn't allow me to login...
-
-* Sat Jan 21 23:50 2017: Split up the parse package intq lexer, parser, and document
-
-  * lexer package contains all the files starting with "lexer_"
-
-  * parser package containes all files starting with "parse_"
-
-  * document package containes all files starting with "node_"
-
-  The parse directory is becoming too large because it contains three packages...
-
-* Fri Feb 19 23:52 2016: Rename *Node to Node* in node.go
-
-  NodeSystemMessage becomes SystemMessageNode
-
-* Thu Oct 20 00:41 2016: CURRENTLY WORKING ON FINISHING GOOD TESTS FOR INLINE MARKUP
+* Thu Oct 20 00:41 2016: IMPLEMENT GOOD TESTS FOR INLINE MARKUP
 
 * Tue Oct 11 00:10 2016: Document creating new tests
 
   * The UltiSnips Snippets should be mentioned.
 
   * document debugging tips
 
-* Mon Oct 10 17:53 2016: sectionLevels.Add() should do one thing
+* Mon Oct 10 17:53 2016: sectionLevels.Add() should be renamed to AddOrReuse()
 
-  In parse.go#399 a section is added or existing is returned. We should rename this function from Add() to AddOrReuse() for
+  In section.go#145 a section is added or existing is returned. We should rename this function from Add() to AddOrReuse() for
   self documentation.
 
 * Mon Oct 10 01:44 2016: Need to add blockquote levels
@@ -60,9 +35,3 @@
   From the spec:
 
   Other whitespace characters (form feeds [chr(12)] and vertical tabs [chr(11)]) are converted to single spaces before processing.
-
-* Sun Feb 28 09:00 2016: renumber-items.sh should only take the test name, it should intelligently find the test and
-  renumber.
-
-* Sun Feb 28 09:21 2016: If I run a lex test directly then "startPosition" is set to 0. It should always be specified in the
-  items.

doc/refactor.rst

@@ -0,0 +1,54 @@
+================
+The Big Refactor
+================
+
+-----
+About
+-----
+
+Deep flaws in the lexer/parser design were discovered when trying to integrate test 06.00.03.00
+
+1. tokenizing is done one slice at a time, instead of one unicode code point at a time.
+
+   This makes it difficult to handle unicode combining characters.
+
+   A workaround was attemted by trying to normalize the input text, but this is not ideal. Mainly because of decomposition;
+   '\u2000' decomposes to '\u2002` when normalized with the NFC form (`Unicode Normalization Forms`_), which according to the
+   unicode spec are functionally identical. For a document syntax such as reStructuredText, this is not good because we want
+   to preserve the original intent.
+
+#. The project directory structure is becoming unweildly.
+
+   It's hard to understand the "flow" of the library. This will make it hard to gain new contributors.
+
+   Also, having separate files for `lexer_*` and `parser_*` add increased complexity.
+
+#.
+
+#. Plugin support (directives)
+
+   Need to start thinking about this...
+
+   The https://github.com/golang-commonmark/markdown repository has a great design.
+
+----
+TODO
+----
+
+This might take a while. Let's plan.
+
+1. rename files to remove "parse\_" and "lexer\_"
+
+#. Use lifo stack for lexer/parser state instead of saveState()
+
+#. Lexer should use scanner.scan()
+
+   * Should loop by unicode code points
+
+   * Remove the need for norm.NFC
+
+#. remove EOL (utf.RuneError)
+
+#. remove the "parse" directory
+
+.. _Unicode Normalization Forms: http://unicode.org/reports/tr15/

lex.go

@@ -0,0 +1 @@
+package rst

parse.go

@@ -0,0 +1,5 @@
+// Package parse is a reStructuredText parser implemented in Go!
+//
+// This package is only meant for lexing and parsing reStructuredText. See the top level package documentation for details on
+// using the go-rst package package API.
+package rst