Skip to content

DataArchitecture

Jump to bottom Edit New page
netrc edited this page Dec 3, 2012 · 19 revisions

Table of Contents

Structure

Church

  • name, address, latlon, mainpic, note
    • Might want a "cityShire" field
    • mainpic ?? maybe just a "thumbpic" reference?
    • mainpic: or ?? other pics are from pic item 'index' value back to church name
    • note: the note will include the pamphlet link, if any; not a separate field
    • note 'theory' below, does not include links to brasses or rubbings....

Brass

  • name, church (namelink), location, year, tags, mainpic
    • church: n.b. back link from brass to church name
    • location: place inside the church

Rubbing

  • vlcnum, brass (namelink) name, status, desc, location, date, mainpic, note
  • status - ok or unknown (used in DataQA page)
  • desc - description of the rubbing; colors, framed/uncut; size
  • note: things like style, inscription, framed, condition, foil are in the text of the note. Again, not separate fields. Should be able to search for standard name, e.g. "foil", "knight"

Note

  • name, tag (category?), date, (versionNum), data
    • tag is a category, e.g. About, blog, brass; (n.b. don't need an index here)
    • name is the 'link' back to church, rubbing, brass.
    • category of About doesn't need a name field value
    • category of Blog uses 'name' field as title of blog entry
    • could be immutable and just add new versions.... (one reason why this is a separate collection (and separate DB wrapper functions) so we can have standard API for getting latest version, getting raw markdown/getting HTML, etc).

Pic

  • name, tag (category), index, thumb, full
    • tag is a category (Brass, Church), index is the key (name) back to the category item
    • tag can include "carousel" ? can we search for one str in multiple tags?
    • could have a caption??
    • might have a category "Carousel" for the front page carousel. On the other hand, want the front page to load very fast

Use Cases

Searching

?

Sorting

?

Blog

Uses Notes. Page should show all Blog posts (using "infinite scroll"). Latest 5 (or so) should be expanded fully. These Notes would have "category" == "Blog" 






Theory






MongoDB uses a document structure - a database consists of collections of documents. Each document can be thought of like a RDBMS row, as the document can have many fields/values. However, the fields can be arbitrarily set per document, and can consist of any type, size, or nested structure. 



For VLCB, our collections and documents are semi-structured.






    

  • 1-to-N relationship is held solely by "backrefs" from each N to 1
    • 

  • 1-to-1 is held solely by "forward ref" (meaning e.g. from Brass to mainpic)
    • 


Seems like the right thing to do is put values like "note" and "pics", which
are really related to just one thing (a church or brass) into their enclosing doc
(the church doc or brass doc). But that makes updating cumbersome - when changing
a picture's owner, e.g. a church, you need to update the old owner and the new owner
doc. By separting into a quasi-"normal" form, you can update in just one place.


One-way relationships, such as from Brass to its note could well be implemented as a HAS-A containment. But I think that there's easier separation of functionality for notes (such as markdown/html generation) from keeping all the notes together. This also helps with getting simple lists of data - returning a list of Brasses will be very lightweight; going back and getting a note for a given Brass, one at a time, seems sensible.



Note that this does mean that we will use the MongoDB characteristic of indexing a lot of each "document". (!! How exactly does this work?). We do expect to want to get a fast response when asking for all (or really just the expected one) notes that have to do with Brass X.



Other idea - a separate "relationship" collection, tying everything together (like composite keys). But still, keeping these indexings straight is an extra layer of work, complexity.



Other idea - keep misc data in a key-value generic collection. For example, "Carousel" key (doc) would contain list of pics for the carousel. But is this easier/better than a field in Pics? 



Note that MongoDB has man/reduce, which could be used to find relationships. But, this is a relatively small database, no sharding/single node. The point of map'ng queries across nodes just isn't the scale for this use.





MongoDB Notes











Impementation Notes





db2.js - file of database access routines.



Locations table. In a truly RDBMS way, I'd have a separate table for "Locations" - RC/basement, Jodi, etc. Just not interested. Keeping this as simple string. Could implement a static drop down list.
Performance: did a test with returning a MongoLab/DB object with _id as is, and _id deleted from properties. The _id property encodes as a very long JSON string; deleting it seems to help performance.
A simple GET rest/Church/St.%20Peter's


    

  • with _id : 20-80ms
    • 

  • with _id property deleted : 3-8ms
    • 



              


              

                
                   Add a custom footer
              

          



  
          

            

              

                
                   Add a custom sidebar
              


            
Clone this wiki locally