Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Party Divisions #514

Open
wants to merge 16 commits into
base: main
Choose a base branch
from
Open

Party Divisions #514

wants to merge 16 commits into from

Conversation

jetilton
Copy link

I am beginner/intermediate python user and have never gone through a pull request, so please correct me if needed. I didn't see any data with regards to the party divisions of the House or Senate. I wanted that data and thought I would try and add to this ecosystem if it is wanted.

This is the party divisions from the house with leadership roles by Congress. Two files have been added, a scraper for all congress links found http://history.house.gov/Institution/Party-Divisions/Party-Divisions/
and the resultant data in a yaml file.

@JoshData
Copy link
Member

Hi there. Thanks for your interest in improving this repository!

There's a lot of information in this pull request so there's a lot to comment on.

  • It might be best to split up the different sorts of data into multiple files: party divisions (+ total membership), party leadership (i.e. leadership positions held by members of the House), and officers (i.e. non-members like clerk and so on).
  • Party leadership might be best not in a separate file but instead merged into the leadership_roles field in the legislator files. This would require mapping the names to the existing records for legislators, and then mapping the Congress numbers to the right entry in the legislator's terms. Wherever the data goes, mapping the names of legislators to their IDs would be important and perhaps a requirement before merging this in. Because this is the hardest thing to do, it might be useful to open a pull request that only adds the party division data and leaves the rest for a separate PR.
  • The Congress numbers (e.g. 100th Congress (1987–1989)) should be turned into integers (100) and the Congresses should be in forward chronological order (to match the order of the legislator files).
  • The party divisions and total membership values should be integers instead of strings.
  • There are some white space issues ('Democrats ': '188').
  • Instead of naming the file party-divisions-house.yaml, it might be better to have a single file that will eventually hold information for both the House and Senate.
  • The bottom of the table on house.gov has some footnotes so we'd want to make sure that information is captured somewhere.
  • Before merging, some documentation should be added to the README.

@jetilton
Copy link
Author

Understood, thanks!

@jetilton
Copy link
Author

How are you envisioning a single file for house and senate?
My preference is to keep it {house:{{1:data},{2:data},{3:data}}, senate:{{1:data},{2:data},{3:data}}}
That way I can have a house scraper and senate scraper since they are in different locations and combine them in a single dictionary afterwards and do one yaml.dump . It seems easier to deal with if one breaks and makes for smaller scripts.

Thoughts?

@JoshData
Copy link
Member

It would be a lot clearer from a user's point of view to do it by Congress and then by chamber within each Congress, but it's not hugely important.

@jetilton
Copy link
Author

Hi Josh, I made the majority of the revisions you requested, but I have not tried to merge the party leadership.

Some things I would appreciate you taking a look at are:

  • congress 107, the senate has some funny dates going on. It isn't very clean right now, but I want to know how you would like it before I try cleaning it up.
  • In some of the congresses it lists which major party the third party caucused with. You can see how I handled this in congress 92 and others. Is this acceptable?

Thanks,
Jeff

@JoshData
Copy link
Member

Hey, I'm sorry I dropped off, I just don't have the time right now to help on this. :(

@konklone
Copy link
Member

I'll try and take a look at this this weekend.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

3 participants