SE2006:group bar:minutes

From Earlham Cluster Department

(Difference between revisions)
Jump to: navigation, search
 
(43 intermediate revisions not shown)
Line 121: Line 121:
==Mar 14==
==Mar 14==
-
* Today we worked on the stubs for the scraping API (scrapi) and had some preliminary discussions about how to create the SQL schemas and connect them to this API.
+
* Today we worked on the stubs for the scraping API (scrAPI) and had some preliminary discussions about how to create the SQL schemas and connect them to this API.
-
* Tomorrow we are planning to work more on this API and improve its relations with our SQL schemas.
+
* Tomorrow we are planning to work more on this API and possibly improve its relations with our SQL schemas.
 +
 
 +
==Mar 15==
 +
 
 +
* We worked more on scrAPI and added more regular expressions into it.  There are some new ideas emerging about some new data types.  It seems that we are making good progress on scrAPI. :-)
 +
 
 +
* We have found some more standardized websites that allow us to automatically scrap them.  Moreover, we also tried to find ideas on how to scrap some of these websites that we have found.  We have communicated them to each other either by e-mail and/or by talking.
 +
 
 +
* Next meeting we are going to try to incorporate the structure of these new websites we have found into our scrAPI design.
 +
 
 +
 
 +
==Mar 27==
 +
 
 +
* We examined some code done and made decisions about upcoming parts.
 +
 
 +
* Next days will be work-on-code days that will happen in XP style pairs.
 +
 
 +
* Created [[SE2006:group_bar:todo|Todo List]]
 +
 
 +
==Mar 28==
 +
 
 +
*[http://zips.sourceforge.net/ The Zip Code Database Project]
 +
*[http://65.39.85.13/google/default.htm Google Maps + US Census Information = Analysis of your Area]
 +
*[http://maps.huge.info/ Maps.Huge.Info Mapping Programs]
 +
*[http://www.gcensus.com/ gCensus]
 +
**FAQ talks about how the data is organized.
 +
**I like the "Totals for viewable area"
 +
*[http://mapki.com/wiki/Main_Page Mapki]
 +
*[http://www-sul.stanford.edu/depts/gis/bookmark.htm bookmarks]
 +
* Created [[SE2006:group_bar:sources|Sources List]]
 +
* http://raws.wrh.noaa.gov/roman/
 +
 
 +
==Mar 28==
 +
 
 +
* We worked on creating regular expressions for scraping data.
 +
 
 +
* We did code clean-up.
 +
 
 +
* Until friday, we will be working on the regular expressions and try finding ways to place our scraped data onto the database.
 +
 
 +
==Mar 31==
 +
 
 +
* We are working on practically everything, tomorrow is the first due date.  Examples of what we are doing includes but not limited to:
 +
** Creating the classes and regular expressions needed for the classes.
 +
** Debugging existing classes
 +
** Error handling
 +
** Documentation
 +
** Code clean-up
 +
** Going crazy
 +
 
 +
==April 3==
 +
(Toby standing in for Aybars)
 +
 
 +
* reassessed our roles in the group: currently we all do a little of everything and we like that.  will remain in current "roles."
 +
* discussed code licenses: we will adopt the [http://www.gnu.org/copyleft/lesser.html LGPL] for the API (and probably the [http://www.gnu.org/copyleft/gpl.html GPL] for the UI)
 +
* we divided upcoming work into 3 projects:
 +
** geocoding API - Colin & Toby will incorporate support for group fu's data store
 +
** user interface - Kevin will start researching possible alternatives to Google maps (both web- and java-based)
 +
** API cleanup and data source additions - Alex & Aybars
 +
*** Alex interested in veterans vs. house value overlay
 +
* Toby will write tests for his code (Scheduler, SourceDefinition, etc.)
 +
 
 +
Questions for class tomorrow:
 +
* what metrics and what dimensions do our customers want in the final product?
 +
* what should the interface look like?
 +
** what kind of data displayed in what way?
 +
** what kind of controls?
 +
* do we need to test each data source individually? (probably)
 +
 
 +
==Apr 4==
 +
 
 +
* Worked out intra-group communication problems.
 +
 
 +
* Found some new sources and found some sources about nuclear power plants.  The last one we found was the most comprehensive, so we will probably end up using it.
 +
 
 +
* Examined Dave's Perl codes and analyzed the output data files (the *.DAT files etc).
 +
 
 +
* Talked about what connections we have between our sources and what connection there should be in the upcoming ones.  We are developing a theme for our sources.
 +
 
 +
* Talked about what to do next.  It seems that more or less all we have is one or two more scrAPI parts to create on one or two more new sources we find around a theme/topic, write a front-end for Dave's geocode, some debugging, a few more documentation and we are mostly done!
 +
 
 +
 
 +
==Apr 5==
 +
 
 +
* Have found more or less all the data sources we need.
 +
 
 +
* Started scrAPIng the new data sourcees we have found.
 +
** Nuclear power plant source
 +
** Wind energy source
 +
 
 +
* Tried to get Geo-coder working
 +
** Waiting on the other group for the geocoder database
 +
 
 +
==Apr 10==
 +
 
 +
* Geocoder ideas (Colin & Toby):
 +
** use http://geocoder.us/service/csv/geocode for city & address geocoding (waiting on e-mail from services@geocoder.us)
 +
** use Dave's database for geocoding states, zip codes, and counties
 +
*** see postgresql get_shape_group function
 +
** create a local cache of at least the geocoder.us data, possibly including Dave's data (to simply access at runtime)
 +
** standardize column names for location data for source tables (e.g., (lat, long), state, county, zip code, city, address)
 +
** write functions that can generically group entities in a data table at different summary levels based on the above column names
 +
** for tomorrow: create geocoding API stubs for Alex
 +
 
 +
* Data formatter (Alex):
 +
** for tomorrow: create XML schema for markers & polylines (to be fed to UI)
 +
 
 +
* Estimation of an average time left per person to be submitted on Tuesday.  According to our statistical extrapolations we have a gigantic amount of more work to do.
 +
 
 +
==Apr 11==
 +
 
 +
* Created a brain-storm-like-map where we created the user interface on board.  We discussed a range of problems and solutions.  Topics included:
 +
**The user interface design
 +
** XML
 +
** Tomcat
 +
** ScrAPI to Tomcat connection
 +
** Tomcat to user interface connection
 +
 
 +
==Apr 12==
 +
 
 +
* We split into groups and:
 +
** Developed the user interface design.  We are trying create:
 +
*** Website intra-communication where different parts of the website can change the other parts on event calls
 +
*** Create the basis for connection with Tomcat server
 +
*** Nice design
 +
** Streaming database insertion
 +
** Worked on the Tomcat server
 +
 
 +
==Apr 17==
 +
 
 +
* Tomorrow is the due date for the beta version, everyone is pretty much stressed out.  We worked on everything, which was mostly the user interface and debugging.
 +
 
 +
* We worked on database and database-user interface relation.
 +
 
 +
* Debugging on scrAPI and some efforts for better standardization.
 +
 
 +
==Apr 19==
 +
 
 +
* We have created a list of tasks, and decided on their priorities and how much more we need to learn to accomplish those tasks.
 +
 
 +
* We created a bunch of design ideas for the user interface.
 +
** How will the data be displayed?
 +
** Where will it be displayed?
 +
** How will a bunch of data be displayed?
 +
** What will be the displaying options?
 +
 
 +
==Apr 24==
 +
 
 +
* At this time, most of our work focuses on debugging.  Most of our ideas of the big picture are implemented, but a lot of things work incorrectly or insufficiently.  Today we concentrated on:
 +
** Testing
 +
** JavaScript related problems (these problems are bugging the group deeply) and understanding JavaScript abnormalities
 +
** General debugging
 +
 
 +
==Apr 25==
 +
 
 +
* We worked again mostly on debugging (we are trying to get this mostly done until our next deadline, thursday morning).
 +
 
 +
* Debugging list includes:
 +
** Bug #2: "Save source metadata in database"
 +
** Some test files
 +
** Objects within JavaScript
 +
 
 +
==Apr 26==
 +
 
 +
* Tomorrow is the last deadline before the actual submission date and the presentation.
 +
 
 +
* We are overstressed and working on getting things done.  Our list includes:
 +
** User Interface, especially the "Regional Stats."  Being the middle box in the right, this gives summarized average information about the points in the screen.
 +
** Some JavaScript debugging
 +
** Some general debugging to get it generally ready
 +
** Metadata/database stuff
 +
 
 +
==Apr 30==
 +
 
 +
* Tomorrow is the actual submission and presentation day.  Today is the last group meeting for software engineering.
 +
 
 +
* We are doing some last minute polishing stuff and getting ready for presentation.  Our software seems mostly ready for presenting and is open to further development.

Latest revision as of 02:12, 1 May 2006

Contents

Feb 15

Possible meeting time: Working M/W/F in the evening for 1 or 2 hours starting the 27th. Toby, Kevin, and I are going to be working this weekend on clustcomp projects and then going to a conference on Feb 22-26.


Feb 28


Mar 1

Mar 6

We compared three of our Kerasotes lab scripts. We apparently all wrote them in PHP. The best innovations we found were Toby's data point definition abstraction and test suite. In essence, the data point definition abstraction allows one to describe a data source and data points that one would like to harvest in a general manner. The test suite is a basis for more general testing of the software that we will write. Also of note, were some of our findings as far as regular expressions. For instance, as a group we learned how to make a regular expression not greedy and how to not save a group (?).
Attached you will find an image of our general model thus far. We have already incorporated some of the ideas discussed in this session, including a more general data point abstraction and streamlined data flow between the programmer API, database, and WUI. As part of our plan, we intend to use Java as the base language, XML for temporary data storage and transfer, PostgreSQL or MySQL for data storage, and XHTML Transitional and JavaScript for the WUI.

Mar 7

Mar 8

Mar 13

Mar 14

Mar 15


Mar 27

Mar 28

Mar 28

Mar 31

April 3

(Toby standing in for Aybars)

Questions for class tomorrow:

Apr 4


Apr 5

Apr 10

Apr 11

Apr 12

Apr 17

Apr 19

Apr 24

Apr 25

Apr 26

Apr 30

Personal tools
Namespaces
Variants
Actions
websites
wiki
this semester
Toolbox