Entering Back Data

Now that all of our existing sheep are in LambTracker and the 2014 lambs’ data is only in LambTracker I started seriously working to enter in the 16 year history of the flock into the database. I’ve discovered a number of issues and developed some workflows that seem to be making the process go much faster. This post will be updated a bit as I get everything documented.

I started LambTracker with the assumption that you would not have to enter in all the history before you started using it. This is good for allowing the users to test the viability of the software first but does make adding reams of back data more painful. Here is the current workflow I am using as I enter in 16 years of history with our flock.

1. Pick a table within the LambTracker database to work on.
2. If the table already has data in it then do a query to get all the data out and save it as a .csv file. If you do not have any data in that table now the get a copy of the blank spreadsheet document that replicates the structure of that table.

Now you have to take one of 2 paths depending on whether you are updating existing records or inserting new ones.

For updating the existing table and its data:

Bring the .csv file into a spreadsheet program. I use LibreOffice. Take care to ensure that fields that might appear to be numbers (NSIP ID, registration ID and so on) are brought in as text fields only.

Edit the spreadsheet file to update and add missing information as required.

Save a copy as a .csv file.

Use a csv to sql tool to create update statements for the file.

Back up your existing LambTracker database.

Run the queries to update the database.

Verify the new data are in correctly.

Back up the database again.

Repeat as necessary for each table to update.

For inserting new records into the table:

Edit a copy of the blank spreadsheet document that replicates the structure of that table adding data as required. Pay particular attention to the format and ensure that all text fields are really stored as text. Also dates must be in the format YYYY-MM-DD stored as text for LambTracker to work properly. Times are stored as text fields too HH:MM:SS

Save a copy as a .csv file.

Use a csv to sql tool to create update statements for the file.

Back up your existing LambTracker database.

Run the queries to insert new records into the database.

Verify the new data are in correctly.

Back up the database again.

Repeat as necessary for each table to create or add to.

There is a lot more that I’ve discovered about how to do this efficiently but this is the gist of it.

Desktop Musings

Been going round and round on several options for the LambTracker Desktop app.

Initial choices, Swing or Java FX. I had about settled on Swing because I really don’t want to keep supporting Oracle. Then Cards or not as a structure. Was still waffling back and forth and then this:

Run Android Apps in Chrome Browser

APK Packager

Started following the details, the required SW is now on the Google Play Store so it’s more official with every passing minute.

A quick test showed that on Linux machines LambTracker runs. The only issue is trying to figure out where the virtual SD card is. I need to know so I can backup and work with the database. I didn’t’ get it running on my Mac yet but I haven’t really had time to play with it much either.

Anyway, I’m attempting to get the existing LambTracker application running on my Mac and do some further testing.

One option is to write an Android Activity that is the desktop side assuming that this will continue as an option and go from there. One advantage, it buys us cross platform capability right away.

Still pondering but this looks promising.

Starting the cycle again

We are about to start our major sheep evaluation cycle. I’ve added most of the NSIP EBV data points into the database and al writing the queries to display the data along with the most recent evaluation. Alex has found several bugs that we never see here and fixed them for which I am grateful. Bug fixes and some needed new features are being implemented but things are taking longer to do as fall farm work is ramping up. Hoping to get the re-vamped evaluate sheep module finished by the middle of October.

Lambing over Now to debugging

Lambing finished 3 July.

I’ve now moved on to going back over each module and working on the user interface issues and bugs identified this first year.

LambTracker Desktop is also being further refined and additional functions are being added. Coding will be starting this fall once I finish some clean-up on the mobile side.

Lambing Lull

We are now in the lull between batches of lambs. Had 9 ewes left to go but now down to 8.

So my focus is shifting to the desktop and cleanup of the mobile application.

Overall design of the desktop side is progressing well. I’ve been writing out a bunch of different structures and doing bench testing and some scenarios to test them. I think I am finally getting a handle on what will work best and allow for future expansion easily. I am also working on the NSIP data collection sections. Most of the data should be collected as a normal part of sheep management but the key is alerting the user to missing data and formatting the data for delivery to LambPlan in Australia. That task has to be done on the desktop side.

I am also starting to go back through all of LambTracker Mobile and pulling out common code into a Utilities class. I’m trying to clean up the rather messy structures I have now. It is slow going as I usually mange to break something with each section I pull out.

There isn’t much activity on the GitHub site as I am not pushing changes until I get to test them more here.

LambTracker Bug Fixes

Bug fixing of the Lambing module is going well. A few things took a bit to correct but it’s all working as expected now. I do keep a running log on twitter of most of the work on LambTracker as well as our farm status. I’ve added a follow button in case anyone wants a running commentary.

LambTracker Meets the Lambs

IMG_0733

We’ve been using LambTracker to document all the births during this lambing season. I’ve uncovered several bugs and there are some workflow issues that are being dealt with but by and large everything is working as planned.

For now I am also keeping my standard paper backup of the data and also entering it into my old spreadsheet record system so there is double work for every birth.

I have fixed the duplicate tag issue for LookUpSheep but need to do some work to make it accessible everywhere. The other 2 issues are not done. I’ve been fixing the minor bugs as they show up during lamb processing rather than adding new features.

Lambing Module First Testing

First Lambs of 2014

Pair of ram lambs. First of 2014 and also via AI.

IMG_0696

 

LambTracker Lambing module encountered live sheep for the first time today. 2 sets of twins. We uncovered a few work flow issues. It’s important to get all the tags ready and enter everything but the weight in LambTracker before you actually tag the lambs or weigh them due to going back and forth with the screen and input. Lamb slime is not good for a touch screen. Neither is snow melting and falling on it causing erroneous key presses.

However, in spite of the weather issues the program worked well and all 4 lambs born so far today were processed properly.

3 known issues that need work:

  1. LookUpSheep only looks up the first sheep with a given tag. Fine for ear tags that are not duplicated but a problem for the paint brands we put on the sheep at birth.
  2. I didn’t get the edit the Lambing History record part finished so for now adding the details of the birth is being done on my Mac by hand directly into the database.
  3. Related to above I also didn’t finish adding the paint brand to the ewe record section. SO I am also adding that by hand on the Mac directly into the database.

The items are not really bugs, just functions that are not written yet or not finished.

We are having snow storms and we have a number of ewes looking ready to go in the next 24 hours so LambTracker is likely to get a workout.

Lambing Module Update

The Lambing Module went to feature freeze about a week ago and I’ve been working on bug fixes. As of yesterday the main Lambing Module and the AddLamb activity are both working properly.

There are some issues with LookUpSheep because handling multiple sheep with the same ID is not supported yet. So looking up a lamb with a paint brand will only get you the first lamb not all lambs with that number.

I did make one modification to LookUpSheep to add display of the birth date, birth type and sex on the main screen.

I will be taking out the debug code today and tomorrow in preparation for testing with real live sheep. Lambing is due to start on Thursday for the early date (142 days gestation) and next Tuesday for our most likely date (147 days gestation).

20140415_173450

As you can see programming is just barely done in time.

Notes vs Evaluations vs Alerts

Just finished a major change to the database structure. We’ve always had the concept of Alerts – things that you want to check next time you work the sheep, Notes – Things that stay with the sheep record for life but don’t really fit into any other table and Evaluations – ranking and scoring and measuring the sheep in some fashion.

What became obvious is that the udder status (none, small, medium or large) is really an evaluation at a given time for a sheep and all that data is not really a note but is another type of user defined evaluation. I had already created user evaluations for the estrus status and pregnancy status so I had the structure there. I’ve added the udder status to the user defined evaluations and completed fixing EvaluateSheep to handle dynamic creation of multiple user defined traits with custom results. Next task was to update the database and fix the historical records already in it to reflect the new structure.

I suspect each flock owner will have a slightly different take on what should be a note vs an alert but that is why the system is designed to support either option. For now the new structure seems to make more sense to me.