There existed an abundance of weekly work with the inventory that was repetitive, laborious, and obligate for the continued operation of the lab. Much of the tasks involved sifting through the inventory for samples and parsing out the garbled entries that would occasionally clutter the final report. These sorting, searching, and reporting tasks were of the brand that makes work an undesirable experience.  These are also the sort of tasks that lend themselves to WHERE, GROUP BY, HAVING, and ORDER BY clauses.

Performing inventory manipulations in a spreadsheet can be especially chaffing when juxtaposed with the knowledge or imagination of any more effective system. No great technical skill or understanding is required to envision any number of better methods of storing and working with the inventory data. The major problem was the gulf of time and effort separating imagination and implementation.

The magnitude of the precarious nature of the spreadsheet-based inventory seemed to be unrealized by most of the staff. This issue was known, but the potential size of the problem was poorly understood. Since the system was apparently operational, the specter of massive data corruption was largely unacknowledged. The possibility that a widespread yet subtle error went undetected for an extended period of time was not in the top worries of the personnel.  All of the people working with the system were attentive, smart, and responsive to problems. Despite the best people and intentions, a few relatively minor data corruptions, that were identified and corrected, foreshadowed the possibility of a catastrophic failure.

Reluctance to move away from using an Excel spreadsheet is understandable.  It is entirely managerial for small projects and seems scalable because it appears very simple.  However, as a project grows in size the logic and complexity required to maintain it falls to the people running the project.  Excel is a good spreadsheet, but it is not a database and has little ability to scale up.

Providing continuity of service while implementing a new system required an implementation that was either parallel, incremental, or otherwise moving in step with the existing system. Previous improvement efforts did not succeed due to a combination of technical aspects, the biological complexities of the samples, the process requirements, and complexity of use for the personnel. While the majority of the affair was that of information technology, the other facets of such an undertaking were vital. The biological details that needed to be considered were sufficiently nuanced that they were beyond the scope of many of the technologically skilled.  Conversely, the technical aspects of the system were not obvious to those accustomed to biological work and required explanation in language understandable by scientists.

A palatable mixture of simplicity, usability, utility, and extensibility was a tall order.  Extending too far or falling short with any of these attributes rendered previous attempts useless for scientific data collection, sample retrieval, sample information entry, or generation of summary reports.  Prior attempts tried systems dependent on MS Access, multiple hyper-linked files, and proprietary software suites targeted to laboratories.  A solution with a viable balance of characteristics was hard to come by in an off the shelf product.

The scope and scale of prior upgrade projects were likely the primary contributors to their collapse. Attempting to tackle the entire inventory system all at once including the migration of all existing data and procedures was just too massive an undertaking. The force of a thousand tiny details that had to be accommodated by any new framework provided a great deal of resistance to forward progress.

The inventory of frozen samples contains 250,000 frozen 1.8 mL Nunc Nalgene cryovials. More than 430,000 frozen vials have passed through the inventory. Information about the contents, location and quantity of these samples was written in 76,500 rows by 22 columns. In 2006, this body of information was slightly smaller (64,000 rows) and contained in a single Excel spreadsheet.

Routine manipulations of the data set included adding new entries, decrementing quantity, changing location, and making corrections to fields discovered to be out of step with the actual inventory samples. The logic and protocols responsible for making these changes were implemented and accomplished primarily by two technicians who were very careful in their administration of this large spreadsheet. Efficiently and effectively managing the inventory required these technicians to be very sensitive to subtle differences in data, extremely attentive to formatting details, and have intimate knowledge of the specific lexicon of the data.

Despite great care and herculean effort in maintaining this body of data, errors were inevitable. Many of these problems were obvious results of manual data entry and unchecked data manipulation. Copy/cut/paste, sorting, and typing data entry by humans lent themselves to precisely the sort of formatting, logical, and systematic problems that were constantly being discovered and repaired.

This vague description of the system in place should be sufficient to clearly illustrate that an Excel spreadsheet is the incorrect tool for the task. Without further details, it aught to be apparent that the tasks and data structure lend themselves to a relational database and semi-automated data entry and manipulation applications. The reaction of a person with the skills to identify these issues would likely be very similar to the reaction of a carpenter who encounters people hammering screws through boards.

Flowchart Summary of TIL Lab Core Processing Group

The previous project wxSQList++ was a complicated attempt to incorporate agreat deal of functionality into a single application.  This was a resounding success as far a demonstration of the procedures a logic to successfully export TheList.xlsb coherently to a local database, but the project was too bulky and too slow to be practical.

The second trimmed down iteration of the project, wxSQList2++, was built from some of the nicer portions of the original.  The scope of the application was simply to extract, analyze, and transform the data from TheList.xlsb.  This application only needed to access TheList.xlsb through use of the M$ Component Object Model (specifically OLE Automation of Excel), and output plain text to be utilized later.  Although this approach seems clunky, it executes quickly and provides data backup in a single step.

The output format was similar to that produced by mysqldump (dev.mysql.com/doc/refman/5.1/en/mysqldump.html).  This was done because at the time I needed a way of making quick snapshots of the data in the inventory, and I was also importing that data into two different databases (one local, one remote).  The output text contains all the commands and data required to rebuild the TIL Lab Inventory in a MySQL database schema.

The remaining functionality of the original wxSQList++ was augmented and included in a batchfile.  The batch file writes a log of the wxSQList2 application debugging output, copies the mysql compatible dump text of TheList.xlsb to an archive on the server, and finally invokes the mysql client in order to execute the sql dump as an import.  This system is mostly plain text and does not require messing with the C++ source code in order to change the archive locations or the database connection information.

Altogether, wxSQList2++, mysql client, and a batchfile take care of the “TheLIST to DB Export procedures” component of the flow chart.

It has been quite a while since I last decided to put something down about the projects I have been working on.  The Wunderlist, which was a java & SQL system, has been put on hold indefinately.  While it would have been easier to do the migration of our labs data to Biofortis’ Labmatrix system from a java base, it was impractical as an intermediate step.  As such, the goal of the project has been preserved, but the implementation has been changed.  The new project is SQList.

The SQList in the current incarnation is a mySQL database that is a copy of the data contained in the monsterous Excel spreadsheet, “The LIST.”  The data is migrated to the database by a C/C++ application titled, “wxSQList++.”  This system is currently used solely by myself, but will soon be migrated off my local machine to a space hosted by the NCIweb group.

The devil and some details of the mySQL schema that is used:

• Schema’s name is currently “lab.”
  I intend on changing this to something such as TILLL (TIL Lab List), TheList, wList, or CG9.  It occurs to me that the naming really does not matter as any name choosen will not retain the original meaning beyond a year or two.  After that, the name will simply be a name.
• The Schema tables are:
  norminv:  Listing of the well formed biomaterial entries.  This holds the majority of the data.
  incoming: A copy of the entries being held in the Incoming sheet of The List.  These entries are manually examine prior to being merged with the main body of the normal inventory.
  abinv: Listing of the entries from The List that could not be parsed well or attributed to a subject.
  patients:  Listing of our subjects.  Having this table removes the need to repeat this information for each biomaterial entry that is attributed to a subject.
  controlsubjects: Listing of subjects that have materials that have been selected to be used as controls for assays (ie: Coculture IFNγ Release ELISA)
• Data Source:
  The wxSQList++ application is an interim tool that drops, creates, repopulates, and cleans all the tables in the schema from the information currently held in The List.  As such, the database is a snap shot of the inventory status and not a real time representation.

A bit about the wxSQList++ application:

This is a text only tool written in C\C++ that is designed for my use only.  I intend to replace it with a much more robust read and backup tool.  It’s purpose is to read the data out of The List excel spreadsheets and create a database representation of the information.

One of the difficulties of this C/Vial column is that there are two formats to be considered. The first describes the number of cells and looks like, “2.00E+6,” while the second describes a volume of serum per vial in the form like, “1.50 mL.”

//set up components
java.text.DecimalFormat decfmtCellConcen = new java.text.DecimalFormat(“###.00E+##”);
java.text.DecimalFromat decfmtSerumConcen = new java.text.DecimalFormat(“##.
decfmtCellConcen.setParseBigDecimal(true);
java.math.BigDecimal bigdecConcen;

//parse concentration column
try{

bigdecConcen= (BigDecimal) decfmtCellConcen .parse(spLine[12]);
}catch(Exception e){
try{
bigdecConcen = (BigDecimal) decfmtCellConcen.parss(spLine[12]);
}catch(Exception x){ myLogger.debug(x.toString()); }
}

Monkey wrenches in parsing compliments of TheLIST.

One of the best part of dealing with a bloated spreadsheet is that sometimes random crap can creep its way into columns it was never supposed to be in. An empty cell or a ‘?’ in a cell is understandable, but finding the word “blue” in the Concentration per Vial column just illustrates how screwed up the data can get. In order to get around this it seems like this string might have to be parsed twice, or simply handle the exception thrown when something unexpected shows up.

The object here is to return the long representation of the date such that it can be wrapped into a java.sql.Date constructor. So ideally, inside the logic of parsing the lines from The LIST we can simply do java.sql.Date( parseListDate( stringFromTheLIST ) ); and insert this into the sql database.

import java.text.SimpleDateFormat
import java.text.DateFormat

public long parseListDate( String dateStr ) Throws ParseException{
SimpleDateFormat simpDateFmt = new SimpleDateFormat("MM/dd/yyyy");
return( simpDateFmt.parse(dateStr).getTime());
}

Of course, the SQL database representation is simply an intermediate for cleaning the data prior to the final import into the proprietary labmatrix database. This will also provide a method for keeping track of which rows were unable to be imported for reasons of confusing, corrupted, or lacking data.