About this "Servicing" Mission

I was one of the IHW Large-Scale Phenomena Network team members, and the task lead for the contract support development staff. It occurred to me not too long ago that I was also one of the few original members of the development team still active in the NASA environment and that I was fast approaching (or past) retirement age. So, it seemed like a good idea to review the original archive with the accrued knowledge of the past 35 years, with Halley recently having passed aphelion, to see if there was technology available which might make access to the archival data easier for our successors in another 35 years, and it also seemed like some "insider" knowledge of the decisions and rationale for what we chose to do with the archive around 1990 might also be helpful.

My original idea was to simply try to pull the index files we'd created for each IHW discipline and subdiscipline from the original 26-disk CD-ROM archive and put it into some sort of coherent database schema, then export it in order to provide what I hoped would be a more user-friendly way of sending the data forward another 35 years - perhaps SQL or JSON would be more prevalent in 2061 than they were in 1985. But once I had a working database, what would be cooler than to wrap a little web application around it. The chosen framework - whatever that might be - could possibly help someone filling my original role with the IHW figure out how to actually read the data with software.

When I started planning this project, I contacted the PDS Small Bodies Node (SBN) and they kindly put me in touch with Anne Raugh, an employee of theirs and a former colleague of mine from back in my IHW days. She has been endlessly helpful in getting me caught up with what the SBN had done with the original CD-ROM archive over the years and how it had evolved to a v2. They'd already started the process of decompressing the compressed LSPN images (a space-saving hack we/I implemented for the original archive), and reattached the FITS headers to the separate data files (an adjustment we put in place to accommodate both FITS and PDS formats for the data). They had also (thankfully) rearranged the directory structure to eliminate the hard CD-ROM disk boundaries and put the data from individual disciplines in their own separate directories. They hadn't quite finished that onerous reformatting task, so I had to adjust my ingest script development efforts to handle that. They'd also added some corrections to the data to fix typos and other glitches, so I wanted to use the most recent version of the data available.

So I downloaded the entire archive from the SBN and set to work. Writing perl scripts to ingest data is pretty straightforward, except for the part about parsing out the various discipline-specific data tables. I was on the verge of developing individual scripts that would read the FITS headers (particularly for tabular data) and writing a parser for each one just to ingest the data into a discipline-specific table in the database, while also pulling common metadata into a single table to allow cross-disciplinary searches, something we'd overlooked in creating the original archive.

My original intention was that this would make a fine post-retirement project since working with computers and software has been a hobby of mine since my time in graduate school back in the late 1970s. I didn't really expect to get it to this point quite so quickly.

This is where AI came to the rescue. I'd been looking for a place where I could learn to use AI for something "useful" for my own purposes, and along came Github with Copilot-cli which provided command-line access to the AI tools, and a USD10 monthly access account. The command-line interface would allow me to point the tools at my entire project at once, rather than using a web interface to upload individual files. So, that's the direction I took. Using Copilot-cli to construct the ingest scripts by walking the directories in the v2 archive, reading FITS headers, building table schemas and populating the database table was significantly easier than writing them by hand. And along the way, I could address attaching FITS headers on the files that had been skipped by the team at the SBN (largely browse and calibration images). The end result was a working database, built in my spare time over a period of a few weeks. I could review the individual tables as they were built to make sure the right data was there and in the proper columns.

The crowning touch was using Copilot-cli to build the web application, written in Python and using the widely-used and highly stable Django framework. I opted to use those because I suspect that Python will probably still at least be understandable by the time 2061 rolls around. I don't hold out much home for Django, but at its core, it's all still just Python. The app itself might or might not be useful in 2061, but I'm more optimistic that the database, or its export products in SQL and JSON will be. Anyway, the web app development literally took a couple of hours of my time to see it built and start testing it.

So, all of the ingest scripts and the code for the web app are at Github. I will donate all of the code and my updated version of the archive back to the Small Bodies Node to do with as they wish. They are our best hope for ensuring that everything is still available to the IHW V2061 team.

Now, I have to figure out what my next post-retirement project will be, if and when I actually retire from my day job.

📚 Additional Resources

• Text Files - Text documents indexed from the original IHW archive
• Documentation - Project-managed notes and text documents stored in the app's documents directory
• PDS Small Bodies Node - Official planetary data archive
• IHW on Wikipedia - Historical background and mission details