Converting DB/TextWorks Data to MARC: Not as Easy as it Sounds!

by Jonathan Jacobsen Wednesday, April 25, 2018 4:55 PM

We recently had a client ask us if we could help them convert their DB/TextWorks library catalogue database to MARC format, for submission to another search system.

Our initial reaction was "Sure, easy-peasy." After all, we're librarians, we know MARC well, we're experts in DB/TextWorks, and we've done this before. How hard could it be?

Ha! Famous last words, and all that.

To be fair, there are a few wrinkles:

  1. The conversion is not a one-time affair, but rather something the client would like automated and running on a regular basis.
  2. Not all records in the database are to be converted.
  3. MARC is not the simplest of formats. Whether MARC or MARCXML, there are a fairly rigid set of rules that must be followed to create 100% valid, standards-compliant MARC records.

Nonetheless, since MARC has been around rather a long time, there are a plethora of tools available for creating MARC records from other data sources. There's Inmagic's own MARC Transformer, which works directly with DB/TextWorks databases, as well as various free or open source tools from the U.S. Library of Congress and other agencies, MARCEdit, Balboa Software's Data Magician, and others.

We also have our Andornot Data Extraction Utility for automatically exporting data from a DB/TextWorks database and manipulating it into a variety of formats.

Ever optimistic, we figured some combination of these tools could be strung together in sequence without too much effort to create a solution for this client. We wanted to avoid developing a DB/TextWorks-to-MARC conversion program from scratch as it would be quite time consuming, mostly due to the MARC format requirements themselves.

Several tools, upon closer investigation, proved to be too ancient to run reliably on a modern Windows server, or couldn't work with the current version of DB/TextWorks. Others proved almost impossible to use in an automated fashion. They could be useful in a one-time manual conversion to MARC format, but not in the hands-off, automated workflow we needed.

The exploration of this issue was an interesting exercise in seeing how old data formats and old programs age and become harder to work with.

The recipe that baked the cake in the end was:

  1. Use the Andornot Data Extraction Utility and the Inmagic ODBC driver to extract data from the DB/TextWorks database to a pseudo-MARC plain text format. 
  2. Use a custom-developed PowerShell script to manipulate the records in this file to handle some of the quirks of the ODBC output and to more closely adhere to the MARC format.
  3. Use the command line interface to MARCEdit to convert the pseudo-MARC to MARC Communications Format files.
  4. Upload the MARC files over FTP to the destination server.
  5. Manage all the moving parts through a PowerShell script and log the steps and results to a file for easy troubleshooting in case of problems.
  6. Run the script nightly as a scheduled task on a Windows server.

When written like that, in hindsight, it sounds so simple. And in the end, it was, and works well. But the journey to arrive at this solution was one of the more challenging small projects we've undertaken, considering how simple the task sounds at first.

We hope this will help you if you have a similar project, but don't hesitate to ask us for help, now that we've worked through this.

Andornot Professional Development Grant for 2018 Awarded to Gayle Graham

by Jonathan Jacobsen Wednesday, March 28, 2018 10:35 PM

We are very pleased to announce a recipient for the Andornot Professional Development Grant for 2018: Gayle Graham of the Nova Scotia Health Authority.

Gayle is relatively new to health sciences librarianship, and has never attended a conference specifically about health libraries. She will use the grant to attend the Canadian Health Libraries Association (CHLA) Conference in St. John’s this June. Gayle notes that “as a new organization, we would really benefit from an update on what's happening in the wider community of health sciences libraries.”

Andornot strongly believes in the value of attending conferences to foster professional development. We attend events across Canada and the United States all year long to learn about new trends and technologies, meet with clients, and share our expertise with like-minded folks.

We inaugurated this grant last year, awarding it to Mark Goodwin of the BC Cancer Agency. We were delighted to be able to offer this grant again this year, and only wish we could send everyone who applied to the conference of their choice. 

We hope that everyone who applied, and all of you, will also be able to attend a conference this year. Check out the list of ones we’ll be at and drop by to say hi if you can.

Tags: events | funding

Library and Archives Canada announces launch of 2018 funding cycle for Documentary Heritage Communities Program

by Jonathan Jacobsen Sunday, December 03, 2017 4:45 PM

Library and Archives Canada has announced the launch of the 2018 funding cycle for the Documentary Heritage Communities Program (DHCP). This is the fourth year of a planned 5 year program, with $1.5 million available this year, as in previous rounds.

The DHCP provides financial assistance to the Canadian documentary heritage community for activities that:

  • increase access to, and awareness of, Canada’s local documentary heritage institutions and their holdings; and
  • increase the capacity of local documentary heritage institutions to better sustain and preserve Canada’s documentary heritage.

The deadline for submitting completed application packages is February 7, 2018. 

This program is a great opportunity for archives, museums, historical societies and other cultural institutions to digitize their collections, develop search engines and virtual exhibits, and other activities that preserve and promote their valuable resources.

The program is aimed at non-governmental organizations specifically, including:

  • Archives; 
  • Privately funded libraries; 
  • Historical societies;              
  • Genealogical organizations/societies;  
  • Professional Associations; and 
  • Museums with an archival component.

Businesses, government and government institution (including municipal governments and Crown Corporations), museums without archives, and universities and colleges are not eligible.

Types of projects which would be considered for funding include:

  • Conversion and digitization for access purposes; 
  • Conservation and preservation treatment; 
  • The development (research, design and production) of virtual and physical exhibitions, including travelling exhibits; 
  • Conversion and digitization for preservation purposes; 
  • Increased digital preservation capacity (excluding digital infrastructure related to day-to-day activities); 
  • Training and workshops that improve competencies and build capacity; and 
  • Development of standards, performance and other measurement activities. 
  • Collection, cataloguing and access based management; and 
  • Commemorative projects.

Further program details, requirements  and application procedures are available at http://www.bac-lac.gc.ca/eng/services/documentary-heritage-communities-program/Pages/dhcp-portal.aspx

How can Andornot help?

Many Andornot clients have obtained DHCP grants in previous rounds, and Andornot has worked on many other projects which would qualify for this grant. Some examples are detailed in these blog posts:

We have extensive experience with digitizing documents, books and audio and video materials, and developing systems to manage those collections and make them searchable or presented in virtual exhibits.

Contact us to discuss collections you have and ideas for proposals. We'll do our best to help you obtain funding from the DHCP program!

Omeka S Out of Beta and Ready for Use

by Jonathan Jacobsen Saturday, November 25, 2017 12:05 PM

About this time last year we blogged about a new version of Omeka, Omeka S, entering beta release. Now we're happy to see that a final 1.0 release of Omeka S has just been released.

Omeka is a free, open-source content management system (CMS) for online digital collections. With Omeka, you can quickly build a searchable repository of archival, artifact or other records and assemble them into virtual exhibits to showcase your holdings.

Andornot uses Omeka with select clients and as the basis of our Digital History Hub platform.

Most content management systems are designed to manage a single website with a hierarchy of pages, in which are placed text and other media. In contrast, Omeka is based around items (e.g. historic documents, photographs, audio or video recordings, etc.) which can be arranged into item sets and pages of items. One Item can be used in multiple ways, as part of different exhibits, for example.

An easy-to-use web interface provides site administrators with access to all the important back-end features: configuring the site appearance and navigation, uploading items (individually or in batches, such as from a database export), changing themes, and creating content pages.

Omeka S offers users a brand-new interface and features such as:

  • Manage multiple separate sites from a single installation of Omeka.
  • Build and publish pages, exhibits, or digital stories by adding and mixing different content blocks.
  • Create relationships between your resources - items, item sets, and media.
  • Use importers to bring in content from a spreadsheet or an Omeka Classic site.
  • Geolocate your content and display maps on sites using Mapping.
  • Connect your installation with Fedora and DSpace repositories, with the ability to update content periodically.
  • Use mobile-ready themes to customize the look of each site.

Omeka is a great choice for museums, archives, historical societies and others with cultural collections who want to make their collections searchable online. It's as easy to use for volunteers with little experience as by professional curators, archivists and historians.

More Information:

DB/TextWorks Still A Popular Choice for Teaching in Schools

by Jonathan Jacobsen Saturday, October 28, 2017 7:44 AM

Inmagic DB/TextWorks continues to be a popular software application taught in schools. For example, the Library Technician programs at Langara College and the University of the Fraser Valley in B.C, as well as at the Southern Alberta Institute of Technology, each include it in some of their cases.

Yesterday I had the pleasure of speaking to students in the Library Technologies and Information Management class at Langara College. These budding library techs will learn to create a database for a class project using DB/TextWorks, hopefully with a bit of inspiration from the ideas I was able to share with them.

The image above shows screens from the Andornot Starter Kit, a ready-to-use DB/TextWorks database suitable for a small library.

Not all software has such longevity as DB/TextWorks, but I think this popular app endures because it remains unique in the market. For clients of ours with a modest budget who need to manage diverse kinds of information and don't have programming skills, it remains an excellent choice, once we heavily recommend to many clients.

We see it used in law firms to create and manage databases of experts, memos, precedents, boilerplate documents, corporate archives, and of course a traditional library catalogue. In hospitals, it's used to manage patient education materials, and libraries with a strong circulation component. Elsewhere, we see it used to manage museum artifact collections, archival documents, databases of digitized historic documents and audio-visual recordings. In municipalities, it manages bylaws, real estate development applications, council documents… the list is endless. 

There are many highly-specific database applications available, tailored to the needs of particular organizations (e.g. Inmagic Genie for specialized libraries, Lucidea's Argus for museums, etc.), but few tools that are as easy to use as DB/TextWorks that can be applied to managing any kind of information. Anyone can learn to create a database and snazzy search and edit screens and have a functional, aesthetically pleasing database in a very short time, with little technical aptitude needed. Managing this information is easy with the many built-in, pre-programmed features, such as validation lists, batch modifications, the URL checker, and so on.

Two other long-standing database programs are of course MS Access, included with almost every copy of the MS Office suite, and Apple's FileMaker. The former is practically free and so ubiquitous that many people use it out of necessity, while the latter is quite visually appealing and with many useful features. However, in our experience, both require a higher level of technical skills to really make useful. DB/TextWorks simply has more of the programming already done.

It's reasons like this that cause it to still be an excellent choice in many cases, when budgets and user skills are modest, and thus is well-worthwhile learning to use in a library technician or similar programm. Paired with a search interface like our Andornot Discovery Interface, VuFind, Omeka, or Inmagic Presto, it becomes a perfect back-end to a highly functional front-end, a great combination for managing and searching information.

Contact us to learn more about any of the above, or if you're a school or student and would like a trial version of DB/TextWorks to use.

Month List