Introduction to the DICOM Standard (using OsiriX)

Wed, 10 Feb 2010 22:44:37 -0600

I decided to write an introductory tutorial on the DICOM standard as there were not many articles on the Internet that singularly cover all aspects of this very large standard through an example based approach which helps the learner relate to the underlying concepts quickly by means of association. Many people I know who work with DICOM on a daily basis often deal only with a few aspects of this vast standard. So, my hope is that by providing a broad overview of all aspects of this standard, there is something useful to take in for everyone.

To illustrate many of the core concepts within this standard, I decided to use a freely available and yet powerful tool called “OsiriX” which has many DICOM capabilities right out of the box. However, please keep in mind that this software runs on the Mac OS X operating system only. So, my apologies if you only have access to some other operating system. However, simply reading the material here as well as viewing the screenshots shown should help you take in the concepts and apply it to any other DICOM software that you choose to utilize. Also, please keep in mind that this is not a programming tutorial, although much like the approach that I took with my earlier HL7 articles, I plan to use this tutorial as a foundation for a future tutorial in which I will illustrate how to write custom software applications using the DICOM standard.

1. What is DICOM

DICOM stands for “Digital Imaging and Communications in Medicine”, and was developed jointly by the National Electrical Manufacturers Association (NEMA) as well as the American College of Radiology (ACR) to permit interoperability between imaging equipment as well as with other devices. This standard is responsible for governing both the image format as well as the various network protocols required for transmission of medical image information generated during the many healthcare-related imaging “modalities” such as magnetic resonance, nuclear medicine, computed tomography and ultrasound. The DICOM standard has existed in one form or the other since 1983, and continues to evolve every year.

1. Image Transmission within Hospital Systems

I feel that the best way to explain the standard is by first describing an imaging-related workflow that occurs very frequently in clinics, hospitals, or large health networks. Later, I will use this scenario as the basis for explaining the “nuts and bolts” of the standard.

Let us say a patient was admitted to a hospital with some chest pains. The attending physician may order an MRI scan, and when this request is recorded on the Hospital Information System (HIS), an electronic request is often transmitted to the Radiology Information System (RIS) located in the imaging centre. This request typically comprises of information about where the request came from, who ordered it, the details of the patient, the type of imaging modality requested, etc. Once the booking is done, the patient then is sent for the scan to the imaging centre. After a scan has been completed, a set of DICOM-compliant images are created from the raw data, and is referred to as a “Study”. A study may itself consist of several acquisitions depending on the scan configurations, and each of these acquisitions is referred to as a “Series”. Each series will consist of a number of images, and each of these images are individually referred to as a “DICOM Information Object”.

After the scanning procedure has been completed, all the images are transmitted for archival to a PACS system (Picture Archival and Communication System). The scanned images may be reviewed for quality before being transmitted to a PACS system, and the reviewing technologist may order a scan again if they are not satisfactory. The archived images can then be retrieved from the PACS system to a workstation for viewing by a radiologist. The radiologist may either view the images directly on the screen, or print these images on film. Later, she may add additional comments about her observations on a report. Once she completes this process, the changes are merged with the original study on the PACS system. An electronic message is also transmitted back to the RIS indicating that the modality request has been completed. Information may also be transmitted back to the originating HIS along with some of the key images to assist in intervention by a cardiologist if necessary.

1. Understanding DICOM Services

The seamless flow of images and image-related information between the many types of hardware and software involved in the workflow described in the earlier section could not be easily achieved without the use of DICOM (and HL7). Let us now dive a little bit into the details to see how the DICOM standard enables this type of information flow to occur.

At its very core, DICOM standard is based on the concept of service providers (also referred to as “Service Class Providers” or “SCP”), service consumers (also referred to as “Service Class Users” or “SCU”) and information objects that are acted upon by the providers and consumers. For example, the printer described in the earlier section is an example of a service provider, and is referred to as having a “C-PRINT SCP” capability. The application requesting the print activity is acting as a service consumer, and has “C-PRINT SCU” capability. The combination of a service class along with the information object template that it acts upon is referred to as “Service-Object Pair” class, also referred to as “SOP”. SOPs form the foundation for DICOM-related compliance (or “conformance”), and allow vendors to disclose what DICOM-related capabilities their software and hardware support. Each SOP class has an unique identification number issued by the DICOM committee. The list of SOP classes is very long already, and continues to grow as more modalities are supported.

The goal of the DICOM standard is to enable a “plug-n-play” architecture within any healthcare information network. The standard therefore mandates that all DICOM-capable devices electronically disclose their capabilities during any initial “handshakes” that occurs between DICOM-compliant devices. This handshaking is often referred to as an “Association Establishment”. Depending on the capabilities supported, this association can be successful or fail. If the association is successful, instances of the SOP classes are exchanged between the consumers and providers of a service. The SOP instance in the print example described previously may carry information such as the patient information, pixel data of the images to be printed as well as any annotations that are required to be printed along with the images. Let us now take a look at the various services offered by DICOM-compliant devices a little further using OsiriX.

1. DICOM Services Offered by OsiriX

OsiriX software has a “C-STORE SCP” capability, and is therefore capable of storing incoming DICOM images into a local database. This allows the software to operate as a rudimentary PACS server. In order to be able to use this feature, you will have configure the settings in the “Preferences” screen to enable OsiriX to run as a DICOM listener. This software has many capabilities including being able to receive images through the HTTP protocol using the “WADO” specification of the DICOM standard (WADO stands for “Web Access to DICOM Objects”). It also offers web service integration through SOAP-based messaging. Another storage-related capability that OsiriX offers is the “C-STORE SCU”. This capability permits us to send DICOM information from OsiriX to other DICOM storage service providers such as PACS systems. The “Local Database” screen (shown below) shows how the images for a patient are organized within OsiriX. The images are always stored in a hierarchy consisting of patients, their studies, and the images in the study (or “series”)

Often, when DICOM images are pushed from a viewing workstation to a PACS system for long-term archival on hard disk, or on backup media such as CD, it is essential that the storage device provides some acknowledgement that information has been successfully received and stored before the images are deleted at the sender. This feature is possible within DICOM using the Storage Commitment Service which is typically achieved using a combination of other services such as “C-MOVE SCU”, “C-MOVE SCP”, “N-ACTION” and “N-EVENT”. This capability is provided by OsiriX,

Another DICOM capability that OsiriX offers is the query/retrieval of DICOM images. This enables users to search a remote PACS system, and later retrieve images that are of interest to them for for viewing locally. The querying capabilities are specified as “C-FIND SCU” and “C-FIND SCP”, and the retrieval capabilities are specified as “C-GET SCU” and “C-GET SCP”. Please see screenshot below showing how these features are implemented within the OsiriX software.

DICOM printing services enable image acquisition devices as well as viewing workstations to share DICOM-compliant printers very similar to how normal printers are often utilized within an information network. DICOM printing enables the use of standard calibration (encoded within the DICOM images themselves) to ensure that there is consistency between various display devices as well as hard copy printouts on which the images are seen. The print capabilities of devices are specified either as “C-PRINT SCU” or “C-PRINT SCP”. There are many types of DICOM printers available, and their capabilities are described through the SOP classes they claim to support. Basic support always includes grayscale printing, colour printing, grayscale printing with lookup table enhancement, and colour printing with lookup table enhancement. Optional features include annotation, image overlays, and print job execution status reporting. OsiriX supports all the basic printing capabilities specified in the standard.

DICOM off-line storage services enable users to exchange DICOM files on removable storage media such as diskettes, CD-ROMs, and optical disks. Different storage mechanisms are preferred based on the imaging context such as coronary angiography, general diagnostic ultrasonography or gastrointestinal endoscopy. OsiriX provides a built-in feature to burn DICOM images onto CD-ROMs (see screenshot below). However, it also permits the ability to export the DICOM images to JPEG, TIFF and QuickTime movie formats which can transferred more easily.

1. Other DICOM Services

Although OsiriX has many DICOM capabilities, it does not offer other more advanced services specified within the DICOM such as the “Modality Worklist” service or the “Modality Performed Procedure Step” service (also known as “MPPS”). These services are typically only required when needing to implement complex workflow scenarios that occur when interacting with a RIS or PACS system.

The modality worklist service is useful when you want to minimize the amount of information that is keyed in manually. For example, it allows the technologist conducting an MRI to automatically transfer the information contained within the requested procedure onto the images collected as part of the scan. This approach not only increases the efficiency of information transfer, but also increases the level of patient safety as there is a reduced chance for incorrect patient information.

The MPPS service is used to communicate messages pertaining to the imaging steps being performed between the device performing a scan and the RIS and/or PACS. There are fundamentally two types of messages that are utilized. There is something called a “N-CREATE” message which is sent at the start of a procedure step, as well as a “N-SET” message is which is transmitted after a procedure step has been completed. Any images acquired as part of the step completion are also transferred as part of this message.

1. DICOM File Format

In addition to the image pixel data, the DICOM file contains other information such as patient identification information, study and series information of the image acquisition, etc. All this information is stored inside a DICOM file in the form of data sets. These data sets are essentially a collection of data objects, and these in return consist of a number of attributes in the form of name/value representations. Information about the image (including patient information, modality information, pixel data for the images) are stored in these attributes which are maintained/managed through a DICOM dictionary. A single DICOM file is capable of storing many images (also known as “frames”) to enable for viewing in the form of a movie or as “cine loop” as they are often referred to in the DICOM world. Image pixel data inside the attributes may be stored in either compressed or uncompressed formats depending on the storage and transmission requirements. Image viewer applications can read and display the image data onto high resolution printers permitting accurate diagnosis of the results. The screenshot below shows how the “meta information” of a DICOM image is displayed within OsiriX.

1. DICOM Structured Reporting

Structured Reports (SR) within DICOM standard enables the exchange of diagnostic reporting between medical devices. These reports are stored in the same format as any other DICOM object. The special SOP classes defined for SR permit an easy way to store the essential diagnostic information based on the images in a study, such as procedure logs, observations, measurements, waveforms can be stored in a seamless manner, and allow us to link these reports to any corresponding images. Based on the complexity of the encoded information contained in the report, there are two types: a Basic Text SR; and an Enhanced SR.

1. Conformance in DICOM

Although not mandatory, vendors who claim that their products adhere to the DICOM standard typically provide a conformance statement describing how their device or software supports the standard. The information in the conformance statement include how the associations are handled (for example, whether it is capable of initiating associations, and how many in parallel, etc), the SOP classes that are supported, as well as other information such as presentation contexts and communication profiles. Customers can use the information contained in these documents to decide whether the vendor product can successfully communicate with other DICOM-compliant devices or software within their network.

1. DICOM Interoperability with Other Standards

There are many more components required to implement an effective workflow within an healthcare information network than is possible using the DICOM standard alone. For example, many complex transactions with a healthcare system are handled through the use of HL7. However, if DICOM and HL7 are to seamlessly integrate, some gaps are still required to be addressed. For this purpose, IHE (Integrating the Healthcare Enterprise) initiative was developed to help improve the cooperation between DICOM, HL7 and many other existing healthcare standards agencies. I will cover IHE in a separate tutorial in the future. For a tutorial on the HL7 standard, see my earlier articles.

1. Conclusion

I hope you found this introductory tutorial useful for gaining a high level understanding of the DICOM standard. There are many additional resources on the Internet as well as many good books that delve much deeper into the areas that I covered at a high level in this tutorial. My recommendation is also that you also take hands on training if possible to help boost your familiarity and confidence in using this standard. If you have any questions or criticisms on the information presented here, please do not hesitate to send me an email. Happy DICOM-ing. ☺

Unit Testing 101 For Non-Programmers

Tue, 19 Jan 2010 01:46:58 -0600

Seeing the many challenges in the field in terms of adopting unit tests, I am totally convinced that a basic awareness needs to exist of the inner workings of unit testing even for the non-programmer who works closely with software delivery teams to ensure that teams use unit testing to help produce high quality software that is easy to enhance or modify. This article explains the fundamentals of unit testing as well as how to adopt unit testing within your organization.

What is Unit Testing?

The goal of unit testing is to take a small piece of code that is responsible for enabling some very specific functionality within the software being developed, and to test it to ensure that it behaves exactly like you would want it to under various conditions. This approach allows us to test internal parts of software that are not typically exposed directly to the end user, for example, through a graphical user interface or screen, and therefore cannot be validated easily through conventional testing. This form of testing also provides early feedback for the programming teams as to whether they are headed in the right direction, and allows them to make small alterations if necessary.

In conventional testing, which can be either manual or automated, validation of functionality typically occurs after the software is developed by which time it becomes nearly impossible to resolve any critical defects or design flaws quickly, and almost always delays the delivery of software. However, with unit testing, the programmer’s work is validated much more quickly by testing small modules of the software as they are developed, allowing for quick changes to be made if defects, or deviations from the original design are detected.

Unit testing has been around for a long time, but tools to support unit testing have gained popularity only since the advent of JUnit (written by Kent Beck). Although Kent did write another Smalltalk-based unit testing tool called “SUnit” many years previously, it was only with the increased popularity of the Java programming language (and the industry’s quest to develop high quality enterprise-wide applications with it) since the late 90s that unit testing came to be increasingly adopted.

There are many challenges to the adoption of unit testing. These range from the fear of doing anything new, lack of desire to increase code quality and productivity by programmers, and also, a lack of awareness of good practices around unit testing as improperly implemented unit tests can sometime do more harm than good. These concerns can be addressed effectively by reading and understanding what others have done, followed by small experimentation before proceeding towards full, or at least limited adoption.

My own experience with building new software as well as in maintaining legacy software over the last 12 years has led to me to believe that the advantages of unit testing are often not plainly visible to programmers and customers alike, and that unit testing is often relegated to lowest priority especially when development timelines are tight. Sadly though, this is precisely when unit tests prove to be most beneficial as it cuts down on the stress levels of the entire team by permitting them to make even drastic changes without the fear of breaking existing or working functionality.

Art of Reasoning, and Why Unit Testing (and Software Development) is Totally Awesome...

Although the stereotypical image of the software programmer often portrayed in the movies is that of someone who is extremely introverted, and morbidly attached to his/her computer, all good programmers I know are friendly and reasonably extroverted individuals, and channel their creative abilities and curious natures on many hobbies, including “extreme” sports. However, more than anything else, the aspect of their lives they enjoy the most is always the challenge of applying their reasoning abilities to solve what many would consider as fundamentally logic-related problems. When developing software, many programmers apply reasoning, and build complex software using what I would call as “axiomatic” principles. That is, you build something small, verify that it works, and later, you build progressively bigger things on top of the previously validated “building blocks”. By doing it this way, in small incremental steps, you can be sure that the logical foundations of the software are always correct.

This is where unit testing comes in, and provides maximum benefit for the software programmer. By taking small portions of code, and by verifying that they work through incremental testing, we can later assemble the pieces of “validated” code blocks together into larger units, and at the same time be assured that software will do what we expect it to do. This approach also provides many opportunities for the programmer to clarify on features that are requested by the customer as it is nearly impossible to anticipate every possible scenario in which the software is intended to be utilized. This helps minimize the problem frequently seen in the industry, which is “paralysis through analysis”, an unfortunate situation in which teams spend way more time in analysis and design phase than necessary due to the fear that they have not captured every possible scenario, and are unable to proceed towards software construction quickly. When unit tests are used in conjunction with another practice called “Continuous Integration” (which I will touch on in the next section), they can help communicate the progress of the development to non-programmers as opposed to status reports which can be quite deceiving/misleading.

Some Concepts

If you are interested in getting a high quality software application developed with the help of unit testing, it might be useful for you to understand some vocabulary that experienced programmers frequently use when dealing with this aspect of software development. Gaining some understanding of this area will also help improve the communication between you and the programming team.

First, there is “System Under Test” (often abbreviated to “SUT”) which refers to whatever small module or piece of code that we are interested in unit testing. And there are other terms including “Class Under Test”, “Object Under Test”, “Method Under Test” and “Application Under Test” which are abbreviated to “CUT”, “OUT”, “MUT” and “AUT” respectively. The first three acronyms refer to the small bits of code, or modules, which the programmer often uses like building blocks, and which are responsible for enabling discrete pieces of functionality . “AUT” refers to the entire software application that is being developed. There is also one more important acronym “DOC” which is “depended-on component”. This refers to parts of the code that are not being unit tested, but still are required to be run in order to be execute the unit test against the SUT.

Another term that you will need to understand is a “test fixture”. This is essentially simulated data to help create the pre-conditions needed in order for the test to work successfully. “Continuous integration” is another term you will often hear in the realm of unit testing, and is an advanced practice utilized amongst teams comprising of many programmers working on different parts of the software simultaneously. This process enables the team to automatically build or assemble the software often to whatever functional state that is realistically possible on any given day. All the unit tests that accompany that code are also automatically run during this time, and this entire “build” of the software is considered a success or failure based on the results of the unit tests. This approach allows project teams to track whether the small modules that programmers are building are able to interact with one another successfully, and provides frequent feedback for team leads to ensure that the team is progressing slowly but steadily towards the overall design specification.

“Refactoring” is yet another term you will hear, and is the practice of changing the structure of existing code without changing its behaviour. This practice is often used in combination with unit testing, and is used to improve the design of existing code while attempting to add more functionality. With a safety net of the unit tests, the programmers feel much more confident about changing the code without worrying whether they broke something in the process. There is also another term “test-driven development” often shortened to “TDD” which is a habit of building one test at a time, and later writing the real code to accomplish the behaviour that the unit test expects. This approach allows the programmer to only build what is absolutely required.

Adopting Unit Testing

If your team does not do unit testing currently, then ask them to try a small pilot preferably on something that is being newly developed. Retro-fitting existing code to be unit testable requires a little bit more experience as well as patience on the part of programmers and customers before they start realizing the many benefits that this practice brings. Bringing in an expert even for a short period of time should help in these situations. Typically, unit testing business logic-related code is typically far more easier than testing user interface-related code or code that is responsible for persisting data into the database as there are far more variables to control during the test. Most often, the programmers who are best suited to be in on successful pilot implementations are people who are curious and eager to learn, and want to improve their productivity and that of their teams. Be supportive of these people as they are bound to struggle with other more cynical programmers who are not likely to try anything new easily. However, some level of cynicism is still good as you are better off adopting this practice in a slow but steady manner. This way, your team can collectively gain proficiency around unit testing, and work more cohesively together.

Conclusion

This is all I can cover in an introductory article, and have taken some liberties at simplifying many concepts for easier understanding. I hope you still found it useful for understanding this practice which is essential for delivering quality software. Unit testing can increase the speed at which defects are fixed, as well as allow new features to be added quickly. Teams that follow unit testing also tend to have better morale as there are fewer opportunities for “finger pointing” as most defects are caught before any code changes made by individual programmers are shared with the rest of the team. This enables these teams to work more productively, and without friction, and produce great software for you. Unit testing also allows for other programmers to join or replace the original developers if they leave, and become productive very quickly. Please do drop me a note if you have any comments or criticisms regarding this article.

People-oriented Software Development...

Sat, 17 Oct 2009 01:25:37 -0600

Recently, I was a lucky recipient of a free book on software development titled “Domain Driven Design: Tackling Complexity in the Heart of Software” by Eric Evans. Many people in the field consider this book a “classic”, and I quickly found out why. Reading the book sparked my motivation to finally do something that I have been meaning to for sometime; this was to capture (at least for my own benefit) a list of good practices for delivering great software.

One decade of reading, working and observing a lot of smart people in the software development field has provided me many insights into why some software development teams “click” and deliver great software, and also why other teams seem to simply whither away after only a few days or months of working together. Some of the key insights that I have are, very oddly enough, nothing to do with how programmers actually code but rather how software is conceived, designed and delivered through a close collaboration between users and programmers. My key insights are (in no particular order):

1. Bring enthusiasm to work, and be friendly

How many times have we all worked in a team where there were at least one or two individuals who considered their work simply a chore, and never had interest in it otherwise. These individuals simply rob all fun and enthusiasm from the rest of the team by their constant grumpiness and moping. I have even had to put up with a few individuals during the course of my career, who although considered very “productive” by some, were single handedly responsible for ruining any shared chemistry and synergies within the team due to their unwillingness to work with others and their total dislike for collective success. However, I have also been incredibly lucky to work with people who always shared an infectious enthusiasm and positive attitude when tackling any challenge that came their way (not just in the software realm), and always managed to be cheerful and friendly even when having a seemingly bad day at work. Teams comprising of such individuals are so rare to come by that most programmers (including me) would gladly consider working even for less pay to share in the fun and learning that almost always happens in such environments.

1. Understand the user’s business/domain

This insight has very special meaning to me these days because this was one area that I really didn’t pay attention at all until much later in my career. I was so focused on developing “frameworks” and other cool gadgetry to support the “structural” foundations of the software that I almost never paid attention to learning a little bit about the problem domain that I was writing the software for. The lack of knowledge sometimes hindered me from truly delivering the kind of software that not only addressed any current needs of the end user, but also one that permitted easy enhancements in the future. The people that I have come to admire these days are the folks who through the course of any software project become so savvy in the end user’s domain, and are able to forge great relationships as a result between the software team and the end users.

Many successful projects that I have participated in included design sessions where the end user (or a business analyst) starts by providing a high level overview of their business area/domain as well as summary of the problem they are attempting to solve. Even when this took a few days (or weeks depending on how large and complex the problem was), this was only a small price or “sacrifice” to pay for the rich interactions that usually almost always followed later since the programmers now had a good understanding of the end user’s domain. This, in turn, permitted them to develop software that addressed their needs. Another advantage that this approach provided was that these interactions often permitted end users to gain insights into the abstraction and modeling processes that are often used by programmers. The combined benefits allowed both parties to have meaningful dialogues and a deeper understanding of the problem at hand. This always led to the formulation of clear requirements, and good software consequently emerged.

1. Never do requirements in isolation

I have a big chuckle these days when I hear a programmer asking for a “detailed software requirements” document before they can begin work. Now, don’t get me wrong here. I am not saying that we shouldn’t bother creating documents containing detailed specifications of the software that we plan to build. But, to ask the end user or a business analyst to write this document in total isolation and simply hand it to you is really asking for trouble. How often have many of us seen “requirements documents” from end users or business analysts that really do not have any requirements at all, but rather contain a hazy/distorted vision of the final solution itself. This only creates frustration for the programmer who now has two tasks in front of him/her: 1) infer the real requirements from this preconceived “solution”, and 2) develop the right design for this software (based on what will now be only considered “his/her” vision). This method often (and unintentionally) leaves end users frustrated (and sometimes offended as well) since it makes them feel as though their ideas were totally ignored, leading to a lack of buy-in later on. So, my point is that we all need to work with the end users as the requirements are captured/formulated.

1. Model after the real world

The book I refer to above is bang-on on this topic. I have witnessed many failed software projects where the development team insisted on using a completely unique set of terminology to aid in their understanding of the user’s domain. Although some abstractions are always necessary when modeling software, abstracting too aggressively will result in confusion during any inter-team communications due to the need to translate the lingo back and forth between the teams. To communicate easily, the design model must be easily understood by both parties. This results in collective ownership of the software due to the investment in this shared model of understanding on which the software is based.

1. If it won’t run, it is garbage

On one project in the past, a team was looking at deploying software that had been written by seemingly bright individuals. The software was throwing cryptic error messages during the install process, and the administrators were making no progress at all. When the support team reviewed the code, they noticed right away that the code was extremely hard to read, understand and modify. When approached for clarification, the original programmers of the software spewed some mumbo-jumbo about all the “advanced techniques” and design patterns they had apparently utilized in developing this software, and directed the blame at administration team that was attempting to deploy this software. My take away from watching this fiasco unfold was that it didn’t matter what is “inside” the software, but if you cannot run and administer the software easily, it is really garbage.

1. Read and talk about software

Most of us who work in great teams will agree that the best software teams chat frequently about software development. Despite constant chatter and passionate discussions within these groups, one can notice that this environment always generates enthusiasm and excitement which forges the teams together. This insight has provided an added benefit for me in that it has kept me very humble due to the realization that happens very quickly that there are many aspects of software development that I have simply no clue at all, and it inspires me to read and understand those topics further

1. Read widely

At a software conference that I attended some years ago, I became convinced of the many benefits of reading outside of one’s area of specialty, or “focus”. A speaker had invested significant amount of time and effort researching a problem that had risen recently in the software industry, and during the course of his research had even read studies conducted in areas that seemed unrelated to software. While doing this, he had found a number of elegant ways of tackling a similar problem that had risen and been successfully addressed in another industry (town planning and building construction). During his very captivating presentation, he first proceeded to illustrate the many “parallels” that existed between these two seemingly unrelated industries, and later explained why the same approaches would work in tackling the problem in the software domain. The breadth of knowledge in this man was extraordinary, and this allowed the speaker to come up with an elegant solution without having to “re-invent the wheel”.

1. Listen for feedback

Very few teams seem to do this, but teams that do gain immensely. By talking to the end users after the software has been deployed for use, we can easily learn what went wrong (or right). With this understanding, teams can either repeat (or not repeat) practices that were followed during the course of a particular software project. In the past, I have been very guilty of not following this practice myself, choosing not to call the end users after the software has been delivered out of fear that I would only hear complaints from them. However, when I did force myself to do this in a disciplined way, I was often pleasantly surprised to hear more good news than bad. Even during the few occasions where there was mostly bad news, the end users were somewhat relieved to see that you shared in their problems, and it helped diffuse any potential tension that would have otherwise been detrimental to the relationship in the long run.

P.S: In the picture with me is James Gosling who I had the good fortune of running into very recently. He is the inventor of the Java programming language and software platform that powers approximately 2.4 billion devices including cellphones, PDAs, computers, computer applications and smartcards. He has a “rockstar” image among programmers for helping develop the “smarts” behind the World Wide Web in its early stages, and yet remains a very humble man to this day. He is one of my heroes. This picture was taken at the recent Oracle OpenWorld software conference in the city of San Francisco, United States.

HL7 Programming using Java A Short Tutorial

Sat, 21 Feb 2009 13:19:59 -0600

Before we get started on this tutorial, have a quick look at my earlier article titled “A Very Short Introduction to the HL7 2.x Standard” for information on the HL7 standard. Please note that this tutorial assumes you know Java or any equivalent object-oriented language. A basic understanding of network as well as thread programming will be useful, but is not necessary.

Introduction

Building custom applications using the HL7 standard can be a very daunting task for a beginner. I have built a number of custom HL7 server applications over the years, and couldn’t have done it without the help of others in the field through my many discussions with them over the Internet. As I got more proficient, I also received a number of emails from other developers/customers asking the same questions that I did when I was getting started on this standard. Therefore, in the spirit of sharing, I decided to put together an example/tutorial illustrating all the basics any programmer dealing with the HL7 standard should know.

A couple of things to note before we proceed. Although the HL7 standard itself does not recommend any specific protocol to use for message communication, most HL7 systems communicate using the Minimum Lower Layer Protocol. This tutorial will show you how to build custom HL7 servers in Java using this protocol. Advanced topics such as robust error handling, error logging and notifications, message translation as well as data persistence are not covered in this tutorial. However, I do plan to write another tutorial on those topics in the near future.

What is Minimum Lower Layer Protocol?

Minimum Lower Layer Protocol (often shortened to “MLLP”) is the most popular protocol used for transmitting HL7 messages using TCP/IP. Because TCP/IP transmits information as a continuous stream of bytes, a wrapping protocol is required for communications code to be able to recognize the start and the end of each message (i.e. headers and trailers). MLLP is often used to address this requirement typically using non-printable characters to serve as wrapping characters around the core HL7 message information.

What are Sockets?

You can think of sockets as an abstraction of “terminals” of a connection between two machines transmitting information through either TCP or the UDP protocols. Most modern languages such as Java, C#, VB.NET and Ruby provide socket-programming support to ease the burden of the application programmers having to deal with lower layers of the OSI (Open Systems Interconnect) model of the ISO (International Standards Organization) standard. When using sockets, the programmer is working with the “Session” layer, and therefore shielded from having to deal with real-time data communication services provided by the lower layer of the OSI stack. Java provides excellent support for socket programming (through the java.net package). We will be using the “Socket” and the “ServerSocket” classes for the examples to be illustrated below.

Tools you need

JDK 1.4 SDK or higher
Eclipse or any other Java IDE (or even a text editor)

Steps (We will take “baby” steps so all the core things are covered)

1. Build a simple TCP/IP client capable of transmission and reception of information
1. Build a simple TCP/IP server capable of reception and echo-ing of received information
2. Build simple threading support to handle many concurrent connections from clients
1. Add MLLP and message acknowledgement features into the server
2. Write a small TCP/IP client that will transmit a dummy MLLP-wrapped HL7 message
6. Modify the server class to build MLLP-based parsing functionality
7. Modify the server class to build a simple acknowledgment functionality
8. Put it all together

Step 1 of 8 - Build a simple TCP/IP client capable of transmission and reception of information

Here is a really simple TCP client that we will use for understanding how sockets work. We will use this client to primarily test our server application through the various steps.

As you can see from the code below, the “Socket” class is used to establish a connection with another listener application running locally, listening on port 1080. Once a connection is established, one can read and write through this connection by using the input and output streams. When we are done, we simply close the socket.

Tip: In industrial strength applications, if one cannot establish a connection right away, or if an existing connection is dropped for unforeseen reasons (if the server application crashes, or is shut down accidently), client applications should retry a certain number of times before giving up. I will not be illustrating this functionality as it is not in the scope of this tutorial, however, it is a very useful feature to include in your HL7 application.

Step 2 of 8 - Build a simple TCP/IP server capable of reception and echo-ing of received information

Now that we have seen a TCP client example in the previous step, here is a really simple TCP server that can accept a connection from the client through a specified port, receive any information transmitted by it, and simply echo it back. It will continue running forever in a loop. We typically build server applications in Java using the “ServerSocket” class. This version of our server is capable of accepting and processing only one connection at a time. Once a client connection is accepted by the server, we can read and write through the input and output streams provided by this connection.

Tip: In real 24/7 HL7 server applications, you will have to ensure that the server does not crash by implementing rugged error handling as well as “auto-restart” feature in case the server does crash. Also, you will want to send error notifications through emails in case manual intervention is required quickly, or write to event logs of the operating system.

Step 3 of 8 - Build simple threading support to handle many concurrent connections from clients

The client and server examples illustrated so far are extremely simplified examples to help you understand the basics of socket communications. The server we have implemented is capable of handling only one connection at a time. In order to handle multiple connections, you will have to use threads. Threads are very easy to implement in Java. But, they are also very easy to misuse. When implementing threads in Java, you will have to ensure that the various threads don’t interfere with one another’s work. This is because threads share the same memory. In this modified version of the server, we instantiate a new thread, and pass every incoming connection that is accepted by the server to its own ConnectionHandler class. This will ensure that the server is quickly ready to accept the next client waiting to connect to the server without any noticeable delay.

Tip: In many HL7 systems in use around the world, multiple clients may attempt to connect to a server at the same time (either on different ports, or on the same port). The server application must be able to accept all these connections without too much delay, and then receive, process and acknowledge any information received from each client.

Step 4 of 8 - Add MLLP and message acknowledgement features into the server

Even though the examples illustrated so far allow the client and server to transmit simple text messages back and forth, this will not be enough to communicate with HL7 systems. Communication with HL7 systems will require the use of MLLP protocol and message acknowledgment features. In this step, we will take our existing server and add code to enable MLLP support. We will also implement functionality in the server to create and transmit a simple acknowledgment message back to the client application.

More on MLLP

At the start of this tutorial, I described the MLLP protocol as a “wrapping” protocol where the core HL7 message is enveloped in special characters to signal the start and end of transmission of message information to the receiving application. When dealing with MLLP, there are essentially three character groups that you will configure and use. They are “Start Block (<SB>)”, “End Block (<EB)>” and “Segment Separator (<CR>)”. HL7 messages transmitted using the MLLP protocol are prefixed with a <SB> character, message segments are terminated with a <CR>, and the messages are terminated with two characters: <EB><CR>.

Most often, the character typically used to signify <SB> is a VT (vertical tab), which is ASCII 11. The FS (file separator), ASCII 28, is used to signify <EB>. And the CR (carriage return), which is ASCII 13, is used to signify <CR>. However, the choice of what character to use is entirely up to you.

Here is an example of an observation request HL7 message “enveloped” for transmission using the MLLP-related wrapping characters:

<SB>MSH^~\&199809091533ORU^R01856369D2.2<CR>
PID139385000343456<CR>
ORC123112312RAD19980909-0005<CR><EB><CR>

Step 5 of 8 - Write a small TCP/IP client that will transmit a dummy MLLP-wrapped HL7 message

Here is the original TCP client modified to transmit a sample HL7 message. Notice how the test message is wrapped in the special characters as explained earlier (highlighted in bold).

Step 6 of 8 - Modify the server class to build MLLP-based parsing functionality

In this step, we will add a method to the server (inside the Connection Handler Class) to handle the parsing of the stream of characters received from the client using the socket stream. The gist of this parsing routine is that it will expect MLLP wrapping characters in the correct order/sequence. If not, it will thrown an exception and close the connection with the client.

Step 7 of 8 - Modify the server class to build a simple acknowledgment functionality

Next, we will add two other methods to the server (again, inside the Connection Handler Class) to build simple HL7 message acknowledgements to transmit back to the client. The first method is to help build a basic message acknowledgement functionality. The method shown below is a verify simplified example to aid in the understanding how acknowledgement messages can be constructed. In reality, you will retransmit other fields in the MSH and MSA segments such as sending facility, sending application, version of the application, and other useful information.

The second supporting method is to help parse the message control id of the received message that will need to be re-transmitted in the acknowledgement message. Some portions of this second routine below may seem like an overkill, but I do this deliberately to illustrate how fields inside a HL7 message can be parsed using the field delimiter (once you can parse one field, you can apply the same concept to do the others). You may also want to raise an exception when the message control id is not found.

Step 8 of 8 - Put it all together

Click here to see the entire code listing for our new Server class.

Conclusion

We are done. We have essentially seen all the bits and pieces needed to assemble a custom HL7 application from scratch if you needed to. Often, instead of re-inventing the wheel, a HL7 programmer will use either freeware or commercial toolkits to build his/her HL7 messaging application. However, even when dealing with toolkits, a basic understanding of sockets, the MLLP protocol as well as message acknowledgement fundamentals that were covered in this tutorial is required.

Building robust HL7 systems capable of running 24/7 and unattended is not easy. For most programmers, once they get past the initial socket-programming hurdles, the bulk of the issues in HL7 2.x-related programming will be around supporting any parsing, translating, formatting and storage requirements. With HL7 3.0 standard starting to come into use increasingly, these issues should go slowly away. However, HL7 2.x standard isn’t going anywhere for now. We should expect it to stay at least for 10 more years before they are fully retired. Until then, we will have to continue on our quest to build more elegant and sturdier solutions for HL7 messaging.

Tip: You will want to design the system so that the MLLP character groupings are configurable. This way, the customer can change them to whatever they require them to be. You should also permit some configuration around the message acknowledgment functionality such as information that is transmitted about the processing system and its location (goes in the “MSH” segment). Also, a configuration to indicate whether the system is in “test” mode versus “production mode”. In the past, things that I have made configurable also include message control id starting number or number pattern, email addresses and error log locations (for error notifications), as well as connection retry attempts.

A Very Short Introduction to the HL7 2.x Standard

Mon, 16 Feb 2009 11:44:04 -0600

The abbreviation “HL7” stands for “Health Level Seven, Inc.”. It is a standards developing organization (SDO) that is accredited by the American National Standards Institute (ANSI). The organization was founded in 1987 to produce a standard for the exchange/integration/sharing/retrieval of electronic health information. Before the HL7 standard arrived, hospitals and other healthcare organizations would typically have many different software applications running on a variety of operating systems and hardware platforms, and running in isolation. Even when these applications supported some network connectivity, they would exchange information using proprietary protocols. This made integration of applications, particularly from different vendors, very difficult. For example, a patient registration system would be unable to communicate with a billing system, or a laboratory system, and therefore, these systems would be unable to provide a consolidated view of a patient’s profile for use by healthcare professionals. As a result, most organizations were often locked into proprietary technologies/solutions, and the costs and risks associated with implementing, integrating and maintaining healthcare systems was extremely high.

All this has changed since 1987 when the HL7 organization was created, and it’s membership has grown from a modest 14 members in 1987 to over 2200 members worldwide, including 500 corporate members today and International Affiliates in 33 countries. The standard published by this organization provided rules on how messages for various events occurring within the healthcare domain should be structured, formatted, transmitted and received. The “transmit/acknowledge” communication model forms the core principle of information delivery in systems following the HL7 standard. Acceptance grew rapidly as healthcare organizations as well as vendors realized the benefits it offered. Since the standard did not impose any specific network protocol for information delivery, software systems could communicate using a variety of transport protocols such as LLP, FTP and SMTP (more on all of this later). Besides developing standards to increase the effectiveness, efficiency and quality of their healthcare information delivery, the group also collaborates with other standards development organizations and national and international sanctioning bodies in the development of supportive and compatible standards (e.g., DICOM, IHE). Any one can become a member, and can get access to published standards and other information at a highly discounted rate. The group has also begun offering a certification exam testing the standard’s core concepts. The certification might be useful to have if you want to make yourself more marketable as a systems integrator, software developer or a consultant. Please visit the organization’s website for more information on the certification.

The HL7 messages sent and received can be any one of the thousands of “message types” that are defined in the HL7 message vocabulary. These HL7 messages are often created and sent by an information system in response to an occurrence of an event such as a patient admission, a patient discharge or a it may be a query to another system. And each message contains information about that event. And further, each of these HL7 messages are themselves made up of “segments”. These segments are considered the real building blocks of HL7 messages although the segments themselves are in turn comprised of “fields”. A segment has a three character name and a pre-defined format of specific fields. The fields are separated by the “|” (pipe) character, and may be further divided into “sub-components” with the “^” character. A “PID” segment for example contains information patient information such as ID numbers, name, address and date of birth. Different HL7 events trigger different message types - each message type has a defined set of segments that are joined together to provide all the required information regarding that event. Some segments are mandatory, and must be included in the message, and other segments are optional.

For example, an “ADT^A01” message (“ADT” stands for “Admission, Discharge, Transfer”) is sent as a result of a patient undergoing the admission process which assigns the patient to a bed. It signals the beginning of a patient’s stay in a healthcare facility. This event can trigger a notification to other auxiliary systems, say, a pharmacy system that a patient has been admitted and may be legitimately prescribed drugs; the nursing system that the patient has been admitted and needs a care plan prepared; the finance system of the start of the billing period; the laboratory, pathology and the radiology systems that a patient has been admitted and is entitled to receive services. This message is comprised of the MSH, EVN, PID, PV1 segments. The MSH segment contains details about the message such as sending and receiving systems, dates and message type. the EVN segment contains details about the triggered event such as event type, event date/time, the person that triggered the event. The PID segment contains details about the relevant patient. PV1 segment contains details of the patient’s visit to hospital, such as visit number, ward/bed, attending doctor, financial classification. So, all relevant information about the patient admission event is communicated to the receiving systems using this message. In the same manner, other information pertaining to patient, observation, finance and billing-related information can also be communicated between various software systems. The HL7 standards manual available from HL7 Inc. provides extensive details on trigger events, message types, and segments. interoperability between local vendors. Please refer to the standards body relevant to your country for more details.

When the incoming message reaches the receiving system, a message acknowledgment (called “ACK” in HL7) is transmitted back to the sending system. This acknowledgment can take many different forms based on how well the receiving system was able to process the incoming message. If the message was successfully processed, a positive ACK message is sent to the sending system and the communication is concluded for that particular message. In HL7, a positive ACK is communicated by using an AA (Application Acknowledgment) value in the first field of the MSA segment of the transmitted The 3rd field in the MSA segment of the acknowledgment contains the same message control ID that was in the message created by the sending system. This permits the sending system to match the acknowledgment to the original message that it transmitted earlier. If there were errors during the processing of the incoming message, then negative acknowledgments (called “NACK” in HL7) are transmitted back to the sending system. The transmitting system can then diagnose, fix, and later resend the failed message back to the receiving system. A negative ACK is indicated by using an AE (Application Error) value in the first field of the MSA segment. This code indicates that were either a problem with the message structure, or the message itself.

This forms the foundation on which the protocol is built. In a later article, I will illustrate, using a small example in Java, how to transmit, receive, parse and acknowledge HL7 messages using the MLLP protocol, which is the most popular form of transmitting and receiving HL7 messages today.

The Broken Windows Story

Sat, 18 Dec 2004 22:03:04 -0600

Some days ago, I was reading an excellent book that was recommended by a colleague/mentor. The book was called "Pragmatic Programmer" (written by Andrew Hunt and Dave Thomas), and one story in the book really stood out for me. It was titled "Don’t live with Broken Windows". The story was based on a crime-related study that showed that when police tackled seemingly small crimes (such as broken windows, graffiti, etc) more aggressively, the rate of big crimes also reduced as a result in those neighbourhoods. People living in those areas felt a heightened sense of pride in their neighbourhood as a result. The authors then draw an excellent analogy to the software industry and to the fact that when quality of software goes unaddressed, even seemingly small problems escalate to large problems eventually, and the team morale also begins to suffer.

Sometime ago, I was an unfortunate participant and witness in one such project, and have learned many valuable lessons from that experience. I went into a project that was only three months from implementation (after nearly 4 years in the works), and a number of features were either still not implemented still or were not working correctly. Because of poor leadership (and due to some bad hiring decisions), almost every member of the original team that had helped to design or develop major portions of the system had either quit, or were let go. The project was nearing its implementation date, and the project owners were anxious to get this done. And so, a bunch of us went on a "rescue mission" to fix things. But, we were in for a huge surprise. We found it nearly impossible to implement the missing features or fix any pending defects for a number of reasons. We noticed extremely hard to read the existing code (some PL/SQL procedures were up to 5000 lines long) filled with hard coded numbers and dates and it was hard to tell their significance without any comments as well. There was also very poor error handling, error logging and other debugging mechanisms in the system making it harder to troubleshoot problems still. Compounding this situation was the fact that most documentation around the project had become really out of date. The code really needed heavy “refactoring” (clean up), but given the looming deadline and other constraints, the software project management team advised us to go ahead and just "ram" our fixes in. So, we did our best, and did implement the system in time although the whole experience was most stressful for everyone involved.

After this implementation, I couldn’t help but wonder after about my many findings during the course of the project, especially about the huge number of clues that had emerged even at the beginning when things appeared to be going smoothly. For example, looking at the defect logs, I could almost trace a “history” of the attitude of the original project team as many team members seem to have operated under the belief that unless a defect was in the area of their direct responsibility, then it was not something they needed to be concerned with. This approach was used to successfully ignore many issues (including some core architectural problems) which were in areas used by everyone. It was not long before these problems grew many fold very quickly, and started to impact almost everyone involved. The team members then started to point fingers at each other instead of collectively working together to solve these problems. No one felt any pride about what had been accomplished overall as a result even though many members of the team sacrificed so much of their family time to help get this project completed.

For me, this particular project will always stand out as a perfect example of what happens when software teams do not place emphasis on code quality right from the beginning. It helped highlight the importance of keeping both the requirements and design documentation current, and the value of unit and automated testing mechanisms. Most importantly, this “fiasco” helped highlight the fact that software projects should provide an atmosphere for people to speak-up and challenge approaches to all aspects of a system being developed (even on areas that may not be strictly in the realm of responsibility for some team members). When such practices and approaches are adopted, software teams should be able to enhance or fix any problems in software much more quickly.