In the first week of May, Prior to the HL7 working group meeting in the beautiful city of Cologne, we published a new Forge release for FHIR STU3 (version 18.6). In this article, we will take a closer look at some of the new features and changes introduced in this release. In addition, we will provide some insight into the underlying architectural changes and discuss the general application roadmap.
Before we discuss how Forge manages profile dependencies, we will briefly look into the workings of the FHIR StructureDefinition resource. Feel free to skip this section if you are already familiar with this key resource.
A FHIR profile has a list of dependencies to external profiles. The StructureDefinition resource defines a dependency to the associated base profile from which the current profile is derived. Each ElementDefinition can also define one or more references to external datatype profiles, including extension definitions. A profile reference is based on the canonical url of the target profile. FHIR does not describe exactly how a system should resolve a canonical url reference to a target profile; this depends entirely on the implementation. A typical FHIR server would probably perform a database lookup. A desktop application such as Forge will try to locate a unique matching profile on local disk. Alternatively, systems could try to resolve profile dependencies remotely from a FHIR profile registry such as simplifier.net.
A FHIR profile specifies a set of constraints on the base profile. These profile constraints are captured by the differential component of a StructureDefinition resource. The differential component represents a sparse view of the profile, defining only the actual constraints with respect to the base profile. Unconstrained elements and properties are inherited as-is from the base profile. This redundant information does not need to be redefined and can be safely omitted from the differential component. A profile with an empty differential component is structurally identical to the base profile.
However, any logic that needs to reason about FHIR profiles requires access to a complete description of all the elements and properties that describe the model. This complete model description is captured by the snapshot component of a StructureDefinition resource. The FHIR standard defines a set of rules for generating the complete snapshot component from the differential component, by merging all the user constraints defined in the differential component with the associated constraints that a profile inherits from the base profile and any element type profiles.
Provided that a system can resolve all referenced profile dependencies, snapshot components can be computed on demand. This allows such a system to exchange and store only the (sparse) differential components. However, if a system is unable to resolve one or more profile dependencies, then a complete snapshot cannot be (reliably) generated. Also, if a system resolves dependencies incorrectly, e.g. to a different version of the target profile than originally intended, then the computed snapshot will also be incorrect and/or different.
Especially once we start collaborating on a set of profiles and introduce versioning, it becomes clear that we could soon find ourselves stuck into – excusez le mot – dependency hell. This explains why reliable dependency management is crucial when processing FHIR profiles.
When you open a profile in Forge, the application completely ignores an existing snapshot component that may exist in the StructureDefinition instance, as the application cannot simply assume that the (externally) generated snapshot is reliable and up to date. Instead, Forge always considers the user-authored differential component as being the ultimate source of truth. After deserializing the profile, Forge calls the FHIR API library to re-generate the snapshot component on the fly. For this process to complete successfully, the application requires access to all referenced external profiles. If any dependency is missing, the resulting snapshot is not guaranteed to be complete and therefore unsafe to be used reliably.
Forge always resolves references to core datatypes and profiles from the core FHIR specification, using the standard core definitions included in the application. References to all other profiles need to be resolved externally. Since the original release of Forge for DSTU1, the application maintains an in-memory list of profiles that are currently open for editing. Originally, Forge would only resolve external references from the in-memory list of currently opened profiles.
In Forge for STU3, we updated the resolving mechanism. When Forge for STU3 needs to generate a snapshot, the application actually scans all the XML files that exist in the directory that contains the profile, and optionally also subdirectories (depending on the application configuration setting “Resolve resources from subdirectories” in the Options menu), in order to discover and index any other external FHIR profiles. Forge will then try to resolve all external profile references from the generated directory index. Snapshot generation occurs whenever you open a profile, and also when you save a profile and the “Save snapshot component” option is enabled.
Note: make sure the option “Resolve resources from subdirectories” is disabled when saving/opening to a folder with many subfolders or non-FHIR content; especially the My Documents folder, Temp folder or a disk root folder. Otherwise Forge will happily start a prolonged and uninterruptible indexing process, wasting lots of precious CPU cycles and memory. We are looking into improving the UI in order to clarify the application behavior and encourage the use of isolated working folders.
To improve support for local dependency management, our team has spent some precious time and effort into designing and implementing powerful new functionality into the FHIR .NET API library that allows us to quickly scan and index FHIR resources stored in a directory folder structure on disk. The API is now capable of quickly and efficiently extracting relevant summary information from resources on disk, regardless of the actual serialization format, without requiring full (slow!) deserialization, from resources inside of Bundles and ZIP archives and even from invalid/incompatible FHIR resources. See the ISummarySource interface definition on GitHub for more information. The new summary indexer proves to be blazingly fast and allows applications such as Forge to quickly explore the contents of a directory (structure) and track base profile hierarchies, without requiring any costly full deserialization. Noice.
Gradually, we are moving Forge towards supporting a folder-based workflow, by rewriting the internal Forge logic to profit from the new disk-based summary index and resolver and updating UI components accordingly. As mentioned earlier, Forge already leverages the new logic when generating snapshots, by resolving external profile dependencies from the containing directory (structure). As of release 18.6, two dialog windows have been redesigned and re-implemented, in order to improve the user interface, leveraging the new summary indexer and removing dependencies to obsolete logic.
The improved New Profile dialog allows you to easily create a new profile, extension definition or logical model. After selecting the base profile, Forge will pre-fill some basic properties such as the canonical url. You can inspect and optionally customize the generated default values before confirming. The same dialog also allows you to create a derived profile by selecting an external base profile from disk. Forge will index the containing directory and list all the discovered profiles, allowing you to select another base profile if necessary.
The Extension Context selection dialog has also been streamlined, improving the user interface and removing dependencies to obsolete logic that we could now remove.
Some profile editing features are still based on the historic in-memory resolving mechanism and remain to be migrated to the new folder-based resolver:
- The Extension Selection drop-down combo-box displays selection options generated from the list of currently loaded extension definitions;
- The list of element types displays compatible type selection options generated from the list of currently loaded profiles.
In the upcoming releases, we will focus on migrating all features to the new folder-based resolver, so we can completely remove the obsolete in-memory resolving logic. Once the resolver migration is complete, we will finish the transition towards the new workflow by introducing a newly designed Folder Explorer that we have been prototyping internally. The new Folder explorer is designed to support and encourage a folder-based workflow and provides complete and detailed insight into the local profile resolving scope. From the folder explorer, a user can easily browse, edit and manage all the FHIR resources that Forge has indexed from the selected directory (structure).
Eventually, we envision moving towards some form of project files, similar to popular programming IDEs. Project files would allow authors to easily manage multiple profiling projects with different configuration options, which is especially useful for power users. However we also want Forge to be intuitive and easy to use for first time users that just want to create their first profile. When adding new features, we need to strike a delicate balance and not further increase the entry barrier and/or learning curve. In order to integrate support for project files, we will have to carefully look into the design first.
At the HL7 Working Group Meeting in New Orleans in January, the FHIR community identified the need for managing releases of (sets of related) official, final artifacts and the apparent lack of FHIR tooling support. Members of the FHIR-I working group, having a strong background in software development, took inspiration from existing programming IDEs that provide a user-friendly package manager for publishing and consuming versioned binary components. Ewout Kramer, Firely CTO and co-founder of the FHIR standard, published an informative article where he explores how FHIR could benefit from using package managers.
At Firely, we are planning to tightly integrate package management into both Simplifier and Forge. We will allow users to publish versioned packages with their projects on Simplifier. And we are also planning to integrate a user-friendly package manager client into Forge that will allow users to quickly browse, find, download and install FHIR packages from Simplifier and/or other package repositories. The Forge package manager will cooperate nicely with the upcoming folder explorer, allowing you to easily browse the contents of imported packages.
Package management is looking quite promising and is going to be a great addition to our FHIR tool stack, bringing us one step closer to our goal of providing user-friendly support for the full profiling workflow, from collaborative design and authoring to publication, release maintenance and archiving. Time permitting, we can bring some improved tooling to the next HL7 FHIR Connectathon in September and evaluate different package management scenarios and interoperability. In the meantime, you are very welcome to join us at FHIR DevDays in Boston, 19 to 21st of June, organized by Firely in cooperation with HL7.