Secondary use of electronic medical records for clinical research: challenges and opportunities

The motivation of our case study is to characterize postoperative pain following different breast cancer treatments using EMRs from two healthcare systems: Stanford Healthcare Medical Center, a tertiary academic hospital, and the Veterans Health Administration (VHA). In the Stanford Healthcare Medical Center, further referred to as AH, patient records were identified from Epic's Clarity relational database. VHA clinical data were extracted from the VA Corporate Data Warehouse (CDW), which includes national data from multiple medical centers [12].

Patients with breast cancer excision surgeries with a previous documented diagnosis of breast cancer, taken from 2008/01/01 to 2016/06/30, were extracted from the EMRs. Breast cancer diagnosis was identified by ICD-9-CM codes '174*' and '175*' and ICD-10-CM code 'C50*'. When multiple surgeries occurred, only the first excision surgery was included. Only patients with body mass index (BMI), preoperative pain, postoperative pain, pain at 30 d (+14/−  7 d), and 3 months (±14 d) were included. The number of patients with excision surgeries filtered by diagnosis codes were at 3387 and 3308 patients for AH and VHA respectively; after all exclusionary criteria was applied, 735 and 1210 patients remained.

To understand different aspects of patient pain after surgery, we assess multiple outcomes: postoperative length-of-stay in days (LOS), discharge day pain (a 0–10 numeric score), prescription of analgesics, physical therapy (PT) visits, and pain management (PM) visits (identified through institution-specific visit department names or classifications) at 30 d (+14/−7 d), 3 months (±14 d), and 1 year (±84 d) outcome time points. Each outcome is meant to be a proxy measure for postoperative pain in different ways. For example, a patient may have an extended LOS if their postoperative pain is not appropriately controlled. Similarly, PM, analgesics prescription, and PT may be prescribed for patients with high pain.

Patient demographics variables included age, race/ethnicity, gender, Charlson score, and BMI at the time of surgery. Race/ethnicity was merged as follows: if hispanic was marked, they were considered hispanic, if there were more than one race marked, they were considered 'other', otherwise the race was given as the final race/ethnicity value. We only took female and male genders in this study. The last BMI prior to surgery was taken. We also included diagnosis of anxiety (supplemental table 1 (stacks.iop.org/CSPO/4/014001/mmedia)) and diagnosis of depression (supplemental table 2), as they have been shown to have longer LOS and postsurgical complications [13]. In AH, medication names are mapped to RxNorm and in the VHA, drugs are mapped to the national drug formulary (supplemental table 3). We developed categories to rate the invasiveness of a breast cancer excision surgery. The list of codes using current procedural terminology (CPT) were identified by examining the CPT ontology as well as including frequent billing codes by the department. A domain expert reviewed the list of codes and classified the excision surgeries as mild, moderate and severe (supplemental table 4). Reconstructions were similarly categorized as one of 3 categories: autologous, implant, and oncoplastic (supplemental table 5).

To control for confounding variables, we added information about other types of patient treatments that may occur from breast cancer patients measured at multiple time ranges. Included are additional breast excision and reconstruction surgeries, as well as common breast cancer treatments such as chemotherapy (supplemental table 6), radiation therapy (supplemental table 7), hormone therapy (supplemental table 8) [10]; in addition to PT, PM and other surgery (identified through HCUP surgery flag excluding excision/reconstruction surgery [14]). Time periods included several time ranges prior to surgery {(anytime up to 1 year prior to surgery), (between 1 year and 3 months prior to surgery), (3 months to surgery date)}, as well as several time periods between surgery date and an outcome time point {(surgery day to 3 months prior to outcome), (3 months to 1 month prior to outcome), (less than 1 month to outcome)}. These are not factored in simple bivariate analysis, however are included as variables in multivariate analysis.

We measured bivariate association for each variable against each outcome using spearman correlation, Fisher's exact test, chi-squared test, and Kruskal–Wallis ANOVA test as appropriate. Specifically, for two numeric variables, we used Spearman correlation; for two categorical variables, Fisher's exact test was used if any values were less than five, otherwise chi-squared test was used; finally, for one categorical and one numeric variable, we used Kruskal–Wallis ANOVA. We developed regression models to test the association between our dependent and independent variables of interest. We applied linear regression for numeric outcomes and binary logistic regression for binary outcomes. For example, dependent variables pain and LOS are numeric variables so a linear regression model was developed; meanwhile, whether or not patients had an analgesic prescription or had a PM visit or a PT visit are binary dependent outcomes and therefore a logistic regression was used. Processing of EMR data was done using SQL and java. Statistical analysis was performed using python packages pandas and scipy and R packages glm and lm. This study was approved by the IRB at both the AH and the VHA.

In addition to the standard statistical analysis, we use our case study to exemplify challenges of clinical studies using EMR. This includes the effects of using multiple institutions with different data organization, the influence of EMR definitions on results, the bias of disease populations on outcomes, the choice of modeling on results, and the difficulty of incorporating long-term information in EMR studies. When possible, we include quantitative analysis in addition to qualitative analysis.

AH is a tertiary care center, and therefore more likely to have patients come for treatment and receive postoperative care elsewhere. This results in a scenario in which many patients may be lost to follow up. On the other hand, VA data represents a national system, including approximately 153 hospital facilities and 788 community-based outpatient clinics [15], in which patients going to multiple VA hospital clinics will still be captured by the system. Furthermore, pain (one of our defining filters) is not always uniformly recorded across visits (pain score were not always collected, and if collected they were collected at different frequencies). In an attempt to capture long-term postoperative pain scores (i.e. 1 year follow-up), the time-window of assessment had to be larger, ranging from 281 to 449. In AH, the drop-off discharge (2277 patients in the beginning), went to 64, 41 and 32% at the 30 d, 3 month, and 1 year follow-up; in contrast, in the VHA, the drop off from discharge (2429 patients) was 74, 54, and 50 percent respectively. Thus, despite such a lenient window, the rates of loss-to follow up were substantial and only 32% had pain information recorded during the 1 year follow-up period.

Table 4 shows the raw number of patients that visit after surgery, prior to filtering by pain information available at different time points. The percentage of patients that drop off is steeper for AH compared to VHA; meanwhile, the percent of patients with pain information is higher in the VHA. The return of patients may signal a bias in the resulting population for people who had worse problems for both locations. VHA pain data because of its larger care system and mandated collection as a vital sign had substantially less of dropout using the same time specifications.

While AH is a private single institution, the VHA CDW data is national data already transformed from individual centers around the country, some data granularity may be lost due to data processing. Biases arise from accommodating for multi-institutions' differences in practice. For example, while 0–10 numeric pain scores were available both in AH and VHA, there were differences in the collection methods. AH pain score data had much more granularity including location, descriptions, alleviating, aggravating factors, etc.

In order to normalize for the substantially less informative VHA CDW data, anatomic locations of pain scores were dropped from the AH data. Therefore, it would be possible (especially for time points further out) that the pain measured from the EMR was not in fact due to the breast excision or reconstruction surgery. The implications of these various constraints led to two-fold results: (1) noisier time windows for further time points and (2) unspecified pain measurements.

Furthermore, the difference in payment can be reflected in records. For example, AH is paid for care it provides (thus it has incentives to maintain coding systems that support its revenue). VHA is capitated and there is no incentive to code effectively. VHA physicians, therefore, tend to rely on more data in notes than in structured codes. For our cohort, the AH population patients were found to have a max, minimum, and average of 552, 16, and 141 ICD-9 codes for their entire record; in contrast it was 355, 1, and 96 in the VHA.

Despite using common vocabularies such as ICD9, ICD10, CPT codes, there remains many different ways in which the same information, e.g. chemotherapy, can be identified. The chief reason for this is that EMR data are imperfect proxies for events we wish to study. An event can be identified using different methods which have different accuracies depending on institution. Even for the same concept, e.g. diagnosis of anxiety, there may be different definitions available [13, 16]. In this study, chemotherapy was defined using CPT codes alone, however it could also be defined using chemotherapy medications alone, or both in conjunction. In table 5, we show that different definitions can lead to different frequency counts. These minor differences may propagate to differences in analytical conclusions. For example, in our bivariate analysis, depending on the identified outcome, e.g. LOS, discharge pain, 30 d, 3 month or 1 year pain, different chemotherapy-related confounding variables were found to be significant and at different levels. For discharge pain, whether or not if the patient received chemotherapy within 1 year to 3 months prior to surgery and whether or not if the patient received chemotherapy within 1 month prior to surgery were significant in using CPT and medication prescription dates, however the significance level for the latter for the former was for an alpha level of 0.1 in contrast to the latter which was at 0.05. For 30 d pain, whether or not the patient received chemotherapy immediately after surgery was shown to be significant within 0.05 alpha levels for CPT codes, but only within 0.15 alpha levels for medications; and only medications showed a 0.05 significance for concurrent chemotherapy within the 30 d window.

Another challenge is that some variables may be institution-dependent which introduces room for data mapping issues. For example, for this study we identified use of PT and PM services as a visit to a particular department(s) within the healthcare setting. This required manual review of institutional-dependent coding. Identification of the closest corresponding data in the VHA were stop-codes, which are classifications of visit type that are assigned after the time of the visit. In another example, for identification of medication related variables such as hormone therapy and analgesic, required using two separate vocabularies (RxNorm in AH and NDF-RT in VHA) which are not easily mapable [17].

In order to compare how certain processes are correlated with good and bad outcomes, various procedures and outcomes must be recorded for fair comparison. However, patients may obtain their health care services from multiple locations, e.g. a patient that receives surgery in one institution but continues care back at their local unconnected clinics. Therefore, the information recorded in one EMR may not be complete or accurate. Furthermore, health care is a service that is typically sought out for when there are health concerns which leads to data representation problems. For example, we can only capture pain scores or use of PT if patients request care—healthy patients are unlikely to return to the hospital for unnecessary care. For our outcomes, the percent of patients with PT or PM visits were lower than 25 percent and average pain scores were lower than pain score of 3 (supplemental table 9). Thus, there is an inherent bias in conclusions derived from EMR studies as they often represented sicker patients.

Besides challenges due to problems in follow-up, outcomes are difficult to measure for several reasons. First, patients do not always report their issues (patients may not seek help for pain). Second, some outcomes may be subjective and between patient variation can be significant. Third, even if reported by the patient, problems may not be recorded in the EMR. If reported, it may not be in structured form, and therefore sophisticated data mining algorithms may be necessary to utilize these data. Finally, another challenge is that typically the number of patients with severe issues is much smaller compared to the number of patients with 'no problems'. In our study, in all time periods, pain score measurements were concentrated towards the low pain scores as indicated by the percentile values (supplemental table 8). For all time periods, the 75 percentiles are at most at a pain score of 4. Meanwhile, use of PM and PT are relatively less populated, especially the PM usage criteria, all less than 23% for both institutions.

The differences in the predictive results for bivariate versus regression analysis and AH and VA populations is indicative of some very challenging issues faced by clinical sciences. The first is the modeling choices and defining which variables need to be controlled for in the population. For example, bivariate analysis gives a very simple view. However, they do not take into account a host of influencing variables available in the EMR, such as additional treatments after initial operation. On the other hand, adding many other variables makes statistical inference more difficult—thus without sufficient representation of various combinations of treatments, true relationships may be masked by idiosyncrasies in the dataset. The second issue is related to the latter point, which is, the influence in sample populations in influencing results. We see that in the VHA population diagnosed depression and anxiety is much more common, meanwhile the representation of breast cancer among men is higher as a result of a large male population in the VA. These can be true indicators of some connection of the disease or they can be spurious relations to the dataset.

The use of multivariate analysis allows for the inclusion of confounding effects such as other treatments and operations. Our choice of simple linear regression for numeric outcomes and binary logistic regression for binary outcomes was based on simplicity and interpretability. While this offers only a first order approximate picture of how individual study variables may have some effect on outcomes, the complexity of human biology guarantees there are many non-linear interaction amongst input variables that can affect outcomes that are difficult to capture in such a simplistic way. Restated, the assumptions of each model (as well as its significance measures) may not be acceptable for every question—and since we do not know the answer, it is impossible to know when they are or are not appropriate. Use of other models that fit to the data, for example decision trees, higher order regression models, or regression models with interaction terms can do better to model nonlinearities in data. Unfortunately, again, it is impossible to know when you are doing the 'right' amount of fitting when you do not know the truth.

Although understanding of long-term treatments outcomes is crucial, longitudinal studies have more confounding variables to control for as patients will undergo various physiological changes and additional treatments. Indeed, we found that many variables that were significantly related to the outcomes, were confounding variables such as recent or current (within the time frame) treatments for chemotherapy, radiation therapy, hormone therapy, reconstruction surgery, and past opioid prescription. However, some variables indicate a more long-term pain problems (e.g. Chemotherapy, PM, or pain prior to surgery). Effectively, longitudinal studies are riddled with a practical problem: if confounding variables are not included, the study is not realistic, if they are included the data becomes sparse, making it even harder to draw conclusions. Table 6 show the amount of patients who have different other treatments, e.g. chemotherapy, within the time-frame of the study for AH, broken down relative to the operation day. For example, from the first row, 12 people had chemotherapy more than 1 year prior to the operative day Models not only have to generalize all the different types of treatments available over time (e.g. each box in the tables), but also have to deal with all permutations of them! Taking a logistic regression model as an example, a plain logistic regression with non-processed inputs will miss many non-linear relationships if no interaction terms are allowed. However, if interaction terms are allowed, then there becomes combinatorially more variables to reason over, for which it is easy to overfit, and for which there is not enough data to provide enough examples.

To get a sense of patients in our selected cohort, we randomly sampled 10 patients from AH and manually reviewed all their medical notes within their 1 year time periods. Only one patient during their 1 year time period was not receiving additional revision surgeries, biopsies, or undergoing chemo- or hormone therapy. This manual review suggests that we are selecting patients who will tend to have additional medical problems whether or not they may be related to the original breast cancer, a metastatic condition, additional medical procedures, or continuing health issues. While one can attempt to control for this in the multivariate analysis, this would be a serious issue of bivariate analysis of long-term outcomes.

Efforts are underway to develop shared data repositories as well as data normalization standards [18–21]. This includes virtual warehouses that can query multiple institutions using the same data model for larger cohorts. These efforts will be critical for obtaining greater cohort population numbers which can be used in retrospective clinical studies. With a greater number of cases, many combinations of patient treatment and characteristic nuances may be taken account of whilst not exasperating the data sparsity problem (small numbers can give arbitrary attribution of clinical significance). Of course, pooling data for meaningful research is nontrivial, and before such efforts can take place, problems such as data security and privacy needs careful attention.

Use of natural language processing applied to biomedicine is increasing in prevalence [22]. As free text is a critical mode of communication among clinical teams, this will mitigate the difficulties of data incompleteness. Moreover, this area offers rich new information as a majority of clinical patient information is locked in text. For example, a clinical note may include information about patients' histories at other institutions or provide detailed and nuanced medical histories. In fact, for several patients who had reconstruction surgeries prior to their first excision surgery, manual review of clinical notes revealed that 4/6 had a previously unrecorded structured data signally an excision surgery. Though careful quantification of errors associated with NLP necessitates additional quality assessments, if applied judiciously, this may lead to an increase in the quality of data, as well as the quantity of data captured.

Another strategy of maximizing information from data, is to decrease noise by increasing the quality of data at the point of capture. However, the burden of entering data generally falls on over-worked clinical staff, whose first priority is to care for their patients. In fact, identified barriers for EMR adoptions includes perceived disruption to clinical workflow, software adaptability and complexity, as well as training and leadership engagement [23]. As the primary purpose of EMRs in a healthcare organization is for billing and record keeping, The result is uneven data capture that is sensitive to local institution protocol, software usability, and changes to billing practices. There is increased awareness towards usability problems and frustrations in EMR software [23, 24]. This is leading to proposals for more user-centric design of EMR software that can shift the burden of making minute, nuanced data input decisions away from clinicians [25, 26]. With design of EMR software with emphasis on usability, clinicians can focus on care of patients with the burden of sifting through correct documentation eased through better human interfaces. Cleaner and more consistent datasets would reduce data definition transferability problems, and with a greater number of data points, would help mitigate large variances in modeling to identify clinically significant variables.