Main Page

From TemporalWiki

Revision as of 21:38, 21 November 2014 by Guergana.savova (Talk | contribs)
Jump to: navigation, search

Contents

Welcome to the THYME project

Welcome to the Temporal Histories of Your Medical Event (THYME) project (THYME is pronounced [taim]).

The overarching long-term vision of our research is to create novel technologies for processing clinical free text. Such technologies will enable sophisticated and efficient indexing, retrieval and data mining over the ever increasing amounts of electronic clinical data. Processing free text poses a number of challenges to which the fields of Artificial intelligence, natural language processing and computer science in general have made advances. Methods for processing free text are informed by linguistic theory combined with the power of statistical inferencing. A key component to the next step, natural language understanding, is discovering events and their relations on a timeline. Temporal relations are of prime importance in biomedicine as they are intrinsically linked to diseases, signs and symptoms, and treatments. Understanding the timeline of clinically relevant events is key to the next generation of translational research where the importance of generalizing over large amounts of data holds the promise of deciphering biomedical puzzles.

The goal of our current proposal is to discover temporal relations from clinical free text through achieving four specific aims:

Specific Aim 1: Develop (1) a temporal relation annotation schema and guidelines for clinical free text based on TimeML, which will require extensions to Treebank, PropBank and VerbNet annotation guidelines to the clinical domain, (2) an annotated corpus (500K words of clinical narrative) following the temporal relations schema with additions to Treebank, PropBank and VerbNet, (3) a descriptive study comparing temporal relations in the clinical and general domains.

Specific Aim 2: Extend and evaluate existing methods and/or develop new algorithms for temporal relation discovery in the clinical domain. Component-level evaluation

Specific Aim 3: Integrate best method and/or a variety of methods for temporal relation discovery into Apache cTAKES (ctakes.apache.org) and release as open source annotators in the pipeline. Functional testing. Dissemination activities.

Specific Aim 4: System-level evaluation. Test the functionality of the enhanced Apache cTAKES (ctakes.apache.org) on translational research use cases, e.g. the progression of colon cancer as documented in clinical notes and pathology reports, the progression of brain tumor as documented in radiology reports.

The methods we will use for the temporal relation discovery are based on machine learning, e.g., Support Vector Machine technology. Such methods require the annotation of a reference standard from which the computations are derived. The best methods will be released as part of the cTAKES for the larger community to use and contribute to. We will test the methods against biomedical queries.

Funding

The project described is supported by Grant Number R01LM010090 from the National Library Of Medicine and the i2b2 project (U54LM008748 from the National Library of Medicine). The content is solely the responsibility of the authors and does not necessarily represent the official views of the National Library Of Medicine or the National Institutes of Health. The project period is October, 2010 - September, 2014.

Who We Are

  • University of Colorado
    • Martha Palmer (MPI)
    • Jim Martin
    • Wayne Ward
    • Steven Bethard (at University of Alabama at Burmingham as of Sept, 2013)
    • Will Styler
    • Arrick Lanfranchi (through August, 2012)
    • Tim O'Gorman
    • Kevin Crooks (through December 2013)
    • Mariah Hamang
    • and several Lingustics and Computer Science graduate students
  • Boston Childrens Hospital/Harvard Medical School
    • Guergana Savova (MPI)
    • Dmitriy Dligach
    • Timothy Miller
    • Sameer Pradhan
    • Sean Finan
    • Chen Lin
    • David Harris
    • Jennifer Green (through December, 2013)
  • Mayo Clinic
    • Piet de Groen
    • Brad Erickson
    • James Masanz
    • Donna Ihrke (through December, 2012)
    • Pauline Funk (through January, 2013)
  • Brandeis University
    • James Pustejovsky

Publications and presentations crediting THYME

2014

  • Lin, Chen; Karlson, Elizabeth; Dligach, Dmitriy; Ramirez, Monica; Miller, Timothy; Mo, Huan; Braggs, Natalie; Cagan, Andrew; Denny, Joshua; Savova, Guergana. 2014. Automatic identification of Methotrexade-induced liver toxicity in Rheumatoid Arthritis patients from the electronic medical records. Journal of the Medical Informatics Association. http://jamia.bmj.com/content/early/2014/10/24/amiajnl-2014-002642.abstract
  • Pascal B. Pfiffner, JiWon Oh, Timothy A. Miller, Kenneth D. Mandl. 2014. ClinicalTrials.gov as a Data Source for Semi-Automated Point-Of-Care Trial Eligibility Screening. PlosOne. DOI: 10.1371/journal.pone.0111055. http://www.plosone.org/article/info%3Adoi%2F10.1371%2Fjournal.pone.0111055#abstract0
  • Pradhan, Sameer; Elhadad, Noemie; South, Brett; Martinez, David; Christensen, Lee; Vogel, Amy; Suominen, Hanna; Chapman, Wendy; Savova, Guergana.2014. Evaluating the state of the art in disorder recognition and normalization of the clinical narrative. Journal of the Medical Informatics Association. http://jamia.bmj.com/content/early/2014/08/21/amiajnl-2013-002544.full.pdf+html
  • Sameer Pradhan, Noemie Elhadad, Wendy Chapman, Suresh Manandhar, Guergana Savova. 2014. SemEval 2014: Task 7. In Proceedings of the International Workshop on Semantic Evaluations, Dublin, Ireland. August. http://alt.qcri.org/semeval2014/cdrom/
  • Finan, Sean; De Groen, Piet; Savova, Guergana. 2014. Narrative event and temporal relation visualization tool. American Medical Informatics Association annual symposium. November 2014. Washington, DC.
  • Bethard, Steven. 2014. The state of the art of temporal relation extraction. Presentation at the NLP workshop at the 4th i2b2 Academic User Group conference. July 9, 2014. Boston, MA.
  • Miller, Timothy. 2014. Methods for temporal relation discovery in the clinical narrative. Presentation at the NLP workshop at the 4th i2b2 Academic User Group conference. July 9, 2014. Boston, MA.
  • Pradhan, Sameer. 2014. Extrinsic evaluation of temporal relation discovery system. Presentation at the NLP workshop at the 4th i2b2 Academic User Group conference. July 9, 2014. Boston, MA.
  • Finan, Sean. 2014. Visualization tool for temporal relations from the clinical narrative. Presentation at the NLP workshop at the 4th i2b2 Academic User Group conference. July 9, 2014. Boston, MA.
  • Chen, Pei. 2014. Modules for temporal relation discovery from the clinical narrative in Apache cTAKES. Presentation at the NLP workshop at the 4th i2b2 Academic User Group conference. July 9, 2014. Boston, MA.
  • Sameer Pradhan, Xiaoqiang Luo, Marta Recasens, Eduard Hovy, Vincent Ng and Michael Strube. 2014. Scoring Coreference Partitions of Predicted Mentions: A Reference Implementation. Short paper. Association for Computational Linguistics Conference. Baltimore, Maryland. http://anthology.aclweb.org//
  • Xiaoqiang Luo, Sameer Pradhan, Marta Recasens and Eduard Hovy. 2014. An Extension of BLANC to System Mentions. Short paper. Association for Computational Linguistics Conference. Baltimore, Maryland. http://anthology.aclweb.org//
  • Chen Lin, Timothy Miller, Alvin Kho, Steven Bethard, Dmitriy Dligach, Sameer Pradhan and Guergana Savova. 2014. Descending-Path Convolution Kernel for Syntactic Structures. Short paper. Association for Computational Linguistics Conference. Baltimore, Maryland. http://anthology.aclweb.org//
  • Savova, Guergana. 2014. Temporal relation discovery from the clinical narrative. Invited talk at the National Library of Medicine Informatics Series. June 4, 2014. Bethesda, MD.
  • Finan, Sean; de Groen, Piet; Savova, Guergana. 2014. Narrative Event and Temporal Relation Visualization Tool. Workshop: Exploring Temporal Patterns in Electronic Health Record Data. 31 Annual Human-Computer Interaction Lab Symposium. May 29 2014. University of Maryland. http://www.cs.umd.edu/hcil/eventflow/workshop2014/
  • Bethard, Steven; Ogren, Philip; Becker, Lee. 2014. ClearTK 2.0: Design Patterns for Machine Learning in UIMA. In: Proceedings of the Ninth International Conference on Language Resources and Evaluation (LREC'14). http://anthology.aclweb.org//
  • Styler, William; Bethard, Steven; Finan, Sean; Palmer, Martha; Pradhan, Sameer; de Groen, Piet; Erickson, Brad; Miller, Timothy; Chen, Lin; Savova, Guergana K.; Pustejovsky, James. 2014. Temporal annotations in the clinical domain. Transactions of the Association for Computational Linguistics. http://www.transacl.org/wp-content/uploads/2014/04/47.pdf
  • Savova, Guergana; Pradhan, Sameer; Palmer, Martha; Styler, Will; Chapman, Wendy; Elhadad, Noemie. (in press). Annotating the clinical text – MiPACQ, ShARe, SHARPn and THYME corpora. In Handbook of Linguistic Annotations. Ed. James Pustejovsky and Nancy Ide. Springer.
  • Miller, Tim. 2014. Discovering narrative containers in clinical text. i2b2 All Hands meeting, Jan 17, 2014. Boston, MA (presentation)

2013

  • Albright, Daniel; Lanfranchi, Arrick; Fredriksen, Anwen; Styler, William; Warner, Collin; Hwang, Jena; Choi, Jinho; Dligach, Dmitriy; Nielsen, Rodney; Martin, James; Ward, Wayne; Palmer, Martha; Savova, Guergana. 2013. Towards syntactic and semantic annotations of the clinical narrative. Journal of the American Medical Informatics Association. 2013;0:1–9. doi:10.1136/amiajnl-2012-001317; http://jamia.bmj.com/cgi/rapidpdf/amiajnl-2012-001317?ijkey=z3pXhpyBzC7S1wC&keytype=ref. PMID: 23355458
  • Miller, Timothy; Dligach, Dmitriy; Bethard, Steven; Pradhan, Sameer; Lin, Chen; Savova, Guergana. 2013. Discovering time expressions in clinical text. Late breaking abstract. American Medical Informatics Association Conference. November, 2014. Washington, DC.
  • Chen, Wei-Te and Styler, Will. 2013. Anafora: A Web-based General Purpose Annotation Tool. Proceeding of the North American Association for Computational Linguistics Conference. Atlanta, GA, June 9-13. http://www.aclweb.org/anthology/N13-3004. Anafora is available open source from https://github.com/weitechen/anafora
  • Miller, Timothy; Bethard, Steven; Dligach, Dmitriy; Pradhan, Sameer; Lin, Chen; and Savova, Guergana. 2013. Discovering narrative containers in clinical text. BioNLP workshop at the Association for Computational Linguistics. http://aclweb.org/anthology/W/W13/W13-1903.pdf
  • Bethard, Steven. 2013. A Synchronous Context Free Grammar for Time Normalization. In: Proceedings of the 2013 Conference on Empirical Methods in Natural Language Processing. http://www.aclweb.org/anthology/D13-1078
  • Bethard, Steven. 2013. ClearTK-TimeML: A minimalist approach to TempEval 2013. In: Second Joint Conference on Lexical and Computational Semantics (*SEM), Volume 2: Proceedings of the Seventh International Workshop on Semantic Evaluation (SemEval 2013). Atlanta, Georgia, USA: Association for Computational Linguistics, pp. 10-14. http://www.aclweb.org/anthology/S13-2002
  • Sameer Pradhan, Alessandro Moschitti, Nianwen Xue, Hwee Tou Ng, Anders Bjorkelund, Olga Uryupina, Yuchen Zhang and Zhi Zhong. 2013. Towards Robust Linguistic Analysis Using OntoNotes. Proceedings of the Conference on Natural Language Learning. Sofia, Bulgaria. August, 2013.
  • Dligach, Dmitriy; Bethard, Steven; Becker, Lee; Miller, Timothy; Savova, Guergana. 2013. Discovering body site and severity modifiers in clinical texts. Journal of the American Medical Informatics Association. http://jamia.bmj.com/content/early/2013/10/03/amiajnl-2013-001766.full
  • Dmitriy Dligach, Timothy A. Miller, Guergana K. Savova. 2013. Active Learning for Phenotyping Tasks. In Proceedings of the 2013 NLP for Medicine and Biology workshop held in conjunction with RANLP-2013. September 2013. Hissar, Bulgaria. http://aclweb.org/anthology//W/W13/W13-5101.pdf
  • Finan, Sean. 2013. Challenges of visually representing rich temporal information of the clinical narrative. Workshop: Exploring Temporal Patterns in Electronic Health Record Data. 30th Annual Human-Computer Interaction Lab Symposium. May 22-23 2013. University of Maryland. http://www.cs.umd.edu/hcil/eventflow/workshop2013/
  • American Medical Informatics Association (AMIA) national webinar. “Towards semantic annotations of the clinical narrative”. National webinar. April 2013 (invited presentation)
  • Natural Language Processing Working Group Pre-Symposium – doctoral consortium and a data workshop. “Shared Annotated Resources for the Clinical Domain”. American Medical Informatics Association. Washington, DC, USA. November 2013.
  • Savova, Guergana; Chapman, Wendy; Elhadad, Noemie; Palmer, Martha. 2013. Shared resources, shared code and shared activities in clinical natural language processing. AMIA Annual Symposium, Panel. Washington, DC.
  • AMIA Fall symposium workshop on Natural Language Processing and data. Dr. Savova presented THYME work as part of the data workshop.

2012

  • Savova, Guergana. 2012. Shared Annotated Resources for the Clinical Domain. Natural Language Processing (NLP) Annotation workshop collocated with the 2nd annual IEEE International Conference on Healthcare Informatics, Imaging and Systems Biology. San Diego, CA, USA. September 2012.
  • Drs. Pustejovsky, Palmer and Savova are members of the Program Committee of the 2012 i2b2 shared task whose topic is temporal relations in the clinical domain. The THYME annotation guidelines are the basis of the annotation guidelines for that shared task.
  • Participation in the State of the Art of Clinical NLP workshop organized by the NLM in April, 2012. Dr. Savova chaired a session, Prof. Pustejovsky was an invited speaker presenting on Temporal relations/TimeML.
  • Participation and presentation in the AMIA Fall symposium workshop on Natural Language Processing and data. Dr. Savova presented THYME work as part of the data workshop.

2011

  • Savova, Guergana; Chapman, Wendy; Elhadad, Noemie; Palmer, Martha. 2011. Shared annotated resources for the clinical domain. AMIA Annual Symposium, Panel. Washington, DC.


Shared NLP Tasks with THYME participation

Getting access to the THYME corpus and gold standard annotations

The THYME corpus with the gold standard annotations is available to others involved in NLP research under a data use agreement (DUA) with Mayo Clinic. Requests for a copy of the THYME corpus should be made to a Mayo NLP investigator (e.g., Dr. Piet de Groen) in order to complete the DUA. After the DUA has been completed, the THYME corpus is available via a secure download mechanism. Distribution of the corpus is supported by grants GM102282 and LM010090 from the NIH; include the funding acknowledgment in your publications.

The corpus is released to an established or junior NLP investigator, formally associated with an institution; thus it is not released to a student. However, all students working with the investigator can have full access to the corpus under the DUA of the investigator. The investigator is urged to have the students work on the corpus on workstations that stay within the laboratory space of the investigator.

The steps for obtaining a DUA are:

Step 1: Send an email to ClinicalNLP@mayo.edu indicating your interest in the THYME corpus. You will receive a form to fill out. Once that form is returned to ClinicalNLP@mayo.edu, your request will be passed on to the Mayo Clinic’s legal contracts group.

Step 2: Mayo Legal Contracts Office will send you a partially filled-out DUA for you to add information to, and for you to have signed by your site's official signatory.

Step 3: After you return the DUA to Mayo Clinic, Mayo Legal Contracts Office will complete the DUA and send you a copy of the fully executed DUA, and ClinicalNLP@mayo.edu will be copied on the email.

Step 4: A Mayo Clinic NLP investigator will arrange to talk with you.

Step 5: Once the DUA is complete and you have talked with a Mayo Clinic NLP investigator, ClinicalNLP@mayo.edu will send you instructions for obtaining the corpus via the secure downloading mechanism

When using the corpus, please cite:

Styler, William; Bethard, Steven; Finan, Sean; Palmer, Martha; Pradhan, Sameer; de Groen, Piet; Erickson, Brad; Miller, Timothy; Chen, Lin; Savova, Guergana K.; Pustejovsky, James. 2014. Temporal annotations in the clinical domain. Transactions of the Association for Computational Linguistics. http://www.transacl.org/wp-content/uploads/2014/04/47.pdf


THYME Gold Standard Annotations

Annotation layers are treebank and propbank annotations as well as temporal annotations for events, temporal expressions and temporal relations.


Annotation guidelines

  • i2b2 Simplified THYME Guidelines (PDF) The guidelines provided to the organizers of the 2012 Temporal relations i2b2 challenge for consideration during planning. They reflect an earlier stage of our guidelines.

Tool for viewing the gold standard annotations - Anafora

We developed a web-based annotation tool. It is open source and available at https://github.com/weitechen/anafora. Use it to view the THYME annotations. Citation for the tool is:

Chen, Wei-Te and Styler, Will. 2013. Anafora: A Web-based General Purpose Annotation Tool. Proceeding of the North American Association for Computational Linguistics Conference. Atlanta, GA, June 9-13. http://www.aclweb.org/anthology/N13-3004.

Viewing the gold standard annotations (Anafora)

(available to the team only)

To view the Temporal-Entity data, use the URL:

https://verbs.colorado.edu/anafora/annotate/Temporal/ColonCancer/TASK_NAME/Temporal.Entity/gold/

TASK_NAME is the filestem, for example, ID074_path_219b

to view Temporal-Relation data:

https://verbs.colorado.edu/anafora/annotate/Temporal/ColonCancer/TASK_NAME/Temporal.Relation/gold/

you could find the available Entity/Relation gold data on verbs by using:

  find /data/anafora/anaforaProjectFile/Temporal/ -name "*Temporal-Entity.gold.completed.xml"
  find /data/anafora/anaforaProjectFile/Temporal/ -name "*Temporal-Relation.gold.completed.xml"


Train/Development/Test splits

Colon Cancer Data

  • Train sets (Residue 0,1,2,3): [1, 2, 3, 8, 9, 10, 11, 16, 17, 18, 19, 24, 25, 26, 27, 32, 33, 34, 35, 40, 41, 42, 43, 48, 49, 50, 51, 56, 57, 58, 59, 64, 65, 66, 67, 72, 73, 74, 75, 80, 81, 82, 83, 88, 89, 90, 91, 96, 97, 98, 99, 104, 105, 106, 107, 112, 113, 114, 115, 120, 121, 122, 123, 128, 129, 130, 131, 136, 137, 138, 139, 144, 145, 146, 147, 152, 153, 154, 155, 160, 161, 162, 163, 168, 169, 170, 171, 176, 177, 178, 179, 184, 185, 186, 187, 192, 193, 194, 195, 200, 201, 202, 203, 208, 209, 210, 211, 216, 217]
  • Development sets (Residue 4,5): [4, 5, 12, 13, 20, 21, 28, 29, 36, 37, 44, 45, 52, 53, 60, 61, 68, 69, 76, 77, 84, 85, 92, 93, 100, 101, 108, 109, 116, 117, 124, 125, 132, 133, 140, 141, 148, 149, 156, 157, 164, 165, 172, 173, 180, 181, 188, 189, 196, 197, 204, 205, 212, 213]
  • Test sets (Residue 6,7): [6, 7, 14, 15, 22, 23, 30, 31, 38, 39, 46, 47, 54, 55, 62, 63, 70, 71, 78, 79, 86, 87, 94, 95, 102, 103, 110, 111, 118, 119, 126, 127, 134, 135, 142, 143, 150, 151, 158, 159, 166, 167, 174, 175, 182, 183, 190, 191, 198, 199, 206, 207, 214, 215]

Brain Cancer Data

  • Train sets: [1, 2, 3, 8, 9, 10, 11, 16, 17, 18, 19, 24, 25, 26, 27, 32, 33, 34, 35, 40, 41, 42, 43, 48, 49, 50, 51, 56, 57, 58, 59, 64, 65, 66, 67, 72, 73, 74, 75, 80, 81, 82, 83, 88, 89, 90, 91, 96, 97, 98, 99, 104, 105, 106, 107, 112, 113, 114, 115, 120, 121, 122, 123, 128, 129, 130, 131, 136, 137, 138, 139, 144, 145, 146, 147, 152, 153, 154, 155, 160, 161, 162, 163, 168, 169, 170, 171, 176, 177, 178, 179, 184, 185, 186, 187, 192, 193, 194, 195, 200, 201]
  • Development sets: [4, 5, 12, 13, 20, 21, 28, 29, 36, 37, 44, 45, 52, 53, 60, 61, 68, 69, 76, 77, 84, 85, 92, 93, 100, 101, 108, 109, 116, 117, 124, 125, 132, 133, 140, 141, 148, 149, 156, 157, 164, 165, 172, 173, 180, 181, 188, 189, 196, 197]
  • Test sets: [6, 7, 14, 15, 22, 23, 30, 31, 38, 39, 46, 47, 54, 55, 62, 63, 70, 71, 78, 79, 86, 87, 94, 95, 102, 103, 110, 111, 118, 119, 126, 127, 134, 135, 142, 143, 150, 151, 158, 159, 166, 167, 174, 175, 182, 183, 190, 191, 198, 199]

THYME Software

The THYME system is available as part of Apache cTAKES at http://ctakes.apache.org/

We are also developing a visualization tool (THYME viz tool) which will be made available in cTAKES. A prototype and details of the THYME vizualization tool was presented by Sean Finan at several annual workshops.

Finan, Sean. 2013. Challenges of visually representing rich temporal information of the clinical narrative. Workshop: Exploring Temporal Patterns in Electronic Health Record Data. 30th Annual Human-Computer Interaction Lab Symposium. May 22-23 2013. University of Maryland. http://www.cs.umd.edu/hcil/eventflow/workshop2013/

Finan, Sean. De Groen, Piet. Savova, Guergana. 2014. Narrative Event and Temporal Relation Visualization Tool. Workshop: Exploring Temporal Patterns in Electronic Health Record Data. 31st Annual Human-Computer Interaction Lab Symposium. May 29 2014. University of Maryland. http://www.cs.umd.edu/hcil/eventflow/workshop2014/

Finan, Sean. De Groen, Piet. Savova, Guergana. 2014. Narrative Event and Temporal Relation Visualization Tool. Natural Language Processing Workshop. Informatics for Integrating Biology & the Bedside (I2B2). 4th Annual Academic User Group Meeting. July 9 2014. Harvard Medical School. https://www.i2b2.org/events/slides/NarrativeVisualizer.pdf

Relevant Background Papers

Relevant Papers


Internal Presentations

Presentations


Venues for manuscript submissions

Venues for manuscript submissions/publications


Project materials

Project Charter

Tasks, leads, teams and deadlines

Progress reports

Annotations - Describes the corpus, the layers of annotations and annotation progress

Annotation Tools - Describes the progress and information pertaining to the Anafora annotation tool


Communication


Meeting Notes

Getting started


Contact

If you need assistance and/or if you have questions about the project, feel free to send e-mail to steven.bethard at colorado dot edu OR Guergana.Savova at childrens dot harvard dot edu

Personal tools