Lauren Fitzharris: Good afternoon, everyone. Thank you for joining us on the March 26 ctsa program webinar
Lauren Fitzharris: for our agenda. Today we will have an update for ncats by Dr. Mike Carrilla, Ccas. Updates by Carrie James, and we will have a presentation by the leads for the learning about the science of translation
Lauren Fitzharris: case studies working group. And also we will have presenters from Emory presenting on at the clinical and translational Science Trainees, at the intersection of academic impact and public attention.
Lauren Fitzharris: so I will go ahead and turn it over to our 1st presenter, Dr. Carrilla
Michael Kurilla: Thank you, Lauren.
Michael Kurilla: Good afternoon, everyone. Okay, let's move through this quickly, because we have much more interesting things to get to. So the big news we have here at Nih is last evening the Senate confirmed the Nih director, Jay Bhattacharya.
Michael Kurilla: And an interesting candidate relative to past Nih directors. He has both a master's and a Phd. In economics. He is currently, but will probably be taking a leave of absence. A professor of medicine at Stanford, with courtesy professorships in both economics and health
Michael Kurilla: research and policy. And, interestingly, his research focus has been on population, health, and well-being.
Michael Kurilla: with a particular attention of the impact of government programs, biomedical innovation and economic factors that contribute to overall
Michael Kurilla: health and well-being. So we welcome him when he's actually going to be sworn in. If the time and date has been set, we haven't been notified of it. But we are moving forward here next slide, please.
Michael Kurilla: Just to give people. And just an update on the ncats budget since we've gone through now.
Michael Kurilla: with the we're on a year long. Cr, so continuing resolution. So that means our budget for 2025 under the Cr is
Michael Kurilla: is the same as it was last year. You can see the total for for ncats overall. And the Ctsa program makes up 68% of that figure. It's just shy of 630 million dollars. Next slide, please.
Michael Kurilla: Just to give you a few some updates. Unfortunately, because of administration priorities, we have, we have had to sunset the deia, the diversity, equity, inclusion, and accessibility. Enterprise Committee.
Michael Kurilla: We have spoken with the leads of the Ec. And we want to thank all the Ec members for their service on this committee. I think they have done a good amount of work that has really informed.
Michael Kurilla: been quite informative and provided guidance moving forward. And so we've spoken with with the lead team and have have shared shared this with the committee membership and just making the rest of the consortium aware of what's going on
Michael Kurilla: next slide, please.
Michael Kurilla: And unfortunately, we have some updates on the Ctsa Spring group meetings. We know that there's been a lot of hard work, particularly the Enterprise Committees into planning their meetings, and we're very grateful for your commitment overall to the Ctsa program. Unfortunately, with everything going on at Nih, we have just been unable to receive approval.
Michael Kurilla: For for our participation in this meeting that includes both being able to. sponsor the group meetings at the acts and as well as provide
Michael Kurilla: because even if it's local travel registration fees make it travel. And our travel is currently
Michael Kurilla: under a pause. So we've been. We've had to cancel the spring Ctsa group meetings, which means no virtual option, for these can be available. The meeting itself translational science 2025, hosted by acts, is still scheduled as as planned, so that there's no impact on that particular meeting, and acts want to make everyone aware, has graciously agreed to provide the opportunity for the Ctsa upis
Michael Kurilla: which we were in the in this planning stage for usually meet for an hour hour and a half but this this the acts has made the the opportunity in terms of meeting space the rooms made that available. So the the Ctsa upis can meet
Michael Kurilla: sans ncat staff and other groups, if they choose, may have the option as well again, there would be no virtual option. But that's going to be something that the group it's the groups themselves are gonna have to decide to do
Michael Kurilla: at. Because of this ncats and C cost we are. We are looking at the possibility of adding Ec meetings to the fall. 2025 ctsa. Program meeting agenda, which is still a
Michael Kurilla: gleam in the program committee. Eye. At this point there's no agenda to really speak of.
Michael Kurilla: and but it would provide an opportunity for the entire Ctsa consortium to to come together, and we'll keep you apprised, as as this evolves next slide, please.
Michael Kurilla: I wanted to make you aware that Nih has a new page. You're probably we're going to want to look at central resources for research grants and funding information. The website. The link is is there. And in particular, the announcements that the implementation of new initiatives and policies. If you click on that link, it'll take you to the next page, which is on the next slide.
Michael Kurilla: and the information contained there is is regarding Nih Grants and funding information status. So you know the grants and funding policies are evolving as Nih aligns with new agency priorities.
Michael Kurilla: This link listed there will keep you up to date.
Michael Kurilla: the most, the most current information. But you're gonna need to check back regularly, because
Michael Kurilla: in some instances things are literally changing on an hour by hour basis. So on that page, you'll find you'll find details and information about communications, applications, some funding opportunities, some Nofos have been closed.
Michael Kurilla: others have been updated to reflect the agency priorities that we are coming to understand, and, as always, please carefully read the funding opportunities for any changes that may have come
Michael Kurilla: when you actually sit down to do this next slide, please.
Michael Kurilla: and so upcoming change one of the upcoming changes, the common forms for biographical sketch and current and pending other support. Nih is adopting a common form for this in in 2025. But this announcement is that that has been postponed. It was originally scheduled
Michael Kurilla: for to to go to to transition in in late in May. But that has been postponed. So again, there's more information with with the link. So just continue to use the existing biographical sketch, and we will make you aware of of any changes to this
Michael Kurilla: that come down in the near future. Next slide, please.
Michael Kurilla: Wanted to highlight, I think at the last program, Webinar. There was a discussion of smart irb version 3 point. Oh, agreement! The reliance system and the website have now gone. Live
Michael Kurilla: the smart Irb agreement must be signed for any new reliance, agreement and indemnification addendum is optional. So you'll need to have internal discussions, probably with general counsel
Michael Kurilla: regarding that that option you will find on the new, smart Irb website. You'll find the new agreement and accompanying resources. There's a new look to the participating institutions, Page, and there are updates to reflect the 3.0 agreement in place. And there's
Michael Kurilla: the the reliance system has been completely updated and replaces the previous jointer and online reliances systems. And it must be used to execute the 3.0 agreements and the next slide
Michael Kurilla: you'll see some additional information
Michael Kurilla: with smart Irb. So we we gone through the 1st week 78 institutions across 33 States have already joined. Interestingly enough, Nih was the 1st signatory of of this. People may recall that Nih was not originally a signatory to version 1.0
Michael Kurilla: version 2.0 was was, actually the result of of contact by the clinical center at Nih, asking me how they could, how they could sign on to this. And that's where 2.0 came. About. 3.0 is is bringing on board
Michael Kurilla: both both the Va. And Dod, as well as some department of energy sites as well. We have 27 ctsa. Hubs already signed on board 50 universities, 18 hospital systems, 5 commercial irbs. And there's 30 institutions that are awaiting signature. And as of today, we're now up to 109, so it's it's percolating very quickly through the system. Next slide, please.
Michael Kurilla: and so upcoming dates. Our next program, webinar, will be April 23, rd and that's all for me, and I'll turn it back over to Lauren. Thank you.
Lauren Fitzharris: Great. Thank you, Dr. Carrilla. We're going to move on to Ccass updates, and I will pass along to Carrie
Kerry James: Thanks, Lauren. Hi, everybody! Go ahead and go to the next slide, please.
Kerry James: Just a notice to folks that the proposal submission period for the working groups closes on March 31st at 3 pm. Eastern time. So if you are planning to submit, please do so before that time and date
Kerry James: next slide.
Kerry James: and we can skip to the next slide. This is a repeat of what Dr. Carrilla announced a little bit ago.
Kerry James: So one update we have for folks, since we have had some inquiries about informal.
Kerry James: you know, networking opportunities at acts for the Ctsa members who will be there, and Ccos has confirmed with acts that the meeting space that was set aside for the Ec. Meetings is still available, and they are generously keeping it open for folks if they would like to use that space acts requests that any groups who want to use their space
Kerry James: to reach out to Kim Stelmazic at that email there, and to arrange the informal meeting by this Friday.
Kerry James: She gave a little wiggle room that if you can't get it in by Friday that they'll they'll sort of accept requests into early next week, but not after April first, st and so, after those requests are received, Acts and Ccos will be in close communication, and we'll communicate out through all the Ccos channels or the Ctsa channels to make sure folks know what's up.
Kerry James: And again, I just want to remind folks these are informal meetings, and they're not sponsored by Ccos or ncats. Of course we have only been able to commit to or get a commitment, of course, just to meeting space, and there will be no additional onsite support provided through Ccos for those meetings just to be really clear next slide.
Kerry James: So currently, these are the meetings that are scheduled for in this informal networking there is. There are 2 meetings on April 14, th the Triad meeting the workforce development. Kpis and tpis are, I believe, meeting at 8 to 11 Eastern time. It was the original time they had had set aside for their meeting.
Kerry James: and then bids the bids. Ec. Is going to meet at 2, 15 to 4, 45, and the Upi meeting is on April 17.th From one to 3 acts has extended the amount of time that was originally set aside for Upis at the request of several Upis. So please reach out if you have questions, and I do want to say. You know, that acts has been incredible.
Kerry James: supportive. And I want to, you know, just vocalize that they a thank you, and and that they've been really generous with their support for for the Ctsa groups
Kerry James: next slide.
Kerry James: we do have 2 new features on the website that are released. And so we're excited to share that. There is a Member Directory that you can search on the website. The link is in the chat and you will be able to search by hub, by institution, by email addresses and by name, but also by group. You can filter by group, by Ctsa program group.
Kerry James: and then also on the website now is a search feature. So this search feature covers most of the content. There's some content that's not included, such as the Member Directory or the Hub Directory. Those have their own search feature, but this is an Mvp. And so let us know. You know, if you're you know, if you have any challenges with it. We're really excited that that's now on the homepage of the seacost website
Kerry James: next slide.
Kerry James: and our team science at University of Rochester is still accepting submissions for proposals by March 31, st for any hubs that may be interested in co-hosting a collaborative workshop. So please do submit your proposal by the 31st
Kerry James: next slide
Kerry James: and to sign up to get access to the web, the Member Directory. You are going to need a Ccos account. So please do go ahead and sign up for a Ccos account. There's links in the chat to do so
Kerry James: next slide, and as noted before, the next webinar is April 23.rd Thanks, Lauren
Lauren Fitzharris: Great thanks so much, Carrie, and thank you, Dr. Carrilla, for both of your updates. We're excited to move forward with our next presenters, who are presenting the translational Science Case Studies, Ctsa working Group, Dr. Palfrey and Dr. Diaz Granadas. I will turn it over to you
Clara Pelfrey: Thank you. Can everyone hear me?
Clara Pelfrey: Okay, good afternoon. My name is Clara Palfrey, and my co-chair is Debbie, and I think there's a slide. Right after this we'll show you it. But we wanted to specify that even though we're called the Translational Science Case Studies Group for short, we do have this long, full title learning about the science of translation, identifying best practices
Clara Pelfrey: to accelerate the translational process through case studies next slide.
Clara Pelfrey: So my co-chair is Deborah Diaz tomatoes. She's from Virginia Commonwealth University.
Clara Pelfrey: and I'm from Case Western Reserve University.
Clara Pelfrey: And next slide, please.
Clara Pelfrey: So just a bit of history on the Case Studies group. We actually started this group in 2017,
Clara Pelfrey: and we were developing a process for conducting case studies and realized there really wasn't a mechanism out there to publish that type of case study. A case study typically is a medical case study. And this was different. So we did develop a protocol paper, as you can see here.
Clara Pelfrey: We then went to the journal of Clinical and Translational science, and asked them if they would be. Consider having a new manuscript category to accommodate this particular type of evaluation research, and if they were willing, and that new manuscript category was called the Translational Science Case study.
Clara Pelfrey: We then went on to publish several individual translational science case studies in the journal.
Clara Pelfrey: and the 1st case was published in 2021.
Clara Pelfrey: So our big question for this working group, then, was, how can we accelerate the translational process? And we thought, if we compared these case, studies a whole bunch of them
Clara Pelfrey: and looked at successful translation
Clara Pelfrey: to learn really what was critical in terms of facilitating that research, enabling it to go forward as well as the challenges and roadblocks and hurdles, and because they've been successful, we knew they'd been able to overcome those hurdles, and we wanted to learn from that
Clara Pelfrey: next slide, please.
Clara Pelfrey: So the goals of our working group were to conduct comparative case studies. So really, it's a cross case analysis of the Translational Science case studies.
Clara Pelfrey: We want to publish at least one paper on the findings of this cross-case analysis.
Clara Pelfrey: And we're hoping to generate new knowledge about the process, about comparing successful case studies and what worked and how they got around it. We also want to produce generalizable knowledge about the process and those facilitators and roadblocks as well as human health impact.
Clara Pelfrey: And finally, we want to develop a best practices, guideline for overcoming those roadblocks and speeding up the process of translational science
Clara Pelfrey: next slide, please.
Clara Pelfrey: So the anticipated impact from our working group was that dissemination would enable future infrastructure, development.
Clara Pelfrey: education, and professional development that would address the potential roadblocks that we discover.
Clara Pelfrey: We want to better understand facilitators and challenges that will allow all the various Ctsa hubs to improve the forward advancement of translational science, and to shorten the time.
Clara Pelfrey: We also want to understand those facilitators and challenges
Clara Pelfrey: allowing the Cts Hub to develop and disseminate common solutions for everyone.
Clara Pelfrey: We want to promote continuous quality improvement by identifying barriers
Clara Pelfrey: and those barriers can be addressed with a multitude of solutions, education, training partnerships, collaborations, funding, and new Grant mechanisms to name just a few that we already know.
Clara Pelfrey: We'd like to demonstrate the impact of the translational science on human health. So we really are hoping to get to impact. And what role the Ctsa consortium has had in that.
Clara Pelfrey: and finally, to provide important information that could benefit all
Clara Pelfrey: kinds of stakeholders, including researchers, teams, institutions, hubs, funders, Congress as well as the public next slide.
Clara Pelfrey: So we started this working group, as I told you in 2017. By the time, not the working group, we started our case studies group. Then, by the time the working group started we had amassed almost everyone on the left side of this list all the way down to Stanford University was already working on case studies.
Clara Pelfrey: and then everyone else was added. Once we started the working group. And so we have a total of 30 unique Ctsa hubs
Clara Pelfrey: here, and 50 members or contributors. Contributors are people who have a successful case study and by successful meaning it's an intervention that's been shown to be successful.
Clara Pelfrey: but they don't necessarily want to be part of the whole process of coding and writing the papers, or don't have the time to do that. So that's what the contributor is. Members are fully functioning members here that help do all the right
Clara Pelfrey: next slide, please.
Clara Pelfrey: So something we discovered in the process of amassing these case studies is that we actually ended up with 2 different kinds of case studies, those on the left that improve health
Clara Pelfrey: and those on the right that improve research. So they're really translational science
Clara Pelfrey: on the right and on the left. We've also found some of them have been demonstrated to work locally at perhaps one health system, and some of them have been implemented beyond research and at other health systems.
Clara Pelfrey: And then, similarly, the improving research one. Many of those have been shown to work at a single Ctsa hub.
Clara Pelfrey: and they're in the process of implementing at other sites. So we have a really interesting variety of case studies to look at
Clara Pelfrey: next slide.
Clara Pelfrey: I'm going to hand it over to Debbie and let Debbie finish the rest of it.
Deborah DiazGranados: All right. Thanks, Clara, for setting us up. Good afternoon, everyone. So we want to share the accomplishments that our working group has made since we started as an official working group, and we believe that we've made significant strides in several areas. And here I'm going to share with you about how we've recruited and identified other collaborating groups as well as started to develop some of the methodology for that cross case analysis.
Deborah DiazGranados: So, as Clara mentioned, to expand our team and bring in fresh perspectives. We focused on targeted outreach as part of the workgroup application requests. We shared the opportunity through our Ctsa hubs. Social media pages. We also targeted evaluators via the evaluators Listserv and
Deborah DiazGranados: we had assistance from seacost because they helped amplify our reach by sharing our social media posts on Linkedin and other platforms. So, as Clara mentioned, we successfully have recruited 50 additional
Deborah DiazGranados: members to the work group that represent 30 unique Ctsas. So this is really strengthen, strengthening our compositional variation and expertise within our group and perspectives. We've also have recognized the importance of leveraging existing case study efforts. So we have identified and worked with other groups that have examined translational science.
Deborah DiazGranados: And this includes the Nih office of science policy. Niehs Washu's translational science benefits, model case studies website, the group that originally developed and put forth the framework that a lot of Ctsas use for their evaluation, as well as Duke University, who has highlighted case studies within their Ctsa website.
Deborah DiazGranados: So these partnerships have really contributed to our collection of case studies that will then ultimately be included in that cross case analysis.
Deborah DiazGranados: So in terms of our methodology, we've we've started by developing 2 committees, one of them being a coding committee. So to ensure a systematic approach to the qualitative analysis. We've established a dedicated coding committee with 9 members of our working group.
Deborah DiazGranados: and this group is focused on determining the best methods for coding and ensuring consistency by using what we've created as a case. Study template, that we're gathering information from all case studies in a very standard
Deborah DiazGranados: format. And this committee meets monthly. Aside from our overall working group, our big working group.
Deborah DiazGranados: and they provide regular progress updates to the broader team. So this structured approach of developing committees to be focused on specific methods has helped us to promote the rigor in our cross case, analysis and alignment within our methodology. We can go to the next slide, please.
Deborah DiazGranados: So in addition to the Coding committee, we formed a 6 Member interview committee. So
Deborah DiazGranados: we had a collection of case studies, some of which didn't have enough information to be included within our Cross case analysis. So this interview committee has been given the the responsibility to identifying case studies that require additional interviews.
Deborah DiazGranados: and they have identified 12 that need further interviews, 7 of which have been completed. And it's provided us with again, just more case studies to add to our collection and to be thorough in terms of the case studies that we're we're adding
Deborah DiazGranados: additionally in terms of our progress and accomplishments. Ncats help us set up a collaborative workspace, which is a very big benefit of being an official work group, and that's been very helpful in order to communicate and keep all our documents
Deborah DiazGranados: posted, and and everyone can keep track, and of our progress.
Deborah DiazGranados: And then, lastly, we what we did, we established a 2 participation pathway.
Deborah DiazGranados: So we understood that individuals that were that were interested in the case study approach and were interested in participating may have different time. You know time available to them for for committing.
Deborah DiazGranados: So we developed 2 pathways, one that is, that we call the full participation member, and they are included in our smart irb. They are the ones really adding and doing, conducting the case, study and completing their case study. And they're really integral in will be integral in the Cross case analysis and in the publication and dissemination.
Deborah DiazGranados: The other pathway is a case study contributor, understanding that folks
Deborah DiazGranados: have case studies that they have, but they'd like to contribute, but aren't interested in conducting additional interviews, identifying barriers and facilitators, and may not want to be included in the coding and the manuscript writing. So we were able to create that those 2 different pathways. And it's helped and really worked with our working group members.
Deborah DiazGranados: The next slide.
Deborah DiazGranados: So in our work group application. We outlined 7 key deliverables. And we want to update everyone on these deliverables. So for our 1st deliverable, we aimed to include, we really wanted to have one case study for each Ctsa hub represented.
Deborah DiazGranados: And so far we have 44 case studies that have been completed or are in progress, and they do represent the 30 hubs that we have in our working group.
Deborah DiazGranados: our second deliverable. We sought case studies again in those organizations, those groups that we we identified, we sought to collect additional case studies from those. And as I previously referenced, we've identified those sources, and we have collected case studies from those sources and have put them into our case. Study template
Deborah DiazGranados: next slide, please.
Deborah DiazGranados: and the another deliverable. Our 3, rd deliverable is focuses on case studies contributed by individual Ctsas. And so some of these case studies again required additional details. We've
Deborah DiazGranados: conducted those additional interviews and and are continue to to
Deborah DiazGranados: finish off the the interviews that are needed to be done for those. So those are, those are progressing well, our 4th deliverable centers on the Cross case analysis. So, as I mentioned, a subset of working groups are leading this effort in terms of establishing an analysis process and conducting pilot analysis on completed case studies.
Deborah DiazGranados: While the final analysis depends on all case. Studies being completed, we have made significant progress by defining a coding strategy and developing an initial codebook for use during coding
Deborah DiazGranados: next slide, which I think is the last slide. So we these are our final 3 deliverables as a working group. They are very dependent on the completion of our cross case analysis. So we have not yet begun.
Deborah DiazGranados: These are and not and cannot necessarily report on the progress right now, but these include, and developing a model or best practices for overcoming translational science. Roadblocks, as well as our dissemination efforts to that will culminate in a publication and presentation to our community.
Deborah DiazGranados: and I believe that was the last slide.
Deborah DiazGranados: So thank you all, we're happy, happy to take any questions that you may have
Lauren Fitzharris: Great. Thank you both so much for this great presentation and for the update on your progress with the working group. We there are 2 questions in the chat. We are right at time to move on to our next presentation. But,
Lauren Fitzharris: There is a question about a link to the existing studies was that in this presentation, or is that available
Clara Pelfrey: The you mean the case studies that we've collected so far
Lauren Fitzharris: Believe. So, yes, yeah.
Clara Pelfrey: No, there is not a general link that we haven't started coding them yet. So right now they're not available widely available. But one of our hopeful outcomes would be that we have a repository for these, that everyone can access
Lauren Fitzharris: Thank you.
Deborah DiazGranados: And I would add that some of the ones that could be accessed are some that have already been published. So if folks are interested in identifying any case studies that have been published. I think most of them have been published in Jcts, so I'd encourage folks to to look through that
Lauren Fitzharris: Great.
Lauren Fitzharris: There's 1 more question, but I think we're going to move on to the next presentation. But Clara or Debbie, if you can type, an answer into the Q. And a. That would be great, and also keep an eye on the Q. And a box in case some additional questions come in in the next couple minutes and thank you again. We're going to move to our last presentation today, and I'm happy to
Lauren Fitzharris: hand it over to Dr. Now.
Eric Nehl: Sure. And can you let me share my screen?
Lauren Fitzharris: Yes, one moment.
Eric Nehl: All right.
Eric Nehl: thank you. And today we're going to talk about publications. So I know about half people drop off immediately when I say that. But we get pretty excited about publications around here
Eric Nehl: and their influence that they have. So we're going to talk about influential scholarship
Eric Nehl: and clinical and translational science trainees
Eric Nehl: and their intersection of both academic impact and public attention. And not only that, we wanted to add on a little bit more about policy, too, because this is a large group that is interested, not just in public attention, but policy attention. So we've added some slides
Eric Nehl: on that topic as well. My name is Eric Neal and my partner in crime here at Emory, at Georgia, Ctsa. Is Nikki Llewellyn, who is on the screen, and we did not plan this, but our other partners are Clara Pelprey and Debbie d'es Granados. The only person not on the screen is G. Dave from Unc. And all of us have been doing this work
Eric Nehl: since about 2018, and what happened was we? We were asked, as all evaluators are, to to examine the impact of the the researches coming from our hubs.
Eric Nehl: And so Nikki and I started looking at the publications that were coming out of our hub and using our Grant numbers to look those up as anyone can do with your Grant numbers. And that's the key. So we decided that
Eric Nehl: if we're using our just our Grant numbers. Why don't we collect all the Grant numbers from all the Ctsa hubs?
Eric Nehl: And so we did that Nikki did that, and then we were able to download all the publications that acknowledge Ctsa support.
Eric Nehl: Not just that year, but from the beginning of the program. And so we were start began. We began to start to ask questions about
Eric Nehl: how the support of the Ctsa hubs can lead to these translational outputs and outcomes. And we started with academic citations because that was the data that were readily available from pubmed. And so that was sort of limited. We looked at content areas, we were able to break it down by different programs within our Ctsa Hub or across Ctsa hubs.
Eric Nehl: We looked at impact factors and what was related to impact factors, but advances in the
Eric Nehl: informatics side. The the library side allowed us to then start to look at
Eric Nehl: things such as policy document references. And so we added that onto our menu of what we were looking at for publications. We can look at patent references now, and we can also look at news and media references.
Eric Nehl: And so we've
Eric Nehl: published a couple handful of articles looking at the Ctsa. As a whole, looking at the Georgia Ctsa. And there are other papers that can then be compared to others outside the Ctsa. So if you have the grant numbers, you can look at the publication impact from those programs.
Eric Nehl: So this is the main question.
Eric Nehl: we assume that publications have influence outside of academia. A lot of us have never even really been concerned about this. But it's becoming increasingly important to understand this.
Eric Nehl: and we really don't know. We really don't know. We write this paper. We send it to a journal, it gets published, it gets cited by other researchers, and that's been the end of it. That's what we need for promotion and tenure. But do we really know if people outside of Academia are using this. So we know there's the support of science publications. And then there's the question, mark what happens after that.
Eric Nehl: And so can we use these cutting edge bibliometric tools
Eric Nehl: to evaluate how the hubs that
Eric Nehl: are being supported by ncats have gone beyond academia. And and that's what we're going to show you today.
Eric Nehl: We have 2 case studies here, or just studies that we're going to present. I'm going to talk about the Kl. 2 scholars and Tl, one trainees and the publications that resulted from that program. Nikki's going to talk about publications that have been from all the Grant numbers across all the years that the Ctsa has been around.
Eric Nehl: We know that publications we have to acknowledge those. We hope that people acknowledge those we go into Nih reporter and pubmed.
Eric Nehl: And then, when we find that data set. We take it into a program called Dimensions Altmetra Explorer, and a newer one called Overton Databases. To get the data that we need to then do these analyses? We like to think it's big data. We're talking in hundreds of thousands and millions of citations from those and all the
Eric Nehl: data that we need to understand impact is in these papers. Scientists are actually pretty good writers. Once you look at what they are putting out there.
Eric Nehl: So first, st I'm going to pass it over to Nikki Llewellyn, and she's going to talk about policy
Nicole Llewellyn: Sure. Hi, everybody! I'm Nikki, and I wanted to start by talking about some of our recently published research, where we demonstrated for the 1st time the policy impact of the whole national Ctsa consortium using some of these latest bibliometric tools.
Nicole Llewellyn: So again, with this one, we collaborated with 2 of our evaluator friends at other Ctsa. Hubs, Clara and Debbie, who we just heard from to publish this paper in academic medicine in 2023
Nicole Llewellyn: for this one. For the 1st time we worked on documenting, drawing these documented lines from the Ctsa. Supported science to the publications that result from that science to the health policy influence of those publications to the next slide.
Nicole Llewellyn: Thanks. So in this project we started with the entire Ctsa. Supported portfolio, which at that time was about 120,000 publications that had cited the Ctsa K's or T's or U Grants, and there are a good bit more of those now that were in 2025. But this was as of 2021
Nicole Llewellyn: using Overton. We 1st queried those 120,000 papers and found that about 13% of those were referenced in policy literature all around the world. And that's what we've mapped here on the left. So even though this research is conducted in the United States, and we see that it is being used all over the world, albeit in greater numbers. In here, in the Us. And in English speaking countries.
Nicole Llewellyn: So we also know, looking at the list on the right, that these references came from a wide variety of source organizations, organizations that work in public health policy, including government organizations like the Cdc. They're all sort of categorized in these 4 bins. Private think tanks like the Rand Corporation intergovernmental organizations like the World Health Organization, and in that other category the main example is the guidelines of pubmed central, which are also included here.
Nicole Llewellyn: Taken together. All of this gives us an idea of the general footprint of policy influence which we can see has been significant over the previous 15 years of Ctsa operations at that time
Nicole Llewellyn: next slide.
Nicole Llewellyn: So another question beyond just that overall influence is that we wanted to look at the kinds of research that were most likely to be taken up in the policy literature. We know it's a small percentage globally. It's really only about 5% of papers that get cited in policy. And that proportion is much higher among Ctsa research. But it's still only 13% of Ctsa papers. So we use logistic regression to take a look at how some different bibliometric indicators might be related to policy uptake.
Nicole Llewellyn: and we found a few different factors that independently predicted the likelihood of a paper being referenced in policy literature.
Nicole Llewellyn: So the 1st one would be having scores that reflect content. That's more closely related to clinical research. So rather than basic science or animal research things that are further along that translational spectrum predicted policy uptake.
Nicole Llewellyn: Second, having higher academic citation scores. So be more influential in those traditional academic terms also predicted policy uptake.
Nicole Llewellyn: and last, having higher Altmetric scores which reflect that public and community attention which Eric's going to go speak more on that in just a little bit, but that also independently predicted uptake in the policy literature. So we have a little bit of a better idea of which kinds of papers, you know are most likely to be having that input and influence
Nicole Llewellyn: in the public arena.
Nicole Llewellyn: So next, just to describe a couple of specific illustrations of what we found. So you can kind of put this into real terms. And these were from our own Georgia Ctsa. Portfolio, looking kind of close to home here.
Nicole Llewellyn: This 1st one is an example of an older article on Flu Vaccines in Pregnant Women, that was carried out in 2011, published in 2011, I should say, and this one was supported by our pediatrics program at Rctsa, and their important findings ended up being referenced in dozens of times by organizations like the World Health organization as well as many different international governments.
Nicole Llewellyn: And an example of a policy document is this, Who evidence review that was listed here. It was published in 2014. So a few years after the paper, and it's by the Global Advisory Committee on Vaccine Safety, and the goal of it was to clarify uncertainty and provide recommendations for the use of vaccines during pregnancy with the implications for influencing policy and practice worldwide.
Nicole Llewellyn: Next slide.
Nicole Llewellyn: And then another more recent example, I hear, was a preliminary findings on the Moderna vaccine for Covid. If you guys can remember that thing that happened, this came. This was supported by our clinical research centers and our informatics programs at the Georgia Ctsa. And it came out during the pandemic in 2020, and was cited many times by organizations like the Cdc. And the Department of Homeland Security. Among others.
Nicole Llewellyn: there's an article here actually was a critical component of the Cdc's policies which examined the pros and cons of recommending that moderna vaccine for emergency authorization. Back in December of 2020 back, when things were really on lockdown and still trying to decide what would be done. As we know it was authorized at that critical time. And this research was part of that decision. So all in all using these bibliometric tools and techniques has allowed us to draw these direct documented lines
Nicole Llewellyn: from our hub support to public health implications of national and even international importance
Nicole Llewellyn: slide. One more slide, all right. And
Nicole Llewellyn: finally, I just wanted to share how we can also use bibliometric tools to see how this paper our own research is being used out in the world. It's kind of coming full circle here. So we know, for example, that Dr. Krilla has used these findings from this academic medicine paper to share how the Ctsa's influence on policy has been used in his own presentations and sharing
Nicole Llewellyn: how the Ctsa program as a whole is making an impact. We can look at tweets or ex posts on this paper. How often they've been viewed Reposted. We know that it was liked by Joni Rutteer, for example.
Nicole Llewellyn: And with that I'll pass this back to Eric so he can talk some more about how Ctsa research is being used in the news and media in this way
Eric Nehl: Thanks, Nikki. And and so the second part of our presentation today is to step away from the policy side and go more toward the public impact side.
Eric Nehl: And so one of our tools, through dimensions and alt Metro explorer allows us to look at things like news stories and tweets and blog posts and Wikipedia, and they've just added blue sky to the mix. And we can then, using this track.
Eric Nehl: how supported research goes to publications to to out to the general public.
Eric Nehl: And so this is focused just on our K's and T's.
Eric Nehl: And what you have on this slide is a figure that shows the growth of the publications each year from the K's and T programs.
Eric Nehl: And so since 2,006, we've had over 30,000, I think it's 32, 32, and a half 1,000 publications that have acknowledged Ctsa support over the course of the program. And you can see, during the course of Covid we had a big spike. And now it's going back to what we consider more of a normal level. So we were obviously very productive.
Eric Nehl: During those few years the Covid was really hot.
Eric Nehl: This next slide are are alt metrics that are linked to those
Eric Nehl: publications, the 1st at the top. Here this blue circle says, 18% have alt metric attention scores greater than 20. This is important. So studies like I usually do, have an alt metric attention score of 0, which means nobody outside of Academia knows anything about it.
Eric Nehl: A score of 20 is considered extremely high, so the work that the Ctsa trainees are doing and acknowledging gets a lot of public attention. And that's 1 of the takeaways we want to have you to have today is that public attention is on. I know this is not surprising to you, but public attention is on science now, and if one out of 5 articles is getting this public attention.
Eric Nehl: what we would traditionally consider a impact. Is this, 30,000 articles over a million academic citations have come from them.
Eric Nehl: 13,000 policy references have come from these articles.
Eric Nehl: 64,000 news articles have been written using these publications, and if you're sort of picking your job up off the floor, there's a lot of work to be done in this area, because it's it's really a wild West of trying to figure out what is causing something to picked up be picked up by the media or not.
Eric Nehl: 7,000 blog posts, almost half a million twitter ex posts. From these 30,000 supported publications, and 27,000 of these have received some sort of Altmetric attention. So it's a significant number of them. Most of them are getting attention
Eric Nehl: as similar to what Nikki showed. I'm showing how the Alt metric attention has laid out across the globe. You can see here the majority of it comes from the Us. But the impact of our programs is really worldwide.
Eric Nehl: News stories have been written in all different languages, in all kinds of different outlets. I put a few around the outside here. New York Times, Yahoo News, BBC. Associated Press.
Eric Nehl: Those news articles are there. We can go in and analyze those, and we need help to do this. So we we would love to go in and analyze those 60,000 something news articles to see the talk that's going on around our research.
Eric Nehl: So one question that we had was what really is the biggest academic influencer
Eric Nehl: papers. What are the ones that are just growing all the attention? What are the ones that are the influencers like you see on social media. And so what we did is we looked at those with the highest Altmetric scores, and we we made a cutoff at Altmetric score of 1,000,
Eric Nehl: which, if you consider a high score, is 21,000, is higher than 99.7% of the rest of the articles. So this is these are the articles that gained the most attention.
Eric Nehl: It won't surprise you that the top of the top was Covid-nineteen. 30% of the articles were related to covid-nineteen that had that high Altmetric score. But it wasn't just covid-nineteen, it was diet and exercise. It was drug overdose. It was genetics. You see the list here, and there's additional analyses that can be done to look how this varies over time. What was important in 2012. Is it important? Now.
Eric Nehl: what's going to be important next year to get this influence? Is it going to be glp, one drugs?
Eric Nehl: What is it going to be. And we can track this and try to figure out, how do we promote our research at our hubs to get that word out that we're having this good research at our programs.
Eric Nehl: A couple more, a couple examples. I pulled this one because if you see the Donut with the different colors, that's the Altmetric score for this article. This is A.
Eric Nehl: A. 2021 article evidence. Review of face masks, maskings, COVID-19. This was supported by the Dartmouth Hub. And A. T. Grant was acknowledged on this on this paper.
Eric Nehl: This has one of the highest Altmetric scores that's ever been recorded. It's a it's a story about face masks and covid-nineteen. This was picked up by 750 news stories, 20,000 times cited on social media. 1,500 men delayed downloads. And this was one of the top 5% articles that's ever been
Eric Nehl: tracked by Altmetric. So this is huge. Our work has been really picked up by the public. A second article is from our Hub. This is from an Mscr graduate and Kl. 2 scholar.
Eric Nehl: And again, it's on Covid-nineteen. If you're wondering if it's related to academic and public influence. Yes, this one has over 800 academic citations in just a few years an Rcr score of 24.6, which is interpreted like an odds ratio.
Eric Nehl: and that Altmetric score of 1593 has 56 news stories, 1,500 social media users, and 6 policy citations. So 6 policies have cited this article.
Eric Nehl: And so we also did multivariable analysis, and are looking at what types of publications gain public attention
Eric Nehl: factors that are independent predictors. The journal impact factor still does have some influence.
Eric Nehl: If it's published after 2020, this is showing that we're in the age of people really having an eye on science. So we need to begin training this within our training programs
Eric Nehl: policy citations. So those that are picked up in policies also are picked up by the general public.
Eric Nehl: those that are downloaded more so I think this really speaks to the importance of all the journals being online now, and you don't have to go to the
Eric Nehl: the library to make copies
Eric Nehl: higher Rcrs or relative citation ratios. So this is comparing an article versus its peers
Eric Nehl: and higher, apt or approximate potential, translate. So is it closer to public health and human research? Or is it closer to lab and those that are closer to public health and community research are more likely to get more public attention. So that sort of makes sense.
Eric Nehl: So in some
Eric Nehl: we have a lot of research going on in this area. We feel like it's meeting one of the Ctsa's goals of the 3 Ds
Eric Nehl: and we want to communicate sciences. We need to do a better job of doing that beyond academia. And we feel like bibliometrics are one way to track and evaluate our programs.
Eric Nehl: Nikki, do you have any other you want to share in our future directions on on this, before we ask answer questions
Nicole Llewellyn: Oh, yeah, I was just gonna say, there's there's so much more to unpack in this, you know, massive world of alt metrics. And you know, we're always excited to collaborate with people. I know that the effect on clinical guidelines is another area we're really interested in, like, how are our research, impacting policy, the news, the public, but also clinical practice as well. So
Nicole Llewellyn: among many things
Eric Nehl: Thank you. Yeah. And we're going to look at clinical guidelines. Clinical trials continue doing the policy lane and continuing down the Altmetric lane, while also looking at specific research areas. And so if you're interested in pediatrics or substance use or physical activity, we can use these same tools to do these analyses
Lauren Fitzharris: Wonderful. Thank you both for that great presentation. We do have time for questions, and there are are already a few in the Q&A box, and we can maybe start from the top. There was a question around hearing more about how you evaluate
Lauren Fitzharris: the metrics, and if you can share the citations referenced
Eric Nehl: Sure. Well, these are really straightforward to to download. You just have to have grant numbers. So it's as easy as entering the Grant numbers in the pubmed, and we're happy to share and and go about. Go about that route. But it's also pretty straightforward to gather this data on your own and replicate it. And that's the the key part of using the Grant numbers and and the site the publication numbers.
Eric Nehl: I see that Emma also asked about cancer centers across the Us.
Lauren Fitzharris: Great
Eric Nehl: That's exactly the kind of thing we can we can do with this. And and possibly, if you wanted to compare trainees at one Hub versus another, or one type of training program, nih versus another. You just have to have the Grant numbers and you can pull the citations
Lauren Fitzharris: Thank you. And I just wanted to add real quick that these slides will be posted on the C cost website. And it does include Eric and Nikki's emails? So that is one way that everyone can contact them. And the next question is, how can other hubs participate or collaborate, to contribute to these metrics and tools, to show you
Eric Nehl: Sure. Yeah. So we present with on the evaluators call each each month just a short update. And so you're welcome to contact us through your evaluator or contact us directly. We're happy to work with folks outside of Ctsa world, too. We've we've worked with other folks across our university and others to begin to show research impact at universities, which is pretty important these days.
Lauren Fitzharris: There's 1 additional question from David. Will these alt metric tools be made available, or the processes for individual sites to use
Eric Nehl: Sure, good, good question. So the basic level is available to most hubs so pubmed.
Eric Nehl: Our our hub, our university has dimensions that we subscribe to. Some of these are add-ons, but we're happy to help out, as we have time
Lauren Fitzharris: Well.
Lauren Fitzharris: it looks like there are no other questions in the Q. And a chat. There are a few kudos to both of you for or to both presentations. Today we are almost at time. I don't see any other questions coming in at the moment.
Lauren Fitzharris: but again, our next webinar will be April 23, rd
Lauren Fitzharris: I. And all of these meeting materials, the slides and the recording will be made available on the Ccos website on the Webinar page.
Lauren Fitzharris: If there are no other questions, I think that we can adjourn to today. But again, I would really like to thank all of the presenters, and I hope and thank everyone for attending, and I hope everyone has a great rest of your day.
Lauren Fitzharris: Take care, everyone