An interview with Kristefer Stojanovski, MPH, PhD(c), Public Health Researcher and Evaluation Specialist, Capacity Building Assistance Program, San Francisco Department of Public Health
by Anna Morgan, RN, BSN, MPH, NaRCAD Program Manager
Tags: Evaluation, HIV, PrEP
NaRCAD: Hi, Kristefer! Thanks for taking the time to chat with us today. Can you tell us a bit about your background and the work you’re currently doing as it relates to academic detailing?
Kristefer: Thanks for having me. I’m a public health researcher and an evaluation specialist with the Capacity Building Assistance Program at San Francisco Department of Public Health. I serve as a specialist and a technical assistance provider for the West region of the United States. My work is focused on data and evaluation of academic detailing programs that are working on topics like sexual health, HIV, and PrEP. My main goal is to help folks measure, or think about, what “success” may look like for an academic detailing program.
NaRCAD: What data do you think is most important for academic detailers to track during their visits when measuring or thinking about success?
Kristefer: I see evaluation data and detailing efforts as one complete package. Detailers should think about their data at a high level and focus on the information they’re collecting and how that information serves the overall goal of detailing, which is to improve knowledge, attitudes, intentions, and behaviors of providers and clinics.
At the same time, detailers should think about how they can show that they’re achieving that goal. For example, it’s useful to track how many providers they’ve seen, how much time was spent with providers, what they talked about during the visit, the resources that were provided, how the providers plan to use those resources, if a follow-up visit was scheduled, and the purpose of that follow-up visit. It’s important to track a mix of quantitative and qualitative data, but the critical components that should be tracked are the outcomes and the process of detailing.
NaRCAD: What about academic detailing programs? What data should they collect?
Kristefer: In a typical detailing program, detailers have a longitudinal, continuous outreach to providers. There’s an interesting conversation to be had about how we can use that temporal matrix as a tool and strategy for evaluation. I’m interested in how we can use the work academic detailers are doing – the actual visits themselves - as pieces of data over time.
For example, if detailers are collecting some of the rich conversation that they’re having with providers or clinics, it’s fascinating to track those conversations over time and see how the detailing program is changing the knowledge, attitudes, intentions, and behaviors of those providers and clinics. There’s a lot of work that detailers are already doing that can be easily turned into data sources for both the detailing effort and the evaluation effort. I like to think about how we can make things as simple as possible.
NaRCAD: Is there a specific platform that makes things simple and is best for collecting data for academic detailers and programs?
Kristefer: That certainly becomes a little bit more individualistic when thinking about the needs, challenges, and abilities of each jurisdiction. Detailing is a conversational effort that is attempting to make concrete behavior change, so it becomes more convoluted when you think about how to track a conversation. Some jurisdictions might have a place to track conversations in their electronic health record, where others may not.
At the most basic level, detailers can chart their conversations on one-page Word documents. The detailers I worked with charted their conversations with providers over time and eventually put it in one large PDF that could be easily uploaded into a qualitative data analysis software. You have to be creative when it comes to tracking this information.
NaRCAD: How would you recommend that programs with limited resources go about data collection and evaluation?
Kristefer: A lot of times we say we want to have high-tech solutions when we don’t actually need them. For a resource constrained department, having that one-page Word document that allows detailers to chart their interactions is more than enough data. Charting for just five minutes after a detailing visit about everything that took place becomes a wealth of information. You can also use an Excel document to input data from provider surveys.
There are many low-tech ways to track information and it’s important to be aware of the low hanging fruit in terms of data collection. You want to be able to easily collect data that serves the detailing efforts, the program, and the evaluation and improvement process.
NaRCAD: That’s a great way to look at it. What are some best practices for using data for leadership buy-in?
Kristefer: I can’t stress enough how important data is in getting leadership buy-in. Data is not only quantitative and qualitative, but also using the information gathered to tell a story. It would be a strong statement if someone was able to go to leadership with a story about how providers have changed their practices based on the detailing effort. Using concrete results and showing leadership that detailing is making a change is extremely helpful for buy-in.
Being able to show crazy big outcomes with your data won’t happen, but sharing stories from providers and clinics about how detailing has helped them is quite moving. I’ve heard some amazing stories during my evaluation. For example, detailers helped providers at some clinics to provide patients with directly observed therapy for PrEP at the same time that they were providing them with medication-assisted treatment for opioid use disorder. It’s impressive that detailing at those sites was able to make the clinics think creatively and be able to provide PrEP to these patients.
NaRCAD: That certainly is impressive! What has surprised you the most about the academic detailing data you’ve evaluated?
Kristefer: I can’t help but to think that pharmaceutical companies spend millions of dollars and resources on this model and they certainly wouldn’t have been doing this for decades if it didn’t work. We’re almost a little late to the game as public health practitioners, but through my experience evaluating some of this work and reading other evaluations, I’ve been shocked by how much providers truly value detailers.
It’s fascinating to see how these health systems and departments are viewed as trusted partners by providers and clinics and how detailing has served as a role to improve that partnership and collaboration. Providers have often said how crucial this information has been in getting access to Department of Public Health resources they didn’t even know existed, which is pretty sad. Seeing public health and the medical system working side by side in this kind of way has been breathtaking.
Kristefer Stojanovski is a PhD Candidate in the Department of Health Behavior & Health Education, School of Public Health at the University of Michigan. Kristefer has been doing community-based mixed methods research since 2010. His research explores the social and structural determinants to sexual health and HIV outcomes among key populations in the U.S. and in Southeastern Europe. Kristefer’s work interrogates how stigma drives HIV risk and infection using complex systems theory, structural equation, agent-based and multilevel modeling. Kristefer also translates his research into policy and decision-making. He is an evaluation specialist with the Capacity Building Assistance program with the San Francisco Department of Public Health.
The NaRCAD Team is excited to kick off the latest episode in our C.O.r.E. Podcast Series, this time featuring the insights of Program Director Rebecca Edelberg, MPH, from the Boston-based non-profit academic detailing organization Alosa Health, as she shares her experiences managing field programs in clinical outreach education.
This episode's 15-minute interview with Rebecca hones in on the "how-to's" of strong AD program management, including:
Tune in here, and sound off on Twitter or in the comments section below with insights, questions for Rebecca, or topics you'd like to see featured on our next podcast. Learn more about Rebecca and Alosa Health below!
Rebecca Edelberg, MPH, Program Director, Alosa Health
Rebecca is responsible for providing technical and operational support to field staff, and for ensuring that field-based clinical education programs are executed to clients' satisfaction. Rebecca previously worked at Boston Medical Center implementing a clinical trial and as a consultant in the electronic medical records (EMR) industry, where she engaged in one-on-one clinician education. She has an undergraduate degree from Tufts University and a Masters in Public Health from Boston University with concentrations in Epidemiology and Health Policy. Learn more about Alosa Health's programs, clinical modules, and expert team on their website.
The National Academic Detailing Service’s
Opioid Overdose Education & Naloxone Distribution (OEND) Program
Guest Blog Authors: Melissa Christopher, PharmD, National Director
Mark Bounthavong, PharmD, MPH, National Clinical Program Manager
Tags: Data, Detailing Visits, Evaluation, Opioid Safety, Substance Use
In 2015, the Department of Veterans Affairs (VA) invested in the National Academic Detailing Service to improve the health of our Veterans to address the call to action for the opioid crisis. Through the Opioid Overdose Education and Naloxone Distribution (OEND) Program, our goals were to reduce harm and risk of life-threatening opioid-related overdose and deaths among Veterans.
Key components of the OEND program include raising awareness about the epidemic, 1:1 academic detailing visits with clinicians to provide education and training regarding opioid overdose prevention, opioid overdose rescue response, and issuing naloxone products. We developed direct-to-consumer marketing and other e-resources, including a video, Introduction to Naloxone for People Taking Prescribed Opioids.
We also created implementation tools, including population management dashboards to aid staff in evaluating risk factors of their patient population and distributing naloxone accordingly. Academic Detailers demonstrated to VA providers these resources to help raise awareness of opioid overdose risk for their patient panel.
Decision-makers believed that funding this program would yield a good return on investment. As part of the National Academic Detailing Service, it’s our responsibility to collect data and supply decision-makers with evidence on the value and success of our program. In other words, we’re accountable for answering the question, “Is academic detailing worth it?”
To answer this question, we performed several program evaluations of the National Academic Detailing Service from 2015 to 2017, one of which we just published in the Journal of American Pharmacists Association (JAPhA) (Trends in naloxone prescriptions prescribed after implementation of a National Academic Detailing Service in the Veterans Health Administration: A preliminary analysis.)
The evaluation found that our program improved naloxone distribution rates at a seven times greater increase for Veterans at risk for opioid overdose. These results provided key empirical evidence that VA’s strategy of academic detailing was working. Just as important, these findings also gave decision-makers what they needed—proof that their investment in an area of high risk to Veterans’ health paid off by improving care.
But we learned that another group of stakeholders was just as important as the decision-makers who funded the program—the clinicians that academic detailers visited to provide outreach education as a service. Academic detailers work with clinicians to help them change practice patterns, focusing on improving health outcomes in alignment with balanced, current evidence.
As clinicians commit to sustainable behavior change, these providers need to hear the feedback about how the time they’ve invested with their patients ultimately improves outcomes and, in this case, saves lives.
Sharing program results with the clinicians in this intervention also encouraged these providers to share their own results, many of which were stories of patients returning to the clinic to relate their experiences of using naloxone to reverse an overdose. These stories, along with reversal reports from the field that tracked the outcomes of naloxone kit distribution and subsequent use, also created a tangible “return on investment” for everyone involved.
We encourage other academic detailing programs to prioritize program evaluation as we have at the VHA—no matter the size of your program, if you’re thinking, “we can’t afford to do program evaluations,“ we stress that you can’t afford not to do them.
Measuring program work builds a case not just for the success of one academic detailing intervention, but for the success of future programs—a case for sustainability. Evaluation measures the quality of a program, analyzing results to look at a program’s impact, and allowing for process improvement adjustments to be made to streamline efforts and strengthen that impact. Evaluation cannot be optional, especially when lives are at stake.
We also recommend that the results from program evaluations are shared with other stakeholders, such as clinicians, in order to encourage and sustain their behavior changes. Leveraging results from well-designed evaluation is essential for academic detailing interventions to illustrate success, share value, and provide stakeholders and community members with a clear “Yes!” in answer to their overarching question: “Was the investment worth it?”
Melissa Christopher, PharmD
National Director, Academic Detailing, US Department of Veterans Affairs Central Office, Pharmacy Benefits Management (PBM) Academic Detailing Service
Dr. Christopher is the National Director of VA Academic Detailing Services, overseeing the implementation efforts for academic detailing expansion across all Veteran Integrated Service Networks since 2014. She received her Doctor of Pharmacy from Duquesne University, Pittsburgh Pennsylvania. She completed a Pharmacy Practice Residency and Post Graduate Year 2 in Pharmacoeconomics and Formulary Management at VA San Diego Healthcare System. Dr. Christopher conducted research in health outcomes and pharmacoeconomic analysis for several chronic disease management areas. In recent years, Dr. Christopher has embraced the mission to expand efforts for educational outreach by clinical pharmacists for improvement of evidence based care in Pain Management, Depression, Schizophrenia, and Posttraumatic Stress Disorder as well as other substance use disorders. Most of her program efforts focus on development of educational materials, outcome monitors, provider specific electronic audit and feedback tools to trend practice patterns with implementation efforts for the newly developed as well as fully implemented AD programs.
Mark Bounthavong, PharmD, MPH
National Clinical Program Manager, Academic Detailing Service, Veterans Affairs
Dr. Bounthavong graduated from the College of Pharmacy at Western University of Health Sciences. He completed a PGY-1 Pharmacy Practice Residency at the Veterans Affairs Loma Linda Healthcare System followed by a fellowship in Outcomes Research and Pharmacoeconomics at Western University of Health Sciences. He started his career at the Veterans Affairs San Diego Healthcare System as a pharmacoeconomics clinical specialist. During his tenure at the VA, Mark worked on identifying cost-effective strategies and formulary management; directed the PGY-1 Managed Care Pharmacy Residency; and completed a Master of Public Health from Emory University. Mark left the VA in order to pursue a PhD in the Pharmaceutical Outcomes Research and Policy Program at the University of Washington. He recently accepted a position at the VA as one of the National Clinical Pharmacy Data Program Managers in the Academic Detailing Service.
NaRCAD's Interview Series: Public Health Detailing Program at New York City Department of Health and Mental Hygiene (DOHMH)
Featuring Michelle Dresser, MPH, Senior Manager, Programming & Strategy
Tags: Detailing Visits, Diabetes, Evaluation, Obesity, Program Management, Smoking Cessation, Training
Thanks for taking the time to share the great clinical outreach education work that’s being done by the NYC Department of Health and Mental Hygiene, Michelle! Tell us a bit about yourself and how you got involved in public health, specifically public health detailing.
Michelle: Thank you for the opportunity to speak about the Public Health Detailing Program. I have over 20 years of public health experience in both the non-profit and government setting, with the last 12 here at the New York City DOHMH. Throughout my professional career, my specialty has been in healthcare marketing and provider education, emphasizing how providers and consumers can better communicate with each other by tailoring complex messages using health literacy principles.
It’s essential our reps have excellent selling and communications skills, so when they engage providers and get their buy-in, providers are then equipped to get their patients “on board”. One-on-one provider engagement helps them understand how important it is to have a 2-way communication with patients.
How can an outreach representative encourage providers to “get on board” and think about care as a dialogue?
Michelle: Let’s use obesity as an example. With obesity, both providers and patients are frustrated, for different reasons. Providers may be frustrated that patients’ comorbid conditions are being exacerbated or don’t have the same kinds of tools to treat obesity as they do other conditions; patients might feel that providers aren’t using great communication techniques, like motivational interviewing (MI), to help them set goals and take small steps towards the goal.
If a patient is only told, “You need to lose weight,” which is such a broad and overarching goal, they’ll be frustrated, and frankly, non-adherent. I know I would be.
Encouraging providers to have specific dialogues using a customized approach for each patient is important. This kind of dialogue takes into account patients’ literacy beyond the written and spoken word—it looks at scientific, fundamental, health and cultural literacy, too.
We work on “coaching scripts”, which take the key recommendations and reframes them in order to custom-tailor the conversation for each patient.
One thing that’s unique about public health detailing is that we detail the whole team through one-on-one interactions. Evidence shows these types of interactions with providers and staff are more effective at changing behavior; however, sometimes due to the makeup of the practice we must conduct group presentations. It’s not ideal, but it still allows us to get the messages and materials out there.
So when an outreach representative goes into an office, they detail...everyone?
Michelle: If there are 15 people who work in an office, we’re going to detail all 15 of them. It’s a lot! Sometimes, the person who is the champion of a new behavior or workflow isn’t going to be the provider. We see the front desk staff as instrumental; they’re interacting with all of the patients. We work with our teams to ensure even the front desk staff receives the materials and information, rather than seeing them merely as a “gatekeeper” to get to the providers.
Sounds like a lot of training goes into preparing for your campaigns, and for thinking about the entire process of effective outreach. Tell us more about your trainings, and about how you prepare outreach representatives on disease content training, as well as in marketing and communications skills.
Michelle: On average, our trainings are about 5 days in length and take place the week prior to launching a new campaign. About 40 percent of the training is disease content, so we work with our internal Health Department experts, as well as external experts, where we learn about prevention strategies, treatment strategies, epidemiology and the landscape around the key recommendations chosen based on the evidence of that topic. We need to know the ‘why’ behind the campaign.
Once we have that under our belt, we shift to sessions on how to frame the issue, how to promote the materials, figuring out the “features and benefits” as well as the “barriers and objections” and finally “gaining a commitment”, which are phrases that come from pharmaceutical marketing. We’re “selling” and promoting public health interactions, so we work on those skills.
We also do a great deal of role playing, including videotaped analysis of each rep. We look at body language, what communication skills are effective, we do knowledge assessments, quizzes—we make sure our team is well-prepared to go out and detail. We take this seriously—they’re representing the New York Department of Health and Mental Hygiene.
What’s a major barrier your program has faced, and how have you tackled it?
Michelle: A big challenge, when starting a detailing program, is access. The landscape of healthcare systems in NYC has drastically changed over the past few years. As an example, several years ago, the majority of our Brooklyn territory was almost entirely made of up of small practices where access wasn’t an issue.
What’s changed since then?
Michelle: Now, many of these sites have become part of larger institutions, so there’s corporate buy-in that needs to happen for people to come in and talk to the staff. As I mentioned before, although we try and limit group presentations, this has proven to be an effective strategy when entering into a new relationship. Once they get to know us and recognize the value of the program, they’re engaged in having us come back to conduct 1:1 visits on the follow-up and subsequent campaigns.
How do you know when a campaign is working and becoming successful?
Michelle: Evaluation is always on the top of our priorities, and can be a challenge for any program to evaluate effectiveness. For every campaign we conduct an initial and follow-up visit where we assess provider practice.
This allows us to see if there has been a change in practice from the initial to the follow-up visit. Additionally, we rate what providers intend to adopt in terms of the key recommendations and supporting tools and resources. We also collect a large amount of qualitative data because it's also critical to gaining a more complete picture of the campaign’s success, especially when reporting on barriers, access and materials.
You can scale this up or down, depending on your need and organizational priorities. Our program focuses on where there’s the greatest need and potential for greatest impact.
Programs should make sure to look at their organization’s agenda and goals. It’s important to look at the data and plan the best course of action within the capacity you have.
Biography: Michelle Dresser. Michelle Dresser is the Senior Manager of Programming and Strategy for the Public Health Detailing Program within the Bureau of Chronic Disease Prevention and Tobacco Control at the New York City Department of Health and Mental Hygiene. In this role, she oversees the overall programmatic direction and strategy of the program. This includes, campaign strategy and timing, campaign content, training and economic incentive development, provider selection, identification of targets to ensure the greatest impact on populations most in need, and identification of “new needs” opportunities to expand program reach and achievement of program goals. She also oversees internal and external strategic relationships to enhance programmatic objectives.
by Joy Leotsakos, PharmD
Tags: Cardiovascular Health, Detailing Visits, Evaluation, Program Management, Training
Who We Are. The Academic Detailing Service (ADS) of the Atrius Health Clinical Pharmacy Program provides clinically appropriate, evidence-based, cost-effective medication management in a multidisciplinary team setting. Our Clinical Pharmacy Program includes 15 clinical pharmacists (CPs) serving nineteen Internal Medicine and Family Medicine (IM/FM) ambulatory care practice locations. In the past four years, our program has evolved and transformed through evaluating our impact, absorbing and implementing internal feedback, and collaborating with others in the field, including NaRCAD.
Our Start. As the program manager of our ADS, I’ve seen our service grow and change. When we began our program in 2011, it was as an administrative mandate to meet with all IM/FM prescribers once per fiscal quarter to deliver messages about cost-effective prescribing and clinical quality. We started by formulating a menu of topics to cover in our ADS work each quarter, including individual clinician prescribing reports reflecting performance on prescribing initiatives from the Pharmacy & Therapeutics Committee, specific questions to survey clinicians on a clinical topic, targeted education for low performers on prescribing initiatives, and various other ‘hot topic’ clinical issues. CPs detailed individual clinicians via formal 1:1 scheduled appointments, and also did so less formally (such as by catching them in the hallways) or in larger groups during department meetings.
Is it Working? We documented our ADS activities by checking off the individual clinicians we detailed each quarter. At that time, there was no formal training for our CPs on how to conduct a detailing meeting. Unfortunately, this method of creating content for visits soon resulted in a large menu of topics so varied that each quarter’s detailing became unwieldy and too broadly focused. And our documentation, while it gave us a general sense of the number of clinicians detailed, did not tell us anything about the quality of this detailing.
Room for Improvement. Our group is fortunate in that our ADS activities have always been accepted and even expected by our IM/FM clinicians. We experienced almost no clinician resistance to our educational meetings. But in 2013, when attending one of NaRCAD’s 2-day Academic Detailing Training sessions, I learned that we could make changes to improve our services, as well as my own skills as a detailer. As a result, we altered the format of our ADS program, choosing to detail clinicians in a 1-1 or small group format of less than 4. We also selected a goal of 90% of clinicians receiving detailing at least once every quarter.
Evaluating Impact. We began evaluating the impact of the changes we’d made to our ADS, specifically choosing to look at its impact on a discretely measurable topic: reducing the unnecessary ordering of an ALT test (alanine transaminase) in patients on the ’statin’ cholesterol-lowering medications. We were able to demonstrate that our detailing of all IM/FM clinicians led to significant reductions in ALT ordering and meaningful cost avoidance for our organization.
Asking for Feedback. With NaRCAD’s support, we further refined our program in 2014 based upon feedback from an internal focus group. By soliciting honest feedback from the CPs about their detailing experiences, I discovered considerable variation in how they approached the menu of topics provided each quarter and came to understand that the continuous process of visiting with each clinician at their sites often felt stale and repetitive.
New Approach, New Results. We revised our ADS workflow to tie each round of clinician appointments directly to a specific and single P&T prescribing initiative. Furthermore, we developed a method to tag low performing clinicians for an ‘intense’ ADS visit and higher performers for a ‘touch’ ADS visit. We began this new workflow with an initiative to improve the use of evidence-based beta-blockers in patients with heart failure, a quality measure for the Medicare Pioneer Accountable Care Organization (ACO) project. Using this new approach, clinical pharmacists were able to deliver a fresh and meaningful message to the right prescribers, resulting in a change from 73.6% to 97.8%prescribing of evidence-based beta-blockers in this patient population.
Partnering with NaRCAD for Ongoing Learning. In March 2015, we coordinated with NaRCAD again, and they provided our group of clinical pharmacists with a 2.5 hour workshop to enhance our AD skills. I’d encourage anyone who does this type of educational outreach to make use of this invaluable resource. Of course, our Atrius Health Academic Detailing Service will continue to grow and change as we find additional ways to improve our workflows and messages. I look forward to continued collaboration with NaRCAD and with others in the field, so that we can all keep learning from each other and improve health outcomes through effective academic detailing.
Bio: Joy Leotsakos is a senior clinical pharmacist and the program manager for the Academic Detailing Service (ADS) of the Atrius Health Clinical Pharmacy Program. Joy joined Atrius Health in 2007 and became the program manager for the ADS program in 2012. Prior to joining Atrius Health, Joy worked as an assistant professor at Massachusetts College of Pharmacy and Health Sciences University in Boston, MA and provided ambulatory care pharmacy services to the South End Community Health Center also in Boston. Joy graduated with a Doctor of Pharmacy degree from Virginia Commonwealth University School of Pharmacy and then completed her residency in Ambulatory Care and Community Pharmacy at the University of Florida College of Pharmacy. Joy is the mother of one son, and enjoys salsa dancing, cycling and running in the summer and skiing in the winter. You can reach Joy by email at email@example.com.
Highlighting Best Practices
We highlight what's working in clinical education through interviews, features, event recaps, and guest blogs, offering clinical educators the chance to share successes and lessons learned from around the country & beyond.