NHC Comments on PFDD Guidance
NHC Comments on PFDD Guidance (PDF)
September 28, 2022
The Honorable Robert M. Califf, M.D.
Administrator
Food and Drug Administration
10903 New Hampshire Ave
Silver Spring, MD 20993-0002
RE: FDA-2022-D-1385: Patient-Focused Drug Development: Selecting, Developing, or Modifying Fit-for-Purpose Clinical Outcome Assessments
Dear Administrator Califf:
The National Health Council (NHC) appreciates the opportunity to provide comments on the third patient-focused drug development (PFDD) Guidance. We are proud of the role the NHC has had in working with the Food and Drug Administration (FDA) to help advance the role of patient-focused drug development in medical innovation and look forward to continuing our partnership in furthering the science of patient engagement.
Created by and for patient organizations over 100 years ago, the NHC brings diverse organizations together to forge consensus and drive patient-centered health policy. We promote increased access to affordable, high-value, equitable, sustainable health care. Made up of more than 150 national health-related organizations and businesses, the NHC’s core membership includes the nation’s leading patient organizations. Other members include health-related associations and nonprofit organizations including the provider, research, and family caregiver communities; and businesses and organizations representing biopharmaceuticals, devices, diagnostics, generics, and payers.
We commend the FDA for producing the third PFDD Guidance in the series. It is particularly noteworthy that it is a collaborative effort by the Center for Drug Evaluation and Research (CDER), the Center for Biologics Evaluation and Research (CBER), and the Center for Devices and Radiological Health (CDRH). It helps stakeholders to know that the three Centers are harmonizing PFDD approaches, which will help create consistency and efficiencies across the agency and the life sciences sectors.
Overall, it is a substantial body of work summarizing the Agency’s views on clinical outcome assessment (COA) selection, development, and use, highlighting the significant role of patient engagement and patient experience data. We also appreciate recognition of the important role of family members and caregivers.
Overarching Supportive Comments
The Guidance introduces the notion of “aspects” of the concept of interest. The NHC very much appreciates this nuance and the recognition that patients’ experiences must inform the selection of the concept(s) and the specific characteristics of those concepts to accurately reflect the impact disease and treatment have on a patient’s health and daily life. For example, we have noted that for one illness, patients report that the most significant issue for them is the constancy of their pain. However, a patient-reported outcome (PRO) instrument widely used for this disease focuses more narrowly on pain intensity, potentially leading to a disconnect between what patients most value and what researchers study.
The NHC appreciates the documented clarification made between proxy- and observer-reported approaches. We have noted confusion on this issue and appreciate it being clarified in the proposed Guidance.
Finally, the NHC appreciates that the revised Roadmap in Figure 2 is improved from earlier versions and more user friendly.
Alignment with Core Impact Set Development
We are very pleased to note that the Guidance includes elements that are consistent with an initiative the NHC began almost two years ago on Patient-Centered Core Impact Sets (PC-CIS). A PC-CIS is a patient-derived and patient-prioritized list of the most important impacts a disease and/or its treatments have on a patient’s health and daily life and that of their family and caregivers. Intentionally broad and inclusive, the term “impacts” are the things patients tell us are important about the way a disease and/or its treatment affects their lives (and that of their family and caregivers). Impacts include short- and long-term health outcomes and any other related implications (e.g., mental health effects, caregiver/family stresses, economic burden, work and career loss, etc.1). The NHC prepared a draft Blueprint2 for developing PC-CIS for which we recently requested public comment and received feedback from various stakeholders, including patient advocacy groups, patient-centered outcomes researchers, and the biopharmaceutical and medical device industries. The final Blueprint will be released this fall.
We note alignment between PC-CIS and the Guidance in the following ways:
- We view a PC-CIS as a precursor, a catalogue of the prioritized impacts patients have told us are most important, that can inform myriad downstream uses, one being COA selection and In Figure 1 of the draft Guidance document, what is labeled as “activity” is what we consider “impacts.”
- The Guidance depicts links along a chain of events connecting patient input to study endpoints. Similarly in our conceptualization of PC-CIS, we view patient engagement as the effort to gather patient experience data, from which impacts are derived, and which then informs concept identification as well as aspects of the concepts of
The NHC respectfully suggests that in the final Guidance the FDA replace the term “activity” with the term “impacts” in depicting and describing what patients report as most important to them about disease and treatment implications for their daily life. This is in alignment with 21st Century Cures legislative language, section 3002, which also specifically uses the term “impacts.”
Overarching Areas of Concern
While the NHC celebrates the progress made in this proposed Guidance, there are some concerns we have about some aspects of the Guidance, and suggestions we would like to offer.
This PFDD Guidance is “intended to help sponsors use high-quality measures of patients’ health in medical product development programs.” The patient voice is an important part of that process, but PFDD is more than just COAs. While it is appropriate (and required by the 21st Century Cures Act) for FDA to dedicate a PFDD Guidance to only COAs, it should be made clear to the public that the FDA does not equate PFDD with only COA selection, development, and use.
We recognize that COAs are an important operationalization of the patient voice. However, the collection of patient experience data and integration of the patient voice as part of PFDD also contribute to medical-product development in ways that go beyond COAs (e.g., asset selection, refining clinical-development plans, protocol development, improvements to trial recruitment, addressing diversity and equity, etc.).
As the NHC commented in its previous letters on PFDD Guidances 1 and 2, we would like to once again strongly encourage the FDA to include introductory statements in this Guidance that clearly convey COAs are only one aspect of PFDD.
In addition, the section on Conceptualizing Clinical Benefit states that when little is known about a patient population or their experiences, it is acceptable to rely on literature or clinical expert input. We are concerned about this recommendation for two reasons. It may discourage needed qualitative research, as it implies it can be bypassed. Qualitative research may add time and costs. This recommendation could be used as a way to short-cut methods, weakening the process. Second, a conceptual model created in this manner could be based on outdated or incorrect perceptions, beliefs, or interpretations since patients are not involved. This can bias the model. We believe the conceptual model must be created with patients. Additionally, researchers need to avoid biasing patients, leading patients in the direction of an already “baked” model.
It also would be helpful to be more explicit about the role of patient input. While the document does a very good job of highlighting the patient’s role in general, PFDD represents a significant paradigm shift that we are only part way through. Thus, there are still cultural barriers of resistance to change to overcome. For example, there is a difference between recommending that “patient perspectives be considered” versus recommending inclusion of “perspectives gathered directly from patients.” The former might be interpreted as it being acceptable to have clinicians provide the patient perspective because they are experts on the disease and provide patient care. Whereas the latter leaves no doubt it should be directly from patients. Areas in the document that could be strengthened on this point include but are not limited to:
- Line 302: The draft Guidance lists possible sources of evidence that may be cited in the rationale for why a COA should be considered fit for purpose. This section should point out that while all/any of the listed “sources” are accepted, the sources should include or report on experience data that comes from patients regarding what is important to them, not documentation of other’s views of what is important to patients.
- Line 332: The section on understanding the disease or condition mentions that patient perspectives should be considered, but again does not reinforce that those perspectives should come directly from patients.
The Guidance should explicitly point out that part of evaluating an existing, legacy measure is to understand if it really captures what is important to patients, the same concept of interest, and the same aspect(s) of the concept of interest. It is important to know/understand how content validity was established. For example, as described earlier, in some disease states, patients report it as constancy of pain that bothers them most, reporting, “It’s always there.” Yet, most COA measures ask about pain intensity, not constancy. Thus, it is easy to find a tool that captures the concept of pain and someone conducting a search could easily locate them. It would, of course, be faster and cheaper to use an existing measure. But it would be measuring the wrong aspect of pain. We suggest this be addressed more overtly in the Guidance.
Specific Recommendations
The following are some specific recommendations for modifications to the proposed Guidance. Each one is preceded by the reference line number from the Guidance.
Line 33-36
The issue of following local laws and institutional policies for protecting patients is important, but as phrased could be misleading. True patient engagement involves partnering with patients, e.g., collaborating with patients who are involved in informing/designing the research and not as research subjects. Both are needed in COA development. However, many institutional review boards (IRBs) are confused about how to handle patient partners in research. Patient partnership (e.g., patients as investigators, advisory board members, etc.) in a research project should not require IRB review any more than including a clinical specialist as an advisor in a research project would involve IRB review.
If the work involves data collection from patients as part of activities such as interviews, focus groups, or survey participation, then IRB review for patient protections would be warranted just as for any other type of research involving patients as study subjects. It also should be noted that just simply collecting data from patients should not be confused with patient engagement in research. See the NHC Patient Engagement Rubric.
It would be very informative to the field if the FDA provided clearer differentiation between those patients involved in patient engagement (e.g., co-investigators, advisors, consultants) to inform the research versus study subjects from whom data are being collected as part of research.
Then, the role of and need for patient protections in research would be clearer.
Line 136
This document is understandably heavily patient-reported outcome (PRO) focused. FDA should clarify if it is to be assumed that what is in this Guidance applies to all COAs even though not all may be specifically addressed throughout in detail. This is particularly important for clinician- reported outcomes (ClinROs) and performance outcomes (PerfOs), to encourage that they be patient centered and not clinician driven.
The FDA should provide caveats for when something is indicated here with PRO relevance, but it might not have relevance for other COAs. Likewise, when there is an issue or nuance relevant for the others that is not relevant for PROs, it also should be called out. Many of these caveats are already mentioned in various places in the draft Guidance. It would be useful to capture and log them together somewhere.
This has relevance to the patient community, to avoid disenfranchisement simply because the Guidance may not have been explicit in some areas.
Line 665
The document describes the importance of qualitative research (e.g., cognitive interviewing) to ensure respondents understand items and response choices. We agree, this is important.
However, the document does not address the role of cognitive interviews in the development (or adaptation) of the tool for concept validation (does the tool capture the concept(s)/aspect(s) patients say is important to them). We have seen cognitive interviewing in COA development where patients are asked if they understand the questions and response choices — but were never asked if they care about the concept they are being asked about. This should be addressed explicitly in the Guidance. Before moving from the conceptual model to the measurement model, patients should confirm the concept(s)/aspect(s).
Line 677
The Guidance recommends good questionnaire design such as avoiding double negatives in item wording. The FDA should also emphasize the importance of being mindful of literacy and numeracy. This is an important equity issue. If patients do not understand the questions and response choices, it is the fault of the researchers that did insufficient cognitive testing. It is not the patients’ fault for having low health literacy.
FDA also should point out that patient engagement in COA design, especially PRO design, is essential to ensure the words and phrases used in items are those patients use and understand. If the concept of interest is truly derived from the impacts of disease and treatment patients experience, there will be fewer problems with patient understanding of items and response choices.
Figure 3
This figure depicts that there is a concept that is not measured, Symptom A. The FDA should add here that they would expect to see rationale on why Symptom A was not selected to discourage cherry-picking of symptoms to measure (or ignore). We also would be concerned and want to avoid the rationale that it did not get measured because it is “too hard” to measure. This could discourage investment in COA development for the concepts patients prioritize as most important but might not ever be measured.
Figure 3 is also confusing with the clinical trial sample’s data flowing, right to left, back into the diagram and it seems to leave out some important elements. While we recognize this is a hypothetical illustration, we believe it may be misleading in several areas and make the following suggestions for its clarification:
- The diagram implies conceptual-model and measurement-model development happen simultaneously when they must be sequential. Furthermore, there is no consideration for how the conceptual model is related to the measurement model in terms of either methods or We believe it would make sense to split the diagram into sections that align with the sequencing of the roadmap, outlining progression and type of work over time.
- Where the header states COI, it would be helpful to indicate here not only the COI but also the aspect(s) of the COI.
- Where the header states “selected” COA and scores, indicate selected “type of COA ”
- By indicating “Patients in Trial Sample,” it implies measurement-model operationalization happens only in trials. We believe that while some data collection in trials is useful, most of the data collection for COA development is best performed before the trials begin, outside of trials. The figure implies otherwise. We suggest indicating that the measurement model should be tested in representative samples of the target population.
- The figure does not indicate where concept validation takes place. We believe this is needed and should happen before the conceptual model is finalized with confirmation of content validity before moving on to the measurement It should indicate that unless the concept(s), the aspect(s) of the concept(s), the response options, and scoring are confirmed by patients through appropriate qualitative research, it does not make sense to move to the measurement model. We suggest this be indicated in the diagram.
Table 1
The table appears to be very useful to guide steps and documentation. But it seems like – in terms of being a tool – that it jumps into the workflow mid-stream and does include the rationale behind selecting the COI and aspects of the COI and does not specifically include concept validation. It would be useful if this table started at the beginning by adding: “A. The concept(s) of interest and meaningful aspects of those concepts were selected based on….. ” We believe this would strengthen the rationale overall.
Conclusion
The NHC appreciates the opportunity to provide input into the draft Guidance. Please do not hesitate to contact Eric Gascho, Senior Vice President of Policy and Government Affairs, if you or your staff would like to discuss these issues in greater detail. He is reachable via e-mail at [email protected].
Sincerely,
Randall L. Ruttaz
Chief Executive Officer
NHC Comments on PFDD Guidance
09/28/2022
NHC Comments on PFDD Guidance (PDF)
September 28, 2022
The Honorable Robert M. Califf, M.D.
Administrator
Food and Drug Administration
10903 New Hampshire Ave
Silver Spring, MD 20993-0002
RE: FDA-2022-D-1385: Patient-Focused Drug Development: Selecting, Developing, or Modifying Fit-for-Purpose Clinical Outcome Assessments
Dear Administrator Califf:
The National Health Council (NHC) appreciates the opportunity to provide comments on the third patient-focused drug development (PFDD) Guidance. We are proud of the role the NHC has had in working with the Food and Drug Administration (FDA) to help advance the role of patient-focused drug development in medical innovation and look forward to continuing our partnership in furthering the science of patient engagement.
Created by and for patient organizations over 100 years ago, the NHC brings diverse organizations together to forge consensus and drive patient-centered health policy. We promote increased access to affordable, high-value, equitable, sustainable health care. Made up of more than 150 national health-related organizations and businesses, the NHC’s core membership includes the nation’s leading patient organizations. Other members include health-related associations and nonprofit organizations including the provider, research, and family caregiver communities; and businesses and organizations representing biopharmaceuticals, devices, diagnostics, generics, and payers.
We commend the FDA for producing the third PFDD Guidance in the series. It is particularly noteworthy that it is a collaborative effort by the Center for Drug Evaluation and Research (CDER), the Center for Biologics Evaluation and Research (CBER), and the Center for Devices and Radiological Health (CDRH). It helps stakeholders to know that the three Centers are harmonizing PFDD approaches, which will help create consistency and efficiencies across the agency and the life sciences sectors.
Overall, it is a substantial body of work summarizing the Agency’s views on clinical outcome assessment (COA) selection, development, and use, highlighting the significant role of patient engagement and patient experience data. We also appreciate recognition of the important role of family members and caregivers.
Overarching Supportive Comments
The Guidance introduces the notion of “aspects” of the concept of interest. The NHC very much appreciates this nuance and the recognition that patients’ experiences must inform the selection of the concept(s) and the specific characteristics of those concepts to accurately reflect the impact disease and treatment have on a patient’s health and daily life. For example, we have noted that for one illness, patients report that the most significant issue for them is the constancy of their pain. However, a patient-reported outcome (PRO) instrument widely used for this disease focuses more narrowly on pain intensity, potentially leading to a disconnect between what patients most value and what researchers study.
The NHC appreciates the documented clarification made between proxy- and observer-reported approaches. We have noted confusion on this issue and appreciate it being clarified in the proposed Guidance.
Finally, the NHC appreciates that the revised Roadmap in Figure 2 is improved from earlier versions and more user friendly.
Alignment with Core Impact Set Development
We are very pleased to note that the Guidance includes elements that are consistent with an initiative the NHC began almost two years ago on Patient-Centered Core Impact Sets (PC-CIS). A PC-CIS is a patient-derived and patient-prioritized list of the most important impacts a disease and/or its treatments have on a patient’s health and daily life and that of their family and caregivers. Intentionally broad and inclusive, the term “impacts” are the things patients tell us are important about the way a disease and/or its treatment affects their lives (and that of their family and caregivers). Impacts include short- and long-term health outcomes and any other related implications (e.g., mental health effects, caregiver/family stresses, economic burden, work and career loss, etc.1). The NHC prepared a draft Blueprint2 for developing PC-CIS for which we recently requested public comment and received feedback from various stakeholders, including patient advocacy groups, patient-centered outcomes researchers, and the biopharmaceutical and medical device industries. The final Blueprint will be released this fall.
We note alignment between PC-CIS and the Guidance in the following ways:
The NHC respectfully suggests that in the final Guidance the FDA replace the term “activity” with the term “impacts” in depicting and describing what patients report as most important to them about disease and treatment implications for their daily life. This is in alignment with 21st Century Cures legislative language, section 3002, which also specifically uses the term “impacts.”
Overarching Areas of Concern
While the NHC celebrates the progress made in this proposed Guidance, there are some concerns we have about some aspects of the Guidance, and suggestions we would like to offer.
This PFDD Guidance is “intended to help sponsors use high-quality measures of patients’ health in medical product development programs.” The patient voice is an important part of that process, but PFDD is more than just COAs. While it is appropriate (and required by the 21st Century Cures Act) for FDA to dedicate a PFDD Guidance to only COAs, it should be made clear to the public that the FDA does not equate PFDD with only COA selection, development, and use.
We recognize that COAs are an important operationalization of the patient voice. However, the collection of patient experience data and integration of the patient voice as part of PFDD also contribute to medical-product development in ways that go beyond COAs (e.g., asset selection, refining clinical-development plans, protocol development, improvements to trial recruitment, addressing diversity and equity, etc.).
As the NHC commented in its previous letters on PFDD Guidances 1 and 2, we would like to once again strongly encourage the FDA to include introductory statements in this Guidance that clearly convey COAs are only one aspect of PFDD.
In addition, the section on Conceptualizing Clinical Benefit states that when little is known about a patient population or their experiences, it is acceptable to rely on literature or clinical expert input. We are concerned about this recommendation for two reasons. It may discourage needed qualitative research, as it implies it can be bypassed. Qualitative research may add time and costs. This recommendation could be used as a way to short-cut methods, weakening the process. Second, a conceptual model created in this manner could be based on outdated or incorrect perceptions, beliefs, or interpretations since patients are not involved. This can bias the model. We believe the conceptual model must be created with patients. Additionally, researchers need to avoid biasing patients, leading patients in the direction of an already “baked” model.
It also would be helpful to be more explicit about the role of patient input. While the document does a very good job of highlighting the patient’s role in general, PFDD represents a significant paradigm shift that we are only part way through. Thus, there are still cultural barriers of resistance to change to overcome. For example, there is a difference between recommending that “patient perspectives be considered” versus recommending inclusion of “perspectives gathered directly from patients.” The former might be interpreted as it being acceptable to have clinicians provide the patient perspective because they are experts on the disease and provide patient care. Whereas the latter leaves no doubt it should be directly from patients. Areas in the document that could be strengthened on this point include but are not limited to:
The Guidance should explicitly point out that part of evaluating an existing, legacy measure is to understand if it really captures what is important to patients, the same concept of interest, and the same aspect(s) of the concept of interest. It is important to know/understand how content validity was established. For example, as described earlier, in some disease states, patients report it as constancy of pain that bothers them most, reporting, “It’s always there.” Yet, most COA measures ask about pain intensity, not constancy. Thus, it is easy to find a tool that captures the concept of pain and someone conducting a search could easily locate them. It would, of course, be faster and cheaper to use an existing measure. But it would be measuring the wrong aspect of pain. We suggest this be addressed more overtly in the Guidance.
Specific Recommendations
The following are some specific recommendations for modifications to the proposed Guidance. Each one is preceded by the reference line number from the Guidance.
Line 33-36
The issue of following local laws and institutional policies for protecting patients is important, but as phrased could be misleading. True patient engagement involves partnering with patients, e.g., collaborating with patients who are involved in informing/designing the research and not as research subjects. Both are needed in COA development. However, many institutional review boards (IRBs) are confused about how to handle patient partners in research. Patient partnership (e.g., patients as investigators, advisory board members, etc.) in a research project should not require IRB review any more than including a clinical specialist as an advisor in a research project would involve IRB review.
If the work involves data collection from patients as part of activities such as interviews, focus groups, or survey participation, then IRB review for patient protections would be warranted just as for any other type of research involving patients as study subjects. It also should be noted that just simply collecting data from patients should not be confused with patient engagement in research. See the NHC Patient Engagement Rubric.
It would be very informative to the field if the FDA provided clearer differentiation between those patients involved in patient engagement (e.g., co-investigators, advisors, consultants) to inform the research versus study subjects from whom data are being collected as part of research.
Then, the role of and need for patient protections in research would be clearer.
Line 136
This document is understandably heavily patient-reported outcome (PRO) focused. FDA should clarify if it is to be assumed that what is in this Guidance applies to all COAs even though not all may be specifically addressed throughout in detail. This is particularly important for clinician- reported outcomes (ClinROs) and performance outcomes (PerfOs), to encourage that they be patient centered and not clinician driven.
The FDA should provide caveats for when something is indicated here with PRO relevance, but it might not have relevance for other COAs. Likewise, when there is an issue or nuance relevant for the others that is not relevant for PROs, it also should be called out. Many of these caveats are already mentioned in various places in the draft Guidance. It would be useful to capture and log them together somewhere.
This has relevance to the patient community, to avoid disenfranchisement simply because the Guidance may not have been explicit in some areas.
Line 665
The document describes the importance of qualitative research (e.g., cognitive interviewing) to ensure respondents understand items and response choices. We agree, this is important.
However, the document does not address the role of cognitive interviews in the development (or adaptation) of the tool for concept validation (does the tool capture the concept(s)/aspect(s) patients say is important to them). We have seen cognitive interviewing in COA development where patients are asked if they understand the questions and response choices — but were never asked if they care about the concept they are being asked about. This should be addressed explicitly in the Guidance. Before moving from the conceptual model to the measurement model, patients should confirm the concept(s)/aspect(s).
Line 677
The Guidance recommends good questionnaire design such as avoiding double negatives in item wording. The FDA should also emphasize the importance of being mindful of literacy and numeracy. This is an important equity issue. If patients do not understand the questions and response choices, it is the fault of the researchers that did insufficient cognitive testing. It is not the patients’ fault for having low health literacy.
FDA also should point out that patient engagement in COA design, especially PRO design, is essential to ensure the words and phrases used in items are those patients use and understand. If the concept of interest is truly derived from the impacts of disease and treatment patients experience, there will be fewer problems with patient understanding of items and response choices.
Figure 3
This figure depicts that there is a concept that is not measured, Symptom A. The FDA should add here that they would expect to see rationale on why Symptom A was not selected to discourage cherry-picking of symptoms to measure (or ignore). We also would be concerned and want to avoid the rationale that it did not get measured because it is “too hard” to measure. This could discourage investment in COA development for the concepts patients prioritize as most important but might not ever be measured.
Figure 3 is also confusing with the clinical trial sample’s data flowing, right to left, back into the diagram and it seems to leave out some important elements. While we recognize this is a hypothetical illustration, we believe it may be misleading in several areas and make the following suggestions for its clarification:
Table 1
The table appears to be very useful to guide steps and documentation. But it seems like – in terms of being a tool – that it jumps into the workflow mid-stream and does include the rationale behind selecting the COI and aspects of the COI and does not specifically include concept validation. It would be useful if this table started at the beginning by adding: “A. The concept(s) of interest and meaningful aspects of those concepts were selected based on….. ” We believe this would strengthen the rationale overall.
Conclusion
The NHC appreciates the opportunity to provide input into the draft Guidance. Please do not hesitate to contact Eric Gascho, Senior Vice President of Policy and Government Affairs, if you or your staff would like to discuss these issues in greater detail. He is reachable via e-mail at [email protected].
Sincerely,
Randall L. Ruttaz
Chief Executive Officer