• Institute of Good Manufacturing Practices India
  • Global accredited training & certification provider
  • Approved by Quality Council of India (QCI) & Accredited Vocational Institution of Ministry of HRD, Government of India
  • Approved Training Institute of Food Safety and Standards Authority of India (FSSAI)
  • Recognized by Department of Industrial Policy & Promotion, Ministry of Commerce & Industry, Government of India
  • An ISO 9001:2015 Certified Organisation registered under The Societies Registration Act,1860 Government of India
  • Empanelled under Ministry of Horticulture and Food Processing, Government of Uttar Pradesh
  • Affiliated with Life Sciences and Food Industry Sector Skills Council (SSC) set up by National Skill Development Corporation (NSDC)
  • Conferred with QUALITY COUNCIL OF INDIA (QCI) – D.L. SHAH NATIONAL QUALITY AWARD & ASSOCHAM Services Excellence Award 2017

Trusted by organizations & training participants in over 30 countries
Training | Certification | Education | Research


Use of indirect treatment comparison/network meta-analysis (NMA) study for assessment of relevance and credibility to be used in clinical research decision making

Updated: Dec 12, 2018

Network meta-analysis has proved to be an important tool for prospective and retrospective observational studies for decision makers of the clinical research industry. The potential value of network meta-analysis is still not known among the healthcare decision makers in clinical trials. For this purpose, task force developed a consensus based questionnaire consisting of 26 questions. The questionnaire was initially modified after internal testing by the task force members during September and October 2012 and then again modified in April and May, 2013. This is the current questionnaire for the relevance and credibility assessment.

Information regarding clinical practice guidelines, management of formulary and reimbursement policies is provided by systematic reviews of the randomized controlled trials (RCT’s) and such reviews take into consideration the network meta-analysis. For sound decision making, all relevant competing interventions must be compared. These comparisons ideally would be done by RCT’s by making comparisons of the interventions of interest. The availability of such studies is almost negligible; therefore, decision making is a complicated process. Therefore, the Questionnaire of network meta-analysis is used to ease the decision making process.

In the cases where the trials involving a direct comparison of the treatments of interest are absent, an indirect comparison can be made which gives evidence between competing interventions of the difference in treatment effects and hence, the best choice is selected. For example, if the comparator is same, a comparison can be made between two treatments indirectly in an RCT. There often exists an argument between direct and indirect comparisons that indirect comparisons are made only in the absence of direct comparisons but available evidence can be used by combining information available from both direct and indirect comparison. A network of evidence is constructed by a collection of RCTs and each RCT corresponds to a treatment subset. Network meta-analysis creates such an evidence of treatments which contains either direct data comparison or indirect data comparison or both. In traditional meta-analysis, same intervention is compared with the same comparator. This concept is extended by network meta-analysis which includes several pair-wise comparisons against a range of interventions. It is accepted by Canadian Agency for Drugs and Technologies in Health, National Institute for Health and Care Excellence in the United Kingdom, Institute for Quality and Efficiency in Health Care in Germany and Haute Autorit© de Sant© in France. For effective decision making process, the assessment of quality of network-meta analysis must be done by uniform and transparent ways.

The task force aimed towards the achievement of two goals for the assessment of the network-meta analysis for creation of a questionnaire for health-care decision makers which can be understood by people knowing basics of clinical research and it should impart knowledge to the users of network meta-analysis and should have educational and instructional value.

A questionnaire is prepared by the task force based on two important characteristics: relevance and credibility. Relevance describes the extent of application of network-meta analysis for viewing the setting of interest for the decision-maker. Â It contains the questions related to population, comparators, end points, time-frame and other differences relevant to reimbursement policy. There are four questions involved in checking the relevance. These are as follows:
a) Is the population relevant?
b) Are any relevant interventions missing?
c) Are any relevant outcomes missing?
d) Is the context (settings and circumstances) applicable?

After ascertaining the relevance of network-meta analysis, its credibility is assessed. Credibility takes into account that to what extent the answers are valid and accurate in comparison to answers, Â a designed questionnaire is supposed to answer. It includes questionnaire for evidence base, analysis, reporting quality and transparency, interpretation and conflict of interest. The task force is dependent on the expertise of its members and scientific literature for generation of the credibility questionnaire. Reviewers were guided by the items and suggested wordings which are informed to users by putting efforts.

The responses of these questions can be eyes, no or can'et answer Can't answer can only be used if the assessor is having insufficient knowledge and training or the details of the question are not adequate. Based on the answers of the questionnaire, judgment about credibility is made. Network meta-analysis results are rated by the following terms- strength means a good network meta-analysis with no shortcomings, neutral means that some issues were there but they were not affecting credibility and hence, ignored by the users, weakness means there are many important shortcomings in the conduct and design of network meta-analysis which provides biased and misleading results and the term fatal flaw means that users considers the analysis to be biased and the results to be misleading.

Following above answers, the credibility and relevance of network meta-analysis will be assessed for decision making, to be either sufficient or insufficient. Here, for final judgment the frequency of each designative term is not measured as it is not a checklist or a score. After consideration of all the inputs, the user decides the incorporation of results provided by network meta-analysis for decision making process.

The questionnaire consisting of 22 questions is depicted below:


Evidence Base a) Attempt to include all relevant RCTs
b) 1 Network?
c) No poor quality RCTs
d) No difference in effect modifiers between direct comparisons?
e) If differences exist, whether these were due to imbalances in effect modifiers across different treatment comparisons?
f) Bias due to selective reporting of outcomes in the study?
Analysis a) Nave comparisons avoided?
b) Consistency assessed?
c) With consistency was direct and indirect evidence included?
d) Account for inconsistency/minimize bias?
e) Valid rational for FE/RE model?
f) Rationale for heterogeneity assumptions in RE model discussed?
e) Subgroup/meta regression analysis?
Reporting Quality and Transparency a) Network and source data presented?
b) Direct and indirect results reported
c) Are all contrasts presented with uncertainty?
d) Ranking of treatments presented?
e) Results by subgroup or levels of effect-modifiers presented?
f) Individual study results reported?
Interpretation Conclusions fair and balanced?
Conflict of Interest a) Conflict of Interest?
If yes, steps taken to address these?

Questionnaire for credibility depicting 22 questions. Two questions are covered in the conflict of interest.

Thus meta-analysis is advantageous and superior to randomized clinical trials (RCT’s) in a way that RCT’s don’t contain sufficient, adequate and complete knowledge of interventions of interest leading to complex decision making process. Therefore, this questionnaire ease the decision making process by giving evidence of competing interventions of the difference in treatment effects and hence, the best choice is selected out of the many interventions. This questionnaire inculcates confidence in the decision makers and gives them knowledge about different types of comparisons and studies involved for taking decisions about clinical research. For building up of evidence which is addressing the same question from several RCT’s many reviews use meta-analysis. Also, these questionnaires are periodically evaluated and modified by the users for further improvement. These questionnaires  are very much useful in clinical research.