Effect of using cardiovascular risk scoring in routine risk assessment in primary prevention of cardiovascular disease: an overview of systematic reviews

Our systematic review protocol was registered with the International Prospective Register of Systematic Reviews (PROSPERO) on 01 September 2016 (registration number CRD42016046898). The full version of the protocol was later published in the BMJ Open [20]. Any deviations from this plan, which were minor, are described in Additional file 1.

Studies were selected according to the prespecified criteria defined below. Those criteria were also mentioned in detail in the full version of the protocol of our study published in the BMJ Open [20].

We searched MEDLINE (Ovid), EMBASE, CENTRAL (Cochrane Central Register of Controlled Trials) and SCOPUS databases from January 1990 to March 2017. All databases from 1990 onwards were scanned because only a very small number of systematic reviews had been conducted before that time [21, 22]. The search was limited to English language literature. It was supplemented by a search for unpublished, ongoing, or recently completed systematic reviews in PROSPERO database. In addition, we reviewed the reference lists of all included systematic reviews. Our detailed search strategy for MEDLINE is presented in Additional file 2.

All identified records were uploaded or manually entered into Mendeley v1.17.6 (Elsevier). After duplicate removal, titles and abstracts from the searches were independently screened by two authors. Full-text articles were retrieved for all potentially includable SRs. Any disagreements were resolved through discussion. In case of lack of consensus, a third author was arbitrated.

The methodology for data extraction and analysis was based on guidance from Preferred Reporting Items for Systematic Reviews and Meta-Analyses (PRISMA) statement [23] and the Cochrane Handbook of Systematic Reviews of Interventions [24]. The PRISMA checklist is presented in an Additional file 3. Two authors independently extracted outcome data from each included SR using a predefined data extraction form. Disagreements during the data extraction process were resolved through discussion or arbitrarily by a third author.

We have extracted the following information: (1) administrative and bibliographic data; (2) the characteristics of each review; (3) methodological details and results of meta-bias assessments (if conducted); (4) reported limitations of each review; (5) results and conclusions of the review.

If meta-analysis was presented in one of the included SRs, we extracted both its results and all relevant methodological aspects (e.g. types and unity of data, effect measured, heterogeneity, sensitivity analysis). As we specified in our overview protocol [20], we did not conduct a meta-analysis of meta-analyses, mainly because of the risk of introducing bias. According to Smith et al. [25] and Pieper et al. [26], such undesired bias can be easily incorporated (due to giving excessive statistical power to primary studies included in more than one systematic review) when performing meta-analysis in review of systematic reviews. Additionally, the same authors highlight that there is no well-established quantification method in this field.

Additionally, we looked at the following information: (1) outcomes reported in a particular SR which were recognised by the authors as evidence of the effectiveness of TRA, (2) reported effect of TRA in different populations, (3) reported effect of different TRA in the same population.

Two reviewers independently appraised risk of bias of the included systematic reviews using a validated assessment of multiple systematic reviews (AMSTAR) checklist, which is the most commonly employed method to assess the quality of systematic reviews included in overviews [27]. We scored each systematic review with a maximum of 11 points. Disagreements were resolved through discussion or arbitrarily by a third author. Each systematic review was assigned to one of three quality levels (0–3 points - low quality, 4–7 points - medium quality and 8–11 points - high quality) [28, 29].

To assess selective outcome reporting within systematic reviews, we compared those which were planned to be assessed in the systematic review protocols (or, if unavailable, in the methods section of the published report of SR) and other outcomes reported in the results section of the published report of SR. To minimise publication bias, we identified all relevant ongoing SRs by searching the PROSPERO database. To assess the degree of overlap in the inclusion of primary studies between systematic reviews, the citation matrix was generated by one reviewer and checked by a second for accuracy (Additional file 4). The degree of overlap was calculated with use of the corrected cover area (CCA) in our citation matrix by an experienced statistician [26].

Characteristics of included systematic reviews are presented in Table 1.

The review performed by Brindle et al. [16] focused both on the accuracy and impact of CVD risk assessment in primary prevention. The authors performed comprehensive searches across 6 databases. The majority of included studies focused on CVD risk assessment accuracy. The authors identified 4 RCTs assessing effectiveness of using CVD risk score. As the quality of evidence was unclear, the authors concluded that it was insufficient to support the claim that CVD risk assessment performed by clinicians improves health outcomes.

Sheridan and Crespo [17] focused on the benefits and harms of physicians’ knowledge of CVD risk scoring. The authors searched one database (MEDLINE) and identified 11 primary studies. They concluded that “physicians’ knowledge of global CHD risk scores may translate into modestly increased prescribing of cardiovascular drugs and modest short-term reductions in CHD risk factors without clinical harm”.

The SR by Sheridan et al. [18] aimed at evaluating the effectiveness of presenting CVD risk information to patients as a primary intervention versus part of a multipart intervention. The researchers studied 4 databases and considered studies of any quantitative experimental design. They identified 18 primary studies of good or fair quality. Results suggested that providing CVD risk information to the patients may “improve accuracy of risk perception and may increase intent to initiate CHD prevention”. However, the effect on more long-term outcomes was not clear.

Waldron et al. [35] tried to assess the effectiveness of using different methods of communicating CVD risk and the impact of these methods on patient-related outcomes. The authors performed comprehensive searches across 6 databases. However, only 4 out of the 15 quantitative studies included assessed individuals’ actual risk. The majority of these studies were analogue studies using hypothetical risk profiles. The authors concluded that there were not many well-designed studies focusing on CVD risk communication.

In their review, Van Dieren et al. [36] aimed to identify all papers presenting CVD prediction models developed in either diabetic or general patient populations that included diabetes as a predictor. The review also identified studies examining the impact of applying a prediction model in clinical practice. They found 3 studies of this type, all of which utilised the Framingham prediction model. Results suggested that it is still unknown whether the use of these models indeed changes the treatment of patients with diabetes mellitus or whether it reduces the number of cardiovascular complications.

Willis et al. [19] aimed to evaluate the effectiveness of using validated CVD risk scores to identify individuals at highest risk followed by intervention as a means of reducing CVD risk or mortality. They included 5 full texts in their review. Two trials reported a reduction in CVD mortality. However one of them showed a lack of statistical significance, and in the second, the p value was not provided. The authors concluded that “evidence suggests that lifestyle interventions aimed at the primary prevention of CVD that use validated risk scores to recruit high risk individuals show potential for lowering CVD mortality and the incidence of cardiovascular events”.

Usher-Smith et al. [37] searched for interventional studies which involved providing a CVD risk assessment to patients or their providers. Final analysis was based on 17 primary studies. Results suggested that providing patients with risk information improves the accuracy of perceived risk without increasing anxiety or decreasing quality of life. Additionally, it leads to a small reduction in cholesterol and blood pressure despite a lack of evidence that this reduction is due to lifestyle changes. Moreover, providing risk information increases the frequency of prescribing lipid-lowering and blood pressure medications.

Tomasik et al. [38] focused only on the effect of using the specifically European total CVD risk estimation from SCORE. A comprehensive search did not reveal any study that compared important clinical outcomes (death, major events, adverse events) between groups that used the SCORE model and those who did not. The authors concluded that a demonstration of the benefits of using SCORE is still lacking and that current use of this tool as a preventive strategy is not supported by evidence.

The Cochrane review by Karmali et al. [39] focused on the effect of physicians, other healthcare professionals or the health system providing a multivariable CVD risk score. They identified 41 RCTs having multiple limitations, substantial heterogeneity and high or unclear risk of bias. Results suggested that the use of CVD risk scores has little or no effect on CVD events compared with standard care, though to a small extent it may reduce risk factor level (systolic BP, total cholesterol), multivariable CVD risk, or adverse events. It may also increase the use of antihypertensive and lipid-lowering pharmacotherapy.

The Collins et al. [40] review of systematic reviews summarises results from 6 SRs on the impact of global CVD risk assessment in primary prevention. No identified review reported results such as mortality or CVD morbidity. The authors also undertook ad hoc meta-analysis and extracted data from 16 primary studies of the identified publications. They showed small and possibly clinically insignificant reductions in systolic BP, cholesterol level and smoking cessation. The quality of evidence was low or very low (GRADE score).

The general conclusions about the included SRs is that there is currently little evidence that providing TRA has a significant impact on important patient outcomes.

AMSTAR ratings ranged from 3/11 to 10/11 (mean 6.1/11); no SRs were excluded based on quality (see Fig. 2). Ten eligible SRs included 66 unique primary studies. Thirty five publications were encompassed in only one SR and 31 in more than one. The measure of overlap by corrected covered area (CCA) was 9%, indicating a moderate overlap [26]. Details of calculations and the list of primary studies are presented in Additional file 4.

In the majority of the SRs, the authors presented the same outcomes in the results sections that they had planned to assess in their initial protocols or in the methodology section of published reports. Collins et al. [40] deviated from their registered protocol, which they reported in Additional file 1, published online. They had originally planned to explore many different outcomes, but instead focused on what they deemed to be the most important ones: CVD-related morbidity and mortality as primary outcomes and systolic blood pressure, cholesterol, and smoking as secondary outcomes. Willis et al. [19] additionally reported cardiovascular disease risk factors which were not in the original scope of their work.

We have identified and analysed 10 SRs which were published between 2006 and 2017. Two of them reported the impact of performing TRA on mortality, fatal and non-fatal CV events. One medium quality SR [19] revealed 2 primary studies reporting reduction of CVD death, but it was statistically insignificant or significance was not reported in the original study. Other high quality SR [39] presented meta-analysis of 3 primary studies which showed no significant differences concerning fatal and non-fatal CV events. Primary studies included in both SRs had important limitations.

Three SRs [17, 37, 39] have analysed the impact of performing TRA on adverse events. It was shown that using TRA causes no difference in presence of psychological or physical events.

The impact of TRA on global CVD risk and individual risk factors is ambiguous, but a tendency towards slight reduction of BP, TC and smoking levels, especially in high risk patients group, was observed. TRA had no influence on lifestyle behaviour.

To the best of our knowledge this is the largest overview of SRs in this area. We included 10 SRs while a previous overview published in March 2017 covered six [40]. Our overview of SRs was conducted strictly according to the protocol, which was registered with the PROSPERO database, and later published in the BMJ Open [20]. Other strengths are a robust search strategy prepared by an experienced librarian, and broad inclusion criteria.

There are some obvious limitations to our overview of SRs. We did not review the quality of each study included in the SRs, nor perform an independent analysis of data (which is standard procedure when performing overview of SRs). The electronic search strategy was limited to 4 main international bibliographic databases and English language literature only. Grey literature for unpublished papers was not studied, however a search for ongoing or recently completed systematic reviews in the PROSPERO database was performed. In addition, we also manually reviewed the reference lists of all included SRs.

Collins et al. [40] completed searches for their overview of SR in October 2016. They did not find SRs concerning their primary outcomes (CVD mortality and morbidity and all-cause mortality). The conclusions of that overview differ from ours and the authors stated that “there is currently no evidence that the prospective use of global cardiovascular risk assessment translates to reductions in CVD morbidity or mortality”. We hold the view that there is a limited evidence for the lack of these reductions.

Results of our overview of SRs can be also compared with evidence constituting the basis for recommendations within national and international guidelines. Major clinical practice guidelines (CCS [13], ESC/ESH [6], ACC/AHA [14], JBS3 [15], NICE [43]) recommend assessing risk of CVD using TRA. It is stated that use of these risk scores increases the accuracy of prediction of CVD events and guides management decisions in primary prevention. Some guidelines indicate that there is no proof for the effectiveness of TRA [6]. Our research has shown the contrary - there is a limited evidence, of low overall quality, for a lack of effectiveness of TRA on CVD events.

There are publications indicating, similar to our study, that some primary prevention interventions may not be effective. The Cochrane review for instance concluded that general health checks performed in healthy individuals do not reduce morbidity or mortality, nor in overall cardiovascular or cancer causes. [32] The Danish Inter99 randomised trial revealed that systematic screening for risk factors of ischaemic heart disease, followed by individually tailored intervention had no effect after 10 years and did not reduce incidence of stroke, ischaemic heart disease, as well as mortality rate [44]. On the other hand Si S. et al. showed that health checks are associated with statistically significant, though clinically small, improvements in surrogate outcomes, especially in high risk individuals [33].

The readers of our overview should take into consideration that the results concerning the effectiveness of TRA comes not from primary studies but are strictly limited to results published in SRs and meta-analyses. Data derived from these types of studies are the most useful evidence in clinical practice [6]. However, our analysis shows a diversity in the quality of included SRs. The majority of them are, according to AMSTAR, of medium quality. We also present the quality of evidence of primary studies as assessed by authors of the included SRs. Most of them reported a low quality of primary studies. In accordance with the authors of these SRs, we agree that conducting further research is still necessary, in order to obtain more persuasive evidence for TRA effectiveness in primary prevention.

There are many reasons why TRA may not be effective and they are described in the literature. Firstly, false positive results of TRA may cause medical treatment which is unnecessary and brings risk of side effects. Secondly, false negative results may cause the lack of appropriate treatment [6]. Thirdly, only a small number of risk variables are included in most of the models (they do not include e.g. family history, TG level, or obesity). Risk categories can be arbitrary (e.g. low, medium, high). In addition, mortality and morbidity rates are declining in most European countries but prediction models are based on historic data and may not apply to the population to be assessed. Moreover, a small change in one of the individual risk factors (e.g. blood pressure) from 1 day to another can markedly change patients’ estimated CVD risk. Finally many physicians identify their patients’ CVD risk factors and launch appropriate treatment without the necessity of TRA [45].

Even though TRA is recommended by major clinical practice guidelines, medical professionals should be aware that current, limited and of low quality, evidence from SRs indicate a possible lack of its clinical effectiveness. Using TRA tools will not harm the patients. There are no grounds to conclude that, in clinical practice, CVD risk assessment based on other than TRA methods, e.g. only on one particular risk factor or counting the number of risk factors instead of using TRA, is inappropriate. However, there are several other potential benefits associated with the use of TRA which are considered in the current guidelines.

Guideline authors should consider re-evaluating the class of recommendations. In the 2016 European Guidelines on CVD prevention [6], the recommendation for routine use of the SCORE algorithm in primary prevention of CVD is marked as class I. This means that there is “evidence and/or general agreement that a procedure is beneficial, useful, effective”. Results of our overview indicate that in this case, class IIb (“usefulness/efficacy is less well established by evidence/opinion”) would be more appropriate. In the British NICE clinical guideline CG181 [43] (concerning lipid modification for the primary and secondary prevention of CVD), the strength of the recommendations in individuals without CVD should also be lowered from “offer” to “consider”.

Researchers should plan and perform impact studies (probably RCTs) which take into account important endpoints e.g. CVD mortality and morbidity, to ultimately confirm the effectiveness or lack of effectiveness of TRA. Development of a new model of TRA and looking for the most appropriate format of its communication also seems inappropriate when effectiveness is not proven. Due to the probability of TRA ineffectiveness, the search for an utterly new method allowing the indication of points of onset and intensification of intervention in CVD primary prevention should be started.

Both researchers and guideline authors should be aware that it is inappropriate to use the models prepared and validated for prediction purposes to make clinical decisions concerning patients. It may be especially misleading if thresholds are based on opinions (which are cheap and quick) and not on impact studies.

Founders and policymakers ought to be aware that spending resources on adopting TRA into practice may be against current evidence which is not strong.

In the protocol of our overview, we expressed optimism that our study might allow patients to have information about a strong scientific background of performed CVD risk assessment. We had hoped that this knowledge would boost patients’ adherence to their physicians’ recommendations. However, these hopes have turned out to have been in vain.