Conducting Health Services Research

Author(s): Ann Dewey , Amy Drahota, Carole Fogg, Sue Halson-Brown, Sally Kilburn, Heather Mackenzie, Chris Markham, Rebecca Stores, Annabel Tremlett

School of Health Sciences and Social Work, University of Portsmouth, James Watson West, 2 King Richard I Road, Portsmouth PO1 2FR, UK

[email protected]

The purpose of this beginner’s guide is to start you off on the research journey by outlining the sequence of steps along the research process (see Figure 1) and providing guidance, including signposting other useful resources that can help support each stage of the process.

Figure 1. Steps along the research process


What is Health Services Research?

Health services research is the scientific investigation of the use, costs, delivery and effects of health care treatments or services for individuals and populations (Lohr and Steinwachs 2002).   It is not about collecting information for information’s sake or transforming facts from one place to another.  Health services research involves systematically seeking knowledge which will lead to improvements in the delivery of health care  (Crombie and .Davies 1996). This type of research requires team members from a wide range of backgrounds, for example, healthcare professionals, University researchers, charities or commercial organisations, social researchers, epidemiologists, economists and statisticians as well as non-professionals such as those who may potentially use the service or treatment.

Good health services research is not accidental. It requires careful planning as well as careful implementation.   Before starting a research project, it is therefore important to prepare a research proposal in which the aims and research questions, background, plans and reasoning for the research are clearly laid out.  Good planning will help you to anticipate and avoid numerous obstacles and pitfalls, save time in the long run, and produce a superior finished product. Remember effective planning for health services research involves seeking views and expertise from a wide range of professionals and non-professional groups.

1.    Getting Started – thinking of an idea

Taking that first step is always the hardest and always starts with a question, “What do I want to research? Where do I get ideas from?”  Ideas can come from anywhere within your practice.  Here are some sources of research ideas you might consider:

  • Service user comments.  What do your service users think about the service?  Do they think they are receiving what you think you are delivering?  Is your service accessible to all?   Can you undertake a reflective cycle to identify key issues for your service users?
  • Personal interest
    • From your practice.  If you have an interesting finding, do other people find the same?
    • From literature you have read.  Is there a question left unanswered?
  • Audit of your own work or the work of others – but remember if you are considering an audit you must have a benchmark or gold standard with which to compare your findings.
  • Topics of local or national interest such as screening programmes.  Are you offering a service which is part of a local or national programme?  What are your findings?  Who attends screening programmes ...who does not attend?  Is there anything that you are doing that is different from others offering the same thing?
  • Service development.  Are you meeting the needs of your population?  What is essential, what is desirable and what is something to aspire to?  Could your research move you or your organisation from providing a basic service to providing an exemplary service?
  • Role extension.  Are you happy in your role?  Are you capable of more?  Could your research provide evidence that you can work at a higher level?
  • Evidence Based Practice (EBP).  Can your research provide the evidence to change practice within your field?  Can you see a pattern of evidence emerging from your own practice that may not currently be evidenced in the literature?
  • Practice specialty.  Are you working in a specialty? Can you share that practice with others?

All you need to get started is: time, an understanding of how far you want to take your research, some local resources and possibly, though not essentially, some funding.  Taking that first step to research can have a profound effect on your practice.  So ... let’s get started!

2. Turn your idea into a research question

A key stage in conducting any piece of research is to spend time at the outset formulating a clear and answerable research question. It is important to be clear in your own mind what question you are intending your research findings to answer. In addition, having a clear question will help you search for previous research efficiently and effectively to discover what work has already been done in your chosen area.  Poor research questions are those that are:

  • Too general (for example: What are the issues affecting child health?)
  • Too vague (for example: Is community care a good thing?)
  • Too ambitious (for example:  What mode of delivery of health care is more effective in terms of health outcomes and cost efficient, community or hospital based?).

First consider what type of question you are asking. This will help you to decide what research design and techniques you should apply. The most common types of research question are:

  • effectiveness questions (which usually evaluate methods of treatment);
  • causal (which determine the cause of a disease or condition);
  • incidence or prevalence (which determine population incidence or prevalence of a disease or characteristics of a disease);
  • screening (where a specific screening programme is evaluated);
  • diagnostic (where different types of tests are evaluated);
  • prognosis (which estimate the likely progress of a condition), cost effectiveness (where the economics of health care is evaluated) and
  • psychosocial (knowledge, behaviour, attitudes and beliefs).

The acronym PICO or PECO  (Richardson et al. 1995)  is often used to highlight the most important aspects of an answerable research question. This helps you break down your question into different components.

  • P stands for ‘population’,
  • I stands for ‘intervention’ and E for ‘exposure’,
  • C for ‘comparison’ and
  • O for ‘outcome’. 

In Table 1 we show how the following question can be broken down to the PICO format: “In pregnant women with uncomplicated malaria, what is the effectiveness and safety of artemisinin combination therapy, e.g. Coartem, compared to quinine?”

Table 1.  Components of a research question


Depending on your research topic, it may not always be appropriate to specify an intervention, but population and outcomes can usually be identified. For complex problems, it may be necessary to map out a series of component parts and ask a succession of questions.

For those asking questions about people’s knowledge, attitudes, behaviour or decision making, the following adaptation to the standard PICO format may be more useful.  In the adapted PICO (Qualitative)  (Aveyard and Sharp 2009) the P stands for people or perspective (for example, women coming for HIV testing and counselling , the I stands for Issue (for example, non-attendance), the C stands for context or setting (for example, local service), and the Outcome for attitudes or opinions (for example, exploring the reasons/rationale behind non-attendance).

Your research questions should be relevant to those using the service and build on the accumulated knowledge available. Most often research projects are not gigantic feats of epic proportion with ground-breaking findings, but rather smaller steps which build upon previous research developments and which contribute valuable information to a growing field of study.  

3.    Review the relevant literature

An in-depth review of the available research evidence and expert opinion is essential in order to:

  • Indicate the scale of the problem you want to research
  • Summarise what is already known about the topic
  • Highlight questions that would inform practice
  • Present an account of relevant research on the same or similar topics

Searching the literature helps to ensure that your research will address an appropriate gap in the evidence (rather than duplicate what has already been done), and also help you refine your research question and choice of methods by learning from what others have done. A good starting point is to seek out systematic reviews in your chosen area (Lancey 2010); the Cochrane Library (Cochrane Collaboration 2010) allows you to search the Cochrane Database of Systematic Reviews (CDSR) and the Database of Abstracts of Reviews of Effects (DARE). Check the reviews for recommendations for future research and consider which of the included studies would be good for you to obtain. Next you can probe deeper into the primary research (Lancey 2010) to identify more recently published studies, or those which may have been outside the scope of the systematic reviews you identified. Consider also checking trial registries (Moja et al. 2009) to see if there is any on-going unpublished research in your area; a useful web site is Trials Central  (Trials Central).

When carrying out your literature review and reading other people’s work, it is important to not only look at what they have done but also how. Through critiquing the methodology in other people’s work you can build up a better understanding and justification for your own research approach. Critique is not just about negative criticism, but should work to empower you to make well-judged decisions for your own research design. It is well worth looking at advice on how to critique, for example the Critical Appraisal Skills Programme, created useful tools as guidance on how to critique all types of research, from qualitative to quantitative frameworks (Solutions for Public Health 2010), and the British Medical Journal produced a series on how to read a paper (British Medical Journal 2011).

4.  Select the study design and methods

The type of research question you are asking will influence the subsequent choice of research design.  Generally we can divide research design into quantitative research and qualitative research.

Quantitative research

Quantitative research  can be defined as “a formal, objective, systematic process in which numerical data are utilised to obtain information about the world" (Crombie and Davies 1996). Quantitative research usually, but not always, sets out to test a hypothesis, for example, ‘In pregnant women with uncomplicated malaria artemisinin combination therapy is more effective than quinine’.   We can further divide quantitative research broadly into two categories, descriptive or analytical study designs.

A descriptive study design is a systematic “look and see” that does not test hypothesis or answer questions about causal relationships.  It can play an important role in describing trends and generating hypothesis about what might be going on.  Descriptive designs include:

  • Descriptive surveys include those that use questionnaires to describe or provide a “snapshot” of what is going on (for example: how many children attended growth monitoring clinics in 2010?).
  • Case reports (data collected on one subject) or case series (data collected on a few subjects) are useful for describing very rare or unique phenomena (for example, HIV associated cervical pre-cancer and cancer. These approaches can be cheap and relatively quick ways of using already available data. 

 Analytical study designs set out to compare two or more groups. You are probably looking to do an analytic study if your research question includes words like “causes, better than, greater than, less than, more likely than, associated with, correlated with”.  Analytical studies can be divided into two main types: Observational and Experimental.

  • Within observational study designs the research protocol does not determine who is exposed to risk factors, or who received the treatments under investigation. The researchers simply record data. These designs  include: 
    • Cross-sectional designs are where a survey is carried out that goes beyond description to include looking at relationships or associations between variables (for example:  “Is back pain associated with level of exercise?” 
    •  Cohort studies are longitudinal observational studies in which those exposed to a suspected risk factor (for example: cooking over smoky fires) compared to those who have not been exposed. Both groups are followed over time to see whether they develop any new outcomes (in this case, the researchers might be looking for respiratory related disease, for example, onset of asthma).  Cohort studies are useful under those conditions when it would be “unethical” or impossible to deliberately expose participants to a particular risk factor.  They can also be useful for directly measuring the incidence of a disease including measuring multiple outcomes of a single exposure.  However, following people over a long period is expensive and risks losing people that withdraw, move away, or may be difficult to trace.
    • Case-control studies are useful if you are interested in a rare outcome. A case-control study design is where you observe differences between those that have the outcome (cases) and those who do not (controls) to look for associations.  One of the problems associated with case-control study design is recall bias, where those with the disease are better able to focus on  potential risks they have been exposed to whereas, those without the disease have difficulty remembering. 
  • Experimental study design in health services research usually refers to clinical trials, either randomised or non-randomised clinical trials.  These are often referred to as intervention studies because the researchers are doing more than observing participants; the study protocol assigns participants to receive the intervention or control treatment.   Random assignment, for example within a Randomised Controlled Trial, is a way of making the study groups similar to each other, so that the only main difference between your study groups will be whether they receive the intervention or control. Sometimes participants, healthcare workers, and/or the researchers are blinded (or masked) to who is receiving which treatment. This is to ensure that both groups are treated the same, which eliminates reactions by patients and researchers who think the treatment is “special”.  Experimental study designs are expensive to conduct so are usually for mature research questions, often informed by preliminary data collected through smaller “pilot” studies which help to inform process and outcome measures first.

Qualitative Research

Unlike quantitative research, qualitative research uses inductive reasoning that does not set out to prove or test a hypothesis but rather emphasises the importance of meaning and experiences. The units of analysis in qualitative research are words not numbers. It collects data (usually verbal) through interviews, direct observation, or analysis of relevant documentation such as patient diaries or case notes (Table 2).  Qualitative research is useful for gaining a different perspective (e.g. asking the patient what they feel about their treatment rather than assuming we already know), and is particularly useful for gaining contextual information and “thick descriptions” in the patient’s own words.  For example, what the everyday experience feels like for those living with leprosy?

            Sometimes qualitative methods are used with other quantitative research designs, like randomised controlled trials, particularly those involving complex healthcare interventions. This is called ‘mixed methods’ research, and can be highly advantageous as it combines the strengths of both forms of research.  Qualitative data may be collected: before the trial to develop and refine the intervention; during a trial to examine to explore the process of implementation; and after a trial, to explore reasons for the findings of the trial (Aveyard and Sharp 2009).

 

Table 2. Differences between qualitative and quantitative research.

Qualitative research

Quantitative  research
  • Inductive
  • Subjective
  • Discovery of meanings
  • Complex and broad focus
  • Element of analysis: words
  • Word interpretation
  • Uniqueness
  • Flexible/iterative process
  • Develops theories
  • Researcher – subjective, part of the research process
  • Deductive
  • Objective
  • Cause & Effect relationships
  • Concise and Narrow
  • Element of analysis: numbers
  • Statistical analysis
  • Generalisation
  • Protocol driven
  • Test Theories
  • Researcher - objective, outside of the research process

 

5.         Writing your research proposal 

Don’t be tempted to miss out on this stage – even if you are not applying for funding.  The research proposal is an essential step to help you formalise your thoughts.  It provides a plan to help navigate your way through the research journey and to communicate the project to others. Even a great idea is unlikely to attract participants, fellow researchers, funding, or approval, unless there is a clear well written proposal. Different groups and organisations may have to read your proposal, and unfortunately each one may require you to structure the writing in a different format. This will be time consuming for you but it will save your assessors time, remember they may have many proposals to review.

Although the length and complexity of your research proposal will vary depending on the purpose and scope, you need to include the following sections:  background, objectives, methodology, outcomes and dissemination. You also need to include a section on ethical issues, service user involvement and project management.

Background section

 You need to start with an introduction or background section to set the scene.  This section provides an overview of research relevant to your question, what is innovative about what you propose to do, and how will the proposed work add to the current body of evidence.  If you are applying for funding, build your case by: showing that you are up-to-date with what is current in the field and that there is a need to answer the question you are posing; demonstrating your capability and familiarity in the area; and convincing the funders that you are the person to carry out this research which should be funded.

Aims and Objectives section

You should go on then to describe what you intend to achieve by doing this piece of work. Your objectives are the small steps you need to reach in order to achieve your aim. Put your most important problem(s) first, the question(s) you want to answer one at a time and don’t be tempted to answer too many questions.  Your aims and objectives should be SMART i.e. Specific, Measurable, Achievable, Realistic, and Timed. Be realistic, consistent, and link your objectives to your outcomes, methods and timetable. 

Methods section

This is how you describe how you will conduct your research.  Ensure that you are using the appropriate method to answer the research question you are asking.  The methods section needs to be sufficiently detailed and thought through to clearly describe and justify your chosen approach, supported with reference to the relevant literature.  

Ethical issues

You need to be clear about what the ethical issues might be and what steps you will take to minimize harm to those who take part (see section 6 below for some helpful hints to get you started).  Provide details of potential benefits to users and/or the work place. You will also need to consider project management – set out how the project will be managed to include timescales, milestones and communication strategy.  You will also need to consider what criteria you will use to measure progress and document processes on how you will manage crisis situations and conflicts that might arise. Finally consider all the outcomes, outputs and dissemination so you can describe the contribution to knowledge and importance for future research. Highlight how results will be disseminated (publications, conferences, commercial exploitation, websites).

‘Service user’ or ‘patient’ inclusion

 It is becoming increasingly recognised in the medical professions and health sciences that services not only should be shaped by practitioners, but also be informed by the experiences of the people who use these services. ‘Service user inclusion’ or ‘patient and public involvement’ (PPI) are now well-used phrases in research, and meaningful participation is often required by research funding bodies. It is important to stress that involvement  should be meaningful with patients or service users given an active role throughout any research process.  Such involvement  can be mapped against a ‘ladder of participation’ model explained in the Supporting People Best Practice Guide (Service User Involvement). There are a number of organisations that describe ways in which users can be involved with service development or research (National Institute for Health and Clinical Excellence , Governance and Social Development Resource Centre 2006).

Proof reading

Finally, don’t forget to proof read carefully before submitting your proposal to a funding body or before your ethical review panel.  Consider having your proposal read by a consumer of health, colleague, or friend.  Their input can help improve clarity and point out jargon that would not be understood by a wider audience. Check for spelling and grammatical errors – failure to attend to detail can suggest sloppy or careless preparation. 

Proof read carefully to see if you have missed out any words or made any typing mistakes.

6. Obtain ethical approval and gain permission from your hosting organisation

The ethics of the research you conduct is not very different to the ethics of clinical practice.  However, one difference is that research ethics concerns activities additional to patients’ normal treatment and is therefore more rigorous.

What do research ethics involve?

  • It starts with the question ‘Is the research of sufficient importance and value to be conducted?’ In other words, does the question lead to research that will be of benefit to the patient?
  • The next step is to consider the chosen methods. For research to be ethical it should be designed and conducted so that the results are significant enough to lead to potential benefit.   Unscientific, poorly designed research is unethical.  Take time to design your study properly, seek advice from a statistician and experienced researchers where-ever possible.    You should consider potential risks and burden to participants.  Are there any side effects? Will they be involved in excessive hospital visits or filling out multiple, lengthy questionnaires. Obviously ‘risk’ should not be avoided in order to conduct ethical research, but it should lead to potential benefit.  This is known as the ‘risk benefit ratio’ and researchers need to achieve a balance between risk and benefit.
  • The next considerations are similar to clinical practice and concern the processes participants go through with research. These are:
  •  Recruitment to the study.  The main point here is that people are not forced into joining a research study.  They should be approached in an environment and manner where they are free to decide if they would like to participate.
  •  Consent.  There is a variety of methods for formally recording a participants’ willingness to join a research study, with written consent being the most common.  Whatever approach is used to record consent, the key issue is that people agree to participate in a study of their own free will.  They must be fully informed about the research. There should be plenty of time to consider any written or verbal information given including an opportunity to ask questions, before a decision is needed.  Information given to participants should be understandable to them. If the research involves vulnerable  groups, such as children, the information should reflect their level of understanding and consider their needs.  Depending on the age of the child, it includes written information, perhaps using story telling or visual images to help understanding and may be best supported by involving a visit from the research team.
  •  Confidentiality. Treat all information obtained from participants confidentially, including secure storage and the use of pseudonyms.

These are just some of the key ethical issues in research and when working on a new study it is essential to consider local and national guidelines and share your ideas with colleagues and service users as well.  You should also consult the guidance and principles provided by the World Medical Association, including the ‘Declaration of Helsinki’ (2008) and their feely available Medical Ethics Manual (World Medical Association 2011).

7. Project Management  

In addition to detailed planning, a multidisciplinary steering group which includes stakeholders from the public (e.g. local leaders, service users, and professionals), can help to plan activities and oversee the project and give ongoing support.

The research budget should include all the items required, including logistical resources, medical supplies, laboratory supplies, staff costs, patient costs, service user engagement costs, and costs for data management and analysis. Making a timetable or GANTT chart will help the team to plan when the stages of the research will occur and how long they will take, and who is responsible for implementing them. Guidelines to describe what you will do, known as Standard Operating Procedures (SOPs), are necessary for training staff and for ensuring that everyone carries out tasks in the same way, which ensures research quality. All documents related to the research need to be kept in a structured and confidential way, so that the research can be audited – guidelines for this and other good research practices are given in the International Conference of Harmonisation Good Clinical Practice guidelines  (ICH GCP 2011).

The progress of the study must be monitored for timescale and quality of actions. Most health services research involves recruiting participants. A pilot study should test the recruitment process and likely take-up rate.  During your research the recruitment rate should be carefully tracked so the team can adjust in a timely fashion, for example by recruiting from other sites or by setting an upper limit of recruitment at which the study team can comfortably manage the patient load, or by employing new staff.

            The research team needs to monitor if participants are not returning for follow-up visits as well as investigating reasons for non-attendance. If the consent rate drops, the study team will need to discuss with local community leaders what the reasons for this could be.

All study documentation that is used for data collection should be double-checked for likely errors or missing information in a timely way to give the team a chance to correct or complete information before it is entered and analysed. Quality control records, for example laboratory refrigerator temperatures or malaria blood slide duplicate reading results, should be closely monitored to identify and address quality issues before they have an impact on the study’s  validity.

Good communication with all partners and workers is essential to facilitate well planned and executed research. For example, if people are recruited through mobile-clinics, the population has to know in advance when the clinic will arrive, and you need to ensure that no other conflicting community events are occurring.

To identify problems early and recognise innovation and success hold regular team meetings. Remember and abide by reporting requirements to local health officials, external sponsors, ethics committees or academic institutions.  It can be motivating for local communities and their leaders to receive feedback on how the study is progressing.

8. Analyse and interpret the data  

Whatever you collect qualitative or quantitative data, it needs to be analysed before you can draw any conclusions about the relationship of the findings to previous research and the implications for future practice. Both qualitative and quantitative data analysis should be conducted in a rigorous manner and recorded clearly so that others can understand and repeat what has been done.

When analysing quantitative data it is vital to use the correct type of statistical test; this is one that is appropriate both to the research question and to the type of data collected (see Table 3). It is important to establish whether the research question is concerned with relationships (which would be analysed by tests of correlation), with differences between groups (which would be analysed either by tests of difference or by calculating odds or risk ratios), or with prediction (which would be analysed by regression). If possible, it is helpful is helpful to involve a medical statistician from the beginning of the study to ensure that the approach to analysis is clear. For some tests you need specialist software (e.g. SPSS, SAS or STATA), you can calculate others by hand or using free Internet software. It is essential that the output from statistical tests is understood and interpreted correctly. Again, a medical statistician is helpful at this stage. Alternatively, reference to a good medical statistics textbook for example ‘Medical Statistics at a Glance’ (Petrie and Sabin 2009) will highlight important considerations and aid interpretation of results.

Although there are a large number of approaches to the analysis of qualitative data, it usually involves some type of thematic analysis. Those working to a particular philosophical approach may analyse their data slightly differently. It is important to understand how you are going to approach the analysis before commencing; a well written qualitative research textbook will provide further guidance, for example Qualitative Research in Health Care (Pope and Mays 2006). Specialist software can facilitate qualitative data analysis, although it is not essential and it can be done by hand or aided by Word processing software. Whatever method is chosen, the analysis remains reliant on the interpretations of the researcher. Allow plenty of time to complete the analysis as it is a time-consuming and flexible process. Qualitative findings should be presented by theme and should be supported by quotes from participants to allow the reader to understand their experience. Qualitative research does not generate evidence for cause and effect mechanisms, therefore ‘do’ or ‘should’ recommendations for practice are not appropriate. Some qualitative research aims to generate ‘food for thought’ for health care practitioners and others aim to generate theory. It is useful to consider what your aims are before writing any conclusions.

9. Disseminate your findings

Once you have completed your research it is good practice to inform others what you have found, especially where your findings have implications for patient care. This could be locally e.g. managers, colleagues and patients, nationally e.g. policy makers, and/or globally e.g. academics in the field.  Locally you might consider disseminating your findings by word-of-mouth, in a local newsletter, or by using your findings to inform a change in local guidelines. To disseminate your findings nationally and/or internationally you may wish to present your findings at a conference (some organisers run a bursary scheme to help with the cost of attendance) and to write your findings as a research paper, published in a peer-reviewed journal.

Editor’s footnote: The Editorial Board would like to have the comments from our readers concerning the use of the advice in this paper. We would welcome examples.


References

Aveyard, H. and Sharp, P. (2009) A Beginner's Guide to Evidence Based Practice in Health and Social Care 1 edition ed., Open University Press.

British Medical Journal (2011) 'How to Read a Paper', [online], available: http://resources.bmj.com/bmj/readers/how-to-read-a-paper/ [accessed July 2011]

Cochrane Collaboration (2010) 'The Cochrane Library', [online], available: http://www.thecochranelibrary.com/view/0/index.html [accessed July 2011]

Crombie, I. K. and .Davies, H. T. O. (1996) Research in Health Care: Design, Conduct and Interpretation of Health Services Research Wiley.

Governance and Social Development Resource Centre (2006) 'Service Delivery: User Involvement and Accountability', [online], available: http://www.gsdrc.org/go/topic-guides/service-delivery/user-involvement-and-accountability [accessed July 2011]

ICH GCP (2011) 'International Conference on Harmonisation of technical requirements for registration of pharmaceuticals for human use.', [online], available: http://ichgcp.net/ [accessed July 2011]

Lancey, A. (2010) 'Evidence based medicine: searching the medical literature Part 1', Southern Soudan Medical Journal, (1).

Lohr, K. N. and Steinwachs, D. M. (2002) 'Health services research: an evolving definition of the field', Health Serv Res, 37(1), 7-9.

Moja, L. P., Moschetti, I., Nurbhai, M., Compagnoni, A., Liberati, A., Grimshaw, J. M., Chan, A. W., Dickersin, K., Krleza-Jeric, K., Moher, D., Sim, I. and Volmink, J. (2009) 'Compliance of clinical trial registries with the World Health Organization minimum data set: a survey', Trials, 10, 56.

National Institute for Health and Clinical Excellence 'Patient and Public Involvement', [online], available: http://www.nice.org.uk/getinvolved/patientandpublicinvolvement/patient_and_public_involvement.jsp [accessed July 2011]

Petrie, A. and Sabin, C. (2009) Medical Statistics at a Glance, 3rd Edition edition ed., Wiley-Blackwell.

Pope, C. and Mays, N., eds. (2006) Qualitative Research in Health Care, 3rd Edition edition ed., Wiley-Blackwell.

Richardson, W. S., Wilson, M. C., Nishikawa, J. and Hayward, R. S. (1995) 'The well-built clinical question: a key to evidence-based decisions', ACP J Club, 123(3), A12-3.

Service User Involvement 'Service User Involvement: Best Practice Guide', [online], available: http://www.serviceuserinvolvement.co.uk [accessed July 2011]

Solutions for Public Health (2010) 'Critical Appraisal Skills Programme', [online], available: http://www.sph.nhs.uk/what-we-do/public-health-workforce/resources/critical-appraisals-skills-programme [accessed July 2011]

Trials Central 'Trials Central', [online], available: http://www.trialscentral.org/ [accessed July 2011]

World Medical Association (2011) 'World Medical Association', [online], available: http://www.wma.net/en/10home/index.html [accessed July 2011]