Entrant;Name;Date;Prototype ;General;Element 1;Importance category;Importance code ;MO/SR comment 1;Key Concepts;Importance category;Importance code ;SR comment;References Andy;;03-Jul;1.2 (Personal experience lesson plan for teachers);"I have had a look at the materials and they look great. By the way, at KWTRP we are currently preparing for COVID-19 trials in Kenya, and I have been working on ppt animations to explain how vaccine/treatment trials work… They are currently being used to engage the MoH but are very widely applicable… my plan was to make the animation slide templates widely available for modification and use. They would be very applicable for use in schools and it just occurred to me that they may be of some use to you. If you’re interested, I’d be happy to share once we get ethics approval over the next week or so, and the gov communication gag on covid-19 related issues is lifted.";Content;Suggestion;Important constructive finding;;"I had a critical look at all 49 concepts because I felt it would be consistent with the project philosophy to be blinded to the teacher views so as not to be influenced by them (honestly, I couldn’t resist it). I confess that I haven’t checked to see if my views concur with the teachers in reducing down to the 29…. Either way, if it does, all well and good, and if it doesn’t perhaps my views may be helpful in clarifying some of the selected concepts. In my opinion, the ones specified below are either: less important than the others; potentially leading to misconceptions; or conceptually too difficult for primary school students 1.1 b Drop – too much detail for lower secondary school kids 1.2 e Drop – potential for conflicting messages because larger studies often do have greater statistical power 1.2 kPotential for misinterpretation 1.3 aThis is too general to everything 1.3 eThis is not normally taught at secondary level – would require a whole lesson on peer review, and I’m not convinced that its important enough at early secondary level level. 2.1 bTo complex 2.1gThere’s a caveat that this may lead kids to misunderstand the voluntary nature of research and that (though undesirable) participants should be made aware that they can withdraw from participation at any time 2.2 a b cI’m not convinced that this level of understanding about systematic review is necessary or useful to most secondary kids 2.3 b c e f g hThis level is too high 3.3 b c d eThese are less important for students to learn";Suggestion;Constructive finding;"1.1b: Large, dramatic effects 1.2e: Associated with 1.2k: Early is better 1.3a: Existing beliefs may be wrong 1.3e: Peer reivewed 2.1b: Indirect comparisons 2.1g: Lots of misssing people 2.2 a: Unsystematic summary 2.2b: Indirect comparisons 2.2c: Dissimilar care 2.2d: Dissimilar expectations 2.3c: Just words 2.3e: Subgroup analysess 2.3f: No confidence interval 2.3g: Statistically significant 2.3h: No evidence 3.3b: What is your baseline risk 3.3c: How important are advantages and disadvantages 3.3d: How sure are you 3.3e: Are there important uncertainties "; Andy;;22-Jun;1.2 (Personal experience lesson plan for teachers);"Thanks for all you are doing here and for making the working papers available so freely and quickly; gold dust ";Overall;Praise;Positive finding;;;;;; Andy;;26-Jun;1.2 (Personal experience lesson plan for teachers);Congratulations on the innovative work!;Overall;Praise;Positive finding;;;;;; Andy;;27-Jun;1.2 (Personal experience lesson plan for teachers);"I have no further comments on all this since I’m not familiar with the African school contexts. However, it sounds great!";Overall;Praise;Positive finding;;;;;; Andy;;30-Jun;1.2 (Personal experience lesson plan for teachers);Overall, I am very impressed with the project and how it is constructed. I am really grateful to be a little part of it. It is highly interesting, important, and the intervention is clearly research based. Below are some comments/considerations. ;Overall;Praise;Positive finding;;"I think the key concepts that are selected seems reasonable. As for leaving some out, I would consider the two below. The reason for this is that, even though important, I may think they can be too abstract for children? Relative effects of treatments alone can be misleading Fair comparisons of treatments in animals or highly selected groups of people may not be relevant.";Suggestion;Constructive finding;;"Demirel, M., & Dagyar, M. (2016). Effects of problem-based learning on attitude: A meta-analysis study. Hebert, M., Bohaty, J. J., Nelson, J. R., & Brown, J. (2016). The effects of text structure instruction on expository reading comprehension: A meta-analysis. Journal of Educational Psychology, 108(5), 609. Melby-Lervåg, M., & Lervåg, A. (2014). Reading comprehension and its underlying components in second-language learners: A meta-analysis of studies comparing first-and second-language learners. Psychological bulletin, 140(2), 409. Schroeder, N. L., Nesbit, J. C., Anguiano, C. J., & Adesope, O. O. (2018). Studying and constructing concept maps: A meta-analysis. Sternberg, R. J., & Frensch, P. A. (1993). Mechanisms of transfer. In D. K. Detterman & R. J. Sternberg (Eds.), Transfer on trial: Intelligence, cognition, and instruction (pp. 25–38). Westport, CT: Ablex Publishing. Swanborn, M. S., & De Glopper, K. (1999). Incidental word learning while reading: A meta-analysis. Review of educational research, 69(3), 261-285." Andy;;01-Jul;1.2 (Personal experience lesson plan for teachers);I fear that there isn’t much particularly constructive in my feedback at this stage, but I feel that’s a reflection on the quality of the planning and resources.;Overall;Praise;Positive finding;;I think the way that the 29 concepts have been ranked seems very sensible, particularly those which to exclude. My one thought was that ‘Fair comparisons of treatments in animals or highly selected groups of people may not be relevant’ (currently listed as ‘consider including’) could be upgraded (‘probably include’ / ‘include’). When we’ve run sessions in secondary schools, albeit in a UK setting, this has been one of the concepts that seemed to most resonate with the children, particularly when we used a few examples of big health claims (e.g. newspaper headlines claiming that: ‘lavender can treat anxiety’) that had been based on studies of mice.;Suggestion;Constructive finding;"Is ""resonate with children"" already a criteria for choosing concepts?"; Andy;;03-Jul;1.2 (Personal experience lesson plan for teachers);I have looked through the resources - which look excellent overall - and made a few brief comments in response to your questions (see below).;Overall;Praise;Positive finding;;Overall, I agree with the suggestions. In general I would include the ‘probably include’ concepts and not include the ‘consider including’ concepts. The exception would be ‘The people being compared should be cared for similarly apart from the treatments being studied’, which I would consider including.;Suggestion;Constructive finding;; Andy;;05-Jul;1.2 (Personal experience lesson plan for teachers);"Thanks for sending through the learning resources - they're looking good! Please find my feedback below. Don’t hesitate to let me know if I can be of further assistance.";Overall;Praise;Positive finding;;"I generally concur with the proposed rating of the Key Concepts as relevant for students in lower secondary school, with a few exceptions: I note that very few Key Concepts within the “Comparison” section have been labelled to “include” or “probably include”. I would reconsider the ones listed below: • If possible, people should not know which of the treatments being compared they are receiving.1 o I am confused as to why this Key Concept rated so poorly. o I think this key concept should definitely be included in the CHOICE learning resources. It is important to understand 1) why blinding participants to their treatment should be a part of a fair assessment and 2) how this step lowers the risk of introducing confounding variables into the research. • The people being compared should be cared for similarly apart from the treatments being studied. 24 o This is an important aspect of research and students in lower secondary school will be able to recognise that treating groups in different ways is “unfair”. • Relative effects of treatments alone can be misleading. 26 o This concept should definitely be included, because we are asking students to understand the impact of research, but they also need to know which numbers will provide useful information. • Outcomes should be assessed in the same way in all the groups being compared. 25 o Again, I would consider this Key Concept particularly important as a basis of understanding how research attempts to minimise bias and confounding variables. I note all the Key Concepts within “Choices: Expected advantages should outweigh expected disadvantages” section were labelled to “include”. I think including the first two would be sufficient. • Weigh the benefits and savings against the harms and costs of acting or not.8 • Consider the baseline risk or the severity of the symptoms when estimating the size of expected effects.9 • Consider how important each advantage and disadvantage is when weighing the pros and cons.10 • Consider how certain you can be about each advantage and disadvantage. 11";Suggestion;Constructive finding;; Andy;;18-Jul;1.2 (Personal experience lesson plan for teachers);"Thanks for the useful update on the IHC project. I am so sorry for getting back to you in the very last minute. Please find my comments to the IHC CHOICE learning resources and your questions in the attached document. I am (as always) impressed by all the good work you all are doing!";Overall;Praise;Positive finding;;I think lower secondary school students should be able to understand that “the results of one study considered in isolation can be misleading”, introducing the idea of systematic reviews (i.e., it should belong to the ‘include’-category). ;Suggestion;Constructive finding;;1Krathwohl (2002). A revision of Bloom’s taxonomy: an overview. Theory into Practice, 41(1): 212-18: Andy;;11-Aug;1.2 (Personal experience lesson plan for teachers);Truly sorry for the late reply here! The good news is that I've just worked my way through all the attached contents (including all the relevant linked reviews) and have not noticed anything that jumps out at as needing amending or tweaking.;Overall;Praise;Positive finding;;;;;; Andy;;22-Jun;1.2 (Personal experience lesson plan for teachers);"I went through this briefly now and looks really exciting. If it is ok, I would get back to you next week, once I get into this in more detail, as I have loads of things to do by the end of this week. Schools have been closed in Croatia as well, and I understand completely. This has been affecting some of our plans we had with delivering education in primary schools, so we are now considering online teaching as a back-up, in case this situation doesn't settle by autumn. Still, hoping it will and we’ll soon get back to our real normal, not the ""new normal"" everyone is talking about. Will let you know about it all.";Overall;;;context;;;;; Andy;;15-Jul;1.2 (Personal experience lesson plan for teachers);"apologies for not being able to get back to you earlier, but I have some good news, though, I was elected as assistant professor at my University. First, congratulations for your recent achievements. I look forward to seeing the results of the overview, and reading your systematic review on critical thinking. As for the materials that you have sent me, I enjoyed going through all of these. This is both, interesting and very useful. Attached, please find the list of Key concepts you are considering for this project, along with some of my comments. Unfortunately, I had trouble viewing all there is about the lesson prototype. Some pages opened, but couldn’t scroll down, and I don’t see the cartoon. Those that I did manage to go through seemed great to me, but would love to see the whole plan.";Overall, prototype;Problem;Positive finding;"Should be split in two, as top part is Praise. Advisory board not being able to review prototypes is a problem we need to deal with";"Your existing beliefs may be wrong. - This seems like it is already covered within the concept Beliefs alone about how treatments work are not reliable predictors of the presence or size of the effects or the one about Widely used treatments. Also, if all these are “consumed” well by the students, than this one seems logical. Just a thought. However confirmation bias is an interesting thing to consider, not just in medicine Opinions alone are not a reliable basis for claims. - Seems like these are covered already within Beliefs alone. Outcomes should be assessed in the same way in all the groups being compared. - I am a bit worried that this one might be complicated for high-schoolers to follow, just something to perhaps consider when explaining this concept. Syntheses of studies need to be reliable. - Were you considering all three concepts or just some. If so, I would go with this one: Reviews of studies comparing treatments should use systematic methods. Average differences between treatments can be misleading. - This is a great concept, but not sure whether appropriate for high-school. I would be happy if the concept on Relative effects goes well at that age. Evidence should be relevant. - I am thinking about this one also, so please see what you think. Attention should focus on all important effects of treatments, and not surrogate outcomes. The reasons we all are familiar with are outcomes that are either surrogate outcomes or those that cannot be measured objectively as thay even lack a proper definition (for example tiny facial wrinkles). Most oftenly seen are treatments for boosting imunity, but there are so many ODC, other treatments, but everyday products as well for all kinds of „outcomes“ Fair comparisons of treatments in animals or highly selected groups of people may not be relevant. - Yes, definitely. Too often, the media report on findings from in vitro or animal studies as promising for treating people, thus offering false hope and misunderstanding.";Suggestion;Constructive finding;; Andy;;15-Jul;1.2 (Personal experience lesson plan for teachers);"finally had a chance to look at your email regarding ICH CHOICE advisory. As you’ll see below, I had technical problems that severely limited my feedback. wonderful, wonderful work.";Overall, prototype;Problem;Showstopper;"Advisory board not being able to review prototypes is a problem we need to deal with Showstopper in terms of gathering feedback";"I was surprised at the extent to which I was unsure what key concepts were best to communicate to lower secondary school children. So, thoughts below come with a lot of uncertainty, even for myself! My favorites among the probably include are: treatment may not be needed (which I would reframe as problems may resolve by themselves); an outcome may be associated with a treatment but not caused by it (this would be among my top choices over all – very easy to grasp when you tell people they had better stay away from hospitals because so many people die there); and small studies may be misleading. Couple I would hesitate among the certainty include (seem lower priority to me): increasing amount of treat not necessarily increase benefits; be clear about the goal";Suggestion;Constructive finding;; Andy;;16-Jul;1.2 (Personal experience lesson plan for teachers);"You asked for feedback on the ""second question"". I could not figure out which question that was. So here are some thoughts about what I've read so far. Please let me know if I can do anything else now. Follow-up email exchange 17/7: I think we disagree. I thought that many of the ""key concepts"" were misleading. Many were of the form, ""X is not a sufficient basis to conclude Y"". But sometimes X _is_ a sufficient basis, when you have to make a decision based on probabilities, with no certainty, and nothing else is available. And it is true that NOTHING is a sufficient basis in the strict sense of removing doubt. RCTs have many well-known problems. And I also thought that the goal the program seems to be to encourage students to think through health decisions on their own, and to believe that, after a few weeks of a special program, they are capable of doing that. I would go to almost the opposite extreme: teach students that they must ""outsource"" this kind of thinking to experts, and the important thing is to learn how to recognize trustworthy experts. For that purpose, it is important to understand how trustworthy expertise comes about, and how it doesn't come about. This is why I thought that the most important thing to learn is how medical science works, how medical knowledge comes about. In particular, it is an explicit self-corrective process that is largely missing from traditional medicine. The idea that we outsource important parts of our thinking is receiving more attention now, including a recent book by Steven Sloman. (I made this argument too in ""Why teach thinking?"") Although I am addicted to news about politics and government policy, I ultimately form most of my political beliefs by reading trusted sources. I am not competent to judge whether a large government deficit is too large. (I used to worry about it, but Paul Krugman and others have convinced me that we do not need to worry right now.) Or whether a minimum wage of $15/hour will increase unemployment enough to do more harm than good. Likewise, I do not trust myself to make decisions about investment, although I know quite a bit about economics and finance. It occurs to me that a highly relevant paper for this project is this one: Horton, R. (1967). African traditional thought and Western science (pts. 1--2). Africa, 37, 50--71, 155--187. As you see, it is long, but beautifully written, hence a good read. It explains why Western science works in ways that traditional thought does not. (And what is most interesting is that the answer is not obvious.) I suggested that one way to learn how medical science works is to trace the history of a number of important discoveries. After I wrote, I thought of another one that might be particularly relevant in Africa: Ebola. The history of how that epidemic was discovered, understood, and dealt with is a rich illustration of many relevant ideas. This includes the final issues, never quite resolved, of how to test possible vaccines. The same issues are coming up now. I have seen it suggested that some African countries are dealing relatively well with COVID-19, despite being quite poor, because of what they learned from prior experience with Ebola.";Overall;Problem;Showstopper;I tried to read the paper he suggested, but I didn't really understand it well. However I think he raises interesting points - particularly one that helps answer the problem of what can people do when they don't have the ability to access reliable information directly: they can think critically about trustworthy expertise.;"My understanding is that this project is for secondary schools, that means ""high schools"" in the U.S. I thought that many of the key concepts would be nearly meaningless to most American high school students, and I can't believe that African students are so much better prepared. A lot of these concepts are more appropriate to PhD students studying epidemiology and bio-statistics. I do not think that high-school students should come to believe that they can evaluate research studies by thinking about appropriate control groups, sample size, potential biases, and many of the things you list. Most people have to trust expert sources on these matters. So the really important things to teach are how to tell whether experts are trustworthy. I'll come back to that. The second problem is that the list is much too negative. You keep saying that we cannot draw conclusions from one thing or another. This is of course true, but studying this list might lead to the impression that we can't draw conclusions from anything! Yet we must, and do. Moreover, almost all the types of evidence from which we ""cannot draw conclusions"" are at least occasionally relevant. Decisions about medical treatments almost always require the balancing of empirical data, clinical experience (including anecdotes), and understanding of biological mechanisms. I don't think that people without advanced education should come to believe that they are capable of doing this balancing. I think that the best way to teach high-school kids about what makes sources trustworthy is to teach them about the history of medical science, probably in a high-school biology or health class. They can thus learn that science, unlike other paths to knowledge, is self-correcting. Sometimes the best advice that it provides is wrong, but this can be discovered. There are wonderful examples: Pasteur's discovery of the cause of anthrax; Semmelweis's discovery of the cause of ""childbed fever"", and the way to prevent it. (He was somewhat wrong about the cause, but this was corrected later.) The modern invention of vaccines (Maurice Hilleman - who got one of them from his own child). The discovery of the smallpox vaccine and, ultimately, the eradication of smallpox (possible because it does not occur in animals). The current problem of polio eradication. An important point here is that sometimes ""poorly done"" science is good enough, e.g., the effectiveness of smallpox vaccine. In the end, the best guides to medical treatment are statements from people who are educated in this tradition and responsive to it. They won't always agree, but that does not imply that any other sources are better. And some people will pretend to be part of it, namely, those who stand to profit from certain ideas, so it is important to look for false claims to expertise. (But, on the other hand, sometimes people with such self-interest happen to be correct.) Of course, we often trust other sources for minor issues. Jewish mothers recommend chicken soup for colds. It doesn't hurt!";Suggestion;Important constructive finding;could we pull many concepts together through the telling of true stories from history? ie. semmelweis, or how Ebola vaccine was tested?;"I highly recommend two articles: Baron, J. (1993). Why teach thinking? - An essay. (Target article, commentary and reply not included here.) Applied Psychology: An International Review, 42, 191-237. https://www.sas.upenn.edu/~baron/papers/WhyTeachThinking.pdf (A candidate for the best article I've ever written - but long.) Perkins, D. (2019). Learning to reason: The influence of instruction, prompts and scaffolding, metacognitive knowledge, and general intelligence on informal reasoning about everyday social and political issues. Judgment and Decision Making, 14, 624-643. http://journal.sjdm.org/19/190925a/jdm190925a.pdf It occurs to me that a highly relevant paper for this project is this one: Horton, R. (1967). African traditional thought and Western science (pts. 1--2). Africa, 37, 50--71, 155--187. As you see, it is long, but beautifully written, hence a good read. It explains why Western science works in ways that traditional thought does not. (And what is most interesting is that the answer is not obvious.)" Andy;;06-Aug;1.2 (Personal experience lesson plan for teachers);"Firstly I must apologise for my late response. I hope this can still be useful to you. From my limited experience of working with teachers in this area, many teachers do not have detailed knowledge of informed health choices. Important barriers for teachers also include the time available in the curriculum. Teachers will need teaching about the content of KCs. I see that you've preparated materials for this but I wasn't clear whether it would entail formal instruction or a separate learning resource for them. I feel that for most school teachers this would be essential. Perhaps a podcast that explained the details of the KCs in more depth. We should also be mindful of the general ""fake news"" climate that abounds. This is an opportunity, but also a potential complication. The IHC resources just deal with health claims, but could potentially be badged as an intervention that addresses broader social issues around critiquing claims and evaluating sources of information. (Although they don't help us as much with other sorts of false claims such as politics, I note that you have that in hand with the That's a Claim suite!) From informal chats with various people (science, social science and history teachers, folks and organisations who are into the public understanding of science game), I think there's an open door to be pushed at. There's a demand, but a lack of detailed knowledge or guidance on what to teach and how to teach it in specific settings. Enough havering, here's the feedback you actually asked for.";Overall, teacher training;Suggestion;Important constructive finding;I like the idea of a podcast for teachers for teacher training. Maybe we could re-use some of the material from the first project, or some of Vanessa's material.;"I tried to envisage what is the minimum set I would want teachers to have a firm grasp of before embarking on this. These could be built out, bringing in more details where appropriate for the audience. This broke down into three ""objectives"" 1.Establishing the problem: why we need claims that are backed up by fair tests of treatments, and what happens when they aren't 2.Describing the solution: what makes a fair test 3.Making a decision: how to decide whether claims are reliable and applicable. Objective 1: Why we need fair tests of treatments •Treatments can harm (1.1) oMore treatment is not always better (1.2j) oEarlier treatment is not always better (1.2k) oTreatment may not be needed (1.2a) •Identifying effects of treatments depends on fair comparisons oAn outcome may be associated with a treatment but not caused by it (1.2d) oBeliefs and (expert) opinions about the effects of treatments can be wrong (1.2b) oCompeting interests may result in misleading claims (1.3b) oAnecdotes are unreliable evidence (1.3d) •People have been, and continue to be, harmed by failure to conduct fair tests of treatments before implementing them Objective 2: What are fair tests of treatments? •Clear statement of the problem, intervention and outcome we are interested in oThe problem, patient or population group should be similar to the people we are thinking about oThe intervention should be clearly described oThe outcomes should be important to us and measured objectively •The study should use a prospective design oThe protocol should be specified in advance and any changes documented oThe sample size should be sufficient to detect a meaningful difference (2.3d?) •The comparison groups should be similar at the start of the study (2.1a) •The comparison groups should treated similarly throughout, apart from the treatment given (2.1c) •Enough people should be followed up (2.1g) •The results should be clearly stated (2.3) oThey should be the outcomes that were pre-specified in the protocol oAverages can be misleading (2.3c) oRelative measures of effects can be misleading (2.3b) oThere should be a statement of uncertainty (confidence interval) around all the results (2.3f) Objective 3: Making Informed Health Choices •What is the study actually telling is about our clearly focused question? (3.1a) oAre you so different from the population that the results can't help you? (3.2d) oIs the intervention practical in your setting? (3.2c) oDo the outcomes matter to you? (3.2a) •Are the benefits worth the harms and costs? (3.3c) •How sensitive is your decision to the uncertainty around the results? (3.3d) My one comment, reflecting on the KCs after a year or so ""away"" from them, is that the PICO* / clearly focused question element is perhaps underplayed. [* Other formulations are available, I'm just old fashioned and always use PICO] In all the EBM teaching I've done (admittedly always at undergraduate or postgraduate level), PICO has always featured. As a way of thinking about evidence, I believe it is the single most useful tool in the EBP box. So, for example, one can present pretty much any health claim, from any source, and start breaking it down into: •Who is this claim about? •What is being done to them? •What is the effect? The components of the PICO are present in the KCs (e.g. 3.1), but does this need to come higher up, e.g. at the start of section 2, as an item on its own? I think this could also help embed a ""research mindset"" in the learning task, in that by formulating the question, the learner ""discovers"" some of the challenges that researchers face in creating fair tests. This could lead logically on (in a research mindset) to specifying a protocol to address the question prospectively.";Suggestion;Important constructive finding;; Andy;;30-Jun;1.2 (Personal experience lesson plan for teachers);"Tusen takk for tilsendt utkast! Det er veldig spennende å lese dette. Jeg synes det ser veldig bra ut og det er fint med den strukturerte oppbygningen. Jeg har noen små kommentarer som jeg har listet under. ";Structure;Praise;Positive finding;;I believe all the blue and green key consepts are important. However, it may take some time to ensure the pupils actually learn and understand all these concepts. It is therefore important that the learning resources and teachers use enough to ensure deep learning.;Suggestion;Constructive finding;; Andy;;07-Jul;1.2 (Personal experience lesson plan for teachers);"I have looked at the learning resources and commented on each of the four questions from my point of view. Hope they can be of some use to you. First of all, I am impressed of how you present the content in a clear and easy-to-follow manner. I also like how you organize the concepts in main groups and subgroups, it gives an immediate understanding of the focus areas. Here is my feedback on the questions:";Structure;Praise;Positive finding;;"I agree to how the concepts are “rated” and categorized in what to include and exclude. My only suggestion is to move one of the concepts under Claims, “Widely used treatments or those that have been used for decades are not necessarily beneficial or safe”, now categorized as Consider including (yellow #23), to Include or Probably include. This is because I believe students (and others) often come across statements like “I, and many others, have always used/done this, and it always works” from family and in their everyday lives. I see the resemblance to the concept “Personal experiences or anecdotes alone are an unreliable basis for most claims”, but it is not quite the same. If you consider downsizing the number of concepts, I suggest that the concept “The results of one study considered in isolation can be misleading”, under Claims (green #15), may be categorized as Consider including or Exclude, as this is somewhat covered under Comparison. ";Suggestion;Constructive finding;; Andy;;06-Jul;1.2 (Personal experience lesson plan for teachers);;;;;;;;;; Andy;;08-Jul;1.2 (Personal experience lesson plan for teachers);Please find attached my feedback in relation to the IHC CHOICE - if you need different / more please do let me know. ;;;;;"Key Concepts being considered for inclusion in CHOICE learning resources •Number 4. BLUE: Competing interests may result in misleading claims oFeedback: I didn’t think that this would feature as a Blue •Number 15 GREEN: The results of one study considered in isolation can be misleading oFeedback: I thought this would feature in the BLUE listing •Number 25, YELLOW: Outcomes should be assessed in the same way in all the groups being compared oFeedback: I thought that this would be a BLUE. •Numbers 8,9,10,11 BLUE oFeedback: These are complex concepts but I understand why they are appropriate and should be included in the CHOICE learning resources. IHC Key Concepts oComparison: 2.1 (h) People’s outcomes should be counted in the group to which they were allocated. oFeedback: I thought that this would be included in the CHOICE learning resources oI was also surprised that none of the concepts from 2.2 Syntheses of studies need to be reliable were included in the CHOICE learning resources.";Suggestion;Constructive finding;; Andy;;10-Jul;1.2 (Personal experience lesson plan for teachers);I’ve had a look at the prototype lesson and the evidence summary and first impressions are below.;;;;;;;;; Andy;;14-Jul;1.2 (Personal experience lesson plan for teachers);I have firstly looked at the lesson plan because this is my area of expertise. I have attached my feedback and am very happy to discuss further if that would help.;;;;;;;;; Andy;;18-Jul;1.2 (Personal experience lesson plan for teachers);in attachment you'll find our feedback about the IHC CHOICE project.;;;;;"Assuming that children from lower secondary schools may not have been taught the Children’s Book lessons, it would be helpful to highlight some of those concepts, starting from the five bad bases that make health claims unreliable. We both agree on all of the 11 key concept included. Among the 7 key concepts being considered for inclusion, we would include: •“Large dramatic effect are rare”: it seems to us a necessary concept because it is useful and preparatory to understand why we need to make many comparisons of treatments. Very few medical practices have large treatment effect, like, e.g. a parachute effect. More often, treatment effects in contemporary medicine are marginal and we need fair comparisons for these effects to be detected. (We used the parachute example to convey this concept to primary school children). •“It is rarely possible to be certain about the effects of treatments”: this makes up a concept of the first Children Book lesson (Chapter 1, page 11: “We cannot be completely sure what the effects of most treatments will be when we use them.”) and underlines the existence of uncertainty about the individual effects of treatments. •“Widely used treatment or those who have been used for decades are not necessarily beneficial or safe”: we would include it, since it is part of the five bad bases for health claims. We would also include the remaining 4 Key concepts being considered for inclusion. Among the excluded Key concepts, we would consider recovering the concept “if possible, people should not know which of the treatments being compared they are receiving”. The concept of blindness in fair comparisons is also present in the Children Book (chapter 6) and, based on our little pilot experience, children immediately became familiar with this concept. Moreover, blindness introduces to other important topics, such as the placebo effect. Finally, blindness is an important requirement for randomization.";;;;