"QuizID"	"AppID"	"AppTitle"	"TaskID"	"TaskText"	"RecordID"	"Record"	"Type"	"Note"
"DSAIRM_fitmodelcomparison"	"fitmodelcomparison"	"Model Comparison"	1	"Take a look at the inputs and outputs for the app. It is similar to the previous fitting apps. To keep things simple, the confidece interval computation part is not present. Each model has 3 parameters which are being estimated (_a_, _b_ and and _r_ for model 1 and _a_, _b_ and _dX_ for model 2). You might wonder why the other parameters are fixed, especially since in previous examples we fitted parameters such as _p_ and _dV_. The only reason here is to keep things simple, and given the small amount of data, it is not possible to estimate more parameters. For a real problem, you would need good outside scientific knowledge to justify fixing some parameters (e.g., those were measured fairly precisely in separate experiments). Otherwise, you would need to fit those parameters, or, if you don't have enough data (i.e., you might be overfitting), you'll have to simplify your model.

For this example, we'll just assume we know the other parameters. Run model 1 for 1 iteration with the default settings (you might want to plot the result with a log y-axis). You should get an SSR of 5.8 and an AICc of 10.7. Now switch to model 2, also run for 1 step. Both SSR and AICc are a bit lower. Those starting values don't really mean anything, it's just an initial check that model and data are reasonably close and we can start fitting. Now run model 1 for 500 iterations. AICc should have dropped to around -7.08. Do the same 500 iterations for model 2. You'll find it fits better, with a lower AICc (remember, the actual value of the AICc doesn't matter, all you can do is compare differences in AICc between models fit to the same data, and lower is better). If your computer allows, you can run more iterations for each model to check that these results hold.

In this example, the model with the lower AICc also have the lower SSR. Why is that so? Is that always true for any 2 models? (It's not: The 2 models here are in some sense special, understand how/why.) If you have a hard time figuring this out, I suggest you take the AICc equation and compute it manually for both models by inserting SSR and K (number of parameters) and data points (here N = 8). "	"T1R1"	"AICc for model 2, 500 interation (round to 2 digits)."	"Rounded_Numeric"	"Round to two significant digits"
"DSAIRM_fitmodelcomparison"	"fitmodelcomparison"	"Model Comparison"	1	"Take a look at the inputs and outputs for the app. It is similar to the previous fitting apps. To keep things simple, the confidece interval computation part is not present. Each model has 3 parameters which are being estimated (_a_, _b_ and and _r_ for model 1 and _a_, _b_ and _dX_ for model 2). You might wonder why the other parameters are fixed, especially since in previous examples we fitted parameters such as _p_ and _dV_. The only reason here is to keep things simple, and given the small amount of data, it is not possible to estimate more parameters. For a real problem, you would need good outside scientific knowledge to justify fixing some parameters (e.g., those were measured fairly precisely in separate experiments). Otherwise, you would need to fit those parameters, or, if you don't have enough data (i.e., you might be overfitting), you'll have to simplify your model.

For this example, we'll just assume we know the other parameters. Run model 1 for 1 iteration with the default settings (you might want to plot the result with a log y-axis). You should get an SSR of 5.8 and an AICc of 10.7. Now switch to model 2, also run for 1 step. Both SSR and AICc are a bit lower. Those starting values don't really mean anything, it's just an initial check that model and data are reasonably close and we can start fitting. Now run model 1 for 500 iterations. AICc should have dropped to around -7.08. Do the same 500 iterations for model 2. You'll find it fits better, with a lower AICc (remember, the actual value of the AICc doesn't matter, all you can do is compare differences in AICc between models fit to the same data, and lower is better). If your computer allows, you can run more iterations for each model to check that these results hold.

In this example, the model with the lower AICc also have the lower SSR. Why is that so? Is that always true for any 2 models? (It's not: The 2 models here are in some sense special, understand how/why.) If you have a hard time figuring this out, I suggest you take the AICc equation and compute it manually for both models by inserting SSR and K (number of parameters) and data points (here N = 8). "	"T1R2"	"TRUE or FALSE: AICc and SSR agree in ranking different models if the models have the same number of parameters."	"Logical"	"Report True/False"
"DSAIRM_fitmodelcomparison"	"fitmodelcomparison"	"Model Comparison"	2	"Based on what we just did and found, we would conclude that model 2 better matches the data, and thus B-cells are more likely to important in this system compared to T-cells. Of course the major caveat is: **Given all other assumptions**. In this case, the assumptions are both how we build the models, and which parameters we assumed to be fixed and the values we gave them. We can't easily explore different model implementations here, but we can explore how other choices for the fixed model parameters might impact our conclusions. Reset all inputs, then set dI=1, dV=8. We are basically now assuming that infected cells live twice as long (24h instead of 12h) and virus is cleared twice as fast. The choices for fixed parameters need to come from the literature, but there is usually some range of reasonable values. Run a single iteration for model 1, you should find AICc = 18.95. Now run it for 500 iterations. The AICc will have gone down (though is still positive). Then run model 2 for 500 iterations. You'll find that again, model 2 is better, though not surprisingly, the best fit estimates for the parameters are different than those found in the previous task. 

You can increase the number of iterations or the starting values for the parameters. I think (but have not exhaustively tested) that model 2 will continue to be able to produce a better fit compared to model 1. This suggests that model 2 might be robustly better than model 1. However, it could be that there are some settings for the fixed parameters for which model 1 performs better."	"T2R1"	"AICc for model 1, 500 interations (round to 2 digits)."	"Rounded_Numeric"	"Round to two significant digits"
"DSAIRM_fitmodelcomparison"	"fitmodelcomparison"	"Model Comparison"	2	"Based on what we just did and found, we would conclude that model 2 better matches the data, and thus B-cells are more likely to important in this system compared to T-cells. Of course the major caveat is: **Given all other assumptions**. In this case, the assumptions are both how we build the models, and which parameters we assumed to be fixed and the values we gave them. We can't easily explore different model implementations here, but we can explore how other choices for the fixed model parameters might impact our conclusions. Reset all inputs, then set dI=1, dV=8. We are basically now assuming that infected cells live twice as long (24h instead of 12h) and virus is cleared twice as fast. The choices for fixed parameters need to come from the literature, but there is usually some range of reasonable values. Run a single iteration for model 1, you should find AICc = 18.95. Now run it for 500 iterations. The AICc will have gone down (though is still positive). Then run model 2 for 500 iterations. You'll find that again, model 2 is better, though not surprisingly, the best fit estimates for the parameters are different than those found in the previous task. 

You can increase the number of iterations or the starting values for the parameters. I think (but have not exhaustively tested) that model 2 will continue to be able to produce a better fit compared to model 1. This suggests that model 2 might be robustly better than model 1. However, it could be that there are some settings for the fixed parameters for which model 1 performs better."	"T2R2"	"AICc for model 2, 500 interations (round to 2 digits)."	"Rounded_Numeric"	"Round to two significant digits"
"DSAIRM_fitmodelcomparison"	"fitmodelcomparison"	"Model Comparison"	3	"Keep exploring. For instance change the bounds for the parameters. As you learned, this shouldn't affect the best fit (as long as the best fit estimate is not at one of the bounds), but you'll see that it might still do so. In practice, you would need to try multiple solvers with multiple starting values to determine which model is better. You could also explore if the model ranking remains if you change assumptions for your non-fitted parameters. You could for instance systematically vary them over ranges, or sample them - using approaches you saw in previous apps. The amount of exploration you do is some kind of cost-benefit trade-off. You could always do more, but at some point you have to decide it's enough to satisfy you (and the reviewers and your audience).

The overall point to take away from this app is that while model comparison can be informative and useful, you always need to keep in mind that it relies on other assumptions you made about model type and structure, and choices of fixed parameter values (if you fixed any)."	"T3R1"	"Nothing"	"None"	""
