Experimental Methods in Computer Science

Recommended Prerequisites

Statistics.

Teaching Methods

The classes contain theoretical exposition about experimental methodologies and their application in the context of computer science in terms of case studies. The assessment consists of designing and analysing experiments from computational systems under several scenarios.

Learning Outcomes

Acquire the knowledge and the competencies on empirical assessment of non-functional attributes (e.g., performance, availability, security, etc.) of computer systems and software considering the needs of the different areas of computer science and informatics engineering. Acquire competencies in analysis and synthesis, design, organization and planning of experiments in informatics engineering.

Work Placement(s)

No

Syllabus

- Introduction: experimental studies in engineering and science

- Data analysis and exploratory data analysis

- Overview of experiment design

- Sampling and data distributions

- Measurements, uncertanties, variability, and confidence intervals

- Contingency tables, measures of association, and correlation

- Hypothesis testing

    – Parametric methods

    – Nonparametric methods

- Linear regression, data transformations

- Simulation experiments

    – Simulation models

    – Simulation languages

    – Pseudorandom number generation

    – Model calibration and validation

    – Design of simulation experiments and analysis of results

- Experiments with people.

Head Lecturer(s)

Luís Filipe dos Santos Coelho Paquete

Assessment Methods

Assessment
Exam: 45.0%
Project: 55.0%

Bibliography

1) T. Bartz-Beielstein, M. Chiarandini, L. Paquete, M. Preuss, Experimental Methods for the Analysis of Optimization Algorithms, Springer, 2010.

2) Natalia Juristo and Ana M. Moreno, Basics of Software Engineering Experimentation, Springer Publishing Company, 2010

3) P. Cohen, Empirical Methods for Artificial Intelligence, MIT Press, 1995

4) R. Jain, The Art of Computer Systems Performance Analysis, Wiley 1991.

5) D.J. Lilja, Measuring Computer Performance, Cambridge University Press, 2000

6) J. Lazar, J. Feng, H. Hochheiser, Research Methods in Human-Computer Interaction, (Chapter11 - Analyzing qualitative data), John Wiley and Sons, 2010.

7) C.C. McGeoch, A Guide to Experimental Algorithmics, Cambrigde University Press, 2012.