-
Notifications
You must be signed in to change notification settings - Fork 0
/
Copy pathchapter4.tex
138 lines (94 loc) · 16.6 KB
/
chapter4.tex
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
72
73
74
75
76
77
78
79
80
81
82
83
84
85
86
87
88
89
90
91
92
93
94
95
96
97
98
99
100
101
102
103
104
105
106
107
108
109
110
111
112
113
114
115
116
117
118
119
120
121
122
123
124
125
126
127
128
129
130
131
132
133
134
135
136
137
138
%===================================== CHAP 4 =================================
\chapter{Research Design and Methodology}
Over the course of this project, both quantitative and qualitative data gathering and analysis methods were employed. Both of the methods offered something valuable to the project, and using both allowed for gathering the data best suited for this master thesis. In the early stages quantitative data, such as questionnaires, were used to gather opinions and thoughts from potential users of the program. Using these surveys, a base could be created to build from, and develop a better prototype. With this prototype finished, more qualitative measures were used to gather more specific and more detailed data. Figure \ref{fig:researchMethod} provided an overview of the research process applied in this thesis, from the strategy, data generation methods and data analysis methods.
\begin{figure}[!ht]
\centering
\includegraphics[width=.8\textwidth]{./fig/researchMethodology/OatesResearch.png}
\captionsetup{width=0.8\linewidth}
\caption{Model of the research process adapted from Oates \cite{oates2005researching}. The red outlined boxes are methods used in this thesis.}
\label{fig:researchMethod}
\end{figure}
\section{Design and Creation Research}
\label{sec:designCreationResearch}
One of the more common research methodologies in computer science, Design and Creation Research uses common development techniques and adapts them for research and is the research methodology that will be used for this thesis. It focuses on development of new IT products, also called artefacts \cite{oates2005researching}. In the case of this paper, the artefact being developed is a prototype aiming to investigate whether or not collaboration is conducive to career guidance in virtual reality. This type of research can be split into a few different types, mostly depending on whether the artefact itself is the end goal, a vehicle for research or if the focus is on the development process itself. The most relevant case for this project is the second case. To expand upon this slightly, to say that the artefact is a vehicle for the research means that while the artefact itself is important, its \textit{usage} in real life is what's really important. By developing something that can be used in real life, it is possible to point to something concrete and measure the perceived effects the artefact has on the system it is introduced to.
While the artefact in this project is a vehicle for the research, the aim is to have one of the pre-existing workplace applications working with collaborative elements at close to 100\% functionality, so that others may continue developing the application later with relative ease. The most important part is still to see whether or not the end product can heighten the efficacy of the VR career guidance project.
\section{Development methodology}
The primary type of development used in the project can best be described as a variant of agile software development. Agile development refers to certain principles that were written down in the 2001 article \textit{Manifesto for Agile Software Development} \cite{beck2001manifesto}. This manifesto decries the old method of rigid development that is very resistant to change. Over the course of this project, feedback is sought at every level, hoping to improve the software both for the customer and the users, which means that it must be open to change when it needs to.
There will be multiple iterations and phases of the project, each building upon the last. Three major phases were roughly planned out at the beginning of the project, with each containing multiple iterations as priorities shift based on feedback. These three phases are discussed in more detail in chapters \ref{chap:phase1}, \ref{chap:phase2} and \ref{chap:phase3}, respectively. Figure \ref{fig:devMethod} outlines the adapted develop methodology.
\begin{figure}[H]
\centering
\includegraphics[width=.9\textwidth]{fig/researchMethodology/devMethod.jpg}
\captionsetup{width=0.9\linewidth}
\caption{The development methodology used in this thesis.}
\label{fig:devMethod}
\end{figure}
\section{Methods to Answer Research Questions}
The primary goal of the paper is to answer the research questions as formulated by us based on the the previous research. To do that, it is important that there is a clear and structured way to answer them. As the matters they pertain to vary to some degree, so too will the ways to answer them.
First and foremost is the primary research question: \textit{"How does collaboration in virtual reality workplaces contribute to the career guidance of young job seekers?"}. This RQ, along with secondary RQ1 and RQ2, will be answered mostly through traditional data gathering and analysis methods, i.e., surveys, user tests and expert tests. They are multi-faceted questions that need to be looked at from multiple angles, and as such, need multiple runs of data gathering to properly answer. Primarily, this will include user testing and interviews, with a side of surveys and focus groups.
Secondary RQ3 and RQ4, are more technical questions, and will be answered with the information gleaned from the design and creation method. Users are less involved with this, as it pertains more to the process of development and the interests of the IMTEL group as well as NAV. The process of development, needs and requirement will therefore be well documented to enable a comprehensive report as to the conversion process from single user experience to multi user experience.
\section{User Testing}
User tests are an important part of agile development. They are one of the primary means of gathering relevant feedback from the people who will actually be using your product. In essence, a user test is way to test the opinions of users who have had a chance to try the artefact being created. By employing user tests, it is possible to glean more unbiased information from users that do not have knowledge of the artefact or preconceived notions of its quality. A common pitfall when evaluating systems is letting your own knowledge of quirks and the intent, rather than the performance, of a functionality cloud your evaluation. With user testing, you can figure out what a mechanic looks like to those the artefact is intended for, gather data and make the proper adjustments to increase the value of the system.
A lot of care goes into creating the tests and making sure that the data extracted from them is sound and valid. The tests needs to specific enough to showcase or test the aspect you are focusing on, but not so specific that it becomes an unnatural situation that does not represent the actual product.
Most of the tests performed with the target audience have been done in collaboration with NAV. With their help, testers in the target group were gathered, and they could then try out the application collaboratively. The experience would be supervised by us, ready to guide or help if they struggled with the basic VR interactions, as the testers had quite varying levels of familiarity with VR. Once they had finished the tasks that were available to them they were asked questions through either a semi-structured interview or a questionnaire. User testing was planned to happen after each iteration as part of the iterative software development process. Data material gathered from the first two phases was used to iterate on the artefact, while the last user testing phase was used for the final evaluation.
This master thesis is approved by the Norwegian Centre for Research Data (NSD). See appendix \ref{appendix:nsd} for the consent form given to user before testing.
\subsection{The Surveys}
The purpose of a survey is to obtain the same data from a large amount of people in a systematic way, and then analyse the data, looking for patterns to generalise to the larger population\cite{oates2005researching}. People often associate surveys with questionnaires, but a survey can consist of many other forms of data gathering. This project has opted to use several of these to be able adapt to the multiple situations.
For surveys, it is important to reach a minimum desired amount of respondents to ensure that you have a large enough sample size. In those situations where we opt to use quantitative data, steps should be taken to increase the response rate where possible. Since the plan is to conduct most of the primary data gathering in person, the response rate will hopefully be quite high. In the situations where we can not gather data in person, questionnaires may be employed instead. For those cases, it is possible to emphasise the importance of the project and thoroughly explain what we hope to achieve in an attempt to appeal to the responder's solidarity and curiosity.
One of the most important parts of research is gathering the data. Often a choice must be made to go for either a smaller quantity of high quality data, i.e., qualitative data, or a larger selection of data points at the cost of the quality, i.e., quantitative data. There are merits to using both, and they will each suit different situations. As such, both quantitative and qualitative data gathering will be used as best suited for the situation.
For this thesis, self-efficacy has been an important measure of the impact collaborative work has on career guidance in VR. According to Bandura et al. self-efficacy refers to one's belief in their ability to execute a task, or more specifically, "Perceived self-efficacy refers to the beliefs in one's capabilities to organise and execute the courses of action required to produce given attainments" \cite{bandura1999self}. Self-efficacy greatly impacts motivation and is important in one's choice of behaviours, including occupational or social behaviours \cite{bandura1999self}.
\subsubsection{Quantitative Data}
When shaping the surveys to gather quantitative data, it is necessary to shape the questions accordingly. The way you allow the respondent to answer will also shape your data. If the survey consists mostly of open questions, where the respondent can answer what they want to you may gain a more detailed answer, but the respondent may also be less likely to answer it at all, as you have raised the barrier for answering \cite{oates2005researching}. Narrow down the possible answers too much, and the respondent may not find an answer they agree with. One of the more common formats for answers called is the \textit{Likert scale}, after psychologist Rensis Likert \cite{likert1932technique}. It is designed to capture the degree of agreement or disagreement of the respondent, and strikes a middle ground between a simple yes/no question and an open one. Likert items are employed heavily in our surveys.
\subsubsection{Qualitative Data}
When quantitative data do not quite cover what you need, qualitative data can instead be of use. Sacrificing the possibility of a larger selection of feedback, one can then dig deeper for the answers you seek, as well as open up the possibility for a dialogue with the respondents, in the case of a focus group or similar situations. To take proper advantage of the qualitative data gathered, some work is required to make it usable. This can take on different forms, but a common version is thematic analysis, where responses are checked for recurring keywords and themes, and categorised accordingly \cite{oates2005researching}. Thematic analysis is one of the main tools that will be used in this paper due to the large amount of variance between responses that qualitative data can produce. Coupled with a significant amount of planned tests and the general size of a transcribed interview, qualitative data can be overwhelming if proper organisation and analysis methods are not used.
\subsubsection{Exploratory Work}
In the beginning phases of the project, it was deemed important to get an overview of the general attitude of the target audience when it came to VR. As such, plans were made to conduct quantitative data surveys of different parts of the target audience. This included both young job seekers, as well as NAV employees. For the first group, the most significant knowledge to be gained was what young job seekers felt was helpful and what was not when it came to VR. The previously developed applications at the IMTEL lab suited that purpose excellently and could be used to convey the possibilities as well as the limitations of VR, and hopefully evoke useful feedback.
For the second group, NAV employees, it was important to figure out how they usually worked with the job seekers. What did their workflow look like before, and how could the artefact of this project fit into that?
\subsection{Expert Evaluations}
Since most of the work of this project was done at the IMTEL lab, opportunities presented themselves for us to gather feedback and advice. Throughout the year, people from multiple fields and disciplines stopped by the lab for various reasons, and most were quite open to some discussion about the project. The multidisciplinary nature of the project meant that even if they were not necessarily well versed in every aspect, they still had some valuable input for us to consider.
Using the contacts of the lab managers, it was also possible to arrange larger scale information gathering sessions, where the application was demonstrated to people working in the field of career guidance and lifelong learning. Following the demonstration, a discussion was held, questions from the audience were fielded, and the participants were asked to answer a survey regarding the application.
By doing this, we intended to gather many different points of view in an effort to ensure that the artefact was interesting and valuable to use for all users.
\section{Covid-19 Ramifications}
While the methods described in this chapter will still be used, their usage and our ability to gather data has been impacted by the outbreak of Covid-19 \cite{FhiCorona}. The primary effects of this is felt in the data gathering work. As of the 13th of March 2019, Norway was under a shelter-in-place directive in an attempt to minimise the spread of the virus. This meant that qualitative data could not be gathered in the amounts we had hoped to. Quantitative data is used more than originally planned, specifically in phase three. In that phase, the plan was to conduct several test days at the IMTEL lab with visitors from NAV using our equipment to gather large amounts of qualitative data. With shelter-in-place in effect this became impossible, and alternative means of conducting tests had to be found. Primarily, this took the form of video conferences coupled with remote usage of the developed software system. The specific changes to each phase will be discussed in detail as it becomes pertinent.
\section{Work distribution}
While working on the project, we made sure to play to the strengths and weaknesses of having two authors of the paper. With our different backgrounds within IT, as much of the work as possible was to be handled by the person most suited for it. The combined expertise covers advanced software architecture, networking, graphics, UX and more. We were therefore well equipped to handle almost every part of the project. Table \ref{table:workDistribution} shows roughly how the work was distributed. Work was done both individually and together (e.g. pair-programming).
\begin{table}[H]
\centering
\begin{tabular}{l|l}
\toprule
\textbf{Author} & \textbf{Task} \\
\midrule
Both & Surveys, interviews and tests \\
Both & PUN2 initialising \\
S. Ulvestad & Network architecture \\
C. Røkke & Prototype game design \\
Both & PUN2 implementation \\
C. Røkke & Interaction design \\
S. Ulvestad & Network optimisation \\
S. Ulvestad & Lobby networking \\
C. Røkke & Game and lobby UX \\
\bottomrule
\end{tabular}
\caption{Distribution of work.}
\label{table:workDistribution}
\end{table}
\section{Data Gathering Schedule}
Table \ref{table:dataGatheringSchedule} shows the data gathering schedule of all dates where tests where performed and what methods was used.
\begin{table}[H]
\centering
\begin{tabular}{l|l|l}
\toprule
{\textbf{Date}} & { \textbf{Place}} & { \textbf{Method}} \\
\midrule
27.09.2019 & NTNU Gløshaugen & Questionnaire \\
04.11.2019 & NTNU Dragvoll & Expert feedback \\
25.11.2019 & NAV Falkenborg & Interviews \\
09.03.2020 & NTNU Dragvoll & Group discussion / questionnaires\\
21.04.2020 & Video conference & Questionnaire \\
24.04.2020 & Video conference & Questionnaire \\
14-24.05.2020 & Video conference & Interviews \\
\bottomrule
\end{tabular}
\caption{Data gathering schedule.}
\label{table:dataGatheringSchedule}
\end{table}
\cleardoublepage