chapter9.tex

%===================================== CHAP 9 =================================

\chapter{Discussion}
\label{chap:discussion}
The outcome of the Design and Creation research strategy used for this thesis is a combination of the working system, methods, models and constructs. Once combined, these IT artefacts can offer knowledge to the research field \cite{oates2005researching}. In this chapter we will discuss the research question presented in chapter \ref{chapter:1} in relation to the artefacts and the analysis results from the final phase. Limitations of the study is also discussed.



\section{Discussion}
The following sections explain and evaluate the results of the analysis done for the final phase of this thesis in relation to the developed artefacts and research questions.

\subsection{The artefact} 
Table \ref{table:comparisonOurApp} lists and compares features from the related work applications presented in chapter \ref{chap:relatedWork} and the features which were implemented in our artefact supporting a collaborative VR environment aimed at facilitating career guidance. A "\ON" symbol indicates the feature is present, while a "\LIM" symbol indicates it is present, but in a limited fashion.

As evident by the table, the artefact supports all the features we identified in the preliminary research study to be important in hopes of answering the research questions set. Although all features provides value to the artefact, some turned out to be more advantageous and useful than others. The ability for remote usage was especially highlighted as a consequence of Covid-19 and stands to be a solution for similar events. While several of the related applications contained features that were useful, they did not have that specific combination that was wanted for this project. The artefact developed for this project contains a fairly unique mix of purpose and features, where multiplayer collaborative features are used to create a different form of career guidance for a different group of people. 

Section \ref{section:electroVR} discusses ElectroVR, a collaborative learning tool in VR that is made to explore magnetism and electricity simulations. It uses co-located VR compared to our remote solution, and is such lacking key features such as voice chat. The other papers in chapter \ref{chap:relatedWork} also have interesting key points, such as \textit{CoVAR}, seen in section \ref{sec:coVAR}, and its focus on interaction and gestures. There are not, however, many examples of a fully multi-user VR system for education and career guidance. The systems using HMDs are rarely made for multiple users, and the multi-user systems are usually a combination of AR and VR, or just AR.

Comparatively, the system created for this application is made to support remote and co-located multi-user career guidance with a focus on collaborative interaction. Users are encouraged to work together, discuss and reflect on the tasks. As mentioned earlier in section \ref{section:VRworkplaceIntership}, this previous project for virtual internships had good results, but more interaction and feedback were some of the more requested features. After implementing precisely that, there is a clear indication that this is positive for the users. The results seen in chapter \ref{chap:phase3} are positive, and the following sections will look at their effects more in-depth.

\begin{table}[]
    \begin{center}
    \begin{tabular}{@{}l c c c c c @{}}
           & \multicolumn{4}{c}{\textbf{Related Work}}
    \\  \cmidrule{2-5}
           & \textbf{Workplace}
    \\       
             \textbf{Features}
           & \textbf{Internship}
           & \textbf{ElectroVR}
           & \textbf{CoVAR}
           & \textbf{VR-JIT}
           & \textbf{Our artefact}
    \\ \midrule
       VR                           & \ON & \ON  & \ON  & \ON & \ON
    \\ Multiplayer                  &     & \LIM & \LIM &     & \ON
    \\ Workplace training           & \ON &      &      & \ON & \ON
    \\ Collaboration                &     & \ON  &      & & \ON
    \\ Real-world simulation        & \ON & \LIM & \ON  & \ON & \ON
    \\ Voice-chat                   &     &      &      & & \ON
    \\ Co-located                   & \ON & \ON  & \LIM & & \ON
    \\ Remote                       &     &      & \ON  & & \ON    
    \\ Symmetric role               & \ON & \ON  & \LIM & & \ON  
    \\ Asymmetric role              &     & \ON  & \LIM & \ON & \ON
    \\ \bottomrule
    \end{tabular}
    \captionsetup{width=1\linewidth}
    \caption{Related work applications and their features compared to our implementation.}
    \label{table:comparisonOurApp}
    \end{center}
\end{table}


\subsection{Research Questions}  
\label{RQDiscussion}

\subsubsection{Primary RQ: \textit{How does collaboration in virtual reality workplaces contribute to the career guidance of young job seekers?}} 

The primary goal of this research paper has been to discover whether or not collaborative VR is conducive to career guidance. Both the primary qualitative data and the secondary quantitative data that was gathered has a high degree of relevance concerning this question. 

First and foremost, how did actual NAV clients and NAV employees compare the collaborative VR experience to a single-user VR experience? While the number of respondents and testers were not as high as wanted, even for a qualitative study, the opinions of the actual users of the system were extremely important. Not only had they tested the single-user VR experience already, many of them were currently in or had been in  a situation where they needed NAV help to find employment. Looking at the qualitative data, we can see that the responses have been quite positive as the theme analysis in table \ref{table:phase3ThemeAnalysis} clearly shows. 

Section \ref{CSCL} discusses the effects of CSCL, and the ways in which it can transform learning for the better. As discussed there, CSCL is an inherently social way of learning, and technology must support this concept in a meaningful way if you are to gain benefits from it. When properly employed, CSCL should lead to increased motivation and learning. Greeenwald et al. found that collaborative visual learning increased the understanding of all involved users \cite{greenwald2017technology}. Considering the gathered data in this project, one can begin to see somewhat similar results. Table \ref{table:phase3SatisfactionAnalysis} shows the perceived value of the concept based on interviews with the testers. One of the points brought up most often here is the concept of \textit{engagement}. Looking at table \ref{table:phase3MethaphorsAnalysis} which shows metaphors and keywords that appeared, engagement is one of only three which was used in every interview. Respondents consequently refer to motivation and engagement as one of the biggest factors for successful use of collaborative VR. As shown in section \ref{quote:engagement} a counsellor said that \textit{"I think VR creates engagement, because I see how gaming how affects how they collaborate and discuss to solve issues."}. 


The engagement of users is an important aspect of learning. During the interviews, one of the most commonly discussed topics were how the inclusion of collaborative tasks would help raise the engagement of users. The NAV employees explained that with current single-user applications, users may grow tired of the VR applications after only a single use, citing boredom or repetitive tasks as the main reasons. They felt that with the inclusion of collaboration in VR, users would be more engaged and serious in their approach to the tasks. The interviewed NAV clients also expressed greater interest in the tasks when they were to be solved alongside someone else. It is also worth noting that the tasks used for the single-user version and the collaborative version did not differ at all, and were originally made to solved by a single person. This points towards collaboration in itself being the interesting and engaging factor, and not because tasks are more complex or engaging in and of themselves. 

By collaborating, users are also placed in a position where the experiential learning cycle is more naturally employed. In the section regarding experiential learning in section \ref{sec:experientialLearning}, the experiential learning model as seen in figure \ref{fig:KolbModel} is highlighted for its relevancy in regards to self-efficacy and learning. The looping nature of the cycle allows learners to grow and reflect on what they have done before, and may work now. The collaborative aspect further strengthens the cycle by allowing users to discuss and reflect on what they have done so far.  By doing this, users are able to proceed through the tasks with a greater sense of self-efficacy and a sense of accomplishment. The application includes light gamification elements such as a score list and checklists detailing what needs to be done. By including some of the aspects seen in \ref{sec:gamification} such as \textit{serious games}, an attempt was made to strengthen the experiential learning cycle by providing objective feedback to give direction to users as well as create some discussion hooks for both job seekers and career guidance counsellors.

Interviewees believed that the collaborative aspects would allow the users to push each other to be better in a peer setting, or allow for more hands on motivation and assistance when collaborating with a career counsellor or similar. One user remarked that \textit{"Engagement definitely increases,  I would say.  It is based on what I said earlier about collaboration in a workplace, it becomes more serious."} as seen in section \ref{quote:engagement}. This contributes to a higher sense of presence as the users become more immersed in their tasks as discussed by Greenwald in his 2017 paper \cite{greenwald2017technology}.




The secondary data also seems to support these points. As seen in figure \ref{fig:phase3_SurveyCollaborationIncreased}, 83.3\% of the survey respondents believed that collaboration would lead to increased career decision learning with 16.7\% responding with \textit{neutral}. The same distribution appears in figure \ref{fig:phase3_SurveyEngagementIncreased}, mirroring the qualitative data results. Comparing to the literature, this matches the finds of Madathil's 2017 paper \cite{madathil2017investigation} as seen in section \ref{sec:collabVR}.

While VR can offer great advantages for learning, there are as outlined in section \ref{learningVR} several aspects to consider. Table \ref{table:awarenessAspects} showcases some of the aspects one needs to consider for learning in VR. Several, if not all, of these aspects must reach a level of quality in order to provide a good learning environment. While the previous VR workplaces appear to have done well in these aspects, the affective aspect seems to hold a lot of weight when it comes to user satisfaction. The affective aspect is concerned with user engagement and confidence in the virtual environment, and it is specifically these aspects that the data would indicate the collaborative features are improving. Collaborating in VR will however do little to the \textit{orientation}, \textit{cognitive} and \textit{technical} aspects, for example. These must be improved in other ways, while the \textit{pedagogical} aspect is more dependent on \textit{who} you are collaborating with and how they work with you.



Furthermore, there generally appears to be a positive outlook on the possibilities of the concept. Figures \ref{fig:phase3_AppBeUsed} and \ref{fig:phase3_RemoteUsecases} showcase a majorly positive result for remote career guidance, highlighting another avenue in which the collaboration in VR can contribute to the career guidance of young job seekers. This is also the case for social skills. Throughout the interviews, respondents agreed that the artefact could function as a safe way to introduce workplaces to those struggling socially, allowing for a safe environment where they can speak to either employers or career counsellors. Adding collaboration into a Virtual Internship application means that it inherently receives an social aspect. As outlined in the background, see section \ref{section:CSCL}, this social aspect influences the  learning, and vice verse. The learning outcome from the application is inherently different from the knowledge which a user would gain from a single-user application lacking collaboration. Although according to our analysis CSCL seems to be a good fit for the VR workplace applications, that may not be the case for every type of career guidance application, and its use needs to be considered on a case-by-case basis.  


%Foundation for VR: Immersion (HMD section => resolution, refresh rate, FOV) and presence, social presence, Workspace Awarenes, CSCL

%VR Collab, career guidance, experiential learning, gamification (balancing act, should consider care of having to much gamification for young job seeker at NAV since it might impact their self-efficacy in a negative way) 

%Decsion learning
%CSCL



\subsubsection{Secondary RQ1: \textit{How is career-guidance affected by seeker-seeker collaboration compared to seeker-counsellor collaboration in virtual reality?}} 

Several features were implemented and numerous aspects were considered so that the artefact could support a wide range of different use cases. This includes a traditional career guidance situation with one seeker and a counsellor, but is not limited to it. The application system can handle any number of seekers and counsellors (up to 16) at any given point, using either VR equipment or a traditional desktop. This means users can utilise the application in a great number of different circumstances. For instance, four young job seekers situated in Trondheim can join a virtual workplace environment from their NAV facility or at home, while a business manager situated in Oslo can show how they operate whilst at the same time a counsellor observers the situation.           
Since the application supports both symmetric and asymmetric roles, plus remote or co-located usage, which based on our research of other applications does not exist for career guidance cases, it would be very interesting to see how this could affect young job seekers and their self-efficacy. 


As previously described, the Covid-19 pandemic heavily impacted our ability to collect data material in regards to this research question. Described in section \ref{section:covid19} the restrictions by the Norwegian government meant proper data collection was not feasible. We were not able to collect the desired amount of interviews from both seeker-seeker and seeker-counsellor pairs. The sample size meant the data material was not suitable for making generalisations of the target group, which generally requires a larger sample size \cite{oates2005researching}. We could there not justify any valid discussion or conclusions related to the research question since we did not possess reliable and rich data.                

This research question will thus become a suggestion for future work. See section \ref{section:futureWork}.  

%Explain why we cannot discuss and conclude this RQ. Lacking data material due to Covid-19. 



\subsubsection{Secondary RQ2: \textit{Which features are effective at facilitating collaborative virtual reality for remote
career guidance?}} 
There are many concepts which are important to consider and contemplate when regarding the use of collaborative VR for remote applications. First and foremost, the subjective feeling of presence, especially social presence, should be considered the bedrock for VR if it is to be applied effectively for collaboration. Explained in section \ref{section:socialPrecence}, a low feeling of social presence greatly impacts the overall experience of the application amongst its users. Factors such as visual representations of users (avatars), interaction with other agents and more behaviourally realistic visual representation has been shown to increase both involvement and engagement \cite{skalski2007role} \cite{oh2018systematic}. 

Interestingly, as shown in our analysis, figure \ref{fig:phase3_SurveyPrecenceIncrease} illustrates that the experts evaluation regarding the affect of social presence on the experience is divided and does not directly align with the research theory. This might be a direct result of experts lack of knowledge in the VR field, as the concept of presence is often misunderstood as being immersed. However, if we look at figure \ref{fig:phase3_SurveyValgkompVideo} we can see that a very similar statement: \textit{To what degree do you feel the presence of another user is useful for developing career decision learning?} was ranked \textit{High degree} by 63.6\%. Although not similar, these statements are concerned about the same concept (social presence), but they show different results depending on how they experienced the application. It is evident they perceive the value of presence differently, and should be taken into consideration. The being said, these data results are regarded as support material due to the circumstances explained before and in an ideal world all testing would have been conducted using VR headsets or desktop mode so that they get a proper feel for the artefact. 
Noticeably,  figure \ref{fig:phase3_socialPresence} shows experts perceived a relatively high degree of presence and social presence from the artefact. Although a few rankings were sub neutral there is a clear common consensus that they feel access to another intelligence and its intentions. This is further strengthened by the mentioned experience of a user shown in section \ref{quote:engagement}, 
\textit{"I have not used VR multiplayer too much before, it has mostly been single-user or turn based multi-user. So I have limited experience, but I felt the presence of another VR user as an avatar during testing.  Movement, and what he was planning was understandable."}.


As we described in the background, workspace awareness is important for how users can collaborate efficiently \cite{gutwin1996workspace}. Thus elements presented in table \ref{table:awarenessPresent}, \ref{table:awarenessPast}  and \ref{table:awarenessActivity} are all especially vital to consider for any application to be used in a remote setting. If a user lacks awareness of another user's intentions then they have trouble understanding what goal and outcome actions they execute is part of. Resulting in a disorienting experience for both users. The awareness of present actions is particularly crucial for counsellors observing other seekers in the workspace. Also, the ability to to explain how tasks work in order to provide assistance within the same virtual environment is a clear benefit compared to existing NAV applications where instructions was coming from the outside environment, resulting in a disconnected and difficult experience. With the artefact created for this thesis counsellors can effortlessly communicate with all users (e.g job seekers) using the built in voice communication component while simultaneously being present in the same virtual environment, which according to table \ref{table:awarenessActivity} benefits the workspace awareness. Experts heavily agreed that verbal communication is a useful quality if the artefact is to be used for remote guidance as seen in figure \ref{fig:phase3_VerbalWritten}. One user said that \textit{"Voice communication worked really well. Obvious who said what."}, see section \ref{quote:engagement}. This strongly indicates that the voice communication implemented is at a sufficient quality level to support the needs of the users.





%It is important to understand that remote career guidance could involve many different use cases depending on the situation. table 7.18

How developers utilises the advantages of VR embedded in the technology is decisive, as it can cater to and empower users that are not situated in the same geographical area to collaborate, share and learn about a workplace or career direction. During the development of artefact we included the ability to apply mechanisms of deixis to demonstrate and collaborate. Section \ref{section:workspaceAwareness} discusses in length what can be done to increase the awareness of users participating in groupware. Mechanisms of deixis fall under the term \textit{simplification of communication}. By enabling communication and coordination, users are able to work together more efficiently. Table \ref{table:awarenessActivity} describes the most important activities of collaborating in a workspace, and can be used to strengthen remote collaboration. This is important as it increases the learning effect \cite{stahl2006computer} and should work seamlessly despite any geographical or networking hindrances. If not, this could heavily distort the artefact's ability to facilitate remote guidance.


For any artefact to be effective for remote usage, we assumed the the process of installing, setup and application use must be easy so that as little as possible of technical support is needed. No one wants to use a system which is hard or difficult to use. It simply becomes too much of a burden so that other solutions such as video conferences are used. As such, during the development we put considerable effort into making sure the artefact could be as intuitive and easy to use as the time-frame allowed. Table \ref{table:phase3ThemeAnalysis}, \ref{table:phase3MethaphorsAnalysis} and figure \ref{fig:phase3_Intuitiveness} show that both survey participants and interviews generally agree that the artefact is intuitive and has a high degree of ease of use. Notably, a young user said during the interview that \textit{"Downloading and getting started was easy. I think it was pretty clear and quite self explanatory."}, see section \ref{quote:engagement}. However, there are inconsistencies with the survey rankings and the SUS score, which provides a scale on the usability of the artefact. The average  SUS score yielded a poor rating of the usability, meaning there is potential for improvement. An important note is that all participants of the survey (which gave the SUS score) was all in the age group of 45-64 and 65\% of those had little to none experience with VR. This could have impacted the results but there is no way of conclusively knowing without doing further studies.   

%With VR technology constantly evolving, allowing HMDs such as the Oculus Quest to disregard the use of a separate computer to hold the VR engine, the ease of use could be further improved. By publishing the application to the Oculus store it could be easier for the users to run the application, but also increase the availability, which was a keyword found in our theme analysis from the interviews (see table \ref{table:phase3MethaphorsAnalysis}). More on this in section \ref{section:futureWork}.

Although VR has become more readily available to consumers, there is still a large percentage who does not have the means to access the equipment needed to experience a VR environment. With the inclusion of the desktop mode in the artefact and through gamification principles we attempted to lower the threshold for the artefact to be used in remote settings. From the survey (where half used desktop mode) it was quite evident that this was supported by the experts as there was a high agreement that such an application could be useful for remote occupational guidance (see figure \ref{fig:phase3_AppBeUsed}). This reasoning is also supported by career counsellors in the interview as seen in table \ref{table:phase3SatisfactionAnalysis}. However, the desktop mode does not have the functionality VR mode has, meaning interaction with the environment is very limited, and might impact the experience. The intent of the desktop mode was primarily to let counsellors observe seekers in the workplace so that not much functionality was put into the feature. Seeing its potential this could be improved further and perhaps become part of a future IMTEL project. 



%social presence
%CSCL
%workspace awareness
%Table 2.1-3

%for it to be effective in use it must be ease to install/setup/use by remote users with minimal support. technologies (VR hardware, Quest would be useful, can be future work)

%desktop mode and VR mode. Desktop mode can lower the threshold (gamification => uses known game principals such as WASD movement). Con: now interaction with VR workspace.



\subsubsection{Secondary RQ3: \textit{What type of collaborative features are technologically feasible for virtual reality workplaces?}} 
\label{discussion:RQ3}

The use of VR as Virtual Internships as a mean to inform and educate young job seekers with the added element of multi-user functionality opens up a whole new challenge for developers. What features can be added so that it supports and enhances collaboration amongst users (e.g. seekers and counsellors) is difficult to answer conclusively, but some are explored in this thesis.

Throughout this project we developed several features based on feedback and analysis of data materiel. The very first collaboration mechanism was gesticulation, specifically hand gesticulation, using the avatar representations of the users. This is a common and effective technique for adding presence and identity in relation to workspace awareness as outlined in section \ref{section:workspaceAwareness}. Virtual 3D environments natively provides opportunities for this and can thus contribute with valuable awareness information according to Dyck
and Gutwin \cite{dyck2002groupspace}. Later, voice communication and laser pointer was added as it was deemed necessary for collaboration to be effective and would satisfy the needs of the counsellors and seekers which was not present in the first prototype. A counsellor remarked that \textit{"A laser pointer is very handy for counsellors, and if a user is wondering about something then you can point at the problem specifically. "}, as seen in section \ref{quote:engagement}. By our analysis, see figure \ref{fig:phase3_CollabHandGestures}, these features were mostly able to fulfil their objective aimed towards collaboration and workspace awareness. Gesticulation and user representation where also keywords found in the theme analysis, see table \ref{table:phase3MethaphorsAnalysis}.


While the difficulty of implementing them into a multi-user project varies, they are technologically feasible. Both gesticulation and voice communication logic was abstracted away and put into reusable prefabs so that little work has to be done for new developers to utilise these collaboration mechanisms. Using the prefabs should exclude the need for data serialisation and synchronisation, while being customisable if the need should arise. This means future developers do not need to be concerned with the underlying client-server architecture as described in section \ref{section:pun2}.
The implementation of laser pointer could, depending of what HMDs are used for the workplace experience, need some configuration to support the same action (press to enable laser) due to differences in controllers and thus different buttons amongst suppliers.  

When developing a single-user VR application there is no need to consider how the user is represented, just the hands are visible to the player. This changes for multi-player applications. There is a distinct need to translate real world movement into corresponding actions represented by an avatar. It provides elements of presence, identity and awareness such as where they are looking or what they are doing. Our solution was simple, a clear and common avatar with a torso, head and hands. This could perhaps be improved by implementation a full body avatar. However it is extremely difficult to implement without using sensors placed on a users body in the real world to translate position data. There was also no mention of this from either the interviews or survey, so this is a feature we would deem not necessary as it also provides little additional value. One must also consider drawbacks such as uncanny valley, yielding an unsettling feeling for the users. Section \ref{section:socialPrecence} discusses the role of an avatar and how the possible different depictions and behaviours can impact a user's feeling of presence. While a full body avatar may be more realistic than a floating head, hands and a body, the increase in presence is not going to be linearly dependent on the realism. As mentioned in the \textit{Social Presence} section in chapter \ref{chap:background}, a commensurate increase in realistic behaviour is needed to prevent it from becoming detrimental to social presence instead \cite{oh2018systematic}. 

The most common VR HMDs today cannot track eye positions, so there no definite way of knowing where users are looking. We can only assume it based on their head position. Gazing, the principle of looking at something for a period of time, is an feature which was not implemented in the artefact. However it could provide some value for collaboration as users could stare at objects and actions could be triggered (change colour etc.). Using ray-casting this could implemented, allowing users to either mark objects, or perhaps career counsellors could see summaries of what objects caught the attention of job seekers. 

An important aspect to consider when implementing collaborative features is that they must be visible and clear for all users. Different resolutions and FOV in HMDs could all impact the success of the features. Table \ref{table:hmdSpecs} outlines that the number of pixels used in the HMD effects the details in the application so that small text or objects can be experienced differently for the users depending on their VR hardware. It is therefore important to keep it in mind so that features does not differ to such an extent that they become useless.


The artefact developed contained both a VR mode and a desktop mode. The collaboration features was mainly implemented for the VR mode.   
The desktop mode had only voice communication and the ability to use their position as a marker. Due to the different circumstances, the limited amount of time meant that this was a low priority. However, as testing was done in the final phase we experienced that it would be very useful to at least include one or two more collaboration mechanisms. That way this mode could be used to a greater extent than just observation for career guidance. Using gamification principles, one could adapt common video game features and thus lower the threshold for its use. This could also help improve the ease of use which was an identified theme in the analysis, see table \ref{table:phase3ThemeAnalysis}.  

%Features (such as laserpointer or hand gesticulation) that can be developed and carried out to fulfil its objective.
%Technologies sections: Unity, PUN2, syncing movement, cross compatible with most HMDs.  

%fullbody avatar, uncanny valley 

%desktop mode

\subsubsection{Secondary RQ4: \textit{What challenges arise when implementing collaborative features in an ongoing
single-user virtual reality project?}} 

As described in the chapter \ref{chapter:1}, this thesis could be regarded as a branch project of the Virtual Internship project. No previous projects have been develop with multi-user capabilities or with that in mind. There was therefore a wide range of obstacles and challenges which was encountered during the three phases of software development. First of all, it is the fact that single-user VR projects do not need to be concerned with anything related to networking. Described in section \ref{section:pun2} and illustrated in figure \ref{fig:ClientServer} and \ref{fig:highLevelArchitecture} a multi-user application such as ours often uses a client-server architecture. Hence, every action in the application done by one user needs to transferred to all other users with matching data so that state is synchronised. This means the foundations of a single-user project are often not suited for it be evolved into a multi-user application which supports collaborative features.

What we found during the development was that there were little to no project structure or followed code standards. It was obvious the application selected was a students bachelors project where Unity and VR development was new for most of them. Thus the lacking structure and inconsistencies should be considered an expected consequence. In the case of IMTEL's future Virtual Internship projects we recommend that they create a basic template for how development should be done and code should be structured. This is however difficult to adhere to as projects differ, but should at least provide some level of similarity between projects resulting in a better foundation if they were to be further developed as multi-user applications.    

There a few technical prerequisites that need to be in place before development can begin to implement collaborative features. However, some aspects determine the intricacies of implementation. These include the complexity of the application and underlying game elements, cohesion of code, numerous elements which needs networking and game state. With respect to technical requirements, there needs to be at least four things in place. First, (1) a  client-network architecture for data transmissions, (2) a lobby system for joining or creating instances of scenes, (3) code (scripts) to handle networked objects and finally (4) managers for for keeping track of game state and general network related requests.
A partial goal of this project was to create a general framework (including prefabs, scripts and lobby system) for future use. With this framework the difficulty of accommodating collaborative features drastically reduces. To further reduce difficulty we created a demo application in our artefact so that it could be used as a guide for development and implementation.    

Consideration must also be made towards what type of project benefits and is suitable for collaborative features. As an example is the \textit{Wind turbine electrician} application \cite{henrichsen2019engaging}. Here there are limitations in regards to virtual space within to the wind-turbine so that having multiple users could present a crammed feeling for the users resulting in a negative experience. Also, the PUN2 cloud service used in the project has a limit of 16 concurrent users (CCU). If an application needs to support more than that there are basically two options: pay for more capacity in the Photon cloud or change framework. However, having more than 16 CCU is an unlikely use case for career guidance, the aim of the Virtual Internship is not to be a massively multiplayer online game (MMO).          

From experience there should also be at least two developers devoted to the task as it is very challenging to test collaboration features, or any other networking solutions with one developer. Additionally, for implementation purposes there should be an agreement of utilising a common VR SDK. Outlined in section \ref{section:sdks} SteamVR is a great choice as it enables support for building applications for major VR headset brands with little change. This yields a higher degree of users which can utilise the collaborating features, although some configuration could be needed. Another consideration could be OpenVR.



%IMTEL standards/prefab for future use. A foundation for future projects
%Discussion about general difficulties
%Thoughts to consider: type arbeidsplass (lite plass i vindølle for flere), mengde brukere (opp til 16 pers, mer enn det må man betale for PUN cloud), synkronisering (komplekse oppgaver med states er krevende) => men finnes metoder som helper med dette (RPC)
% bør være minst to stykker for utvkling, da testing av multiplayer er vanskelig som en

%ProgArk pensum: high cohesion, low coupling osv., code standard







\section{Limitations}
While the overall project has been successful and produced a functional artefact as planned, testing was impacted by Covid-19. As a direct result of the pandemic, the amount of testers decreased, which also may have lead to some selection bias.

Originally, the plan was to test with NAV clients, which would allow for a good distribution of testers. This would include people from the entire spectrum of familiarity with technology and make sure that the artefact was usable to everyone, not just those who have used VR before. With the stay-at-home situation, testers were significantly harder to gather. Those that agreed to join us for remote testing already had VR headsets at home, or at least an interest in the technology, which introduces bias in the tester pool on top the already small selection. While it is hard to pinpoint the exact reason for this, and most likely it is simply based on the low amount of testers, we were not able to test the final application with female job seekers. On the other hand, female career counsellors were over represented compared to males. There was also no opportunity to test the artefact in schools, as was originally planned.

%Covid-19
%Data gathering
%Limited testers to NAV clients, not tested at schools
%- Small scale tests
%- Alle testere hadde VR headset, eller var involverte med prosjektet fra før av (Bias)








\cleardoublepage