An essay written for the course Bioinformatics for Molecular Biology (MBV-INF4410, University of Oslo) in 2013 discussing the possibilities of protein visualization by VR, the Oculus Rift in particular. Includes useful links for people interested in the use of VR in molecular biology. You can download this text in .pdf format here.
Protein Visualization and Virtual Reality
|
Introduction Since the 1950’s, structural prediction of protein structures has been essential to furthering our understanding of molecular biology. In the 1970’s, the first crystallographically solved protein structure was visualized on a computer (Beem KM, 1977), and advances in the protein modeling field has been led by bioinformatics ever since. Schwede (2013) argues that protein modeling has gone through a paradigm shift during the last two decades. The trend has moved from few available structures to automatic generation of protein models for most amino acid sequences. Three dimensional structures have been assigned for the majority of model organism proteins. This shift coincides with another change – the parallel decrease in the cost of computer hardware and the leap in processing power and memory, making modeling and visualization of protein structures readily available to any motivated researcher. At present, the main challenges in protein modeling is correct interpretation of the structure and enabling a realistic visualization of the structures themselves. In many cases, the final details of a polypeptide structure cannot be solved by algorithms alone, but requires human intervention. This requires educated human intuition, and is greatly enhanced by comparative modeling. However, visualizing proteins in a manner allowing for comparison and exploration is challenging, and as of yet not optimal. The Cambridge Dictionary defines virtual reality as “a set of images and sounds produced by a computer, which seem to represent a place or a situation that a person can take part in”. In this text, I will explore how virtual reality technologies could be relevant to this challenge. I will use the CAVE and the Oculus Rift as examples of relevant VR technologies and compare these. As virtual reality is a vibrant technology under rapid development, I will limit my discussion mainly to the relevance for protein modeling, and explore how this technology is being applied at present and what applications it could potentially have in the future. Visualization of Proteins Protein function involves interactions and conformational changes. In order to understand these functions, researchers need satisfactory visual representation that can be explored dynamically. While a variety of desktop and browser programs are available for examining three-dimensional structures, these have the disadvantage of inadequate depth perception and difficulty of manipulating molecular orientations, especially when the two-dimensional screen gets cluttered with several models (Anderson A, Weng Z, 1999). The programs are also presented with a challenge when attempting to present transient protein interactions and intrinsically disordered proteins – situations where small changes make a particularly big difference. Even during the early 90’s, developers saw better immersion as a solution to this problem. The argument was that 3D visualization would only be optimized if the researcher experienced the impression of being “in there” with the protein. Virtual Reality Technology The CAVE – A virtual reality environment The earliest attempts at creating virtual reality environments involved isolating a part of the real world and, figuratively speaking, transforming it into a different one. This approach prevailed in scientific VR throughout the 1990s and 2000s – exemplified through the Cave automatic virtual environment (CAVE) system (Cruz-Neira et al, 1993). The name is fitting, as this VR is generated from a room with wall-fitted screens. Immersion in the CAVE is created by motion capturing the position of the user in the room. Implementing a system where the user can use his or her whole body to interact was seen as an advantage over head-mounted «eyes only» systems. For bioinformatics, the scientist or student could walk around objects and crouch or perch to observe processes on different levels – a handy feature for models of molecular dynamics and membrane receptor binding (Cruz-Neira et al, 1993). In 1993, the CAVE overcame the limitations of contemporary virtual reality technology. The image resolution was improved drastically in comparison with other systems, while the user was freer to move and interact with the virtual reality world than by using clunky and slow head mounts and hand-held controllers (Cruz-Neira et al, 1993). Scientists reported previously undiscovered molecular features «popping out» as they explored the VR environment (Schulze et al, 2011). However, the disadvantages of the CAVE were many. Memory and processing limitations hindered complicated rendering of molecular structure interactions. Most importantly, the CAVE was a costly setup (Cruz-Neira et al, 1993). While the hardware limitations are less relevant today, the economic aspects of the CAVE are still problematic. The Star CAVE wall, built in 2007, cost $1 million, while the less powerful NexCAVE setup cost “a few thousand dollars” (Schulze et al, 2011). Data processing is problematic for virtual reality representations. In the CAVE, users reported annoying delays in the redrawing of molecular structures. Virtual realities also require a “fixed point” giving the user the impression of where he/she is. If you are unsure of your position in the “world”, you will have difficulty steering, but also feel disoriented.(Anderson A, Weng Z, 1999). This user issue could be particularly problematic for the CAVE, as the most common setup is to have a user stand upright in a room, not allowing for a “fixed point” in the representation (as opposed to the user sitting down and being fixed to a certain environment). These user problems, in combination with the prohibitive high cost and complicated nature of the CAVE, explains why this setup is not common at universities. The Oculus Rift – a stereoscopic display suitable for normal workstations. The composition of a virtual reality system remains unchanged from the early 1990s. The system at its most basic consists of three hardware modules: A display (usually with two screens, delivering stereoscopic output resulting in a 3D effect), a computer (generating the graphical output to the headset) and a tracker (Bryson, 1996). A tracker module integrated in the head set enables movement of the visual field as the user changes his or her head position. In August 2012, Oculus VR created a crowd funded campaign asking for donations to fund the development of a new virtual reality headset. $2.4 million was raised, and developer kits were first shipped out in late 2012 [1]. Weighing in at 369 gram, the headset is both light and comfortable to wear, in contrast to older head mounted display technology. [2] The principle of the Oculus Rift is creating virtual worlds through using stereoscopic 3D – the presentation of two images, one to each eye, through a single screen. The images are slightly warped, creating a three-dimensional effect. Movement is achieved by head tracking, and by moving his or her head the user has a realistic range of vision. Software Since its release, there has been a immense development in game support for the Oculus Rift. While Oculus VR and its developers have focused on how this technology could be optimized for computer gaming, utilizing the Oculus Rift for non-gaming applications is also feasible. For example, DigiCortex is an interactive neural pathway visualizer under active development [3]. Virtual reality support has been developed for PyMOlL by Virtalis [4] both for CAVE-like wall displays and head mounted displays. The Virtalis software is a commercial product, not freely available to researchers or students. Developing PyMOL support for the Oculus Rift is challenging, as the newest versions of PyMOL are notably not open source (only the outdated builds are). An alternative is viewing three-dimensional stereoscopic content modified for Oculus Rift directly through your browser. Vr.js is a simple plugin that wraps multimedia content into an Oculus Rift format [5]. This javascript library has been successfully implemented for Google Street View. Vr.js also allows for stereoscopic 3D presentation of panorama images, 3D models and interactive environments. It is feasible to imagine this plug in integrated with other .js applications, such as protein visualizers. More complex visualizers are under development by utilizing raytracing techniques. In normal three-dimensional graphics, light rays are computed as triangles, but raytracing computes a “light ray” from every pixel in the scene [6]. This makes the rendering intensely realistic compared to older techniques. These types of visualizations require slightly stronger computers than a standard desktop computer, but with the low cost and rapid developments in CPU and memory technology, we can expect these applications to run on standard desktop computers in the near future. Why Virtual Reality as Bioinformatics Visualization Tool? The technology for generating virtual realities has become both available and affordable, but a central question remains – is virtual reality technology useful in science, and if so, why? The current molecular modeling programs are based on visualizing three-dimensional structures in a two-dimensional space. When manipulating and examining structures in this way, the researcher will often experience a «fuzzy effect», where the display gets cluttered with structures and a clear overview becomes difficult. Even in the 1990s, researchers reported feeling that receptor docking simulations were improved by virtual reality visualization. Engaging a researchers motor- and visual- skills could generate an intuitive docking setting. Humans are naturally skilled at visual pattern recognition in a manner not (as of yet) replicable by computer algorithms, and engaging this skill is a potential tool for speeding up automatic searching. Conceivably, researchers in a virtual reality environment could choose to restrict a search to an interesting site determined by the researcher’s knowledge and interest, not the automatic search function in itself. This could be very useful in cases where, for example, a binding site is unknown (Anderson A, Weng Z, 1999). While a complete virtual reality setting with full immersion could be a holy grail for protein visualization and experimental modeling in the future, there are inherent difficulties with virtual realities at present. First of all, the performance of the virtual reality representation is limited by the processing power of the computer used to generate the outputs. At the moment, a standard desktop computer can easily run simple virtual environments, but higher level simulations will require a more sophisticated graphics card, memory and processing power. Secondly, latency is an important concept in virtual reality. Simply explained, when you move your head in a virtual reality environment, the images around your head must be redrawn. Latency describes how much time this process takes. A high latency causes the visual input to lag as you move your head to change your surroundings. Early virtual reality technologies suffered from high latencies, as the brain interprets lags to mean that the visual input is not realistic. A 5-15ms latency would be ideal for a total seamless exploration of a virtual world, and the Oculus Rift is getting quite close by lowering the latency to well below 20ms [7]. In comparison, the CAVE system has a 20ms latency [8]. The Oculus Rift Consumer Edition is expected to have even lower latencies, and this one of the reasons that the Oculus Rift might be more suited for molecular visualization than the CAVE. Finally, two-dimensional interface paradigms could be problematic to implement in an immersive virtual reality. Input controls by mouse, keyboard or touchscreens are hard to implement if you are navigating a virtual reality through a head-mounted display. With modern virtual reality technology, these problems are decreasing rapidly, as evidenced by existing control functions such as the Razer Hydra [9] and LeapMotion [10]. Head tracking in itself allows for a certain degree of control of the environment. As Kin (et al, 2004) explains, a user utilizing a head mount is not interacting with a two-dimensional screen display at all, but rather directly manipulating a model in three-dimensional virtual reality. This could for example be using to combine a ligand with a receptor, an example of docking modeling. In this way, the researcher could observe interactions at the active site from many directions, circle around these, and also focus on related sites of potential interest. Another exciting possibility is the use of eye tracking – the Oculus Rift has the possibility of tracking the user’s focus, and activating functions by identifying which aspect of the environment is being looked at. It is interesting to imagine an interactive protein modeling tool with a HUD (head-up display) inside the visualization. In this thought experiment, the researcher could manipulate the program settings by simply navigating the virtual menu by eyesight. If the researcher could manipulate settings by visual input, manipulation of the protein itself could be carried out through motor input (hand tracking), such as selecting certain areas of a protein by hand and directing his gaze to change the coloration or representation of this selection alone. Such a setup would both be more intuitive and less time consuming that navigating a standard two-dimensional screen by keyboard and mouse. The thought might seem fantastical, but the technology for such a setup already exists and awaits implementation. Conceptually, virtual reality allows researchers to use their own educated intuition to a larger degree than when using two-dimensional screens (Anderson A, Weng Z, 1999). Correct protein and ligand modeling (for example of amino acid side chains) require human intuition and experience. In virtual reality space, ligand design and docking could be performed by hands alone. As mentioned above, other controls and overlays could be implemented. Discussion In comparison to gaming and others types of visualizations, scientific imaging stands in a unique position, often containing complicated three-dimensional data. Virtual reality offers a method of unambiguously displaying this. Exploring data in this manner allows for a much richer set of depth and spatial cues than would be possible with a two-dimensional screen. An argument for using the CAVE system and not head mounted VR displays has been that using your whole body to navigate an actual «virtual room» is more realistic than the latter. However, while this argument might be valid for driving a car in a racing game, it might not apply to academic utilization of VR. Bryson (1996) argues that virtual reality works better for science if not occupied with expressing a realistic world. A virtual reality for science can thus be oriented towards representing accurate rather than realistic representations. This allows for abstract experiments and investigations – such as protein interaction simulations in different environments – impossible in a “real” world. These investigations could themselves be of intrinsic value. Another aspect of the virtual reality in science is that a researcher would be able to explore regions not “computationally and mathematically” expected to be of interest (Bryson, 1996). In this sense, it can be argued that virtual reality technology and science makes for a far better combination that virtual reality and gaming. However, despite promising developments, virtual reality technology is challenging to implement. Most new users are not accustomed to wearing a head mounted display, and neither the software nor hardware are as of yet sophisticated enough to be of serious academic use. One might imagine the affordable equipment and the open source traditions of bioinformatics would create a natural environment for active development. However, bioinformatics stands in a challenging position when it comes to attracting developers for open source applications (Stahl M.T., 2005). One would need to draw developers and visionaries from two quite specific pools of specialists. In the case of virtual reality for protein modeling, these would be biologists and programmers. The virtual reality developers have primarily been interested in gaming, and do perhaps not easily see the value of applying their talents to an unfamiliar field such as biology. Mentioned above, the Molecular Visualizer is to date the only application to model molecular structures for the Oculus Rift. As a prototype for the future possibilities, this application is impressive in its details and immersive effect. The developer of this application, is currently implementing a set of control functions for protein visualization, inspired by programs such as PyMOL (Favreau, pers. comm., 28 november 2013). Conclusion In 1987, Box and Draper noted that «Essentially, all models are wrong, but some are useful». When considering protein models, three-dimensional models are certainly useful, even on two-dimensional displays. However, so are virtual reality representations of these models. The key point is that points of view matter – in a virtual reality environment, new aspects of a model could be discovered. Virtual reality might be especially interesting for visualizing topologies and interactions in real time. For example, Oculus Rift is currently integrated with Google Street View. One could imagine a similar setup for a three-dimensional view of cellular organization, or even walking around a genomic landscape. While computationally taxing, one could also envision a real time landscape with controls for pausing, speeding up – or even “terraforming” epigenetic elements. At the moment, software for academic purposes is sparse, but there are several programs in development. Favreau likens the usefulness of virtual reality in academic settings to music, suggesting that our brains are capable of understanding the contents of big data if it is presented in a way we can comprehend, just like we easily identify one song among thousands in seconds (pers. comm., 28 november 2013). Suggesting virtual realities as the best way of visualizing protein structures might seem ambitious, but historically speaking the concept is not new to bioinformatics. In 1980, the following lines were written about stereoscopic images in the Teaching aids for macromolecular structures manual [11]: “As you look at the stereoscopic views you will see beauty beyond any previous experience. 34 years later, this idea still holds true, with a certain modification: it seems plausible that we – students and researchers alike – will not only understand, but interact with macromolecules as they really exist. References Articles and Books Anderson A., Weng Z., (1999) VRDD: Applying virtual reality visualization to protein docking and design. Journal of Molecular Graphics and Modelling 17, 180–186. Bryson, S. (1996) Virtual Reality in Scientific Visualization. Vol. 39, No. 5, Communications of the ACM. Cruz-Neira, C., Leigh, J., Papka, M., Barnes, C., Cohen, S.M., Das, S., Engelmann, R., Hudson, R., Kim, J., Park, S., Lee, J., Choi, Y., and Jung, S. (2004) Development of a Gesture-Based Molecular Visualization Tool Based on Virtual Reality for Molecular Docking. Bull. Korean Chem. Soc. 2004, Vol. 25, No. 10. Roy, T., Siegel, L., Vasilakis, C., DeFanti, T.A., Sandin, D.J. (1993) Scientists in Wonderland: A Report on Visualization Applications in the CAVE Virtual Reality Environment. IEEE Electronic Visualization Laboratory, Department of Electrical Engineering and Computer Science, University of Illinois at Chicago, Chicago, IL 60680. Schulze, J. P., Kim, H.S., Weber, P., Prudhomme, A., Bohn, R.E., Seracini, M., DeFanti. (2011) Advanced Applications of Virtual Reality. Advances in Computers, Vol. 82. Burlington: Academic Press, 2011, pp. 217-260. Schwede, T. (2013). Protein modeling: what happened to the “protein structure gap”? Cell Press Structure, 21(9): 1531-1540. Stahl, M.T. (2005) Open-source software: not quite endsville. Drug Discovery Today, Volume 10, Issue 3, 1 February 2005, Pages 219–222 Beem, K.M., Richardson, D.C., Rajagopalan, K.V. (1977) Metal sites of copper-zinc superoxide dismutase, Biochemistry, 1977 May 3;16(9):1930-6. Webpages 1. Kickstarter 2012, viewed 5. December 2013, <http://www.kickstarter.com/projects/1523379957/oculus-rift-step-into-the-game> 2. Oculus VR 2013, viewed 5. December 2013 <https://developer.oculusvr.com/wiki/HardwareSpecs> 3. Digicortex, viewed 5. December 2013 4. Virtalis xx, viewed 5. December 2013 <http://www.virtalis.com/academic-rad.php> 5. Ben Vanik, GitHub, viewed 5. December 2013 6. Hindriksen, V. Scientific Visualisation of Molecules Steam Computing, viewed 5. December 2013 <http://streamcomputing.eu/blog/2012-10-31/scientific-visualisation-of-molecules/> 7. LaValle, S., The Latent Power of Prediction. Oculus VR, viewed 5. December 2013 <http://chpsw.temple.edu/chpsw/vepolab/index_files/Page588.htm> 9. Razer Hydra, PC Gaming Motion Sensing Controllers, viewed 5. December 2013 <http://www.razerzone.com/gaming-controllers/razer-hydra/> 10.LeapMotion, The Leap Motion Controller, viewed 5. December 2013 11. Martz, E., Francoeur, E., History of Visualization of Biological Macromolecules, Viewed 5. December 2013 |
lizardphunk 