At the beginning of the year, Blooloop.com, a professional resource providing information about museums, amusement parks and other attractions, named six technology trends in leisure activities for 2020. This list includes virtual and augmented reality (VR/AR), artificial intelligence (AI), interactivity, as well as robots and e-sports. The latter replaced the bots and voice control that were among the trends for 2019. Although the forecast for this year had been made before the world faced the coronavirus pandemic, it describes a further shift in the development of cultural entertainment and tourism to online.
ICT.Moscow asked the developers of digital solutions for cultural institutions and representatives of museums about the specific features of their interaction and asked how they evaluate digital platforms in this area. The experts spoke about the changing demand for digital solutions in tourism and cultural leisure activities during the pandemic. Moreover, the material contains practices and examples of the application of emerging technologies in this area in Moscow and abroad.
Vladimir Opredelenov, Deputy Director for IT at the Pushkin State Museum of Fine Arts and Head of the Department of Information Technologies in Culture at the Higher School of Economics, highlighted an important issue in terms of interaction of the museum as a customer with IT developers - understanding what each party needs. The situation is complicated due to the fact that cultural institutions rarely have clearly defined business processes.
Opredelenov emphasizes that after the terms of reference are created, it is necessary to prepare a large amount of time for its debugging, not to skip the stages of testing, commissioning and training of employees.
The experience of the “Future History” studio - the creator of the immersive theater application “Mobile Art Theater” - can be an example of successful interaction and complete understanding of aims and objectives. Maria Shashayeva, COO of “Future History”, claims that usually the company completely comes up with what the project should look like and how it should work, then it hires programmers or a team to implement these ideas, and then the company employees become engaged in filling the project with content and its further development.
Moscow museums often initiate new digital projects on their own, especially when it comes to solutions that are close to the user, says Sergey Shakryl, curator of IT projects in the field of culture and business of the Moscow Department of Information Technologies. The agency initiates solutions affecting the entire industry. The DIT of Moscow notes that there is a large number of requests from cultural institutions for revision and improvement of the current functionality; it is attributed to the active implementation of IT solutions.
Maxim Panin, head of the project office of digital transformation of the Ministry of Culture of Russia, claims that cultural institutions make different requests. According to him, when considering such applications, the department identifies “common problems and tasks which the business may not be so interested in, but at the same time there is a need to automate and digitalize a number of processes”.
Egor Yakovlev, co-founder and development director of izi.TRAVEL, a street and museum audio guide platform, says that when it comes to upgrading this application, he most often hears about VR/AR and iBeacon. However, these requests are not so common and much more often “stem from the fantasies of museums, not from the real requests of visitors”. The expert believes that it is more important for museums to deal with the quality of content, for example, focus on making storytelling more interesting.
In the field of culture, as in many other areas, digital platforms are popular, which allows partners - in this case cultural institutions - to use ready-made solutions, providing their customers with convenient services. Such examples include Google Arts and Culture or Artefact.
Maxim Panin, head of the project office of digital transformation of the Ministry of Culture of Russia, which is the project customer of Artefact, recalls that the department’s strategy includes the creation of platform solutions. He calls Artefact the UGC platform, where User is a museum that can create a multimedia guide for free with the help of a WYSIWYG editor, using only a computer and a camera.
Maxim Yakovlev from Datastack (developer of the Artefact application) believes that, starting from a certain level of the task, a digital platform can be its solution. He claims that “it depends on the scale and tasks, everything always starts with individual developments and only then becomes a platform solution”.
CEO of Datastack
The platforms allow to combine various functionalities, including universal solutions that are not directly related to culture. For example, street and museum audio guides platform izi.TRAVEL provides its museum content via API to third-party applications with other technologies: Yandex.Alice and the Atrefact application using image recognition and AR technologies. Egor Yakovlev from izi.TRAVEL considers integration with voice assistants one of the most important strategic tasks of his service.
Sergey Shakryl from the DIT of Moscow agrees that platform solutions are suitable for meeting some mass requests. However, he is convinced that such solutions will not be able to replace individual developments, since in the field of culture there will always be unique projects, which require their own unique solutions.
Alexey Steblev from RDI.Digital agrees with Sergey Shakryl. He believes that platforms cannot replace individual IT developments in the field of culture. He claims that “every cultural object is unique, and its creators or keepers are not always ready to fit it into a common, even the best, interface”.
The pandemic brought us several projects, including the creation of a marketplace for the Vienna contemporary art fair Viennacontemporary and an educational project for the Pushkin Museum of Fine Arts.
Sergey Shakryl from the DIT of Moscow says that the introduction of technologies in certain areas was put on hold because of the pandemic, but at the same time the pandemic contributed to the increase in the number users of existing online cultural sites. He notes that the number of virtual tours and exhibitions has increased significantly.
The Pushkin State Museum of Fine Arts is trying to bring all the products that the museum creates to online. The museum has difficulties with funding, since the off-budget income has drastically decreased, Vladimir Opredelenov says. This made the museum switch to more rational use of existing resources and think over new business processes. For example, Opredelenov says that during the pandemic, it has become clear that the system of 3D modeling and design of future expositions of the Smart Museum can be used for full-fledged virtual exhibitions.
Along with the introduced restrictions, the demand for online projects has also increased, including in the field of cultural entertainment. Egor Yakovlev from izi.TRAVEL says that during the times of severe restrictions people started to listen to virtual tours of museums more actively. He attributes this to the fact that museums quickly began to redirect offline visitors to online. He adds that when the restrictions were lifted, the demand for local travel and audio guides to all places that could be reached by car on weekends increased sharply. He also said that in August 2020 izi.TRAVEL had reached an absolute record for the number of stories people listened to - more than 2 million.
The head of the company that developed Artefact platform claims there was a three-fold increase in the attendance of digital resources of the platform in the first months of quarantine. However, after a couple of months, the indicators began to decline, but still became higher than before the start of the pandemic.
Sergey Shakryl, curator of IT projects in the field of culture and business of the DIT of Moscow, also notes there was an increase in demand for museum online projects, although there was no sharp growth, and the user’s behavioral model remained the same.
The fact that the growth in demand during the coronavirus pandemic is associated with the uniqueness of the content is confirmed by the data of the Pushkin State Museum of Fine Arts (46th most visited in the world in 2019): in the first month of the pandemic, in the interest in VR expositions and excursions increased by 1500%.
Users actively watch virtual museum tours online, but are not yet ready to pay for it, Vladimir Opredelenov stated. The museum notes that with the introduction of fees for VR tours after the quarantine was canceled, their popularity has declined.
According to Alexey Steblev from RDI.Digital, the introduction of technologies will dramatically expand the audience of the cultural sphere, and in tourism it will provide great opportunities for individualizing user experience, leading to an the expansion of both audience and the geographic scope.
Head of the project office of digital transformation of the Ministry of Culture of Russia
Access to museum collections in digital format was already available even before the spread of the Internet, when such image collections were stored on CD-ROMs. The Museum of Computer Art (MOCA), founded in 1993, can be considered the first online project that is not a digital version of a traditional cultural institution. Today, according to the results of the VCIOM (Russia Public Opinion Research Center) poll, among Russians with Internet access (83% of the population), one in five people visited a virtual museum (19%). Among them, 57% took a virtual tour during the self-isolation regime.
Practically all residents of Moscow (94%) are aware of the possibilities of virtual museums. In the capital a single resource called Museums of Moscow online was launched in 2017, it has more than 50 thousand photographs of exhibit items. During the period of self-isolation due to the spread of COVID-19, the project Culture of Moscow online was also opened, it contains video versions of performances, concerts, lectures, etc.
Virtual exhibitions and interactive tours are of much greater interest in terms of technology applications. The most famous global project in this field is Google Arts and Culture - a platform with panoramic views of halls and images of exhibits in high resolution. There you can find more than 20 collections from Moscow, including, for example, the Museum of Cosmonautics or the Pushkin State Museum of Fine Arts (Pushkin Museum also has its own large VR project). In total, the Google project includes over 2500 museums and galleries from different countries. From 2016 onwards, the application supports VR format and is compatible with virtual reality headsets.
Yandex has a similar service based on the Yandex.Maps functionality - about 15 Moscow museums are available in it for the moment.
The American Alliance of Museums named virtual tours and VR/AR among the four ways of digital museum content promotion during coronavirus pandemic.
In Moscow in 2016 the Pushkin Museum launched its own virtual tours. The experience of the Tretyakov Gallery in using immersive technology is also interesting: in 2018, an interactive exhibition in virtual reality “In Three Dimensions: Goncharova and Malevich” was held there, it was organized jointly with the VRTech Group of Companies.
in their VR projects do not repeat the existing exposition, but offer original
content and experiences. An example is the panoramic VR film “Journey to the
Past”, presented in 2017, which became part of the permanent exhibition of the
Jewish Museum and Tolerance Center. And in the “Smart City” pavilion at VDNH,
you can take a virtual trip in an unmanned vehicle.
The Consumer Technology Association predicts that VR and AR in 2020 will return to double-digit growth for the first time in four years. The opinions of Russian experts on the prospects of this field can be found in the review material of ICT.Moscow.
VR in cultural entertainment
Virtual reality technology is originally focused on entertainment. However, it can be used not only in gaming and is increasingly being used in more traditional areas. For example, VR in theaters. In the fall of 2019, the State Theatre of Nations hosted the premiere of an experimental performance.
This year, due to the pandemic, the organizers of the Venice Film Festival have also used VR technology. Moscow was one of 15 cities where special platforms were opened (in the Russian capital it is the Flacon design factory) for viewing immersive works selected by the curators of the Biennale.
Another immersive technology popular in the area of cultural recreation and tourism is augmented reality, for which it is enough to have a smartphone. Many museums in the capital, including the Tretyakov Gallery, the Pushkin State Museum of Fine Arts, the Museum-panorama the Borodino Battle, the Darwin and Historical Museums, use the Artefact application for this. Museums are free to join this interactive exhibition guide. The solution was developed by the Moscow company Datastack at the request of the Ministry of Culture of Russia.
At the end of 2018, the federal project “Digital Culture” was approved in Russia, it is expected to last until the end of 2024. The document, in particular, entails the creation of virtual concert halls and multimedia guides on expositions and exhibition projects using AR technology.
An example of a separate AR-development is the mobile application of the Museum of Cosmonautics - a virtual guide that introduces the main exhibit items in game form.
In addition to navigating inside exhibition spaces with virtual annotations to exhibits, AR technology is also used to guide tourists between attractions in city streets. In the capital, the application “Discover Moscow” has been developed for these purposes. In addition to geographical points with descriptions of monuments, the application contains a 3D-model of the Ostankino TV tower: being near this tourist attraction, users can get acquainted with its history and internal structure in AR mode.
Here is another scenario of using AR for tourists: the possibility to take a picture with a virtual character (similar to catching Pokémon in a mobile game which was popular a few years ago). Such functionality in Moscow is now available in the Histars application, but it is planned to be transferred to the “Discover Moscow” app as well.
In May 2019, the Moscow Samsung Artificial Intelligence Center presented an algorithm that helps to create “living” portraits. A moving Mona Lisa was shown as an example. You can read more about the prototype created by Russian developers here.
At the end of 2019, the Metropolitan Museum of Art, Massachusetts Institute of Technology and Microsoft company held a joint hackathon to find new ways of using artificial intelligence to present the collection of the New York Museum to the visitors. As a result, several prototypes were developed. Among them there was the Artwork of the Day service, which selects a piece of art, one way or another corresponds the current information agenda in the world, taking into account the preferences and characteristics of each user. Another interesting solution is Storyteller. It selects suitable images from the museum’s collections that can serve as illustrations for what the user is talking about. For this purpose, speech recognition algorithms in 60 languages are used.
Technologies and art
The application of technology in the visual arts industry is called ArtTech. According to Deloitte estimates, startups in this area in 2011-2019 received almost $600 million in investments. Experts predict that the dynamics of the emergence of such initiatives will increase over the next two years. An overview of the 10 most unusual ArtTech startups of 2020 is available here.
The Salvador Dali Museum in Saint Petersburg (Florida, USA) in 2019 used artificial intelligence to create the image of a revived artist. The algorithm was trained on the basis of archival records, interviews, quotes, to recreate the appearance and behavior of Dali. His virtual image helps visitors to get acquainted with the collection of the museum.
Artificial intelligence was also used in the Tretyakov Gallery’s VR-exhibition “In Three Dimensions: Goncharova and Malevich”, which has already been mentioned above. Visitors could compose their own still-life painting from virtually proposed objects. AI helped them, processing the compositions created by visitors in real time. The neural network was trained using a dataset of art objects images presented in the museum.
Another example of using AI in a museum is Viktor Vasnetsov’s “revived” paintings as part of a multimedia exhibition in the same Tretyakov Gallery. To do this, two machine learning algorithms trained on the artist’s paintings were used.
Computer vision algorithms were used in another cultural project. With the help of the face recognition technology Vision which was developed by Mail.ru Group, archival photographs of the Great Patriotic War were processed, and users could find images of their relatives.
Cases of AI application in the field of culture and art in the knowledge base of ICT.Moscow
Artificial intelligence was also an independent creator of visual art works. For example, last spring, Yandex opened the Gallery of neural network art. The neural network was trained on works of different painting styles. In total, there were about 40 thousand paintings in the training sample. On their basis, AI generated 4 thousand new images collected in an online gallery.
guides and rental devices for listening to recorded excursions are gradually
being replaced by smartphones through the use of chatbots and voice assistants.
In 2019, the Hermitage launched “Liza” chatbot. Visitors of the museum can communicate with it via messages in social network “Vkontakte”.
Alice, a voice assistant from Yandex, can act as a guide to the Pushkin Museum exposition. It is able to recognize a picture from a photograph and tell about it.
Chatbots are also used to help tourists in urban space. For example, for the 2018 FIFA World Cup in the Russian capital, “Boris” chatbot was created.
Distributed ledger technologies are used not only in corporations: in 2019, a charity project of the Tretyakov Gallery with a blockchain was announced, it was supposed to be implemented by the Russian company RDI.Digital Russia and the Austrian Riddle & Code. However, later it was decided not to use blockchain, and Riddle & Code left the project.
The Pushkin State Museum of Fine Arts also had plans to implement blockchain. In 2018, the director of the museum spoke about the possible use of this technology in the processes of buying and collecting art objects.
Previously, the blockchain has already been used by Christie’s auction house in New York during the sale of art projects.
Analysts from the Consumer Technology Association predict that fifth-generation communications will provide significant impetus to the use of VR and AR, which will also affect the areas of tourism and cultural recreation. Among the many demonstrations of 5G capabilities there are a plenty of such examples.
Spokesman of DEUS
In a very traditional area of cultural recreation, new ways of contacting visitors with exhibits are appearing, and technological solutions themselves become units of cultural value. Tourism which has always been associated with movement in space turns into a remote multimedia experience.
For example, social networks are used in the field of tourism and recreation not only as platforms for chatbots or information and reference resources, but also as part of independent cultural and technological projects. At the end of 2018, the Russian Museum, together with the “Odnoklassniki” social network, launched “the Masterpieces of the Russian museum” virtual project. It is a separate site with audio guides of ten works from the museum collection, voiced by various Russian celebrities. The site also offers a virtual tour of the museum in interactive panoramic mode.
The coronavirus pandemic and mobility restrictions have forced people to use virtual tourism opportunities. Among other online excursions, the Faroe Islands initiative should be highlighted. The local Ministry of Trade and Industry has launched an online tour, the specific feature of which is that the user sees a panorama from a camera carried by a guide. At the same time, the virtual tourist can give the guide instructions in real time where to turn or where to go.
You can see Moscow and its architectural monuments without leaving your home using street panoramas on Yandex.Maps or Google.Maps. The self-isolation mode allowed Yandex to make panoramas of empty streets of the Russian capital with the opportunity to compare them with filming of previous years.
An interesting technological project is the Moscow gigarama - panoramas of the city from a bird’s eye view and “infinite” zoom, created with gigapixel photography technology.
VOMA (Virtual Online Museum of Art) has been opened in September. The VR space is available free of charge from anywhere in the world, it displays digital copies of classical art objects from various collections (for example, the Musée d'Orsay and the New York Metropolitan Museum of Art), as well as works by contemporary authors. The project uses modern computer graphics for video games and distributed computing.
The coronavirus pandemic seems to have encouraged a return to a fully virtual environment that simulates an exhibition space. It is curious that such an immersive experience (on a more primitive technical level) was offered by Apple almost 30 years ago by releasing a CD with a three-dimensional model of a non-existent science museum.