Unity / Microsoft Trip 8/19 - 8/21/2019

USC ICT Combined Notes

Contributors to this report

• Ed Fast - Computer Scientist
• Mark Core - Computer Scientist
• David Nelson - Creative Director
• Kevin Chang - Research Programmer
• Rajay Kumar - Research Programmer
• Aakash Shanbhag - Computer Vision Engineer
• Ramon Paz - IT Manager
• Daniel Anguiano - Systems Administrator

Unity visit

Contacts

• Greg Melling - greg.melling@unity3d.com - Biz dev for of all non-gaming, dod, auto
• Jeff Hansen - jeff.hensen@unity3d.com - Head of Global Sales & Business Development
• Seth Gibson - seth@unity3d.com - Solution engineer - dev questions
• Arisa Scott - graphics
• Nick Rapp - ECS/DOTS
• Rick Johnson - XR Architecture / AR Foundation
• Mike Durand - XR Architecture / AR Foundation
• Kevin Saito, Principal PM - Machine Learning
• James Ford

Unity development happens in SF, Copenhagen, Seattle (AI & ML)

Overview of core Unity Tech - ECS, DOTS, HDRP, AR Architecture, AR Foundation, etc.

Arisa Scott - Graphics

Scriptable rendering pipeline - SRP

• HDRP and Universal RP (formerly Lightweight)
• No longer black box - C# source available
• Don't need to be a graphics programmer
• Fog system for Lightweight

Shader Graph system

• Node based approach
• Complex transparency and refraction options - Late December
• Subsurface scattering 
• Node-based shader authoring
• Diffraction shaders
• Vertex animation
• Advanced real-time materials, iridescence, sub-surface scattering, cloth, layered materials, decals, micro-shadows, screen-space reflections, de-noising

Ray Tracing in HDRP in C#

• Control on the number of rays and filters 
• Integration of DXR with DX12 implementation
• Vulkan supported

Terrain Tool API

• Texture mapping and Virtual texture maps
• Creating the world maps - Efficient vegetation and detail rendering GPU pipeline with dynamic response
• Terrain - nondestructive, layer-based
• Destructive: from a collaboration point-of-view
• Paint, Stamp

Official Stadia build target will be coming in a couple months

XR Overview - US Combat Capabilities

Nick Rapp - ECS/DOTS

Rick Johnson / Mike Durand - XR Architecture / AR Foundation

• Keep iteration times down
• Can reconcile an iPad in a place with lighthouse beacons setup
• HoloLens and HoloLens2 don't support plane tracking, but will
• Standardization?
• OpenXR is standard, but glacial
• Mainly around VR, but AR features not defined
• MS defines own extensions 
• Hand/gesture tracking -> HoloLens and Magic Leap
• XRSDK
• Alternative to direct integration of AR engines into Unity
• Let XR devices be custom integrated by own devs, not Unity
• Future support for "Tiny" Unity to allow small footprint / quick load AR apps
• XR Cloud
• Put a spot in middle of table, share with other devices, come back tomorrow and get same spot
• Unity Spatial References & Mapping data
• Google Cloud Anchors
• Azure Spatial Anchors

STE Problem Statement

Chris McGroarty

• STE pillars
• Scalability
• Architecture
• Models
• Behaviors

Unity Simulations - Overview

Kevin Saito, Principal PM - Machine Learning

Unleash power of visually rich simulation

Simulation Service - eSim Platform

• Find black swan events and replay them
• Playtesting
• Robotics (TBD)
• RL as a service (TBD)
• Cloud based
• Run headless
• Is it correct to say this can be used for simulated playtesting + RL training?  Yes
• Need own agent - ML agents is one way, Could build own bot
• Use REST APIs
• SQueue = Social Queue
• eSim SDK that enables "headless" rendering for CPU & GPU + additional optimizations
• kubernetes - cloud agnostic way to manage compute, storage, & network resources
• PaaS

ML Agents

• Learn from experience
• Trial and error until optimal
• agents learn from demonstrations
• mimics behavior quickly
• Open source Unity plugin - lower barrier of entry
• v1.0 early 2020 - Still in Beta
• Reinforcement Learning
• Imitation Learning

Unity for AI research

• Deepmind
• Stanford
• Q: ML Agents support for training outside of tensorflow and importing models into Unity aside from .nn files? General flexibility
• Ask eng team - reach out to Kevin Saito

SimViz - Unity for perception training

James Ford

• LIDAR example
• Point clouds
• Tree/Foliage cover example - See through environment
• Rays find opponents/targets

DOTS

Nick Rapp -- Desktop, DOTS, ECS Leads Desktop + Web teams in platforms group in R&D

• Data-Oriented Tech Stack
• Built on Entity Component System (ECS)
• MegaCity demo
• ECS at core of future
• C# Job System
• Runs a subset of C# -- smaller C# profile
• Access to specific types like native array
• Small enough to take adv of burst compiler
• ECS
• Burst Compiler
• Custom IL to Machine Code compiler
• Enable C# code to run at Native Speeds or faster
• Above and beyond when comes to SIMD optimizations
• For GPU? A goal
• Built to be entirely deterministic
• Snapshot worlds in deterministic way
• Modular approach to engine dev
  • Havoc can be added on from asset store
• Each module as a package
• Instead of monolithic engine
• Different modules for parts, like physics, renderer, audio, 2d, animation, editor
• Unity physics, cowrote with Havoc
• Pick and choose aspects of engine needed for game
• Should function when removing any aspect of package
• Dedicated server implementation in DOTS
• Built on top of IL2CPP
• Project Tiny
• Build instant games and experiences that are small, light, and fast
• Handful of project specific packages
• By 2022, DOTS fully integrated into main unity build

Q&A - AMA

Seth Gibson - Solutions Engineer

• UI Elements
• Version 3 of the current UI
• Performant
• Package manager
• NPM server but not supported all features; can't do authentication?
• Documentation - How to run servers and how to set up certificates
• Make it into a product / feature?
• VSP - Verified Solution Partner
• Addressable Asset system
• Mark assets "on S3 bucket"
• Asset bundles are opaque binary blob
• Can access asset by URI
• AES - Architecture, Engineering Systems

Paris use case - addressables, streaming, terrain, dots

Floating License - can be hosted locally

Enterprise Support - more flexible options - per seat

General Feedback

This would make our rehab games easier to be distributed from PC to mobile or VR/MxR devices such as Oculus Quest, HoloLens 2, and Magic Leap. They will have OpenXR SDK option in the editor, which would support Human hand tracking and gesture recognition on Microsoft HoloLens 2 and Magic Leap. This would help rehab work by providing a built-in solution for doing these types of tasks.

Microsoft visit

Contacts

• Seana Murray – HoloLens department seanam@microsoft.com
• Stephen Gordon – Gamin/Simulation DoD services stepheng@microsoft.co
• Robert Freeman – Microsoft DoD Solution Eng / Github dev engineer robert@github.com
• Lucas Boyd - DoD GitHub Sales Rep
• Brendon V. - PlayFab

Azure / PlayFab

PlayFab Goals

• Buy
• Compete
• Play
• Chat
• Watch
• Create

PlayFab Features

• Analytics
• Content
• Automation
• Players
• Commerce
• Community
• Multiplayer

“LiveOps” as a mentality

• Games are living entities - a Live Game
• Connected Experience
• A way of thinking about players
• Approach to product design and directions
• Drive games online -- take inputs of what players are doing to update game
• Live games + operations get increasing player engagement over time
• Sine-wave levels of difficulty
• Hard-easy-hard, etc.
• Feel relief after doing hard level
• Feel smart
• Gamification everywhere

Data is stored in PlayFab as events

Use Triggers on those events to generate unique content (eg, show different menus based on experience points)

Handles Login/Account Management

Use Postman examples to test functionality.  All PlayFab functionality is handled via Rest API.

Vanguard Outrider is a sample project, can be used as example data

3BCTU - code for enabling multiplayer hosting

PlayFab party service for voice comm

• Uses relay protocol or reliable udp

Telemetry Service - high-bandwidth data

Chat Service - but no audio storage

Working with Azure Functions: not currently, but is future work

Would like a PDF that shows all playfab features

Mixer - Live Viewing - future integration?

Video AI - future playfab integration?

Azure / PlayFab Feedback

The introduction to Azure was very basic (e.g., no one from the Cognitive Services, Machine Learning, or Data Science groups was there) so it is hard to say whether it would be useful or not. Azure supports a variety of databases including Cosmos DB, which has a MongoDB API. However, I saw no indication of specialized support for training. Specifically, the Learning Sciences at ICT supports the DoD xAPI protocol and currently we are using the open source Learning Locker as a Learning Record Store (i.e., database). It does not appear Microsoft provides any special support for xAPI or any kind of Learning Record Store.

Azure docs: https://docs.microsoft.com/en-us/azure/guides/developer/azure-developer-guide

Microsoft acquired PlayFab about a year ago. It handles: 

1. Real-time event streams (analytics, liveops, match making)
2. Telemetry (goes to data warehouse)
3. Chat (text/speech): speech gets transcribed via speech recognition. Original audio may not be warehoused.

The analytics are basic and focused on video games. There are a few common event types such as log in and log out, but generally, each game has a custom set of events. You can browse and search the events via a dashboard and see basic reports/graphs (e.g., histograms) as well as graphical displays (e.g., a heat map of player locations). PlayFab supports A/B tests (i.e., compare analytics from two different versions of the game). PlayFab supports web API calls.

Liveops are dynamic/intelligent changes to games to engage players. They defined engagement as attention while playing, and coming back to play again. One pattern is a sin wave in which players are challenged then given a break with some easier tasks. Liveops are driven by hand-authored rules triggered by spending patterns, engagement patterns, game performance, and k-factor (social influence).

Match making is logic for putting users together in a multiplayer game. These rules must be flexible and trade-off quality of match with time to match players.

The PlayFab has at least one training customer (Level Ex: https://www.level-ex.com) but for the most part there seemed to be no specific support for training. For example, we would have to implement custom xAPI events for every training application to be supported. Furthermore, there is no built-in support for machine learning only hand-authored rules.

• Free PlayFab accounts (with limited resources): https://developer.playfab.com/en-US/sign-up
• PlayFab docs: https://docs.microsoft.com/en-us/gaming/playfab
• Fake data generator for PlayFab: https://www.vanguardoutrider.com

Notion of LiveOps is interesting and somewhat aligned with what MxR plans to do with the SCOPE platform – utilize trainee/player data to improve training content design and trainee performance. However, MS is not pulling any biometric/physiological data.

Azure seems like a promising Cloud compute option for the DOD, but MS will first have to tackle security solutions before anything real can happen.

Playfab - I believe it would work on Bravemind by logging the button events on the control panel. It might help therapists or researchers or developers to understand the behavior of Bravemind users' usages to have more information either to give the feedback or to improve the software in the future, or do more research by analyzing this information.

Ravenswood Solutions

Stacy

Not part of Microsoft but was invited by Chris to give a presentation.  Subsidiary of SRI International

https://www.ravenswoodsolutions.com/capabilities

Their Mobile Ground Truth System collects data from Soldiers during live training (e.g., GPS, MILES) and can present data during AAR using ORION Viewer. They work with National Guard units and could potentially collaborate with us in future efforts and provide data.

Question:  Can we use this to make imitation models? How about better procedural/heuristic agents?

Need to followup with Chris to see if we can obtain any of their data.

HoloLens 2

Steven - Hololens developer answering questions

New hardware features (from HoloLens 1):

• Eye tracking - precise, updates frequently (compared to ML)
• Wider FOV
• Better balance
• Better response
• Better depth perception
• Better hand gesture controls

New software features (from HoloLens 1):

• Remote assist
• Mixed reality toolkit
• Arm compiled DLL
• Azure Remote Rendering
• Azure Spatial Anchors
• Local anchors

Could this be used for imitation training?  Yes, but eye tracking only to capture information from soon-to-be retirees

HoloLens 2 Feedback

HoloLens 2 is a much more usable AR device than the first generation of the product in terms of weight, field of view, and ease of calibration. The user interface accepts voice commands as well as tracking hand movements (e.g., you can grab and manipulate virtual objects). The main push seems to be procedural tasks in which trainees work in the real world with virtual assistance (e.g., highlights, blueprints, video calls). 

HoloLens 2 Demo was impressive, mainly insofar as the new form factor of the HMD. Lighter and more comfortable. The demos were good but they were under very controlled, nearly ideal, in-door conditions, so it was not easy to assess whether the newly touted increased field-of-view made much of a difference or if the luminance has increased. I think we need to get some units in-house to experiment with. 

From the demo experience, it is a big improvement for rehab work for daily life. For example, it has a wider view of the screen and finger tracking, which we could have users manipulate the virtual objects in the real world. 

GitHub

Robert Freeman - Technologist DoD/IC - engineer

Lucas Boyd - sales

GitHub

• 27 million developers - 0.35% of world population

DevOps/DevSecOps

• Developers + Operations
• Dev: Plan/Code/Build/Test/Release
• Ops: Release/Deploy/Operate/Monitor/Plan
• Automated testing using automated test tools
• CI = continuous improvement
• CICD = Continuous Improvement, Continuous Delivery/Deployment

Challenges

• Size of repos
• Storage of binary assets
• Still have issues handing diverse files types (binaries, images, videos) - optimized for handling source code
• Art / assets / binary
• Can store everything in GitHub
• GPR = Github Package Repository
• GIT LFS will help, but not solve - LFS = Large File Storage
• Alternative - Git for code / SVN for binaries
• Teaching non-technical people how to use it

Github Actions

• Built into CICD
• Trivy Scanner
• Vulnerability Scanner
• Nessus scan plugin capability, can trigger security scan from git commit
• Can incorporate build/smoke tests
• Can trigger on pushes -or- trigger on pushes with particular keywords
• Jenkins - "Is a little dated, but it works"

Github Connect - for the social aspects

xCloud

Play XBox games - streamed wirelessly on different media

content, community, cloud

Platform advantages

• end users - already has a base
• XBox game pass / 3rd party dev/pub relationships / Distribution partners
• XBox content
• other content
• lift n shift / cloud aware / cloud native content
• xcloud streaming platform

How to optimize experience

• managing provisioning
• hot loading
• inject signed on user

Hardware

• based on XBox hardware
• v1
  • 4boards per 1u
  • 1gb nic
  • ssd
• v2
  • 8 hovisized boards per 2u
  • 10gb nic
  • flash storage
  • centralized storage
• 5mpbs down / 1mpb up
• 720p
• Controller input it BT connected to client device, client sends input data to server
• Client hardware is extremely flexible - Just need good video decoder for streaming

Dates

• 5/3 - Dev Experience v1
• 7/29 - Content Version Flighting
• 8/13 - Dev Experience v2
• 8/30 - Cloud Aware SDK to Game Core
• Nov - Public Preview

xCloud Feedback

It is impressive that they seem to be able to run Xbox games on server farms and stream the graphics to mobile phones. No specific support for training management though.

Once we upload our Rehab App to xCloud, the software will become more reachable, especially for those who have a stroke. It will become easier for them to download and setup the system by themselves, and they will have the opportunity to do the treatment practice independently when they have the difficulty to commute between their home and clinical hospital. 

Project Acoustics

• What makes players and teams good?
• Halo knows best players
• Looking for purpose of balance
• Which players make other players better?
• NDA access to Acoustics is available

Random notes

Tencent - huge player in the distribution space

MS Projects: Bonzai + Airsim = Socrates

Super exclusive invite-only XFest show - November - London

Microsoft has a partnership with USC Environmental Science - for doing AI work.

• Can ICT utilize this partnership?

Digital Advisors at Microsoft who can assist Army / ICT

Azure Batch

Microsoft Game Stack  - youtube channel

Studios Production Framework

Summary Feedback

Microsoft focused the workshop on game development. Azure cloud services were mentioned and it is worthwhile exploring more, but they did not present anything more compelling than what we are already using. We spent much time discussing PlayFab, their new game analytics acquisition. However, its capabilities seem limited (e.g., no support for standards such as xAPI, no support for machine learning). Unfortunately, there was little discussion of Microsoft’s IVAS effort so it is difficult to say how much using these tools would help us with future IVAS integration. On the Learning Sciences side, I’d recommend a “wait and see” approach for these technologies.

Slides

Photos