The Creation Engine: Microthreading Mono

« Free To Play! Pay For Stuff | Main | Second Life Kitchen, Real Life Cooking »

May 05, 2006

Microthreading Mono

Being able to add an independent, event driven program to any object in the world makes scripting in Second Life really intuitive: you add a script and your object becomes an independent agent able to sense its world through event handlers, sit and think and then effect the world through library calls. It's a model that works equally well whether you're coding a fish which swims around looking for food or an information post that just hands out notecards whenever it's poked.

It's an approach that quickly leads to lots of running scripts. Some of the more complex regions constantly run 1000s of scripts. Abbotts Aerodrome runs around 2000 scripts and while most of these are of the give notecard when poked variety they could just as well be calculating PI or running SETI at Home, which means they all need to behave as if they were independent processes running concurrently within the simulator.

Supporting such high levels of concurrency has been one of the major challenges in integrating Mono in to Second Life. Although Mono and the CLI support the concept of independent threads of execution, each Mono thread maps on to an operating system thread which tend to be fairly heavyweight. By default most operating systems reserve megabytes of memory for each thread and don't cope with more than a few dozen running concurrently. Even reducing the allocated stack size to 32K and using the latest thread libraries on Linux doesn't get you much past 1000 running threads before the system starts to wheeze.

The other difficulty with Second Life scripts is that they can migrate between simulators while they are running. You are perfectly entitled to write a script which never stops running and then to tie it to a rocket that you fire in to another region. This is relatively easy to do when all of your script state is in a single 16K block, but much harder when your script has been Just In Time compiled to machine code and its state is scattered throughout memory, registers and an operating system thread.

The solution we implemented for Second Life was to use RAIL on top of .NETs Reflection and Reflection.Emit facilities to inject microthreading in to the script assemblies, an approach used by the JavaGoX and Brakes projects to implement mobile agents in Java. Our microthread injector searches through the script assembly finding points where the script should yield and inducing the types on the stack at those points. It can then inject extra opcodes in to the assembly which copy the stack in to a heap object and cause the script to yield. Whenever a script yields we can restore another script from its previously saved heap object and start it running again, allowing us to schedule many microthreaded scripts on a single operating system thread. By marking the heap objects as serializable we can then just use the standard .NET communication facilities to migrate scripts to remote simulators where they can continue running.

The approach is memory efficient as the memory needed to save the microthread is allocated when it yields, so a large amount of stack space doesn't need to be conservatively reserved up front for the thread. The problems with this approach is that it is expensive in CPU time spent switching microthreads and the space taken up by injected opcodes.

An alternative approach to microthreading Mono has been taken by Tomi Valkeinen who has written some very funky code (inspired by Eve Online's use of Stackless Python) to save and restore the native stack to switch between microthreads. This approach should be a lot faster than opcode injection. We can't currently use Tomi's microthreads for Second Life as the saved stack may contain pointers which can't be transferred to a remote machine, but it may be possible in the future to use Mono's garbage collector to identify the pointers in the native stack and twizzle them in to a form that can be sent to a remote machine.

It would be very cool to build a library which automatically switched between Tomi's microthreading implementation on Mono and Linden Lab microthreading on .NET. People could then build portable microthreaded applications which take advantage of Mono's native microthreading when it's available.

May 05, 2006 in Web/Tech | Permalink

Comments

What was the rationale for choosing Mono over Stackless Python?

Will

Posted by: William Wise | June 07, 2006 at 09:11 AM

It's address space, not physical memory, that most operating systems reserve for each thread.

Posted by: anon | June 08, 2006 at 08:11 AM

Because python is slow as shit that's why

Posted by: duh | June 08, 2006 at 11:34 AM

Perhaps, "duh", but your flippant attitude doesn't inspire confidence in your assertion. Stackless Python is used by the folks at Eve who run the largest non-sharded MMORPG and thus has a proven track record so I think it's a reasonable question to ask.

Personally I prefer mono and am glad Linden Labs is putting in the time to add these capabilities to mono. Nevertheless, I'd like to know what advantages mono will have over SP. If there are good reasons I'll personally lobby the Eve devs to consider using mono in the future.

Will

Posted by: William Wise | June 10, 2006 at 05:52 PM

Will, Stackless may well have been a good choice too. The stackless microthreading implementation is certainly mature and proven through its use in Eve. The attraction of Mono is that we want to support the use of mainstream languages to script Second Life alongside maintaining compatability with existing LSL scripts and allowing LSL to be used to develop new scripts in the future. The CLI and Mono are particularly attractive from this perspective. Hilmar Petursson, the CEO of CCP was very interested in the prospect of a microthreaded CLI when I spoke to him about it at the Austin Games Conference last year.

Posted by: Jim Purbrick | June 11, 2006 at 10:34 AM

I believe the decision for using mono is based more on then just threading. It's the JIT that most appeals to LL and garabage collection, as well as the common byte code (CIL) that make up Mono that makes it a win over python. Now if you like Python, you can always use IronPython. :-)

Posted by: Zac Bowling | June 11, 2006 at 06:01 PM

Well, if you can't change the LSL VM cause you don't own it, then yeah, I'd go with SP.

Mono is not what you're looking for..

Posted by: [email protected] | June 19, 2006 at 01:24 AM

I'm delighted you're thinking Mono. I think every day about how wonderful it will be to use it instead of LSL. I'm just not as big a fan of Python as C#. So don't be discouraged by these naysayers!

Posted by: Unoti Quonset | June 21, 2006 at 02:07 PM

Let me reframe the problem.. can you think of an environment similar to SL in which Mono has been a success in?

Posted by: [email protected] | June 22, 2006 at 12:29 AM

I like where this is headed, although I don't understand the comment that "Python is slow." Maybe Stackless Python is slow, but the Eve crew seem to have done fine with it. But IronPython and C# take exactly the same amount of time to chew through my ridiculous recursive Sudoku solver... whatever THAT number's worth...

On the topic of Mono, though, I understand LSL is going to remain the sole scripting language of SL for some time, with no ETA on other langauges being added... BUT I'm optimistic, and hope IPy makes it at least. As it stands, I'm using my own converter ("slython") to generate LSL from Py source because I got sick of having to call a function just to access a list member. =)

Posted by: Jon Molnar | July 11, 2006 at 01:36 AM

"I think every day about how wonderful it will be to use it instead of LSL."

I think it unlikely that you'll be able to use anything other than LSL even after the underlying implementation is using Mono.

What I'd like to know is whether their microthreading scheduler is still going to poll every script every frame for triggering events. And if not, why they don't start out by fixing their current bytecode interpreter?

Posted by: Peter da Silva | July 14, 2006 at 05:05 PM

With a few caveats you can use any language that targets the CLI to script the Mono version of SL now. I've written scripts in C# and CIL assembler already.

The biggest block to using other languages is currently Monos incomplete bytecode verifier, but that is being worked on. If we're going to allow scripts to be compiled using compilers we don't control, we need to be able to verify that the bytecode they produce is safe before we run it.

Posted by: Jim Purbrick | July 25, 2006 at 03:15 AM

"twizzle"? I think you're thinking of "swizzle" - but I thought that usually refers to mapping global address space pointers to local virtual address spaces, not the other way around (?).

Posted by: Cenji Neutra | September 14, 2006 at 05:23 PM

It seems the injection across sims with its slowdown or complexity is due to the native measures of datatypes and the methods to gather them into a packet. The problem is simple, it is an attempt to use a CIL that has been further compiled natively for its final execution. As a CIL is intermediate, the process needs to return back to the intermediate state with current stack and data before transfer of the packet. That wouldn't need to be done if the original CIL is not discarded. The original CIL provides the appropriate indexes an data maps, which is what is tried to be reconstructed as described in the blog. Avoid the reconstruction steps and keep the original copy as a map, anotherwords.

On microthreading... moving away from one thread per process - good. However, it sounds risky with any need to rewrite code on the fly and harder to debug. Again with above ideas of the original CIL, having the maps would also help context switches. It is a little extra memory overhead but with the feature of much greater speed improvement. Once a packet can be made for like transfering sims, a similar packet can be context switched. The difference between the packet transfer and a context switch is that you don't need to check dependencies in a context switch, and you can expect memory structures already exist for each index in the maps - very fast in actual sims.

Just sharing some trade practices...

Posted by: Dzonatas | October 01, 2006 at 06:12 PM

best penis enlargement pill http://www.marviva.org/chs-bin/msboard.cgi?ID=SOLONAUTICA&msg=56498 - - fda approved penis enlargement pill
natural penis enlargement pill http://www.marviva.org/chs-bin/msboard.cgi?ID=SOLONAUTICA&msg=56498 - - australia enlargement penis pill
penis enlargement pill review http://www.marviva.org/chs-bin/msboard.cgi?ID=SOLONAUTICA&msg=56498 - - best penis enlargement pill
do penis enlargement pill work http://www.marviva.org/chs-bin/msboard.cgi?ID=SOLONAUTICA&msg=56498 - - penis growth pill
http://www.marviva.org/chs-bin/msboard.cgi?ID=SOLONAUTICA&msg=56498 - Penis Growth Patch Rx

Posted by: pinismanow | July 30, 2007 at 05:50 AM

Three phrases should be among the most common in our daily usage. They are: Thank you, I am grateful and I appreciate.

Posted by: penis stretcher | August 04, 2007 at 01:56 AM

That ad's interjection, was an inverse wavelet echo from some other interSIM perturbation?

Funny....

Posted by: cognicio merlin | August 09, 2007 at 04:34 AM

If you don't want to see this links, please email me dnsk_mail [ A t ] yahoo.com
And give URL of your site. Snxs.

Posted by: Stive Angelo | August 24, 2007 at 07:32 AM

Posted by: ronelos | August 31, 2007 at 02:08 AM

ao credits
Tales Of Pirates gold
Tales Of Pirates money
gaia online gold
gaia gold
gaia money
last chaos gold
last chaos money
hero online gold
hero online money
buy hero online money
hero online powerleveling
hero online power leveling
eve isk

Posted by: qq | September 05, 2007 at 11:34 PM