The first session of the day was led by Joe Miller, VP of Platform Development and Technology. Joe's team includes QA, release management and what we have been referring to as 'maintenance' -- the improvement of features within Second Life that can be done quickly by one developer.
Joe's explained the development process, outlined Linden priorities, and asked for feedback on a variety of testing questions.
Follow through for notes, and please feel free to add you own thoughts in the comments.
Joe's session - Development methodology
Development: Linden Lab uses an agile dev model particularly suited to large software project like SL. Goal is to focus on smaller pieces that allow frequent releases, because large releases tend to bog us down (e.g. v1.9). This allows greater control for maintaining a stable platform. Good example: OS10 vs Vista.
Approach: dev resources grouped into studios so each group has a manageable number of projects, and a scaffolding to support the project execution. 5-7 projects per studio. 4 studios: 3 on platform one on web. Also a maintenance that focuses on smaller things, including bugs, that can be cleared out in a week. Project management is the glue that makes the whole process work. They communicate and advocate the projects to other parts of the company. QA is the final arbiter/gatekeeper on when projects are ready to go, and taking more responsibility for the stability of the platform. Dedicated member of the QA team for every 4 engineers. Currently 8 members of QA, plus outsourced support. Unit testing on developers to test a given code module to take risk out of functional test process. Regression testing on all new code, then Preview testing, then production grid. Any bugs that show up in the production grid are either dealt with immediately (crucial) or are put into maintenance.
Key governor to our ability to scale: performance and stability. This is our number one priority. Crash logs, engineering projects to increase performance on a particular class of machine, kill features that negatively impact performance. We're working closely with ATI, nVidia, Intel about our issues. We're doing things that no one else does. Our feedback to them is valuable because we're pushing the envelope.
Discussion
- a place on the production grid where users can put content to be tested which would show up automatically in the preview grid
- LL has focused on qa in the past, some usability testing. Need to think about what effect new features have on residents, not just functionality but usability.
- (LL) We know the viewer UI is complex and often unapproachable. Should we have 3 levels or modes in the interface, or even 3 different clients? EG builder, business owner, casual user.
- The UI is very difficult to teach to other users. Two building modes is very confusing.
- customizable UI could open up opportunity for different interfaces for different types of use.
- (LL) goal is to have an initial UI that's stripped down, then have to find a way to open up more complexity when they gain more experience
- would be cool to allow people to take over your UI show they can show you something
- a set of videos would be very helpful to walk people through the UI
- (LL) how do we test usability better?
- (LL) it's been difficult to open up land and currency purchasing in Preview because of needing to hit the external website, without touching RL billing. Now have that capability so we could test those actions.
- (LL) will be launching multiple previews. today the first phase of the extensible UI are going into preview. challenge is that we're running in parallel the grid with group and land estate sales tools.
- weekly downloads on Wednesday. can we have patches instead of full clients?
- (LL) that's a high priority, will only download bits that are needed
- (LL) need reminders about preview, maybe in MOTD.
- preview needs to be fun, an event, to get a broader audience
- I only go to preview if the feature affects something I work on
- attract audience who will use the new features, e.g. flexi prims. He tried to recruit fashion people for flexi prims but very few tried it. They could have had a competitive advantage. The ability to transfer objects (e.g. through download/upload) to the main grid would be a big draw.
Test HUD
Created a HUD that stores all public test scripts. Allows team testing to test group tools. Allows for much bester testing through metrics, tracking of failure points. Shortens the cycle of getting information to us by skipping bug testing; maybe voice mail in the preview grid
Q: How can we make it easier to teach the next wave of users how to use SL?
- sometimes changes we make cause older users to be very unhappy, expressed as anger in the forums
Ideas
- if you're relying on something and it changes it can be very disruptive
- there are 10 million people in front of us. we need to be ready to address them.
Q: What is the low-hanging fruit that we need to fix?
- is the known issues page helpful? there's a repro there so you can verify if it is the bug you're dealing with
- is the feature voting tool useful?
Ideas:
- now that there are so many alts there are too many duplicate posts, silly questions, and gaming of the voting system
- (maybe this isn't the best place because people can't find existing suggestions, changes to policy and social systems
- hasn't used it but people have indicated they want more feedback on the reasons why we're making these changes
- emergence of different classes of users in SL, subsets of different content creators, people who run businesses and have support systems, casual social users, sub-cultures are emerging that are unique even with their own chat idioms; the feature voting tool doesn't pay attention to how people use SL or how much; it needs a way to integrate this sort of information and weight the feedback.
- with respect to verifying identity, it ignores the fact that profiles offer a lot of information
This should have been a very interesting presentation on how the whole development and QA process works :)
It's also nice to see that the voting tool has been mentioned. Somehow, I feel that the current stage of SL development is very roughly between a "perpetual beta" and an open-source project — teams push ideas forward, they get experimented with, a larger audience (the Preview grid(s)) tests it out and gives feedback, then new ideas pop up at the feature voting tool — very amusing to watch and follow!
At this point I wonder who designs the specifications of a particular "project"? (ie. a new feature). With an estimated number of 50,000 programmers (from amateurs to pros) as residents in SL, and a large number of project managers and development team leaders, I wonder if the feature voting tool couldn't become more "professional" somehow? Perhaps evolving towards a "standardised" format of specification, with more (and better) comments/discussion for each feature... this would hopefully weed out the "cool ideas" that are so often posted but that are impossible to implement (in a reasonable timeframe and with a limited number of developers).
Posted by: Gwyneth Llewelyn | July 19, 2006 at 05:52 PM
That's right, Gwyn, let the tekkies run the society, let them make the features proposals incomprehensible, let them "weed out" ideas that they find just "impossible to implement," often for political/caste/economic interests of their own, let only a "limited number of developers" post, sure, because the rest of us feebs and choads are too ignorant to reason out ideas for features or have any inspirations unless we are not only certified tekkies but also on that oh-so-short list of "developers" who aren't "amateurs" (whatever *that* means in this neo-field of virtual worlds).
Oh, but then why even have such a page open to the public? Just hide it somewhere on a password you only give out in the IRC channel, and make it another Wiktatorship Wiktopia.
Posted by: Prokofy Neva | July 19, 2006 at 11:41 PM
considering the number of new releases and their impact on gameplay, i would like to see a little more inside the QA and release management processes. it was specifically said that new code is regression tested, but what about code that "shouldn't" change? how do you make sure you didn't break anything with fixes if you don't test old code (especially areas that are critical or have been historically problematic).
additionally, the last few releases have been messy affairs at best, which makes me concerned about release management and scm methodologies. what regression testing is done after new code is rolled out but prior to opening the grid to the public?
just some things i would have liked to have heard more about...
Posted by: Chris Kuhr | July 23, 2006 at 06:59 AM
At the moment there is a huge credibility gap between what is said above and what the users are experiencing. There is undoubtedly a lot of work which is invisible to the user, but there is also no doubt tha bugs are reported, remain with an update, are reported again. Residents have reported bugs on the preview grid, only to find the bugs intact when the version is implemented in-world.
If users are going to make the world, they have to have working tools to do it, and currently the tools aren't working. something is wrong with the QA and bug process, and needs fixing. Who was at this SecondLife views? They seems to have been rather tame, unless a lot of table thumping wasn't reported.
Posted by: Caliandris | July 23, 2006 at 11:01 PM