Meet the Press...

In what could be a series of fortunate events, I missed a plane on Jan 1st. So instead of flying, I was working when CIO Senior Editor was gathering opinions about another QA mess. The last one I knew of was from MySQL. This new one was from MS.

So I decided to chip in my two cents since not many people was supposed to be working, the published article is here.

ISB

I hope it is useful!!!

Total Quality Slow Down:

It is always comforting to see that pretty reputable companies of all sizes get to efeKctive via searches like: "total quality management in software industry".

It makes easier the solo/self-funding effort. The summer months will be slower than previous months. Too many hours sitting in front of the keyboard take a toll on anyone's back, wrists, etc.

Hopefully, my new surfboard will be here by our arrival.

So tomorrow, we are heading to Amsterdam to visit my sister. Pictures will be coming...

The Referencer

Is still working on fixing bugs. The logic to implement new/old method matching is uncovering a lot of bugs. Not as close to beta as I thought or would like...

5,000,000

After tweaking some generation logic to economize the memory consumption, DevRiot reached the 5 million tests mark in less than 34 seconds.

Sounds fast, isn't it? Now, really going back to suite serialization...

500,000 and counting...

The Suite serialization is slowly starting. I am polishing some performance features.

But I wanted share with the latest readers some of the brute-force finesse of the tool:

500,000 tests, no coding, no wasted time. Enjoy.

VUT Threads...

This is almost identical to the previous demo but with the difference that the Load/Stress testing engine is hooked up.

Note that the video follows the same script as the previous one, so no PrivateObject API or coding or scripting. Even the thread creation, execution, and harvest.

The other difference is that we intentionally tried to add a test without pre-compiling a parameter so the tool could complain :-] Three threads are created and used against the method under test.

To open source or not to open source...

I was im'ing with a good friend from Austin, catching up after a few months off.

She was somewhere on the west coast on a business trip. She mentioned that I should open source the whole thing. She is working on SOA a lot lately. The whole thing is hard to test. So buddy, open source it!

I have toyed with the idea before, and yes SOA is hard to test. But then one looks into WPF and the ball-and-chain (to paraphrase Jobs) Microsoft is putting on these widgets just to make sure they are testable and one realizes that before testing SOA we would need to make sure that simplest things work.

BTW, I am been trying to figure out for several days why would anybody want to take that approach to no avail.

Beta FAQ 2:

For those research units interested in what is coming down the pipe:

*) The .Net beta will include Gui-Off: that is the ability of running the gui execution engine off the screen to increase throughput.

*) The Test Writing Engine will not be included in yet. We need field-proven execution engines before design on the writing engine continues.

*) The eclipse/java version will be overhauled after the .Net beta is available: it will become an Eclipse feaure, etc.

Off-topic:

Well, one thing led to the next. It started with using .Net 3.0 and the GUI testing engine, and ended up in: "we need to make sure that the off-screen engine works".

The off-screen engine is a feature of the GUI testing engine that allows these type of tests to run at speeds close to that of the unit test engine. The main goal is to increase the throughput of the process without losing accuracy.

BTW, it works...

Prototipo; fundamentals, what fundamentals

The VS tool is coming along smoothly but a little bit slow. Phurnace's project has been quite interesting: started with a prototype with an rcp front-end and has turned into a WebSphere process server, web-services thing.

I have to admit that is highly interesting.

In this astonishing summer of news, there was this particular one about onshoring on cnn: an indian company was offshoring to Ohio some call center operations because the lower wages were not enough.

If the process and where economic value is created are not understood, there is little chance to success by trying anything else...

50 to 80 Marketing

Well, between working on DevRiot for VS 2005 and my friends' RCP project, I am having time to compare the two most used IDEs in the market.

It amazes me how the new bells and whistles added over time to these IDEs address so little the costs that are crippling most software shops. Testing and debugging shared 50% of the costs of a software project back when the first edition of the MMM came out. Now the same pair share up to 80% of the costs of a software project (according to NIST figures)

It is nice to have code-completion to the max but how does that reduce that 80%?

It seems that the game is about who messes things up the least, instead of who improves things the most.

In the bay for more than a day...

Due to unexpected family emergencies, we will need to extend our stay in the Bay area for another week, were it not for the sad situation this would be a great change in plans.

We will try to make the most out of it. Some fine sushi dinner is waiting for us...

The trip to Austin was as expected, one meeting went well (the RCP project one) the other a no-go.

This friend of mine really needs to get her QA process faster, more manageable, and easier on the developers. But they have already started using junit and when things got difficult they turned to xml.

It felt like the moving sand trap, the more they moved deeper they sank. It was also challenging to explain how one could write tests without writing code and not to spill the beans.

Anyways, we will be around the San Andreas fault for another week.

.Net Progress, back to Bay Area, and other great escapes chapter...

Well, this RCP work does take its time but it is fun and it helps us. I have started to get the new architecture on Visual Studio together. This a little snapshot of the toolbar. (Not particularly different from DevRiot for Eclipse. And that's is the idea: lowering the learning curves...)

I also have been in touch with my friend to put DevRiot through the grinder of DataMining development. Progress but it will take time to show.

On another front, I have been meeting with some researchers from Santiago to start looking at the "writing engine" (so users will do nothing but look at test results and develop)

Finally, after 18 months I will be back to the States. The rest of the crew has been back several times but not me.

We will be visiting San Carlos/Bay Area. A new nephew is on the way :-}

I will post some pictures of an early spring trip to Costa da Morte which was quite pleasant: great waves and great seafood :-}

Variable Load!

Interesting visitors to our blog lately. Some have registered in the beta program, some have not.

We are looking at ways to cleanly allow the user invoke methods with a variable number of parameters. So far we are leaning towards double-clicking the parameters tab to insert a new parameter state tab.

The design on Load & Stress is going too. Before we try to measure memory and all that, we should make sure the user agrees with the state the object is after the beating. It requires some thought.

Why all that bloating?

As I move along with the VS add-in of DevRiot, I started to look again at the built-in features of VS.

What a mess! To end up asking the user the same information that DevRiot directly asks, there is an endless sequence of menus, options!

Orcas is following the same path. Why does the user need to deal with code? Because it executes faster? Obviously not.

That automatic code generation just works with native types. After that is back to set/get methods to set objects ready to tests. Or play PrivateObject nesting game.

Does it provide GUI testing for PC or devices? Nope.

Does it analyze build trees or AST? Nope.

Again, it is a step forward but short. Otherwise people would not still be asking "why white-box testing is ignored"

Beta FAQ:

A few questions have come up:

Do you Support Swing?

Yes. Actually, we are far along in that regard. We just wanted to release something that worked in J2SE and J2ME uniformly.

Swing is not an official J2ME package so we decided to remove it from the release.

How does the Private members access work?

Think of it as Visual Studio's PrivateObject but on steriods. And for Eclipse, too.

It is always there working for the user. But the user does not know it. The feature is transparent and does not require coding. It is also immune to refactoring changes.

It also works for all the types involved in a test (gui or unit): parameters, fields.

It also works like a drill: the user can click into an object state as deep as needed.

Is it only the plug-in/add-in?

No. The plug-in/add-in has two things in it: the plug-in/add-in logic and the reduced version of the engine. The plan would be to provide an automation server where the output of the plug-ins/add-ins are consumed and value added stats are gathered.

This could be provided as a service or as an attachement to SCM.

Virtual Solution 2209

Well, the actual performance increase over Nunit is 2209 times. Our average cost per test is around 8 ticks, while for Nunit 2.4 is 1700+ ticks per test.

So, I decided to look at the Test tools offered by VS 2005. Although a step in the right direction, not even close to the streamlining that most software operations need.

It is still a solution based on writing code (as if figuring out how to make code fail does not take time), they have to learn a never-ending list of attributes.

No wonder, people google for "why+white-box+testing+is+ignored" :-}

It allows to access private members (fields and methods) but their approach seems a babushka doll. It works fine if those fields are native types.

But if those fields are something more complex? Can the users apply the PrivateObject trick to types not-under-test?

After all, the type-under-test methods are affected by the internal state of any of its internal state components. Ah, the babushka doll!

For load testing, Visual Studio does not "require" to write code. It just puts the all the tests the user already wrote in a bundle.

What about daisy-chaining tests from the data layer or business logic layer to the gui to verify behavior across the layers? Even some tools, like code coverage, are there with the wrong crowd.

Hey, I am just trying to differentiate our product! ;-}

600+ Faster

As we polish the new architecture in .Net, we decided to compare the execution speeds of Nunit 2.4 and DevRiot.

DevRiot is still cookin!! Just considering the overall times, not the average, DevRiot is in the order of 600+ times faster. Plus it is also a GUI testing engine that runs in Windows as well as PocketPC.

Why do we consider this important? Testing and quality represents a huge percentage, according to NIST it could reach 80%, of a given project costs. Vista is another example of this efficiency problem.

The startling thing is very few, if any, high-tech strategy courses or implementations really address this. So which kind of strategy ignores altogether 80% of the problem?

A former, I think, SAP CEO mentioned the software industry needs to copy the auto industry. McKinsey & Co has written articles about it too. Microsoft Research aims in that direction also. But the facts seem to say something else. Just a look at the different internet forums will confirm this.

So if we want to improve the situation we need to use the scarce human time in more efficient ways, and let the computers use their cycles in something useful.

Anyways enough ramblings, here is the snapshot of DevRiot .Net