Available!

We are happy to announce that DevRiot for Visual Studio 200X(Beta) is available for downloading at:

http://efekctive.com/beta_public/

The documentation can be found at:

http://efekctive.com/docs/

Don't forget you will need a working email address, a working internet connection, and a SQL Server instance available to install and use DevRiot (the email address is required during setup only)

We hope you find it useful and worth the time to try it! If you have any questions please contact us at this address: beta at efekctive dot com.

Rewiring in progress!

This is a snapshot of the gui testing logic partially rewired. As mentioned, it does not rely on capture-and-replay, and shares the same user interface as the unit test logic.



BTW: What is going on in the open source world today?

Except me, Mac!

These are a couple of snapshots of DevRiot Exception handling. It allows to click on one the exceptions thrown, and to verify that the exception gives the expected message. I have seen released product display pretty dummy messages :-}

I will jump to the gui testing engine as soon as some problems between Eclipse and Mac OS X I am experiencing are solved...

Selecting an Exception

Setting the expected message

Parameters more, parameters less:

This a "show-me-the-progress" couple of snapshots. The second one shows a slight change from the initial approach to parameters. Now everyhting, except Exceptions and Load/Stress, will have the same tree approach.

Now off to wire the GUI testing engine with the rest...

One Click-Interfacing

The following 3 snapshots (all from Windows, it was Sunday night, what can I say?) show the flexibility of DevRiot when handling interfaces, abstract classes, and Object types.

Snapshots 1 & 2 show how easily is to nest these types while describing the pre-state, post-state of an object. Even parameter generation works this way.

Ideally, the tool should generate these types on the fly if the user does not have them handy. But this would future feature :-}

Snapshot #1

Snapshot #2

Snapshot #3

One Click away - Too much noise!

The following screenshots show how DevRiot takes away almost of the work from the user. Everything is just a click away. Well except for interfaces and abstract classes where the tool needs some guidance from the user.

On Mac OSX, about to set pre-test state of the private field:



On Windows, showing how DevRiot deals with compiling errors:



On Linux, finally a success!



Well, given how many hits the Amber postings have taken in the last few days and the people reading them, it makes me wonder if it should hibernate at all :-}

Amber's hibernation, DevRiot update

There are a couple of progress snapshots of DevRiot's new UI: precompiling code and related tasks. Although, sometimes is discouraging put too much effort on the UI since QA tools should generate tests automatically, data and plumbing work :-} but...

Amber is in hibernation, I do not know how deep. The Mono people started a direct implementation of Indigo/WCF. Amber's goal was to avoid the IP restriction imposed on WCF/Indigo but since there is an open-source version of them on its way...

Their work is part of Mono Olive, they seem to be cruising on that front. There is also a google group for Mono Olive.

Well that's all for now...

On Mac OS!!

Although, given the problems that exist between Mac, AWT/Swing, and SWT, it will be a while before we could support this configuration.

Improved UI on Linux and Windows!!

Well, the snapshots managed to have some text to go along with them :-}

During this time we have had some useful feedback that allowed us to change the UI of the Eclipse plugin for the better, outside and inside. It looks more integrated with the Eclipse IDE

We had been able to delete a bunch of code that was more fat than substance.

The Linux snapshots show how devriot deals with pre-test conditions and post-test conditions when invoking a constructor.

The pre-test conditions are not allowed while the post-test conditions are. Not big surprise there. Still the state of the object can be accessed/set through the UI without access restrictions.

The Windows snapshots show how methods and its invoking parameters are dealt with. A series of tabs and useful data about the type at hand will be available to the users.

The Eclipse versions that will be available for the beta are 3.0x, 3.1X, and 3.2. The operating systems will be Windows and Linux/GTK (I have not run the GUI logic in a Motif box, so the behavior there is unknown), After this the .Net version will come.

Thanks!!

Looking at the wide variety of organizations downloading the whitpapers and postings, specially the more technical ones, it is encouraging after a relatively "light & silent" period where some dead weights had to be let go.

Sometimes less is more :-}

The beta of our java QA engine is coming together: the footprint is still around 50Kb (full, stripped down less than that) this should be enough to get inside a good number of handleds, PC, etc.

The advantage of avoiding the capture & replay model is that there is a lot of room for becoming lean & mean :-}

Typ-o:

the actual spelling of the path is:

http://efekctive.com/blogging/wp/quality_advantage_V2.pdf

Web Advange!!

The downloads of the whitepaper keep on coming which is nice to see. After all I have been in stealth mode lately.

I do not know if it was because word of mouth or a typo but there have been a bunch of attempts to download

http://efekctive.com/blogging/wp/quality_advange_V2.pdf (which does not exist)

the actual spelling of the path is:

http://efekctive.com/blogging/wp/quality_advantage_V2.pdf


Hope this helps...

The Keystone cops strike back! (Addedum):

The other tidbit of wisdom that Deming left around to be used was the fact that quality is in the process not in the product.

The Keystone Cops strike back!!

A couple of weeks ago, there was this exchange on one of the usenet forums where the original poster was at a loss asking for help. He was the only and new QA/QC in a software software shop that had 30+ developers.

Tall order. A lot of people chipped in, myself included. But there was one posting that struck me because it was odd. The post argument was that because QA/QC is a "support activity" (that encompases with debugging around 80% of any software project) the solution was for the guy to start his own quality movement in the hope that help would jump in after a while. Apparently, this approach was suggested in a tech book.

If one were to contrast that approach with Deming's TQM (something hard to argue against: Toyota, etc...) is a failed Quality 101. Deming's work proved that quality has to come from the top of the company to be really successful.

So what is a "support" guy to do?

Then the answer came through this posting on GK blog: the fad du jour. It is not a secret that a lot of companies manage by fads. These ones do come from the top.

So until quality becomes the fad du jour (FDJ) and stays that way long enough the "support" guy has to go with the FDJ, or risk being fired, or quit.

And top management will not support Quality until...

The cycle repeats itself: Vista is nowhere to be seen and this is the most visible one. What about the ones that do not get as much press coverage?

Real-Politik plus 150

Reading The Economist's article about MS and the beating that its stock price took, I couldn't stop thinking about the realpolitik that should be going on inside.

Traditionally, within an organization, groups become important and gain budgetary weight if their size increases. It is a matter of visibility.

This is important because the leader of that group, say the VP of engineering in a software company, will get stock options or bonuses depending on that visibility.

People are rewarded by the responsibility and visibility they seem to have, not by the improvements in their groups. So, for a Director of Quality Assurance of Acme Software, having lots of people is the way up.

And for this Director, becoming more efficient by testing more with less people is not a particularly enticing proposition. Especially if the credit for that improvement goes to someone else.

If on top of that there is a chance that in the past, some releases were less than acceptable and the CTO or VP needs to do some cyb'ing, what we have is the makings of a royal mess: The VP/CTO trying to cover her/his back, the Director of QA protecting the turf, and a product's quality going down hill.

It woudn't surprise me that something like that is going on inside MS.

_________________

Well, we have reached 150 downloads of the white paper.

This number is more than we expected, and we appreciate the feedback we've gotten.

We would love to see more feedback. Please feel free to email me directly with anything, we mean anything, you have to say about the paper.

http://efekctive.com/blogging/wp/quality_advantage_V2.pdf

Good Ol' Bean Counter, where art thou:

There is this study published few years ago by Cap Gemini Ernst & Young which states the following: 65% of the in-memory image of a software program belongs to software developed by a third party.

Put this figure together with the NIST stats (80% of a project is testing and debugging) and the magical number is 52%. Any software project could use up to 52% of its resources (time, people, etc) executing somebody else's software.

And doing the marketing research update for the BP, I ran into Mercury's QTP 8.X.

This tool allows users to continue using capture & replay through another layer of indirection (their ActiveScreen feature)

So a capture & replay user just interested in:

button1_clicked(because is the only logic that she/he can modify)

has to wait for the tool to execute who knows how many screen repaints, OS and specific tool calls (all of which are third party) to get to the real test target:

the fabled "button1_clicked".

How many more tests could be executed instead? Aren't we always short of time for quality? Are we using the time we have wisely?

On the same token, if we could get rid of the extra layers of indirection. Could we test everything all the time?

The Mythical Man Month considered this not economically/physically feasible. But back then the computing power was in its infancy and most of the production of software was done manually. Is it possible now?

Software Development & Europe, a mirror

I am about to hit the BP again but could not stop thinking an exchange that I had with someone on usenet. This person who also teaches proposed that automated testing is not relieable and the only solution is manual testing (not exactly with these words but that's the only conclusion that could be drawn)

But when confronted with the economic reality of the fixed physical throuput capacity of humans got into trouble: what to do with a growing codebase? was he going to force people to move their hands and eyes faster? So his only solution is to use more inputs to produce the same good?

This is bad lesson in economics 101. It should take less inputs to create a good over time. That is in essence efficiency.

This is the same approach as "pair programming": adding more people to see if they make it work. If recent ship date delays are a guide, adding people to a flawed process just does not work.

So here we are several years after the tech bust we are, in short, running to stand still:

Manual testing advocates think they can get it right after 25 years.

Traditional testing tool companies have "somewhat fixed" the problems they created (quoting a Forrester report) But what about non PC gadgets? Can they run inside a Pocket PC?

An XP advocates think that the solution is to add more people to:

1. Ask developers to write their tests, as if they had time to write their own code.
2. Ask developers to add special code so they can write their own tests.
3. Ask testers to write code that is going to become useless in the next round of refactoring.

And then came France and its PCE (first employment contract, or close enough) the students and the unions kept the PM from turning PCE into law. The PCE was an attempt make France's laber market more fluid and create jobs.

PCE is not a law which may have been a bad law but France and Europe in general need something along the changes that are needed in the software industry:

The population is growing old, so more immigration or more natality is needed.

In 2004 just Spain had positive demographic growth.

According to the OCDE, just 2 european universities are among the top 20/top 50 in the world. The rest are in USA, Japan, India, etc.

These two are english. According to a survey published recently when Europeans emmigrate, they mostly choose England because of the jobs and pay.

With China and India growing into world powers, what is Europe going to bring in the future to the table of big decision makers?

Education? Job opportunities? a healthy work force?

Anyways that business plan is waiting...

About the Keystone Cops, The Cart and The Horse:

The vast majority of the software projects that I have seen, in one way or another, always made me think of the Keystone Cops (the early 20 century comedians) trying to compete in F1 with a car with square tires.

It doesn’t matter how hard the Keystone’s try to push, or how many of them are pushing, the car will not run at all or fast.

If the tires were round, the requirement of Keystones will be less and there will be a chance of winning the race.

Now let’s go back to software. What happened during the last 20 odd years?

People trying to reach first mover advantage set up companies that could not maintain that advantage. In other words they set up themselves for failure in the future: near, medium, or long.

Everything needed to be fast: quality? Effieciency? Next release. But once the ball got rolling, who was the one to improve the process when there were two more deadlines in the coming months?

So the sound marketing principle of “It is cheaper to keep an existing customer happy than acquiring a new one” went down the drain by consistently having quality problems.

The MS case is interesting because they are the ones who tried to use the 1:1 ratio or something close to it. But these inefficiency problems are everywhere.

I guess this event it is going to put a new twist in to outsourcing. Why look for cheaper labor, in the hope to increase the head-count, if increasing the head count does not solve the problem?

I do not have anything against outsourcing. But does it make sense to outsource a flawed process? Do we need more people or a more rational use of all the resources at hand?

The economic impact of delayed schedules and bad quality software (lost sales and extra expenses) should be measured not only for MS but for the whole industry. I guess the figure would be mind-boggling.

On to other things: I just found a beta version of VS .Net 2005 (a year to use after installing) Hopefully it would work still, so I can go back to our .Net engine or Amber.

I was just checking the new Junit release (4.0) It’s looking like Nunit but in Java.

But the same flaws remain in both: why would a user care about anything else but the input and the output of a test? Why expend so many resources executing the inner workings of these two when so much logic under test remains unexecuted in gthe overall process?

Correction: the new engine is faster (25-200 times)

We have been updating the performance comparisons of our new architecture against the competition. As mentioned before, we currently have numbers for the java/eclipse product only.

The performance comparison was to measure how much time would it take to execute a suite that exercises 19 methods of the form:

public XX ReturnXX(XX param){
return param;
}

Where XX is an native type or its equivalent in the java.lang package. We ran one set where every test was a failed one and another set where all were successes.

The idea was to measure how efficient the engines were at handling different amounts of work. A failure should take more work than a success because the engine has to report where the failure is.

The numbers are better than we thought. In the all-failures scenario our engine is, consistently, 25 times faster than Junit. In the all-successes scenario our engine is 200 times faster than Junit.

With the advantage that we can handle all types of access to methods and fields, and strict gui testing.

We will be updating the white paper with these new figures and snapshots of the product, soon.

Well, that's all for now...

A Snapshot of the Eclipse plug-in!!

NOTE: The GUI capture here is obsolete for a newer version please visit:

http://efekctive.com/picture_library/blogging/devriot-ui-new-linux1.gif

Copyright José Cornado (c) 2005-2006

There are a couple of things to notice, we think:

The user just needs to provide the parameter and expected return values. The rest is optional information: an ID to execute the test in a particular order, pre-conditions and post conditions of the object under examination, number of times to execute, and load/stress level.

The method shown is private, but our technology can still reach it. It is really poweful and dynamic. One of the advantages of this feature is that your QA investment is immune to Refactoring.

As several people have noted over different lists and forums, Refactoring can (and does) make public or protected methods private.

In this case the QA efforts are lost because Junit, or Nunit for that matter, just can not handle the change. What to do? wait and let the code base grow without testing? risk to lose several weeks of QA development? or just do not use refactoring?

Another is the ability of testing even the exceptions (type and contents) thrown by the logic under scrutiny.

Also, this is the same interface that will be used to handle GUI testing. Nothing to capture :-}

The new architecture/engine is fast

The new architecture that I mentioned in the previous post refers to our automated testing engine. Please note that:

• It is a testing engine, not a GUI only testing engine or an unit-testing only engine. It's both, so users have to deal with one test tool only, we hope.
• It works as an IDE plug/add-in or as a automation server.
• It does not force Capture & Replay, writing code or scripts on the user. It uses what we call Reduce Input/Output Testing.
• It is platform and OS independent. For example, the gui tests created under Linux can be executed in a Windows environment.
• It is language independent.

Now the numbers: the average execution time per test (unit or gui) of our Java product is less than 1 millisecond. Junit throws in the same environment a lower limit of 20 milliseconds per test.

This is a performance increase in excess if 2,000%, so imagine that your QA technology performed 2000% faster?

To put in a nutshell, our goal is to provide a technology that minimizes the users' effort (input), minimizes the post-test creation maintenance (input), and executes faster than any competitor (output) thus making the user more productive.

The .Net version, even though is still under the old architecture, offers similar performance numbers when compared against Nunit.

Well, that's all for now. Hopefully, I will be able to get back to Amber and .Net pretty soon...

BTW, our most recent white paper is this (password: AUSTIN).
(The images are old though. We will update them shortly)