On Mac, again!

A renewed interest on DevRiot for Eclipse (Mac) has appeared in the last couple of weeks which is great. 

The problem was that the demo link inside the whitepaper was broken. The link has been fixed now. I apologize for the incovenience. Enjoy!

White-paper

Demo

May I Compare...

It is funny but blog-post.html from 05/07 seems to be pretty popular around the globe these days. I wonder why it is so special...

And it is interesting how readers compare the execution speeds of DevRiot .Net with DevRiot Java, specially since the Java data comes from a 1,000 tests run on Mac PowerBook (PPC) .

I have to admit that I have been redoing some code that looked good few weeks ago and now not so much...

Slow drip...

Sometimes things happen in a way that makes you wonder.

I have not worked on DevRiot's eclipse plug-in in ages. Obviously, I am concentrating on .Net since Visual Studio 2008 is going to be released soon.

But the funny thing is that the demo of DevRiot running the GUI testing engine on J2ME has become pretty popular this month.

How information flows in the business world is amazing!

Needless to say, it is going to be a lot easier to implement the GUI testing engine for .Net than for J2ME. Everyhting is already in the IDE (VS 200x) On the other hand getting the same device support on Eclipse/J2ME is like secretively impossible. Back to work...

Variable Load!

Interesting visitors to our blog lately. Some have registered in the beta program, some have not.

We are looking at ways to cleanly allow the user invoke methods with a variable number of parameters. So far we are leaning towards double-clicking the parameters tab to insert a new parameter state tab.

The design on Load & Stress is going too. Before we try to measure memory and all that, we should make sure the user agrees with the state the object is after the beating. It requires some thought.

Beta FAQ:

A few questions have come up:

Do you Support Swing?

Yes. Actually, we are far along in that regard. We just wanted to release something that worked in J2SE and J2ME uniformly.

Swing is not an official J2ME package so we decided to remove it from the release.

How does the Private members access work?

Think of it as Visual Studio's PrivateObject but on steriods. And for Eclipse, too.

It is always there working for the user. But the user does not know it. The feature is transparent and does not require coding. It is also immune to refactoring changes.

It also works for all the types involved in a test (gui or unit): parameters, fields.

It also works like a drill: the user can click into an object state as deep as needed.

Is it only the plug-in/add-in?

No. The plug-in/add-in has two things in it: the plug-in/add-in logic and the reduced version of the engine. The plan would be to provide an automation server where the output of the plug-ins/add-ins are consumed and value added stats are gathered.

This could be provided as a service or as an attachement to SCM.

Beta Available for Download

DevRiot for Eclipse Beta is now available for download.

Some notes on the product:

• DevRiot is a testing engine that combines unit testing and GUI testing (without the problems that capture-and-replay cause).
  
• It does not require coding or scripting. Tests are automatically created from minimal user input.
  
• It allows the user to reach any member without regard to their protection level. It also has an internal maintenance process that keeps the tests continuously updated.
• It is small: 56Kb (plugin + engine). This makes it possible to run on both J2SE and J2ME devices.
• It is fast: approximately 200 times faster than Junit.
  
• Features planned for future development: support for arrays and generics, EJB(with dynamic type generation), WS, and JDBC.
  
• Both unit testing and GUI testing interfaces are available for Linux and Windows. Currently, only the unit testing interface is available for Mac users.

Your feedback is essential to us, so we are asking those that download the DevRiot Beta to fill out a short registration form. We will not share your information with anyone - promise!

Depending on the version you are evaluating, you can contact us with feedback or questions at the following addresses:

• beta_linux AT efekctive.com
• beta_mac AT efekctive.com
• beta_windows AT efekctive.com

Please state in the subject line if you have a problem to report [Bug], or a feature request [Feature].

Any general questions can be sent to: beta AT efekctive.com.

Thanks for your interest in DevRiot!

Current beta is DevRiot for Visual Studio 200X

DevRiot .Net gets going, Beta for Eclipse almost available!!

3 People got their hands on DevRiot for Eclipse 3.X (Windows version) meanwhile I decided to get the .Net version a jolt.

Here is a snapshot of the GUI in the .Net environment. Pretty much the same as in Eclipse/Java. That's the idea: no new tricks to learn, no new tests to write. They are portable right off the start.

One down, two more to go...

The first install image of DevRiot is ready, some more sanity checks while I work on the other two and the documentation before it is ready for downloading!

DevRiot for Eclipse Beta 0 = 55.5 Kb

It is not bad, isn't it? To hold the plugin logic and the engine logic of a Unit/Gui testing tool, that is. Much of the footprint is the plugin/user interface logic. The engine is a reduced version.

A full fledged engine should not be much more than. It should fit in some devices, or it could be customized :-}

Well, next in line are the installers and documentation...

Micro pudding!!

As we get closer to packaging the beta, features become more usable. This sample video shows DevRiot running under J2ME CDC.

It is a GUI testing only video but the unit testing engine works fine as well. We previously mentioned that our approach to J2ME was coarse or brute force. In fact, we just run the Ide under J9 to force our plug-in to use.

As you will be able to see: it works and it is stable. It would be nice to have access to some emulators with Robot-like support but the main point is there.

Finally, Dynamic building update!!!

After a lot of trying on video formats and embedding, we have a another video update, we are pretty close to work on the installers. Things have smoothed over a lot.

We will create a couple of tests: one for the text field with a simple verification, and another for the button that is a little bit more complex. Here we will try to verify that after a click the field "AbstractMe"'s "ReturnMe" field has an arbitrary value.

Then, we will change projects to delete "ReturnMe" from Abstract.java and change back to run the suite again and witness that the tests were rebuilt automatically.

So if it was already an effortless task to create those tests, combined with the widget hound technology and the trimming/dynamic building, the net result is an extremely effective way to let people concentrate their time on more value added task.

I hope you enjoy it :-}

Please bear with me...

The internal build chain is pretty stable right now.

If a test case TC for type A in project P refers for some reason type B in project P1, any change in B is communicated to DevRiot and its internal build process (the trimmer to be exact checks any) updates TC with those changes.

We need to add a couple of buttons to the plugin so adding, running, and loading tests cases are separate tasks.

We are also changing the layout of the blog a little bit :-}

The Widget Hound!!

The progress has not stopped around here. We have another little video to share meanwhile other features steady a little bit more.

This time we have a video of a feature called "Widget Hound". It is in WMV format and is less than 400K. The feature shows one of our approaches to test problematic GUIS.

In the video we will show a button-clicked method that every time is invoked, it changes position. In a capture-and-replay model this would break a lot of things, but we can rest assured that DevRiot can handle it.

The story line of the video is as follows, with the cursor already placed on the target method.

1) Launch debug session an click 3 times the button to show changing positions.

2) Launch DevRiot and a test case that generates the same 3 button clicks.

As you will be able to see, DevRiot can chase the running widget down and honor the clicks required by the test case. The maintenance cost of test suites drop dramatically with this approach.

The link to the video is here...

Code Generation + the perils of Political Marketing...

Well, apparently the Spanish government is starting the contacts with other political forces to create an anti-terrorism policy that has wide support. The funny thing is that they said it as if they were talking about where to buy bread. They are running a country.

So, what were they doing before? No back-up plan? After the 3rd broken cease-fire? Thinking that they were smarter, more illuminated than the rest of human kind?

The funny thing is that they take the same, or worse, attitudes of the foes they most criticized to get into office. Does it sound familiar?

And no, I do not like early elections. These are matters that transcend a poll.

Well, like a marketeer friend of mine, who has worked in political campaigns, said: there is so much you can do with the product you got.

I am currently working the IO/Instrumentation Units compiling/Trimmer. It is coming along fine. It was not as stable as I thought it was.

Running natively on Linux...

The following link will take you to another video of DevRiot running on Linux. Notice that by getting around the limitations of capture-and-replay, things are a lot simpler. For instance, there is no need to use Windows to test a Gui in Linux, or use JavaBeans for that matter.

The same applies to other devices like PDA, etc.

Well, here is the link

A small video sample ...

This is a small video of DevRiot Gui Engine running on Windows. The action is:

1. Clicking 6 times on a Button and verify that t == 7. This verification fails because the tool clicked 6 times only
2. Type 3 letters on a TextField and verify that t == 9. This verification succeeds.
3. Delete the logic handling the clicking on the button, and work on the Choice widget.
4. The "trimmer" kicks in, finds out that the first test is not useful anymore, and deletes it. The tool points out the obsolete is deleted
5. Select item 5 in the Choice widget and force a failure to show that the event actually happened.

The interesting things are: there is no need to capture-and-replay anything. And when the underlying code changes, the tool updates the suites automatically so there is no maintenance.

This is the link. It is in Windows Media format and a little bit over 1.5 Mb

Happy New Year!!

Pre-Christmas gift!!

Well, the GUI testing engine (inside the eclipse plug-in) still does not work in Mac OS X. Despite upgrading to DP1. I guess the SWT/AWT compatibility has some more tweaking ahead.

On the brighter side, I am finishing wiring what I call "the gardener": code changes, method signatures change, methods are deleted. The gardener takes care of this by keeping the tests up to date with the real code.

I am still shooting to get a GUI engine up and running for Christmas :-}

Movie time!!

This is a little demo of DevRiot - Eclipse running on a Mac. The movie is in QuickTime format and about 4.5Mb in size.

It shows DevRiot at work:

1. finds where the cursor is
2. picks the method to test
3. generates a test form
4. lets the user fill the form with ease
5. add and run

Hopefully for Christmas we will have a similar demo running the GUI testing engine. click on the image below to launch the demo:

Not so fast son!!

Well, the swt/awt/apple problems are not solved with the compatibility library released by apple. Eclipse keeps on crashing and when it does not crash the AWT/Swing widgets-under-test miss events or get them too late.

I know the problem is not the gui-engine logic because it delivers the events orderly and timely on other widget types (SWT, ...)

So DevRiot will not support GUI automation on Apple for now. It will support the unit testing automation which runs good.

Who wants to get in the middle of the fight of this two? Not us :-}

1001 Tests...

Just to see where the engine started to show signs of effort, I decided to wire up some stress/load logic. I used a simple method that takes an int and returns it.

The rest is on the progress snapshots:

Enjoy :-}

The Proof is in the pudding...

This is a snapshot of DevRiot running on Eclipse 3.3M3 and Java 1.5. Note that the engine executed 100 tests and all of them failed. Despite the extra effort of reporting the failures, the little engine still needs a good set of brakes:-}

We are getting closer to the point where we can release this eclipse/java beta. We do want to move on to the VS/.Net beta because the performance gains here a orders of magnitude bigger than in Java.

Stay tuned...

Still rocking on java 5.0!!

We just got the first taste of DevRiot running on java 5.0 and we like it. The speed (ours) is still blazing fast.

There are a couple of bugs in java 5.0 that were affecting DevRiot and needed to address so they didn't hurt performance wise. That was done.

We will try to show how fast it goes before the weekend is over... :-}


Sorry for the unusual formatting. Blogger is not showing the formatting toolbar at this moment!

A shorter update...

Still wiring the gui, unit, and activation logic. The activation logic will work like this, or close enough:

When a user clicks on the DevRiot button if the currently selected file is a source file then it continues, if the cursor is NOT inside a method then the tests are run. If the cursor is INSIDE a method, the form (that is on the screenshots) reconfigures itself and lets the user add new tests. When the popup is closed then the tests are run...

Back to coding :-}

Rewiring in progress!

This is a snapshot of the gui testing logic partially rewired. As mentioned, it does not rely on capture-and-replay, and shares the same user interface as the unit test logic.



BTW: What is going on in the open source world today?

Except me, Mac!

These are a couple of snapshots of DevRiot Exception handling. It allows to click on one the exceptions thrown, and to verify that the exception gives the expected message. I have seen released product display pretty dummy messages :-}

I will jump to the gui testing engine as soon as some problems between Eclipse and Mac OS X I am experiencing are solved...

Selecting an Exception

Setting the expected message

Parameters more, parameters less:

This a "show-me-the-progress" couple of snapshots. The second one shows a slight change from the initial approach to parameters. Now everyhting, except Exceptions and Load/Stress, will have the same tree approach.

Now off to wire the GUI testing engine with the rest...

One Click-Interfacing

The following 3 snapshots (all from Windows, it was Sunday night, what can I say?) show the flexibility of DevRiot when handling interfaces, abstract classes, and Object types.

Snapshots 1 & 2 show how easily is to nest these types while describing the pre-state, post-state of an object. Even parameter generation works this way.

Ideally, the tool should generate these types on the fly if the user does not have them handy. But this would future feature :-}

Snapshot #1

Snapshot #2

Snapshot #3

One Click away - Too much noise!

The following screenshots show how DevRiot takes away almost of the work from the user. Everything is just a click away. Well except for interfaces and abstract classes where the tool needs some guidance from the user.

On Mac OSX, about to set pre-test state of the private field:



On Windows, showing how DevRiot deals with compiling errors:



On Linux, finally a success!



Well, given how many hits the Amber postings have taken in the last few days and the people reading them, it makes me wonder if it should hibernate at all :-}

On Mac OS!!

Although, given the problems that exist between Mac, AWT/Swing, and SWT, it will be a while before we could support this configuration.

Improved UI on Linux and Windows!!

Well, the snapshots managed to have some text to go along with them :-}

During this time we have had some useful feedback that allowed us to change the UI of the Eclipse plugin for the better, outside and inside. It looks more integrated with the Eclipse IDE

We had been able to delete a bunch of code that was more fat than substance.

The Linux snapshots show how devriot deals with pre-test conditions and post-test conditions when invoking a constructor.

The pre-test conditions are not allowed while the post-test conditions are. Not big surprise there. Still the state of the object can be accessed/set through the UI without access restrictions.

The Windows snapshots show how methods and its invoking parameters are dealt with. A series of tabs and useful data about the type at hand will be available to the users.

The Eclipse versions that will be available for the beta are 3.0x, 3.1X, and 3.2. The operating systems will be Windows and Linux/GTK (I have not run the GUI logic in a Motif box, so the behavior there is unknown), After this the .Net version will come.

Dynamic Array Manipulation Snapshot!!

Slowly but surely the dynamic array manipulation through DevRiot's gui is coming together.

This will allow the user just concentrate on the most important things, in this case what the array contents should be. The for loops, etc, etc, are on the house :-]

This one is short because I am about to go for an afternoon surfing session.



Copyright José Cornado (c) 2005-2006

Correction: the new engine is faster (25-200 times)

We have been updating the performance comparisons of our new architecture against the competition. As mentioned before, we currently have numbers for the java/eclipse product only.

The performance comparison was to measure how much time would it take to execute a suite that exercises 19 methods of the form:

public XX ReturnXX(XX param){
return param;
}

Where XX is an native type or its equivalent in the java.lang package. We ran one set where every test was a failed one and another set where all were successes.

The idea was to measure how efficient the engines were at handling different amounts of work. A failure should take more work than a success because the engine has to report where the failure is.

The numbers are better than we thought. In the all-failures scenario our engine is, consistently, 25 times faster than Junit. In the all-successes scenario our engine is 200 times faster than Junit.

With the advantage that we can handle all types of access to methods and fields, and strict gui testing.

We will be updating the white paper with these new figures and snapshots of the product, soon.

Well, that's all for now...

A Snapshot of the Eclipse plug-in!!

NOTE: The GUI capture here is obsolete for a newer version please visit:

http://efekctive.com/picture_library/blogging/devriot-ui-new-linux1.gif

Copyright José Cornado (c) 2005-2006

There are a couple of things to notice, we think:

The user just needs to provide the parameter and expected return values. The rest is optional information: an ID to execute the test in a particular order, pre-conditions and post conditions of the object under examination, number of times to execute, and load/stress level.

The method shown is private, but our technology can still reach it. It is really poweful and dynamic. One of the advantages of this feature is that your QA investment is immune to Refactoring.

As several people have noted over different lists and forums, Refactoring can (and does) make public or protected methods private.

In this case the QA efforts are lost because Junit, or Nunit for that matter, just can not handle the change. What to do? wait and let the code base grow without testing? risk to lose several weeks of QA development? or just do not use refactoring?

Another is the ability of testing even the exceptions (type and contents) thrown by the logic under scrutiny.

Also, this is the same interface that will be used to handle GUI testing. Nothing to capture :-}

The new architecture/engine is fast

The new architecture that I mentioned in the previous post refers to our automated testing engine. Please note that:

• It is a testing engine, not a GUI only testing engine or an unit-testing only engine. It's both, so users have to deal with one test tool only, we hope.
• It works as an IDE plug/add-in or as a automation server.
• It does not force Capture & Replay, writing code or scripts on the user. It uses what we call Reduce Input/Output Testing.
• It is platform and OS independent. For example, the gui tests created under Linux can be executed in a Windows environment.
• It is language independent.

Now the numbers: the average execution time per test (unit or gui) of our Java product is less than 1 millisecond. Junit throws in the same environment a lower limit of 20 milliseconds per test.

This is a performance increase in excess if 2,000%, so imagine that your QA technology performed 2000% faster?

To put in a nutshell, our goal is to provide a technology that minimizes the users' effort (input), minimizes the post-test creation maintenance (input), and executes faster than any competitor (output) thus making the user more productive.

The .Net version, even though is still under the old architecture, offers similar performance numbers when compared against Nunit.

Well, that's all for now. Hopefully, I will be able to get back to Amber and .Net pretty soon...

BTW, our most recent white paper is this (password: AUSTIN).
(The images are old though. We will update them shortly)