Emulators

There will be another UI element in DevRiot command set. A combo box that will allow the user which emulators are available to use given the current project. If a project is a smartphone one, then only smartphone SDK emulators will populate the combo box.

Moving to VS 2008

Adding support to the different devices that exist in the Compact and Micro Frameworks is forcing us to move development to VS 2008 from VS 2005 it seems a doable task in a short period of time :-}

Starting on .Net Compact and Micro Frameworks!

This afternoon we started the wiring of DevRiot to the .Net Compact and Micro Framework emulators.

Some considerations:

• The execution speed will not be as good as DevRiot is on the Desktop Framework. Those are resource constrained frameworks.
• There will be an smaller amount of threads available to the Load & Stress Engine.
• The maximum number of tests will not be affected.

Available!

We are happy to announce that DevRiot for Visual Studio 200X(Beta) is available for downloading at:

http://efekctive.com/beta_public/

The documentation can be found at:

http://efekctive.com/docs/

Don't forget you will need a working email address, a working internet connection, and a SQL Server instance available to install and use DevRiot (the email address is required during setup only)

We hope you find it useful and worth the time to try it! If you have any questions please contact us at this address: beta at efekctive dot com.

Just waiting...

Everything is just waiting for a final delivery hurdle to clear and the beta will be available....

Almost there...

The obfuscator has been selected. Building + obfuscating + packaging + installing + monkey testing. In short, making sure it works before actually buying it...

Wiking!!

For the last few days we have been putting together a simple documentation for this beta. It can be found at: http://www dot efekctive dot com/docs.

Mild and off-target...

After a harmless hack attempt at the wrong site, it was suggested by my isp to harden the site logic. We did so it is robust enough to avoid the easy gotchas, we believe.

Now off to put a little documentation together with the previous images and posts and it should be ready for the public to break...

Deployment is steady...

Well deployment of the engine is steady, as well as registering and unregistering. A few runs of these three together and some poking around the engine to make sure one last time and it should be ready to be downloaded! Happy new year!!

Registration is there...

The first part of deployment is ready, or almost ready. Tomorrow we will start on deployment of the engine itself. Happy Holidays!

Shallow waters

Finally we packaged a shallow/dummy version of DevRiot for .Net. The goal is to make sure the installers work correctly and package the resources needed by the tool to work. No functionality yet.

We are working on the full deployment logic.

They can be found here

Feel free to try it and let us know what you think at beta [at] efeKctive dot com

Done.

The generics engine, and the engine in general, is stable enough for somebody else to break it. The install/deploy phase begins. A video with the full shebang of features will come in a few...

One Bug Down...

DevRiot's generics compiler has a couple of bugs that make it behave different from .Net's standard behavior. One is already down. The other should come down soon and deployment start...

Random thoughts

The GUI acts now as it should be (more stable). Just a couple of scenarios/bugs in the GG engine and the deployment phase should start in earnest.

The federal catch 22: labour market is in the doldrums. Stimulus vs. outsourcing? Spend more tax payer money that could be sent overseas in 2/3 years? Black Friday purchases shrunk almost 10% in compared to 2008. Not a seat for the faint of heart.

FormMinder!

Some messy code on our behalf renders DevRiot's GUI unusable. And it looks random so far. Working on it...

Generics Generator getting a life...

We have hit a can of worms, actually a couple of weeks back, that made the automatic recompilation of tests algorithm working for generic types only or non-generic types only. Obviously this was not acceptable.

This algorithm was rewritten and it looks cleaner and working...

Crawling back to deployment

The recompilation process has hit a couple of obstacles while dealing with generic methods but we are working on them. But we have found a clean, simple way of deployment that is worthwhile trying...

Glad to be back!

After a short trip where I broke the only surfboard I had on the first session and was chased out of the line-up by a shark on the third day finding the last changes made to the engine were less than ideal is a piece of cake.

The trouble is already spotted, thoroughly examined, and almost fixed. The deployment logic will have to wait a tiny bit.

It is nice to see how big companies, and big research institutions continue using "Quality Advantage" on a regular basis.

DevRiot Beta is wating for deployment...

We fixed the nested generics problems. It looks good enough to let people really find its problems.

This leaves DevRiot beta ready to be used once the deployer method is ready. Will be off the grid on a 2-week friends & family & surfing trip. A break is called for.

Nested Generics...

We are experiencing a little bit of trouble with the way nested generic types are being stored. In the meantime a couple of buttons/shortcuts were added so the engine can be disengaged when a series of builds trigger continuously DevRiot's automatic build.

Next week will be a simple install and finish the engine deployment method. Beta is just a short steps away...

1,600 T

I have been trying to squeeze more than 1,600 concurrent threads out the engine without changing any settings. No luck. Obviously, it will need a closer, slower look and better tools. Still 1,600 is a number that can be useful, specially when one invests 30 seconds or less to create, use, and harvest.

Pit Stop!!

From time to time things deadlock, etc. Well, for those occasions DevRiot lets you deal with them seamlessly...

Measuring up...

The L&S engine is working just fine. Some changes were made to allow users to measure on a per-thread basis the execution times. Like the Time-To-Live concept: if a method takes more than X ticks, a failure is reported.

Progress one thread at a time...

Arrays and Dynamic fixtures seem to be stable. A couple of bugs that we'll need to get back to but nothing to keep us from moving on to threading/Load & Stress...

Objects

This is an snapshot of DevRiot latest build. It shows a run with a large amount of dynamic fixtures. Mainly System.Object in various shapes and forms (arrays, block arrays, etc) Depsite the number of failures, the execution times still are minimal. Enjoy!

Short Flight update

The block arrays logic is smooth now. There is a little glitch when mixing and nesting different type of arrays like TypeA[,,][][,,][]. But we are working on it.

Short Stop.

Block arrays have become stable. Cut and Paste of tree nodes will wait.

Tweeterish update

Making great progress. Generics is stable. Block arrays (of the form [,,,]) not so much.

Generics core

The interesting thing about last post is that the cpu utilization averaged 50% during the entire rebuilding of the suite (3.4 million tests) If memory is managed with common sense the throughput of DevRiot in multi-core machines with several cpu would be hard to match (mine is a duo)

On the down side the internal compiler choked at complex generic types like:

"ns1.t2<ns1.t1<system.byte[][],system.byte>,System.Byte,System.Int32>[][][]"

Problem solved. Smooth..

3,400,003 and counting...

We spent the weekend fixing an old disconnection problem when running large amount of tests. We fixed it. It works like a champ, it executed some really ineffient code for 24 hours straight without a bleep.

We also managed to squeeze 3.4 million tests with some complexity: runtime generics, different levels of validation and the engine ran like a champ. The pudding:

 

DevRiot for .Net Beta Update...

Having a little bit of trouble with nested generic types and bulk creation (25,000 or more) but other than stable an smooth.

BTW properties will not be supported for this beta. 

Another Short Update

DevRiot for .Net is coming along well. We have spent extra time ironing out the code that deals with System.Object, interfaces, and abstract types because it is adding more dynamic input to an already dynamic engine. 

Since the user never writes code, tests are generated/executed on the fly, the former types add more complexity to the logic so the ironing of bugs turns more complex too.

Hit a unexpected behavior between the recompiling process of DevRiot and VS debugger. Specially annoying when handling 10,000 tests or more.

Me too!!

There will be no public beta for DevRiot .Net. Just access to the people who kindly signed up to try it out.

So for those me-too with a laundry list of me-too features, stop poking around!

Great Progress!!

The last couple of weeks have seen a lot of bug fixing, a more transparent GAC manipulation. Th engine is a lot more stable. It needs to be because it will also handle the GUI tests. Back to work...

The Road to Badajoz...

This is going to be my last post from Galicia, hopefully not last ever. I will be in Denver by the end of the month. The rest of the family is already there.

Moving a household across the ocean is quite an undertaking, even with movers. I have worked on DevRiot some: fixed several bugs, on the .Net Compact side.

Sendoff surfing session with the crew tomorrow. Next posting will be from the Rockies.

To Shake!

The weekend was busy as any impending across-the-Atlantic move would demand but I managed to download Chrome (I have been using Firefox for years) and poked around the web.

I tried to run the technorati searches that we appear on but it didn't work. It may be a matter of sgining-in, the reading was interesting nevertheless.

Some of the posts were pro unit test other against, Agitar, Agile, etc.

A blogger explained that Agile is an incremental methodology where the developer decides what to work on next. A couple of them stated in other posts that a lot of their collegues just don't write unit-tests.

It is not surprising: as soon as the decision of what to work on is made by the over-pressured project manager, something's got to give.

Agitar's demise may be a consequence of the previous, but it also missed some crucial requirements: it just worked unit tests, the underlying framework was somewhat limited. It also solved just part of a customer problem: product quality. By only adding a lock statement the whole thing lost compass.

BTW, Chrome seems very useful...

Possibilities

In case any of you were wondering, DevRiot runs in Java and .Net but it is not limited to those runtimes. It could run on C or C++.

It is a matter of resources available:-}

Recompile The Move...

Ok it's hard to move across the ocean again and fix bugs but we have managed to fix some areas that were choking while recompiling: abstract classes + interfaces.

Still no departure date.

Params, Threads, Generics This is another progress demo. It involves: dynamically allocating params, genereics, threads, and switching between classes. Everything is handled transparently and effortlessly to the user. With such a small learning curve, ease of use, and blasting speeds the testing "Total Costs of Ownership" should be extremely low while attaining effectiveness never seen before. Enjoy your Sunday morning...

Smooth...

After cleaning up some bugs, a task that is still going, here is another sampler: a 500 thread session, with generics included, ran twice under 3 minutes from scratch.

That is generics type, tests + threads creation, and run. Everything completely synchronized.

General Dynamic Fixtures

This is the first work-in-progress report in a while. It shows how DevRiot dynamically generates, stores, and retrieves tests between sessions.

The "fixtures" are handled dynamically. We can mix and match the generics parameter types on-the-fly, keeping the engineers effort easy, flexible, and quick. Extremely quick.

Very few tools on the market can offer such a low Cost of Ownership, in the short or long run :-}

Enjoy...

Summer Blues Report...

I must admit that moving suites/tests from storage to binary form has been a challenge. It has become more difficult than DevRiot for Java because it has more features to take care of.

Generics Generation has posed its own set of binarization challenges.

But still snailing away...

Still working out the bugs...

...inside the generics engine to recompile the tests from storage. Not being able to ask questions makes the work slow down a lot.

Will have another demo soon.

The Referencer

Is still working on fixing bugs. The logic to implement new/old method matching is uncovering a lot of bugs. Not as close to beta as I thought or would like...

Plugging the plug-in

We are paying the error of working while sick on a key piece of code regeneration. Bad code, or at least worse than usual, that needs rewriting.

But it also has given us the opportunity of thinking about deployment.

We are toying with the idea of adopting "perpetual delivery" mode. Since the engine is not larger than 150K, downloading it can part of the add-in setup.

Generating Generics...

Stuck here. Trying to combine serialization/de-serialization of suites with generics and not generics is proving to have several traps...

Deploying the report,

Still making a lot of progress with suite recompilation and integrity.

Also some time has been spent on figuring out how to deploy the tool while minimizing the IP exposure.

There is VS Gallery and other channels. Obfuscation is a good deterrent but it needs help from other angles...

From the lonely tree to the red sound (Progress report)

The recompilation process is moving slowly but solidly. As anyone might have guessed bugs have been uncovered and are about to disappear...

The S option seems to be a viable option!

Good ol' SQL improved the serialization of tests by leaps and bounds.

Now it takes less than minute and a half to create and store 5,000,000 tests. Before this much time was needed for 100,000.

The S Option

Given the enormous amount of tests can be generated with DevRiot, our old serialization architecture is proving not up-to-par.

SQL seems to be most logical way to handle this.

Well keep you posted. Sorry for the inconvenience!

Less than two minutes...

We have been tweaking around the serializer (getting rid of duplicate IO work) and writing, storing 100,000 tests takes less than 2 minutes. The development costs aren't why they used to be...

A more realistic approach to this feature can be: there is a method that takes a matrix and returns a modified version of it. Given the nature of matrices, a user could create a "base" test batch with a baseline matrix then modify the individual matrices with minimum effort.

Going academic!!!

It is always encouraging to see that not only businesses find the white paper useful. A lot of universities, institutes of technology from all over the world have been downloading the white paper (even for course work)

Even a national police from South America :-?

Well, the serialization server is moving forward nicely, although the weekend was lost with a clogged up laptop. It has been deleting *.tst file for the last 48 hours...

Storage...

I must have to say that developing code to serialize large number of objects can be quite time consuming.

My laptop seems to handle 100,000 or less files quite well. When the number jumps to 1,000,000, it becomes messy molasses to continue working.

Will keep you posted...

Gluttony...

Being able to handle large number of tests (5,000,000 in my dev machine, but it could be higher) is that serialization of suites becomes a little bit harder.

Well, that is what I am working on now...

5,000,000

After tweaking some generation logic to economize the memory consumption, DevRiot reached the 5 million tests mark in less than 34 seconds.

Sounds fast, isn't it? Now, really going back to suite serialization...

I tripled THAT...

1,500,000 in less than ten seconds. Back to Suite serialization...

500,000 and counting...

The Suite serialization is slowly starting. I am polishing some performance features.

But I wanted share with the latest readers some of the brute-force finesse of the tool:

500,000 tests, no coding, no wasted time. Enjoy.

It does not register

Well after having a question/post mysteriously disappear from the VSX forum, which is a first, I decided not to wait more on that.

Two registry entries will allow the user to gauge according their HW configuration what the max number of user/threads DevRiot can handle and the emergency TTL (when all current users/threads are harvested to avoid hosing down the box)

To be a product that has a really small exposure, there a lot of big copycats :-}

Time To Change...

Well after fiddling around with the load/stress engine + 1,600 threads/users for a while, I think it makes more sense to let users set the TTL per test.

It is easier for the user to set it according to their knowledge of the code than use any predefined formula.

It also becomes part of the verification data of the test which should be useful.

Thread Quartet in K reverse...

The following demo shows a Load/Stress session on a object using four of its methods X 400 threads/users (1600 total).

In addition, there is the Assertion, Out-Of-Scope, etc, in a codeless, easy-to-use, low-skills-needed sort of way.

It is a little bit longer than the previous ones but worth watching (it has been edited to avoid the initial setup of the threadpool) :-}

1,500 Threads in action

Well , we wanted to try a higher number on our load/stress engine. So, we tried 1,500. Watch and enjoy the execution times after the initial run (when all the setup occurs).

58 Seconds...

That is how long it takes to create, launch, execute, synchronize, and verify a load/stress session with DevRiot. Check it out for yourself...

Why wouldn't you want that level of efficiency in your SDLC? :-}

Visual Threading!!

This is another progress demo with assertions, out-of-scope verifications, multiple threads were involved in a series of tests. Not a single line of code was written.

Try to achieve the same with other technology: Microsoft's, etc. See how much time it takes you to achieve the same. Then check how much more time it takes to get the results back.

How do you want to spend your time and budget? :-}

VUT Threads...

This is almost identical to the previous demo but with the difference that the Load/Stress testing engine is hooked up.

Note that the video follows the same script as the previous one, so no PrivateObject API or coding or scripting. Even the thread creation, execution, and harvest.

The other difference is that we intentionally tried to add a test without pre-compiling a parameter so the tool could complain :-] Three threads are created and used against the method under test.

Almost Context Full...

The context manager is 60% done. 4 snapshots of the context manager at work: assertions, cleaning up, etc.

Automation?

We are currently working on the ContextMenu manager. It should be a breeze since the Generics support actually meant a major addition of functionality.

GUI support should come soon too, its major piece of functionality is ready to wire.

Which lead me to look at the Automation library that MS has on .Net 3.5-VS 2008. It is not a surprise , and not a particularly interesting competitor.

First is stuck in PC land. None of the major classes are supported by the Compact Framework.

Second, it is another hard to digest class/method behemoth.

I guess they are not competing on usability and product definition/differentiation...

Compare me if you can...

These two are a couple of progress snapshots from the Generics Engine.

To show the flexibility and ease-of-use of DevRiot's Visual Unit Testing(tm) technology here we have a generic method that takes 3 generic parameters. Each one of them is a bound generic type A made of an array of bound generic type B. Type B is made of an array int.

Just a few clicks away. No Source code...

That ugly description turns into an easy to use tree widget:

Short...

Now the generics engine can handle pretty much everything it is thrown at it.

The first ever Visual Unit Testing tools is coming your way...

General Complexity

The Generics Generator is pretty smooth now:

This is an snapshot of a that involves: generic method, out parameter, multidimensional arrays, out of scope data (PrivateObject in VS parlance).

And it is the same as if it a method that takes and returns an integer. The user just spends time on what it matters most: substance and not boiler plate noise.

None of the competitors can achieve such a level of usability and ease-of-use. Let alone the execution speeds. We are pretty close to wire up the GUI testing engine...

Anything VUT...

Visual Unit Testing, this is the acronym that best describes the visual nature of DevRiot's user experience. It also applies to the GUI testing engine but there are more salient features that describe DevRiot's GUI testing engine: device independence and ditching the capture-and-replay paradigm.

Back to ironing out some bad bugs in the generics generator logic...