Automated Test Framework for Unit, integration and system test?

Hi,

I need to do a unit test for the EmonLib C++ class.

I was thinking about the Boost framework, but I would like to hear what experiences there is out there with various test automation frameworks in relation to the OEM project and EmonLib in particular.

I am new to the OEM Project however I did stumble on a defect.

https://github.com/openenergymonitor/EmonLib/issues/11

I prefer to make test cases, when defects are found. Both for the broken functionality, but also for the functionality to be preserved after the codechanged.

I can't seem to find any automated test for the OEM project.

 

looking forward to your input.

 

Andreas Markussen

 

 

Robert Wall's picture

Re: Automated Test Framework for Unit, integration and system test?

Are you sure that the code of which you complain is incorrect? Both values being compared are of type "unsigned long", therefore the difference will be of type unsigned long, which by definition cannot be negative. It will overflow, but it cannot be negative. For the comparison, timeout will be promoted to unsigned long (and it should never be set as a negative value anyway, because that makes no sense, so I agree with you that it should be of type unsigned int). But as far as I can see, it will handle the rollover correctly.

Perhaps you can tell me where my logic has gone wrong.

Robert Wall's picture

Re: Automated Test Framework for Unit, integration and system test?

By way of illustration (not proof) the attached sketch gives this output:

millis() overflow test
OpenEnergyMonitor.org

j= 0, mytime = FFFFFFF8, (clock()-start) = 0, st = 0
j= 1, mytime = FFFFFFF9, (clock()-start) = 1, st = 0
j= 2, mytime = FFFFFFFA, (clock()-start) = 2, st = 0
j= 3, mytime = FFFFFFFB, (clock()-start) = 3, st = 0
j= 4, mytime = FFFFFFFC, (clock()-start) = 4, st = 0
j= 5, mytime = FFFFFFFD, (clock()-start) = 5, st = 0
j= 6, mytime = FFFFFFFE, (clock()-start) = 6, st = 0
j= 7, mytime = FFFFFFFF, (clock()-start) = 7, st = 0
j= 8, mytime = 0, (clock()-start) = 8, st = 0
j= 9, mytime = 1, (clock()-start) = 9, st = 0
j= 10, mytime = 2, (clock()-start) = 10, st = 0
j= 11, mytime = 3, (clock()-start) = 11, st = 1
j= 12, mytime = 4, (clock()-start) = 12, st = 1
j= 13, mytime = 5, (clock()-start) = 13, st = 1
j= 14, mytime = 6, (clock()-start) = 14, st = 1
j= 15, mytime = 7, (clock()-start) = 15, st = 1

Done

I had no wish to wait 50 days, I think clock( ) is equivalent to millis( ) for the present purpose - it does return the same type.

You can clearly see the time (mytime) overflowing yet the difference, (millis() - start)  is still handled correctly, and the boolean is set in the correct place.

 

andreasm_dk's picture

Re: Automated Test Framework for Unit, integration and system test?

My post had a purpose to discuss the Test Framework, and I would still like this.

Choice of test framework

I will continue with Boost, until somebody pitch in. (Please do! )

The fact that Robert and I, are both doing various analysis, just shows the value for a test framework. 

Comment on Robers input on Issue 11

However I also need to reply to Roberts input. I haven't done the test case, however I have done the math, in an Excel Sheet.

Lets assume that the wrap happens at 100, and the time is 95, and the timeout is 6. So in this case, it should time out after 6 iterations. 





Limit Start time Time Out      
99 95 6  
millis start timeout Diff Return Comment
97 95 6 2 FALSE All fine
98 95 6 3 FALSE All fine
99 95 6 4 FALSE Diff 4, two more rounds
1 95 6 -94 FALSE Ups! Wrap?
2 95 6 -93 FALSE Wrapped, however should have timed out.
3 95 6 -92 FALSE This will never 
4 95 6 -91 FALSE  
5 95 6 -90 FALSE  
6 95 6 -89 FALSE  
7 95 6 -88 FALSE  
8 95 6 -87 FALSE  
           

However we should continue the discussion of Issue 11, on Issue 11. 

So what do you suggest, for automatically showing if an assumption is true or false.

 

dBC's picture

Re: Automated Test Framework for Unit, integration and system test?

C guarantees the desired result in this case.  When an unsigned integer subtraction underflows, it will effectively add 0x100000000 to the result, assuming 32-bit (which in practice means just ignore the underflow and do nothing).  

Undetected underflow has been the cause of plenty of buffer overrun vulnerabilities for network coders, but in this case, the guaranteed behaviour is exactly what the coder is expecting.

Robert Wall's picture

Re: Automated Test Framework for Unit, integration and system test?

andreasm_dk:

What makes you think that Excel obeys the same rules as C & C++? Clearly you have not read the language specification, of that I am quite certain. Here is a reference if you need it: "The C Programming Language", Second Edition, Kernighan & Ritchie, Prentice Hall, ISBN 0-13-110362-8, Appendix A6.5 (page 198). And as far as I am aware, those rules are unchanged since 1988 and have been inherited by C++. [Edit:] The C++ November 2014 draft standard is the same (Section 5.10, page 89).
And I think a review of binary arithmetic and how overflow and underflow is handled (as dBC points out) might not come amiss, either.

Comment viewing options

Select your preferred way to display the comments and click "Save settings" to activate your changes.