Current Cost Support Forum

I have a system consisting of a whole house sensor, three mini transmitters, and a single IAM.

I'm noticing that I'm losing approximately 10%-20% of my data at the base unit, and I believe this is caused by collisions on transmission. I'm guessing you're using the Zigbee protocol given you have the badge on your website.

How are collisions resolved on the transmitters? Does the system try to minimize them or is it just random?
What baud rate do the sensors transmit to the base station at? Is there actually enough bandwidth for 10 sensors?

I thought I posted a message about this here yesterday, but it seems that I must have closed the window before posting, as it's not here any longer.

I think only their next generation of products will use the Zigbee.

The current ones on offer only transmit and no handshake takes place to resend lost or corrupt data

I would also guess other gadget that may be in the home could also cause data loss.

Automan.

----------------------
These are the results from the 3 beta testers of my program live_sol. Yes, the number of readings per hr does reduce quite rapidly (more rapidly than I'd expected) as you add transmitters. The nominal is of course 1200, but SteveH & I get between 1000-1100 (roughly half from each transmitter) from an envi with 2 separate transmitters each nominally every 6 sec eg for house and solar power, 'Seeker' gets only 850 or so on a system with 2 transmitters at 6 sec, plus one from a dev board at a line every 2.5 sec, but for some reason they aren't split 50%:50% between house and solar so that's another issues.

It's just random I think - it's intended to be a very simple system, no protocol. The envi data is all transmitted between monitor and pc at 57k baud, so that'll be 6kbytes/s or so. Each xml line is about 150 bytes, assuming 1 channel on 1 sensor, a lot more 1000 bytes for the (occasional) history lines so in theory the b/w is enough for 10 sensors. I'm guessing that if another transmission happens during the 30ms taken to transmit & process a message in the pc, it'll be lost. If you want to do the stats you can predict what'll happen, but my guess is that using more than say a few eg <5 sensors at every 6 sec there'd be too many collisions & system wouldn't be very good, tho' it'd still work... Of course it doesn't necessarily impact the accuracy, if you take account of the 'missing' lines and the reduced rate as I (try to) do in live_sol.

hth John

Thanks for the replies guys.

I don't think it's inherently a problem with the PC processing the message, as the RS232 will just transmit the data. There isn't any handshaking as far as I can tell.

Also, we still don't know what the over the air baud rate is. PC may be 57600, but there's no reason to expect the transmitters work at that speed. If they worked at a higher baud rate, there'd be less chance of a collision. Flip side of course is that if they're actually slower, there's more chance.

The over air baud rate isn't a standard RS-232 style baud rate that would normally be used for serial comms and as far as i can remember, it's actually quite low, somewhere in the regeon of 3 to 4 Kbaud. Somewhere around 3.3K rings a bell.

Theoretically a higher baud rate would shorten any transmission time and hence less oppertunity for collisions, but it's likely that there would be an increased risk of data loss because of the way 'similar' RF receiver modules work. IE similar radio modules have quite a slow processing speed.

The data packet that is sent, is usually about 23-24 bytes of data. The first 3 bytes are often used as a pre-amble, essentially 0xAA, 0xAA, 0xAA (forming an alternate 10101010 bit pattern) to allow a receiver module to lock on and syncronise to the signal. The next two bytes of data are required by the RF receiver module for it's "synchron" word, which enables the receiver module to identify that the transmission is intended for that particular style of receiver.

This part of the data packet is fairly standard for this style of radio module and can be found from the data sheets for many similar RF modules.

Following the above is the data payload, consisting of, for example, power data and a few other bytes of data, possibly a checksum or other info that i haven't bothered trying to decifer.

It was quite some back when i breifly looked into this out of curiosity, but my 'paid' work takes precidence and unfortunately most of my time, so i haven't had a great deal of spare time to investigate anything further.

Datasheets for extremely similar 'off the shelf' radio modules are freely available to the public, which openly the describe their standard data formats used for transmission, such as the pre-amble and synchron word, preceding any data payload.

That's about all i can say about any "over air" details but i'm sure further info may be found from datasheets for similar radio modules.

Here is a quick analysis on a dump of messages coming from my CC128. It covers a 2hour period from 6am to 8am on DSB 37 which would be around May 2010. [I'm currently on DSB 534 ]. I've used this file as it was straight from the cable (doesn't use the software splitters etc I am using today).
The following table shows, for a single transmitter, the number of messages received after a gap of n seconds from the previous msg for the same sensor (i.e. excludes history msgs in hour 7).


If all msgs were spaced at 6 seconds I would expect 1200 msgs in the 2 hours, so I have lost 90 msgs approximately evenly distributed across the two hours. In hour 7 there were an additional 1040 history records but they do not appear to have affected the number of msgs lost. Because there is only a single transmitter the problem cannot be two msgs arriving together. That does not preclude other interference. I live in an electrically noisy house -- wireless telephones, dorbells, and LAN are all likely candidates for causing interference.

I also questioned whether the ENVI is juggling times to avoid collisions. The next table shows the distribution of msgs in each 10 second interval. In the 7:01 - 7:18 period there are a large number of history msgs all with a time which is an exact multiple of 10 seconds.

There is apparently no bunching of times to avoid the busy periods.

I'll try to repeat this analysis with my present set up of 3 transmitters and possibly add more (I've got 5 spare dev boards).

_________________
Seeker
"The Truth is out there!"

Seeker, sorry if my mention of your system's data rates above was confusing/ or wrong & thanks for the analysis, but guess I missed your overall conclusion and a comment on what the previous emailer was saying about the (v low!) baud rates over air. Guess no-one (probably including most of the cc guys) knows how envi really works!.

qs
1) why does the addition of the extra data cause your solar data/hr to be (consistently) different from the house data and much lower that w/o the extra data?

2) what's the likely answer to the original q - ie would envi support 10 sensors, in a usable fashion?

3) A practical q:- given the way that the data rates reduce with additional sensor lines, what is the correct/best/most accurate way to calculate the power for a sensor from the live sensor readings. Is it still, as cc recommend, to multiply the sensor reading in xml by the interval in secs since last reading, bearing in mind that maybe readings are lost between clamp/transmitter (by collisions) and pc, or is it better add up the readings and to scale according to the average no of readings received in an hour somehow? Or what.

John

Hi John, No confusion, I was just trying to show that the commonly held view (that it is collisions on the radio side causing the loss) does not apply in the example.

There are many places the problem can be introduced; Noisy or weak radio signals, insufficient buffering in the UART, PC running near 100% CPU are just a few. We just have to program around those things we can't fix.

Who Knows? Possibly a lower powered transmitter?

I'm sure it would, but need to define usable. Accepting there will always be some loss, is 10% ok? is 20% too much?

Well this is the crux of it. Usability depends on the Mission. So if I want an estimate of Energy used each hour or 30 minutes, a single sample in each minute will probably suffice (This is what Techtonic Energy Station does). Assuming the samples are normally distributed a sample of 30 will give an approximation of the mean +/- 2SD ie about 99.5% confidence. (N.B. If you average the 6 second readings in each minute you can be certain that it is normally distributed).
On the other hand, if my mission is to identify appliances in use at any particular time I would need to pattern match the instantaneous power at a high frequency. so 6 second intervals is probably the outside of what is acceptable. (Doesn't negate the approach, but reduces confidence in result).

Wow! that was a bit esoteric wasn't it? I didn't delete it cos I put a lot of thought into it .
In practical terms the difference in the average value between your two approaches is marginal though the second approach is likely to have more variance.
In my home-brew system reading the inverter (CC not involved), i read the inverter each minute multiply the average reading before and after the interval by the interval length and sum over 10 minutes. I have special handling if I miss a reading on the minute boundary.

_________________
Seeker
"The Truth is out there!"

3) A practical q:- given the way that the data rates reduce with additional sensor lines, what is the correct/best/most accurate way to calculate the power for a sensor from the live sensor readings. Is it still, as cc recommend, to multiply the sensor reading in xml by the interval in secs since last reading, bearing in mind that maybe readings are lost between clamp/transmitter (by collisions) and pc, or is it better add up the readings and to scale according to the average no of readings received in an hour somehow? Or what. [/quote]
Well this is the crux of it. Usability depends on the Mission. So if I want an estimate of Energy used each hour or 30 minutes, a single sample in each minute will probably suffice (This is what Techtonic Energy Station does). Assuming the samples are normally distributed a sample of 30 will give an approximation of the mean +/- 2SD ie about 99.5% confidence. (N.B. If you average the 6 second readings in each minute you can be certain that it is normally distributed).
On the other hand, if my mission is to identify appliances in use at any particular time I would need to pattern match the instantaneous power at a high frequency. so 6 second intervals is probably the outside of what is acceptable. (Doesn't negate the approach, but reduces confidence in result).

Wow! that was a bit esoteric wasn't it? I didn't delete it cos I put a lot of thought into it .
In practical terms the difference in the average value between your two approaches is marginal though the second approach is likely to have more variance.
In my home-brew system reading the inverter (CC not involved), i read the inverter each minute multiply the average reading before and after the interval by the interval length and sum over 10 minutes. I have special handling if I miss a reading on the minute boundary.

No not too esoteric at all imo - to quote the great man 'as simple as possible, but not simpler'. I'm always pleased to see someone quoting the 'central limit' theory - one of the most amazing theories in stats IMO.

My 'mission' in live_sol is to do the best estimate for a house with solar, of the power saved and 'wasted' (ie going back to grid at 0 credit) wrt the user, and to get the best estimate of the billable power each day.

The system is non linear because of the threshold effect ie if solar-house > 0 then some saving & waste else saving only, so I concluded that I need to do the calc in realtime every 6 (or so secs) to get best estimates and potentially just sampling might be a good deal less accurate imo. Of course there are still errors, only some of which cancel via averaging, eg the time is only to nearest sec, plus PF & clamp errors, potential data losses as above, and of course we are doing the difference of nos with errors hence some error magnification. Given all of this, and with testing I've done vs my meter (SteveH has also done these, and I hope to do them on your data too soon), I'm mostly getting the billable power estimate to within (+ or -) < 5kwh + about (+ or -) <5% in my house power of 15-20kwh per day, so not too bad, and better than is reported many others using other apps...but I'd like to improve it if possible, so all ideas welcome.

John

Hmm, I was going to talk to you about the terminology.

To you, power back to the grid is "wasted"; to me it is "exported". That's because I receive $0,66 for each Kwh I export. I only pay $0.32/KwH in peak times (M-F 7am-11pm) and $0.10/Kwh for off-peak. That makes my mission to use as little power as possible while the sun shines and shift as much import to the wee small hours as I can (e.g. run dishwasher on a timer). I hope Live_sol will accommodate both missions.

_________________
Seeker
"The Truth is out there!"