Hi all,
I would like to have a piece of code to check for data quality or if every scan got data OK.
I currently have a counter in the 250ms scan, totalize every minute.
But now and then the total differ from previous total, how is that possible with no skipped scans ?
Sometimes i see the counter go from e.g. 23 to 25.
Currently the total is 7036, but can be lower is Well...
With the kindest regards
forgive,
but the post is not clear to me. What are 23 ... 25? and 7036?
if possible paste the program here to understand better.
At 4 Hz (250mSec scan rate) there are 240 scans per minute
Smile
PreserveVariables
PipeLineMode
AngleDegrees
Dim AirTC_2
Dim SPkPa
Dim Twg
Dim Twpg
Dim Vpg
Dim Vp
Dim SVp
Dim Twch
Dim VpgVpd
Dim Top
Dim Bottom
Dim SVpW
Dim N
Dim HITF
Dim WCTF
Dim WCWSMPH
Public BattV
Public PTemp_C
Public WS_ms
Public Avg_3s_ms
Public WindDir
Public BP_mbar
Public SlrW
Public Rain_mm
Public RainTotal_sec
Public Precepitation_type
Public Lufft_R2S_Mode
Dim Mode(6)
Dim DestMaxSpa(2)
Public AirTC
Public RH
Public TdC
Public TwC
Public HI_C
Public SunHrs
Public PotSlrW
Public RTime(9)
Public SolPos(5)
Public WC_C
Public Tot24
Public PrecepitationHrs
Public counter
Public Rain_mm_tot_hour
Public Rain_inch_tot_hour
Public Tot24_inch
Public BP_inch
Public Tdf
Public AirTf
Public WS_mph
Public WS_kts
Alias RTime(1)=Year
Alias RTime(2)=Month
Alias RTime(3)=DayOfMonth
Alias RTime(4)=HourOfDay
Alias RTime(5)=Minutes
Alias RTime(6)=Seconds
Alias RTime(7)=Microseconds
Alias RTime(8)=DayOfWeek
Alias RTime(9)=DayOfYear
Alias SolPos(1)=SolarAzimuth
Alias SolPos(2)=SunElevation
Alias SolPos(3)=HourAngle
Alias SolPos(4)=Declination
Alias SolPos(5)=AirMass
Units BattV=Volts
Units PTemp_C=Deg C
Units WS_ms=meters/second
Units WindDir=degrees
Units BP_mbar=mbar
Units SlrW=W/m^2
Units Rain_mm=mm
Units AirTC=Deg C
Units RH=%
Units TdC=Deg C
Units TwC=Deg C
Units HI_C=Deg C
Units SunHrs=hours
Units PotSlrW=W/m^2
Units WC_C=Deg C
Units Year=years
Units Month=months
Units DayOfMonth=days
Units HourOfDay=hours
Units Minutes=minutes
Units Seconds=Seconds
Units Microseconds=Microseconds
Units DayOfWeek=days
Units DayOfYear=days
Units SolarAzimuth=degrees
Units SunElevation=degrees
Units HourAngle=radians
Units Declination=radians
Units AirMass=unitless
'Define Data Tables
DataTable(Table1,True,-1)
DataInterval(0,1,Min,0)
WindVector(1,Avg_3s_ms,WindDir,FP2,False,0,0,0)
FieldNames("WS_ms_S_WVT,WindDir_D1_WVT,WindDir_SD1_WVT")
Maximum(1,WS_ms,FP2,False,False)
Minimum(1,WS_ms,FP2,False,False)
Maximum(1,Avg_3s_ms,FP2,False,False)
Minimum(1,Avg_3s_ms,FP2,False,False)
Average(1,BP_mbar,IEEE4,False)
Maximum(1,BP_mbar,IEEE4,False,False)
Minimum(1,BP_mbar,IEEE4,False,False)
Average(1,SlrW,FP2,False)
Maximum(1,SlrW,FP2,False,False)
Minimum(1,SlrW,FP2,False,False)
Totalize(1,Rain_mm,FP2,False)
Maximum(1,RainTotal_sec,FP2,False,False)
Average(1,RainTotal_sec,FP2,False)
Average(1,AirTC,FP2,False)
Maximum(1,AirTC,FP2,False,False)
Minimum(1,AirTC,FP2,False,False)
Sample(1,RH,FP2)
Average(1,TdC,FP2,False)
Maximum(1,TdC,FP2,False,False)
Minimum(1,TdC,FP2,False,False)
Average(1,TwC,FP2,False)
Maximum(1,TwC,FP2,False,False)
Minimum(1,TwC,FP2,False,False)
Average(1,HI_C,FP2,False)
Maximum(1,HI_C,FP2,False,False)
Minimum(1,HI_C,FP2,False,False)
Totalize(1,SunHrs,FP2,False)
Totalize(1,PrecepitationHrs,FP2,False)
Average(1,PotSlrW,FP2,False)
Maximum(1,PotSlrW,FP2,False,False)
Minimum(1,PotSlrW,FP2,False,False)
Average(1,WC_C,FP2,False)
Maximum(1,WC_C,FP2,False,False)
Minimum(1,WC_C,FP2,False,False)
Sample(1,Tot24,FP2)
Sample(1,Lufft_R2S_Mode,FP2,False)
Sample(1,Mode(1),FP2,False)
Sample(1,Mode(2),FP2,False)
Sample(1,Mode(3),FP2,False)
Sample(1,Mode(4),FP2,False)
Sample(1,Mode(5),FP2,False)
Sample(1,Mode(6),FP2,False)
Sample(1,counter,FP2,False)
Totalize(1,counter,FP2,False)
EndTable
DataTable(Table2,True,-1)
DataInterval(0,10,Min,0)
Minimum(1,BattV,FP2,False,False)
EndTable
DataTable(upload,True,1)
DataInterval(0,1,Min,10)
Average(1,AirTC,FP2,False)
Average(1,AirTf,FP2,False)
Average(1,TdC,FP2,False)
Average(1,Tdf,FP2,False)
Average(1,RH,FP2,False)
Average(1,BP_mbar,IEEE4,False)
Average(1,BP_inch,IEEE4,False)
WindVector(1,Avg_3s_ms,WindDir,FP2,False,0,0,0)
Maximum(1,Avg_3s_ms,FP2,False,True)
SampleMaxMin(1,WindDir,FP2,False)
Average(1,WS_mph,FP2,False)
Average(1,WS_kts,FP2,False)
Maximum(1,WS_mph,FP2,False,True)
Maximum(1,WS_kts,FP2,False,True)
Average(1,SlrW,IEEE4,False)
Sample(1,Tot24,IEEE4,False)
Sample(1,Tot24_inch,IEEE4,False)
Sample(1,Rain_mm_tot_hour,IEEE4,False)
Sample(1,Rain_inch_tot_hour,IEEE4,False)
EndTable
'Main Program
BeginProg
'Main Scan
Scan(250,mSec,4,0)
'Default CR1000 Datalogger Battery Voltage measurement 'BattV'
Battery(BattV)
'Default CR1000 Datalogger Wiring Panel Temperature measurement 'PTemp_C'
PanelTemp(PTemp_C,250)
'05305 Wind Speed & Direction Sensor measurements 'WS_ms' and 'WindDir'
VoltDiff(WS_ms,1,mV2500,1,True,0,_60Hz,0.025333,-12.5)
VoltDiff(WindDir,1,mV2500,2,True,0,_60Hz,0.18,-0)
PulseCount(Rain_mm,1,1,2,0,0.01,0)
PulseCount(Precepitation_type,1,2,0,1000,1,0)
AvgRun(Avg_3s_ms,1,WS_ms,12)
WindDir = WindDir + 140
If WindDir > 360 Then WindDir = WindDir - 360
If WindDir < 0 Then WindDir = WindDir + 360
If WindDir>=360 Then WindDir=0
counter=counter+1
NextScan
SlowSequence
Scan(1000,mSec,50,0)
VoltDiff(SlrW,1,mV250,7,True,0,_60Hz,1,0)
VoltDiff(AirTC,1,mV2500,3,True,0,_60Hz,0.05,-65)
VoltDiff(RH,1,mV2500,4,True,0,_60Hz,0.05,-25)
VoltDiff(BP_mbar,1,mV2500,6,True,0,_60Hz,0.184,600)
If TimeIntoInterval(0,60000,mSec) Then counter=0
SlrW=SlrW*43.30879
If SlrW<0 Then SlrW=0
'Dew Point and Wet-Bulb calculation prep
AirTC_2=AirTC
SPkPa=101.325
SatVP(SVp,AirTC_2)
Vp=RH*SVp/100
'Dew Point calculation 'TdC'
DewPoint(TdC,AirTC_2,RH)
If TdC>AirTC_2 Or TdC=NAN Then TdC=AirTC_2
'Find Wet-Bulb 'TwC'
Top=AirTC_2
Bottom=TdC
For N = 1 To 25
Twpg=Twg
Twg=((Top-Bottom)/2)+Bottom
WetDryBulb(Vpg,AirTC_2,Twg,SPkPa)
VpgVpd=Vpg-Vp
Twch=ABS(Twpg-Twg)
If VpgVpd>0 Then
Top=Twg
Else
Bottom=Twg
EndIf
If Twch<0.01 Or N=25 Then ExitFor
Next
TwC=Twg
'Heat Index calculation 'HI_C'
HITF=1.8*AirTC+32
HI_C=-42.379+2.04901523*HITF+10.14333127*RH-0.22475541*HITF*RH-6.83783*10^-3*HITF^2-5.481717*10^-2*RH^2+1.22874*10^-3*HITF^2*RH+8.5282*10^-4*HITF*RH^2-1.99*10^-6*HITF^2*RH^2
If HITF<80 Or RH<40 Or HI_C0.4*PotSlrW AND SIN(SunElevation)>0.1 Then
'Calculate sun hours for scan time in seconds
SunHrs=1/3600
Else
'Set sun hours for scan time in seconds to 0
SunHrs=0
EndIf
If Precepitation_type>0 Then
'Calculate sun hours for scan time in seconds
PrecepitationHrs=1/3600
Else
'Set sun hours for scan time in seconds to 0
PrecepitationHrs=0
EndIf
'Wind Chill calculation 'WC_C'
WCTF=1.8*AirTC+32
WCWSMPH=WS_ms*2.236936
WC_C=35.74+0.6215*WCTF-35.75*WCWSMPH^0.16+0.4275*WCTF*WCWSMPH^0.16
If WC_C>WCTF Or WC_C=NAN Then WC_C=WCTF
If WCTF>50 OR WCWSMPH<3 Then WC_C=WCTF
WC_C=(5/9)*(WC_C-32)
'1 second running total calculation
TotalRun (RainTotal_sec,1,Rain_mm,4)
'24 hour running total calculation 'Tot24'
Tot24=Tot24+Rain_mm
If TimeIntoInterval(0,1440,Min) Then Tot24=0
'Call Data Tables and Store Data
'Upload Data
TotalRun (Rain_mm_tot_hour,1,RainTotal_sec,3600)
Rain_inch_tot_hour = Rain_mm_tot_hour / 25.4
Tot24_inch = Tot24 / 25.4
BP_inch = BP_mbar / 33.864
Tdf = TdC * 1.8 + 32
AirTf = AirTC * 1.8 + 32
WS_mph = Avg_3s_ms * 2.2369362912
WS_kts = Avg_3s_ms * 1.9438444924406
'count the occurrences of each mode and put the counts into
'an array
If Precepitation_type = -10 Then Mode(1) = Mode(1) + 1
If Precepitation_type = 10 Then Mode(2) = Mode(2) + 1
If Precepitation_type = 20 Then Mode(3) = Mode(3) + 1
If Precepitation_type = 30 Then Mode(4) = Mode(4) + 1
If Precepitation_type = 40 Then Mode(5) = Mode(5) + 1
If Precepitation_type = 50 Then Mode(6) = Mode(6) + 1
'use MaxSpa to find the maximum of the Mode array
'the second destination variable is the Location
'within the array where the Max occurs
MaxSpa (DestMaxSpa, 6, Mode(1))
'Return a mode, based on which index in Mode() holds the max
If DestMaxSpa(2)= 1 Then Lufft_R2S_Mode = 0
If DestMaxSpa(2)= 2 Then Lufft_R2S_Mode = 1
If DestMaxSpa(2)= 3 Then Lufft_R2S_Mode = 2
If DestMaxSpa(2)= 4 Then Lufft_R2S_Mode = 3
If DestMaxSpa(2)= 5 Then Lufft_R2S_Mode = 4
If DestMaxSpa(2)= 6 Then Lufft_R2S_Mode = 5
CallTable Table1
CallTable Table2
CallTable upload
If TimeIntoInterval(0,1,Min) Then Mode(1)=0
If TimeIntoInterval(0,1,Min) Then Mode(2)=0
If TimeIntoInterval(0,1,Min) Then Mode(3)=0
If TimeIntoInterval(0,1,Min) Then Mode(4)=0
If TimeIntoInterval(0,1,Min) Then Mode(5)=0
If TimeIntoInterval(0,1,Min) Then Mode(6)=0
If TimeIntoInterval(0,1,Min) Then Lufft_R2S_Mode=0
If TimeIntoInterval(0,1,Min) Then DestMaxSpa(2)=0
NextScan
EndProg
Hi Smile thanks for your reply .
The 23 to 25 is the counter missing one, 24.
The 7036 is the counter total over one minute.
Thats what i use tot check for missing data/quality check, but i think it doesnt.
Bit i dont know why the counter total differs now and then.
With the kindest regards,
Mark
Seem the question is due to the fact that the main table has priority over the slow scan, so the line where counter is reset to 0 is not executed exactly at the exact minute x,00 but could also be completed after the completion of the next main table. The same is true for datatable table1 which is not certain that it will be called at x,00 of the minute.
Smile
Hi Smile,
That makes sense, thank you.
Regarding the data quality check, am i doing it right with the counter method or are there better ways to do that?
With the kindest regards,
Mark
Hi,
Can I mention that the "TimeIntoInterval" operates on the dataloggers realtime clock at the start of the scan. This is true for both the Main Scan and any SlowSequence Scans. From the CRBasic Help :
"TimeIntoInterval statement is evaluated True (-1) or False (0) based on the datalogger's real-time clock at the start of a scan."
Rather than the instruction below not being True and so it would not execute the "counter=0" part, I would suspect that the scan itself is perhaps not being executed?
If TimeIntoInterval(0,60000,mSec) Then counter=0
It would probably be better to put the above reset of the counter, in the main scan? Then it should always happen on 1min, as long as that scan has not been skipped.
Do you see any skipped scans or skipped slowSequence scans in the Status table? If so, then the sMain Scan or SlowSeq scan is taking longer than its alloted scan time to complete. In this case, it will skip the next time slot when it was due to start and wait for the next time slot.
More details can be found here :-
Understanding CRBasic Program Compile Modes: Sequential and... (campbellsci.co.uk)
Would it be better to put the CallTable instructions in the Main scan? You then know for sure you get all the 250ms measurements included in the averages, totals, etc. If you call the tables from the 1s scan, you do not know if the call to the table happens after the 250ms scan has completed or the 500ms scan or 750ms scan?
BR