Custom Data Verification
Since version 8.1.0, when importing a meteorological file through custom import, PVsyst runs a set of quality control checks to help you assess the reliability of your data. The verification is split into two stages depending on the time resolution of the file:
- Basic range checks — applied to hourly values (hourly and sub-hourly data inputs)
- K-index and BSRN tests 1 — applied to sub-hourly values only
Except for mistakes in unit choice in the .MEF configuration, the verification process issues warnings. It is the user's responsibility to review the flagged timestamps and decide whether to clean the data externally, exclude the flagged intervals, or use the file as-is.
Variable definitions
The following irradiance variables and derived indices are used throughout the checks below.
| Symbol | Name | Unit | Description |
|---|---|---|---|
| GHI | GlobHor: Global Horizontal Irradiance | W/m² | Total solar irradiance on a horizontal surface |
| DHI (DIF) | DiffHor: Diffuse Horizontal Irradiance | W/m² | Diffuse (scattered) component on a horizontal surface |
| DNI | DNIMeas: Measured Direct Normal Irradiance | W/m² | Direct beam irradiance on a surface perpendicular to the sun |
| ETN | Extraterrestrial Normal Irradiance | W/m² | Solar irradiance above the atmosphere, corrected for Earth–Sun distance |
| SZA | Solar Zenith Angle | ° | Angle between the sun and the vertical. See Solar Geometry |
| Kt | Clearness index | ratio | Kt = GHI / (ETN × cos(SZA)). Clearness index or effective global horizontal transmittance2 |
| K | Diffuse fraction | ratio | K = DHI / GHI. Effective diffuse horizontal transmittance2 |
| Kn | Direct clearness index | ratio | Kn = DNI / ETN. Direct beam transmittance2 |
Basic range checks
These checks run on hourly values only (hourly and sub-hourly file imports). Their primary purpose is to catch obvious unit errors and gross physical inconsistencies. Checks are listed in the table below
| Variable | Checks | Warning | Action / Interpretation |
|---|---|---|---|
| GHI | GHI > ETN | Variable XX : outside of physical range | Hourly GHI exceeds extraterrestrial irradiance. Affected hour marked invalid. Verify unit. Verify site and time shift for POA imports |
| GHI | GHI > 5 W/m² at night | GlobHor value (XX w/m2) is invalid | Irradiance recorded at night. Affected hour marked invalid. Clean your data. |
| GHI | GHI ≤ 0 W/m² during the day | GlobHor value (0 w/m2) is invalid | No irradiance during the day. Check if missing data was expected. |
| DHI | DHI > GHI | Diffuse irradiance exceeds global irradiance | Verify units, column order and instrument calibration |
| Ambient temperature | Value > 60 °C | Variable XX : outside of physical range | Verify units and instrument calibration |
| Wind speed | Value > 35 m/s | Variable XX : outside of physical range | Possible unit mistake (e.g. km/h or mph instead of m/s). Can be ignored once unit is verified & wind expected to be that high |
| Relative humidity, Linke turbidity, AOD, Ground albedo | ratio in range [0-1] | Variable XX : outside of physical range | Verify unit (e.g. measurement in % imported as a ratio). Verify measurement validity (e.g. albedo <= 1 ). |
K-index and BSRN tests
These checks run on sub-hourly values only. They implement the quality control framework described in:
Forstinger, A. et al. (2021). Expert quality control of solar radiation ground data sets. Proceedings of the ISES Solar World Congress 2021. DOI: 10.18086/swc.2021.38.02
Two families of tests are applied: BSRN limits (physical and extremely-rare upper/lower bounds) and K-index tests that check for physical consistency between GHI, DHI and DNI using the normalised indices Kt, K and Kn.
The tracker-off tests and BSRN's closure tests are not implemented as PVsyst does not allow the import of all 3 GHI, DHI and DNI components at the same time.
BSRN limits
These limits are computed dynamically at each timestep as a function of ETN and cos(SZA). Two tiers exist:
- Extremely rare limits (ERL) — values that are physically possible but statistically exceptional. A flag here warrants attention.
- Physical possible limits (PPL) — values that cannot be produced by the real atmosphere. A flag here almost certainly indicates a measurement error. A Measurement flagged here will also be flagged with the corresponding (ERL flag).
The list of flags and limit is given below
| Flag | Variable | Lower bound | Upper bound | Interpretation |
|---|---|---|---|---|
flagERLGHI | GHI | −2 W/m² | \(1.2 × ETN × cos(SZA)^{1.2} + 50\) | GHI is in the extremely-rare range: very high but not strictly impossible, or small negative |
flagPPLGHI | GHI | −4 W/m² | \(1.5 × ETN × cos(SZA)^{1.2} + 100\) | GHI exceeds physical limits: almost certainly a sensor or unit error, or large negative clearly erroneous. Data cleaning advised |
flagERLDIF | DHI | −2 W/m² | \(0.75 × ETN × cos(SZA)^{1.2} + 30\) | DHI is in the extremely-rare range, or small negative likely |
flagPPLDIF | DHI | −4 W/m² | \(0.95 × ETN × cos(SZA)^{1.2} + 50\) | DHI exceeds physical limits, or large negative clearly erroneous. Data cleaning advised |
flagERLDNI | DNI | −2 W/m² | \(0.95 × ETN × cos(SZA)^{0.2} + 10\) | DNI is in the extremely-rare range, or small negative |
flagPPLDNI | DNI | −4 W/m² | ETN | DNI cannot exceed extraterrestrial normal irradiance, or large negative clearly erroneous. Data cleaning advised |
K-index tests
These six tests check for physical consistency between the normalised irradiance indices. They require that at least one of Kt, K or Kn be available and GHI > 50 W/m2 in most cases.
| Flag | limit | Domain | Interpretation |
|---|---|---|---|
flagKnKt | Kn < Kt | GHI > 50 W/m², Kn > 0, Kt > 0 | The direct beam transmittance exceeds the total clearness index — physically impossible, as the direct component cannot account for more of the extraterrestrial flux than the total |
flagKn | Kn < ((1100 + 0.03 × altitude) / ETN) | GHI > 50 W/m², Kn > 0 | DNI is unrealistically high relative to the solar constant. The limit is slightly relaxed at high-altitude sites where the atmosphere is thinner |
flagKt | Kt < 1.35 | GHI > 50 W/m², Kt > 0 | GHI exceeds 135% of the horizontal extraterrestrial component. Typical clear-sky Kt values are around 0.7–0.8 |
flagKlowSZA | K < 1.05 | SZA < 75°, GHI > 50 W/m², K > 0 | When the sun is high, diffuse irradiance should not exceed GHI. A diffuse fraction above 1.05 is anomalous |
flagKhighSZA | K < 1.10 | SZA ≥ 75°, GHI > 50 W/m², K > 0 | At low sun angles, the tolerance is slightly relaxed to 1.10 to account for enhanced forward scattering near the horizon |
flagKKt | K < 0.96 | Kt > 0.6, GHI > 150 W/m², SZA < 85°, K > 0 | Under clear-sky conditions (high Kt), most irradiance is direct — a diffuse fraction above 0.96 under clear skies suggests a sensor error |
How to act on warnings
The warnings from basic range checks often mean there is an issue in the .MEF configurations. Please read them carefully and try to correct the .MEF file
For K-index and BSRN tests:
- Isolated flags : High irradiance values can be due to cloud enhancement and the presence of a few rare BSRN flags can be ignored. PPL flags should still be investigated and understood.
- Systematic flags : Data should be checked and the source of the problem should be identified. It is not recommended to run a simulation on the generated file without addressing the flagged issue.
Updates
- Added in PVsyst 8.1.0
References
-
A. Forstinger, S. Wilbert, A.R. Jensen et al.
Expert quality control of solar radiation ground data sets.
Proceedings of the ISES Solar World Congress 2021. DOI: 10.18086/swc.2021.38.02 ↩ -
N. Geuder, F. Wolfertstetter, S. Wilbert et al.
Screening and flagging of solar irradiation and ancillary meteorological data.
Energy Procedia, Vol. 69, pp. 1989–1998, 2015. DOI: 10.1016/j.egypro.2015.03.205 ↩↩↩