Skip to content

Custom Data Verification

Since version 8.1.0, when importing a meteorological file through custom import, PVsyst runs a set of quality control checks to help you assess the reliability of your data. The verification is split into two stages depending on the time resolution of the file:

  • Basic range checks — applied to hourly values (hourly and sub-hourly data inputs).
  • K-index and BSRN tests 1 — applied to sub-hourly values only.

Except for unit-choice mistakes in the .MEF configuration, the verification process issues warnings. It is the user's responsibility to review the flagged timestamps and decide whether to clean the data externally, exclude the flagged intervals, or use the file as-is.

Variable definitions

The following irradiance variables and derived indices are used throughout the checks below.

Symbol Name Unit Description
GHI GlobHor: Global Horizontal Irradiance W/m² Total solar irradiance on a horizontal surface
DHI (DIF) DiffHor: Diffuse Horizontal Irradiance W/m² Diffuse (scattered) component on a horizontal surface
DNI DNIMeas: Measured Direct Normal Irradiance W/m² Direct beam irradiance on a surface perpendicular to the sun
ETN Extraterrestrial Normal Irradiance W/m² Solar irradiance above the atmosphere, corrected for Earth–Sun distance
SZA Solar Zenith Angle ° Angle between the sun and the vertical. See Solar Geometry
Kt Clearness index ratio Kt = GHI / (ETN × cos(SZA)). Clearness index or effective global horizontal transmittance 2
K Diffuse fraction ratio K = DHI / GHI. Effective diffuse horizontal transmittance 2
Kn Direct clearness index ratio Kn = DNI / ETN. Direct beam transmittance 2

Basic range checks

These checks run on hourly values only (hourly and sub-hourly file imports). Their primary purpose is to catch obvious unit errors and gross physical inconsistencies. The checks are listed in the table below.

Variable Checks Warning Action / Interpretation
GHI GHI > ETN Variable XX : outside of physical range Hourly GHI exceeds extraterrestrial irradiance. Affected hour marked invalid. Verify unit. Verify site and time shift for POA imports
GHI GHI > 5 W/m² at night GlobHor value (XX w/m2) is invalid Irradiance recorded at night. Affected hour marked invalid. Clean your data.
GHI GHI ≤ 0 W/m² during the day GlobHor value (0 w/m2) is invalid No irradiance during the day. Check if missing data was expected.
DHI DHI > GHI Diffuse irradiance exceeds global irradiance Verify units, column order and instrument calibration
Ambient temperature Value > 60 °C Variable XX : outside of physical range Verify units and instrument calibration
Wind speed Value > 35 m/s Variable XX : outside of physical range Possible unit mistake (e.g. km/h or mph instead of m/s). Can be ignored once unit is verified & wind expected to be that high
Relative humidity, Linke turbidity, AOD, Ground albedo ratio in range [0-1] Variable XX : outside of physical range Verify unit (e.g. measurement in % imported as a ratio). Verify measurement validity (e.g. albedo <= 1 ).

KTcs checks

These checks run on hourly values only (hourly and sub-hourly file imports). This test checks that the clearness index (Kt) values for the best clear days of the dataset lie within reasonable bounds. If they are too far out of bounds, warnings or errors will be displayed.

Error type Message Default limit
Fatal error The clearness index of the best clear days is way too high with respect to the clear day model. Kt > 1.2
Warning The clearness index of the best clear days seems high with respect to the clear day model. Kt > 1.1
Fatal error The clearness index of the best clear days is way too low with respect to the clear day model. Kt < 0.85
Warning The clearness index of the best clear days seems low with respect to the clear day model. Kt < 0.93

Interpretation

Interpretation may be difficult, as high or low Kt values can result from either valid physical causes or from measurement/model errors.

Valid causes of high Kt values include:

  • Cloud enhancement
  • High reflectance from the ground (snow, large water bodies)
  • High altitude

Valid causes of low Kt values include:

  • Persistent thick cloud cover
  • High aerosol content (dust, pollution, smoke). See NASA map

Action

  1. Check that you have selected the correct irradiance column, irradiance unit, and multiplier values.
  2. Fix any time shift issues you might have.
  3. For POA import, check that your site location and POA orientation are correct. If you also have a GHI measurement, import GHI instead.
  4. Open the Best clear days KTcs plots and identify the days flagged.
  5. Open the graphs and inspect the GHI and DHI on these days.

It is up to the user to identify whether they are facing calibration issues or are observing one of the phenomena described above. When unsure, you may import data from another source to check for systematic bias.

K-index and BSRN tests

These checks run on sub-hourly values only. They implement the quality control framework described in 1

Two families of tests are applied: BSRN limits (physical and extremely-rare upper/lower bounds) and K-index tests, which check for physical consistency between GHI, DHI and DNI using the normalised indices Kt, K and Kn.

The tracker-off tests and BSRN's closure tests are not implemented, as PVsyst does not allow the simultaneous import of all three components (GHI, DHI and DNI).

BSRN limits

These limits are computed dynamically at each timestep as a function of ETN and cos(SZA). Two tiers exist:

  • Extremely rare limits (ERL) — values that are physically possible but statistically exceptional. A flag here warrants attention.
  • Physically possible limits (PPL) — values that cannot be produced by the real atmosphere. A flag here almost certainly indicates a measurement error. A measurement flagged here will also be flagged with the corresponding ERL flag.

The list of flags and limits is given below.

Flag Variable Lower bound Upper bound Interpretation
flagERLGHI GHI −2 W/m² \(1.2 × ETN × cos(SZA)^{1.2} + 50\) GHI is in the extremely-rare range: very high but not strictly impossible, or small negative
flagPPLGHI GHI −4 W/m² \(1.5 × ETN × cos(SZA)^{1.2} + 100\) GHI exceeds physical limits: almost certainly a sensor or unit error, or large negative clearly erroneous. Data cleaning advised
flagERLDIF DHI −2 W/m² \(0.75 × ETN × cos(SZA)^{1.2} + 30\) DHI is in the extremely-rare range, or small negative likely
flagPPLDIF DHI −4 W/m² \(0.95 × ETN × cos(SZA)^{1.2} + 50\) DHI exceeds physical limits, or large negative clearly erroneous. Data cleaning advised
flagERLDNI DNI −2 W/m² \(0.95 × ETN × cos(SZA)^{0.2} + 10\) DNI is in the extremely-rare range, or small negative
flagPPLDNI DNI −4 W/m² ETN DNI cannot exceed extraterrestrial normal irradiance, or large negative clearly erroneous. Data cleaning advised

K-index tests

These six tests check for physical consistency between the normalised irradiance indices. They require that at least one of Kt, K or Kn be available, and GHI > 50 W/m² in most cases.

Flag Limit Domain Interpretation
flagKnKt Kn < Kt GHI > 50 W/m², Kn > 0, Kt > 0 The direct beam transmittance exceeds the total clearness index — physically impossible, as the direct component cannot account for more of the extraterrestrial flux than the total.
flagKn Kn < ((1100 + 0.03 × altitude) / ETN) GHI > 50 W/m², Kn > 0 DNI is unrealistically high relative to the solar constant. The limit is slightly relaxed at high-altitude sites where the atmosphere is thinner.
flagKt Kt < 1.35 GHI > 50 W/m², Kt > 0 GHI exceeds 135% of the horizontal extraterrestrial component. Typical clear-sky Kt values are around 0.7–0.8.
flagKlowSZA K < 1.05 SZA < 75°, GHI > 50 W/m², K > 0 When the sun is high, diffuse irradiance should not exceed GHI. A diffuse fraction above 1.05 is anomalous.
flagKhighSZA K < 1.10 SZA ≥ 75°, GHI > 50 W/m², K > 0 At low sun angles, the tolerance is slightly relaxed to 1.10 to account for enhanced forward scattering near the horizon.
flagKKt K < 0.96 Kt > 0.6, GHI > 150 W/m², SZA < 85°, K > 0 Under clear-sky conditions (high Kt), most irradiance is direct — a diffuse fraction above 0.96 under clear skies suggests a sensor error.

How to act on warnings

The warnings from basic range checks often indicate an issue in the .MEF configuration. Please read it carefully and correct the .MEF file accordingly.

For K-index and BSRN tests:

  • Isolated flags : High irradiance values can be due to cloud enhancement and the presence of a few rare BSRN flags can be ignored. PPL flags should still be investigated and understood.
  • Systematic flags : Data should be checked and the source of the problem should be identified. It is not recommended to run a simulation on the generated file without addressing the flagged issue.

Updates

  • Added in PVsyst 8.1.0

  1. Anne Forstinger, Stefan Wilbert, Birk Kraas, Carlos Fernández Peruchena, Chris A. Gueymard, Elena Collino, Jose A Ruiz-Arias, Jesús Polo Martinez, Yves-Marie Saint-Drenan, Dario Ronzio, Natalie Hanrieder, Adam R. Jensen, and Dazhi Yang. Expert Quality Control of Solar Radiation Ground Data Sets. In Proceedings of the ISES Solar World Congress 2021, 1–12. International Solar Energy Society, 2021. URL: http://proceedings.ises.org/citation?doi=swc.2021.38.02, doi:10.18086/swc.2021.38.02. ↩↩

  2. N. Geuder, F. Wolfertstetter, S. Wilbert, D. Schüler, R. Affolter, B. Kraas, E. Lüpfert, and B. Espinar. Screening and flagging of solar irradiation and ancillary meteorological data. Energy Procedia, 69:1989–1998, May 2015. doi:10.1016/j.egypro.2015.03.205. ↩↩↩