Q- band Array Integration,
4.4 Data Selection and Reduction
4.4.6 Weather Cut
Description and Design
We designed a cut to remove data taken during poor weather conditions. This section describes the development of the weather cut, the final product, and studies performed to ensure this cut did not bias the data set.
Contributions from weather are assessed with the double-demodulated time stream, downsampled to one second. We process the data first by binning the data into 10 second bins for one scan and one diode, and computing the standard deviation of each bin. We then compute the standard deviation of the distribution of standard deviation values. This yields a single value which encodes the variability of noise between 10 second time scales. We will call this the weather variable. We repeat this computation for all scans, giving a distribution of the weather variable for a particular diode over the observing season. We fit a Gaussian to the distribution of the weather variable for all scans of a given patch, and compute the gaussian width (σ) and mean (µ). We note any scans which have a weather variable greater than 5σ from the mean.
We repeat this for all diodes, and any scan for which 70% or more of the diodes lie outside of the 5σ limit is cut from the data set. We repeat this for a 30 second bin size, and for each patch, such that a scan can be cut by either the 10-second or 30-second bin size distributions. The distribution of the weather variable for module RQ09 (the central polarimeter) for all diodes is shown in Figure 4-6 for the 10-second bin size, the red vertical lines are the 5-σ limit. We performed various studies to assess the accuracy of the weather cut and that we were not biasing the data set.
Those studies are described in the following sections.
(a)
Patch Coordinates Time
RA DEC Hours
6a 0h 48m −48◦ 900
4a 5h 12m −39◦ 768
2a 12h 4m −39◦ 1002
7b 22h 44m −36◦ 243
Gb (Galactic) 16h 0m −53◦ 320 Gc (Galactic) 17h 46m −29◦ 110
Calibration 142
Total CMB 2913
(b)
Figure 4-3: a: QUIET sky patches (circles), plotted over the WMAP Q-band tem-perature map (Ref. [36]) b: Hours spent on each QUIET patch with no data cuts imposed and coordinates in J2000. Because it is far from the other patches, Patch 2a was observed almost without interruption each day from the time it rose to the time it set and has the most integrated hours. Patch 7b, which had overlapping scan times with Patch 6a, was observed less frequently than the other CMB patches.
(a)
Figure 4-4: Total power vs. demodulated time stream before and after de-glitching for module RQ15, Q1 for scan 437.2. The cyan line shows the location of the glitch;
the χ2 was 49.2 before de-glitching, and 1.9 afterwards. Courtesy Immanuel Buder (Ref. [18]).
(a) (b)
(c) (d)
Figure 4-5: Maximum PS21 current for all scans for a: RQ11, as a function of time, b: RQ11, the distribution of currents, c: RQ12, as a function of time, and d: RQ12, the distribution of currents. The red vertical lines in the distributions denote the chosen maximum current value in mA for the phase-switch cut.
Figure 4-6: Histogram of standard deviation of standard deviation of binned data (10 second bins), Module 9. The red lines indicate 5-σ of the distribution.
Studies
Time Scales for Weather Variable
The temperature of the enclosure drifts on a variety of time scales, and with it, the polarimeter data stream. This effect can be corrected in further analysis steps, and so we must choose a weather variable which selects only periods of bad weather, and does not flag data which is varying only from the enclosure temperature. The two effects are illustrated in Figures 4-7(a) and 4-7(b); these show the time-streams for scan 404, which has a clear spike originating from a cloud, and scan 1776, which has a signal envelope dependent only on the enclosure temperature and is not an example of bad weather.
To isolate and cut scans which are affected by bad weather, we investigated a variety of binning time scales: 5 seconds, 10 seconds, 30 seconds, 60 seconds, and 120 seconds. The standard deviation of each bin for these bin sizes is shown for scan 404 (Figure 4-8(a)) and scan 1776 (Figure 4-8(b)). The significance of the weather variable for each of these bin sizes for both scans is given in Table 4.2. The spike from weather in scan 404 was detected at all bin sizes. Enclosure temperature variation was apparent by a bin size of 60 seconds as it includes the rise of the enclosure temperature in the RMS statistic. The 30 second bin size generally had the highest significance for weather. We included the 10 second time bin because it is near the scan frequency, and so will have sensitivity to stationary weather patterns. The overlap between the two bin-size cuts is �80%, and is dominated by the Q-diodes (which have higher leakage and make up a larger percentage of the weather cut). A visual inspection of all scans which were cut by only one showed that both cuts were removing bad data, so both cuts were retained.
Bin size CES 404.5 CES 1776.1
5 19σ 0.1σ
10 30σ 0.6σ
30 33σ 0.9σ
60 33σ 2.5σ
120 35σ 10.7σ
Table 4.2: The significance of the weather variable for a set of different bin sizes for scans 404 (bad weather) and 1776 (enclosure drift), RQ09 diode Q1 (DD1).
(a)
(b)
Figure 4-7: Demodulated stream for module RQ09 diode Q1 (DD1) binned into 5, 10, 30, and 120 second time bins for a: Scan 404, segment 5, which has a spike from weather in all bin sizes and b: Scan 1776, segment 1, which varies only with enclosure temperature.
(a)
(b)
Figure 4-8: a: Standard deviation per bin for module RQ09 diode Q1 (DD1) for scan 404 segment 5, for bin sizes of 5 seconds, 10 seconds, 30 seconds, and 120 seconds.
The spike is from weather (likely a cloud). b: The same for scan 1776 segment 1. The envelope in the standard deviation comes from variation with enclosure temperature.
Bi-modal Distributions
We found many modules had distinctly different distributions in the weather variable between the two halves of the season, however there was nothing apparent in the data stream. We investigated whether this was due to enclosure temperature variation or differing weather conditions between the two halves of the season, however neither of these were contributing factors to the bimodal distributions. The underlying cause of the change in noise properties over the season was not resolved. We may be able to tailor the weather cut to each half of the season, this is currently under investigation.
Leakage
Water vapor is linearly polarized to only a small degree, �1% (Ref.[34]), while the high-leakage modules have I→Q leakage of order 1-2% (discussed in sections 2.2.4, 5.8), such that the polarization TODs are sensitive to water vapor and cloud-cover primar-ily through I→Q/U leakage. Because the weather cut is based on the (unfiltered) demodulated stream, and hence is sensitive to only the linear polarization of the atmosphere and the leakage, the majority of the fluctuations present in the RMS statistic come from leakage from the total power weather-based fluctuations in the atmosphere into the polarized data stream. As a result, the majority of the diodes which comprise the 70% of diodes in the weather cut will tend to be those with rel-atively higher leakage. This is shown in Figure 4-9, which shows how frequently a diode was included in the 70% of modules contributing to cutting a particular scan as a function of leakage. Q-diodes have higher leakage and so are preferentially used in this statistic.
If we were cutting diode-by-diode or module-by-module, this would introduce a large systematic effect of only cutting modules or diodes with high leakage. However, the weather cut removes all diodes in a flagged scan, so we are not biasing the data set by cutting on a diode-by-diode basis. In addition, the weather cut requires at least 70% of the diodes to be cut such that it requires lower-leakage diodes to flare up as well for the scan to be removed.
Figure 4-9: The weather cut requires that 70% of the diodes lie outside of a 5-σ threshold, this shows which diodes make up that 70% as a function of leakage. It is apparent that higher-leakage modules appear more frequently in the list of modules cut. Because weather effects both the demodulated and total power streams, and leakage is contamination from total power into the demodulated stream, this isn’t unexpected. This study was done with patch 2a data only.
Bias
We created a set of simulated time-ordered-data with noise only (no signal) using the same simulation code we use in the Maximum-likelihood analysis pipeline for power spectrum analysis (section 6.4.3). The simulation code uses the pointing and calibration information for a set of selected scans (in our case 44, ideally we would draw a larger sample size but we have been limited by computation time), and uses the noise model (described below in section 4.4.7) and an input power spectrum to generate a set of TODs. In our case, the signal spectrum is null, allowing us to test
whether or not the weather monitor will bias the data set by removing scans which only contain noise. We used identical noise properties between the 44 scans, with νknee = 10mHz, α = -2.0, and σ0=1×10−5. For each scan, an FFT was generated and then transformed back to TOD space. The resulting TOD for each scan and each diode were analyzed by the weather cutting program. If the weather cut had removed a scan, this would indicate it cuts on random noise, which would bias the data set.
There were no cases where 70% of the diodes all had 5σ outliers for a given scan, so no data was cut, and the weather cut is not contributing to bias in the data set.