Dalitz analysis of D0→K-π+η decays at Belle

Dalitz analysis of D

⁰

→ K

⁻

π

⁺

η decays at Belle

Y. Q. Chen,⁷³L. K. Li ,^27,8W. B. Yan,⁷³I. Adachi,^18,14H. Aihara,⁸⁵S. Al Said,^79,37D. M. Asner,⁴H. Atmacan,⁸ V. Aulchenko,^5,65T. Aushev,⁵⁵R. Ayad,⁷⁹V. Babu,⁹ I. Badhrees,^79,36S. Bahinipati,²³P. Behera,²⁵J. Bennett,⁵² V. Bhardwaj,²²T. Bilka,⁶ J. Biswal,³³A. Bozek,⁶²M. Bračko,^49,33T. E. Browder,¹⁷ M. Campajola,^30,57 L. Cao,³ D. Červenkov,⁶ M.-C. Chang,¹⁰V. Chekelian,⁵⁰A. Chen,⁵⁹B. G. Cheon,¹⁶K. Chilikin,⁴⁴H. E. Cho,¹⁶ K. Cho,³⁹ S.-K. Choi,¹⁵Y. Choi,⁷⁷ S. Choudhury,²⁴D. Cinabro,⁹⁰S. Cunliffe,⁹ N. Dash,²³ G. De Nardo,^30,57F. Di Capua,^30,57 Z. Doležal,⁶ T. V. Dong,¹¹S. Eidelman,^5,65,44 D. Epifanov,^5,65 J. E. Fast,⁶⁷T. Ferber,⁹ D. Ferlewicz,⁵¹B. G. Fulsom,⁶⁷ R. Garg,⁶⁸V. Gaur,⁸⁹N. Gabyshev,^5,65A. Garmash,^5,65A. Giri,²⁴P. Goldenzweig,³⁴B. Golob,^45,33Y. Guan,⁸O. Hartbrich,¹⁷

K. Hayasaka,⁶⁴H. Hayashii,⁵⁸ W.-S. Hou,⁶¹C.-L. Hsu,⁷⁸K. Inami,⁵⁶G. Inguglia,²⁸A. Ishikawa,^18,14 R. Itoh,^18,14 M. Iwasaki,⁶⁶ Y. Iwasaki,¹⁸W. W. Jacobs,²⁶E.-J. Jang,¹⁵ H. B. Jeon,⁴¹S. Jia,² Y. Jin,⁸⁵ K. K. Joo,⁷ K. H. Kang,⁴¹ G. Karyan,⁹T. Kawasaki,³⁸ D. Y. Kim,⁷⁶S. H. Kim,¹⁶ T. D. Kimmel,⁸⁹K. Kinoshita,⁸ P. Kodyš,⁶S. Korpar,^49,33 P. Križan,^45,33 R. Kroeger,⁵²P. Krokovny,^5,65T. Kuhr,⁴⁶R. Kulasiri,³⁵R. Kumar,⁷¹A. Kuzmin,^5,65Y.-J. Kwon,⁹² K. Lalwani,⁴⁸J. S. Lange,¹² I. S. Lee,¹⁶S. C. Lee,⁴¹Y. B. Li,⁶⁹L. Li Gioi,⁵⁰J. Libby,²⁵K. Lieret,⁴⁶D. Liventsev,^89,18 J. MacNaughton,⁵³C. MacQueen,⁵¹M. Masuda,⁸⁴D. Matvienko,^5,65,44M. Merola,^30,57K. Miyabayashi,⁵⁸R. Mizuk,^44,55

S. Mohanty,^80,88M. Mrvar,²⁸R. Mussa,³¹M. Nakao,^18,14 Z. Natkaniec,⁶²M. Nayak,⁹³S. Nishida,^18,14 S. Ogawa,⁸² H. Ono,^63,64P. Oskin,⁴⁴P. Pakhlov,^44,54 G. Pakhlova,^44,55S. Pardi,³⁰H. Park,⁴¹S. Patra,²²S. Paul,⁸¹T. K. Pedlar,⁴⁷ R. Pestotnik,³³L. E. Piilonen,⁸⁹T. Podobnik,^45,33V. Popov,^44,55E. Prencipe,²⁰M. T. Prim,³⁴A. Rabusov,⁸¹M. Ritter,⁴⁶ M. Röhrken,⁹N. Rout,²⁵G. Russo,⁵⁷D. Sahoo,⁸⁰Y. Sakai,^18,14T. Sanuki,⁸³V. Savinov,⁷⁰O. Schneider,⁴³G. Schnell,^1,21 J. Schueler,¹⁷C. Schwanda,²⁸A. J. Schwartz,⁸ Y. Seino,⁶⁴K. Senyo,⁹¹M. E. Sevior,⁵¹M. Shapkin,²⁹V. Shebalin,¹⁷ J.-G. Shiu,⁶¹A. Sokolov,²⁹E. Solovieva,⁴⁴M. Starič,³³Z. S. Stottler,⁸⁹M. Sumihama,¹³T. Sumiyoshi,⁸⁷W. Sutcliffe,³

M. Takizawa,^74,19,72 K. Tanida,³²F. Tenchini,⁹K. Trabelsi,⁴² M. Uchida,⁸⁶T. Uglov,^44,55S. Uno,^18,14 P. Urquijo,⁵¹ G. Varner,¹⁷V. Vorobyev,^5,65,44E. Waheed,¹⁸C. H. Wang,⁶⁰ E. Wang,⁷⁰M.-Z. Wang,⁶¹P. Wang,²⁷M. Watanabe,⁶⁴

E. Won,⁴⁰X. Xu,⁷⁵S. B. Yang,⁴⁰H. Ye,⁹ J. H. Yin,²⁷C. Z. Yuan,²⁷Y. Yusa,⁶⁴Z. P. Zhang,⁷³V. Zhilich,^5,65 V. Zhukova,⁴⁴and V. Zhulanov^5,65

(Belle Collaboration)

1University of the Basque Country UPV/EHU, 48080 Bilbao

2Beihang University, Beijing 100191

3University of Bonn, 53115 Bonn

4Brookhaven National Laboratory, Upton, New York 11973

5Budker Institute of Nuclear Physics SB RAS, Novosibirsk 630090

6Faculty of Mathematics and Physics, Charles University, 121 16 Prague

7Chonnam National University, Gwangju 61186

8University of Cincinnati, Cincinnati, Ohio 45221

9Deutsches Elektronen–Synchrotron, 22607 Hamburg

10Department of Physics, Fu Jen Catholic University, Taipei 24205

11Key Laboratory of Nuclear Physics and Ion-beam Application (MOE) and Institute of Modern Physics, Fudan University, Shanghai 200443

12Justus-Liebig-Universität Gießen, 35392 Gießen

13Gifu University, Gifu 501-1193

14SOKENDAI (The Graduate University for Advanced Studies), Hayama 240-0193

15Gyeongsang National University, Jinju 52828

16Department of Physics and Institute of Natural Sciences, Hanyang University, Seoul 04763

17University of Hawaii, Honolulu, Hawaii 96822

18High Energy Accelerator Research Organization (KEK), Tsukuba 305-0801

19J-PARC Branch, KEK Theory Center, High Energy Accelerator Research Organization (KEK), Tsukuba 305-0801

20Forschungszentrum Jülich, 52425 Jülich

21IKERBASQUE, Basque Foundation for Science, 48013 Bilbao

22Indian Institute of Science Education and Research Mohali, SAS Nagar, 140306

23Indian Institute of Technology Bhubaneswar, Satya Nagar 751007

24Indian Institute of Technology Hyderabad, Telangana 502285

25Indian Institute of Technology Madras, Chennai 600036

26Indiana University, Bloomington, Indiana 47408

27Institute of High Energy Physics, Chinese Academy of Sciences, Beijing 100049

28Institute of High Energy Physics, Vienna 1050

29Institute for High Energy Physics, Protvino 142281

30INFN—Sezione di Napoli, 80126 Napoli

31INFN—Sezione di Torino, 10125 Torino

32Advanced Science Research Center, Japan Atomic Energy Agency, Naka 319-1195

33J. Stefan Institute, 1000 Ljubljana

34Institut für Experimentelle Teilchenphysik, Karlsruher Institut für Technologie, 76131 Karlsruhe

35Kennesaw State University, Kennesaw, Georgia 30144

36King Abdulaziz City for Science and Technology, Riyadh 11442

37Department of Physics, Faculty of Science, King Abdulaziz University, Jeddah 21589

38Kitasato University, Sagamihara 252-0373

39Korea Institute of Science and Technology Information, Daejeon 34141

40Korea University, Seoul 02841

41Kyungpook National University, Daegu 41566

42LAL, Univ. Paris-Sud, CNRS/IN2P3, Universit´e Paris-Saclay, Orsay 91898

43École Polytechnique F´ed´erale de Lausanne (EPFL), Lausanne 1015

44P.N. Lebedev Physical Institute of the Russian Academy of Sciences, Moscow 119991

45Faculty of Mathematics and Physics, University of Ljubljana, 1000 Ljubljana

46Ludwig Maximilians University, 80539 Munich

47Luther College, Decorah, Iowa 52101

48Malaviya National Institute of Technology Jaipur, Jaipur 302017

49University of Maribor, 2000 Maribor

50Max-Planck-Institut für Physik, 80805 München

51School of Physics, University of Melbourne, Victoria 3010

52University of Mississippi, University, Mississippi 38677

53University of Miyazaki, Miyazaki 889-2192

54Moscow Physical Engineering Institute, Moscow 115409

55Moscow Institute of Physics and Technology, Moscow Region 141700

56Graduate School of Science, Nagoya University, Nagoya 464-8602

57Universit `a di Napoli Federico II, 80055 Napoli

58Nara Women’s University, Nara 630-8506

59National Central University, Chung-li 32054

60National United University, Miao Li 36003

61Department of Physics, National Taiwan University, Taipei 10617

62H. Niewodniczanski Institute of Nuclear Physics, Krakow 31-342

63Nippon Dental University, Niigata 951-8580

64Niigata University, Niigata 950-2181

65Novosibirsk State University, Novosibirsk 630090

66Osaka City University, Osaka 558-8585

67Pacific Northwest National Laboratory, Richland, Washington 99352

68Panjab University, Chandigarh 160014

69Peking University, Beijing 100871

70University of Pittsburgh, Pittsburgh, Pennsylvania 15260

71Punjab Agricultural University, Ludhiana 141004

72Theoretical Research Division, Nishina Center, RIKEN, Saitama 351-0198

73Department of Modern Physics and State Key Laboratory of Particle Detection and Electronics, University of Science and Technology of China, Hefei 230026

74Showa Pharmaceutical University, Tokyo 194-8543

75Soochow University, Suzhou 215006

76Soongsil University, Seoul 06978

77Sungkyunkwan University, Suwon 16419

78School of Physics, University of Sydney, New South Wales 2006

79Department of Physics, Faculty of Science, University of Tabuk, Tabuk 71451

80Tata Institute of Fundamental Research, Mumbai 400005

81Department of Physics, Technische Universität München, 85748 Garching

82Toho University, Funabashi 274-8510

83Department of Physics, Tohoku University, Sendai 980-8578

84Earthquake Research Institute, University of Tokyo, Tokyo 113-0032

85Department of Physics, University of Tokyo, Tokyo 113-0033

86Tokyo Institute of Technology, Tokyo 152-8550

87Tokyo Metropolitan University, Tokyo 192-0397

88Utkal University, Bhubaneswar 751004

89Virginia Polytechnic Institute and State University, Blacksburg, Virginia 24061

90Wayne State University, Detroit, Michigan 48202

91Yamagata University, Yamagata 990-8560

92Yonsei University, Seoul 03722

93School of Physics and Astronomy, Tel Aviv University, Tel Aviv 69978

(Received 17 March 2020; accepted 18 June 2020; published 6 July 2020)

We present the results of the first Dalitz plot analysis of the decay D⁰→K⁻π^þη. The analysis is performed on a data set corresponding to an integrated luminosity of 953fb⁻¹ collected by the Belle detector at the asymmetric-energye^þe⁻KEKB collider. The Dalitz plot is well described by a combination of the six resonant decay channelsK¯ð892Þ⁰η,K⁻a₀ð980Þ^þ,K⁻a₂ð1320Þ^þ,K¯ð1410Þ⁰η,Kð1680Þ⁻π^þ and K₂ð1980Þ⁻π^þ, together withKπ and KηS-wave components. The decaysKð1680Þ⁻→K⁻ηand K₂ð1980Þ⁻→K⁻η are observed for the first time. We measure ratio of the branching fractions,

BðD⁰→K⁻π^þηÞ

BðD⁰→K⁻π^þÞ ¼0.5000.002ðstatÞ 0.020ðsystÞ 0.003ðBPDGÞ. Using the Dalitz fit result, the ratio

BðKð1680Þ→KηÞ

BðKð1680Þ→KπÞis measured to be 0.110.02ðstatÞ^þ0.06_−0.04ðsystÞ 0.04ðBPDGÞ; this is much lower than the theoretical expectations (≈1) made under the assumption thatKð1680Þis a pure1³D₁state. The product branching fractionBðD⁰→½K₂ð1980Þ⁻→K⁻ηπ^þÞ ¼ ð2.2^þ1_−1.9^.7Þ×10⁻⁴is determined. In addition, theπη⁰ contribution to the a₀ð980Þ resonance shape is confirmed with10.1σ statistical significance using the three-channel Flatt´e model. We also measureBðD⁰→K¯ð892Þ⁰ηÞ ¼ ð1.41^þ0.13₋₀.12Þ%. This is consistent with, and more precise than, the current world averageð1.020.30Þ%, deviates with a significance of more than 3σ from the theoretical predictions of (0.51–0.92)%.

DOI:10.1103/PhysRevD.102.012002

I. INTRODUCTION

The understanding of hadronic charmed-meson decays is theoretically challenging due to the significant nonpertur- bative contributions, and input from experimental mea- surements thus plays an important role[1–3]. We present a Dalitz plot (DP) analysis[4]to study the dynamics of three body decayD⁰→K⁻π^þη. This decay is Cabibbo-favored (CF) and proceeds via the c→sud¯ transition. Because of isospin symmetry, intermediate states of this decay (e.g., excited kaon states decaying into Kπ or Kη), and a-family mesons decaying intoπη, are similar to those in D⁰→K⁰_Sπ⁰η. The DP analysis of the latter channel has previously been performed, and the intermediate channels K⁰_Sa₀ð980Þ⁰andK¯ð892Þ⁰η[5]were found to be dominant, but additional components of a nonresonant amplitude, K₀ð1430Þη, K⁰_Sa₂ð1320Þ, κη, and combinations of these processes, were found to contribute significantly. However, the statistical power of that sample was too limited for

precise measurements to be made. TheD⁰→K¯⁰ηdecay is sensitive to theW-exchange diagram, which is important for the theoretical understanding of charm decays.

The theoretical predictions of the branching fraction of this mode vary in the range (0.51–0.92)% depending on the method[1–3]. This is consistent with, but smaller than, the current experimental result of ð1.020.30Þ%

[6] obtained in the D⁰→K⁰_Sπ⁰η final state [5]. A more precise measurement of this branching fraction from D⁰→K⁻π^þηdecays would test the theoretical predictions.

The K₀ð1430Þ →Kη decay was observed by the BABAR experiment [7] and is awaiting confirmation.

Experimentally, the ratio of K₀ð1430Þ decaying into Kη and Kπ is 0.09^þ0.03_−0.04 [6], which is consistent with the theoretical prediction of 0.05 [8] which was made with the assumption that it is a pure1³P₀ state.

Decays of some other excited kaons to Kη, including Kð1410Þ, Kð1680Þ and K₂ð1980Þ, were predicted by Refs. [8,9] but have not yet been observed. These states may have some interesting properties;Kð1410Þ may not be a simple2³S₁ state, and Kð1410Þ andKð1680Þ are predicted to be a mixture of the 2³S₁ and 1³D₁ states.

AssumingKð1410ÞandKð1680Þare pure2³S₁and1³D₁ states, respectively, the relative branching ratio of Kð1680Þ to Kη andKπ should be close to one (1.18 in Published by the American Physical Society under the terms of

the Creative Commons Attribution 4.0 International license.

Further distribution of this work must maintain attribution to the author(s) and the published article’s title, journal citation, and DOI. Funded by SCOAP³.

Ref.[8]and 0.93 in Ref.[9]) Experimentally, no branching ratio measurement for the former channel has previously been made.

The nature of thea₀ð980Þis still not clear. Since it is a dominant intermediate resonance inD⁰→K⁻π^þη, we can collect a large sample of a₀ð980Þ^þ decays to study its character further, e.g., to confirm the πη⁰ contribution to the a₀ð980Þ line shape in a Flatt´e model as measured by BESIII[10]. Such a study can also help determine theπη andKK¯ contributions toa₀ð980Þprecisely and understand its quark component.

Wrong-sign (WS) decays play an important role in studies of D⁰-D¯⁰ mixing and CP violation such as the first observation ofD⁰-D¯⁰mixing[11]. One possible mode for this, D⁰→K^þπ⁻η, will be reconstructed at Belle II, which aims at a data set fifty times [12] larger than that currently available from Belle. A time-dependent Dalitz analysis of this mode can be used to measure charm-mixing parameters, and for such a measurement an amplitude analysis of the right-sign decay,D⁰→K⁻π^þη, is needed to obtain the CF decay model.

This paper is organized as follows. Section II briefly describes the Belle detector and data samples, and Sec.III discusses event selection and parametrizations of signal and background and presents the measurement of the overall branching fraction. In Sec.IV, we report the results of the DP analysis. The evaluation of the systematic uncertainties are discussed in Sec.V. Further study and discussion of the Dalitz fit results are presented in Sec. VI. Finally, the conclusions are presented in Sec.VII.

II. BELLE DETECTOR AND DATASETS We perform a first Dalitz analysis of the decaysD⁰→ K⁻π^þη[13]using953fb⁻¹of data collected at or near the ϒðnSÞ resonances (n¼1, 2, 3, 4, 5), where 74% of the sample is taken at theϒð4SÞpeak, with the Belle detector [14] operating at the KEKB asymmetric-energy e^þe⁻ collider [15]. The Belle detector is a large-solid-angle magnetic spectrometer that consists of a silicon vertex detector (SVD), a 50-layer central drift chamber (CDC), an array of aerogel threshold Cherenkov counters (ACC), a barrel-like arrangement of time-of-flight scintillation coun- ters (TOF), and an electromagnetic calorimeter comprised of CsI(Tl) crystals located inside a superconducting solenoid coil that provides a 1.5 T magnetic field. An iron flux-return located outside of the coil is instrumented to detect K⁰_L mesons and to identify muons. A detailed description of the Belle detector can be found elsewhere[14].

III. EVENT SELECTION AND YIELDS The signal decay chain consists of D^þ→D⁰π^þs with D⁰→K⁻π^þη and η→γγ; D mesons are produced in e^þe⁻→cc¯ processes, and the charge of the slow pionπs

tags the flavor of the D⁰ meson [16]. To ensure charged

tracks are well reconstructed, each is required to have at least two associated hits of the SVD in the beam and azimuthal directions, separately. The slow-pion candidates are required to have the signed distances from the pivotal point to the helix to be within1.0cm in the transverse plane and within3.0cm along the direction opposite to the positron beam. A charged track is identified as a kaon by requiring a ratio of particle identification likelihoods [17] LK=ðLKþLπÞ>0.7, which are constructed using CDC, TOF, and ACC information; otherwise the track is assumed to be a pion. This requirement has efficiencies of 85% and 98% and misidentification rates of 2% and 10%

for kaons and pions, respectively. The photon candidates are reconstructed from ECL clusters unmatched to any charged track. The ratios of their energy deposits in a3×3 array of CsI(Tl) crystals to that in a 5×5 array centered on the crystal with maximum deposited energy are required to be more than 0.8. The energies of photon candidates used to form η, E_γ, must exceed 60 or 120 MeV in the barrel or endcap region. The η candidates must have γγ mass within ⁻⁰_þ0.05^.06 GeV=c² of the nominal mass[6]which takes into account the asymmetric resolution, and to have momentum in the laboratory frame, p_η, larger than 1GeV=c. Furthermore, we requirejcosθηj ¼ j_E^E_γ1^γ1−E_þE^γ2_γ2j·^E_p^η

η

to be less than 0.8, which is optimized to suppress combi- natorial background. A large set of simulated signal Monte Carlo (MC) samples, more than 25 million events, is produced to study the efficiency. These are generated uniformly in phase space with theEVTGEN[18]andJETSET

[19]software packages, and the detector response is mod- eled by theGEANT3[20]. The final-state radiation effect is taken into account using thePHOTOS[21]package.

Kaon and pion tracks with opposite charge are required to form a common vertex (theD⁰decay position) with fit qualityχ²v. Theηcandidates are mass-constrained assuming that they are produced at this decay vertex, and the resultant ηmomentum is added to the Kπ system to obtain the D⁰ momentum. The invariant mass ofKπη,M, is required to satisfy the condition 1.80GeV=c²< M <1.92GeV=c². Then, theD⁰production vertex is constrained to thee^þe⁻ interaction point, with fit qualityχ²_b. Theπstrack is refit to thisD⁰production vertex, with a fit quality denotedχ²s, to improve the resolution of the released energy inD^þdecay, Q≡M_Kπηπs −M_Kπη−m_πs. The value of Qis required to be less than15MeV=c²to suppress further combinatorial background. The D momentum in the center-of-mass frame,pðDÞ, is required to be greater than 2.4, 2.5, or 3.1GeV=cfor data below, on, or aboveϒð4SÞenergy, to reduce high-multiplicity events and combinatorial back- ground. A consequence of this requirement is that theD⁰ candidates fromBdecays are removed. After applying all of these selection criteria, there are on average 1.3 signal decay candidates per event. A best-candidate selection (BCS) method is applied to multicandidate events,

retaining as the best candidate the one with the smallest sum of vertex-fit qualities, χ²vþχ²_bþχ²s. A mass-con- strained fit is then applied to the D⁰ meson to improve resolution on the Dalitz variables, M²_Kπ andM²_πη.

To extract yields of signal and background, a fit of the two-dimensional distribution ofMandQis performed. For the signal, the probability density function (PDF) in Mis described by the sum of a double Gaussian and a double bifurcated Gaussian, with a common mean value (μ); the PDF inQis described by the sum of a bifurcated Student function, a bifurcated Gaussian function, and a bifurcated Cruijff function[22], where the mean values and widths are correlated to the M value by a second-order polynomial function ofjM−μj. For a real signalD⁰combined with a random πs (named the random πs background), the M distribution uses the same PDF as for the signal and theQ distribution uses a threshold function, fðQÞ ¼Q^αe^−βQ. This random background will be treated as signal, as it nearly consists of the sameD⁰decay as the signal when the tiny fraction of DCS decay relative to CF decay is neglected. The combinatorial background is considered to have two components. A PDF smoothed by bilinear interpolation [23] is used for correlated combinatorial background, which has a correctly reconstructed πs from D decay, but incorrectly reconstructed D⁰, whereas for other combinatorial background a third-order polynomial function of Mand a threshold function of Qis used as a parameterization. The ratio between these two combinato- rial backgrounds is fixed to that found using the generic MC. Figure 1 shows the M-Q combined fit for the experimental data. We obtain a signal yield of 105197 990in theMandQtwo-dimensional (2D) signal region of 1.85GeV=c²< M <1.88GeV=c² and 5.35MeV=c²<

Q <6.35MeV=c² with a high purity ð94.60.9Þ%. These are the combinations that will be used for the fit to the Dalitz plot.

To measure the branching fraction of the decay D⁰→K⁻π^þη, we normalize the signal yield by the number of D⁰ mesons produced in the decay D^þ→D⁰π^þ. For normalization, we choose theD⁰→K⁻π^þ channel, which has a well-known rate of B¼ ð3.9500.031Þ% [6]. We use the same selection criteria as are usedD⁰→K⁻π^þηbut without the η. We extract the signal yield from the distribution ofD⁰invariant mass in 1.78GeV=c²< M <

1.94GeV=c² and Q wide signal region jQ−5.85j<

1.0MeV=c² and find signal yields of 116302510 for D⁰→K⁻π^þη, and25973431669forD⁰→K⁻π^þ(with a high purity 98.3%) based on theϒð4SÞon-resonance data set. The efficiencyϵðD⁰→K⁻π^þηÞ ¼ ð5.340.01Þ%and ϵðD⁰→K⁻π^þÞ ¼ ð23.490.02Þ%are determined based on Dalitz signal MC produced with the nominal Dalitz fit result shown in TableIforD⁰→K⁻π^þηand signal MC for D⁰→K⁻π^þ. Taking into account the branching fraction Bðη→γγÞ ¼ ð39.410.20Þ% [6], we find the ratio of branching fractions to be

BðD⁰→K⁻π^þηÞ

BðD⁰→K⁻π^þÞ ¼0.5000.002ðstatÞ 0.020ðsystÞ 0.003ðBPDGÞ; ð1Þ where the three uncertainties shown are statistical, system- atic, and the uncertainty of branching fraction of η→γγ, respectively. Using the known D⁰→K⁻π^þ branching fraction, we measure the branching fraction

BðD⁰→K⁻π^þηÞ ¼ ð1.9730.009ðstatÞ 0.079ðsystÞ 0.018ðBPDGÞÞ%; ð2Þ where the last error is associated with uncertainty of the branching fractions of D⁰→K⁻π^þ and η→γγ. Many systematic uncertainties are canceled in the ratio

2) M (GeV/c

1.8 1.82 1.84 1.86 1.88 1.9 1.92

2 Events / 0.002 GeV/c

103

104

2) Q (MeV/c

0 2 4 6 8 10 12 14

2 Events / 0.2 MeV/c

103

104

Exp. data Signal

πs

Random Combinatorial

FIG. 1. TheD⁰→K⁻π^þηreconstructed mass,M (in5.35GeV=c²< Q <6.35GeV=c²) and release energy of D decay,Q(in 1.85GeV=c²< M <1.88GeV=c²) for experimental data (points with error bars) and fitted contributions of signal, randomπs and combinatorial backgrounds.

measurement, and the dominant uncertainty is that of theη reconstruction efficiency (4%).

IV. DALITZ ANALYSIS

The isobar model [24] is applied for the amplitude of D⁰→ðR→ABÞCthrough a resonanceRwith spin-J(A,B andCare pseudoscalar particles). The decay amplitude is given by a coherent sum of individual contributions, consisting of a constant terma_NRe^iϕNR for the nonresonant three-body decay, and different quasi-two-body resonant decays:

M¼a_NRe^iϕNRþX

R

a_Re^iϕRMRðm²_AB; m²_BCÞ: ð3Þ Here m²_AB and m²_BC are Dalitz variables, and a_Re^iϕR is a complex amplitude for the contribution of an individual intermediate resonance R. The amplitude and phase of K¯ð892Þ⁰, having the largest fit fraction, are fixed to a_Kð892Þ⁰ ¼1 and ϕKð892Þ⁰¼0. The matrix element MR

for an intermediate resonant decay is given by

MðABCjRÞ ¼F_D×F_R×T_R×ΩJ; ð4Þ whereT_R×ΩJis a resonance propagator.T_Ris a dynami- cal function for a resonance, described by a relativistic Breit-Wigner (RBW) with mass-dependent width,

T_R ¼ 1

M²_R−m²_AB−iM_RΓAB

; ΓAB ¼Γ^R₀

p_AB p_R

_2Jþ1

M_R m_AB

F²_R; ð5Þ wherep_AB(p_R) is the momentum of either daughter in the AB(orR) rest frame, andM_RandΓ^R₀ are the nominal mass and width, ΩJ describes the angular momentum that

depends on the spin J by using the Zemach tensor [25,26], and F_D and F_R are Blatt-Weisskopf centrifugal barrier factors [27,28], describing the quark structure of theD⁰ meson and intermediate resonance. The param- eter of meson radius, R, is set to 5.0ðGeV=cÞ⁻¹ and 1.5ðGeV=cÞ⁻¹ for the D⁰ meson and the intermediate resonances, respectively[26]. For thea₀ð980Þcontribution description, we use the Flatt´e formalism with three coupled channels,πη, K¯⁰K andπη⁰ [10]

T_RðsÞ ¼ 1

m²_a0−s−iðg²_πηρ_πηþg²_K¯0KρK¯⁰Kþg²_πη0ρ_πη⁰Þ; ð6Þ

where ffiffiffi ps

is the invariant mass ofπη;g_iandρiare coupling constants and phase-space factors, respectively. For exam- pleρ_πη¼ ffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffi

½1−ðm_πþm_ηÞ²=M²_πη½1−ðm_π−m_ηÞ²=M²_πη q

. The generalized LASS model [29,30] is used to para- metrize theKπ andKη S-wave contributions:

AgLASSðsÞ ¼ ffiffiffis p

2q·½BsinðδBþϕBÞe^{iðδBþϕBÞ}

þsinðδRÞe^{iðδRþϕRÞ}e^{2iðδBþϕBÞ}; ð7Þ

where s is the invariant mass squared of the Kπ or Kη system, q is the momentum of K in the Kπ or Kη rest frame, andδB andδR are phase angles of the nonresonant component and K₀ð1430Þ component, respectively.

They are defined as tanðδRÞ ¼M_rΓðmabÞ=ðM²r−m²_abÞand cotðδBÞ ¼1=ðaqÞ þrq=2, where a, r, B, ϕB andϕR are real parameters and may be determined by amplitude analysis.

The DP fit is performed by an unbinned maximum likelihood method with

lnL¼Xⁿ

i¼1

ln½fⁱ_sðM_i; Q_iÞ·P_sðm²_Kπ;i; m²_πη;iÞ

þ ð1−fⁱ_sðMi; Q_iÞÞ·P_bðm²_Kπ;i; m²_πη;iÞ; ð8Þ

wherenis the number ofD⁰candidates in theMandQ2D signal region andfⁱ_sis the event-by-event fraction of signal obtained from theM-Q fit; the combinatorial background function,P_b, is a smoothed PDF[23], determined from the DP in the M sideband region (1.755GeV=c²< M <

1.775GeV=c² or 1.935GeV=c²<M <1.955GeV=c²) and theQsignal region (5.35MeV=c²< Q <6.35MeV=c²).

The signal PDF,P_s, is calculated taking the reconstruction- efficiency dependence on the Dalitz-plot variables into account, and normalized in the Dalitz plot region.

TABLE I. Magnitude and phase of intermediate components, and their fit fraction from Dalitz-plot fit of D⁰→K⁻π^þη. The quoted uncertainties on the fit fractions are statistical, systematic, and the uncertainty due to the Dalitz model, respectively.

Component Magnitude Phase (°) Fit fraction (%) K¯ð892Þ⁰ 1 0 47.611.32^þ0.24þ3.64_{−0.49−2.71} a₀ð980Þ^þ 2.7790.032 310.31.1 39.281.50^þ1₋₀.^.51−3^58þ4.^.30³⁸

KπS-wave 10.820.23 50.05.7 31.921.21^þ1.47þ2.75_{−0.53−2.87} KηS-wave 1.700.082 113.813.6 3.370.50^þ0_{−0.27−1.21}^.77þ3.20 a₂ð1320Þ^þ 1.270.079 283.44.7 0.740.09^þ0.06þ0.37₋₀.04−0.17

K¯ð1410Þ⁰ 4.840.36 352.72.8 6.940.85^þ0.55þ2.37_{−1.61−3.22} Kð1680Þ⁻ 2.560.18 232.26.6 1.070.16^þ0_{−0.10−0.36}^.11þ0.58 K₂ð1980Þ⁻ 9.290.69 207.74.0 1.130.15^þ0.05þ0.88_{−0.05−0.98} Sum 132.13.4^þ1₋₀.^.7−4^6þ8.^.5³

P_s¼ jMðm²_Kπ; m²_πηÞj²ϵðm²_Kπ; m²_πηÞ

ZZ

dm²_Kπdm²_πηjMðm²_Kπ; m²_πηÞj²ϵðm²_Kπ; m²_πηÞ: ð9Þ

This efficiency distribution ϵðm²_Kπ;i; m²_πη;iÞ is obtained from a high-statistic signal-MC sample and takes into account the known difference in particle identification efficiency for charged tracks between MC and data.

These correction factors depend on the momentum and polar angle of individual charged track. The fit fractions (FF) of each intermediate component are calculated across the DP region as

FF¼ RR

DPja_Re^iϕRMRðm²_Kπ; m²_πηÞj²dm²_Kπdm²_πη RR

DPjMðm²_Kπ; m²_πηÞj²dm²_Kπdm²_πη : ð10Þ The FF uncertainties are evaluated using a Toy MC method in which the sampling takes into account the considerations among all the fitted parameters by propagating the full covariance matrix obtained by the DP fit.

Fifteen possible intermediate resonances [31] were initially considered in the Dalitz analysis. We found K¯ð1410Þ⁰ and K¯ð1680Þ⁰ have a phase-angle difference

of approximately 180° and similar behavior in the DP, therefore, it is hard to separate them. In order to ensure stability of the fit to the Dalitz plot, onlyK¯ð1410Þ⁰is kept, while a possible K¯ð1680Þ⁰ contribution is considered as a source of systematic uncertainty. Therefore for the rest of this paper, Kð1410Þ⁰ represents the contribution ofKð1410Þ⁰, Kð1680Þ⁰ and their possible interference.

Then, the resonances not contributing to the amplitude significantly are eliminated one by one based on signifi- cance-level testing. Significances of individual contribu- tions are determined as the likelihood difference, Δð−2lnLÞ, that arises when an individual contribution is removed from the model taking into account the degrees of freedom (d.o.f). Only components with significances in excess of5σ, i.e., Δð−2lnLÞ>28.74withΔðd:o:fÞ ¼2, are retained in the Dalitz model. Of the resonances which were eliminated, K₂ð1430Þ⁻ had the largest significance (3.8σ). A model with eight components is chosen as our nominal model, and this is presented in Fig.2. It includes six resonances [a₀ð980Þ^þ, a₂ð1320Þ^þ, K¯ð892Þ⁰, K¯ð1410Þ⁰, Kð1680Þ⁻,K₂ð1980Þ⁻] and two S-wave components (Kπ andKη). The fit quality of this nominal model isχ²=d:o:f ¼ 1638=ð1415−24Þ ¼1.18across the Dalitz plane, and the three Dalitz plot projections are shown in Fig.2(b–d). The statistical significance of each component is larger than10σ.

0 10 20 30 40 50 60

4)

2/c (GeV

π K

m2

0.4 0.6 0.8 1 1.2 1.4 1.6 1.8

)4 /c2 (GeVηπ2m

0.4 0.6 0.8 1 1.2 1.4 1.6 1.8 (a)

0 2 4 6 8 10 12 14 16 18 20 22

4)

2/c (GeV

π K

m2

0.4 0.6 0.8 1 1.2 1.4 1.6 1.8

)4 /c2 (GeVηπ2m

0.4 0.6 0.8 1 1.2 1.4 1.6 1.8 (b)

4)

2/c (GeV

π K

m2

0.4 0.6 0.8 1 1.2 1.4 1.6 1.8

4/c2Events / 0.030 GeV

0 2 4 6 8 10 12 14 16

103

×

Exp. data Total fit Combinatorial

*(892)0

K (980)+

a0

(1320)+

a2

*(1410)0

K K*(1680)-

*(1980)-

K2

S-wave π K

S-wave η K

(c)

4)

2/c (GeV

η π2

m

0.4 0.6 0.8 1 1.2 1.4 1.6 1.8

4/c2Events / 0.031 GeV

0 0.5 1 1.5

2 2.5 3 3.5 4 4.5

103

×

(d)

4)

2/c (GeV

η K

m2

1 1.2 1.4 1.6 1.8 2 2.2 2.4 2.6 2.8 3 4/c2 Events / 0.042 GeV

0 1 2 3 4 5

103

×

(e)

FIG. 2. The Dalitz plot ofD⁰→K⁻π^þηin (a)M-Qsignal region and (b)Msideband region, and projections on (c)m²_Kπ, (d)m²_πηand (e) m²_Kη. In projections the fitted contributions of individual components are shown, along with contribution of combinatorial background (grey-filled) from sideband region.

In particular, the statistical significance of the KηS-wave component with K₀ð1430Þ⁻ is greater than 30σ, and Kð1680Þ⁻→K⁻η and K₂ð1980Þ⁻ →K⁻η are observed for the first time and have statistical significances of16σand 17σ, respectively. The fitted magnitudes and phases of intermediate components are listed in TableI, together with corresponding fit fractions, where statistical uncertainties are obtained from 500 sets of toy MC samples, and systematic uncertainties take into account model uncertain- ties and other systematic uncertainties as discussed in Sec.V.

The fact that the sum of fit fractions is greater than 100%

indicates significant destructive interference. TableIIshows the fitted parameters of LASS model in Eq. (7)and their correlation coefficient matrix for the Kπ and Kη S-wave components. The left coefficients in the full correlation matrix from Dalitz fit are shown in TableIIIincluding the correlation coefficients among the magnitudes and phases of resonances and the LASS model parameters. Various Dalitz models, including the nominal model used in the fit to final experimental data, are produced using MC to perform tests for any possible bias, and to check that the input and output Dalitz parameters are consistent. We also checked for the existence of possible multiple solutions in the fit, with likelihood scanning of each of the free parameters. In addition, 100 sets of Dalitz fits were performed by sampling the initial values of free parameters uniformly in an interval around their final values. No multiple solutions were found.

To investigate the parameters of the Flatt´e formulation of the a₀ð980Þ^þ line shape, the Dalitz fit based on the nominal model with free g_πη is also performed and this yieldsg_πη¼0.5960.008ðstatÞGeV=c². This value is consistent with the measurement of BESIII, 0.607 0.011GeV=c²[10]. The significance of the πη⁰ contribu- tion is tested and the results with floated g_πη0 and fixed g_πη⁰ ¼0giveΔð−2lnLÞ ¼102withΔðd:o:fÞ ¼1, which indicates a πη⁰ contribution with 10.1σ statistical

significance. The fitted g_πη0¼0.4080.018ðstatÞGeV=c² is also consistent with the BESIII measurement of g_πη0 ¼0.4240.050GeV=c²[10].

V. SYSTEMATIC UNCERTAINTIES

The systematic uncertainties are divided into the uncer- tainties arising from the Dalitz model used in the fit and those from other sources. The model systematic uncertain- ties arise from the choice of individual components in the model, and from the parametrization of intermediate structures. The effective barrier radial parameter, R, is varied between 0 and 3.5ðGeV=cÞ⁻¹ for intermediate resonances, where the maximum value is chosen as the measuredRvalue for the narrowest resonance, theKð892Þ (R¼3.00.5ðGeV=cÞ⁻¹[6]), increased by its statistical error. Three coupling factors of the Flatt´e function are varied within the quoted uncertainties, and the largest difference with respect to the nominal model is assigned as the systematic uncertainty due to this source. The masses and widths of intermediate resonances are varied within their uncertainties [6]. To account for the Kπ and Kη S-wave components, the model used in the fit is modified by adding a wide resonanceκdescribed by a complex pole function[32]for aKπ S-wave, andK₀ð1950Þ⁻ described by RBW for aKη S-wave. The nonsignificant resonance a₀ð1450Þ^þ is added to evaluate theπηS-wave component uncertainty. We also use aK¯ð1680Þ⁰resonance instead of aK¯ð1410Þ⁰contribution.

The systematic uncertainty due to the Dalitz distribution of combinatorial background is evaluated by (1) varying theMsideband region within a shift of5MeV=c², and by (2) correcting the Dalitz distribution of experimental data in the M sideband by the ratio of combinatorial background in the M signal and sideband regions from generic MC. The larger difference is assigned as the TABLE II. Fitted parameters of the LASS model (with statistical uncertainties only) and their correlation coefficient matrix forKπand KηS-wave components.

Parameters Fitted values Correlation coefficient

B^Kπ ϕ^KπB ϕ^KπR a^Kπ r^Kπ B^Kη ϕ^KηB ϕ^KηR a^Kη r^Kη B^Kπ 0.2390.010 1

ϕ^KB^π (°) −2.10.8 0.094 1

ϕ^KπR (°) −0.71.8 0.134 0.738 1

a^Kπ (GeV⁻¹c) 5.360.29 0.172 0.784 0.754 1

r^Kπ (GeV⁻¹c) −3.300.10 −0.385 0.484 0.409 0.452 1

B^Kη 0.6930.108 −0.021 0.309 0.351 0.278 −0.185 1

ϕ^KB^η (°) 1.33.4 −0.318 0.387 0.340 0.432 0.529 −0.338 1

ϕ^KηR (°) 25.59.1 −0.210 −0.746 −0.756 −0.804 −0.250 −0.199 −0.447 1

a^Kη (GeV⁻¹c) 0.2930.048 −0.373 −0.711 −0.696 −0.790 −0.214 −0.173 −0.509 0.784 1 r^Kη(GeV⁻¹c) −15.92.6 −0.381 −0.694 −0.675 −0.776 −0.218 −0.092 −0.528 0.774 0.995 1

TABLEIII.Thecorrelationcoefficientsamongtheresonantparameters:themagnitudes(mag.)andphases,andtheLASSmodelparametersforKπandKηS-wavecomponents fromDalitzfit. Parametersa0ð980Þþa2ð1320Þþ¯Kð1410Þ0Kð1680Þ−K 2ð1980Þ−ðKπÞS-waveðKηÞS-wave MagnitudesPhaseMagnitudesPhaseMagnitudesPhaseMagnitudesPhaseMagnitudesPhaseMagnitudesPhaseMagnitudesPhase a0ð980ÞþMagnitudes1 Phase−0.0111 a2ð1320ÞþMagnitudes−0.2500.1881 Phase−0.378−0.143−0.1141 ¯Kð1410Þ0Magnitudes−0.6990.130−0.0240.5111 Phase0.555−0.513−0.196−0.406−0.6661 Kð1680Þ−Magnitudes−0.5160.2440.2310.1940.596−0.2401 Phase0.455−0.495−0.045−0.237−0.8270.707−0.5721 K 2ð1980Þ−Magnitudes0.080−0.5850.0920.097−0.3450.469−0.3020.5661 Phase0.208−0.606−0.514−0.164−0.2320.62−0.2370.4410.3501 ðKπÞS-waveMagnitudes0.330−0.2180.095−0.483−0.5820.655−0.3270.5020.5460.2461 Phase−0.2220.6490.0660.1430.571−0.650.521−0.812−0.652−0.461−0.6091 ðKηÞS-waveMagnitudes0.493−0.503−0.189−0.326−0.6870.765−0.4720.8040.5010.5280.587−0.8391 Phase0.375−0.451−0.194−0.089−0.6150.515−0.6150.7900.4190.3760.315−0.8530.7681 LASSofKπ S-wave

B0.051−0.383−0.1370.170−0.1570.026−0.3500.3710.1170.345−0.375−0.1990.1710.488 ϕB0.210−0.5380.014−0.186−0.5440.607−0.4380.7290.6540.4010.611−0.8580.7310.687 ϕR0.149−0.475−0.011−0.207−0.4860.561−0.3730.6440.5500.3770.689−0.8420.7380.715 a0.302−0.6530.022−0.187−0.6220.677−0.5340.8120.7020.4820.637−0.9340.7770.739 r0.335−0.1760.159−0.258−0.4250.422−0.1840.3010.4610.0860.678−0.2850.2550.020 LASSofKη S-wave B−0.064−0.290−0.2010.2680.1120.065−0.0260.1560.1520.125−0.012−0.4430.4400.417 ϕB0.325−0.1790.163−0.599−0.5660.686−0.1060.4050.2860.2570.729−0.3450.3690.071 ϕR−0.4540.4410.1290.3290.716−0.7130.560−0.784−0.451−0.414−0.5840.872−0.786−0.890 a−0.3510.5280.0400.2580.712−0.6940.524−0.845−0.516−0.443−0.5230.830−0.736−0.779 r−0.3290.5170.0240.2620.702−0.6810.519−0.830−0.512−0.435−0.5200.807−0.692−0.752